Abstract
Software tools for generating digital sound often present users with high-dimensional, parametric interfaces, that may not facilitate exploration of diverse sound designs. In this article, we propose to investigate artificial agents using deep reinforcement learning to explore parameter spaces in partnership with users for sound design. We describe a series of user-centred studies to probe the creative benefits of these agents and adapting their design to exploration. Preliminary studies observing users’ exploration strategies with parametric interfaces and testing different agent exploration behaviours led to the design of a fully-functioning prototype, called Co-Explorer, that we evaluated in a workshop with professional sound designers. We found that the Co-Explorer enables a novel creative workflow centred on human–machine partnership, which has been positively received by practitioners. We also highlight varied user exploration behaviours throughout partnering with our system. Finally, we frame design guidelines for enabling such co-exploration workflow in creative digital applications.
- Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation. 265--283. Google ScholarDigital Library
- Pieter Abbeel and Andrew Y. Ng. 2004. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st International Conference on Machine Learning. ACM, 1. Google ScholarDigital Library
- Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine 35, 4 (2014), 105--120.Google ScholarDigital Library
- Saleema Amershi, Max Chickering, Steven M. Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. Modeltracker: Redesigning performance analysis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 337--346. Google ScholarDigital Library
- Saleema Amershi, James Fogarty, and Daniel Weld. 2012. Regroup: Interactive machine learning for on-demand group creation in social networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 21--30. Google ScholarDigital Library
- Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, and Blaine Christian. 2011. CueT: Human-guided fast and accurate network alarm triage. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 157--166. Google ScholarDigital Library
- Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Google ScholarDigital Library
- Kristina Andersen and Peter Knees. 2016. Conversations with expert users in music retrieval and research challenges for creative MIR. In Proceedings of the 17th International Society for Music Information Retrieval Conference. 122--128.Google Scholar
- Kumaripaba Athukorala, Alan Medlar, Antti Oulasvirta, Giulio Jacucci, and Dorota Glowacka. 2016. Beyond relevance: Adapting exploration/exploitation in information retrieval. In Proceedings of the 21st International Conference on Intelligent User Interfaces. ACM, 359--369. Google ScholarDigital Library
- Kumaripaba Athukorala, Alan Medlar, Antti Oulasvirta, Giulio Jacucci, and Dorota Glowacka. 2016. Beyond relevance: Adapting exploration/exploitation in information retrieval. In Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI ’16). ACM, New York, NY, 359--369. DOI:https://doi.org/10.1145/2856767.2856786 Google ScholarDigital Library
- Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. 2016. Unifying count-based exploration and intrinsic motivation. In Proceedings of the Advances in Neural Information Processing Systems. 1471--1479. Google ScholarDigital Library
- Mark Blythe, Kristina Andersen, Rachel Clarke, and Peter Wright. 2016. Anti-solutionist strategies: Seriously silly design fiction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4968--4978. Google ScholarDigital Library
- Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010. A Bayesian interactive optimization approach to procedural animation design. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 103--112. Google ScholarDigital Library
- Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540 (2016).Google Scholar
- Giuseppe Amato, Malte Behrmann, Frédéric Bimbot, Baptiste Caramiaux, Fabrizio Falchi, Ander Garcia, Joost Geurts, Jaume Gibert, Guillaume Gravier, Hadmut Holken, Hartmut Koenitz, Sylvain Lefebvre, Antoine Liutkus, Fabien Lotte, Andrew Perkis, Rafael Redondo, Enrico Turrin, Thierry Vieville, and Emmanuel Vincent. 2019. AI in the Media and Creative Industries. Doctoral dissertation. New European Media (NEM).Google Scholar
- Mark Cartwright, Bryan Pardo, and Josh Reiss. 2014. Mixploration: Rethinking the audio mixer interface. In Proceedings of the 19th International Conference on Intelligent User Interfaces. ACM, 365--370. Google ScholarDigital Library
- Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the creativity support index. ACM Transactions on Computer-Human Interaction 21, 4 (2014), 21. Google ScholarDigital Library
- John M. Chowning. 1973. The synthesis of complex audio spectra by means of frequency modulation. Journal of the Audio Engineering Society 21, 7 (1973), 526--534.Google Scholar
- Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems. 4299--4307. Google ScholarDigital Library
- Jacob W. Crandall, Mayada Oudah, Tennom, Fatimah Ishowo-Oloko, Sherief Abdallah, Jean-François Bonnefon, Manuel Cebrian, Azim Shariff, Michael A. Goodrich, and Iyad Rahwan. 2018. Cooperating with machines. Nature Communications 9, 1 (2018), 233.Google ScholarCross Ref
- Mihaly Csikszentmihalyi. 1997. Creativity: Flow and the Psychology of Discovery and Invention. Harper Perennial, New York.Google Scholar
- Nicholas Davis, Chih-PIn Hsiao, Kunwar Yashraj Singh, Lisa Li, and Brian Magerko. 2016. Empirically studying participatory sense-making in abstract drawing with a co-creative cognitive agent. In Proceedings of the 21st International Conference on Intelligent User Interfaces. ACM, 196--207. Google ScholarDigital Library
- Nicholas M. Davis, Yanna Popova, Ivan Sysoev, Chih-Pin Hsiao, Dingtian Zhang, and Brian Magerko. 2014. Building artistic computer colleagues with an enactive model of creativity. In Proceedings of the International Conference On Computational Creativity.Google Scholar
- Stefano Delle Monache, Davide Rocchesso, Frédéric Bevilacqua, Guillaume Lemaitre, Stefano Baldan, and Andrea Cera. 2018. Embodied sound design. International Journal of Human-Computer Studies 118, (2018), 47–59.Google Scholar
- Ruta Desai, Fraser Anderson, Justin Matejka, Stelian Coros, James McCann, George Fitzmaurice, and Tovi Grossman. 2019. Geppetto: Enabling semantic design of expressive robot behaviors. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 369. Google ScholarDigital Library
- Christoph Sebastian Deterding, Jonathan David Hook, Rebecca Fiebrink, Jeremy Gow, Memo Akten, Gillian Smith, Antonios Liapis, and Kate Compton. 2017. Mixed-initiative creative interfaces. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM. Google ScholarDigital Library
- Mark d’Inverno and Jon McCormack. 2015. Heroic versus collaborative AI for the arts. In Proceedings of the 24th International Conference on Artificial Intelligence. Google ScholarDigital Library
- Alan Dix. 2007. Designing for appropriation. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI… but not as we know it-Volume 2. British Computer Society, 27--30. Google ScholarDigital Library
- Kees Dorst and Nigel Cross. 2001. Creativity in the design process: Co-evolution of problem–solution. Design Studies 22, 5 (2001), 425--437.Google ScholarCross Ref
- Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX design innovation: Challenges for working with machine learning as a design material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 278--288. Google ScholarDigital Library
- Jerry Alan Fails and Dan R. Olsen Jr. 2003. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces. ACM, 39--45.Google Scholar
- Rebecca Fiebrink. 2019. Machine learning education for artists, musicians, and other creative practitioners. ACM Transactions on Computing Education 19, 4 (2019), 1--32. Google ScholarDigital Library
- Rebecca Fiebrink and Baptiste Caramiaux. 2016. The machine learning algorithm as creative musical tool. In Handbook of Algorithmic Music. Oxford University Press.Google Scholar
- Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, 147--156. DOI:https://doi.org/10.1145/1978942.1978965 Google ScholarDigital Library
- Rebecca Fiebrink, Daniel Trueman, N. Cameron Britt, Michelle Nagai, Konrad Kaczmarek, Michael Early, M. R. Daniel, Anne Hege, and Perry R. Cook. 2010. Toward understanding human-computer interaction in composing the instrument. In Proceedings of the International Computer Music Association.Google Scholar
- Tesca Fitzgerald, Ashok Goel, and Andrea Thomaz. 2017. Human-robot co-creativity: Task transfer on a spectrum of similarity. In Proceedings of the 8th International Conference on Computational Creativity.Google Scholar
- David B. Fogel. 2006. Evolutionary Computation: Toward a New Philosophy of Machine Intelligence. Vol. 1. John Wiley 8 Sons. Google ScholarDigital Library
- Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. 2018. Noisy networks for exploration. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Jules Francoise and Frederic Bevilacqua. 2018. Motion-sound mapping through interaction: An approach to user-centered design of auditory feedback using machine learning. ACM Transactions on Interactive Intelligent Systems 8, 2 (2018), 16. Google ScholarDigital Library
- Rémy Frenoy, Yann Soullard, Indira Thouvenin, and Olivier Gapenne. 2016. Adaptive training environment without prior knowledge: Modeling feedback selection as a multi-armed bandit problem. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization. ACM, 131--139. Google ScholarDigital Library
- Jérémie Garcia, Theophanis Tsandilas, Carlos Agon, and Wendy Mackay. 2012. Interactive paper substrates to support musical creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1825--1828. Google ScholarDigital Library
- Marco Gillies. 2019. Understanding the role of interactive machine learning in movement interaction design. ACM Transactions on Computer-Human Interaction 26, 1 (2019), 5. Google ScholarDigital Library
- Marco Gillies, Rebeca Fiebrink, Atau Tanaka, Jérémie Garcia, Frédéric Bevilacqua, Alexis Heloir, Fabrizio Nunnari, Wendy E. Mackay, Saleema Amershi, Bongshin Lee, Nicolas D'Alessandro, Joëlle Tilmanne, Todd Kulesza, and Baptiste Caramiaux. 2016. Human-centred machine learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 3558--3565. Google ScholarDigital Library
- Dorota Glowacka, Tuukka Ruotsalo, Ksenia Konuyshkova, Kumaripaba Athukorala, Samuel Kaski, and Giulio Jacucci. 2013. Directing exploratory search: Reinforcement learning from user interactions with keywords. In Proceedings of the 2013 International Conference on Intelligent User Interfaces (IUI’13). ACM, New York, NY, 117--128. DOI:https://doi.org/10.1145/2449396.2449413 Google ScholarDigital Library
- Yuval Hart, Avraham E. Mayo, Ruth Mayo, Liron Rozenkrantz, Avichai Tendler, Uri Alon, and Lior Noy. 2017. Creative foraging: An experimental paradigm for studying exploration and discovery. PloS One 12, 8 (2017), e0182133.Google ScholarCross Ref
- Xu He, Haipeng Chen, and Bo An. 2020. Learning behaviors with uncertain human feedback. arXiv preprint arXiv:2006.04201 (2020).Google Scholar
- Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 159--166. Google ScholarDigital Library
- Andy Hunt and Ross Kirk. 2000. Mapping strategies for musical performance. Trends in Gestural Control of Music 21, 2000 (2000), 231--258.Google Scholar
- Andy Hunt and Marcelo M. Wanderley. 2002. Mapping performer parameters to synthesis engines. Organised Sound 7, 2 (2002), 97--108. Google ScholarDigital Library
- Hilary Hutchinson, Benjamin B. Bederson, Allison Druin, Catherine Plaisant, Wendy Mackay, Helen Evans, Heiko Hansen, Stéphane Conversy, Michel Beaudouin-Lafon, Nicolas Roussel, Loïc Lacomme, Björn Eiderbäck, Sinna Lindquist, Yngve Sundblad, and Bosse Westerlund. 2003. Technology probes: Inspiring design for and with families. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 17--24. Google ScholarDigital Library
- Ian Jolliffe. 2011. Principal component analysis. In International Encyclopedia of Statistical Science. Springer, 1094--1096.Google Scholar
- Sergi Jorda. 2005. Digital Lutherie Crafting Musical Computers for New Musics’ Performance and Improvisation. Ph.D. Dissertation. Universitat Pompeu Fabra.Google Scholar
- Anna Kantosalo, Jukka M. Toivanen, Ping Xiao, and Hannu Toivonen. 2014. From isolation to involvement: Adapting machine creativity software to support human-computer co-creation. In Proceedings of the 5th International Conference on Computational Creativity. 1--7.Google Scholar
- Ashish Kapoor, Bongshin Lee, Desney Tan, and Eric Horvitz. 2010. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1343--1352. Google ScholarDigital Library
- Simon Katan, Mick Grierson, and Rebecca Fiebrink. 2015. Using interactive machine learning to support interface development through workshops with disabled people. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 251--254. Google ScholarDigital Library
- Andrea Kleinsmith and Marco Gillies. 2013. Customizing by doing for responsive video game characters. International Journal of Human-Computer Studies 71, 7–8 (2013), 775--784. Google ScholarDigital Library
- W. Bradley Knox and Peter Stone. 2009. Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the 5th International Conference on Knowledge Capture. ACM, 9--16. Google ScholarDigital Library
- Janin Koch. 2017. Design implications for designing with a collaborative AI. In Proceedings of the 2017 AAAI Spring Symposium.Google Scholar
- Janin Koch, Andrés Lucero, Lena Hegemann, and Antti Oulasvirta. 2019. May AI?: Design ideation with cooperative contextual bandits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 633. Google ScholarDigital Library
- Janin Koch and Antti Oulasvirta. 2018. Group cognition and collaborative AI. In Human and Machine Learning. Springer, 293--312.Google Scholar
- Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender systems. Computer 8 (2009), 30--37. Google ScholarDigital Library
- Yuki Koyama. 2016. Computational design driven by aesthetic preference. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 1--4. Google ScholarDigital Library
- Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, and Scott R. Klemmer. 2011. Bricolage: Example-based retargeting for web design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2197--2206. Google ScholarDigital Library
- Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In Proceedings of the Symposium of the Austrian HCI and Usability Engineering Group. Springer, 63--76. Google ScholarDigital Library
- Yuxi Li. 2018. Deep reinforcement learning. arXiv preprint arXiv:1810.06339 (2018).Google Scholar
- Changchun Liu, Pramila Agrawal, Nilanjan Sarkar, and Shuo Chen. 2009. Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. International Journal of Human-Computer Interaction 25, 6 (2009), 506--529.Google ScholarCross Ref
- Wanyu Liu, Rafael Lucas d’Oliveira, Michel Beaudouin-Lafon, and Olivier Rioul. 2017. Bignav: Bayesian information gain for guiding multiscale navigation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5869--5880. Google ScholarDigital Library
- J. Derek Lomas, Jodi Forlizzi, Nikhil Poonwala, Nirmal Patel, Sharan Shodhan, Kishan Patel, Ken Koedinger, and Emma Brunskill. 2016. Interface design optimization as a multi-armed bandit problem. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4142--4153. Google ScholarDigital Library
- Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579--2605.Google Scholar
- Wendy E. Mackay. 1990. Users and Customizable Software: A Co-adaptive Phenomenon. Ph.D. Dissertation. Citeseer.Google Scholar
- Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.Google Scholar
- Stefano Delle Monache, Pietro Polotti, and Davide Rocchesso. 2010. A toolkit for explorations in sonic interaction design. In Proceedings of the 5th Audio Mostly Conference: A Conference on Interaction with Sound. ACM, 1. Google ScholarDigital Library
- Yael Niv. 2009. Reinforcement learning in the brain. Journal of Mathematical Psychology 53, 3 (2009), 139--154.Google ScholarCross Ref
- Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. 2016. Deep exploration via bootstrapped DQN. In Proceedings of the Advances in Neural Information Processing Systems. 4026--4034. Google ScholarDigital Library
- François Pachet, Pierre Roy, Julian Moreira, and Mark d’Inverno. 2013. Reflexive loopers for solo musical improvisation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2205--2208. Google ScholarDigital Library
- Kayur Patel, Steven M. Drucker, James Fogarty, Ashish Kapoor, and Desney S. Tan. 2011. Using multiple models to understand data. In Proceedings of the International Joint Conference on Artificial Intelligence, Vol. 22. 1723. Google ScholarDigital Library
- Jonas Frich Pedersen, Michael Mose Biskjaer, and Peter Dalsgaard. 2018. Twenty years of creativity research in human-computer interaction: Current state and future directions. In Designing Interactive Systems. Association for Computing Machinery. Google ScholarDigital Library
- Claire Petitmengin. 2006. Describing one’s subjective experience in the second person: An interview method for the science of consciousness. Phenomenology and the Cognitive Sciences 5, 3–4 (2006), 229--269.Google ScholarCross Ref
- Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine (Bean). 2001. New interfaces for musical expression. In Proceedings of the CHI’01 Extended Abstracts on Human Factors in Computing Systems. ACM, 491--492. Google ScholarDigital Library
- Landy Rajaonarivo, Matthieu Courgeon, Eric Maisel, and Pierre De Loor. 2017. Inline co-evolution between users and information presentation for data exploration. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. ACM, 215--219. Google ScholarDigital Library
- Mitchel Resnick. 2007. All I really need to know (about creative thinking) I learned (by studying how children learn) in kindergarten. In Proceedings of the 6th ACM SIGCHI Conference on Creativity 8 Cognition. ACM, 1--6. Google ScholarDigital Library
- Mitchel Resnick, Brad Myers, Kumiyo Nakakoji, Ben Shneiderman, Randy Pausch, Ted Selker, and Mike Eisenberg. 2005. Design principles for tools to support creative thinking. Working Paper.Google Scholar
- Horst W. J. Rittel. 1972. On the Planning Crisis: Systems Analysis of the “First and Second Generations”. Institute of Urban and Regional Development.Google Scholar
- Tuukka Ruotsalo, Giulio Jacucci, Petri Myllymäki, and Samuel Kaski. 2014. Interactive intent modeling: Information discovery beyond search. Communications of the ACM 58, 1 (Dec. 2014), 86--92. DOI:https://doi.org/10.1145/2656334 Google ScholarDigital Library
- Diemo Schwarz and Norbert Schnell. 2009. Sound search by content-based navigation in large databases. In Proceedings of the Sound and Music Computing. 1--1.Google Scholar
- Hugo Scurto, Frédéric Bevilacqua, and Baptiste Caramiaux. 2018. Perceiving agent collaborative sonic exploration in interactive reinforcement learning. In Proceedings of the 15th Sound and Music Computing Conference (SMC’18).Google Scholar
- Hugo Scurto and Rebecca Fiebrink. 2016. Grab-and-play mapping: Creative machine learning approaches for musical inclusion and exploration. In Proceedings of the 2016 International Computer Music Conference.Google Scholar
- Burr Settles. 2010. Active learning literature survey. University of Wisconsin, Madison 52, 55–66 (2010), 11.Google ScholarDigital Library
- Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando De Freitas. 2016. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE 104, 1 (2016), 148--175.Google ScholarCross Ref
- Michael Shilman, Desney S. Tan, and Patrice Simard. 2006. CueTIP: A mixed-initiative interface for correcting handwriting errors. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology. 323--332. Google ScholarDigital Library
- Ben Shneiderman. 2007. Creativity support tools: Accelerating discovery and innovation. Communications of the ACM 50, 12 (2007), 20--32. Google ScholarDigital Library
- David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529, 7587 (2016), 484.Google Scholar
- Malcolm Strens. 2000. A Bayesian framework for reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning. 943--950. Google ScholarDigital Library
- Simone Stumpf, Vidya Rajaram, Lida Li, Margaret Burnett, Thomas Dietterich, Erin Sullivan, Russell Drummond, and Jonathan Herlocker. 2007. Toward harnessing user feedback for machine learning. In Proceedings of the 12th International Conference on Intelligent User Interfaces. ACM, 82--91. Google ScholarDigital Library
- Simone Stumpf, Vidya Rajaram, Lida Li, Weng-Keen Wong, Margaret Burnett, Thomas Dietterich, Erin Sullivan, and Jonathan Herlocker. 2009. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies 67, 8 (2009), 639--662. Google ScholarDigital Library
- Harini Suresh and John V. Guttag. 2019. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv:1901.10002 (2019).Google Scholar
- Richard S. Sutton and Andrew G. Barto. 2011. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA. Google ScholarDigital Library
- Andrea L. Thomaz and Cynthia Breazeal. 2008. Teachable robots: Understanding human teaching behavior to build more effective robot learners. Artificial Intelligence 172, 6–7 (2008), 716--737. Google ScholarDigital Library
- Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. 2018. Deep TAMER: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the Association for the Advancement of Artificial Intelligence.Google Scholar
- Christopher John Cornish Hellaby Watkins. 1989. Learning From Delayed Rewards. Ph.D. Thesis, Cambridge.Google Scholar
- Geraint A. Wiggins. 2006. A preliminary framework for description, analysis and comparison of creative systems. Knowledge-Based Systems 19, 7 (2006), 449--458. Google ScholarDigital Library
- Weng-Keen Wong, Ian Oberst, Shubhomoy Das, Travis Moore, Simone Stumpf, Kevin McIntosh, and Margaret Burnett. 2011. End-user feature labeling: A locally-weighted regression approach. In Proceedings of the 16th International Conference on Intelligent User Interfaces. 115--124. Google ScholarDigital Library
- Matthew Wright. 2005. Open sound control: An enabling technology for musical networking. Organised Sound 10, 3 (2005), 193--200. Google ScholarDigital Library
- Qian Yang, Nikola Banovic, and John Zimmerman. 2018. Mapping machine learning advances from HCI research to reveal starting places for design innovation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 130. Google ScholarDigital Library
- Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating how experienced UX designers effectively work with machine learning. In Proceedings of the 2018 Designing Interactive Systems Conference. ACM, 585--596. Google ScholarDigital Library
- Georgios N. Yannakakis, Antonios Liapis, and Constantine Alexopoulos. 2014. Mixed-initiative co-creativity. In Proceedings of the 9th International Conference on the Foundations of Digital Games.Google Scholar
- Mehmet Ersin Yumer, Siddhartha Chaudhuri, Jessica K. Hodgins, and Levent Burak Kara. 2015. Semantic shape editing using deformation handles. ACM Transactions on Graphics 34, 4 (2015), 86. Google ScholarDigital Library
- Bruno Zamborlin, Frederic Bevilacqua, Marco Gillies, and Mark D’inverno. 2014. Fluid gesture interaction design: Applications of continuous recognition for the design of modern gestural interfaces. ACM Transactions on Interactive Intelligent Systems 3, 4 (2014), 22. Google ScholarDigital Library
- Xiang Sean Zhou and Thomas S. Huang. 2003. Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems 8, 6 (2003), 536--544.Google ScholarCross Ref
Index Terms
- Designing Deep Reinforcement Learning for Human Parameter Exploration
Recommendations
Exploration in deep reinforcement learning: A survey
AbstractThis paper reviews exploration techniques in deep reinforcement learning. Exploration techniques are of primary importance when solving sparse reward problems. In sparse reward problems, the reward is rare, which means that the agent ...
Highlights- Exploration in deep reinforcement learning is investigated comprehensively.
- New ...
Designing with Only Four People in Mind? --- A Case Study of Using Personas to Redesign a Work-Integrated Learning Support System
INTERACT '09: Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part IIIn this paper we describe and reflect on the use of personas to redesign the 3rd prototype of APOSDLE --- a system to support informal learning and knowledge transfer in the workplace. Based on the results of a formative evaluation of the 2nd prototype ...
Comments