Skip to main content
SpringerLink
Log in

iRoPro: An interactive Robot Programming Framework

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

The great diversity of end-user tasks ranging from manufacturing environments to personal homes makes pre-programming robots for general purpose applications extremely challenging. In fact, teaching robots new actions from scratch that can be reused for previously unseen tasks remains a difficult challenge and is generally left up to robotics experts. In this work, we present iRoPro, an interactive Robot Programming framework that allows end-users with little to no technical background to teach a robot new reusable actions. We combine Programming by Demonstration and Automated Planning techniques to allow the user to construct the robot’s knowledge base by teaching new actions by kinesthetic demonstration. The actions are generalised and reused with a task planner to solve previously unseen problems defined by the user. We implement iRoPro as an end-to-end system on a Baxter Research Robot to simultaneously teach low- and high-level actions by demonstration that the user can customise via a Graphical User Interface to adapt to their specific use case. To evaluate the feasibility of our approach, we first conducted pre-design experiments to better understand the user’s adoption of involved concepts and the proposed robot programming process. We compare results with post-design experiments, where we conducted a user study to validate the usability of our approach with real end-users. Overall, we showed that users with different programming levels and educational backgrounds can easily learn and use iRoPro and its robot programming process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://github.com/ysl208/iRoPro

  2. https://github.com/jstnhuang/rapid_pbd

  3. http://docs.ros.org/indigo/api/pddl_planner

  4. https://github.com/jstnhuang/surface_perception

  5. Video can be seen at https://youtu.be/YCDrC0UFX38

  6. All participants were different from the first experiment.

References

  1. Abdo N, Kretzschmar H, Spinello L, Stachniss C (2013) Learning manipulation actions from a few demonstrations. In: International conference on robotics and automation, pp 1268–1275. IEEE

  2. Ahmadzadeh SR, Paikan A, Mastrogiovanni F, Natale L, Kormushev P, Caldwell DG (2015) Learning symbolic representations of actions from human demonstrations. Int Conf Robot Autom 5:3801–3808

    Google Scholar 

  3. Akgun B, Cakmak M, Jiang K, Thomaz AL (2012) Keyframe-based learning from demonstration. Int J Social Robot 4(4):343–355

    Article  Google Scholar 

  4. Alexandrova S, Cakmak M, Hsiao K, Takayama L (2014) Robot programming by demonstration with interactive action visualizations. Robot Sci Syst 3:10

    Google Scholar 

  5. Argall BD, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robot Auton Syst 57(5):469–483

    Article  Google Scholar 

  6. Bangor A, Kortum PT, Miller JT (2008) An empirical evaluation of the system usability scale. Int J Hum Comput Interact 24(6):574–594

    Article  Google Scholar 

  7. Biggs G, Macdonald B (2003) A survey of robot programming systems. Austral Conf Robot Autom 3:1–3

    Google Scholar 

  8. Billard A, Calinon S, Dillmann R, Schaal S (2008) Robot programming by demonstration. In: Springer handbook of robotics, pp 1371–1394. Springer

  9. Brooke J (2013) SUS: a retrospective. J Usability Stud 8(2):29–40

    Google Scholar 

  10. Calinon S, Billard A (2007) Incremental learning of gestures by imitation in a humanoid robot. In: International conference on human-robot interaction, pp 255–262

  11. Calzado J, Lindsay A, Chen C, Samuels G, Olszewska JI (2018) Sami: interactive, multi-sense robot architecture. In: 2018 IEEE 22nd international conference on intelligent engineering systems (INES), pp 317–322

  12. Cashmore M, Fox M (2015) ROSPlan: planning in the robot operating system. In: International conference on automated planning and scheduling, pp 333–341

  13. Fikes RE, Nilsson NJ (1971) STRIPS: a new approach to the application of theorem proving to problem solving. Artif Intell 2(3–4):189–208

    Article  Google Scholar 

  14. Fraser N (2013) Blockly: a visual programming editor. https://developers.google.com/blockly/. Accessed 01 April 2019

  15. Freedman DH (2012) The Rise of the Robotic Work Force. https://www.inc.com/magazine/201210/david-h-freedman/the-rise-of-the-robotic-workforce.html. Accessed 11 April 2019

  16. Pignaton de Freitas E, Bermejo-Alonso J, Khamis A, Li H, Olszewska JI (2020) Ontologies for cloud robotics. Knowl Eng Rev 35:e25. https://doi.org/10.1017/S0269888920000338

    Article  Google Scholar 

  17. Gaisne A (2018) Baxter eyes. https://github.com/Anne-Gaisne/baxter_eyes. Accessed 11 April 2019

  18. Ghallab M, Nau D, Traverso P (2004) Automated planning: theory and practice. Elsevier, London

    MATH  Google Scholar 

  19. Gosavi A (2009) Reinforcement learning: a tutorial survey and recent advances. J Comput 21(2):178–192

    MathSciNet  MATH  Google Scholar 

  20. Hoffmann J, Nebel B (2001) The FF planning system: fast plan generation through heuristic search. J Artif Intell Res 14:253–302

    Article  Google Scholar 

  21. Jetchev N, Lang T, Toussaint M (2013) Learning grounded relational symbols from continuous data for abstract reasoning

  22. Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285

    Article  Google Scholar 

  23. Konidaris G, Kaelbling LP, Lozano-Perez T (2018) From skills to symbols: learning symbolic representations for abstract high-level planning. J Artif Intell Res 61:215–289

    Article  MathSciNet  Google Scholar 

  24. Kuhner D, Aldinger J, Burget F, Göbelbecker M, Burgard W, Nebel B (2018) Closed-loop robot task planning based on referring expressions. In: International conference on intelligent robots and systems, pp 876–881. IEEE

  25. Lego: Lego mindstorms. http://mindstorms.lego.com (2003). Accessed 12 March 2019

  26. Li C, Berenson D (2016) Learning object orientation constraints and guiding constraints for narrow passages from one demonstration. In: International symposium on experimental robotics, pp 197–210. Springer

  27. Liang YS, Pellier D, Fiorino H, Pesty S (2017) Evaluation of a robot programming framework for non-experts using symbolic planning representations. In: International symposium on robot and human interactive communication

  28. Liang YS, Pellier D, Fiorino H, Pesty S (2017) A framework for robot programming in cobotic environments: First user experiments. In: International conference on mechatronics and robotics engineering, pp 30–35. ACM

  29. Liang YS, Pellier D, Fiorino H, Pesty S (2019) End-user programming of low- and high-level actions for robotic task planning. In: International symposium on robot and human interactive communication

  30. Liang YS, Pellier D, Fiorino H, Pesty S, Cakmak M (2018) Simultaneous end-user programming of goals and actions for robotic shelf organization. In: 2018 international conference on intelligent robots and systems

  31. Majed M (2014) Learn to program with Scratch: a visual introduction to programming with games, art, science, and math. No Starch Press Inc

  32. Mandran N, Dupuy-Chessa S (2017) THEDRE: a traceable process for high quality in human centred computer science research. In: International conference of System Development

  33. Niekum S, Osentoski S, Konidaris G, Barto AG (2012) Learning and generalization of complex tasks from unstructured demonstrations. In: International conference on intelligent robots and systems, pp 5239–5246. IEEE

  34. Olszewska JI, Barreto M, Bermejo-Alonso J, Carbonera J, Chibani A, Fiorini S, Goncalves P, Habib M, Khamis A, Olivares A, de Freitas EP, Prestes E, Ragavan SV, Redfield S, Sanz R, Spencer B, Li H (2017) Ontology for autonomous robotics. In: 2017 26th IEEE international symposium on robot and human interactive communication (RO-MAN), pp 189–194

  35. Pais AL, Argall BD, Billard AG (2013) Assessing interaction dynamics in the context of robot programming by demonstration. Int J Social Robot 5(4):477–490

    Article  Google Scholar 

  36. Pastor P, Hoffmann H, Asfour T, Schaal S (2009) Learning and generalization of motor skills by learning from demonstration. In: International conference on robotics and automation, pp 763–768

  37. Paxton C, Hundt A, Jonathan F, Guerin K, Hager GD (2017) CoSTAR: instructing collaborative robots with behavior trees and vision. In: International conference on robotics and automation, pp 564–571

  38. Perzylo A, Somani N, Profanter S, Kessler I, Rickert M, Knoll A (2016) Intuitive instruction of industrial robots: semantic process descriptions for small lot production. In: International conference on intelligent robots and systems, pp 2293–2300

  39. Plch T, Chomut M, Brom C, Barták R (2012) Inspect, edit and debug PDDL documents: simply and efficiently with PDDL studio. In: International conference on automated planning and scheduling: system demonstration

  40. Quigley M, Faust J, Foote T, Leibs J (2009) ROS: an open-source Robot Operating System. In: ICRA workshop on open source software

  41. Sauro J (2012) Predicting net promoter scores from system usability scale scores. https://measuringu.com/nps-sus/. Accessed 16 April 2019

  42. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  43. She L, Cheng Y, Chai JY, Jia Y, Yang S, Xi N (2014) Teaching robots new actions through natural language instructions. In: International symposium on robot and human interactive communication, pp 868–873

  44. Simpson RM, Kitchin DE, McCluskey TL (2007) Planning domain definition using GIPO. Knowl Eng Rev 22(2):117–134

    Article  Google Scholar 

  45. Stenmark M, Haage M, Topp EA (2017) Simplified programming of re-usable skills on a safe industrial robot: Prototype and evaluation. In: International conference on human-robot interaction, pp 463–472. ACM

  46. Suay HB, Toris R, Chernova S (2012) A practical comparison of three robot learning from demonstration algorithm. Int J Social Robot 4(4):319–330

    Article  Google Scholar 

  47. Ugur E, Piater J (2015) Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning. In: International conference on robotics and automation, pp 2627–2633

  48. Vaquero TS, Silva JR, Tonidandel F, Beck JC (2013) itSIMPLE: towards an integrated design system for real planning applications. Knowl Eng Rev 28(2):215–230

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Nadine Mandran, research engineer at the Laboratoire d’Informatique de Grenoble, for her support and guidance during for the experimental studies.

Author information

Authors and Affiliations

  1. Université Grenoble Alpes, LIG, 38000, Grenoble, France

    Ying Siu Liang, Damien Pellier, Humbert Fiorino & Sylvie Pesty

Authors
  1. Ying Siu Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Damien Pellier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Humbert Fiorino
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sylvie Pesty
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Siu Liang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, Y.S., Pellier, D., Fiorino, H. et al. iRoPro: An interactive Robot Programming Framework. Int J of Soc Robotics 14, 177–191 (2022). https://doi.org/10.1007/s12369-021-00775-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-021-00775-9

Keywords

Search

Navigation