Abstract
The main goal in task planning is to build a sequence of actions that takes an agent from an initial state to a goal state. In robotics, this is particularly difficult because actions usually have several possible results, and sensors are prone to produce measurements with error. Partially observable Markov decision processes (POMDPs) are commonly employed, thanks to their ability to model the uncertainty of actions that modify and monitor the state of a system. However, since solving a POMDP is computationally expensive, their usage becomes prohibitive for most robotic applications. In this article, we propose a task planning architecture for service robotics. In the context of service robot design, we present a scheme to encode knowledge about the robot and its environment, that promotes the modularity and reuse of information. Also, we introduce a new recursive definition of a POMDP that enables our architecture to autonomously build a hierarchy of POMDPs, so that it can be used to generate and execute plans that solve the task at hand. Experimental results show that, in comparison to baseline methods, by following a recursive hierarchical approach the architecture is able to significantly reduce the planning time, while maintaining (or even improving) the robustness under several scenarios that vary in uncertainty and size.
Similar content being viewed by others
Code Availability
References
Balai, E., Gelfond, M., Zhang, Y.: Towards answer set programming with sorts. In: International Conference on Logic Programming and Nonmonotonic Reasoning, pp 135–147. Springer (2013)
Bellman, R.E.: Dynamic programming (1957)
Chin, R.T., Harlow, C.A.: Automated visual inspection: a survey. IEEE Trans. Patt. Anal. Mach. Intell 6, 557–573 (1982)
Eversheim, W., Herrmann, P.: Recent trends in flexible automated manufacturing. J. Manuf. Syst. 1(2), 139–148 (1982)
Fikes, R.E., Nilsson, N.J.: Strips: a new approach to the application of theorem proving to problem solving. Artif. Intell. 2(3-4), 189–208 (1971)
Fine, S., Singer, Y., Tishby, N.: The hierarchical hidden markov model: analysis and applications. Mach. Learn. 32(1), 41–62 (1998)
Fox, M., Long, D.: Pddl2. 1: an extension to pddl for expressing temporal planning domains. J. Artif. Intell. Res. 20, 61–124 (2003)
Gelfond, M., Kahl, Y.: Knowledge Representation, Reasoning, and the Design of Intelligent Agents: The Answer-Set Programming Approach. Cambridge University Press, Cambridge (2014)
Hanheide, M., Göbelbecker, M., Horn, G.S., Pronobis, A., Sjöö, K., Aydemir, A., Jensfelt, P., Gretton, C., Dearden, R., Janicek, M., et al.: Robot task planning and explanation in open and uncertain worlds. Artif. Intell. 247, 119–150 (2017)
Ingrand, F., Ghallab, M.: Deliberation for autonomous robots: a survey. Artif. Intell. 247, 10–44 (2017)
Kaelbling, L.P., Littman, M.L., Cassandra, A. R.: Planning and acting in partially observable stochastic domains. Artif. Intell. 101(1-2), 99–134 (1998)
Kress-Gazit, H., Fainekos, G.E., Pappas, G.J.: Temporal-logic-based reactive mission and motion planning. IEEE Trans. Robot. 25(6), 1370–1381 (2009)
Kurniawati, H., Hsu, D., Lee, W. S.: Sarsop: efficient point-based pomdp planning by approximating optimally reachable belief spaces. In: Robotics: Science and Systems, vol. 2008. Zurich, Switzerland. (2008)
McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., Wilkins, D.: Pddl-the planning domain definition language (1998)
Miller, S.A., Harris, Z.A., Chong, E.K.: A pomdp framework for coordinated guidance of autonomous uavs for multitarget tracking. EURASIP J. Adv. Sig. Process. 2009(1), 724597 (2009)
Ng, A.Y., Russell, S.J.: Algorithms for inverse reinforcement learning. In: Proceedings of the Seventeenth International Conference on Machine Learning, pp 663–670 (2000)
Pajarinen, J., Kyrki, V.: Robotic manipulation of multiple objects as a pomdp. Artif. Intell. 247, 213–228 (2017)
Pandey, A.K., Gelin, R.: A mass-produced sociable humanoid robot: pepper: The first machine of its kind. IEEE Robot. Autom. Mag. 25(3), 40–48 (2018)
Papadimitriou, C.H., Tsitsiklis, J.N.: The complexity of markov decision processes. Math. Oper. Res. 12(3), 441–450 (1987)
Pineau, J., Gordon, G., Thrun, S., et al.: Point-based value iteration: an anytime algorithm for pomdps. In: IJCAI, vol. 3, pp 1025–1032 (2003)
Pineau, J., Roy, N., Thrun, S.: A hierarchical approach to pomdp planning and execution. In: Workshop on Hierarchy and Memory in Reinforcement Learning (ICML), vol. 65, p 51 (2001)
Pineau, J., Thrun, S.: An integrated approach to hierarchy and abstraction for pomdps (2002)
Png, S., Pineau, J.: Bayesian reinforcement learning for pomdp-based dialogue systems. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 2156–2159. IEEE (2011)
Poudel, D.B.: Coordinating hundreds of cooperative, autonomous robots in a warehouse. Jan 27, 1–13 (2013)
Puterman, M.L.: Markov Decision Processes.: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2014)
Shani, G., Pineau, J., Kaplow, R.: A survey of point-based pomdp solvers. Auton. Agent. Multi-Agent Syst. 27(1), 1–51 (2013)
Smith, T., Simmons, R.: Heuristic search value iteration for pomdps. In: Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pp 520–527 (2004)
Spaan, M.T., Vlassis, N.: Perseus: Randomized point-based value iteration for pomdps. J. Artif. Intell. Res. 24, 195–220 (2005)
Sridharan, M., Gelfond, M., Zhang, S., Wyatt, J.: Reba: a refinement-based architecture for knowledge representation and reasoning in robotics. J. Artif. Intell. Res. 65, 87–180 (2019)
Sridharan, M., Wyatt, J., Dearden, R.: Planning to see: a hierarchical approach to planning visual actions on a robot using pomdps. Artif. Intell. 174(11), 704–725 (2010)
Theocharous, G., Mahadevan, S.: Hierarchical Learning and Planning in Partially Observable Markov Decision Processes. PhD thesis, Michigan State University. Department of Computer Science & Engineering (2002)
Theocharous, G., Rohanimanesh, K., Maharlevan, S.: Learning hierarchical observable markov decision process models for robot navigation. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), vol. 1, pp 511–516. IEEE (2001)
Younes, H.L., Littman, M.L.: Ppddl1. 0:, An extension to pddl for expressing planning domains with probabilistic effects, vol. 2 (2004)
Zhang, S., Khandelwal, P., Stone, P.: Dynamically constructed (Po) Mdps for adaptive robot planning. In: AAAI, pp 3855–3863 (2017)
Zhang, S., Sridharan, M., Bao, F.S.: Asp+ Pomdp: integrating non-monotonic logic programming and probabilistic planning on robots. In: 2012 IEEE International Conference On Development and Learning and Epigenetic Robotics (ICDL), pp 1–7. IEEE (2012)
Zhang, S., Stone, P.: Integrated commonsense reasoning and probabilistic planning. In: Proceedings of the 5th Workshop on Planning and Robotics (PlanRob), pp 111–114 (2017)
Acknowledgements
Elizabeth Santiago deeply thanks to the postdoctoral scholarship and Sergio A. Serrano thanks the scholarship No. 489044, both granted by CONACYT, and also to the Robotics Laboratory of the National Institute of Astrophysics, Optical and Electronic which were important for the realization of this work.
Funding
This work was supported with postdoctoral and master scholarships provided by CONACYT.
Author information
Authors and Affiliations
Contributions
Sergio A. Serrano, Elizabeth Santiago, Jose Martinez-Carranza, Eduardo F. Morales and L. Enrique Sucar contributed to the study conception, design and implementation. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Competing Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Conflict of interests
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Serrano, S.A., Santiago, E., Martinez-Carranza, J. et al. Knowledge-Based Hierarchical POMDPs for Task Planning. J Intell Robot Syst 101, 82 (2021). https://doi.org/10.1007/s10846-021-01348-8
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-021-01348-8