Embodiment is an important characteristic for all intelligent agents, while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment. In this work, we propose the Embodied Scene Description, which exploits the embodiment ability of the agent to find an optimal viewpoint in its environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is established to teach the intelligent agent to generate corresponding sensorimotor activities. The proposed framework is tested on both the AI2Thor dataset and a real-world robotic platform for different scene description tasks, demonstrating the effectiveness and scalability of the developed method. Also, a mobile application is developed, which can be used to assist visually-impaired people to better understand their surroundings.

体现是所有智能代理的一个重要特征，而现有的场景描述任务主要集中在被动地分析图像，场景的语义理解与代理与环境之间的交互分离。在这项工作中，我们提出了体现场景描述，它利用代理的体现能力在其环境中为场景描述任务找到最佳视点。建立了具有模仿学习和强化学习范式的学习框架，以教智能代理生成相​​应的感觉运动活动。所提出的框架在 AI2Thor 数据集和真实世界机器人平台上针对不同的场景描述任务进行了测试，证明了所开发方法的有效性和可扩展性。此外，还开发了一款移动应用程序，可用于帮助视障人士更好地了解周围环境。