Interactions between an intelligent software agent (ISA) and a human user are ubiquitous in everyday situations such as access to information, entertainment, and purchases. In such interactions, the ISA mediates the user’s access to the content, or controls some other aspect of the user experience, and is not designed to be neutral about outcomes of user choices. Like human users, ISAs are driven by goals, make autonomous decisions, and can learn from experience. Using ideas from bounded rationality (and deploying concepts from artificial intelligence, behavioural economics, control theory, and game theory), we frame these interactions as instances of an ISA whose reward depends on actions performed by the user. Such agents benefit by steering the user’s behaviour towards outcomes that maximise the ISA’s utility, which may or may not be aligned with that of the user. Video games, news recommendation aggregation engines, and fitness trackers can all be instances of this general case. Our analysis facilitates distinguishing various subcases of interaction (i.e. deception, coercion, trading, and nudging), as well as second-order effects that might include the possibility for adaptive interfaces to induce behavioural addiction, and/or change in user belief. We present these types of interaction within a conceptual framework, and review current examples of persuasive technologies and the issues that arise from their use. We argue that the nature of the feedback commonly used by learning agents to update their models and subsequent decisions could steer the behaviour of human users away from what benefits them, and in a direction that can undermine autonomy and cause further disparity between actions and goals as exemplified by addictive and compulsive behaviour. We discuss some of the ethical, social and legal implications of this technology and argue that it can sometimes exploit and reinforce weaknesses in human beings.

中文翻译：

---

智能软件代理与人类用户交互分析

智能软件代理 (ISA) 和人类用户之间的交互在日常情况中无处不在，例如访问信息、娱乐和购买。在此类交互中，ISA 调解用户对内容的访问，或控制用户体验的某些其他方面，并且并非旨在对用户选择的结果保持中立。与人类用户一样，ISA 受目标驱动，做出自主决策，并且可以从经验中学习。使用有限理性的想法（并部署来自人工智能、行为经济学、控制理论和博弈论的概念），我们将这些交互构建为 ISA 的实例，其奖励取决于用户执行的操作。这些代理通过将用户的行为导向最大化 ISA 效用的结果而受益，这可能与用户的一致，也可能不一致。视频游戏、新闻推荐聚合引擎和健身追踪器都可以成为这种一般情况的实例。我们的分析有助于区分交互的各种子案例（即欺骗、胁迫、交易和轻推），以及可能包括自适应界面诱发行为成瘾和/或改变用户信念的可能性的二阶效应。我们在一个概念框架内介绍了这些类型的交互，并回顾了说服性技术的当前示例及其使用中出现的问题。我们认为，学习代理通常用于更新其模型和后续决策的反馈的性质可能会引导人类用户的行为偏离对他们有利的行为，并且朝着可能破坏自主性并导致行动和目标之间进一步差距的方向发展，例如成瘾和强迫行为。我们讨论了这项技术的一些伦理、社会和法律影响，并认为它有时会利用和加强人类的弱点。