An artificial general intelligence (AGI) might have an instrumental drive to modify its utility function to improve its ability to cooperate, bargain, promise, threaten, and resist and engage in blackmail. Such an AGI would necessarily have a utility function that was at least partially observable and that was influenced by how other agents chose to interact with it. This instrumental drive would conflict with the strong orthogonality thesis since the modifications would be influenced by the AGI’s intelligence. AGIs in highly competitive environments might converge to having nearly the same utility function, one optimized to favorably influencing other agents through game theory. Nothing in our analysis weakens arguments concerning the risks of AGI.

中文翻译：

---

违反强正交论修改其效用函数的AGI

人工智能（AGI）可能会发挥工具作用，以修改其效用功能，以提高其协作，讨价还价，承诺，威胁，抵抗和参与勒索的能力。这样的AGI必须具有至少部分可观察到的效用功能，并且受其他代理选择与之交互的方式的影响。这种工具性的驱动将与强正交性论证相冲突，因为修改将受到AGI情报的影响。在竞争激烈的环境中，AGI可能会趋同于具有几乎相同的效用功能，该功能经过优化可以通过博弈论对其他主体产生有利影响。我们的分析并没有削弱有关AGI风险的论点。