As AI agents gain the ability to shop, negotiate and make decisions on behalf of users, Stanford researchers are warning that the systems create a new problem: The agent acting for you may have incentives to benefit someone else.
A policy brief published by the Stanford Institute for Human-Centered Artificial Intelligence (HAI) argues that developers and companies deploying certain AI agents should have a “duty of loyalty” requiring them to act in users’ interests and address conflicts that could influence an agent’s recommendations or actions.
The issue becomes more significant as AI shifts from providing information to taking action. An agent recommending a product, for example, could potentially be influenced by commercial relationships, commissions or the business interests of the company operating the system.
That creates risks familiar in industries such as finance, where advisers and brokers operate under rules governing conflicts of interest, but potentially at far greater scale as software agents begin making millions of decisions automatically.
The Stanford researchers – Ella Genasci Smith, Victor Y. Wu and Jennifer King – argue that conventional disclosure requirements may be inadequate because users cannot realistically evaluate every potential conflict influencing an autonomous system. Instead, they propose placing responsibility on developers and deployers to identify, mitigate or eliminate conflicts.
The proposal is particularly relevant for enterprises building agents capable of purchasing products, selecting vendors, negotiating transactions or otherwise acting with financial consequences. It raises a governance question companies may increasingly have to answer before deploying autonomous agents: Who is the agent actually working for?
Author
Get the latest insights about enterprise AI.
Subscribe to our newsletter. Thank you.
