Salesforce SVP on How to Keep AI Agents From Quietly Going Rogue

Salesforce is rolling out a new capability designed to help companies detect when their AI agents are quietly going off track – and intervene before the problem spreads.

The company is unveiling Agent Evals, a capability that continuously tests agents, large language models and Model Context Protocol servers after they have been deployed. It is a new feature of Agent Fabric, software Salesforce introduced last fall that lets companies manage their AI agents.

Unlike conventional software, AI agents continue to operate even when components under them change – say a switch to a cheaper AI model. Since much of the agent’s work happens autonomously, companies may not know whether or not an agent continues to perform at the same level, such as accurately summarizing the data it retrieved.

Agent Evals acts as quality control. When performance falls below thresholds set by a company, it can alert the user or trigger interventions, including taking an agent out of service. The feature, which works on agents from different vendors, enters beta in November and is expected to become generally available in January.

Agent Fabric is also adding API Evals, designed to detect changes in APIs that agents depend on, as well as spending controls that can stop requests when a budget limit is reached. API Evals are scheduled for general availability in January 2027.

As companies create more AI agents to tackle workflows, they have to learn to manage them even if they are connected to different systems and given varying degrees of authority to perform work.

“You’ve got AI agents running everywhere. How do you have confidence in what they’re doing?” Andrew Comstock, senior vice president and general manager of MuleSoft at Salesforce, told The AI Innovator.

He said Agent Evals is more than just about testing the output; it also manages the quality of the results over time, such as whether answers become degraded when the user switches AI models, or if the agent’s tone has shifted from, say, friendly to sarcastic as it mimics its users.

“Is it evaluating in the same way that it always has?” he said. “If I change the model, if I change the context, how do I make sure … I’m not getting a fundamentally different experience? How do I get that consistency?”

Why agents are harder to manage

The agents’ ability to reason is what makes them useful – and harder to manage.

Traditional software is largely deterministic: Give it the same input under the same conditions and it should produce the same result. Change a data type or an interface in a way the software doesn’t understand and it will often fail visibly.

An AI agent may do something different. It may adapt.

If an underlying service changes, for instance, the agent may interpret the new information and keep working instead of showing an error. The problem is that its behavior can gradually move away from what the company originally authorized it to do.

Comstock recalled a famous AI failure in the state of Washington. Callers to the Department of Licensing who requested Spanish on an automated line instead heard an AI voice speaking English in a Spanish accent, according to the Associated Press. (The government agency has since apologized.)

The outcome was “about the worst you could probably imagine,” Comstock said. “It would have been better if it just said, ‘I can’t speak Spanish, I’m sorry.’”

In a conventional system, choosing Spanish would result in failure. In Washington’s AI system, it adapted, to disastrous results. “No user sees (the AI drifting) because it’s happening behind the scenes,” he added.

From answering questions to taking action

The issue becomes more consequential as agents move beyond generating answers and begin acting inside corporate systems.

An employee-onboarding agent, for example, could contact a new hire by email or text message to determine where to ship a computer, check an address against information in a resume and escalate the request to a manager if the employee wants the computer shipped somewhere else.

That seemingly mundane process requires the agent to cross several systems and make several decisions.

It needs access to some HR information. It needs to read the resume. It needs permission to communicate with the employee. And it needs to recognize when a decision should be handed to a human.

The challenge is giving it enough authority to perform the task without handing it the keys to the HR department.

Comstock said that if an agent needs HR access solely to verify an address, “we really only want it to be able to check your address. We don’t want to give it your information on your payroll or other people’s payroll or other people’s addresses.”

Agent Fabric is designed to put rules around those actions. In the onboarding example, a company could expose a single HR-system function – an address lookup – through MCP and allow only the onboarding agent to use it, and only when a shipment is actually being prepared.

The system could block the lookup when those conditions aren’t met.

That is an important distinction as companies move toward more autonomous AI. Rather than simply determining whether someone can use an agent, businesses increasingly have to decide precisely what that agent can do on a person’s behalf.

Comstock said an agent shouldn’t necessarily inherit all of a user’s permissions. A worker who can read thousands of company emails, for example, may want an agent authorized to send one particular email – not given unrestricted access to the entire mailbox.

The multi-agent problem

Salesforce first introduced Agent Fabric in 2025 on the premise that enterprise AI would become a multi-vendor environment.

That prediction is increasingly visible. Agents are being built by business application providers, cloud hyperscalers, data platforms and internal development teams, leaving technology departments with systems from multiple vendors rather than one standardized AI stack.

Agent Fabric is intended to serve as a layer above those systems, providing an inventory of agents and tools, common governance policies and controls over what they can access and how they interact. It is separate from Salesforce’s Agentforce platform, which is for building agents, and it can manage agents built outside Salesforce.

Salesforce said Agent Fabric is handling more than 1.2 billion requests a day. The company also said that 96% of enterprises it surveyed use more than one agent platform.

Capita, the British business-process outsourcing company, has taken an inventory-first approach.

Comstock said Capita used Agent Fabric’s registry to identify AI assets running across environments including Microsoft, Google Cloud and Amazon Web Services. That gives the company a central inventory before it layers on governance, cost monitoring, observability and evaluations.

The approach speaks to a basic problem in large organizations: Before a company can govern its AI, it has to know what AI it has.

“So many people don’t even know … how big the problem is,” Comstock said.

Comstock acknowledged that Agent Fabric isn’t the only system attempting to govern enterprise AI. Cloud and software providers increasingly offer their own monitoring, security and agent-management tools. Salesforce argues that its advantage is Agent Fabric isn’t limited to its own agents and is designed to be vendor-agnostic.

Companies still have to become Salesforce customers to buy Agent Fabric, although Comstock said they don’t have to use other Salesforce products.

Agent Fabric is priced based on the agents, actions and tools that Salesforce manages rather than by user seats, Comstock said.