Akamai Cloud Evangelist: The Case for Distributed AI Inference

As enterprises race to deploy AI agents into production, a new battleground is emerging far from the large AI training clusters that have dominated the industry’s attention over the past two years.

The challenge is inference — the process of running trained AI models to generate responses on new data in real time. As organizations increasingly embed AI into core business operations, there is a growing realization that the centralized cloud architectures that powered the generative AI boom may not be fast enough for business-critical applications.

That is the premise behind Akamai’s latest push into AI infrastructure.

“The internet that was designed for people … is now starting to be consumed by machines,” Akamai Cloud Evangelist Ari Weil said in an interview with The AI Innovator. “We’re calling it the agentic web.”

The company, best known for its content delivery network (CDN), argues that the same distributed architecture that helped websites deliver images and video globally can also distribute AI inference closer to users and applications. This reduces lag times – which is especially important for core, customer-facing and critical operations.

In March, Akamai became the first Nvidia partner to implement its AI Grid architecture to distribute AI inference workloads across edge, regional and core infrastructure instead of relying solely on centralized AI data centers. The goal is to speed up the response times of AI applications through intelligent workload orchestration based on factors such as latency, cost and available compute.

Inference needs rapid responses

Unlike AI training, which typically runs in centralized data centers over days or weeks, inference often requires responses in fractions of a second. That distinction becomes increasingly important for applications such as fraud detection, computer vision, autonomous systems, financial quoting engines and AI agents that interact continuously with enterprise software.

Speed is even more critical when dealing with AI agents instead of human users.

“If you look at all the different types of agents that are semi-autonomous or autonomous, they’re simplistic today but they’re not waiting,” Weil said. “They don’t want to see something spooling text and tokens because that is a poor use of time and compute power. … What we’re trying to optimize for in those cases is for when those agents need access to information as part of an agentic work stream.”

That means AI infrastructure increasingly needs to place compute resources physically closer to users and enterprise systems, Weil said.

Today, many enterprises are still centralizing their applications on hyperscalers such as AWS, Microsoft Azure and Google Cloud to simplify operations and reduce costs. Akamai argues AI applications need to be closer to users, much as companies distributed web content across networks in the early internet to reduce latency.

“It is early, but we’re already seeing them coming to us for a variety of reasons,” Weil said.

Weil does not argue those platforms are becoming obsolete. Rather, he contends they were largely optimized around centralized compute rather than geographically distributed inference.

Proximity key for inference workloads

Akamai’s ‘The State of AI Inference 2026’ survey appears to reflect that tension. The report found that 60% of practitioners consider proximity important for inference workloads, yet 46% still run those workloads from a single data center. Among organizations deploying business-critical AI applications, however, 77% said proximity becomes important, while reliance on a single data center fell 14%.

The findings suggest many organizations recognize the importance of low-latency inference but have not yet redesigned their infrastructure accordingly.

Exactly how much distributed inference enterprises ultimately require remains an open question.

Major cloud providers continue expanding regional infrastructure while investing heavily in custom AI hardware and networking. AWS, Microsoft and Google all operate globally distributed cloud regions, although Akamai argues its network reaches significantly deeper into internet infrastructure because of its CDN heritage.

Whether that advantage proves decisive will likely depend on the application.

Some AI workloads — including model training and long-running batch jobs — remain well suited for centralized infrastructure. Others, including interactive AI agents, security monitoring and real-time video analytics, may benefit more from compute positioned closer to users.

Weil pointed to computer vision systems that detect anomalies from security cameras, AI-assisted sports broadcasting and financial services applications generating instant insurance quotes as examples where milliseconds matter.

AI’s next phase resembles early internet

The infrastructure itself is evolving as well.

Historically, Akamai built thousands of relatively small edge locations optimized for content delivery. AI is leading the company to rethink that model by adding larger GPU deployments, expanding data center capacity and integrating storage, networking and security more tightly.

“Distributing content is what we did, and now we’re distributing intelligence by putting GPUs and CPUs, and in some cases VPUs (Vision Processing Units) across our network by orchestrating the connections between all of those, and then making sure that, based on the application, we are routing our customers’ traffic however it meets their SLAs (Service Level Agreements),” he added.

Security also becomes more complex as AI agents begin acting on behalf of users.

Rather than simply authenticating people, organizations increasingly must determine whether an autonomous agent is legitimate, acting within approved guardrails and accessing only authorized resources.

“A lot of the problems that businesses find themselves in is based on doing the same old thing and then trying to retrofit security to it,” Weil said. “There’s a wake-up call that has come with agents now where people are sufficiently worried that they truly understand they have to apply these foundational principles right from the onset.”

Governance is becoming a priority as AI agents are increasingly embedded into operations.

That shift is already underway, according to Akamai’s report. “Inference has moved from experimentation into live business operations. Success is no longer just about model quality. It’s about delivering responses fast enough, reliably enough, and consistently enough to support real customer experiences and real operational decisions.”