AI agents are putting the CPU back on the critical path

An agent is a chain, not a single request

The familiar picture of AI infrastructure is simple: a prompt goes to a model on a GPU, the GPU produces an answer, and the server sends it back.

An agent makes that picture untidy. It asks a model what to do, runs a tool, checks the result, may hand a subtask to another agent, then calls a model again. One user request becomes a changing chain of small jobs.

A new preprint from researchers who studied Microsoft Azure argues that the host CPU is therefore no longer just feeding an accelerator. It schedules model calls, keeps track of the workflow and runs tools. The task keeps crossing the CPU-GPU boundary until it is done.

That matters because improving model inference alone cannot remove time spent waiting on the rest of the chain.

Quiet averages can hide short bottlenecks

The authors used two views of the problem. They collected a 24-hour trace of production agent services across Microsoft Azure, then ran four open-source systems on a server with a 96-core AMD EPYC CPU and eight NVIDIA A100 GPUs.

The controlled set covered different patterns: sequential coding agents, a four-agent parallel workflow and a mixed system with text, vision and speech roles.

Across those runs, median host CPU use ranged from 6% to 31%. GPU compute activity stayed below 55%. Those averages sound comfortable, but the traces were bursty. Tool calls could suddenly push the CPU, while some model roles filled a GPU and others left one idle.

The researchers' point is not that the CPU always replaces the GPU as the main bottleneck. It is that either one can briefly sit on the critical path, and static capacity leaves a lot unused between those moments.

Three choices change the workload

The paper sorts agent systems along three practical dimensions.

First, orchestration: does ordinary software decide the next step, or does a model make that decision? Second, execution: does work happen in sequence or fan out in parallel? Third, model composition: do the agents share one model or keep several different models ready?

A sequential workflow leaves gaps that other work may use, but every slow step delays the answer. A parallel workflow creates useful concurrency and sharper bursts. A shared model can reuse weights and cache, while several model families add memory pressure and make balancing harder.

This taxonomy is probably the paper's most durable contribution. It gives infrastructure teams a better question than simply asking how many GPUs an agent needs.

The prototype worked, but not everywhere

The team built a prototype runtime called Agora to test three ideas: lend idle CPU cores to other work, consolidate agents onto fewer GPUs when their busy periods do not overlap, and keep scheduling, orchestration and tools in separate CPU pools.

At low load across the four frameworks, CPU harvesting recovered 95% of a co-located workload's standalone throughput while keeping average agent slowdown below 3%. In one Owl configuration, GPU consolidation freed a third of the GPUs, raised generation throughput by 82% and cut tail latency by 2.5 times.

There is an important counterexample in the same paper. Removing half the GPUs from the highly parallel CORAL workflow cut throughput by 71% and made tail latency 16 times worse. Agora's own policy would avoid harvesting there.

So the result is not a general promise to use fewer GPUs. The useful result is that the workflow's shape can tell a runtime when spare capacity really exists, and when it does not.

What is confirmed, found and still open

Confirmed: the public preprint describes a 24-hour Microsoft Azure fleet trace and controlled runs of SWE-Agent, Trae, CORAL and Owl on specified hardware. It reports separate CPU, GPU and role-level measurements, plus prototype experiments.

The research finding: in the workloads studied, agent execution was fragmented and bursty. CPU-side orchestration and tools repeatedly interrupted GPU inference, leaving both kinds of hardware underused on average while still producing short bottlenecks.

Still open: how representative one day of Azure traffic is, how Agora behaves on other clouds and hardware, and whether the same gains survive production constraints the paper does not expose. The underlying fleet trace is not publicly available for independent reanalysis.

The paper is a preprint and has not been peer-reviewed. The authors also evaluate a limited set of agent frameworks. This is an early systems result, not a finished blueprint for every data centre.

Still, it changes the mental model in a useful way. An agent is not one large request. It is a small, unpredictable procession of work, and the server has to follow all of it.