Eliminate GPU Waste by Cutting the Retry Tax

In my previous article, I proposed an agent-native load balancer built around the unique traffic patterns of agents.

GitHub recently showed why this matters. A latent retry bug amplified Copilot traffic from roughly 7-9K requests per second to 70-100K RPS, delaying recovery during an outage.¹

Aquifer is designed to mitigate this kind of retry amplification. Instead of immediately rejecting excess work and forcing every client to retry independently, it queues requests and dynamically paces them against downstream capacity.

Dynamic pacing turns excess demand into waiting time instead of excess attempts.

That becomes especially interesting with GPUs.

At an illustrative $5 per GPU-hour, a fleet of 100,000 GPUs represents $500,000 of compute every hour. Even a small percentage of wasted capacity caused by retries, duplicate inference, or abandoned work can become expensive quickly.

We don't currently have good public numbers showing how much GPU capacity the industry wastes on retries. Inference providers should measure it.

Microsoft recently demonstrated how extreme retry amplification can become. During its May 29 Azure OpenAI outage, which lasted 7 hours and 26 minutes, Microsoft reported that a single failed request could generate up to 48 additional retry attempts.²

Those retries lacked sufficient backoff and jitter, and both remain important defenses. AWS has shown that exponential backoff with jitter dramatically reduces wasted work under contention.³

But retries still aren't free.

Agents make this more important because they operate at machine speed, fan out into parallel requests, and retry automatically. Cockroach Labs has described this as a new thundering herd problem for agent infrastructure.⁴ Datadog also found that 60% of the LLM-call span errors it observed in February 2026 were rate-limit errors.⁵

For GPU inference, repeated work can be particularly expensive. NVIDIA explicitly describes KV-cache recomputation as wasted compute and designs its inference infrastructure to avoid unnecessary prefill computation.⁶

Putting the burden of retrying on the client creates a tax for both the consumer and the provider. Consumers pay in latency and repeated work. Providers pay in capacity, congestion, slower recovery, and potentially wasted inference.

So I think inference operators should start measuring a simple metric:

How many physical inference attempts does it take to produce one successful logical request?

And ultimately:

How many GPU-seconds are being spent on retries, duplicate requests, abandoned inference, and recomputation?

That's the GPU retry tax.

---

Traditional load balancers are good at deciding where a request should go.

For expensive workloads that can tolerate waiting, we also need infrastructure that decides when that request should run.

Aquifer and EZThrottle Local explore that model by queueing excess work and dynamically pacing execution instead of relying entirely on failure followed by client retries.⁷⁸

This won't fit every workload. Real-time voice or interactive chat may require extremely low latency.

But a huge class of AI work does not.

Image generation, video generation, music generation, research agents, document processing, batch inference, and long-running background agents can often tolerate waiting.

For these workloads, waiting may be cheaper than retrying.

Instead of provisioning GPUs around every instantaneous spike, we can queue work, pace execution against available capacity, and scale when sustained useful demand actually requires it.

That could mean fewer retry storms, higher GPU utilization, less wasted compute, and potentially a greener inference stack.

Before buying another GPU, calculate your retry tax.