The catch is who owns that cache. The cached KV belongs to sessions that sit idle in tool calls, and admitting waiting work evicts it. A burst can tip the engine into a trap: every turn misses, the queue grows, the gap between a session's turns grows with it, and the backlog does not drain, at a load the engine served before.

Quasi-static models read the trap as the falling branch of $\dot n=\lambda-\mu(n)$, with $\mu(n)$ the engine's throughput when $n$ sessions are kept running. On our engines, where every trapped turn misses, that reading gets the numbers wrong. Trapped runs complete sessions faster than the all-miss capacity, by up to 2.4 times (one from-empty run at 6.5 $\lambda_{\mathrm{ref}}$), and over the 57 stuck runs of the paired-test protocol their backlog grows, on average, at about two thirds of the quasi-static rate. An operator should care: in model F below, for this workload, a cut to just below the throughput measured in an all-miss trap does not clear it. We therefore ask two questions:

• What is the trapped state? We idealize it as model F, a fluid FIFO server on which every turn misses, and derive its law exactly: along FIFO orbits its cycles obey a linear renewal recursion, and every trapped state converges to a known self-similar regime. In the model, for our workload, a trapped server completes sessions faster than its all-miss capacity: a youth premium.

• How does an engine get out? We test two exits on real engines. Refusing arrivals drains the queue by shedding sessions; a session cap restores reuse, and rescued all 9 established traps it was tried on without turning sessions away.

We claim none of the eviction, cold-prefill and queueing mechanisms that agent-serving systems describe; what we add is the law of the state they lead to, and a test of the ways out. Lu et al. (arXiv:2611.02543), who posted first, studied the same collapse concurrently and independently, on prefill–decode disaggregated SGLang. The table below collects our main results.

Model F: A Renewal Law for the Trapped State

To say what the trapped state is, we idealize it. We first give the intuition for why a trap holds, then the model and what it implies.

Why the Trap Holds

Once the burst has passed, why does the engine not recover? Our answer here is a heuristic consistency argument, not part of the theorem. In a trap the KV pool turns over every $R_M=M/\mathrm{churn}\approx 26$–78 s (a turnover scale, not a pathwise eviction bound; $M$ is the KV pool in tokens, and churn is the rate at which new KV is written: uncached prompt plus completion tokens per second), while the gap between a session's turns keeps growing: in every BURST run of the first three registered tests the gap was 4.1–9.7 times $R_M$, so an idle session's KV is overwritten long before it returns. In the good arms the same ratio is 0.010–0.028.

Drag the slider below to compare the two time scales in a good arm and in a trapped one. In the good arms a session comes back long before the pool has turned over; in the trapped arms it comes back after several turnovers.

A Renewal Recursion along FIFO Orbits

Model F is a fluid FIFO server on which every turn misses. Turn $k$ costs $c_k>0$ GPU seconds when it misses, and $G_k=P(T\ge k)$. The all-miss capacity and the load are

Its assumptions (F1)–(F6): every turn misses; the server works at unit rate while anything is pending; service is first come, first served at the request level; a completed turn $k<T$ rejoins the tail at once as turn $k+1$; new sessions arrive as turn 1 at rate $\lambda$; and the initial pending mass is finite with positive work. A request joining at time $u$ completes at $f(u)=u+V(u)$, where $V(u)$ is the pending work, so $V$ is also the gap between a session's consecutive turns. A FIFO orbit is $u_{n+1}=f(u_n)$, with cycles $\Delta_n=u_{n+1}-u_n$.

FIFO maps the requests that join during one cycle onto the requests that complete during the next, and the turn-$k$ mass joining there is $\lambda G_k\Delta_{n-k}$. Lemma 1 turns this into a linear recursion for the cycles,

with $K$ the largest possible turn count (60 for the standard workload), $h_n\ge 0$, $h_n=0$ for $n\ge K$, and $\sum_n h_n=V_{\mathrm{tot}}(u_0)$, where $V_{\mathrm{tot}}$ is the remaining all-miss work of every session present, which changes at rate $\rho-1$ while the server is busy. Its growth rate is a Perron root:

Where Every Trapped State Goes

Theorem 1(a): every trapped state converges to a known self-similar regime, in which

The gap between a session's turns grows linearly at rate $\gamma$, the number of sessions present $N$ grows at $\alpha$, sessions complete at rate $D$, and the turns served follow the tilted mix $\nu^z$ rather than the stationary mix $\nu_k\propto G_k$: the turns served are young and short. Proposition 1(a), classical in substance, covers the other side: the queue empties if and only if $\rho<1$, at $V_{\mathrm{tot}}(0)/(1-\rho)$, and at $V_{\mathrm{tot}}(0)$ with arrivals stopped. Near threshold, Proposition 1(c) gives $\gamma=(\rho-1)/\bar k_c+O((\rho-1)^2)$ and $\alpha/(\lambda-\lambda_{\mathrm{miss}})\to E[T]/\bar k_c$, where $\bar k_c=\sum_k k\,c_kG_k\big/\sum_k c_kG_k$.

The Youth Premium

In model F the turns served are young, and in our workload young turns are cheap, so a trapped server completes sessions faster than its all-miss capacity: a youth premium. Theorem 2 splits it into a turn-count factor $H$ and a cost-growth factor $\Gamma$:

Near threshold, $D/\lambda_{\mathrm{miss}}-1=(\rho-1)\big(1-E[T]/\bar k_c\big)+O((\rho-1)^2)$. For our workload $E[T]/\bar k_c$ lies in 0.60–0.62, and the premium is positive at every load checked: proved by Theorem 2's monotonicity for $\rho\ge1.04$; below that it rests on a floating-point bracket and a first-order expansion. The premium is not automatic. With slowly growing costs $c_k=1+10^{-3}(k-1)$ it is negative at every load checked, and for an untruncated geometric $T$ with constant costs there is none.

This is what an operator needs. Corollary 3: after a switch to $\lambda'$ at $t_0$, the queue empties if and only if $\lambda'<\lambda_{\mathrm{miss}}$, after the busy period $V_{\mathrm{tot}}(t_0)/(1-\lambda'/\lambda_{\mathrm{miss}})$. The corollary is not new as mathematics, but with a premium it bites: a cut to any rate in $[\lambda_{\mathrm{miss}},D(\lambda))$, below the asymptotic trapped throughput, does not empty model F's queue. In the model, a trap's measured throughput overstates the recovery threshold. Remark 1 adds that $z$ solves $\rho\sum_k\omega_kz^{-k}=1$ with $\omega_k=c_kG_k/\sum_jc_jG_j$, so the trapped composition and $D/\lambda_{\mathrm{miss}}$ are unchanged by a uniform GPU speed-up at equal $\rho$; this is not a general hardware claim.

The costs come from the paper's calibrated prefill model, $t_P(x,y)=a\max(1,x/B_c)+\max(f_0,bx)+c_p\,x\,\big(y+(x+1)/2\big)$, with $a=9.24$ ms, $f_0=1.26$ ms, $b=62.6\,\mu$s/token, $c_p=2.24$ ns per causal pair and chunk $B_c=8192$. Then $c_k=t^m_k+\delta$, the miss prefill plus $\delta$, the all-miss per-turn decode share, chosen so that $\lambda_{\mathrm{miss}}$ matches the first-principles model's capacity. The calculator below evaluates these statements in your browser.

Experiments

The registered tests on real engines come first, then the theory against every stuck run, a blind test, and the two ways out.

Setup and Baselines

The registered tests drive real engines on Iowa State's Nova cluster, via Slurm, with one NVIDIA A100 80GB PCIe or H200 NVL per engine, under a synthetic agent workload. A session has $T$ turns, and turn $k$ sends a prompt of $L_k$ tokens:

so $E[T]=14.76$. The engines are SGLang 0.5.22 and vLLM 0.32.0, serving Qwen2.5-7B-Instruct or Llama-3.1-8B-Instruct, with the KV pool capped at 300k tokens (150k or 600k in the pool sweep; SGLang's native 955,757 tokens in the native cell). Rates are multiples of each cell's reference rate $\lambda_{\mathrm{ref}}$, the all-miss session rate measured with prefix caching disabled.

The main protocol, E2, runs three arms per seed on common arrivals: BURST (200 sessions within 60 s at 1200 s), NOBURST, and CAPPED, with a client-side FIFO session cap from the start. Runs last 8700 s and are judged over $[5700,8700)$ s. BURST counts as trapped if its uncached prefill token rate is within ±15% of the all-miss rate and its backlog slope is above $0.5(\lambda-\lambda_{\mathrm{ref}})$. Each test's rule was committed to git before its jobs were submitted; there is no third-party registry. Pre-registered means the rule was also pushed to GitHub before submission; registered means it was committed before and pushed later. For the blind tests the record is a git tag on each frozen prediction or rule, listed below with its commit and the time GitHub recorded its push (all on 2026-11-05, CST); the rest is in Appendix E.12–E.13 of the paper and in the provenance/ folder of the public repository.

The quasi-static baseline rests on the engine's closed-population throughput curve $\mu(n)$, the completion rate with $n$ sessions kept running, which has a cliff. The 300k-token pool limit did not create that cliff; it moved it. On an H200 at the 300k pool the window between the all-miss capacity and the curve's peak narrows: the all-miss capacity rose 2.71-fold and the peak of $\mu(n)$ only 1.63-fold. A first-principles model of the curve put its 0.5-hit crossing off by 1.5 sessions on average (Lu et al.'s curve, read post hoc: 4.1); five of its cells were blind (three in time, two by access only), one of them partly seen.

Registered Results

Table 1 lists the registered tests and their outcomes, and Figure 7 shows the trap on SGLang.

Table 1. The registered tests, condensed from the paper's Table 1. Rates are nominal multiples of each cell's $\lambda_{\mathrm{ref}}$. Seeds 0–2 are shared across tests, so counts across tests are descriptive. Across the five E2 tests, 5 + 6 + 4 + 5 + 3 = 23 BURST runs were trapped by the rule. The other 7 of the 30 were also at zero hits; they missed only the rule's backlog-slope condition, since each one's uncached prefill rate was within ±15% of the all-miss rate (paper Table 7 and App. E.5). The slope line is $0.5(\lambda-\lambda_{\mathrm{ref}})$, and at 1.35 $\lambda_{\mathrm{ref}}$ model F puts the true slope only about 25% above it. Six of the 7 runs below the line are seed 1, whose arrival plan delivers 0.8–8.9% fewer arrivals than nominal over the window in every cell and rate (paper App. B). This account is post hoc and changes no registered verdict.

Once formed, the trap held. Over 50 trapped runs before the blind tests, about 100 trapped hours, no 300 s window after the trap formed reached a follow-up hit fraction of 0.5; the highest was 0.16 (a post hoc summary). The registered dose response places the basin boundary between bursts of 40 and 60 sessions in every seed, with a burst of 50 trapping 1 of 3. Not every registered rule was met: lengthening every tool call that starts in $[1200,2100)$ s by 30 s trapped 1 of 3 runs, so that rule (traps in 2 of 3) failed, and at the tenth rate, Llama-3.1-8B at the lower rate, the trap rule failed on its backlog-growth condition.

The Theory against 57 Stuck Runs

We derived the theory after the stuck runs were read, so this comparison is retrospective. No parameter is fitted to any trap measurement, and model F's capacity is within 4% of the all-miss capacity measured on the engine with prefix caching disabled and 32 sessions kept running. Across the 57 stuck runs:

• the mean served prompt is within 3% of model F's in 50 of the 57 runs: the trapped engine serves turns 9–41% shorter than the stationary mix;

• gap growth, measured over predicted, has median 1.01 (mean 0.96); per run it ranges from 0.04 to 1.81;

• every stuck run completes sessions above $\lambda_{\mathrm{ref}}$, at 1.10–2.03 $\lambda_{\mathrm{ref}}$ (up to 2.40 in the from-empty runs), and net of the congested engine's faster token rate, completions still exceed $\lambda_{\mathrm{ref}}$ by a factor of 1.08–1.81 in every one of the 57;

• the backlog slope, over model F's $\alpha$, has mean 0.99 and median 1.06, where the quasi-static reading gives 1.36–1.59.

Table 2. Model F against the engine, per cell (paper App. B, Table 3): measured ranges over seeds against model F's prediction; $\bar L$ is the mean served prompt in tokens. Retrospective; no parameter fitted to any trap measurement. The last column gives the measured backlog slope over model F's $\alpha$, with the quasi-static reading in parentheses.

Blind Tests

We then tested the theory blind in outcome: the predictions were frozen before the runs they score existed. Each test carries its status from the paper's evidence ledger, with its caveats.

• Tool-slowdown traps (pre-registered, blind in outcome; tolerances set on earlier runs). Across the four blind runs trapped by a tool slowdown alone, pre-set scores S1–S4 pass in both cells and S5 is close in 3 of 4. The per-run slope check failed in 2 of 4 runs.

• Which seeds a slowdown traps (pre-registered, two frozen predictors). Outcome: NO-TRAP, as the simulator said, seed by seed; the first-principles fluid verdict, TRAPS, missed.

• Cut traps, CUT-A and CUT-B (pre-registered, blind in outcome; the rule was fixed after the predictions were read). Cutting arrivals into $[\lambda_{\mathrm{miss}},D)$: all three CUT-A runs stayed stuck to the end at zero hits, and all three seeds were theory-favoured. Cutting below capacity: all three CUT-B runs were still stuck at 8700 s, with 140, 77 and 89 sessions present against model F's 139, 79 and 94. Intermediate readings fit as well; what singles out model F against them is post hoc.

Two Ways Out

In model F, Corollary 3 says a cut in arrivals must land below $\lambda_{\mathrm{miss}}$ to drain the queue. We illustrate it in the all-miss fluid for the 300k cell at 1.65 $\lambda_{\mathrm{ref}}$, where $D=1.223\,\lambda_{\mathrm{miss}}$, cutting at 5000 s with a backlog of 251. This is model output, not engine data.

Table 3. Corollary 3 illustrated (paper App. A). Every cut at or above $\lambda_{\mathrm{miss}}$ leaves the trap growing or level; only a cut below it drains, after a busy period that diverges as the cut approaches $\lambda_{\mathrm{miss}}$.

On the engine we tested two exits. Refusing arrivals works by shedding: a 600 s stop refused 56–59 sessions and left both runs stuck; a 3600 s stop refused 355–378 and left both good. A session cap restores reuse instead. Engaged inside an established trap, it holds the admitted sessions below the all-miss onset $n_b$, so their turns hit again, and the excess waits at the client.

Table 4. The rescue tests R1 and R4 (paper App. E.7). After release, the rescued runs match NOBURST, while the never-capped BURST runs stay at zero hits.

Visualizing the Trap and the Rescue

To see a trap form and a session cap rescue it, we run the paper's simulator on two copies of the same engine, side by side.

The run was chosen from 40 seeds as one in which every burst session finishes before the end, the cap comes off inside the paper's engine range, and the completion rate is closest to model F's. Over 40 further seeds that played no part in the choice, every engine that gets the burst and no cap is stuck at zero hits, the trapped engines' served prompts average 1.03 times model F's, and their completion rates average 1.12 times the all-miss capacity, from 0.95 to 1.31.

Conclusions and Limitations

A burst of agent sessions can leave a prefix-cached LLM serving engine missing every turn, at a load it served before. In registered tests on two engines, two GPUs and two models, a 200-session burst left the engine at zero hits to the end in all 30 runs. For an idealization of the trapped state, model F, we proved that the cycles along FIFO orbits obey a linear renewal recursion whose Perron root fixes the dynamics: every trapped state converges to a self-similar regime in which the turns served are young and short. For our workload this gives a youth premium, so in the model a trap's measured throughput overstates the recovery threshold. Of the two exits, refusing arrivals sheds sessions, while a session cap rescued 9 of 9 established traps at the same load without shedding, and the engine stayed good after release.

Limitations:

• Model F is an idealization, and the paper does not prove that the engine converges to it. It leaves out concurrency, preemption and the transitions into and out of the trap. In the registered window, about 33–37 generations after the burst, a transient of order 10% remains for this workload.

• The premium is proved positive for $\rho\ge1.04$; below that it rests on a floating-point bracket and a first-order expansion. The repository's theory tests check most statements numerically, but not the exact identity of Lemma 1, Theorem 1(b), Remark 1, Corollary 3 beyond the subcritical busy period, or the monotonicity statements of Theorem 2.

• Why the trap persists, the gap outrunning the pool turnover, is a heuristic consistency argument, not part of the theorem. "Stays stuck" always means within the run's horizon.

• The match with the 57 stuck runs is retrospective. Among the blind tests, the CUT rule was fixed after the predictions were read, the tool-slowdown per-run slope check failed in 2 of 4 runs, and MIRROR's workload was chosen with the model.

• One registered trap rule failed (Llama-3.1-8B at the lower rate), as did the first tool-slowdown rule. The tests use a synthetic workload with random token ids and no shared prefixes (MIRROR caps its steps), on two engines, two GPUs and two models.

Code, Data and Registrations

Our public repository holds the code, the data and the registrations behind the paper. Tag trap-arxiv-v1 is the commit the preprint describes. (The repository's name comes from an earlier draft of the paper, withdrawn before it was announced.)

One command, run from the root of a clone, checks every registration against the commits GitHub's push log recorded. It should end with 28 proofs, 0 failures (exit status 0).

python provenance/registration-proofs/verify.py

The first command below recomputes 13 of the 16 tables in paper/derived/; the second rebuilds Figures 1 and 5–8 of the paper.

python figures/paper/derived_numbers.py
bash figures/paper/build_all.sh

Repository    Tag trap-arxiv-v1    arXiv

Abstract

Prefix caching lets an LLM agent's follow-up turns reuse cached KV, but a burst can tip an engine into a trap in which every turn misses and the backlog does not drain, at a load it served before. In registered paired tests on two engines, two GPUs and two models (passed at 9 of 10 rates), a 200-session burst left the engine at zero hits to the end in all 30 runs, while every matched no-burst run stayed good and a session cap prevented the trap in all 30 capped runs. We analyse an idealization of the trapped state exactly: for a fluid FIFO server on which every turn misses, we prove that along FIFO orbits its cycles obey a linear renewal recursion. As is classical, the trap drains if and only if arrivals are below the all-miss capacity; above it, a Perron root fixes the dynamics: every trapped state converges to a known self-similar regime in which the gap between turns grows linearly and the turns served are young and short. For our workload the backlog grows at 0.6 to 0.8 of the excess load and sessions complete faster than the all-miss capacity, a youth premium with turn-count and cost-growth factors, so in the model a trap's measured throughput overstates the recovery threshold. Retrospectively, with nothing fitted to trap data, the theory matches 57 stuck runs: the mean served prompt within 3% in 50 runs, completions above the all-miss capacity in every run, and the pooled gap growth. Blind in outcome, 4 runs trapped by a tool slowdown met 4 of 5 pre-set scores, cut traps persisted, and with a 20-step budget, trapped runs completed below the all-miss capacity, as predicted. Of two exits, refusing arrivals sheds sessions; a session cap restores reuse: in registered tests it rescued 9 of 9 established traps at the same load without shedding, and the engine stayed good after release. We do not prove that the engine converges to the model.

Citation

To cite the preprint:

Akshat Valse and Ali Jannesari. Inside the Trap: A Renewal Law for Prefix-Cache Collapse in LLM Agent Serving. arXiv:2611.08902 [cs.DC], November 2026.

@misc{valse2026inside,
  title         = {Inside the Trap: A Renewal Law for Prefix-Cache Collapse in {LLM} Agent Serving},
  author        = {Valse, Akshat and Jannesari, Ali},
  year          = {2026},
  eprint        = {2611.08902},
  archivePrefix = {arXiv},
  primaryClass  = {cs.DC},
  url           = {https://arxiv.org/abs/2611.08902}
}

   All research    Langevin Sampling