[Submitted on 12 Feb 2026 (v1), last revised 29 Jul 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at this https URL.

Submission history

From: Rome Thorstenson [view email]
[v1] Thu, 12 Feb 2026 08:09:14 UTC (617 KB)
[v2] Tue, 30 Jun 2026 02:56:57 UTC (669 KB)
[v3] Wed, 29 Jul 2026 19:22:20 UTC (670 KB)