[2026 年 2 月 12 日提交 (v1)，最后修订于 2026 年 7 月 29 日（当前版本，v3）]

查看 PDF HTML（实验性）

摘要：Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat1M and WildChat-4.8M lack long-context, high prefix-reuse data, we release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During held-out evaluation windows, we fix the parameter values learned from tuning and improve p95 TTFT by 6.9-15.5% and p95 end-to-end (E2E) latency by 14.3-30.9% over baseline load-balancing policies such as simple session affinity and prefix-cache. The code and ART-Chat-2.5M dataset can be found at this https URL.

提交历史

来自：Rome Thorstenson [查看邮箱]
[v1] 2026 年 2 月 12 日 星期四 08:09:14 UTC (617 KB)
[v2] 2026 年 6 月 30 日 星期二 02:56:57 UTC (669 KB)
[v3] 2026 年 7 月 29 日 星期三 19:22:20 UTC (670 KB)