Scaling LLM Inference: Multi-Node KV Cache Offloading with GKE & Managed Lustre

To scale LLM inference, a multi-node KV cache offloading solution was implemented using GKE and Managed Lustre. This approach bypasses host-level capacity limits and reduces networking overhead. It achieved over 50% TCO savings and 60% reduction in GPU-hour requirements for Llama-3.3-70B inference. To further improve performance, a hybrid approach that offloads to CPU RAM can be used, delivering a 40% improvement in TTFT and 30% reduction in end-to-end latency.