Engineering teams at the world's largest streaming platforms are currently grappling with a fundamental tension in AI infrastructure: the trade-off between the rigid performance of proprietary engines and the rapid iteration cycles of open-source frameworks. As large language models move from experimental playgrounds to production environments that serve millions of concurrent users, the overhead of managing disparate serving stacks becomes a primary bottleneck. This week, the industry is looking closely at how Netflix has navigated this divide by constructing a unified serving layer that bridges the gap between high-level orchestration and low-level GPU acceleration.

The Architecture of a Unified Model Scoring Service

Netflix operates a JVM-based integrated serving system designed to handle the entire lifecycle of a request within a single pipeline. This system manages routing, A/B testing, feature lookups, inference, post-processing, and logging. To maintain efficiency, the platform differentiates execution paths based on model scale. Small CPU-based models are executed directly within the process to eliminate the latency of remote calls. In contrast, larger GPU-dependent models utilize a local pre- and post-processing layer before delegating the heavy lifting to a remote Model Scoring Service (MSS).

The MSS acts as a universal interface for a variety of frameworks, including XGBoost, TensorFlow, PyTorch, and LLMs. At the core of this service is the NVIDIA Triton Inference Server, which manages model loading and GPU scheduling. While Netflix initially relied on TensorRT-LLM, the company transitioned to vLLM by the summer of 2025. This shift was driven by the closing performance gap between open-source engines and proprietary ones, as well as the need to support diverse workloads such as embedding generation, prefill-only inference, and autoregressive decoding. A critical advantage of vLLM is its ability to load custom model architectures without requiring multi-stage compilation, which significantly accelerates the development cycle for non-standard models.

To ensure compatibility across the organization, the platform provides two parallel interfaces. It maintains gRPC for existing client libraries and deployment pipelines, while offering an OpenAI-compatible HTTP API to integrate with the broader LLM ecosystem. This HTTP API is implemented by leveraging NVIDIA Triton's OpenAI-compatible front-end, where a TritonLLMEngine converts request schemas into Triton inference requests and delivers responses via FastAPI.

Packaging vLLM within Triton is handled through two distinct paths: the Python backend and the vLLM backend. The Python backend requires developers to manually define input and output tensor specifications, meaning any front-end I/O change necessitates a corresponding update to the packaging code. The vLLM backend simplifies this by using a JSON configuration containing model weights and tokenizer information, allowing the Triton backend to generate I/O tensor specifications dynamically. This decoupling enables independent updates to the model and the front-end. However, this flexibility introduces versioning risks. Because the Triton backend is compiled against specific vLLM APIs, a version mismatch can prevent the backend from loading. For instance, if Triton 25.09 references `vllm.engine.metrics` but vLLM 0.11.2 has removed that module, the system will fail to initialize.

Deployment strategies are split between cost-efficiency and stability. For stable interfaces, Netflix employs Red-Black deployment, where a new version is validated before traffic is incrementally shifted. When I/O schema changes are unavoidable, the team uses Versioned deployment, maintaining independent deployments for every `(modelId, modelVersion)` pair to decouple consumer updates from model releases. To mitigate cold start latency, models are materialized on Amazon FSx, allowing the system to load weights from a high-performance file system rather than a slower object store.

Solving the GIL Bottleneck and Metric Invisibility

Despite the robust architecture, the transition to production revealed a critical performance ceiling in the decoding loop of vLLM V0. In the V0 architecture, the logit processor was Python-based. The GPU would generate logits for a batch, which were then copied to the CPU to be executed sequentially for each request. This design hit a wall due to Python's Global Interpreter Lock (GIL). As batch sizes increased, the CPU processing time grew linearly, leading to a spike in tail latency that degraded the user experience.

Netflix addressed this by migrating to vLLM V1, which fundamentally restructured logit processing. By moving the processing to a batch-level operation and rewriting the critical path in multi-threaded C++, the team ensured that logit processing time remains constant regardless of the batch size. The V1 API introduces `update_state(batch_update)` to explicitly track request states within dynamic batches. This is particularly vital for handling partial prefills caused by chunked prefilling or state machine resets during preemption events triggered by memory shortages.

Operational visibility presented another hurdle. The built-in Triton bridge only exposed 9 of the 40+ available vLLM metrics, leaving engineers blind to critical data such as KV cache utilization and prefix cache hit rates. To solve this, Netflix implemented a lightweight HTTP proxy to fetch Triton metrics while using a Prometheus MultiProcessCollector to read vLLM metrics directly from `.db` files on disk. These two streams are then merged into a single `/metrics` endpoint for comprehensive monitoring.

Finally, a bug was discovered in the Triton front-end where the `response_format` specified in the schema was dropped before reaching vLLM. This meant that requests for JSON output were executed without guided decoding constraints, often resulting in malformed JSON responses. Netflix resolved this by importing the front-end as a Git subtree and applying a custom patch to ensure `response_format` requests are correctly converted into vLLM guided decoding parameters.

For engineers designing similar infrastructure, the primary lesson lies in the management of I/O schema variability. If configuration changes, such as tensor shapes, can be encapsulated within the model to remain version-independent, the lower-cost Red-Black strategy is preferable. Versioned deployment should be reserved strictly for cases where interface changes are inevitable, as the associated increase in GPU memory overhead is significant.