Instead of relying only on externally hosted APIs, Netflix built the entire LLM serving stack directly within the existing production environment, from model deployment to inference, operation, and observation. This article explains the process of deciding on engine selection, model packaging, API design, non-disruptive deployment, and constraint decoding, as well as the problems revealed in actual operation. Ultimately, Netflix created an integrated serving platform that combines vLLM and NVIDIA Triton and provides gRPC and OpenAI-compatible HTTP API.

1. Integrating LLM into existing ML serving environment

Netflix's machine learning system for large-scale members operates centered on JVM-based integrated serving system. This system is responsible for the entire flow required by downstream consumers.

• Request routing and A/B testing logic
• Candidate creation
• Feature search
• Model inference
• Post-processing
• Logging of each step
• Real-time processing and cached batch processing

Rather than operating LLM in a completely separate system from the existing machine learning model, Netflix chose to handle XGBoost, TensorFlow, PyTorch, and LLM within a single serving system.

Small CPU models run directly inside the integrated serving system to reduce remote call costs. On the other hand, since large models require a GPU, pre-processing and post-processing are handled by the existing serving system, and the actual inference is transferred to a remote backend called Model Scoring Service (MSS).

Under MSS, NVIDIA Triton Inference Server is responsible for model loading, batching, and GPU scheduling. A Java-based control plane is placed on top of that to manage the following tasks:

• Model deployment and version management
• Status check
• Autoscaling
• Rollout across multiple regions
• Non-disruptive upgrades

Model developers only need to prepare model artifacts and deployment settings, and the control plane creates a GPU instance, configures Triton, and deploys the new version.

"The LLM model should not be an exceptional model that receives special treatment."

Netflix's core goal was to handle LLM within the same operating system as the existing model. Existing consumers can use the gRPC route through the integrated serving system, and newer LLM applications can use the direct HTTP route.

2. Reasons for switching from TensorRT-LLM to vLLM

The initial platform was built on top of TensorRT-LLM, which was highly performant at the time and was already integrated with Triton. But things changed around the summer of 2025.

Open source inference engines have significantly reduced the performance gap with specialized commercial/compilation-based stacks, and Netflix's LLM workload has also become more diverse. In addition to creating a simple autoregressive, we had to handle the following tasks:

• Create embeddings
• Prefill-only inference for ranking and search
• Autoregressive decoding
• Custom model requiring complex step-by-step constraint logic

After re-benchmarking against the new workload, Netflix selected vLLM as the default engine, or 'paved path', considering its operational suitability.

The main reasons for choosing vLLM are as follows:

1. Can load custom architectures without complex multi-step compilation
   We were able to quickly experiment and deploy models that deviated from the standard.

2. Provides an extension point for custom decoding logic
   It was essential to implement constraint decoding, which will be explained later.

3. High debuggability
   It was easier to examine the cause of failure and intermediate states than in the past TensorRT-LLM's compile-based engine.

4. High level of familiarity
   Many researchers and ML engineers are already using vLLM during the research phase, reducing the cost of moving research results to production.

3. How to package vLLM models in Triton

After choosing vLLM, the next important question was how to package the model into Triton. There are two main methods available in Triton.

Python backend

In the Python backend, the model creator directly defines the specifications of the input and output tensors at packaging time. This specification is anchored within the model artifact and must exactly match the format expected by the external frontend's request generator.

Therefore, if the frontend changes the input/output format, the packaging code must also be modified. If you miss this, the request will fail at runtime after the model is deployed.

vLLM backend

In the vLLM backend, the model artifact is more of a JSON configuration file that points to model weights and tokenizer locations. Triton's vLLM backend dynamically generates input and output tensor specifications at deployment time, so model authors do not need to define them directly.

In this structure, the model and frontend can develop independently of each other. Therefore, the vLLM backend was a more suitable default for general Hugging Face compatible models.

However, two problems emerged in actual operation.

Version mismatch issue

Triton's vLLM backend is compiled against a specific vLLM API. If the two versions are different, the backend itself may not load. For example, there was a case where Triton 25.09 tried to import vllm.engine.metrics, but the module was removed in vLLM 0.11.2.

Accordingly, Netflix applied the following principles:

• When creating a service image, fix compatible Triton·vLLM versions together.
• Prevent model creators from arbitrarily changing the vLLM version at packaging time.

Custom model issues

The vLLM backend assumes a standard Hugging Face compatible model and handles the entire inference lifecycle. However, the following models deviate from the standard flow:

• Ensemble Pipeline
• Custom tokenizing
• Non-standard pre-processing and post-processing
• Models that require special execution methods

In this case, you must directly control execute() using the Python backend. Netflix plans to make the vLLM backend the default path, but maintain an escape hatch called the Python backend for some models.

4. Coexistence of OpenAI compatible API and existing gRPC

Netflix made it possible for all models to be inferred with the same gRPC call for integration with existing systems. At the same time, OpenAI-compatible API, which has become a de facto standard in the LLM ecosystem, was also provided as an additional frontend.

OpenAI-compatible APIs are already supported by the following tools and ecosystem:

• Inference Engine
• Orchestration framework
• Assessment tools
• Various client libraries

Thanks to this, code changes can be minimized when switching from an externally hosted model to a model fine-tuned within Netflix. Whether you choose to self-host for reasons such as quality, latency, cost, or data privacy, the journey from experiment to production is virtually seamless.

"The transition from a hosted model to a fine-tuned self-hosted model is almost seamless. Because we use the same API, code changes are minimal."

NVIDIA's Triton OpenAI compatible frontend was used for implementation. This frontend operates with the following structure:

1. Embed the Triton server within an OpenAI-compatible frontend process.
2. TritonLLMEngine converts the API request schema into a Triton inference request.
3. Provide a response through FastAPI.
4. Activate the KServe HTTP/gRPC frontend as well.
5. Java control plane accesses the same Triton instance with gRPC.

Silent failure discovered in response_format

While applying Triton's default OpenAI compatible frontend, a significant issue was discovered. Although response_format was allowed in the API schema, this value was not actually passed to vLLM and was quietly discarded.

As a result, even if the caller requested a JSON response, guided decoding constraints were not applied, and incorrect JSON could be generated. The bigger problem was that the platform did not flag the error.

Netflix integrated the frontend into the code repository and modified it to convert response_format to vLLM's guided decoding parameters at the time of request.

This case shows that just because an API is supported in form, it does not mean that the meaning is conveyed to the actual engine. In particular, the option silently ignored can be very dangerous in production.

5. Non-disruptive deployment strategy for GPU model

GPU-based LLM services have longer startup times than CPU services. In addition, the input and output schema may change as the model version changes, requiring adjustments beyond simply launching a new instance.

Netflix offers two distribution strategies.

Red-Black Distribution

Deploy the new version side-by-side with the current version, then gradually shift traffic to the new instance as it passes health checks. It matches the rate at which the new version expands with the rate at which the existing version shrinks, and if any step fails, an atomic rollback is performed.

The Red-Black method is suitable for models with a stable interface. However, in actual operation, coordination problems related to schema changes emerged.

For example, if a new model changes the tensor dimension, the parent consumer must also change its settings. However, consumer settings cannot be changed until the new model is fully ready. As a result, during the transition process, the parent system may send requests in the old format to the new deployment, and these requests may fail.

Versioned distribution

The Versioned method maintains independent distribution for each (modelId, modelVersion) combination. Because new and existing versions process requests simultaneously, model deployment and consumer configuration changes can be separated.

Consumers can switch settings once the new version is fully ready, while the old version continues to serve existing traffic. Clean up old, unused distributions, but always keep the latest version.

However, during the transition period, both versions will use the GPU simultaneously, so GPU costs will temporarily increase.

Netflix's recommended method is as follows:

"It is better to include mutable settings, such as tensor types, directly inside the inference model to make them version-independent. Then, you can use the cheaper Red-Black route."

In other words, design the model to avoid breaking interfaces as much as possible, and use Versioned distribution only when there are unavoidable compatibility changes.

6. Boot process and integration metrics

Two additional operational issues were discovered that could be easily overlooked in the design phase, but are important in actual production.

Model caching and boot order

To launch vLLM and Triton instances, several steps must be completed to open the gRPC port. In particular, if you download it directly from S3 or Hugging Face every time you start a large LLM, the startup time may become too long and exceed the scheduler's tolerance range.

Netflix pre-stores the model in Amazon FSx when the model is announced for distribution. Then, when the actual instance starts, the model can be read from the file system, which is faster than object storage, allowing for a warm start.

For deployments that require an OpenAI-compatible API, Triton is embedded within the OpenAI front-end process; otherwise, Triton runs as an independent process. This method is set at the time of distribution packaging.

Other boot steps are as follows:

• Unzip the model package
• Custom vLLM plugin installation via Python entry_points
• Prometheus multiprocess directory cleanup
• Delay opening gRPC ports until the engine is ready

Triton and vLLM metrics integration

vLLM and Triton expose metrics in different ways.

• vLLM records metrics in the form of .db files in PROMETHEUS_MULTIPROC_DIR.
• Triton provides server metrics through its own Prometheus endpoint.

Triton's default bridge exposed only 9 of vLLM's 40+ metrics. Therefore, the following important indicators could not be confirmed.

• Token throughput
• KV cache utilization
• Prefix cache hit rate

Netflix has created a lightweight HTTP proxy that combines both metrics. This proxy pulls metrics from Triton over HTTP, reads vLLM metrics from disk using Prometheus's MultiProcessCollector, and returns them to a single /metrics endpoint.

Thanks to this, it is now possible to observe LLM-related indicators without modifying the existing dashboard and notification system.

7. Challenges of large-scale constraint decoding

Some of Netflix's production workloads require fine-grained control over which tokens a model can generate. Normally, if a model generates an incorrect result, it can be post-processed or retried, but this approach also comes at the cost of generating an incorrect result.

Netflix chose to place constraints inside the decoding loop rather than applying them after inference. The goal is to make the model generate only results that meet the conditions from the beginning.

Each constraint is represented by a state machine. The state machine calculates the tokens allowed in the next step based on the history of tokens generated so far, and creates a mask indicating whether a token can be selected at each step. Because different rules may apply to each request, each request has its own constraint processor.

This feature was initially implemented in vLLM V0. This is because at the time, V1 was not mature enough in the necessary functions. Afterwards, V1 was developed and migrated to V1 in 4th quarter of 2025.

CPU bottleneck in V0

The initial Python implementation was functional, but did not scale as the number of requests increased.

In V0, the GPU calculates the logit for the entire batch and then copies it to the CPU. Afterwards, the constraint logic of each request is sequentially executed. Because of Python's GIL, it was difficult to effectively parallelize processing for each request.

Therefore, as the batch size increased, the logit processing time performed by the CPU increased linearly, and ultimately the overall delay time increased due to CPU constrained processing rather than the model inference itself.

Although it was not found in single request benchmarks, it became a serious problem in real concurrency environments, worsening tail latency.

Batch unit processing in V1

vLLM V1 moved logit processing to batch level rather than per request. Netflix rewrote its custom processor accordingly.

• Process data from multiple requests in a batch unit structure
• Calculate masks for multiple requests at once
• Reimplement core operations in C++
• Avoid the influence of Python GIL through multithreading

In V1, we had to explicitly manage update_state(batch_update) to track the members of a dynamically changing batch. Although the implementation was more complex than V0, it was a necessary change to maintain the state accurately even when requests were added or removed from the batch.

As a result, even as the batch size increased, the logit processing time did not increase significantly and remained stable.

Additional issues resolved during operation

An unexpected problem occurred in the constraint logic that maintains the state even after resolving the performance bottleneck.

Partial prefill

V1 processes the prefill by dividing it into several pieces. Therefore, prefilling of one request can progress through multiple engine stages.

However, it was difficult to sufficiently distinguish whether the request was completely prefilled or only partially prefilled using BatchUpdate alone. For this purpose, Netflix separately tracked the prefill status internally.

Preemption and KV cache regeneration

When memory pressure occurs, vLLM can evict the KV cache of unfinished requests and reschedule them for a later time. At this point, the request may resume with a different prompt and output token list than before.

This causes problems when the state machine assumes that the list of output tokens always grows monotonically. When Netflix detects that the token history has decreased between decoding steps, it initializes the state machine and restarts based on a new prompt.

8. Construction results and future plans

The goal of the platform Netflix created was to simultaneously meet a wide range of production ML requirements.

• Low latency
• Custom logic support
• Integration with existing infrastructure
• Fast transition from experiment to production
• Large-scale GPU operation
• Stable version distribution and observation

The final system is based on vLLM and Triton and serves existing models and LLM in a consistent way. Integrates with existing Java serving systems via gRPC and connects with the latest LLM applications via OpenAI-compatible HTTP API.

During the construction process, Netflix confirmed that the following details, rather than performance itself, determine operational stability.

• Fixed versions of Triton and vLLM
• Combination of model packaging method
• Verify that API fields are delivered to the actual engine
• Schema changes and adjustments to consumer settings
• GPU boot time and model caching
• Integration of metrics from multiple engines
• Handling CPU constraints in a concurrent environment

The main future investment directions are as follows.

• System Prompt Compression to reduce prompt length while maintaining quality.
• Asynchronous Scheduling in vLLM V1
• Vectorized logit processor that runs as a fused GPU kernel instead of CPU code
• Low-precision model version to reduce memory usage and increase throughput

Netflix has actively utilized open source projects such as Triton, vLLM, and PyTorch, and plans to continue to develop the platform by collaborating with the open source community.

Conclusion

Netflix's case shows that rather than simply choosing the fastest engine in LLM serving, integration with existing ML platforms, debugging possibilities, API compatibility, deployment stability, and operational observability must be considered together. In particular, in actual production, performance and stability were affected by issues that were easy to miss at the design stage, such as version mismatches, silently ignored API options, schema changes, and batch processing methods.

In the end, the biggest lesson Netflix learned is that incorporating LLM naturally into the existing model serving system, rather than making it a special exception system, improves long-term operational efficiency and developer experience.