How NVIDIA MPS Cuts Amazon EC2 ASR Inference Costs by 75%

Solving GPU Underutilization with NVIDIA MPS

Automatic Speech Recognition (ASR) inference often leaves hardware resources dormant. A single ASR request typically occupies only 15% to 20% of a GPU's total compute capacity, leaving 80% of the hardware idle. This inefficiency forces organizations to deploy more instances than necessary to meet throughput demands. By transitioning from traditional time-slicing to NVIDIA CUDA Multi-Process Service (MPS), teams can consolidate workloads that previously required 16 GPU instances down to just four, resulting in a 75% reduction in infrastructure requirements.

Standard CUDA time-slicing forces processes to wait in a queue, granting them exclusive access to the GPU. This creates overhead due to frequent context switching, where the GPU processes only one task at a time while other resources sit idle. For lightweight models like the Parakeet TDT 0.6B V2, this bottleneck prevents the full utilization of the 142 Streaming Multiprocessors (SM) available on the hardware.

NVIDIA CUDA MPS overcomes these limitations by acting as a binary-compatible implementation of the CUDA API. It allows multiple processes to share a single GPU context, enabling parallel execution. By partitioning the GPU into multiple concurrent execution instances, MPS maximizes SM occupancy and significantly increases Requests Per Second (RPS) within a fixed hardware footprint.

Triton Inference Server and MPS Integration

To bridge the gap between resource allocation and performance, developers can map NVIDIA Triton Inference Server scheduling directly to MPS partitions. In a production environment, transcription instances are allocated 25% of the SMs to run four concurrent processes, while diarization instances are assigned 12% of the SMs to run eight concurrent processes. Each transcription instance consumes approximately 2.5GB of VRAM, with diarization instances using about 1.8GB.

This setup utilizes Triton’s dynamic batching for transcription tasks and sequence batching for diarization, favoring batch sizes of [4, 8, 16] with a maximum latency allowance of 50,000 microseconds. To deploy this architecture, the system requires a specific sequence of operations to initialize the environment:

bash

(1) start the CUDA MPS daemon
(2) run auto_config.py to set the Triton instance count and SM percentage
(3) launch tritonserver

By integrating Triton’s request scheduler with the GPU partitioning layer, the system minimizes idle cycles. Detailed implementation guides and build instructions are available in the official repository.

Optimizing Compute with ONNX Runtime and TensorRT

Performance in an ASR pipeline depends on how efficiently model components occupy hardware. This optimization strategy employs ONNX Runtime for the compute-intensive Conformer encoder, utilizing NVIDIA TensorRT as the execution provider to accelerate processing. This approach converts model graph nodes into hardware-specific kernels, significantly reducing execution time.

TensorRT performs kernel fusion—combining multiple operations into a single step to reduce memory access—and applies FP16 precision. This optimizes the 24 layers and 1024 hidden dimensions of the Conformer encoder on NVIDIA L40S GPUs. For the RNN-T decoder, which requires the flexibility to handle variable-length token generation, the pipeline switches to a PyTorch CUDA native environment. By utilizing CUDA graph caching, the system records repetitive operation calls to minimize overhead, effectively balancing static acceleration with dynamic generation requirements.

Deployment Architecture on Amazon EC2

Deploying on Amazon EC2 g6e.4xlarge and g7e.4xlarge instances involves a three-tier pipeline consisting of a FastAPI gateway, the Triton inference server, and the MPS daemon, all containerized via Docker Compose. The FastAPI gateway acts as the entry point, providing an OpenAI Whisper-compatible REST API that converts audio files into 16kHz mono float32 tensors before passing them to Triton via gRPC.

This architecture allows the system to maintain sub-second latency while processing 92.1 requests per second per GPU. To deploy this in a production environment, developers can clone the repository and execute the following commands: