For years, the most capable large language models have existed behind a curtain of proprietary APIs. Enterprises facing strict data residency requirements or needing deep architectural customization have been forced to choose between the convenience of a closed-source giant and the limited capabilities of smaller, open-source alternatives. The industry has long awaited a tipping point where a trillion-parameter class model could be hosted on private infrastructure without sacrificing the reasoning depth typically reserved for the world's most guarded AI systems.

The Architecture of a 2.8 Trillion Parameter Behemoth

Moonshot AI shifted this landscape on July 27, 2026, with the official release of Kimi K3. This is not merely another incremental update but a massive leap in open-weights availability, featuring a staggering 2.8 trillion parameters. To make a model of this magnitude computationally feasible, Moonshot AI utilized a Mixture of Experts (MoE) architecture. In this design, the model does not activate its entire parameter set for every single token; instead, it selectively engages a subset of its expert networks based on the input. Specifically, Kimi K3 consists of 896 expert models, but only 16 are activated per token. This means that while the total knowledge capacity is 2.8 trillion parameters, the actual compute cost per forward pass is limited to approximately 104 billion active parameters.

The model weights are publicly accessible via the moonshotai/Kimi-K3 repository on Hugging Face, allowing organizations to bypass external API dependencies entirely. Beyond raw size, Kimi K3 is engineered for high-complexity reasoning and agentic workflows. It introduces a dedicated Thinking Mode, which allows the model to execute internal reasoning steps and self-correct logical errors before delivering a final response. This is paired with native tool calling and structured output capabilities, enabling the model to interact with external APIs and backend systems using strict formats like JSON, which significantly reduces the friction of integrating AI into production data pipelines.

The Efficiency Paradox and the B300 Requirement

Deploying a model with 2.8 trillion parameters would typically be a memory nightmare, but Kimi K3 solves this through a combination of aggressive quantization and architectural innovation. The model is distributed in MXFP4 (Microscaling Floating Point 4-bit) format. Unlike standard 4-bit quantization, MXFP4 applies scaling factors to small blocks of data, preserving numerical precision while drastically reducing the memory footprint. This allows the model to maintain high-quality inference while fitting within the constraints of modern GPU VRAM.

To further optimize throughput, Moonshot AI implemented Kimi Delta Attention (KDA), Gated Multi Head Latent Attention (MLA), and the Stable LatentMoE framework. These three components work in tandem to alleviate memory bottlenecks and increase scaling efficiency. When compared to its predecessor, Kimi K2, Kimi K3 demonstrates a 2.5x improvement in scaling efficiency. However, this efficiency does not mean the hardware requirements are trivial. To run Kimi K3 at production speeds, the infrastructure must provide massive interconnect bandwidth to handle tensor parallelism across multiple GPUs.

This is where the NVIDIA B300 Blackwell Ultra GPU becomes mandatory. The recommended deployment target is the `ml.p6-b300.48xlarge` instance on AWS, which houses eight B300 GPUs. Because these instances are in high demand, they cannot be acquired via standard on-demand requests. Instead, engineers must use the Flexible Training Plan (FTP) to reserve B300 GPU nodes in advance through the SageMaker console, ensuring that the cluster is available for the specific duration of the deployment.

Orchestrating the Inference Pipeline

Managing the deployment of a 2.8 trillion parameter model manually would be an operational disaster. To solve this, the deployment process is abstracted through SageMaker HyperPod and the Inference Operator. The Inference Operator automates the heavy lifting of container orchestration, model loading, and endpoint management. Rather than manually configuring Kubernetes pods or moving massive weight files across servers, the operator handles the lifecycle of the model deployment.

For the inference engine, Kimi K3 relies on vLLM, a high-performance library optimized for LLM serving and memory management. Because Kimi K3 requires specific optimizations not yet present in the main vLLM branch, developers must use a dedicated container image:

vllm/vllm-openai:kimi-k3

Once the infrastructure is provisioned, the actual deployment is triggered via a configuration file. By applying the `kimi-k3.yaml` file through kubectl, the entire environment is stood up with an OpenAI-compatible API:

kubectl apply -f kimi-k3.yaml

This compatibility is critical for adoption. It means that any existing application designed for OpenAI's API can be migrated to a private Kimi K3 instance simply by changing the base URL, eliminating the need to rewrite integration logic or redesign prompt templates.

Building Autonomous Infrastructure with EKS and Terraform

For teams requiring a more autonomous and reproducible setup, the path leads through Amazon EKS and Infrastructure as Code (IaC). By utilizing EC2 Capacity Blocks, teams can secure `p6-b300.48xlarge` instances for specific windows without long-term contracts. The deployment is then codified using Terraform modules from the AI on EKS project, which automatically provisions the VPC network, managed node groups, and the necessary IAM roles and policies. This ensures that the high-speed networking required for GPU-to-GPU communication is configured correctly from the start.

Once the cluster is live, the NVIDIA device plugin and the latest B300 drivers are installed to allow Kubernetes to schedule workloads across the eight GPUs. The final layer of the stack is deployed via Helm charts, where the vLLM container is configured with a tensor parallel size of 8 to distribute the 2.8 trillion parameters across the B300 cluster. To prevent network bottlenecks during the initial boot, model weights are synchronized via Amazon S3, allowing the nodes to pull data directly from high-speed object storage.

With the LoadBalancer or Ingress controller configured to expose port 8000, the model becomes accessible via the `/v1/chat/completions` endpoint. Engineers can verify the deployment instantly using a simple curl command:

curl -X POST <ENDPOINT_URL>/v1/chat/completions

This end-to-end pipeline—from Terraform-led provisioning to vLLM serving—transforms the deployment of a trillion-parameter model from a research experiment into a repeatable engineering process. The ability to host Kimi K3 privately on AWS B300 hardware marks a shift in power, giving enterprises the same reasoning capabilities as the largest closed-source models while maintaining absolute control over their data and infrastructure.

The era of the trillion-parameter API is ending, replaced by the era of the trillion-parameter private cluster.