The modern AI developer is currently fighting a silent war against the noisy neighbor. In any production environment where multiple users share a pool of LLM resources, a single power user or a runaway recursive loop can consume the entire request quota in seconds. This results in a catastrophic failure where the service becomes unresponsive for everyone else, not because the infrastructure is down, but because one identity has monopolized the throughput. For teams managing multi-tenant AI applications, the challenge has shifted from simply scaling the backend to implementing sophisticated traffic shaping that ensures fairness without sacrificing performance.

The Architecture of the Bedrock AgentCore Gateway

Amazon Bedrock AgentCore addresses this instability by introducing a comprehensive rate-limiting framework within its gateway. As a fully managed serverless AI gateway, AgentCore serves as the single secure entry point for AI traffic, removing the need for developers to build and maintain their own custom proxy infrastructure. The gateway acts as an intelligent router, distributing incoming requests across a variety of targets including managed web searches, knowledge bases, LLMs, agents, and HTTP endpoints. A critical component of this ecosystem is the Model Context Protocol (MCP) server, which allows AI models to connect to external data sources and tools using a standardized interface. By centralizing these connections, AgentCore can enforce security policies and traffic controls at the very edge of the infrastructure.

Traffic control within the gateway is applied across three primary target types: inference targets, MCP targets, and HTTP passthrough targets. To provide granular control, the system tracks three distinct metrics. The first is Requests Per Minute (RPM), which limits the raw volume of calls. The second is Concurrent Session count (CPS), which manages the number of active, open connections—a vital metric for streaming responses. The third is token throughput, which monitors the actual volume of text processed over a specific duration. By combining these metrics, operators can prevent backend overload during traffic spikes and maintain a consistent latency profile for all users.

These limits are not applied blindly but are tied to identity. The gateway leverages OAuth and IAM (Identity and Access Management) credentials to define who is making the request. Whether using OAuth for external customer authentication or IAM for internal organizational permissions, the system maps every request to a specific identity, allowing the gateway to block excessive resource consumption at the account or user level before the request ever reaches the expensive inference engine.

Dimension Keys and the Logic of Traffic Bucketing

To manage this traffic at scale, Bedrock AgentCore employs a system of Dimension Keys and Entries. A Dimension Key acts as the identifier for a traffic bucket. When a request hits the gateway, the system scans the request context for these keys to determine which bucket the traffic belongs to. Supported keys include targetName, toolName, and qualifiedModelId. For more complex identity-based routing, the gateway can extract values from JWT claims using `$.context.jwt.<claim>`, or use IAM identifiers such as `$.context.iam.principal` and `$.context.iam.sourceIdentity`.

Entries are the actual rules that define the throughput for these buckets. The system supports a hierarchical matching logic where explicit names take precedence over wildcards. For instance, if an operator defines a specific entry for a Booking MCP server with a limit of 100 RPS (Requests Per Second) and sets a wildcard entry for all other targets at 10 RPS, the Booking server will enjoy high-speed processing while every other target is isolated into its own 10 RPS bucket. This ensures that a surge in requests to a minor tool cannot starve a mission-critical service of its allocated bandwidth.

However, these customer-defined limits do not exist in a vacuum. Bedrock AgentCore implements a two-stage verification process that compares customer-defined rate limits against AWS Service Quotas. The final throughput is always determined by the lower of the two values: `min(customer-defined limit, service managed limit)`. If a developer sets a limit of 100 RPM but the AWS account quota is capped at 50 RPM, the effective limit remains 50. This hierarchical structure acts as a safety valve, ensuring that a configuration error—such as accidentally entering an extra zero in a limit field—cannot crash the underlying AWS infrastructure. While service quotas act as the absolute ceiling, they can be increased via the AWS Service Quotas console if the business scale requires more headroom.

Solving the Intra-Group Monopoly with Dual-Layer Control

The true innovation in the AgentCore approach is the shift from simple capping to a dual-layer control model. In many enterprise setups, users are grouped by tier, such as Basic, Advanced, and Beta. In a traditional system, a Basic group might share a pool of 100 RPM. The flaw here is that one aggressive user in the Basic group could consume 80 RPM, leaving only 20 for everyone else in that tier. This is the intra-group monopoly problem.

Bedrock AgentCore solves this by applying an AND semantic to group and individual limits. For example, the Basic group is assigned a shared bucket of 100 RPM and 50 CPS. Simultaneously, the system applies a per-user limit using the `$.context.jwt.sub` claim as a dimension key. Even if the group pool has plenty of capacity, an individual user might be capped at 20 RPM and 10 CPS. If that user hits their 20 RPM limit, the gateway returns a throttling response immediately, regardless of how much of the 100 RPM group quota remains.

This creates a fair-share environment. The group ceiling prevents a specific tier from exhausting the entire system's resources, while the individual ceiling prevents a single user from dominating their peers. This is particularly important for the Beta group, which is often granted higher CPS limits to handle heavy streaming workloads during benchmarking. By isolating these users into their own high-capacity buckets, AWS ensures that experimental benchmarking does not interfere with the stability of the production Advanced or Basic tiers.

Enterprise Governance and Identity Integration

For large-scale deployments, Bedrock AgentCore integrates with Microsoft Entra ID to handle identity provision and JWT issuance. The gateway validates these tokens and enforces Role-Based Access Control (RBAC) policies, ensuring that users can only access the models and targets permitted by their organizational role. The AgentCore Identity service further manages the issuance of tokens for requests heading toward external targets, creating a secure chain of custody for every AI interaction.

To achieve maximum precision, operators can use multi-dimensional keys. By combining the target name with the JWT role, the gateway can create highly specific traffic lanes:

dimensionKeys: ["targetName", "$.context.jwt.role"]

In this configuration, a Basic user hitting their quota for a specific model will not affect an Advanced user accessing the same model, as they are operating in entirely different buckets. This level of granularity transforms the AI gateway from a simple pass-through into a sophisticated governance tool. All these configurations are managed via the AWS CLI, allowing infrastructure teams to treat their traffic shaping as code.

Ultimately, the move toward dual-quota systems signals a maturation of AI infrastructure. As LLMs move from experimental playgrounds to core business logic, the ability to predict, limit, and guarantee resource availability becomes as important as the model's intelligence itself. Developers must now balance their own custom limits with the overarching account quotas to accurately forecast the performance and cost of their AI pipelines.