Most enterprise teams using LLMs like ChatGPT or Claude eventually hit the same wall. They have the intelligence of the model, but the data remains trapped in fragmented spreadsheets and legacy databases. This creates a tedious cycle where employees manually copy-paste project contexts, customer names, and status updates into a prompt just to get a usable answer. The friction is not a lack of AI capability, but a lack of structural access. Corporate data is rarely designed for an AI agent to read, leaving a gap between the model's reasoning and the company's actual records.
Scaling the AI Gateway with MCP and AWS Fargate
Smartsheet addressed this disconnect by implementing a remote Model Context Protocol (MCP) server hosted on AWS infrastructure. MCP serves as a standardized bridge, allowing AI models to interact with external tools and data sources through a consistent interface. By deploying this server, Smartsheet integrated its system with Anthropic's Claude Desktop and Amazon Quick, enabling users to analyze project data, update tasks, and manage workspaces using natural language. This shift transformed the AI from a passive chatbot into an active agent capable of autonomous coordination, such as capturing requirements, assigning tasks to owners, and attaching test results directly within the sheets.
The technical foundation relies on a serverless architecture using Amazon ECS and AWS Fargate. Because AI agents exhibit highly unpredictable traffic patterns, Smartsheet utilized an Application Load Balancer (ALB) to distribute incoming requests across the Fargate cluster. This removes the overhead of managing physical servers and allows the system to scale instantly as demand spikes. To ensure stability during deployment, the team integrated Amazon ECR with an automated CI/CD pipeline. They implemented ECS deployment circuit breakers that monitor container health in real-time, triggering an automatic rollback to the last stable version if a failure is detected during rollout.
Beyond the infrastructure, Smartsheet introduced a specialized AI-optimized interface sitting atop their existing APIs and central intelligence layer. This interface acts as a critical filter, processing corporate data into a format that LLMs can digest more efficiently. By refining the data before it reaches the model, the system minimizes hallucinations and strips away redundant information. This optimization is the primary driver behind the company's reported saving of over 3 billion tokens since the service launched, as it prevents the model from processing unnecessary noise.
Solving the Bursty Traffic and Non-Deterministic Loop
Traditional API traffic is generally predictable, reacting to discrete user clicks. AI agents, however, operate on a bursty pattern. An agent might trigger a sequence of five or six tool calls within a single second to gather context, followed by a period of total silence while the model performs inference. This creates a volatile load that can crash standard proxy servers or lead to massive over-provisioning of resources.
To manage this, Smartsheet applied target tracking policies to their ECS auto-scaling. Rather than scaling based on simple request counts, they implemented compute-aware scaling. Since LLM-optimized serialization is more resource-intensive than simple data proxying, the system monitors actual CPU and memory utilization to ensure performance does not degrade during a burst. This ensures that the infrastructure reacts to the computational weight of the AI's logic rather than just the number of hits to the endpoint.
Testing this environment required a fundamental shift in quality assurance. Because LLM responses are non-deterministic, a tool that works once might fail the next time due to a slight variation in the model's reasoning. Smartsheet solved this by introducing end-to-end (E2E) tests that keep the LLM in the loop. These tests simulate real business scenarios, such as workspace creation or complex data queries, verifying the entire flow from the model's interpretation to the final API execution. To maintain production stability, the team runs canary tests every 15 minutes, executing multi-step MCP workflows through the full authentication and gateway path to detect performance regressions before they impact users.
Governance in an autonomous agent environment requires a balance between autonomy and safety. Smartsheet introduced `readOnlyHint` and `destructiveHint` annotations to its tool definitions. When an AI client encounters a `destructiveHint`, it is programmed to automatically trigger a confirmation prompt for the human user before executing the action. This prevents the agent from accidentally deleting critical data. Access control is managed hierarchically at the organizational level, allowing administrators to toggle between full AI access or restricted, non-destructive operations.
For observability, the entire pipeline is instrumented with OpenTelemetry. Every tool call, including user identity, organization ID, and execution results, is captured as a trace. These logs stream through Amazon Kinesis into Amazon OpenSearch Service, allowing engineers to visualize the chain of calls an agent makes to complete a single request. Infrastructure health is monitored via Amazon CloudWatch and Datadog APM, with PagerDuty handling critical alerts. Furthermore, all call events are routed through Amazon SQS to an intelligence layer, creating a feedback loop that identifies which tools are most utilized and where bottlenecks occur.
Security is handled at the edge using AWS WAF and AWS Shield to block external attacks. Within the VPC, Smartsheet uses private subnets and mutual TLS (mTLS) for service-to-service communication, while an OAuth2 proxy rejects unauthorized requests immediately. To prevent a single user from monopolizing the shared proxy server, they implemented layered rate limiting in AWS WAF. This system tracks traffic based on identity headers rather than just IP addresses, ensuring that a burst of requests from one session does not degrade the service for others. Finally, the system ensures that AI agents cannot be used as a backdoor to access restricted data by synchronizing MCP permissions with the existing UI-level domain services.
The success of an enterprise AI agent depends less on the raw intelligence of the model and more on the efficiency of the data pipeline. By combining the Model Context Protocol with a compute-aware AWS Fargate architecture, Smartsheet proved that the cost and scalability challenges of corporate AI can be solved through rigorous infrastructure design.
