Most businesses today treat their customer touchpoints as isolated silos. A customer might ask a question via a chat bot, follow up with a voice note, and eventually call a support line, only to find that the human agent on the phone has no record of the previous two interactions. This fragmentation creates a friction-filled experience where the user is forced to repeat their context multiple times. The industry is currently shifting toward a unified multimodal interface where the medium of communication changes, but the intelligence and memory of the agent remain constant.

The Multimodal Engine of Nova 2 Lite and Sonic

Building a seamless agent for a platform with over 2 billion users like WhatsApp requires a backend capable of handling diverse data streams through a single business identity. The architecture leverages Amazon Bedrock to deploy a dual-model strategy tailored to the specific constraints of each communication medium. For text-based interactions, the system utilizes the Amazon Nova 2 Lite model via the Bedrock Converse API. When an inbound webhook receives a text message, Nova 2 Lite analyzes the context and generates a response, which is then routed through a Sender Lambda to the WhatsApp API. The choice of the Lite model is a strategic decision to minimize inference latency and reduce operational costs for high-volume text queries.

Voice notes require a fundamentally different approach to avoid the latency inherent in traditional pipelines. Instead of the standard sequence of Speech-to-Text, LLM processing, and Text-to-Speech, this system employs the Amazon Nova 2 Sonic model. Sonic utilizes a speech-to-speech capability that allows for a direct voice-in, voice-out workflow. The technical pipeline involves a worker downloading OGG Opus bytes from WhatsApp, decoding them into 16kHz PCM, and feeding them directly into a Nova 2 Sonic session. By removing the transcription layer, the agent preserves the speaker's original nuance and significantly accelerates the response time.

Despite the different model paths for text and voice, the system maintains a unified output channel. All generated responses, regardless of whether they originated from Nova 2 Lite or Sonic, pass through the same Sender Lambda. This ensures that while the processing logic is optimized for the medium, the final delivery mechanism remains consistent and manageable.

Decoupling Intelligence from Infrastructure via MCP

The true technical pivot in this architecture is the move away from channel-specific bots toward a centralized agent runtime. Most AI implementations bind the model logic directly to the API of the messaging platform, meaning a change in the business logic requires a rewrite of the bot's integration code. This system solves that by implementing a three-layer separation: the WhatsApp layer for the interface, the agent runtime for context interpretation, and the backend for data management. This structure ensures that the core business logic—handling menus, shopping carts, and order histories—remains untouched even if the company decides to add new communication channels.

At the intersection of the agent and the restaurant backend lies the Model Context Protocol (MCP) gateway. MCP serves as a standardized interface that allows the AI model to access external data sources and tools without needing to know the specific API signatures of the underlying database. If the backend API changes its schema, developers only need to update the MCP gateway configuration rather than retraining the model or modifying the runtime code. This decoupling transforms the AI from a rigid script into a flexible operator capable of interacting with any tool provided by the gateway.

To handle the scale of a global platform, the system employs an asynchronous processing model. The traffic receiver is a single HTTPS webhook that immediately returns a `200 OK` response to WhatsApp to prevent request blocking. The actual heavy lifting—multimodal inference and backend database calls—is handled by a queue and worker system. This prevents the system from hitting WhatsApp's strict response timeouts during periods of high traffic.

For real-time voice calls, the complexity increases with the introduction of WebRTC. When a user initiates a call, the Meta Calling API sends a webhook containing a Session Description Protocol (SDP) offer. Because the runtime often operates in internal networks without public IP addresses, it functions in `turnOnly` mode, relying on relay servers to bridge the connection. The system uses Amazon Kinesis Video Streams (KVS) to manage TURN credentials. Since Meta does not support Trickle ICE, the `aiortc` answerer waits for all network paths to be gathered before sending a single-shot SDP response back to Meta. Once the connection is established, voice data is encrypted via DTLS and SRTP and streamed through the KVS TURN relay, where Nova 2 Sonic provides near-instantaneous speech-to-speech responses.

Deployment is managed through a five-stage AWS CDK workflow that sequentially builds the VPC, the backend stack (including DynamoDB and Location Service), the Gateway and shared memory, the runtime, and finally the webhooks and notification systems. A critical constraint is that all resources must be deployed in the `us-east-1` (N. Virginia) region to ensure access to the latest Nova 2 multimodal APIs. The agent containers are built for ARM64 architecture via AWS CodeBuild and stored in Amazon ECR, allowing for a standardized environment that eliminates the need for local audio toolchain installations.

Security and authentication are handled through the Meta Business Platform. Since Meta's app creation and business verification cannot be automated via AWS APIs, these steps are performed manually in the Meta console. The resulting Meta Access Token, App Secret, and Verify Token are stored in AWS Secrets Manager, ensuring the runtime can maintain a secure connection without hardcoding credentials.

This architecture proves that the future of AI agents is not about building better bots, but about building a unified intelligence layer that treats the interface as a mere commodity.