The modern data engineer often lives in a state of perpetual backlog, caught between the urgent demands of business analysts and the rigid constraints of legacy infrastructure. For the team at F1, this tension reached a breaking point within their Customer 360 marketing technology platform. The environment was a fragmented landscape where data engineers, scientists, and analysts operated in silos, each using different accounts and disparate environments. As the team attempted to integrate 12 critical data sources, the friction of manual engineering created a bottleneck so severe that it resulted in an 18-month backlog of pending tasks. The gap between a business requirement being identified and that data actually appearing in a dashboard had become a multi-week odyssey.
The Architecture of the Data Accelerator
To dismantle this bottleneck, F1 partnered with AWS to build the Data Accelerator, an agentic AI solution powered by Amazon Bedrock AgentCore. The primary objective was to transform the manual process of data onboarding into an automated pipeline capable of generating infrastructure code, data transformation logic, and governance policies autonomously. By moving away from a manual maintenance system toward an observable, integrated data environment, F1 aimed to minimize human intervention in the repetitive stages of the pipeline.
The results were immediate and quantifiable. The AI agents now handle 95% of the onboarding workload autonomously. What previously took eight weeks of manual effort has been compressed into approximately 40 minutes of automated code generation, followed by a few hours of human deployment and review. This shift fundamentally altered the workflow: instead of engineers designing and implementing from scratch, they now act as auditors who review and approve AI-generated outputs.
The technical execution begins when a team member uploads a Business Requirements Document (BRD) to an Amazon S3 bucket. This action triggers an AWS Lambda function, which in turn calls the Amazon Bedrock AgentCore Runtime. The agent analyzes the text within the BRD to extract system configuration parameters and generates a standardized configuration file. To ensure traceability and version control, the agent uses a GitHub App to create a Pull Request (PR) in a standardized Git repository and issues a Jira ticket via REST API to link the PR to a specific task. Every decision made by the agent and every API call is logged in Amazon CloudWatch, providing a full audit trail for post-verification.
Once the initial configuration file is approved by an engineer, the second stage of the automation pipeline activates. Using the approved configuration as a single source of truth, the system initiates a branching process that creates PRs across three distinct repositories. The first is the infrastructure repository, which defines the necessary cloud resources. The second is the DBT repository, which handles the data transformation logic. The third is the governance repository, which defines data access permissions and security protocols. This structure allows F1 to maintain physical separation between different layers of the data pipeline while ensuring that a single business requirement is applied consistently across the entire technical stack. All three PRs are linked to the original Jira ticket, allowing the engineer to review infrastructure, DBT, and governance changes in one unified view.
From Code Generation to Autonomous Governance
While automating boilerplate code is a significant win, the true power of the Data Accelerator lies in its ability to handle complex regulatory requirements. F1 transitioned its manual data column analysis and GDPR (General Data Protection Regulation) classification tasks to the AI agent. The agent now automatically tags data as personal information, sensitive information, or pseudonymized information. These tags are then published directly to the governance registry within SageMaker Unified Studio, providing real-time visibility into data compliance.
This capability is driven by a modular skill definition framework. Rather than relying on a single, monolithic prompt, the system utilizes independent functional units for specific tasks such as schema mapping, quality validation, and governance enforcement. These modules can be flexibly combined and called based on the context of the request. This modularity ensures that if a complex compliance rule changes, the team only needs to update the specific module rather than redesigning the entire pipeline, drastically reducing long-term maintenance costs.
To ensure the precision required for legal compliance, F1 implemented a multi-pass reasoning structure. The process is divided into three distinct stages. In Pass-0, the agent performs token scrubbing to remove unnecessary noise from the data. In Pass-1, it summarizes the outputs from the available tools. Finally, in Pass-2, it synthesizes all previous information to produce a final, comprehensive evaluation. While this multi-pass approach increases total processing time compared to a single-response model, it significantly enhances the accuracy of data classification and strengthens the organization's regulatory posture.
Proactive Schema Evolution and the New Engineering Role
Beyond the initial onboarding, the Data Accelerator addresses the volatility of live data sources through an event-driven schema evolution system. By combining AWS Lambda with Amazon EventBridge, F1 built a trigger mechanism that detects changes in data source structures in real time. When a column is added or a data type is modified, EventBridge captures the event and immediately executes a Lambda function. This shifts the team from a reactive posture—where they would analyze logs after a failure occurred—to a proactive one where they are notified the moment a change happens.
Upon detection, the system performs an impact assessment to determine how the structural change affects downstream pipelines and consumers. By tracing the data lineage, the agent identifies exactly which dashboards or API endpoints will be impacted. In an environment with countless data touchpoints, this automated impact analysis allows the team to identify the propagation path of a potential failure and prioritize responses instantly.
The agent then generates the corrected code to match the new schema, submits a PR to the relevant repository, and issues a Jira ticket to the responsible engineer. The engineer receives a notification containing the change details, the impact analysis, and the proposed code fix. This architecture moves the starting point of problem-solving from root-cause analysis to solution review, reducing the resolution time for schema-related issues from several days to just a few hours.
This evolution fundamentally redefines the role of the data engineer. The primary task is no longer the manual mapping of schemas or the construction of pipelines, but the logical review and approval of agent-generated PRs. The physical labor of writing boilerplate code has been replaced by the intellectual labor of judging the appropriateness of a result. Consequently, the core competency required for the role has shifted. The ability to write code quickly is now less valuable than the ability to define business requirements precisely within a BRD and verify that the AI's output adheres to internal governance and security policies.
For organizations facing massive onboarding backlogs or frequent schema changes, this model offers a blueprint for operational efficiency. However, the critical success factor is the explicit design of a Human-in-the-loop stage. By ensuring that no AI-generated code is deployed without human approval, F1 maintains data integrity and consistency while maximizing throughput. The goal is not to remove the engineer from the process, but to move the bottleneck from the act of creation to the act of verification.
The shift toward agentic workflows transforms the data pipeline from a fragile chain of manual dependencies into a resilient, self-healing system where the engineer serves as the final gatekeeper of quality.
