Every enterprise IT department maintains a digital graveyard: the ITSM ticket history. Thousands of resolved incidents sit in databases, containing the exact steps needed to fix a critical server crash or a recurring software bug. Yet, this tribal knowledge rarely makes it into a formal knowledge base. Engineers find themselves solving the same problem for the tenth time, not because the solution doesn't exist, but because it is buried in a fragmented ticket from two years ago. This gap between incident resolution and institutional knowledge is where most operational efficiency dies.

The Closed-Loop Architecture of KnowledgeForge

KnowledgeForge addresses this inefficiency by implementing a closed-loop knowledge lifecycle that transforms raw ticket data into curated technical documentation. The system is split into two primary subsystems: Generation and Curation. The Generation subsystem analyzes clusters of tickets to identify common symptoms and resolution paths, synthesizing them into a single Root Cause Analysis (RCA) document or a knowledge base article. This process turns isolated work notes into a standardized asset that the entire organization can leverage.

Once a document is generated, it enters the Curation subsystem. This stage involves a four-step refinement process: type classification, deduplication, quality scoring, and the rewriting of weak content. A critical aspect of this loop is that the curated vector data is fed back into the Generation stage as a reference for Retrieval-Augmented Generation (RAG). By referencing verified existing knowledge, the system ensures that new documents maintain high accuracy and consistency. The final output is pushed to ServiceNow, where it awaits a final sign-off from a knowledge manager before being published.

At the heart of the generation flow is Anthropic Claude 4.5 Sonnet, orchestrated via Amazon Bedrock. The input data consists of customer-specific ticket themes, keywords, and work notes stored as JSON files in Amazon S3. To prevent the LLM from hallucinating terminology or creating conflicting instructions, KnowledgeForge employs a RAG structure that retrieves the five most similar existing documents from an S3 Vectors index. This ensures that regardless of who wrote the original ticket or when it was created, the resulting documentation adheres to a unified technical vocabulary.

To handle the bursty nature of ticket processing, the entire pipeline runs on Amazon ECS and AWS Fargate. By utilizing a serverless container engine, the system scales compute resources automatically based on the volume of ticket themes requiring processing, removing the need for manual server provisioning. Furthermore, the system implements response streaming from Amazon Bedrock. Rather than waiting for the LLM to complete the entire document, the container assembles the text in real-time as tokens arrive, significantly reducing latency in the high-volume conversion pipeline.

The Shift to Serverless Vector Indexing

While most RAG implementations rely on dedicated vector databases like Pinecone or Milvus, KnowledgeForge makes a strategic pivot to Amazon S3 Vectors. The primary driver here is infrastructure efficiency. Traditional vector databases require maintaining server nodes, which introduces fixed costs and operational overhead. In contrast, S3 Vectors operates on a cost model based on query volume and storage capacity in gigabytes. By storing embedding vectors and metadata directly in S3, the system eliminates the need to manage a separate database cluster while maintaining the ability to perform efficient customer-level filtering.

Every document is processed through Amazon Titan Text Embeddings V2 to create a 1,024-dimensional embedding. These vectors are stored in customer-isolated S3 Vectors indices, allowing the system to perform both document retrieval and deduplication simultaneously. Because vector search calculates semantic similarity rather than keyword matches, KnowledgeForge can identify duplicate documents even if the phrasing differs slightly.

To distinguish between a related document and a literal duplicate, the system uses a strict cosine distance threshold of 0.05, meaning a similarity score of 0.95 or higher is required to trigger a deduplication event. The system utilizes a Top-K 5 configuration to compare the new vector against the five closest matches. If a duplicate is confirmed, the system retains only the most recent version and discards the obsolete entry. This deduplication is handled in a single API call that filters for active documents belonging to a specific customer:

response = s3_vectors_client.query(index_name='customer-index', vector=new_article_vector, top_k=5, filter={'status': 'active'})

This approach provides the economic foundation to assign and manage individual vectors for every document in a massive library without the financial burden of an always-on database server. For those looking to implement similar storage patterns, the Amazon S3 User Guide provides the necessary configuration details.

Managing this massive GenAI pipeline requires a sophisticated orchestration layer to handle dependencies and failures. The process begins with Amazon EventBridge triggering a schedule, which prompts an AWS Lambda function to identify customers with new or modified documents. These tasks are pushed into Amazon SQS FIFO to ensure strict processing order per customer, preventing data contention during large batch runs.

To manage the distribution of these tasks, KnowledgeForge utilizes the Distributed Map feature of AWS Step Functions. This allows the system to fan out thousands of concurrent processing items while maintaining a precise state for each. The Distributed Map handles retries and error logic automatically, ensuring that a single LLM timeout doesn't crash the entire batch. Detailed implementation patterns for this architecture can be found in the AWS Step Functions Developer Guide.

Because LLM calls to Amazon Bedrock can frequently exceed five minutes, the item processors are configured in STANDARD mode to preserve detailed execution histories for debugging. To bypass the payload size limits inherent in Step Functions, the ItemReader reads manifest files directly from S3, allowing the system to pass large amounts of metadata to the processing containers without hitting API limits.

By combining the reasoning power of Claude 4.5 with the serverless efficiency of S3 Vectors and Step Functions, KnowledgeForge transforms the cost center of IT support into a self-sustaining knowledge engine.