Modern data engineering is currently split into two disconnected worlds. On one side, developers use high-performance tools like SQL or PySpark to filter millions of rows of structured data with microsecond precision. On the other side, they write fragile Python wrappers to send chunks of text to a Large Language Model for classification or extraction, often struggling with rate limits, inconsistent schemas, and massive API bills. This fragmentation forces engineers to build complex glue code that manages the handoff between the rigid world of dataframes and the fluid world of semantic reasoning.
The Architecture of Semantic Querying
fenic introduces a query engine designed to collapse this divide by treating LLM calls as first-class operators within a dataframe model. The engine allows developers to execute PySpark-style operations and semantic operators within a single, unified query model. This approach transforms unstructured sources such as documents, logs, and API responses into type-specified rows and structured workflows. To facilitate this, fenic incorporates built-in AI operators including extract, classify, and summarize, which function as native transformations on the data stream.
Under the hood, the engine relies on a high-performance stack to ensure that the overhead of LLM orchestration does not bottleneck the data processing. It utilizes Polars for high-speed dataframe manipulation and DuckDB for in-process analytical SQL queries. To ensure seamless data exchange between these components without the cost of serialization, fenic employs Apache Arrow. This architecture allows the engine to handle the heavy lifting of data movement while reserving the LLM for the high-value semantic tasks.
To solve the problem of LLM hallucinations and inconsistent output, fenic binds unstructured text to Pydantic schemas. By using Python type hints for data validation, the engine ensures that the results of a semantic operation are returned as structured columns rather than raw strings. This allows the output of an AI operator to be immediately used as an input for a subsequent SQL filter or a join operation, maintaining strict type safety across the entire pipeline. The project is released under the Apache-2.0 license, ensuring it remains accessible for open-source integration.
Bridging the Gap Between Logic and Meaning
The true innovation of fenic lies not in the mere addition of LLM calls to a dataframe, but in the intelligent orchestration of how those calls are executed. In a traditional pipeline, a developer might accidentally send thousands of rows to an LLM only to filter out 90 percent of them in the next step. fenic prevents this inefficiency by implementing a specific execution order where standard filters are always prioritized over semantic filters. By pruning the dataset using traditional logic first, the engine drastically reduces the number of tokens sent to the model, directly lowering operational costs.
Beyond simple filtering, the engine introduces the concept of semantic joins. While traditional joins rely on exact key matches, semantic joins allow the engine to link datasets based on meaning and context. This enables the connection of disparate data sources that may refer to the same entity using different terminology, a task that previously required manual mapping or complex vector database lookups. To further optimize performance, fenic applies automatic batching and caching to LLM requests, ensuring that redundant semantic operations are not repeated across the dataset.
This capability transforms the pipeline from a static script into a dynamic tool. By registering these pipelines in a catalog, fenic allows them to be exposed as tools via the Model Context Protocol (MCP). This means an AI agent can autonomously trigger a complex data pipeline, perform semantic extraction on a massive log file, and return a structured summary to the user without the developer needing to manually define every step of the interaction. The pipeline ceases to be a backend process and becomes a functional capability of the LLM itself.
This integration of rigid data structures and fluid semantic reasoning marks the transition from traditional ETL to semantic data engineering.
