Modern developers building search interfaces often face a frustrating trade-off between latency and intelligence. To implement semantic search, the standard pipeline requires sending a user's query to a remote inference server, waiting for a high-dimensional embedding vector to return, and then querying a vector database. Even with optimized APIs, the network round-trip time creates a perceptible lag that kills the fluidity of search-as-you-type experiences. This dependency not only introduces a single point of failure but also forces sensitive user data to leave the local device, complicating privacy compliance and increasing operational costs.

The Architecture of Localized Embeddings

Ternlight solves this bottleneck by moving the entire embedding pipeline into the client side. It integrates the inference engine, the model weights, and a BERT tokenizer into a single .wasm file. This design allows the tool to execute immediately upon loading, eliminating the need for separate fetch requests or complex post-install steps during runtime. The system is delivered in two distinct tiers to balance the tension between precision and performance. The quality-focused @ternlight/base model occupies 7.2MB when gzipped, while the high-speed @ternlight/mini version is stripped down to 5.0MB.

Both tiers output 384-dimensional L2 normalized Float32Arrays, supporting a maximum input length of 128 tokens, which translates to roughly 95 words. The developer interface is stripped down to three essential primitives. The embed function converts a raw string into an embedding vector, cosineSim calculates the similarity between two vectors, and similar performs a nearest-neighbor search across a corpus, allowing developers to specify the number of results via a topK option. To ensure maximum portability, Ternlight supports Node.js 18 and above, standard web browsers via bundlers, Cloudflare Workers, Vercel Edge, Deno, and Bun, utilizing optimized loaders for each specific environment.

The Ternary Weight Breakthrough

The true technical shift in Ternlight lies in its departure from traditional floating-point arithmetic. Most embedding models rely on heavy matrix multiplications that demand significant CPU or GPU resources. Ternlight instead utilizes a distillation process, using all-MiniLM-L6-v2 as a teacher model, and applies Quantization-Aware Training (QAT) based on the BitNet b1.58 architecture. This approach compresses the model size by approximately 30 times while maintaining a surprising level of accuracy.

By implementing ternary weights, Ternlight restricts every weight in the model to one of three values: -1, 0, or +1. This architectural choice fundamentally changes the underlying math of the inference process. Instead of performing complex floating-point multiplications, the engine replaces these operations with simple additions and subtractions. Because the model is trained as a ternary system from the start, it avoids the typical quality collapse associated with post-training quantization. To further accelerate this, the inference engine is written in Rust and compiled to WASM SIMD, allowing the CPU to process these additions and subtractions in parallel using vector instructions without requiring a GPU or NPU.

Performance benchmarks on M-series Mac hardware illustrate the efficiency of this approach. The @ternlight/mini tier, featuring a 2-layer architecture with a d_model of 256, 4 heads, and 9.5M parameters, achieves a p50 latency of 2.5ms and a single-threaded throughput of 400 embeddings per second. It maintains a Spearman correlation of 0.820 and a SciFact NDCG@10 of 0.439. The @ternlight/base tier, which scales up to a d_model of 384, 6 heads, and 15.4M parameters, offers higher precision with a Spearman correlation of 0.844 and a SciFact NDCG@10 of 0.465, though it increases p50 latency to 5.1ms and reduces throughput to 195 embeddings per second.

This shift eliminates the network round-trip entirely, enabling a level of responsiveness that was previously impossible for semantic search. For developers, the implementation is a simple package installation:

npm install @ternlight/base

Once installed, the logic for a local semantic search is concise:

import { embed, similar } from '@ternlight/base';

const recipes = [ /* list of embedded documents */ ];

const results = similar('easy weeknight dinner ideas', recipes, { topK: 3 });

Beyond speed, this architecture provides a critical privacy advantage. Since the query and the document corpus never leave the user's device, the risk of data leakage is eliminated, making it an ideal choice for applications handling sensitive personal or corporate information. From an infrastructure perspective, deploying the model on edge runtimes like Cloudflare Workers or Vercel Edge allows the request handler and the embedding model to coexist in the same execution context, further simplifying the stack. Because the ternary operations are highly efficient on ARM cores, Ternlight can even run semantic search on low-power IoT hardware like Raspberry Pi or industrial kiosks without dedicated AI accelerators.

Developers must simply choose their tier based on the specific needs of their application: @ternlight/mini for maximum speed and smallest bundle size, or @ternlight/base for superior retrieval quality. Since both tiers share the same API, switching between them requires only a change in the import statement.

Local AI is moving from simple keyword matching to full semantic understanding without the cloud tax.