For companies managing massive fleets of AI-powered hardware, the cost of real-time inference is often dictated by the memory footprint of the underlying infrastructure. Tomofun, the company behind the Furbo pet-monitoring smart camera, found itself constrained by a 32 GB memory requirement for its GPU-based inference workloads. As the number of connected devices scaled into the hundreds of thousands, the overhead of maintaining high-memory GPU instances became a significant financial burden. Rather than attempting to rewrite the core architecture of their AI models, the engineering team pivoted to a hardware-centric strategy, migrating their inference pipeline to AWS Inferentia2, Amazon’s custom-designed silicon for machine learning.

BLIP Model Modularization and Compilation Strategy

The core of Tomofun’s AI stack is the BLIP (Bootstrapping Language-Image Pre-training) model, a vision-language model that integrates image understanding and generation, as detailed in the original research paper. To make this model compatible with the specialized architecture of AWS Inferentia2, the team broke the monolithic BLIP structure into three distinct components: the image encoder, the text encoder, and the text decoder. By isolating these modules, the team could independently compile each part using torch_neuronx, the library designed to execute PyTorch models on Neuron-based hardware. To ensure the pre-trained logic remained untouched, the team implemented lightweight wrapper classes that act as adapters, standardizing the input and output formats for each component.

Optimization Through Inference Wrappers

Transitioning from a unified GPU execution block to a component-based pipeline required a shift in how the model handles data flow. The wrappers serve a critical function: they satisfy the specific tensor input and output requirements mandated by the `torch_neuronx.trace()` API. This approach allows the existing PyTorch codebase to remain intact while preparing the model for deployment in an Inferentia2 environment. The following snippet demonstrates how the text encoder is compiled and prepared for the Neuron runtime:

python

텍스트 인코더를 Neuron 최적화 형식으로 컴파일

models.text_encoder = TextEncoderWrapper.from_model(

torch.jit.load(os.path.join(directory, 'text_encoder.pt')))

Real-World Gains in Inference Efficiency

The most significant shift for the development team is the increased portability of the model. During the compilation phase, the system references the original sub-modules to perform hardware-specific optimizations, while the wrapper classes manage the data pipeline during deployment to ensure stability. By decoupling the model from the underlying hardware, Tomofun maintains the same level of accuracy and throughput as their previous GPU-based setup, but at a substantially lower cost. This modular architecture proves that scaling AI services does not necessarily require a complete overhaul of the model’s internal logic, but rather a strategic approach to how that logic interfaces with specialized hardware accelerators.

Hardware acceleration can be achieved through architectural modularity without the need for complex, model-breaking code modifications.