For most developers, the modern Large Language Model remains a black box. The sheer scale of these systems, often requiring thousands of H100 GPUs and proprietary datasets, creates a psychological barrier that suggests AI is the exclusive domain of trillion-dollar companies. This perception has turned the act of model training into a theoretical exercise rather than a practical skill for the individual engineer. However, a shift is occurring as the community realizes that the fundamental mechanics of a 175-billion parameter model are identical to those of a model small enough to run on a laptop.

The 10M Parameter Blueprint for Local Learning

This workshop transforms the abstract concept of generative AI into a tangible project by building a GPT-style training pipeline from the ground up. The primary objective is to create a model capable of generating text in the style of William Shakespeare, but the real value lies in the architectural constraints. While projects like nanoGPT often aim to replicate the 124M parameter scale of GPT-2, this specific implementation optimizes the model down to a 10M parameter ceiling. This reduction ensures that the entire lifecycle—from data ingestion to text generation—can be completed in under an hour on standard consumer hardware.

The project is structured around three essential files: `model.py`, which defines the neural network architecture; `train.py`, which handles the optimization loop; and `generate.py`, which manages the inference process. To ensure the environment is reproducible and lightweight, the workshop utilizes the uv Python package manager. Users can initialize their environment with a single command:

uv sync

For those without a dedicated local GPU, the pipeline is compatible with Google Colab. Users can simply upload the files and execute the training process using the following command:

!python train.py

The hardware abstraction layer is designed to be agnostic, automatically detecting and utilizing Apple Silicon GPU via MPS, NVIDIA GPUs via CUDA, or falling back to the CPU. Performance varies based on the chosen model scale. On an M3 Pro chipset, the Tiny model with 0.5M parameters trains in approximately 5 minutes, the Small model with 4M parameters takes about 20 minutes, and the Medium model with 10M parameters completes in roughly 45 minutes.

Why Character-Level Tokenization Changes the Game

The critical divergence between this educational pipeline and industrial-scale LLMs lies in the tokenization strategy. Most commercial models employ Byte Pair Encoding (BPE), which breaks text into frequent chunks of characters to manage massive vocabularies efficiently. However, BPE is counterproductive for small-scale learning on limited datasets. When the vocabulary is too large relative to the data, the model struggles to find meaningful statistical relationships between tokens.

By switching to a character-level tokenizer, this workshop fixes the vocabulary size at exactly 65 characters. This design choice allows the model to effectively learn patterns even from a dataset as small as 1MB. The data flow begins by converting input text into token IDs, which then pass through token embeddings and positional embeddings to provide the model with a sense of sequence. These inputs travel through multiple Transformer blocks, where the model calculates the probability of the next token via logits.

To prevent the common pitfalls of training small models, such as vanishing or exploding gradients, the pipeline integrates several stability mechanisms. LayerNorm is used to normalize the activations, while a Multi-Layer Perceptron (MLP) handles the non-linear transformations. The optimization is driven by the AdamW algorithm, paired with gradient clipping to ensure that weight updates remain stable. Once trained, the model generates text using temperature settings to control randomness and top-k sampling to filter out low-probability candidates, ensuring the output remains coherent.

This implementation draws its technical lineage from the nanoGPT project, the foundational Attention Is All You Need paper, and the TinyStories research, which proves that small models can exhibit sophisticated linguistic capabilities if the data is curated correctly.

Moving from observing benchmarks to writing the actual training loop replaces abstract intuition with technical certainty. The ability to manipulate parameters and witness the immediate effect on text generation is the only way to truly understand the fragile balance between data quality and model scale.