The current era of generative AI is defined by a pursuit of statistical perfection. When we prompt a frontier model for an image, we typically receive a high-fidelity, polished result that feels mathematically averaged—smooth gradients, perfect symmetry, and a sterile clarity. Yet, a recent video by Surya Narreddi, which garnered 1.5 million views, captured the public's imagination by doing the exact opposite. Instead of a diffusion-based masterpiece, it showed an AI painstakingly writing JavaScript code to paint a watercolor. The charm lay in the imperfection: the bleeding edges, the translucent washes, and the palpable sense of a digital brush hitting a virtual canvas. This shift from pixels to process suggests that the next frontier of AI creativity isn't found in larger datasets, but in the precise orchestration of constraints and reinforcement learning.

The Architecture of an Artistic RL Loop

To move this experiment from a viral video to a reproducible framework, a full open-source pipeline has been constructed using TRL (Transformer Reinforcement Learning) and OpenEnv. The goal is not merely to provide a set of weights, but to offer an end-to-end ecosystem hosted on Hugging Face that includes the reference pool datasets, the RL environment, and the training scripts. This allows developers to see exactly how a model is incentivized to prefer a specific aesthetic over another.

The technical loop begins when the model generates approximately 150 lines of JavaScript code. This code is not interpreted as text but is executed within a Headless Chromium environment. This browser-based sandbox is critical because it provides the physical ground truth; the code must be rendered into an actual image before any evaluation can occur. This environment also acts as a security gate, ensuring the model does not attempt to bypass the artistic constraints through programmatic shortcuts or invalid API calls.

Once the image is rendered, it enters a dual-stage reward system. The first layer is HPSv3, a 7B-parameter open preference model. HPSv3 functions as a proxy for general human taste, scoring the image based on how a broad demographic would likely perceive its quality. Because it was trained on massive pairs of image preferences, it provides a stable, baseline metric of aesthetic appeal. However, general appeal is not the same as a specific artistic style. To inject a particular watercolor identity, the pipeline employs a second reward model: Qwen3-VL-30B-A3B-Instruct, accessed via HF Inference Providers.

Unlike HPSv3, Qwen3-VL acts as a Pairwise Judge. It does not assign a raw score but instead compares the model's output against four randomly selected images from a curated reference pool. The judge is given specific criteria to look for, such as translucent washes, soft edges, and characteristic pigment bleeds. By swapping the order of images and comparing them in pairs, the system calculates a win rate. The final reward is a weighted mix of the general HPSv3 score and this specific pairwise win rate, guiding the small LLM to refine its code until it satisfies both general beauty and the creator's specific artistic vision.

Constraint as a Catalyst for Style

One of the most counterintuitive aspects of this pipeline is the aggressive restriction of the model's vocabulary. The p5.brush library, which simulates the physics of watercolor on paper, offers 47 different methods. In a typical LLM implementation, providing the full API documentation would be the logical choice. However, this often leads to hallucinations or the model using tools that clash with the desired aesthetic. Instead, the developers implemented a strict allowlist, permitting only 10 specific functions:

scaleBrushes, noStroke, fill, noFill, fillBleed, fillTexture, beginShape, vertex, endShape, circle

By stripping away 37 methods—particularly those related to sharp line drawing or hatching—the system forces the model to think in terms of shapes and fills. In watercolor, the beauty comes from the absence of hard boundaries. When the model is forbidden from drawing a line, it must rely on the library's automatic bleeding effects applied to filled shapes. This constraint transforms the model's behavior from a general-purpose coder into a specialist that understands the physics of pigment diffusion.

This approach allows for a level of precision that prompt engineering cannot match. In a text-to-image model, adding the word watercolor to a prompt is a suggestion that the model interprets statistically. In this RL pipeline, the style is baked into the grammar. For example, when the model calls `fillBleed(0.25)`, it is directly controlling the diffusion of the ink, simulating the water-to-pigment ratio and the absorbency of the paper. To achieve depth, the model is instructed to layer its work, using `beginShape` and `vertex` to create a large base path and then overlaying two or three smaller, more opaque paths on top. This layering creates the natural variance in saturation and transparency that defines professional watercolor art.

From Prompt Engineering to Taste RL

The transition from HPSv3-only training to a Reward Mix reveals a fundamental truth about AI alignment: curation is more powerful than hyperparameter tuning. Initial tests focused on HPSv3 to ensure the pipeline was stable and the reward signals were healthy. Once the baseline was established, the focus shifted to the reference pool. The quality of the final output was determined not by the learning rate or the batch size, but by which images the human curator decided were beautiful. This mirrors the work of artists like Anna Ridler, who treat the curation of a dataset as the primary act of artistic creation.

This methodology solves a recurring problem in LLM-based tool use. When models are given exhaustive documentation, they often invent non-existent methods. By replacing documentation with a verified allowlist and refining it through 200 iterations of Guided Agent Optimization (GEPA), the developers eliminated hallucinations. The model no longer guesses how to use the library; it is physically unable to use it incorrectly.

Ultimately, this project demonstrates that small models can achieve domain-specific expertise that rivals frontier models if the reward function is designed around taste rather than truth. By defining a style as a set of programmatic constraints and a curated preference pool, the pipeline moves beyond the unpredictability of prompting. It proves that RLHF (Reinforcement Learning from Human Feedback) can be applied to the subjective realm of art, provided there is a clear reference point to guide the descent. The result is a system that doesn't just simulate the look of a painting, but learns the logic of the medium.