How a 250M model got trained on one RTX 2060.

Training a 250 million parameter model normally needs more graphics memory than an RTX 2060 has. This is how Kestrel got trained on one anyway.

The problem

The RTX 2060 has about 5.6 GB of memory we can use. Kestrel has 249,396,480 parameters. Training it with standard FP32 AdamW needs 3.72 GB for the weights, the gradients and the optimizer's state alone. That leaves too little of the card for activations, the working memory of each training step, and the run does not survive.

The fix

EPICYCLE, our training-efficiency system, includes a memory-efficient optimizer. It rotates the first moments across a small set of rings and stores the second moments in a factored form, so the optimizer keeps far less state than AdamW. In the configuration we measured on this architecture, weights, gradients and optimizer state came to 2.09 GB, which is 44% less than 3.72 GB. That leaves about 3.2 GB for activations, enough to train.

What the savings cost

The optimizer is an approximation. It does not behave identically to AdamW, and it trades exact dense optimizer state for the memory to fit a bigger model. The figures above cover weights, gradients and optimizer state only, not activations. We measured them for this model on this card, and we are not claiming they transfer to other models or hardware.

Why it matters

All 5.72 billion tokens went through that one card. Kestrel itself is not publicly available, but the training approach is described in our technology notes.