All selected work

Deep learning / Model internals / Project notes

ShakespeareGPT

A tiny GPT, built from the inside out. A character-level transformer trained from random weights on Shakespeare, with attention, training, and generation implemented in PyTorch.

PythonPyTorchTransformers
ShakespeareGPTDocumented training run
ShakespeareGPT training and validation loss over 1,000 optimizer steps; validation loss falls from 4.214 to 2.060 nats per character.
4.214 → 2.060Estimated validation loss / nats per character1,000Optimizer steps
PyTorch · RTX 3050 Laptop GPURepository training results

The problem

Understanding a language model means understanding what happens between the input text and the next prediction. I built a small GPT from its component parts to explore that process directly.

How the system works

A 65-character tokenizer feeds learned token and position embeddings into four transformer blocks. Multi-head causal self-attention, residual connections, layer normalization, and feed-forward layers produce next-character predictions.

A key engineering decision

Causal self-attention is implemented with tensor operations instead of a high-level transformer implementation. Training starts from random weights, making the complete learning process visible in the notebook.

Implemented scope

The documented 1,000-step run on an RTX 3050 Laptop GPU reduced estimated validation loss from 4.214 to 2.060 nats per character. The notebook includes loss curves, before/after generation, checkpoint loading, and a causal masking check.

Scope & limitations

This is a small learning model with limited spelling and coherence. Validation loss is estimated over ten fixed sampled batches per split. Checkpoints support inference but do not preserve optimizer state for exact training resumption.

Next project

FitFlow

Explore case study