Deep learning / Model internals / Project notes
ShakespeareGPT
A tiny GPT, built from the inside out. A character-level transformer trained from random weights on Shakespeare, with attention, training, and generation implemented in PyTorch.

The problem
Understanding a language model means understanding what happens between the input text and the next prediction. I built a small GPT from its component parts to explore that process directly.
How the system works
A 65-character tokenizer feeds learned token and position embeddings into four transformer blocks. Multi-head causal self-attention, residual connections, layer normalization, and feed-forward layers produce next-character predictions.
A key engineering decision
Causal self-attention is implemented with tensor operations instead of a high-level transformer implementation. Training starts from random weights, making the complete learning process visible in the notebook.
Implemented scope
The documented 1,000-step run on an RTX 3050 Laptop GPU reduced estimated validation loss from 4.214 to 2.060 nats per character. The notebook includes loss curves, before/after generation, checkpoint loading, and a causal masking check.
Scope & limitations
This is a small learning model with limited spelling and coherence. Validation loss is estimated over ten fixed sampled batches per split. Checkpoints support inference but do not preserve optimizer state for exact training resumption.
Next project