Language Modelling

Transformer Implementation [code]

KV Cache [code]
Stateless transformer design with external cache management
Rotary Positional Embeddings [blog] [code]
Both interleaved and half-flipped rotations; direct application and factory patterns for different use cases

Toy Models of Superposition
Reproduce 5→2→5 experiments, plotting feature directions in the compressed activation space
Sparse Autoencoders
ReLU, TopK and BatchTopK implementations. Trained to recover features from toy models

BPE Tokeniser [blog] [code]
6 training optimisations to go from 8 hours to 13s.
Test Suite [code]
Comprehensive testing for transformer and tokeniser implementations

Name	Name	Last commit message	Last commit date
Latest commit History 62 Commits 62 Commits
.devcontainer	.devcontainer
.vscode	.vscode
configs	configs
data	data
scripts	scripts
src	src
tests	tests
.gitignore	.gitignore
LICENSE	LICENSE
README.md	README.md
pyproject.toml	pyproject.toml