OpenAI Parameter Golf Competition – Adaptive Recurrent Transformer
Not every token deserves equal compute.
Standard language models allocate depth uniformly. Yet intuitively, some tokens are easier to predict than others. Take the following fragment:
“After lunch, he was looking forward…”
As humans, we easily recognize the next word to likely be “to”. Accordingly, I approached this challenge with a singular thesis: not every token deserves equal compute – and dynamic recurrence can provide this adaptability.
My initial idea was to use universal transformers as a sort of artificial thinking time; each recursive step refines the output. Taking some lessons from diffusion models, I implemented a universal transformer with Fourier step embeddings. I observed diminishing returns from additional recursion and theorized that gains accrue mostly for “hard” tokens. To exploit this, I trained a gating network to stop recursion early by predicting delta-loss from the current token sequence and step embedding.
I eventually determined my goal to be broader than stopping recursion. I sought to develop a deep, memory-efficient architecture that still benefits from the step-specific expressivity of a traditional LLM. After all, transformers act differently depending on the input; why shouldn’t architectures be variable as well?
| Variation | Best BPB | Hardware |
|---|---|---|
| Soft ART on PR1855 | 1.0643 | 8xH100 |
| Simple ART on PR1855 | 1.1999 | 1xH100 |
| Adaptive Recurrent Transformer 1 | 1.3168 | 1xH100 |
Selected versions of the ART architecture. Scored in bits per byte (BPB) compression, lower is better. 10 minutes of compute for all trials.
