Skip to content
More
Share
Explore
Open Problems in Mechanistic Interpretability
The purpose of this doc is to let people quickly browse through problems in Neel’s .
There are filters for difficulty, existing work (blank/not blank), help wanted, currently working on, etc. There are also individual pages accessible from the top/side menu of this doc for you to browse specific difficulty level of problems, an individual category of problems, or a card view of this page.
The “Existing Work” column is for completed posts, papers, or other documents. The “Currently Working On” column is for drafts, brainstorms, and people who want to work on it but haven’t produced anything yet. The “Help Wanted” column is for people who’re working on a problem but would like additional collaborators or mentors. Please specify what help you’re looking for if you add to that column. Please add a date if you’re currently working on something so it’s clear if you expressed interest yesterday or two years ago. Write it up as a comment and I'll approve it fairly promptly - people will still see the comment until then.
This sequence is long! What this means is that not all relevant information is contained in this spreadsheet! There is lots of great context for the sequence as a whole and each section in general, including motivation and useful resources. Some problems are copied word-for-word - many are not.
If you are interested in a problem, please take a look at the problem in the original post before deciding to tackle it! Often it includes relevant context or links that didn’t make it into the spreadsheet for space reasons. I’d also recommend looking at the first part of the relevant post, before seriously tackling one of its problems.
Huge thanks to Neel Nanda for his work in creating this sequence and building the field.
Open Problems
Category
Difficulty
Existing Work
Currently working
Help Wanted?
Search
Category
Subcategory
Difficulty
Number
Problem
Existing Work
Currently working
Help Wanted?
Category
Subcategory
Difficulty
Number
Problem
Existing Work
Currently working
Help Wanted?
Toy Language Models
Understanding neurons
B
1.1
How far can you get deeply reverse engineering a neuron in a 1L model? 1L is particularly easy since each neuron's output adds directly to the logits.
Toy Language Models
Understanding neurons
B
1.2
Find an interesting neuron you think represents a feature. Can you fully reverse engineer which direction should activate that feature, and compare to neuron input direction?
Toy Language Models
Understanding neurons
B
1.3
Look for trigram neurons and try to reverse engineer them. in a 1L model.(e.g, "ice cream -> sundae")
Toy Language Models
Understanding neurons
B
1.4
Check out the SoLU paper for more ideas on 1L neurons to find and reverse engineer.
Toy Language Models
Understanding neurons
C
1.5
How far can you get deeply reverse engineering a neuron in a 2+ layer model?
Toy Language Models
Understanding neurons
A
1.6
Hunt through Neuroscope for the toy models and look for interesting neurons to focus on.
Toy Language Models
Understanding neurons
A
1.7
Can you find any polysemantic neurons in Neuroscope? Explore this.
Toy Language Models
Understanding neurons
B
1.8
Are there neurons whose behaviour can be matched by a regex or other code? If so, run it on a ton of text and compare the output.
Toy Language Models
How do larger models differ?
B
1.9
How do 3-layer and 4-layer attention-only models differ from 2L? (For instance, induction heads only appeared with 2L. Can you find something useful that only appears at 3L or higher?)
Toy Language Models
How do larger models differ?
B
1.1
How do 3-layer and 4-layer attention-only models differ from 2L? Look for composition scores - try to identify pairs of heads that compose a lot.
Toy Language Models
How do larger models differ?
B
1.11
How do 3-layer and 4-layer attention-only models differ from 2L? Look for evidence of composition.
Toy Language Models
How do larger models differ?
B
1.12
How do 3-layer and 4-layer attention-only models differ from 2L? Ablate a single head and run the model on a lot of text. Look at the change in performance. Do any heads matter a lot that aren't induction heads?
Toy Language Models
How do larger models differ?
B
1.13
Look for tasks that an n-layer model can't do, but an n+1-layer model can, and look for a circuit that explains this. (Start by running both models on a bunch of text and look for per-token probability differences)
Toy Language Models
How do larger models differ?
B
1.14
How do 1L SoLU/GELU models differ from 1L attention-only?
Toy Language Models
How do larger models differ?
B
1.15
How do 2L SoLU models differ from 1L?
Toy Language Models
How do larger models differ?
B
1.16
How does 1L GELU differ from 1L SoLU?
Toy Language Models
How do larger models differ?
B
1.17
Analyse how a larger model "fixes the bugs" of a smaller model.
Toy Language Models
How do larger models differ?
B
1.18
Does a 1L MLP transformer fix the skip trigram bugs of a 1L Attn Only model? If so, how?
Toy Language Models
How do larger models differ?
B
1.19
Does a 3L attn only model fix bugs in induction heads in a 2L attn-only model? Try looking at split-token induction, where the current token has a preceding space and is one token, but the earlier occurrence has no preceding space and is two tokens. E.g " Claire" vs. "Cl" "aire"
Toy Language Models
How do larger models differ?
B
1.2
Does a 3L attn only model fix bugs in induction heads in a 2L attn-only model? Look at misfiring when the previous token appears multiple times with different following tokens
Toy Language Models
How do larger models differ?
B
1.21
Does a 3L attn only model fix bugs in induction heads in a 2L attn-only model? Look at stopping induction on a token that likely shows the end of a repeated string (e.g, . or ! or ")
Toy Language Models
How do larger models differ?
B
1.22
Does a 2L MLP model fix these bugs (1.19 -1.21) too?
Toy Language Models
A
1.23
Choose your own adventure: Take a bunch of text with interesting patterns and run the models over it. Look for tokens they do really well on and try to reverse engineer what's going on!
Circuits In The Wild
Circuits in natural language
B
2.1
Look for the induction heads in GPT-2 Small that work with pointer arithmetic. Can you reverse engineer the weights?
Circuits In The Wild
Circuits in natural language
B
2.2
Continuing sequences that are common in natural language (E.g, "1 2 3 4" -> "5", "Monday\nTuesday\n" -> "Wednesday"
I did some preliminary work on this during a hackathon this July, and found components shared between sequence contnuation tasks such as head 9.1 that were found to output the “next member” of a circuit. The work was rushed and crude but I am looking to polish and continue it in the future. A link to it can be found here:
Pablo Hansen- April 18- 2024
Circuits In The Wild
Circuits in natural language
B
2.3
A harder example would be numbers at the start of lines, like "1. Blah blah blah \n2. Blah blah blah\n"-> "3". Feels like it must be doing something induction-y!
Circuits In The Wild
Circuits in natural language
B
2.4
3 letter acronyms, like "The Acrobatic Circus Group (ACG) and the Ringmaster Friendship Union (" -> RFU
Circuits In The Wild
Circuits in natural language
B
2.5
Converting names to emails, like "Katy Johnson <" -> "katy_johnson"
Circuits In The Wild
Circuits in natural language
C
2.6
A harder version of 2.5 is constructing an email from a snippet, like Name: Jess Smith, Email: last name dot first name k @ gmail
Circuits In The Wild
Circuits in natural language
C
2.7
Interpret factual recall. Start with ROME's work with causal tracing, but how much more specific can you get? Heads? Neurons?
Circuits In The Wild
Circuits in natural language
B
2.8
Learning that words after full stops are capital letters.
Circuits In The Wild
Circuits in natural language
B
2.9
Counting objects described in text. (E.g, I picked up an apple, a pear, and an orange. I was holding three fruits.)
Circuits In The Wild
Circuits in natural language
C
2.1
Interpreting memorisation. Sometimes GPT knows phone numbers. How?
Circuits In The Wild
Circuits in natural language
B
2.11
Reverse engineer an induction head in a non-toy model.
Circuits In The Wild
Circuits in natural language
B
2.12
Choosing the right pronouns (E.g, "Lina is a great friend, isn't")
Alana Xiang - 5 May 2023
Circuits In The Wild
Circuits in natural language
A
2.13
Choose your own adventure! Try finding behaviours of your own related to natural language circuits.
Circuits In The Wild
Circuits in code models
B
2.14
Closing brackets. Bonus: Tracking correct brackets - [, (, {, etc.
Circuits In The Wild
Circuits in code models
B
2.15
Closing HTML tags
Circuits In The Wild
Circuits in code models
C
2.16
Methods depend on object type (e.g, x.append a list, x.update a dictionary)
Circuits In The Wild
Circuits in code models
A
2.17
Choose your own adventure! Look for interesting patterns in how the model behaves on code and try to reverse engineer something. Algorithmic flavored tasks should be easiest.
Circuits In The Wild
Extensions to IOI paper
A
2.18
Understand IOI in the Stanford mistral models. Does the same circuit arise? (You should be able to near exactly copy Redwood's code for this)
Circuits In The Wild
Extensions to IOI paper
A
2.19
Do earlier heads in the circuit (duplicate token, induction, S-inhibition) have backup style behaviour? If we ablate them, how much does this damage performance? Will other things compensate?
Circuits In The Wild
Extensions to IOI paper
B
2.2
Is there a general pattern for backup-ness? (Follows 2.19)
Manan Suri - 14 July, 2023
Circuits In The Wild
Extensions to IOI paper
A
2.21
Can we reverse engineer how duplicate token heads work deeply? In particular, how does the QK circuit know to look for copies of the current token without activating on non-duplicates since the current token is always a copy of itself?
Circuits In The Wild
Extensions to IOI paper
B
2.22
Understand IOI in GPT-Neo. Same size but seems to do IOI via MLP composition.
Circuits In The Wild
Extensions to IOI paper
C
2.23
What is the role of Negative/Backup/regular Name Mover heads outside IOI? Are there examples where Negative Name Movers contribute positively?
Circuits In The Wild
Extensions to IOI paper
C
2.24
What are the conditions for the compensation mechanisms where ablating a name mover doesn't reduce performance much to occur? Is it due to dropout?
Circuits In The Wild
Extensions to IOI paper
B
2.25
GPT-Neo wasn't trained with dropout - check 2.24 on this.
Circuits In The Wild
Extensions to IOI paper
B
2.26
Reverse engineering L4H11, a really sharp previous token head in GPT-2-small, at the parameter level.
Circuits In The Wild
Extensions to IOI paper
C
2.27
MLP layers (beyond the first) seem to matter somewhat for the IOI task. What's up with this?
Circuits In The Wild
Extensions to IOI paper
C
2.28
Understanding what's happening in the adversarial examples, most notable S-Inhibition Head attention pattern (hard)
Circuits In The Wild
Confusing things
B
2.29
Why do models have so many induction heads? How do they specialise, and why does the model need so many?
Circuits In The Wild
Confusing things
B
2.3
Why is GPT-2 Small's performance ruined if the first MLP layer is ablated?
Circuits In The Wild
Confusing things
B
2.31
Can we find evidence of the residual stream as shared bandwidth hypothesis?
Circuits In The Wild
Confusing things
B
2.32
Can we find evidence of the residual stream as shared bandwidth hypothesis? In particular, the idea that the model dedicates parameters to memory management and cleaning up memory once it's used. Are there neurons with high negative cosine sim (so the output erases the input feature) Do they correspond to cleaning up specific features?
Circuits In The Wild
Confusing things
B
2.33
What happens to the memory in an induction circuit? (See 2.32)
Circuits In The Wild
Studying larger models
C
2.34
GPT-J contains translation heads. Can you interpret how they work and what they do?
Circuits In The Wild
Studying larger models
C
2.35
Try to find and reverse engineer fancier induction heads like pattern matching heads - try GPT-J or GPT-NeoX.
Circuits In The Wild
Studying larger models
C
2.36
What's up with few-shot learning? How does it work?
Circuits In The Wild
Studying larger models
C
2.37
How does addition work? (Focus on 2-digit)
Circuits In The Wild
Studying larger models
C
2.38
What's up with Tim Dettmer's emergent features in the residual stream stuff? Do they map to anything interpretable? What if we do max activating dataset examples?
Interpreting Algorithmic Problems
Beginner problems
A
3.1
Sorting fixed-length lists. (format - START 4 6 2 9 MID 2 4 6 9)
Interpreting Algorithmic Problems
Beginner problems
A
3.2
Sorting variable-length lists. (What's the sorting algorithm? What's the longest list you can get do? How does length affect accuracy?)
Interpreting Algorithmic Problems
Beginner problems
A
3.3
Interpret a 2L MLP (one hidden layer) trained to do modular addition. (Analogous to Neel's grokking work)
Interpreting Algorithmic Problems
Beginner problems
A
3.4
Interpret a 1L MLP trained to do modular subtraction (Analogous to Neel's grokking work)
Interpreting Algorithmic Problems
Beginner problems
A
3.5
Taking the minimum or maximum of two ints
Interpreting Algorithmic Problems
Beginner problems
A
3.6
Permuting lists
Interpreting Algorithmic Problems
Beginner problems
A
3.7
Calculating sequences with Fibonnaci-style recurrence (predicting next element from the previous two)
Interpreting Algorithmic Problems
Harder problems
B
3.8
5-digit addition/subtraction.
Interpreting Algorithmic Problems
Harder problems
B
3.9
Predicting the output to simple code function. E.g, problems like "a = 1 2 3. a[2] = 4. a -> 1 2 4"
Interpreting Algorithmic Problems
Harder problems
B
3.1
Graph theory problems like this. Unsure of the correct input format. Try a bunch. See here
Interpreting Algorithmic Problems
Harder problems
B
3.11
Train a model on multiple algorithmic tasks we understand (like modular addition and subtraction). Compare to a model trained on each task. Does it learn the same circuits? Is there superposition?
Joshua ; jhdhill@uwaterloo.ca ; jan 31 2024
Interpreting Algorithmic Problems
Harder problems
B
3.12
Train models for automata tasks and interpret them. Do your results match the theory?
Interpreting Algorithmic Problems
Harder problems
B
3.13
In-Context Linear Regression - the transformer gets a sequence (x_1, y_1, x_2, y_2, ...) where y_i = Ax_i + b. A and b are different for each prompt, and need to be learned in-context. (Code here)
Interpreting Algorithmic Problems
Harder problems
C
3.14
Problems in In-Context Linear Regression that are in-context learned. See 3.13.
Interpreting Algorithmic Problems
Harder problems
C
3.15
5 digit (or binary) multiplication
Interpreting Algorithmic Problems
Harder problems
B
3.16
Predict repeated subsequences in randomly generated tokens, and see if you can find and reverse engineer induction heads.
Interpreting Algorithmic Problems
Harder problems
C
3.17
Choose your own adventure! Find your own algorithmic problem. Leetcode easy is probably a good source.
Interpreting Algorithmic Problems
B
3.18
Build a toy model of Indirect Object Identification - train a tiny attention-only model on an algorithmic task simulating IOI - and reverse-engineer the learned solution. Compare it to the circuit found in GPT-2 Small.
Interpreting Algorithmic Problems
C
3.19
Is 3.18 consistent across random seeds, or can other algorithms be learned? Can a 2L model learn this? What happens if you add more MLP's or more layers?
Interpreting Algorithmic Problems
C
3.2
Reverse-engineer Othello-GPT. Can you reverse-engineer the algorithms it learns, or the features the probes find?
Interpreting Algorithmic Problems
Questions about language models
A
3.21
Train a 1L attention-only transformer with rotary to predict the previous token and reverse engineer how it does this.
5/7/23: Eric (repo: https://github.com/DKdekes/rotary-interp)
Interpreting Algorithmic Problems
Questions about language models
B
3.22
Train a 3L attention-only transformer to perform the Indirect Object Identification task. Can it do the task? Does it learn the same circuit found in GPT-2 Small?
Interpreting Algorithmic Problems
Questions about language models
B
3.23
Redo Neel's modular addition analysis with GELU. Does it change things?
Interpreting Algorithmic Problems
Questions about language models
C
3.24
How does memorisation work? Try training a one hidden layer MLP to memorise random data, or training a transformer on a fixed set of random strings of tokens.
Interpreting Algorithmic Problems
Questions about language models
C
3.25
Compare different dimensionality reduction techniques on modular addition or a problem you feel you understand.
Interpreting Algorithmic Problems
Questions about language models
B
3.26
In modular addition, look at what different dimensionality reduction techniques do on different weight matrices. Can you identify which weights matter most? Which neurons form clusters for each frequency? Anything from activations?
Interpreting Algorithmic Problems
Questions about language models
C
3.27
Is direct logit attribution always useful? Can you find examples where it's highly misleading?
Interpreting Algorithmic Problems
Deep learning mysteries
D
3.28
Explore the Lottery Ticket Hypothesis
Interpreting Algorithmic Problems
Deep learning mysteries
D
3.29
Explore Deep Double Descent
Interpreting Algorithmic Problems
Extending Othello-GPT
A
3.3
Try one of Neel's concrete Othello-GPT projects.
Interpreting Algorithmic Problems
Extending Othello-GPT
C
3.31
Looking for modular circuits - try to find the circuits used to compute the world model and to use the world model to compute the next move. Try to understand each in isolation and use this to understand how they fit together. See what you can learn about finding modular circuits in general.
Interpreting Algorithmic Problems
Extending Othello-GPT
B
3.32
Neuron Interpretability and Studying Superposition - try to understand the model's MLP neurons, and explore what techniques do and don't work. Try to build our understanding of transformer MLP's in general.
Interpreting Algorithmic Problems
Extending Othello-GPT
C
3.33
Transformer Circuits Laboratory - Explore and test other conjectures about transformer circuits - e.g, can we figure out how the model manages memory in the residual stream?
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
A
4.1
Does dropout create a privileged basis? Put dropout on the hidden layer of the ReLU output model and study how this changes the results.
Post
14 April 2023: Kunvar (firstuserhere)
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
B
4.2
Replicate their absolute value model and study some of the variants of the ReLU output models.
May 4, 2023 - Kunvar (firstuserhere)
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
B
4.3
Explore neuron superposition by training their absolute value model on a more complex function like x -> x^2.
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
B
4.4
What happens to their ReLU output model when there's non-uniform sparsity? E.g, one class of less sparse features and another of very sparse
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
A
4.5
Explore neuron superposition by training their absolute value model on functions of multiple variables. Make inputs binary (0/1) and look at the AND and OR of element pairs.
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
B
4.6
Explore neuron superposition by training their absolute value model on functions of multiple variables. Keep the inputs as uniform reals in [0, 1] and look at max(x, y)
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
A
4.7
Adapt their ReLU output model to have a different range of feature values, and see how this affects things. Make the features 1 (i.e, two possible values)
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
B
4.8
Adapt their ReLU output model to have a different range of feature values, and see how this affects things. Make the features discrete (1, 2, 3)
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
B
4.9
Adapt their ReLU output model to have a different range of feature values, and see how this affects things. Make the features uniform [0.5, 1]
April 30, 2023; Kunvar(firstuserhere)
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
A
4.1
What happens if you replace ReLU's with GeLU's in the toy models?
May 1, 2023 - Kunvar (firstuserhere)
Exploring Polysemanticity and Superposition
Confusions to study in Toy Models of Superposition
C
4.11
Can you find a toy model where GELU acts significantly differently from ReLU?
May 1, 2023 - Kunvar (firstuserhere)
Exploring Polysemanticity and Superposition
Building toy models of superposition
C
4.12
Build a toy model of a classification problem with cross-entropy loss
Exploring Polysemanticity and Superposition
Building toy models of superposition
C
4.13
Build a toy model of neuron superposition that has many more hidden features than output features
Exploring Polysemanticity and Superposition
Building toy models of superposition
C
4.14
Build a toy model that needs multiple hidden layers of ReLU's. Can computation in superposition happen across several layers? Eg max (|x|, |y|)
Exploring Polysemanticity and Superposition
Building toy models of superposition
C
4.15
Build a toy model of attention head superposition/polysemanticity. Can you find a task where the model wants to do different things with an attention head on different inputs? How does it represent things internally / deal with interference?
Exploring Polysemanticity and Superposition
Building toy models of superposition
D
4.16
Build a toy model with a mdoel needs to deal with simultaneous interference, and try to understand how it does it, or if it can.
Exploring Polysemanticity and Superposition
Making toy model counterexamples
C
4.17
Make toy models that are counterexamples in MI. A learned example of a network with a non-linear representation.
Exploring Polysemanticity and Superposition
Making toy model counterexamples
C
4.18
Make toy models that are counterexamples in MI. A network without a discrete number of features.
Exploring Polysemanticity and Superposition
Making toy model counterexamples
C
4.19
Make toy models that are counterexamples in MI. A non-decomposable neural network.
Exploring Polysemanticity and Superposition
Making toy model counterexamples
C
4.2
Make toy models that are counterexamples in MI. A task where networks can learn multiple different sets of features.
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
B
4.21
Induction heads copy the token they attend to the output, which involves storing which of 50,000 tokens it is. How are these stored in a 64-dimensional space?
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
B
4.22
How does the previous token head in an induction circuit communicate the value of the previous token to the key of the induction head? Bonus: What residual stream subspace does it take up? Is there interference?
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
B
4.23
How does the IOI circuit communicate names/positions between composing heads?
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
B
4.24
Are there dedicated dimensions for positional embeddings? Do any other components write to those dimensions?
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
A
4.25
Can you find any examples of the geometric superposition configurations in the residual stream of a language model?
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
C
4.26
Can you find any examples of locally almost-orthogonal bases?
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
C
4.27
Do language models have "genre" directions that detect the type of text, and then represent features specific to each genre in the same subspace?
Exploring Polysemanticity and Superposition
Studying bottleneck superposition in real language models
D
4.28
Can you find examples of a model learning to deal with simultaneous interference?
Exploring Polysemanticity and Superposition
Studying neuron superposition in real models
B
4.29
Look at a polysemantic neuron in a 1L language model. Can you figure out how the model disambiguates what feature it is?
Exploring Polysemanticity and Superposition
Studying neuron superposition in real models
C
4.3
Look at a polysemantic neuron in a 2L language model. Can you figure out how the model disambiguates what feature it is?
Exploring Polysemanticity and Superposition
Studying neuron superposition in real models
B
4.31
Take a feature that's part of a polysemantic neuron in a 1L language model and try to identify every neuron that represents that feature. Is it sparse or diffuse?
Exploring Polysemanticity and Superposition
Studying neuron superposition in real models
C
4.32
Try to fully reverse engineer a feature discovered in 4.31.
Exploring Polysemanticity and Superposition
Studying neuron superposition in real models
C
4.33
Can you use superposition to create an adversarial example for a neuron?
Exploring Polysemanticity and Superposition
Studying neuron superposition in real models
C
4.34
Can you find any examples of the asymmetric superposition motif in the MLP of a 1-2 layer language model?
Exploring Polysemanticity and Superposition
C
4.35
Pick a simple feature of language (e.g, is number, is base64) and train a linear probe to detect that in the MLP activations of a 1L language model.
Exploring Polysemanticity and Superposition
D
4.36
Look for features in Neuroscope that seem to be represented by various neurons in a 1-2 layer language model. Train probes to detect some of them. Compare probe performance vs. neuron performance.
Exploring Polysemanticity and Superposition
Comparing SoLU/GELU
A
4.37
How do TransformerLens SoLU / GeLU models compare in Neuroscope under the SoLU polysemanticity metric? (What fraction of neurons seem monosemantic)
Exploring Polysemanticity and Superposition
Comparing SoLU/GELU
B
4.38
Can you find any better metrics for polysemanticity?
Exploring Polysemanticity and Superposition
Comparing SoLU/GELU
B
4.39
The paper speculates LayerNorm lets the model "smuggle through" superposition in SoLU models by smearing features across many dimensions and letting LayerNorm scale it up. Can you find evidence of this?
Exploring Polysemanticity and Superposition
Comparing SoLU/GELU
B
4.4
How similar are the neurons between SoLU/GELU models of the same layers?
Exploring Polysemanticity and Superposition
Comparing SoLU/GELU
C
4.41
How does GELU vs. ReLU compare re: polysemanticity. Replicate SoLU analysis.
Exploring Polysemanticity and Superposition
Getting rid of superposition
C
4.42
If you train a 1L/2L language model with d_mlp = 100 * d_model, does superposition go away?
Exploring Polysemanticity and Superposition
Getting rid of superposition
C
4.43
Study the T5 XXL. It's 11B params and not supported by TransformerLens. Expect major infrastructure pain.
Exploring Polysemanticity and Superposition
Getting rid of superposition
D
4.44
Can you take a trained model, freeze all weights except an MLP layer, x10 that layer's width, copy each neuron 10 times, add noise, and fine-tune? Does this remove superposition / add new features?
Exploring Polysemanticity and Superposition
Getting rid of superposition
C
4.45
Pick an open problem at the end of Toy Models of Superposition.
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
B
5.1
Understanding why 5 digit addition has a phase change per digit (so 6 total?!)
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
C
5.2
Why do 5-digit addition phase changes happen in that order?
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
B
5.3
Look at the PCA of logits on the full dataset, or the PCA of a stack of flattened weights. If you plot a scatter plot of the first 2 components, the different phases of training are clearly visible. What's up with this?
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
C
5.4
Can we predict when grokking will happen? Bonus: Without using any future information?
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
C
5.5
Understanding why the model chooses specific frequencies (and why it switches mid-training sometimes!)
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
B
5.6
What happens if we include in the loss one of the progress measures in Neel's grokking post? Can we accelerate or stop grokking?
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
B
5.7
Adam Jermyn provides an analytical argument and some toy models for why phase transition should be an inherent part of (some of) how models learn. Can you find evidence of this in more complex models?
Analysing Training Dynamics
Algorithmic tasks - understanding grokking
B
5.8
Build on and refine Adam Jermyn's arguments and toy models - think about how they deviate from a real transformer, and build more faithful models.
Analysing Training Dynamics
Algorithmic tasks - lottery tickets
B
5.9
For a toy model trained to form induction heads, is there a lottery-ticket style thing going on? Can you disrupt induction head formation by messing with the initialisation?
Analysing Training Dynamics
Algorithmic tasks - lottery tickets
C
5.1
All Neel's toy models (attn-only, gelu, solu) were trained with the same data shuffle and weight initialisation. Many induction heads aren't shared, but L2H3 in 3L and L1H6 in 2L always are. What's up with that?
Analysing Training Dynamics
Algorithmic tasks - lottery tickets
B
5.11
If we knock out the parameters that form important circuits at the end of training on some toy task, but knock them out at the start of training, how much does that delay/stop generalisation?
Analysing Training Dynamics
Algorithmic tasks - lottery tickets
B
5.12
Analysing how pairs of heads in an induction circuit compose over time - Can you find progress measures which predict these?
Analysing Training Dynamics
Algorithmic tasks - lottery tickets
B
5.13
Analysing how pairs of heads in an induction circuit compose over time - Can we predict which heads will learn to compose first?
Analysing Training Dynamics
Algorithmic tasks - lottery tickets
B
5.14
Analysing how pairs of heads in an induction circuit compose over time -Does the composition develop as a phase transition?
Analysing Training Dynamics
Understanding fine-tuning
C
5.15
Build a toy model of fine-tuning (train on task 1, fine-tune on task 2). What is going on internally? Any interesting motifs?
Analysing Training Dynamics
Understanding fine-tuning
A
5.16
How does model performance change on the original training distribution when finetuning?
Analysing Training Dynamics
Understanding fine-tuning
B
5.17
How is the model different on fine-tuned text? Look at examples where the model does much better after fine-tuning, and some normal text.
Analysing Training Dynamics
Understanding fine-tuning
B
5.18
Try activation patching between the old and fine-tuned model and see how hard recovering performance is.
Analysing Training Dynamics
Understanding fine-tuning
B
5.19
Look at max activating text for various neurons in the original models. How has it changed post fine-tuning?
Analysing Training Dynamics
Understanding fine-tuning
B
5.2
Explore further and see what's going on with fine-tuning mechanistically.
Analysing Training Dynamics
Understanding fine-tuning
C
5.21
Can you find any phase transitions in the fine-tuning checkpoints?
Analysing Training Dynamics
Understanding training dynamics in language models
B
5.22
Can you replicate the induction head phase transition results in the various checkpointed models in TransformerLens? (If code works for attn-only-2l it should work for them all)
Analysing Training Dynamics
Understanding training dynamics in language models
B
5.23
Look at the neurons in TransformerLens SoLU models during training. Do they tend to form as a phase transition?
Analysing Training Dynamics
Understanding training dynamics in language models
C
5.24
Use the per-token loss analysis technique from the induction heads paper to look for more phase changes.
Analysing Training Dynamics
Understanding training dynamics in language models
A
5.25
Look at attention heads on various texts and see if any have recognisable attention patterns, then analyse them over training.
Analysing Training Dynamics
Finding phase transitions
A
5.26
Look for phase transitions in the Indirect Object Identification task. (Note: This might not have a phase change)
Analysing Training Dynamics
Finding phase transitions
B
5.27
Try digging into the specific heads that act on IOI and look for phase transitions. Use direct logit attribution for the name movers.
Analysing Training Dynamics
Finding phase transitions
B
5.28
Study the attention patterns of each category of heads in IOI for phase transitions.
Analysing Training Dynamics
Finding phase transitions
B
5.29
Look for phase transitions in simple IOI-style algorithmic tasks, like few-shot learning, addition, sorting words alphabetically...
Analysing Training Dynamics
Finding phase transitions
B
5.3
Look for phase transitions in soft induction heads like translation.
Analysing Training Dynamics
Finding phase transitions
C
5.31
Look for phase transitions in benchmark performance or specific questions from a benchmark.
Analysing Training Dynamics
Finding phase transitions
D
5.32
Hypothesis: Scaling laws happen because models experience a ton of tiny phase changes which average out to a smooth curve due to the law of large numbers. Can you find evidence for or against that?
Analysing Training Dynamics
Studying path dependence
A
5.33
How much do the Stanford CRFM models have similar outputs on a given text?
Analysing Training Dynamics
Studying path dependence
B
5.34
How much do the Stanford CRFM models differ with algorithmic tasks like Indirect Object Identification?
Analysing Training Dynamics
Studying path dependence
A
5.35
Look for Indirect Object Identification capability in other models of approximately the same size.
Analysing Training Dynamics
Studying path dependence
B
5.36
When model scale varies (e.g, GPT-2 small vs. medium) is there anything the smaller model can do that the larger one can't do? (Look at difference in per token log prob)
Analysing Training Dynamics
Studying path dependence
B
5.37
Try applying the Git Re-Basin techniques to a 2L MLP trained for modular addition. Does this work? If you use Neel's grokking work to analyse the circuits involved, how does the re-basin technique map onto the circuits?
Analysing Training Dynamics
Studying path dependence
A
5.38
Can you find some problem where you understand the circuits and Git Re-Basin does work?
Techniques, Tooling, and Automation
Breaking current techniques
A
6.1
Try to find concrete edge cases where a technique breaks - start with a misleading example in a real model or training a toy model with one.
Techniques, Tooling, and Automation
Breaking current techniques
B
6.2
Break direct logit attribution - start by looking at GPT-Neo Small where the logit lens (precursor to direct logit attribution) seems to work badly, but works well if you include the final layer and the unembed.
Techniques, Tooling, and Automation
Breaking current techniques
C
6.3
Can you fix direct logit attribution in GPT-Neo small, e.g, by finding a linear approximation to the final layer by taking gradients? (Eleuther's tuned lens in #interp-across-depth would be a good place to start)
Techniques, Tooling, and Automation
Breaking current techniques
B
6.4
Find edge cases where linearising LayerNorm breaks. See some work by Eric Winsor at Conjecture.
Techniques, Tooling, and Automation
Breaking current techniques
B
6.5
Find edge cases where activation patching breaks. (It should break when you patch one variable but there's dependence on multiples)
Techniques, Tooling, and Automation
Breaking current techniques
C
6.6
Find edge cases where causal scrubbing breaks.
Techniques, Tooling, and Automation
Breaking current techniques
A
6.7
Find edge cases where ablations break. (Start w/ backup name movers in the IOI circuit, where we know zero ablations break)
Techniques, Tooling, and Automation
Breaking current techniques
B
6.8
Can you find places where one ablation (zero, mean, random) breaks but the others don't?
Techniques, Tooling, and Automation
Breaking current techniques
B
6.9
Find edge cases where composition scores break. (They don't work well for the IOI circuit)
Techniques, Tooling, and Automation
Breaking current techniques
B
6.1
Find edge cases where eigenvalue copying scores break.
Techniques, Tooling, and Automation
Breaking current techniques
C
6.11
Automate ways to identify heads that compose. Start with IOI circuit and the composition scores in A Mathematical Framework.
Techniques, Tooling, and Automation
B
6.12
Try looking for composition on a specific input. Decompose the residual stream into the sum of outputs of previous heads, then decompose query, key, value into sums of terms from each previous head. Are any larger than the others / matter more if you ablate them / etc?
Techniques, Tooling, and Automation
C
6.13
Can you automate direct path patching as used in the IOI paper?
Techniques, Tooling, and Automation
B
6.14
Compare causal tracing to activation patching. Do they give the same outputs? Can you find situations where one breaks and the other doesn't? (Try IOI task or factual recall task)
Techniques, Tooling, and Automation
ROME activation patching
A
6.15
In the ROME paper, they do activation patching by patching over the outputs of 10 adjacent MLP or attention layers. (Look at logit difference after patching). How do results change when you do single layers?
Techniques, Tooling, and Automation
ROME activation patching
A
6.16
In the ROME paper, they do activation patching by patching over the outputs of 10 adjacent MLP or attention layers. (Look at logit difference after patching). Can you get anywhere when patching specific neurons?
Techniques, Tooling, and Automation
ROME activation patching
B
6.17
In the ROME paper, they do activation patching by patching over the outputs of 10 adjacent MLP or attention layers. (Look at logit difference after patching). Can you get anywhere when patching some set of neurons? (E.g, the neurons that activate the most within the 10 layers?)
Techniques, Tooling, and Automation
Automatically find circuits
A
6.18
Automate ways to find previous token heads. (Bonus: Add to TransformerLens!)
Techniques, Tooling, and Automation
Automatically find circuits
A
6.19
Automate ways to find duplicate token heads. (Bonus: Add to TransformerLens!)
Techniques, Tooling, and Automation
Automatically find circuits
A
6.2
Automate ways to find induction heads. (Bonus: Add to TransformerLens!)
Techniques, Tooling, and Automation
Automatically find circuits
A
6.21
Automate ways to find translation heads. (Bonus: Add to TransformerLens!)
Techniques, Tooling, and Automation
Automatically find circuits
B
6.22
Automate ways to find few shot learning heads. (Bonus: Add to TransformerLens!)
Techniques, Tooling, and Automation
Automatically find circuits
B
6.23
Can you find an automated way to detect pointer arithmetic based induction heads vs. classic induction heads?
Techniques, Tooling, and Automation
Automatically find circuits
B
6.24
Can you find an automated way to detect the heads used in the IOI Circuit? (S-inhibition, name mover, negative name mover, backup name mover)
Techniques, Tooling, and Automation
Automatically find circuits
B
6.25
Can you automate detection of the heads used in factual recall to move information about the fact to the final token? (Try activation patching)
Techniques, Tooling, and Automation
Automatically find circuits
B
6.26
(Infrastructure) Combine some of the head detectors from 6.18-6.25 to make a "wiki" for a range of models, with information and scores for each head for how it falls into different categories. MVP: Pandas Dataframes with a row for each head and a column for each metric.
Techniques, Tooling, and Automation
Automatically find circuits
C
6.27
Can you automate the detection of something in neuron interpretability? E.g, trigram neurons
Techniques, Tooling, and Automation
Automatically find circuits
C
6.28
Find good ways to find the equivalent of max activating dataset examples for attention heads. Validate on induction circuits, then IOI. See post for ideas.
Techniques, Tooling, and Automation
Refine max activating dataset examples
C
6.29
Refine the max activating dataset examples technique for neuron interpretability to find minimal or diverse examples.
Techniques, Tooling, and Automation
Refine max activating dataset examples
B
6.3
Using 6.28: Corrupt different token embeddings in a sequence to see which matter.
Techniques, Tooling, and Automation
Refine max activating dataset examples
B
6.31
Using 6.28: Compare to randomly chosen directions in neuron activation space to see how clustered/monosemantic things seem.
Techniques, Tooling, and Automation
Refine max activating dataset examples
B
6.32
Using 6.28: Validate these by comparing to direct effect of neuron on the logits, or output vocab logits most boosted by that neuron.
Techniques, Tooling, and Automation
Refine max activating dataset examples
B
6.33
Using 6.28: Use a model like GPT-3 to find similar text to an existing example and see if they also activate the neuron. Bonus: Use them to replace specific tokens.
Techniques, Tooling, and Automation
Refine max activating dataset examples
B
6.34
Using 6.28: Look at dataset examples at different quantiles for neuron activations (25%, 50%, 75%, 90%, 95%). Does that change anything?
Techniques, Tooling, and Automation
Refine max activating dataset examples
C
6.35
Using 6.28: (Infrastructure) Add any of 6.29-6.34 to Neuroscope. Email Neel (neelnanda27@gmail.com) for codebase access.
Techniques, Tooling, and Automation
Refine max activating dataset examples
A
6.36
Using 6.28: Finding the minimal example to activate a neuron by truncating the text - how often does this work?
Techniques, Tooling, and Automation
Refine max activating dataset examples
A
6.37
Using 6.28: Can you replicate the results of the interpretability illusion for Neel's toy models by finding seemingly monosemantic neurons on Python code or C4 (web text), but are polysemantic when combined?
Techniques, Tooling, and Automation
Refine max activating dataset examples
B
6.38
Using 6.28: In SoLU models, compare max activating results for pre-SoLU, post-SoLU, and post LayerNorm activations. ('pre', 'mid', 'post' in TransformerLens). How consistent are they? Does one seem more principled?
Techniques, Tooling, and Automation
Interpreting models with LLM's
B
6.39
Can GPT-3 figure out trends in max activating examples for a neuron?
Techniques, Tooling, and Automation
Interpreting models with LLM's
B
6.4
Can you use GPT-3 to generate counterfactual prompts with lined up tokens to do activation patching on novel problems? (E.g, "John gave a bottle of milk to -> Mary" vs. "Mary gave a bottle of milk to -> John")
Techniques, Tooling, and Automation
Interpreting models with LLM's
D
6.41
Choose your own adventure - can you find a way to usefully use an LLM to interpret models?
Techniques, Tooling, and Automation
Apply techniques from non-mechanistic interpretability
B
6.42
How well does feature attribution work on circuits we understand?
Techniques, Tooling, and Automation
Apply techniques from non-mechanistic interpretability
C
6.43
Can you use probing to get evidence for or against predictions in Toy Models of Superposition?
Techniques, Tooling, and Automation
Apply techniques from non-mechanistic interpretability
C
6.44
Pick anything interesting from Rauker et al and try to apply the techniques to circuits we understand.
Techniques, Tooling, and Automation
Apply techniques from non-mechanistic interpretability
D
6.45
Wiles et al gives an automated set of techniques to analyse bugs in image classification models. Can you get any traction adapting this to language models?
Techniques, Tooling, and Automation
C
6.46
Take existing circuits and explore quantitative ways to characterise that it's a true circuit (or disprove it!) Try causal scrubbing to start.
Techniques, Tooling, and Automation
C
6.47
Build on Arthur Conmy's work to automatically find circuits via recursive path patching
Techniques, Tooling, and Automation
B
6.48
Resolve some of the open issues/feature requests for TransformerLens.
Techniques, Tooling, and Automation
Taking the "diff" of two models
C
6.49
Build tooling to take the "diff" of two models, treating them as a black box mapping inputs to outputs, so it works with models with different internal structure
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.5
Using 6.49, run it on a bunch of text and look at the biggest per-token log prob difference.
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.51
Using 6.49, run them on various benchmarks and compare performance.
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.52
Using 6.49, try "benchmarks" like performing algorithmic tasks like IOI, acronyms, etc. as from Circuits In the Wild.
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.53
Using 6.49, try qualitative exploration like just generating text from the models and look for ideas.
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.54
Build tooling to take the diff of two models with the same internal structure. Includes 6.49 but also lets you compare model internals!
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.55
Using 6.54, look for the largest difference in weights.
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.56
Using 6.54, run them on a bunch of text and look for largest difference in activations.
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.57
Using 6.54, look at the direct logit attribution of layers and heads on various texts, and look for the biggest differences.
Techniques, Tooling, and Automation
Taking the "diff" of two models
B
6.58
Using 6.54, do activation patching on a piece of text where one model does much better than the other - are some parts key to improved performance?
Techniques, Tooling, and Automation
C
6.59
We understand how attention is calculated for a head using the QK matrix. This doesn't work for rotary attention. Can you find a principled alternative?
Image Model Interpretability
Reverse engineering image models
C
7.1
Using Circuits techniques, how well can we reverse engineer ResNet?
Image Model Interpretability
Reverse engineering image models
C
7.2
Vision Transformers - can you smush together transformer circuits and image circuits techniques? Which ones transfer?
Image Model Interpretability
Reverse engineering image models
C
7.3
Using Circuits techniques, how well can we reverse engineer ConvNeXt, a modern image model architecture merging ResNet and vision transformer ideas?
Image Model Interpretability
Building on Circuits thread
C
7.4
How well can you hand-code curve detectors? Can you include color? How much performance can you recover?
Image Model Interpretability
Building on Circuits thread
C
7.5
Can you hand-code any other circuits? Start with other early vision neurons
Image Model Interpretability
Building on Circuits thread
D
7.6
What happens if you apply causal scrubbing to the Circuits thread's claimed curve circuits algorithm? (This will take significant conceptual effort to extend to images since it's harder to precisely control input!)
Image Model Interpretability
Building on Circuits thread
B
7.7
Look for equivariance in late layers of vision models, symmetries in a network with analogous families of neurons. Likely looks like hunting in Microscope.
Image Model Interpretability
Building on Circuits thread
C
7.8
Digging into polysemantic neuron examples and trying to understand better what's going on there.
Image Model Interpretability
Building on Circuits thread
B
7.9
Look for a wide array of circuits using the weight explorer. What interesting patterns and motifs can you find?
Image Model Interpretability
Multimodal models (CLIP interpretability)
B
7.1
Look at the weights connecting neurons in adjacent layers. How sparse are they? Are there any clear patterns where one neuron is constructed from previous ones?
Image Model Interpretability
Multimodal models (CLIP interpretability)
C
7.11
Can you rigorously reverse engineer any circuits, like the Curve Circuits paper?
Image Model Interpretability
Multimodal models (CLIP interpretability)
C
7.12
Can you apply transformer circuits techniques to understand the attention heads in the image part?
Image Model Interpretability
Multimodal models (CLIP interpretability)
B
7.13
Can you refine the technique for generating max activating text strings? Could it be applied to language models?
Image Model Interpretability
C
7.14
Train a checkpointed run of Inception. Do curve detectors form as a phase change?
Image Model Interpretability
B
7.15
Does activation patching work on Inception?
Image Model Interpretability
Diffusion models
B
7.16
Apply feature visualisation to neurons in diffusion models and see if any seem clearly interpretable.
Image Model Interpretability
Diffusion models
B
7.17
Are there style transfer neurons in diffusion models? (E.g, activating on "in the style of Thomas Kinkade")
Image Model Interpretability
Diffusion models
B
7.18
Are different circuits activating when different amounts of noise are input in diffusion models?
Interpreting Reinforcement Learning
AlphaZero
C
8.1
Replicate some of Tom McGrath's AlphaZero work with LeelaChessZero. Use NMF on the activations and trying to interpret some. See visualisations here.
Interpreting Reinforcement Learning
AlphaZero
D
8.2
Try applying 8.1 to an open source AlphaZero style Go playing agent
Interpreting Reinforcement Learning
AlphaZero
D
8.3
Train a small AlphaZero model on a simple game like Tic-Tac-Toe, and try to apply 8.1 there. (Training will be hard! See this tutorial.)
Interpreting Reinforcement Learning
AlphaZero
D
8.4
Can you extend the work on LeelaZero? Can you find anything about how a feature is computed? Start by looking for features near the start or end of the network.
Interpreting Reinforcement Learning
Goal misgeneralisation
C
8.5
Intrepret one of the examples in the goal misgeneralisation papers (Langosco et al and Shah et al). Can you concretely figure out what's going on?
Interpreting Reinforcement Learning
Goal misgeneralisation
B
8.6
Using 8.5: Possible starting point, Tree Gridworld and Monster Gridworld from Shah et al.
Interpreting Reinforcement Learning
Goal misgeneralisation
C
8.7
Using 8.5: Possible starting point - CoinRun. Interpreting RL Vision made significant progress and Langosco et al found it was an example of goal misgeneralisation - can you build on these to predict the misgeneralisation?
Interpreting Reinforcement Learning
Decision Transformers
B
8.8
Can you apply transformer circuits to a decision transformer? What do you find?
Interpreting Reinforcement Learning
Decision Transformers
B
8.9
Try training a 1L decision transformer on a toy problem, like finding the shortest path in a graph.
Interpreting Reinforcement Learning
C
8.1
Train and interpret a model from the In-Context Reinforcement Learning and Algorithmic Distillation paper. They trained small transformers where they input a sequence of moves for a "novel" RL task and the model outputs sensible answers for that task.
10/april/2023-Victor Levoso and others , working on reinplementing AD to try this, we have a channel for it on this discord: https://discord.gg/cMr5YqbU4y
Interpreting Reinforcement Learning
Interpreting RLHF Transformers
D
8.11
Go and interpret CarperAI's RLHF model (forthcoming). What's up with that? How is it different from a vanilla language model?
Interpreting Reinforcement Learning
Interpreting RLHF Transformers
C
8.12
Can you find any circuits in CarperAI's RLHF model corresponding to longer term planning?
Interpreting Reinforcement Learning
Interpreting RLHF Transformers
C
8.13
Can you get any traction on interpreting CarperAI's RLHF model's reward model?
Interpreting Reinforcement Learning
Interpreting RLHF Transformers
D
8.14
Train a toy RLHF model (1-2 layers) to do a simple task. Use GPT-3 for human data generation. Then try to interpret it. (Note: This will be hard to train, but Neel would be super excited to see the results!) Bonus: Try bigger models like GPT-2 Medium to XL.
Interpreting Reinforcement Learning
C
8.15
Try training and interpreting a small model from Guez et al. They trained model-free RL agents and showed evidence they spontaneously learned planning. Can you find evidence for/against this?
Interpreting Reinforcement Learning
Interpreting policy gradients
B
8.16
Can you interpret a small model trained with policy gradients on a gridworld task?
Interpreting Reinforcement Learning
Interpreting policy gradients
B
8.17
Can you interpret a small model trained with policy gradients on an OpenAI Gym task?
Interpreting Reinforcement Learning
Interpreting policy gradients
B
8.18
Can you interpret a small model trained with policy gradients on an Atari game (e.g, Pong)?
Interpreting Reinforcement Learning
C
8.19
Can you interpret a model on a task from 8.16-8.18 using Q-Learning?
Interpreting Reinforcement Learning
C
8.2
Take an agent trained with RL and train another network to copy the output logits of that agent. Try to reverse engineer the clone. Can you find the resulting circuits in the original?
Interpreting Reinforcement Learning
C
8.21
Once you've got traction understanding a fully trained agent on a task elsewhere in this category, try to extend this understanding to study it during training. Can you get any insight into what's actually going on?
Interpreting Reinforcement Learning
B
8.22
Choose your own adventure! There's lots of work in RL - pick something you're excited about and try to reverse engineer something!
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.1
Explore random neurons! Use the interactive neuroscope to test and verify your understanding.
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.2
Look for interesting conceptual neurons in the middle layers of larger models, like the "numbers that refer to groups of people" neuron.
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.3
Look for examples of detokenisation neurons
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.4
Look for examples of trigram neurons (consistently activate on a pair of tokens and boost the logit of plausible next tokens)
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.5
Look for examples of retokenization neurons
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.6
Look for examples of context neurons (eg base64)
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.7
Look for neurons that align with any of the feature ideas in 9.13-9.21
Studying Learned Features in Language Models
Exploring Neuroscope
B
9.8
Look for examples of neurons with a naive (but incorrect!) initial story that have a much simpler explanation after further investigation
Studying Learned Features in Language Models
Exploring Neuroscope
B
9.9
Look for examples of neurons with a naive (but incorrect!) initial story that have a much more complex explanation after further investigation
Studying Learned Features in Language Models
Exploring Neuroscope
A
9.1
How much does the logit attribution of a neuron align with the dataset example patterns? Is it related?
Studying Learned Features in Language Models
Exploring Neuroscope
B
9.11
If you find neurons for 9.10 that seem very inconsistent, can you figure out what's going on?
Studying Learned Features in Language Models
Exploring Neuroscope
B
9.12
For dataset examples for neurons in a 1L network, measure how much its pre-activation value comes from the output of each attention head vs. the embedding (vs. positional embedding!). If dominated by specific heads, how much do those heads attend to the tokens you expect?
Studying Learned Features in Language Models
Seeking out specific features
A
9.13
Basic syntax (Lots of ideas in post)
Studying Learned Features in Language Models
Seeking out specific features
A
9.14
Linguistic features (Try using spaCy to automate this) (Lots of ideas in post)
Studying Learned Features in Language Models
Seeking out specific features
A
9.15
Proper nouns (Lots of ideas in post)
Studying Learned Features in Language Models
Seeking out specific features
A
9.16
Python code features (Lots of ideas in post)
Studying Learned Features in Language Models
Seeking out specific features
B
9.17
From 9.16 - level of indent for a line (harder because it's categorical/numeric)
Studying Learned Features in Language Models
Seeking out specific features
B
9.18
From 9.16 - level of bracket nesting (harder because it's categorical/numeric)
Studying Learned Features in Language Models
Seeking out specific features
B
9.19
General code features (Lots of ideas in post)
Studying Learned Features in Language Models
Seeking out specific features
A
9.2
LaTeX features. Try common commands (\left, \right) and section titles (\abstract, \introduction, etc.)
Studying Learned Features in Language Models
Seeking out specific features
B
9.21
Features in compiled LaTeX, e.g paper citations
Studying Learned Features in Language Models
Seeking out specific features
B
9.22
Any of the more abstract neurons in Multimodel Neurons (e.g Christmas, sadness, teenager, anime, Pokemon, etc.)
Studying Learned Features in Language Models
Seeking out specific features
A
9.23
Diambiguation neurons - Foreign language disambiguation (e.g, "die" in Dutch vs. German vs. Afrikaans)
Studying Learned Features in Language Models
Seeking out specific features
A
9.24
Disambiguation neurons - words with multiple meanings (e.g, "bat" as animal or sports equipment)
Studying Learned Features in Language Models
Seeking out specific features
A
9.25
Search for memory management neurons (high negative cosine similarity between w_in and w_out). What do their dataset examples look like? Is there a pattern?
Studying Learned Features in Language Models
Seeking out specific features
A
9.26
Search for signal boosting neurons (high positive cosine similarity between w_in and w_out). What do their dataset examples look like? Is there a pattern?
Studying Learned Features in Language Models
Seeking out specific features
C
9.27
Search for neurons that clean up superposition interference.
Studying Learned Features in Language Models
Seeking out specific features
A
9.28
Can you find split-token neurons? (I.e, " Claire" vs. "Cl" and "aire" - the model should learn to identify the split-token case)
Studying Learned Features in Language Models
Seeking out specific features
B
9.29
Can you find examples of neuron families/equivariance? (Ideas in post)
Studying Learned Features in Language Models
Seeking out specific features
B
9.3
Neurons which link to attention heads - Induction should NOT trigger (e.g, current token repeated but previous token is not, different copies of current string have different next tokens)
Studying Learned Features in Language Models
Seeking out specific features
B
9.31
Neurons which link to attention heads - fixing a skip trigram bug
Studying Learned Features in Language Models
Seeking out specific features
A
9.32
Neurons which link to attention heads - duplicated token
Studying Learned Features in Language Models
Seeking out specific features
B
9.33
Neurons which link to attention heads - splitting into token X is duplicated for many common tokens
Studying Learned Features in Language Models
Seeking out specific features
B
9.34
Neurons which represent positional information (not invariant between position). Will need to input data with a random offset to isolate this.
Studying Learned Features in Language Models
Seeking out specific features
B
9.35
What is the longest n-gram you can find that seems represented?
Studying Learned Features in Language Models
Seeking out specific features
C
9.36
Try training linear probes for features from 9.13-9.35.
Studying Learned Features in Language Models
Seeking out specific features
C
9.37
Using 9.36 - How does your ability to recover features from the residual stream compare to MLP layer outputs vs. attention layer outputs? Can you find features that can only be recovered from some of these?
Studying Learned Features in Language Models
Seeking out specific features
C
9.38
Using 9.36 - Are there features that can only be recovered from certain MLP layers?
Studying Learned Features in Language Models
Seeking out specific features
C
9.39
Using 9.36 - Are there features that are significantly easier to recover from early layer residual streams and not from later layers?
Studying Learned Features in Language Models
Curiosities about neurons
A
9.4
When you look at the max dataset examples for a specific neuron, is that neuron the most activated neuron on the text? What does it look like in general?
Studying Learned Features in Language Models
Curiosities about neurons
A
9.41
Look at the distributions of neuron activations (pre and post-activation for GELU, and pre, mid, and post for SoLU). What does this look like? How heavy tailed? How well can it be modelled as a normal distribution?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.42
Do neurons vary in terms of how heavy tailed their distributions are? Does it at all correspond to monosemanticity?
Studying Learned Features in Language Models
Curiosities about neurons
A
9.43
How similar are the distributions between SoLU and GELU?
Studying Learned Features in Language Models
Curiosities about neurons
A
9.44
What does the distribution of the LayerNorm scale and softmax denominator in SoLU look like? Is it bimodal (indicating monosemantic features) or fairly smooth and unimodal?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.45
Can you find any genuinely monosemantic neurons? That are mostly monosemantic across their entire activation range?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.46
Find a feature where GELU is used to calculate it in a way that ReLU couldn't be (e.g, approximating a quadratic)
Studying Learned Features in Language Models
Curiosities about neurons
B
9.47
Can you find a feature which seems to be represented by several neurons?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.48
Using 9.47 - what happens if you ablate some of the neurons? Is it robust to this? Does it need them all?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.49
Can you find a feature that is highly diffuse across neurons? (I.e, represented by the MLP layer but doesn't activate any particular neuron a lot)
Studying Learned Features in Language Models
Curiosities about neurons
B
9.5
Look at the direct logit attribution of neurons and find the max dataset examples for this. How similar are the texts to max activating dataset examples?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.51
Looka t the max negative direct logit attribution. Are there neurons which systematically suppress the correct next token? Can you figure out what's up with these?
Studying Learned Features in Language Models
Curiosities about neurons
A
9.52
Try comparing how monosemantic the neurons in a GELU vs SoLU model are. Can you replicate the results SoLU does better? What are the rates for each model?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.53
Using 9.52, can you come up with a better and more robust metric? How consistent is it across reasonable metrics?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.54
The GELU and SoLU toy language models were trained with identical initialisation and data shuffle. Is there any correspondence between what neurons represent in each model?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.55
If a feature is represented in one of the GELU/SoLU models, how likely is it to be represented in the other?
Studying Learned Features in Language Models
Curiosities about neurons
B
9.56
Can you find a neuron whose activation isn't significantly affected by the current token?
Studying Learned Features in Language Models
Miscellaneous
B
9.57
An important ability of a network is to attend to things within the current clause or sentence. Are models doing something more sophisticated than distance here, like punctuation? If so, are there relevant neurons/features?
Studying Learned Features in Language Models
Miscellaneous
C
9.58
Replicate Knowledge Neurons in Pretrained Transformers on a generative model. How much are these results consistent with what Neuroscope shows?
Studying Learned Features in Language Models
Miscellaneous
A
9.59
Can you replicate the results of the interpretability illusion on SoLU models, which were trained on a mix of web text and Python code? (Find neurons that seem monosemantic on either but with importantly different patterns)
Studying Learned Features in Language Models
Miscellaneous
B
9.6
Try doing dimensionality reduction over neuron activations across a bunch of text, and see how interpretable the resulting directions are.
Studying Learned Features in Language Models
Miscellaneous
B
9.61
Pick a BERTology paper and try to replicate it on GPT-2! (See post for ideas)
Studying Learned Features in Language Models
Miscellaneous
B
9.62
Make a PR to Neuroscope with some feature you wish it had!
Studying Learned Features in Language Models
Miscellaneous
B
9.63
Replicate the part of Conjecture's Polytopes paper where they look at the top eg 1000 dataset examples for a neuron across a ton of text and look for patterns. (Is it the case there are monosemantic bands in the neuron activation spectrum?)
No results from filter
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.