Announcing Open-Source SAEs for Llama 3.3 70B and Llama 3.1 8B

Following our announcement of Goodfire Ember, we're excited to release state-of-the-art, open-sourced sparse autoencoders (SAEs) for Llama 3.1 8B and Llama 3.3 70B. SAEs are interpreter models that help us understand how language models process and represent information internally [1].

We're releasing SAEs for Llama 3.1 8B and Llama 3.3 70B. These models build on top of our earlier work on Llama-3-8B, where we demonstrated the effectiveness of training an SAE on the LMSYS-Chat-1M dataset [2]. Our SAEs are designed to decompose complex neural activations into interpretable features, making it possible to understand and steer model behavior at a granular level.

Our starting point for parameterisation strategy was the Anthropic April update [3]. Our SAEs were evaluated on typical sparsity and fidelity metrics, as well as on overall feature quality for tasks like steering 1. The L0 count varies across model scales, reaching 91 for the Llama 3.1 8B SAE and increasing to 121 in Llama 3.3 70B. For our SAE training, we targeted layer 19 in the 8B model and layer 50 in the 70B variant.