Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
canrager authored Sep 27, 2024
1 parent 2e2e9f8 commit 98d5f57
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
This repo contains the template we would like to use for the SAE Bench project. The `template.ipynb` is a good starting point explaining the input to your custom eval (SAEs hosted on SAELens) and the output (a standardized results file).

The `sparse_probing` folder contains a full example implementation of a custom eval. In `sparse_probing_eval.py`, we have a function that takes a list of SAELens SAE names (defined in `eval_config.py`) and an sae release and returns a dictionary of results in a standard format. This folder contains some helper functions, like pre-computing model activations in `activation_collection.py`, that might be useful for you, too! We try to reuse functions as much as possible across evals to reduce bugs. Let Adam and Can know if you've implemented a helper function that might be useful for other evals as well (like autointerp, feauture scoring).
The `sparse_probing` folder contains a full example implementation of a custom eval. In `sparse_probing_eval.py`, we have a function that takes a list of SAELens SAE names (defined in `eval_config.py`) and an sae release and returns a dictionary of results in a standard format. This folder contains some helper functions, like pre-computing model activations in `activation_collection.py`, that might be useful for you, too! We try to reuse functions as much as possible across evals to reduce bugs. Let Adam and Can know if you've implemented a helper function that might be useful for other evals as well (like autointerp, feature scoring).

`python3 sparse_probing_eval.py` should run as is and demonstrate how to use our SAE Bench SAEs with Transformer Lens and SAE Lens. It will also generate a results file which can be graphed using `graph_sae_results.ipynb`.

Expand Down

0 comments on commit 98d5f57

Please sign in to comment.