Be consistent when setting default dtype

README.md

@@ -54,7 +54,7 @@ For a tutorial of using SAE Lens SAEs, including calculating L0 and Loss Recover
 
 ## Custom SAE Usage
 
-Our goal is to have first class support for custom SAEs as the field is rapidly evolving. Our evaluations can run on any SAE object with encode(), decode(), and a few config values. For example custom SAE implementations and more info, refer to the `baselines/README.md`.
+Our goal is to have first class support for custom SAEs as the field is rapidly evolving. Our evaluations can run on any SAE object with encode(), decode(), and a few config values. For example custom SAE implementations and more info, refer to the `custom_saes/README.md`.
 
 There are two ways to evaluate custom SAEs:
 
@@ -68,7 +68,7 @@ There are two ways to evaluate custom SAEs:
    - Simpler interface requiring only model, SAE, and config values
    - Graphing will require manual formatting
 
-The script `run_all_evals_custom_saes()` will run approach 1 on all SAE Bench evals. We currently have a suite of SAE Bench SAEs on layers 3 and 4 of Pythia-70M and layers 5, 12, and 19 of Gemma-2-2B, each trained on 200M tokens. These SAEs can serve as baselines for any new custom SAEs. We also have baseline eval results, saved at TODO.
+The script `run_all_evals_custom_saes()` will run approach 1 on all SAE Bench evals. We currently have a suite of SAE Bench SAEs on layers 3 and 4 of Pythia-70M and layers 5, 12, and 19 of Gemma-2-2B, each trained on 200M tokens with checkpoints at various points. These SAEs can serve as baselines for any new custom SAEs. We also have baseline eval results, saved at TODO.
 
 ## Training Your Own SAEs
 

evals/autointerp/eval_config.py

@@ -76,7 +76,7 @@ class AutoInterpEvalConfig:
         description="Split up total tokens into batches of this size",
     )
     llm_dtype: str = Field(
-        default="bfloat16",
+        default="float32",
         title="LLM Data Type",
         description="The data type to use for the LLM",
     )

evals/mdl/eval_config.py

@@ -15,6 +15,6 @@ class MDLEvalConfig:
     sae_batch_size: int = 64
 
     model_name: str = "pythia-70m-deduped"
-    llm_dtype: str = "bfloat16"
+    llm_dtype: str = "float32"
 
     mse_epsilon_threshold: float = 0.01

evals/scr_and_tpp/eval_config.py

@@ -87,7 +87,7 @@ def ensure_min_probe_test_batch_size(cls, value: int) -> int:
         description="LLM batch size, inference only",
     )
     llm_dtype: str = Field(
-        default="bfloat16",
+        default="float32",
         title="LLM Dtype",
         description="",
     )