EditCoT: Knowledge Editing through Chain of Thought

EditCoT accomplishes knowledge editing by modifying the chain of thought generated by the model. In this repository, we provide:

• Complete framework code for EditCoT.
• All code for training the CoT Editor, including data generation, post-processing, and model training.
• Code for evaluating EditCoT on MQuAKE-CF-3k.

Getting Started

Install Environment

conda create -n editcot python=3.10
conda activate editcot
pip install -r ./requirements.txt

Repository Structure

The ./all_prompts directory contains all the prompts involved in generating datasets and editing chains of thought for EditCoT.

The ./dataset_construct directory includes the code for generating datasets with EditCoT.

The ./train_editor directory contains the code for training the CoT Editor.

The ./eval directory includes the code for evaluating EditCoT on MQuAKE-CF-3k.

Dataset Generation

Run the following command:

bash ./dataset_construct/data_construct.sh

Note that this command will generate intermediate data and the final dataset (dataset.json) in the ./dataset_construct/output directory.

For training convenience, we provide a dataset constructed from Meta-Llama-3-8B-Instruct, located at ./dataset_construct/output/llama3_dataset.json (note: as mentioned in the paper, this dataset samples 80% of the generated data).

Training the CoT Editor

Run the following command:

bash ./train_editor/train.sh

Evaluation

Run the following command:

cd ./eval
python eval.py

Note that you need to set the --editor_path parameter to the path of the trained CoT Editor from the previous step. If you need to modify the model path, maximum epochs, or other parameters, please refer to the parameter settings in eval.py.

Results

The results on MQuAKE-CF-3k are as follows (using Multihop-Accuracy as the evaluation metric):

Method	Meta-Llama-3-8B-Instruct	Qwen2.5-14B-Instruct
EditCoT	35.4	34.2