BiasMonkey Example: Initial Jupyter Notebook

[zaidsheikh]

Description

This is a replication of the experiments from BiasMonkey (Tjuatja et al. 2023), which investigates whether LLMs exhibit human-like response biases in survey questionnaires. It is based on the original repo.

Status:

• Generation using HuggingFace models (Llama 2 etc.)
• Generation using GPT-3.5 models
• Add analysis code:
  • Plot heatmap comparing LLMs’ behavior on bias types with their respective behavior on the set of perturbations
  • Compute human and model distributions for all relevant questions and wasserstein distance between the two distributions.
  • Generate uncertainty measures for all models across response biases and non-bias perturbations.

References

• Part of Implement BiasMonkey Paper #31
• Generation using GPT-3.5 models blocked by:
  • Implement API-based LLM using litellm #37
  • Implement rate limiting on API-based LM #38

[zaidsheikh]

Blocked by #54