skycamp2023_completed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>\n",
    "\n",
    "# DSPy: Tutorial @ SkyCamp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains the **DSPy tutorial** for **SkyCamp 2023**.\n",
    "\n",
    "Let's begin by setting things up. The snippet below will also install **DSPy** if it's not there already."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import sys\n",
    "import os\n",
    "\n",
    "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
    "    import google.colab  # noqa: F401\n",
    "    repo_path = 'dspy'\n",
    "    !git -C $repo_path pull origin || git clone https://github.com/stanfordnlp/dspy $repo_path\n",
    "except:\n",
    "    repo_path = '.'\n",
    "\n",
    "if repo_path not in sys.path:\n",
    "    sys.path.append(repo_path)\n",
    "\n",
    "# Set up the cache for this notebook\n",
    "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = os.path.join(repo_path, 'cache')\n",
    "\n",
    "# import pkg_resources # Install the package if it's not installed\n",
    "# if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",
    "#     !pip install -U pip\n",
    "#     # !pip install dspy-ai\n",
    "#     !pip install -e $repo_path\n",
    "\n",
    "import dspy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n",
      "./cache/compiler\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import sys; sys.path.append('/future/u/okhattab/repos/public/stanfordnlp/dspy')\n",
    "\n",
    "from dspy.evaluate import Evaluate\n",
    "from dspy.teleprompt import BootstrapFewShot, BootstrapFewShotWithRandomSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1) Configure the default LM and retriever\n",
    "\n",
    "We'll start by setting up the language model (LM) and retrieval model (RM). **DSPy** supports multiple API and local models.\n",
    "\n",
    "In this notebook, we will use `Llama2-13b-chat` using the HuggingFace TGI serving software infrastructure. In principle you can run this on your own local GPUs, but for this tutorial all examples are pre-cached so you don't need to worry about cost.\n",
    "\n",
    "We will use the retriever `ColBERTv2`. To make things easy, we've set up a ColBERTv2 server hosting a Wikipedia 2017 \"abstracts\" search index (i.e., containing first paragraph of each article from this [2017 dump](https://hotpotqa.github.io/wiki-readme.html)), so you don't need to worry about setting one up! It's free.\n",
    "\n",
    "**Note:** _If you run this notebook as instructed, you don't need an API key. All examples are already cached internally so you can inspect them!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "llama = dspy.HFClientTGI(model=\"meta-llama/Llama-2-13b-chat-hf\", port=[7140, 7141, 7142, 7143], max_tokens=150)\n",
    "colbertv2 = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')\n",
    "\n",
    "# # NOTE: After you finish this notebook, you can use GPT-3.5 like this if you like.\n",
    "# turbo = dspy.OpenAI(model='gpt-3.5-turbo-instruct')\n",
    "# # In that case, make sure to configure lm=turbo below if you choose to do that.\n",
    "\n",
    "dspy.settings.configure(rm=colbertv2, lm=llama)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Create a few question–answer pairs for our task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = [('Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?', 'Kevin Greutert'),\n",
    "         ('The heir to the Du Pont family fortune sponsored what wrestling team?', 'Foxcatcher'),\n",
    "         ('In what year was the star of To Hell and Back born?', '1925'),\n",
    "         ('Which award did the first book of Gary Zukav receive?', 'U.S. National Book Award'),\n",
    "         ('What documentary about the Gilgo Beach Killer debuted on A&E?', 'The Killing Season'),\n",
    "         ('Which author is English: John Braine or Studs Terkel?', 'John Braine'),\n",
    "         ('Who produced the album that included a re-recording of \"Lithium\"?', 'Butch Vig')]\n",
    "\n",
    "train = [dspy.Example(question=question, answer=answer).with_inputs('question') for question, answer in train]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev = [('Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?', 'E. L. Doctorow'),\n",
    "       ('Right Back At It Again contains lyrics co-written by the singer born in what city?', 'Gainesville, Florida'),\n",
    "       ('What year was the party of the winner of the 1971 San Francisco mayoral election founded?', '1828'),\n",
    "       ('Anthony Dirrell is the brother of which super middleweight title holder?', 'Andre Dirrell'),\n",
    "       ('The sports nutrition business established by Oliver Cookson is based in which county in the UK?', 'Cheshire'),\n",
    "       ('Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.', 'February 13, 1980'),\n",
    "       ('Kyle Moran was born in the town on what river?', 'Castletown River'),\n",
    "       (\"The actress who played the niece in the Priest film was born in what city, country?\", 'Surrey, England'),\n",
    "       ('Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.', 'Portrait of a Marriage'),\n",
    "       ('What year was the father of the Princes in the Tower born?', '1442'),\n",
    "       ('What river is near the Crichton Collegiate Church?', 'the River Tyne'),\n",
    "       ('Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?', 'Renault'),\n",
    "       ('André Zucca was a French photographer who worked with a German propaganda magazine published by what Nazi organization?', 'the Wehrmacht')]\n",
    "\n",
    "dev = [dspy.Example(question=question, answer=answer).with_inputs('question') for question, answer in dev]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Key Concepts: Signatures & Modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(\n",
       "    answer='Berlin'\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define a dspy.Predict module with the signature `question -> answer` (i.e., takes a question and outputs an answer).\n",
    "predict = dspy.Predict('question -> answer')\n",
    "\n",
    "# Use the module!\n",
    "predict(question=\"What is the capital of Germany?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above, we used the `dspy.Predict` module **zero-shot**, i.e. without compiling it on any examples.\n",
    "\n",
    "Let's now build a slightly more advanced program. Our program will use the `dspy.ChainOfThought` module, which asks the LM to think step by step.\n",
    "\n",
    "We will call this program `CoT`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CoT(dspy.Module):  # let's define a new module\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "        # here we declare the chain of thought sub-module, so we can later compile it (e.g., teach it a prompt)\n",
    "        self.generate_answer = dspy.ChainOfThought('question -> answer')\n",
    "    \n",
    "    def forward(self, question):\n",
    "        return self.generate_answer(question=question)  # here we use the module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's compile this using our six `train` examples. We will us the very simple `BootstrapFewShot` in DSPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:00<00:00, 29.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 7 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "metric_EM = dspy.evaluate.answer_exact_match\n",
    "\n",
    "teleprompter = BootstrapFewShot(metric=metric_EM, max_bootstrapped_demos=2)\n",
    "cot_compiled = teleprompter.compile(CoT(), trainset=train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's ask a question to this new program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(\n",
       "    rationale='determine the capital of Germany. We know that the capital of Germany is Berlin, so the answer is Berlin.',\n",
       "    answer='Berlin'\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cot_compiled(\"What is the capital of Germany?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might be curious what's happening under the hood. Let's inspect the last call to our Llama LM to see the prompt and the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Given the fields `question`, produce the fields `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "Answer: Kevin Greutert\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "Answer: U.S. National Book Award\n",
      "\n",
      "Question: What documentary about the Gilgo Beach Killer debuted on A&E?\n",
      "Answer: The Killing Season\n",
      "\n",
      "Question: In what year was the star of To Hell and Back born?\n",
      "Answer: 1925\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "Answer: Foxcatcher\n",
      "\n",
      "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
      "Answer: Butch Vig\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Question: ${question}\n",
      "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
      "Answer: ${answer}\n",
      "\n",
      "---\n",
      "\n",
      "Question: Which author is English: John Braine or Studs Terkel?\n",
      "Reasoning: Let's think step by step in order to determine which author is English. We know that John Braine is English, so we need to determine if Studs Terkel is English. After researching, we found that Studs Terkel was an American author, so the answer is John Braine.\n",
      "Answer: John Braine\n",
      "\n",
      "---\n",
      "\n",
      "Question: What is the capital of Germany?\n",
      "Reasoning: Let's think step by step in order to determine the capital of Germany. We know that the capital of Germany is Berlin, so the answer is Berlin.\n",
      "Answer:\u001b[32m Berlin\n",
      "\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "llama.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the prompt ends with the question we asked (\"What is the capital of Germany?\"), but before that it includes few-shot examples.\n",
    "\n",
    "The final example in the prompt contains a rationale (step-by-step reasoning) self-generated from the LM for use as a demonstration, for the training question \"Which author is English: John Braine or Studs Terkel?\".\n",
    "\n",
    "Now, let's evaluate on our development set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_THREADS = 32\n",
    "evaluate_hotpot = Evaluate(devset=dev, metric=metric_EM, num_threads=NUM_THREADS, display_progress=True, display_table=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's evaluate the compiled `CoT` program with Llama. Feel free to replace `cot_compiled` below with `CoT()` (notice the paranthesis) to test the zero-shot version of CoT."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1): 100%|██████████| 13/13 [00:00<00:00, 117.05it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1%)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_47c09 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_47c09 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_47c09_row0_col0, #T_47c09_row0_col1, #T_47c09_row0_col2, #T_47c09_row0_col3, #T_47c09_row0_col4, #T_47c09_row1_col0, #T_47c09_row1_col1, #T_47c09_row1_col2, #T_47c09_row1_col3, #T_47c09_row1_col4, #T_47c09_row2_col0, #T_47c09_row2_col1, #T_47c09_row2_col2, #T_47c09_row2_col3, #T_47c09_row2_col4, #T_47c09_row3_col0, #T_47c09_row3_col1, #T_47c09_row3_col2, #T_47c09_row3_col3, #T_47c09_row3_col4, #T_47c09_row4_col0, #T_47c09_row4_col1, #T_47c09_row4_col2, #T_47c09_row4_col3, #T_47c09_row4_col4, #T_47c09_row5_col0, #T_47c09_row5_col1, #T_47c09_row5_col2, #T_47c09_row5_col3, #T_47c09_row5_col4, #T_47c09_row6_col0, #T_47c09_row6_col1, #T_47c09_row6_col2, #T_47c09_row6_col3, #T_47c09_row6_col4, #T_47c09_row7_col0, #T_47c09_row7_col1, #T_47c09_row7_col2, #T_47c09_row7_col3, #T_47c09_row7_col4, #T_47c09_row8_col0, #T_47c09_row8_col1, #T_47c09_row8_col2, #T_47c09_row8_col3, #T_47c09_row8_col4, #T_47c09_row9_col0, #T_47c09_row9_col1, #T_47c09_row9_col2, #T_47c09_row9_col3, #T_47c09_row9_col4, #T_47c09_row10_col0, #T_47c09_row10_col1, #T_47c09_row10_col2, #T_47c09_row10_col3, #T_47c09_row10_col4, #T_47c09_row11_col0, #T_47c09_row11_col1, #T_47c09_row11_col2, #T_47c09_row11_col3, #T_47c09_row11_col4, #T_47c09_row12_col0, #T_47c09_row12_col1, #T_47c09_row12_col2, #T_47c09_row12_col3, #T_47c09_row12_col4 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_47c09\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_47c09_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_47c09_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_47c09_level0_col2\" class=\"col_heading level0 col2\" >rationale</th>\n",
       "      <th id=\"T_47c09_level0_col3\" class=\"col_heading level0 col3\" >pred_answer</th>\n",
       "      <th id=\"T_47c09_level0_col4\" class=\"col_heading level0 col4\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_47c09_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_47c09_row0_col1\" class=\"data row0 col1\" >E. L. Doctorow</td>\n",
       "      <td id=\"T_47c09_row0_col2\" class=\"data row0 col2\" >determine who has a broader scope of profession. We know that E. L. Doctorow was a novelist, but Julia Peterkin was a journalist and a...</td>\n",
       "      <td id=\"T_47c09_row0_col3\" class=\"data row0 col3\" >Julia Peterkin</td>\n",
       "      <td id=\"T_47c09_row0_col4\" class=\"data row0 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_47c09_row1_col0\" class=\"data row1 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_47c09_row1_col1\" class=\"data row1 col1\" >Gainesville, Florida</td>\n",
       "      <td id=\"T_47c09_row1_col2\" class=\"data row1 col2\" >determine the answer. We know that the singer was born in Minneapolis, so the answer is Minneapolis.</td>\n",
       "      <td id=\"T_47c09_row1_col3\" class=\"data row1 col3\" >Minneapolis</td>\n",
       "      <td id=\"T_47c09_row1_col4\" class=\"data row1 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_47c09_row2_col0\" class=\"data row2 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_47c09_row2_col1\" class=\"data row2 col1\" >1828</td>\n",
       "      <td id=\"T_47c09_row2_col2\" class=\"data row2 col2\" >determine the year the party of the winner of the 1971 San Francisco mayoral election was founded. We know that the party was founded in...</td>\n",
       "      <td id=\"T_47c09_row2_col3\" class=\"data row2 col3\" >1971</td>\n",
       "      <td id=\"T_47c09_row2_col4\" class=\"data row2 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_47c09_row3_col0\" class=\"data row3 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_47c09_row3_col1\" class=\"data row3 col1\" >Andre Dirrell</td>\n",
       "      <td id=\"T_47c09_row3_col2\" class=\"data row3 col2\" >determine which super middleweight title holder Anthony Dirrell is the brother of. We know that Anthony Dirrell is a professional boxer, and after researching, we...</td>\n",
       "      <td id=\"T_47c09_row3_col3\" class=\"data row3 col3\" >Andre Dirrell</td>\n",
       "      <td id=\"T_47c09_row3_col4\" class=\"data row3 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_47c09_row4_col0\" class=\"data row4 col0\" >The sports nutrition business established by Oliver Cookson is based in which county in the UK?</td>\n",
       "      <td id=\"T_47c09_row4_col1\" class=\"data row4 col1\" >Cheshire</td>\n",
       "      <td id=\"T_47c09_row4_col2\" class=\"data row4 col2\" >determine the county in the UK where Oliver Cookson's sports nutrition business is based. We know that Oliver Cookson is a British entrepreneur, so we...</td>\n",
       "      <td id=\"T_47c09_row4_col3\" class=\"data row4 col3\" >Surrey</td>\n",
       "      <td id=\"T_47c09_row4_col4\" class=\"data row4 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_47c09_row5_col0\" class=\"data row5 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_47c09_row5_col1\" class=\"data row5 col1\" >February 13, 1980</td>\n",
       "      <td id=\"T_47c09_row5_col2\" class=\"data row5 col2\" >determine the birth date of the actor. We know that the actor was born in the 1950s, so we need to narrow down the possible...</td>\n",
       "      <td id=\"T_47c09_row5_col3\" class=\"data row5 col3\" >August 12, 1955</td>\n",
       "      <td id=\"T_47c09_row5_col4\" class=\"data row5 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_47c09_row6_col0\" class=\"data row6 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_47c09_row6_col1\" class=\"data row6 col1\" >Castletown River</td>\n",
       "      <td id=\"T_47c09_row6_col2\" class=\"data row6 col2\" >determine where Kyle Moran was born. We know that Kyle Moran was born in the town on the Delaware River.</td>\n",
       "      <td id=\"T_47c09_row6_col3\" class=\"data row6 col3\" >Delaware River</td>\n",
       "      <td id=\"T_47c09_row6_col4\" class=\"data row6 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_47c09_row7_col0\" class=\"data row7 col0\" >The actress who played the niece in the Priest film was born in what city, country?</td>\n",
       "      <td id=\"T_47c09_row7_col1\" class=\"data row7 col1\" >Surrey, England</td>\n",
       "      <td id=\"T_47c09_row7_col2\" class=\"data row7 col2\" >determine the answer. We know that the actress was born in a city, so we need to determine the country. After researching, we found that...</td>\n",
       "      <td id=\"T_47c09_row7_col3\" class=\"data row7 col3\" >Los Angeles, California</td>\n",
       "      <td id=\"T_47c09_row7_col4\" class=\"data row7 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_47c09_row8_col0\" class=\"data row8 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_47c09_row8_col1\" class=\"data row8 col1\" >Portrait of a Marriage</td>\n",
       "      <td id=\"T_47c09_row8_col2\" class=\"data row8 col2\" >determine the name of the movie. We know that the daughter of Noel Harrison plays Violet Trefusis, so we need to determine the name of...</td>\n",
       "      <td id=\"T_47c09_row8_col3\" class=\"data row8 col3\" >The Remains of the Day</td>\n",
       "      <td id=\"T_47c09_row8_col4\" class=\"data row8 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_47c09_row9_col0\" class=\"data row9 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_47c09_row9_col1\" class=\"data row9 col1\" >1442</td>\n",
       "      <td id=\"T_47c09_row9_col2\" class=\"data row9 col2\" >determine the answer. We know that the father of the Princes in the Tower was born before 1483, so we need to find the correct...</td>\n",
       "      <td id=\"T_47c09_row9_col3\" class=\"data row9 col3\" >1452</td>\n",
       "      <td id=\"T_47c09_row9_col4\" class=\"data row9 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_47c09_row10_col0\" class=\"data row10 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_47c09_row10_col1\" class=\"data row10 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_47c09_row10_col2\" class=\"data row10 col2\" >determine what river is near the Crichton Collegiate Church. We know that the church is located in Scotland, so we need to determine which river...</td>\n",
       "      <td id=\"T_47c09_row10_col3\" class=\"data row10 col3\" >River Tyne</td>\n",
       "      <td id=\"T_47c09_row10_col4\" class=\"data row10 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_47c09_row11_col0\" class=\"data row11 col0\" >Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?</td>\n",
       "      <td id=\"T_47c09_row11_col1\" class=\"data row11 col1\" >Renault</td>\n",
       "      <td id=\"T_47c09_row11_col2\" class=\"data row11 col2\" >determine who purchased the team. We know that Michael Schumacher raced for the Benetton team in the 1995 Monaco Grand Prix, so we need to...</td>\n",
       "      <td id=\"T_47c09_row11_col3\" class=\"data row11 col3\" >Renault</td>\n",
       "      <td id=\"T_47c09_row11_col4\" class=\"data row11 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_47c09_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_47c09_row12_col0\" class=\"data row12 col0\" >André Zucca was a French photographer who worked with a German propaganda magazine published by what Nazi organization?</td>\n",
       "      <td id=\"T_47c09_row12_col1\" class=\"data row12 col1\" >the Wehrmacht</td>\n",
       "      <td id=\"T_47c09_row12_col2\" class=\"data row12 col2\" >determine which Nazi organization André Zucca worked with. We know that he worked with a German propaganda magazine, so we need to determine which Nazi...</td>\n",
       "      <td id=\"T_47c09_row12_col3\" class=\"data row12 col3\" >SS</td>\n",
       "      <td id=\"T_47c09_row12_col4\" class=\"data row12 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fc00c2420d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "23.08"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_hotpot(cot_compiled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Bonus 1: RAG with query generation\n",
    "\n",
    "As a bonus, let's define a more sophisticated program called `RAG`. This program will:\n",
    "\n",
    "- Use the LM to generate a search query based on the input question\n",
    "- Retrieve three passages using our retriever\n",
    "- Use the LM to generate a final answer using these passages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class RAG(dspy.Module):\n",
    "    def __init__(self, num_passages=3):\n",
    "        super().__init__()\n",
    "\n",
    "        # declare three modules: the retriever, a query generator, and an answer generator\n",
    "        self.retrieve = dspy.Retrieve(k=num_passages)\n",
    "        self.generate_query = dspy.ChainOfThought(\"question -> search_query\")\n",
    "        self.generate_answer = dspy.ChainOfThought(\"context, question -> answer\")\n",
    "    \n",
    "    def forward(self, question):\n",
    "        # generate a search query from the question, and use it to retrieve passages\n",
    "        search_query = self.generate_query(question=question).search_query\n",
    "        passages = self.retrieve(search_query).passages\n",
    "\n",
    "        # generate an answer from the passages and the question\n",
    "        return self.generate_answer(context=passages, question=question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of curiosity, we can evaluate the **uncompiled** (or **zero-shot**) version of this program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1): 100%|██████████| 13/13 [00:00<00:00, 45.09it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1%)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "23.08"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_hotpot(RAG(), display_table=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now compile this RAG program. We'll use a slightly more advnaced teleprompter (automatic prompt optimizer) this time, which relies on random search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Going to sample between 1 and 2 traces per predictor.\n",
      "Will attempt to train 8 candidate sets.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1): 100%|██████████| 13/13 [00:00<00:00, 155.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1%)\n",
      "Score: 23.08 for set: [0, 0]\n",
      "New best score: 23.08 for seed -3\n",
      "Scores so far: [23.08]\n",
      "Best score: 23.08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1): 100%|██████████| 13/13 [00:00<00:00, 72.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1%)\n",
      "Score: 23.08 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08]\n",
      "Best score: 23.08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 86%|████████▌ | 6/7 [00:00<00:00, 13.07it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 7 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5): 100%|██████████| 13/13 [00:00<00:00, 45.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5%)\n",
      "Score: 38.46 for set: [7, 7]\n",
      "New best score: 38.46 for seed -1\n",
      "Scores so far: [23.08, 23.08, 38.46]\n",
      "Best score: 38.46\n",
      "Average of max per entry across top 1 scores: 0.38461538461538464\n",
      "Average of max per entry across top 2 scores: 0.46153846153846156\n",
      "Average of max per entry across top 3 scores: 0.46153846153846156\n",
      "Average of max per entry across top 5 scores: 0.46153846153846156\n",
      "Average of max per entry across top 8 scores: 0.46153846153846156\n",
      "Average of max per entry across top 9999 scores: 0.46153846153846156\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:00<00:00, 19.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 7 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 6 / 13  (46.2): 100%|██████████| 13/13 [00:00<00:00, 42.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 6 / 13  (46.2%)\n",
      "Score: 46.15 for set: [7, 7]\n",
      "New best score: 46.15 for seed 0\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 14%|█▍        | 1/7 [00:00<00:00, 21.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8): 100%|██████████| 13/13 [00:00<00:00, 68.72it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8%)\n",
      "Score: 30.77 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15, 30.77]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 29%|██▊       | 2/7 [00:00<00:00, 21.89it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8): 100%|██████████| 13/13 [00:00<00:00, 67.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8%)\n",
      "Score: 30.77 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15, 30.77, 30.77]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43%|████▎     | 3/7 [00:00<00:00, 21.61it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 4 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1): 100%|██████████| 13/13 [00:00<00:00, 61.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1%)\n",
      "Score: 23.08 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15, 30.77, 30.77, 23.08]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 14%|█▍        | 1/7 [00:00<00:00, 21.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5): 100%|██████████| 13/13 [00:00<00:00, 44.95it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5%)\n",
      "Score: 38.46 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15, 30.77, 30.77, 23.08, 38.46]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:00<00:00, 22.62it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 7 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8): 100%|██████████| 13/13 [00:00<00:00, 66.59it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8%)\n",
      "Score: 30.77 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15, 30.77, 30.77, 23.08, 38.46, 30.77]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 57%|█████▋    | 4/7 [00:00<00:00, 23.46it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 5 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8): 100%|██████████| 13/13 [00:00<00:00, 68.29it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8%)\n",
      "Score: 30.77 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15, 30.77, 30.77, 23.08, 38.46, 30.77, 30.77]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 7/7 [00:00<00:00, 20.87it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 7 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8): 100%|██████████| 13/13 [00:00<00:00, 70.76it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 4 / 13  (30.8%)\n",
      "Score: 30.77 for set: [7, 7]\n",
      "Scores so far: [23.08, 23.08, 38.46, 46.15, 30.77, 30.77, 23.08, 38.46, 30.77, 30.77, 30.77]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.5384615384615384\n",
      "Average of max per entry across top 3 scores: 0.5384615384615384\n",
      "Average of max per entry across top 5 scores: 0.5384615384615384\n",
      "Average of max per entry across top 8 scores: 0.5384615384615384\n",
      "Average of max per entry across top 9999 scores: 0.5384615384615384\n",
      "11 candidate programs found.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "teleprompter2 = BootstrapFewShotWithRandomSearch(metric=metric_EM, max_bootstrapped_demos=2, num_candidate_programs=8, num_threads=NUM_THREADS)\n",
    "rag_compiled = teleprompter2.compile(RAG(), trainset=train, valset=dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now evaluate this compiled version of RAG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 6 / 13  (46.2): 100%|██████████| 13/13 [00:00<00:00, 137.18it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 6 / 13  (46.2%)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_bb3b1 th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_bb3b1 td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_bb3b1_row0_col0, #T_bb3b1_row0_col1, #T_bb3b1_row0_col2, #T_bb3b1_row0_col3, #T_bb3b1_row0_col4, #T_bb3b1_row1_col0, #T_bb3b1_row1_col1, #T_bb3b1_row1_col2, #T_bb3b1_row1_col3, #T_bb3b1_row1_col4, #T_bb3b1_row2_col0, #T_bb3b1_row2_col1, #T_bb3b1_row2_col2, #T_bb3b1_row2_col3, #T_bb3b1_row2_col4, #T_bb3b1_row3_col0, #T_bb3b1_row3_col1, #T_bb3b1_row3_col2, #T_bb3b1_row3_col3, #T_bb3b1_row3_col4, #T_bb3b1_row4_col0, #T_bb3b1_row4_col1, #T_bb3b1_row4_col2, #T_bb3b1_row4_col3, #T_bb3b1_row4_col4, #T_bb3b1_row5_col0, #T_bb3b1_row5_col1, #T_bb3b1_row5_col2, #T_bb3b1_row5_col3, #T_bb3b1_row5_col4, #T_bb3b1_row6_col0, #T_bb3b1_row6_col1, #T_bb3b1_row6_col2, #T_bb3b1_row6_col3, #T_bb3b1_row6_col4, #T_bb3b1_row7_col0, #T_bb3b1_row7_col1, #T_bb3b1_row7_col2, #T_bb3b1_row7_col3, #T_bb3b1_row7_col4, #T_bb3b1_row8_col0, #T_bb3b1_row8_col1, #T_bb3b1_row8_col2, #T_bb3b1_row8_col3, #T_bb3b1_row8_col4, #T_bb3b1_row9_col0, #T_bb3b1_row9_col1, #T_bb3b1_row9_col2, #T_bb3b1_row9_col3, #T_bb3b1_row9_col4, #T_bb3b1_row10_col0, #T_bb3b1_row10_col1, #T_bb3b1_row10_col2, #T_bb3b1_row10_col3, #T_bb3b1_row10_col4, #T_bb3b1_row11_col0, #T_bb3b1_row11_col1, #T_bb3b1_row11_col2, #T_bb3b1_row11_col3, #T_bb3b1_row11_col4, #T_bb3b1_row12_col0, #T_bb3b1_row12_col1, #T_bb3b1_row12_col2, #T_bb3b1_row12_col3, #T_bb3b1_row12_col4 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_bb3b1\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_bb3b1_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_bb3b1_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_bb3b1_level0_col2\" class=\"col_heading level0 col2\" >rationale</th>\n",
       "      <th id=\"T_bb3b1_level0_col3\" class=\"col_heading level0 col3\" >pred_answer</th>\n",
       "      <th id=\"T_bb3b1_level0_col4\" class=\"col_heading level0 col4\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_bb3b1_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_bb3b1_row0_col1\" class=\"data row0 col1\" >E. L. Doctorow</td>\n",
       "      <td id=\"T_bb3b1_row0_col2\" class=\"data row0 col2\" >answer this question. We know that E. L. Doctorow and Julia Peterkin are both authors, but we also know that E. L. Doctorow is known...</td>\n",
       "      <td id=\"T_bb3b1_row0_col3\" class=\"data row0 col3\" >E. L. Doctorow.</td>\n",
       "      <td id=\"T_bb3b1_row0_col4\" class=\"data row0 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_bb3b1_row1_col0\" class=\"data row1 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_bb3b1_row1_col1\" class=\"data row1 col1\" >Gainesville, Florida</td>\n",
       "      <td id=\"T_bb3b1_row1_col2\" class=\"data row1 col2\" >answer this question. We know that Beyoncé is the singer who co-wrote the lyrics of \"Right Back At It Again\". We also know that Beyoncé...</td>\n",
       "      <td id=\"T_bb3b1_row1_col3\" class=\"data row1 col3\" >Houston.</td>\n",
       "      <td id=\"T_bb3b1_row1_col4\" class=\"data row1 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_bb3b1_row2_col0\" class=\"data row2 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_bb3b1_row2_col1\" class=\"data row2 col1\" >1828</td>\n",
       "      <td id=\"T_bb3b1_row2_col2\" class=\"data row2 col2\" >answer this question. We know that the winner of the 1971 San Francisco mayoral election was a member of the Democratic Party. We also know...</td>\n",
       "      <td id=\"T_bb3b1_row2_col3\" class=\"data row2 col3\" >1828.</td>\n",
       "      <td id=\"T_bb3b1_row2_col4\" class=\"data row2 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_bb3b1_row3_col0\" class=\"data row3 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_bb3b1_row3_col1\" class=\"data row3 col1\" >Andre Dirrell</td>\n",
       "      <td id=\"T_bb3b1_row3_col2\" class=\"data row3 col2\" >answer this question. We know that Anthony Dirrell is a professional boxer. We also know that he held the WBC super middleweight title from 2014...</td>\n",
       "      <td id=\"T_bb3b1_row3_col3\" class=\"data row3 col3\" >Andre Dirrell.</td>\n",
       "      <td id=\"T_bb3b1_row3_col4\" class=\"data row3 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_bb3b1_row4_col0\" class=\"data row4 col0\" >The sports nutrition business established by Oliver Cookson is based in which county in the UK?</td>\n",
       "      <td id=\"T_bb3b1_row4_col1\" class=\"data row4 col1\" >Cheshire</td>\n",
       "      <td id=\"T_bb3b1_row4_col2\" class=\"data row4 col2\" >answer this question. We know that Oliver Cookson established Myprotein, a sports nutrition business. We also know that Myprotein was sold for a reported £58...</td>\n",
       "      <td id=\"T_bb3b1_row4_col3\" class=\"data row4 col3\" >Cheshire.</td>\n",
       "      <td id=\"T_bb3b1_row4_col4\" class=\"data row4 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_bb3b1_row5_col0\" class=\"data row5 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_bb3b1_row5_col1\" class=\"data row5 col1\" >February 13, 1980</td>\n",
       "      <td id=\"T_bb3b1_row5_col2\" class=\"data row5 col2\" >answer this question. We know that the actor played roles in \"First Wives Club\" and \"Searching for the Elephant\". We also know that the actor's...</td>\n",
       "      <td id=\"T_bb3b1_row5_col3\" class=\"data row5 col3\" >October 17, 1976.</td>\n",
       "      <td id=\"T_bb3b1_row5_col4\" class=\"data row5 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_bb3b1_row6_col0\" class=\"data row6 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_bb3b1_row6_col1\" class=\"data row6 col1\" >Castletown River</td>\n",
       "      <td id=\"T_bb3b1_row6_col2\" class=\"data row6 col2\" >answer this question. We know that Kyle Moran is an actor who was born in Livingston. We also know that Livingston is a town in...</td>\n",
       "      <td id=\"T_bb3b1_row6_col3\" class=\"data row6 col3\" >River Forth.</td>\n",
       "      <td id=\"T_bb3b1_row6_col4\" class=\"data row6 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_bb3b1_row7_col0\" class=\"data row7 col0\" >The actress who played the niece in the Priest film was born in what city, country?</td>\n",
       "      <td id=\"T_bb3b1_row7_col1\" class=\"data row7 col1\" >Surrey, England</td>\n",
       "      <td id=\"T_bb3b1_row7_col2\" class=\"data row7 col2\" >answer this question. We know that Lily Collins is an actress and the daughter of Phil Collins. We also know that she was born in...</td>\n",
       "      <td id=\"T_bb3b1_row7_col3\" class=\"data row7 col3\" >Surrey, England.</td>\n",
       "      <td id=\"T_bb3b1_row7_col4\" class=\"data row7 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_bb3b1_row8_col0\" class=\"data row8 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_bb3b1_row8_col1\" class=\"data row8 col1\" >Portrait of a Marriage</td>\n",
       "      <td id=\"T_bb3b1_row8_col2\" class=\"data row8 col2\" >answer this question. We know that Noel Harrison is the father of Dhani Harrison. We also know that Dhani Harrison is a member of the...</td>\n",
       "      <td id=\"T_bb3b1_row8_col3\" class=\"data row8 col3\" >The daughter of Noel Harrison plays Violet Trefusis in the movie \"The Killing Season\".</td>\n",
       "      <td id=\"T_bb3b1_row8_col4\" class=\"data row8 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_bb3b1_row9_col0\" class=\"data row9 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_bb3b1_row9_col1\" class=\"data row9 col1\" >1442</td>\n",
       "      <td id=\"T_bb3b1_row9_col2\" class=\"data row9 col2\" >answer this question. We know that the father of the Princes in the Tower was King Richard III of England. We also know that he...</td>\n",
       "      <td id=\"T_bb3b1_row9_col3\" class=\"data row9 col3\" >1452.</td>\n",
       "      <td id=\"T_bb3b1_row9_col4\" class=\"data row9 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_bb3b1_row10_col0\" class=\"data row10 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_bb3b1_row10_col1\" class=\"data row10 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_bb3b1_row10_col2\" class=\"data row10 col2\" >answer this question. We know that Crichton Collegiate Church is situated in Midlothian, Scotland. We also know that the church is near the hamlet of...</td>\n",
       "      <td id=\"T_bb3b1_row10_col3\" class=\"data row10 col3\" >River Tyne.</td>\n",
       "      <td id=\"T_bb3b1_row10_col4\" class=\"data row10 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_bb3b1_row11_col0\" class=\"data row11 col0\" >Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?</td>\n",
       "      <td id=\"T_bb3b1_row11_col1\" class=\"data row11 col1\" >Renault</td>\n",
       "      <td id=\"T_bb3b1_row11_col2\" class=\"data row11 col2\" >answer this question. We know that Michael Schumacher raced for the Benetton team in the 1995 Monaco Grand Prix. We also know that Gilberto Benetton...</td>\n",
       "      <td id=\"T_bb3b1_row11_col3\" class=\"data row11 col3\" >Gilberto Benetton.</td>\n",
       "      <td id=\"T_bb3b1_row11_col4\" class=\"data row11 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_bb3b1_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_bb3b1_row12_col0\" class=\"data row12 col0\" >André Zucca was a French photographer who worked with a German propaganda magazine published by what Nazi organization?</td>\n",
       "      <td id=\"T_bb3b1_row12_col1\" class=\"data row12 col1\" >the Wehrmacht</td>\n",
       "      <td id=\"T_bb3b1_row12_col2\" class=\"data row12 col2\" >answer this question. We know that André Zucca was a French photographer who worked with a German propaganda magazine. We also know that the magazine...</td>\n",
       "      <td id=\"T_bb3b1_row12_col3\" class=\"data row12 col3\" >Nazi organization.</td>\n",
       "      <td id=\"T_bb3b1_row12_col4\" class=\"data row12 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fbfd47102b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "46.15"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_hotpot(rag_compiled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect one of the LM calls for this. Focus in particular on the structure of the last few input/output examples in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Given the fields `context`, `question`, produce the fields `answer`.\n",
      "\n",
      "---\n",
      "\n",
      "Question: Which author is English: John Braine or Studs Terkel?\n",
      "Answer: John Braine\n",
      "\n",
      "Question: The heir to the Du Pont family fortune sponsored what wrestling team?\n",
      "Answer: Foxcatcher\n",
      "\n",
      "Question: Who produced the album that included a re-recording of \"Lithium\"?\n",
      "Answer: Butch Vig\n",
      "\n",
      "Question: In what year was the star of To Hell and Back born?\n",
      "Answer: 1925\n",
      "\n",
      "Question: What documentary about the Gilgo Beach Killer debuted on A&E?\n",
      "Answer: The Killing Season\n",
      "\n",
      "Question: Who was the director of the 2009 movie featuring Peter Outerbridge as William Easton?\n",
      "Answer: Kevin Greutert\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context: ${context}\n",
      "\n",
      "Question: ${question}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
      "\n",
      "Answer: ${answer}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «The Dancing Wu Li Masters | The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the book is mostly concerned with the work of pioneers in western physics down through the ages.»\n",
      "[2] «Gary Zukav | Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award.»\n",
      "[3] «Li Junfeng | Master Li Junfeng (born October 13, 1938 in Gaocheng, Hebei) is a qigong master, the founder of Sheng Zhen Qigong, and a world-renowned wushu coach. He has also starred-in and choreographed several Chinese martial arts films.»\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "\n",
      "Reasoning: Let's think step by step in order to answer this question. We know that Gary Zukav is the author of \"The Dancing Wu Li Masters\". We also know that this book was awarded a U.S. National Book Award in category of Science. Therefore, the answer is the U.S. National Book Award.\n",
      "\n",
      "Answer: U.S. National Book Award.\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Democratic Party (United States) | The Democratic Party is one of the two major contemporary political parties in the United States, along with the Republican Party. Tracing its heritage back to Thomas Jefferson and James Madison's Democratic-Republican Party, the modern-day Democratic Party was founded around 1828 by supporters of Andrew Jackson, making it the world's oldest political party.»\n",
      "[2] «Democratic Party (South Korea, 2008) | The Democratic Party (Hangul: 민주당 hanja: 民主黨 ) was a liberal political party in South Korea. Since its foundation in 2008, it was the main opposition party in the 18th Assembly. In late 2011, it merged into the Democratic United Party.»\n",
      "[3] «Democrat Party (Turkey, current) | The Democratic Party (Turkish: \"Demokrat Parti\" ), abbreviated to DP, is a centre-right, conservative Turkish political party, established by Süleyman Demirel in 1983 as the True Path Party (Turkish: \"Doğru Yol Partisi\" or DYP). It succeeded the historical Democratic Party and the Justice Party, two parties with similar ideologies.»\n",
      "\n",
      "Question: What year was the party of the winner of the 1971 San Francisco mayoral election founded?\n",
      "\n",
      "Reasoning: Let's think step by step in order to answer this question. We know that the winner of the 1971 San Francisco mayoral election was a member of the Democratic Party. We also know that the Democratic Party was founded around 1828. Therefore, the answer is 1828.\n",
      "\n",
      "Answer:\u001b[32m 1828.\n",
      "\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "rag_compiled(\"What year was the party of the winner of the 1971 San Francisco mayoral election founded?\")\n",
    "llama.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Bonus 2: Multi-Hop Retrieval and Reasoning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now build a simple multi-hop program, which will interleave multiple calls to the LM and the retriever.\n",
    "\n",
    "Please follow the **TODO** instructions below to implement this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dsp.utils.utils import deduplicate\n",
    "\n",
    "class MultiHop(dspy.Module):\n",
    "    def __init__(self, num_passages=3):\n",
    "        super().__init__()\n",
    "\n",
    "        self.retrieve = dspy.Retrieve(k=num_passages)\n",
    "        self.generate_query = dspy.ChainOfThought(\"question -> search_query\")\n",
    "\n",
    "        # TODO: Define a dspy.ChainOfThought module with the signature 'context, question -> search_query'.\n",
    "        self.generate_query_from_context = dspy.ChainOfThought(\"context, question -> search_query\")\n",
    "\n",
    "        self.generate_answer = dspy.ChainOfThought(\"context, question -> answer\")\n",
    "    \n",
    "    def forward(self, question):\n",
    "        passages = []\n",
    "        \n",
    "        search_query = self.generate_query(question=question).search_query\n",
    "        passages += self.retrieve(search_query).passages\n",
    "\n",
    "        # TODO: Use self.generate_query_from_context to generate a search query.\n",
    "        # Note: Modules require named keyword arguments (e.g., context=..., question=...).\n",
    "        search_query = self.generate_query_from_context(context=passages, question=question).search_query\n",
    "\n",
    "        # TODO: Use self.retrieve to retrieve passages. Append them to the list `passages`.\n",
    "        passages += self.retrieve(search_query).passages\n",
    "\n",
    "        return self.generate_answer(context=deduplicate(passages), question=question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1): 100%|██████████| 13/13 [00:00<00:00, 40.91it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1%)\n",
      "Score: 23.08 for set: [0, 0, 0]\n",
      "New best score: 23.08 for seed -3\n",
      "Scores so far: [23.08]\n",
      "Best score: 23.08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1): 100%|██████████| 13/13 [00:00<00:00, 53.59it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 3 / 13  (23.1%)\n",
      "Score: 23.08 for set: [7, 7, 7]\n",
      "Scores so far: [23.08, 23.08]\n",
      "Best score: 23.08\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 57%|█████▋    | 4/7 [00:00<00:00, 11.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 5 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 6 / 13  (46.2): 100%|██████████| 13/13 [00:00<00:00, 27.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 6 / 13  (46.2%)\n",
      "Score: 46.15 for set: [7, 7, 7]\n",
      "New best score: 46.15 for seed -1\n",
      "Scores so far: [23.08, 23.08, 46.15]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.46153846153846156\n",
      "Average of max per entry across top 3 scores: 0.46153846153846156\n",
      "Average of max per entry across top 5 scores: 0.46153846153846156\n",
      "Average of max per entry across top 8 scores: 0.46153846153846156\n",
      "Average of max per entry across top 9999 scores: 0.46153846153846156\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43%|████▎     | 3/7 [00:00<00:00, 15.41it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 4 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Average Metric: 5 / 13  (38.5): 100%|██████████| 13/13 [00:00<00:00, 27.45it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5%)\n",
      "Score: 38.46 for set: [7, 7, 7]\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46]\n",
      "Best score: 46.15\n",
      "Average of max per entry across top 1 scores: 0.46153846153846156\n",
      "Average of max per entry across top 2 scores: 0.46153846153846156\n",
      "Average of max per entry across top 3 scores: 0.46153846153846156\n",
      "Average of max per entry across top 5 scores: 0.46153846153846156\n",
      "Average of max per entry across top 8 scores: 0.46153846153846156\n",
      "Average of max per entry across top 9999 scores: 0.46153846153846156\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 14%|█▍        | 1/7 [00:00<00:00, 16.50it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 7 / 13  (53.8): 100%|██████████| 13/13 [00:00<00:00, 37.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 7 / 13  (53.8%)\n",
      "Score: 53.85 for set: [7, 7, 7]\n",
      "New best score: 53.85 for seed 1\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46, 53.85]\n",
      "Best score: 53.85\n",
      "Average of max per entry across top 1 scores: 0.5384615384615384\n",
      "Average of max per entry across top 2 scores: 0.6153846153846154\n",
      "Average of max per entry across top 3 scores: 0.6153846153846154\n",
      "Average of max per entry across top 5 scores: 0.6153846153846154\n",
      "Average of max per entry across top 8 scores: 0.6153846153846154\n",
      "Average of max per entry across top 9999 scores: 0.6153846153846154\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 29%|██▊       | 2/7 [00:00<00:00, 17.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 8 / 13  (61.5): 100%|██████████| 13/13 [00:00<00:00, 55.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 8 / 13  (61.5%)\n",
      "Score: 61.54 for set: [7, 7, 7]\n",
      "New best score: 61.54 for seed 2\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46, 53.85, 61.54]\n",
      "Best score: 61.54\n",
      "Average of max per entry across top 1 scores: 0.6153846153846154\n",
      "Average of max per entry across top 2 scores: 0.6153846153846154\n",
      "Average of max per entry across top 3 scores: 0.6923076923076923\n",
      "Average of max per entry across top 5 scores: 0.6923076923076923\n",
      "Average of max per entry across top 8 scores: 0.6923076923076923\n",
      "Average of max per entry across top 9999 scores: 0.6923076923076923\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43%|████▎     | 3/7 [00:00<00:00, 17.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 4 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 8 / 13  (61.5): 100%|██████████| 13/13 [00:00<00:00, 50.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 8 / 13  (61.5%)\n",
      "Score: 61.54 for set: [7, 7, 7]\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46, 53.85, 61.54, 61.54]\n",
      "Best score: 61.54\n",
      "Average of max per entry across top 1 scores: 0.6153846153846154\n",
      "Average of max per entry across top 2 scores: 0.6153846153846154\n",
      "Average of max per entry across top 3 scores: 0.6153846153846154\n",
      "Average of max per entry across top 5 scores: 0.6923076923076923\n",
      "Average of max per entry across top 8 scores: 0.6923076923076923\n",
      "Average of max per entry across top 9999 scores: 0.6923076923076923\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 14%|█▍        | 1/7 [00:00<00:00, 11.73it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 2 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5): 100%|██████████| 13/13 [00:00<00:00, 38.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5%)\n",
      "Score: 38.46 for set: [7, 7, 7]\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46, 53.85, 61.54, 61.54, 38.46]\n",
      "Best score: 61.54\n",
      "Average of max per entry across top 1 scores: 0.6153846153846154\n",
      "Average of max per entry across top 2 scores: 0.6153846153846154\n",
      "Average of max per entry across top 3 scores: 0.6153846153846154\n",
      "Average of max per entry across top 5 scores: 0.6923076923076923\n",
      "Average of max per entry across top 8 scores: 0.6923076923076923\n",
      "Average of max per entry across top 9999 scores: 0.6923076923076923\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 71%|███████▏  | 5/7 [00:00<00:00, 17.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 6 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5): 100%|██████████| 13/13 [00:00<00:00, 34.45it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5%)\n",
      "Score: 38.46 for set: [7, 7, 7]\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46, 53.85, 61.54, 61.54, 38.46, 38.46]\n",
      "Best score: 61.54\n",
      "Average of max per entry across top 1 scores: 0.6153846153846154\n",
      "Average of max per entry across top 2 scores: 0.6153846153846154\n",
      "Average of max per entry across top 3 scores: 0.6153846153846154\n",
      "Average of max per entry across top 5 scores: 0.6923076923076923\n",
      "Average of max per entry across top 8 scores: 0.6923076923076923\n",
      "Average of max per entry across top 9999 scores: 0.6923076923076923\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 29%|██▊       | 2/7 [00:00<00:00, 20.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 1 full traces after 3 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5): 100%|██████████| 13/13 [00:00<00:00, 56.06it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 5 / 13  (38.5%)\n",
      "Score: 38.46 for set: [7, 7, 7]\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46, 53.85, 61.54, 61.54, 38.46, 38.46, 38.46]\n",
      "Best score: 61.54\n",
      "Average of max per entry across top 1 scores: 0.6153846153846154\n",
      "Average of max per entry across top 2 scores: 0.6153846153846154\n",
      "Average of max per entry across top 3 scores: 0.6153846153846154\n",
      "Average of max per entry across top 5 scores: 0.6923076923076923\n",
      "Average of max per entry across top 8 scores: 0.6923076923076923\n",
      "Average of max per entry across top 9999 scores: 0.6923076923076923\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43%|████▎     | 3/7 [00:00<00:00, 20.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 2 full traces after 4 examples in round 0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 7 / 13  (53.8): 100%|██████████| 13/13 [00:00<00:00, 26.05it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 7 / 13  (53.8%)\n",
      "Score: 53.85 for set: [7, 7, 7]\n",
      "Scores so far: [23.08, 23.08, 46.15, 38.46, 53.85, 61.54, 61.54, 38.46, 38.46, 38.46, 53.85]\n",
      "Best score: 61.54\n",
      "Average of max per entry across top 1 scores: 0.6153846153846154\n",
      "Average of max per entry across top 2 scores: 0.6153846153846154\n",
      "Average of max per entry across top 3 scores: 0.6153846153846154\n",
      "Average of max per entry across top 5 scores: 0.8461538461538461\n",
      "Average of max per entry across top 8 scores: 0.8461538461538461\n",
      "Average of max per entry across top 9999 scores: 0.8461538461538461\n",
      "11 candidate programs found.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "multihop_compiled = teleprompter2.compile(MultiHop(), trainset=train, valset=dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Average Metric: 8 / 13  (61.5): 100%|██████████| 13/13 [00:00<00:00, 92.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Metric: 8 / 13  (61.5%)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_930ac th {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_930ac td {\n",
       "  text-align: left;\n",
       "}\n",
       "#T_930ac_row0_col0, #T_930ac_row0_col1, #T_930ac_row0_col2, #T_930ac_row0_col3, #T_930ac_row0_col4, #T_930ac_row1_col0, #T_930ac_row1_col1, #T_930ac_row1_col2, #T_930ac_row1_col3, #T_930ac_row1_col4, #T_930ac_row2_col0, #T_930ac_row2_col1, #T_930ac_row2_col2, #T_930ac_row2_col3, #T_930ac_row2_col4, #T_930ac_row3_col0, #T_930ac_row3_col1, #T_930ac_row3_col2, #T_930ac_row3_col3, #T_930ac_row3_col4, #T_930ac_row4_col0, #T_930ac_row4_col1, #T_930ac_row4_col2, #T_930ac_row4_col3, #T_930ac_row4_col4, #T_930ac_row5_col0, #T_930ac_row5_col1, #T_930ac_row5_col2, #T_930ac_row5_col3, #T_930ac_row5_col4, #T_930ac_row6_col0, #T_930ac_row6_col1, #T_930ac_row6_col2, #T_930ac_row6_col3, #T_930ac_row6_col4, #T_930ac_row7_col0, #T_930ac_row7_col1, #T_930ac_row7_col2, #T_930ac_row7_col3, #T_930ac_row7_col4, #T_930ac_row8_col0, #T_930ac_row8_col1, #T_930ac_row8_col2, #T_930ac_row8_col3, #T_930ac_row8_col4, #T_930ac_row9_col0, #T_930ac_row9_col1, #T_930ac_row9_col2, #T_930ac_row9_col3, #T_930ac_row9_col4, #T_930ac_row10_col0, #T_930ac_row10_col1, #T_930ac_row10_col2, #T_930ac_row10_col3, #T_930ac_row10_col4, #T_930ac_row11_col0, #T_930ac_row11_col1, #T_930ac_row11_col2, #T_930ac_row11_col3, #T_930ac_row11_col4, #T_930ac_row12_col0, #T_930ac_row12_col1, #T_930ac_row12_col2, #T_930ac_row12_col3, #T_930ac_row12_col4 {\n",
       "  text-align: left;\n",
       "  white-space: pre-wrap;\n",
       "  word-wrap: break-word;\n",
       "  max-width: 400px;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_930ac\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_930ac_level0_col0\" class=\"col_heading level0 col0\" >question</th>\n",
       "      <th id=\"T_930ac_level0_col1\" class=\"col_heading level0 col1\" >example_answer</th>\n",
       "      <th id=\"T_930ac_level0_col2\" class=\"col_heading level0 col2\" >rationale</th>\n",
       "      <th id=\"T_930ac_level0_col3\" class=\"col_heading level0 col3\" >pred_answer</th>\n",
       "      <th id=\"T_930ac_level0_col4\" class=\"col_heading level0 col4\" >answer_exact_match</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_930ac_row0_col0\" class=\"data row0 col0\" >Who has a broader scope of profession: E. L. Doctorow or Julia Peterkin?</td>\n",
       "      <td id=\"T_930ac_row0_col1\" class=\"data row0 col1\" >E. L. Doctorow</td>\n",
       "      <td id=\"T_930ac_row0_col2\" class=\"data row0 col2\" >answer this question. We know that E. L. Doctorow is an American novelist, editor, and professor, and he has been described as one of the...</td>\n",
       "      <td id=\"T_930ac_row0_col3\" class=\"data row0 col3\" >E. L. Doctorow.</td>\n",
       "      <td id=\"T_930ac_row0_col4\" class=\"data row0 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_930ac_row1_col0\" class=\"data row1 col0\" >Right Back At It Again contains lyrics co-written by the singer born in what city?</td>\n",
       "      <td id=\"T_930ac_row1_col1\" class=\"data row1 col1\" >Gainesville, Florida</td>\n",
       "      <td id=\"T_930ac_row1_col2\" class=\"data row1 col2\" >answer this question. We know that Beyoncé is an American singer, songwriter, dancer, and actress, and she was born in Houston, Texas. Her album \"Beyoncé\"...</td>\n",
       "      <td id=\"T_930ac_row1_col3\" class=\"data row1 col3\" >Houston.</td>\n",
       "      <td id=\"T_930ac_row1_col4\" class=\"data row1 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_930ac_row2_col0\" class=\"data row2 col0\" >What year was the party of the winner of the 1971 San Francisco mayoral election founded?</td>\n",
       "      <td id=\"T_930ac_row2_col1\" class=\"data row2 col1\" >1828</td>\n",
       "      <td id=\"T_930ac_row2_col2\" class=\"data row2 col2\" >answer this question. We know that the Democratic Party is one of the two major contemporary political parties in the United States, and it was...</td>\n",
       "      <td id=\"T_930ac_row2_col3\" class=\"data row2 col3\" >1828.</td>\n",
       "      <td id=\"T_930ac_row2_col4\" class=\"data row2 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_930ac_row3_col0\" class=\"data row3 col0\" >Anthony Dirrell is the brother of which super middleweight title holder?</td>\n",
       "      <td id=\"T_930ac_row3_col1\" class=\"data row3 col1\" >Andre Dirrell</td>\n",
       "      <td id=\"T_930ac_row3_col2\" class=\"data row3 col2\" >answer this question. We know that Anthony Dirrell is a professional boxer who held the WBC super middleweight title from 2014 to 2015. We also...</td>\n",
       "      <td id=\"T_930ac_row3_col3\" class=\"data row3 col3\" >Andre Dirrell.</td>\n",
       "      <td id=\"T_930ac_row3_col4\" class=\"data row3 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_930ac_row4_col0\" class=\"data row4 col0\" >The sports nutrition business established by Oliver Cookson is based in which county in the UK?</td>\n",
       "      <td id=\"T_930ac_row4_col1\" class=\"data row4 col1\" >Cheshire</td>\n",
       "      <td id=\"T_930ac_row4_col2\" class=\"data row4 col2\" >answer this question. We know that Oliver Cookson is a UK entrepreneur who established the sports nutrition business Myprotein. We also know that Myprotein was...</td>\n",
       "      <td id=\"T_930ac_row4_col3\" class=\"data row4 col3\" >Cheshire.</td>\n",
       "      <td id=\"T_930ac_row4_col4\" class=\"data row4 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_930ac_row5_col0\" class=\"data row5 col0\" >Find the birth date of the actor who played roles in First Wives Club and Searching for the Elephant.</td>\n",
       "      <td id=\"T_930ac_row5_col1\" class=\"data row5 col1\" >February 13, 1980</td>\n",
       "      <td id=\"T_930ac_row5_col2\" class=\"data row5 col2\" >answer this question. We know that the actor's name is Jo Dong-hyuk, and he was born on December 11, 1977. Therefore, the answer is December...</td>\n",
       "      <td id=\"T_930ac_row5_col3\" class=\"data row5 col3\" >December 11, 1977.</td>\n",
       "      <td id=\"T_930ac_row5_col4\" class=\"data row5 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_930ac_row6_col0\" class=\"data row6 col0\" >Kyle Moran was born in the town on what river?</td>\n",
       "      <td id=\"T_930ac_row6_col1\" class=\"data row6 col1\" >Castletown River</td>\n",
       "      <td id=\"T_930ac_row6_col2\" class=\"data row6 col2\" >answer this question. We know that Kyle Moran is an Irish footballer who plays as a forward for Perth SC in the NPL Western Australia....</td>\n",
       "      <td id=\"T_930ac_row6_col3\" class=\"data row6 col3\" >River Dundalk.</td>\n",
       "      <td id=\"T_930ac_row6_col4\" class=\"data row6 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_930ac_row7_col0\" class=\"data row7 col0\" >The actress who played the niece in the Priest film was born in what city, country?</td>\n",
       "      <td id=\"T_930ac_row7_col1\" class=\"data row7 col1\" >Surrey, England</td>\n",
       "      <td id=\"T_930ac_row7_col2\" class=\"data row7 col2\" >answer this question. We know that Lily Collins is an actress, and she was born in Surrey, England. Therefore, the answer is Surrey, England.</td>\n",
       "      <td id=\"T_930ac_row7_col3\" class=\"data row7 col3\" >Surrey, England.</td>\n",
       "      <td id=\"T_930ac_row7_col4\" class=\"data row7 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_930ac_row8_col0\" class=\"data row8 col0\" >Name the movie in which the daughter of Noel Harrison plays Violet Trefusis.</td>\n",
       "      <td id=\"T_930ac_row8_col1\" class=\"data row8 col1\" >Portrait of a Marriage</td>\n",
       "      <td id=\"T_930ac_row8_col2\" class=\"data row8 col2\" >answer this question. We know that Cathryn Harrison is the daughter of Noel Harrison, and she is an English actress. One of her roles was...</td>\n",
       "      <td id=\"T_930ac_row8_col3\" class=\"data row8 col3\" >First Daughter.</td>\n",
       "      <td id=\"T_930ac_row8_col4\" class=\"data row8 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_930ac_row9_col0\" class=\"data row9 col0\" >What year was the father of the Princes in the Tower born?</td>\n",
       "      <td id=\"T_930ac_row9_col1\" class=\"data row9 col1\" >1442</td>\n",
       "      <td id=\"T_930ac_row9_col2\" class=\"data row9 col2\" >answer this question. We know that the father of the Princes in the Tower was King Richard III of England, and he was born on...</td>\n",
       "      <td id=\"T_930ac_row9_col3\" class=\"data row9 col3\" >1452.</td>\n",
       "      <td id=\"T_930ac_row9_col4\" class=\"data row9 col4\" >❌ [False]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_930ac_row10_col0\" class=\"data row10 col0\" >What river is near the Crichton Collegiate Church?</td>\n",
       "      <td id=\"T_930ac_row10_col1\" class=\"data row10 col1\" >the River Tyne</td>\n",
       "      <td id=\"T_930ac_row10_col2\" class=\"data row10 col2\" >answer this question. We know that Crichton Collegiate Church is situated about 0.6 mi south west of the hamlet of Crichton in Midlothian, Scotland. We...</td>\n",
       "      <td id=\"T_930ac_row10_col3\" class=\"data row10 col3\" >River Tyne.</td>\n",
       "      <td id=\"T_930ac_row10_col4\" class=\"data row10 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_930ac_row11_col0\" class=\"data row11 col0\" >Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?</td>\n",
       "      <td id=\"T_930ac_row11_col1\" class=\"data row11 col1\" >Renault</td>\n",
       "      <td id=\"T_930ac_row11_col2\" class=\"data row11 col2\" >answer this question. We know that Michael Schumacher raced for the Benetton team in the 1995 Monaco Grand Prix. In 2000, the Benetton team was...</td>\n",
       "      <td id=\"T_930ac_row11_col3\" class=\"data row11 col3\" >Renault.</td>\n",
       "      <td id=\"T_930ac_row11_col4\" class=\"data row11 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_930ac_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_930ac_row12_col0\" class=\"data row12 col0\" >André Zucca was a French photographer who worked with a German propaganda magazine published by what Nazi organization?</td>\n",
       "      <td id=\"T_930ac_row12_col1\" class=\"data row12 col1\" >the Wehrmacht</td>\n",
       "      <td id=\"T_930ac_row12_col2\" class=\"data row12 col2\" >answer this question. We know that André Zucca was a French photographer who worked with a German propaganda magazine called \"Signal\". Therefore, the answer is...</td>\n",
       "      <td id=\"T_930ac_row12_col3\" class=\"data row12 col3\" >Wehrmacht.</td>\n",
       "      <td id=\"T_930ac_row12_col4\" class=\"data row12 col4\" >✔️ [True]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7fbfd45d8ca0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "61.54"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_hotpot(multihop_compiled, devset=dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now inspect the prompt for the second-hop search query for one of the questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Given the fields `context`, `question`, produce the fields `search_query`.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context: ${context}\n",
      "\n",
      "Question: ${question}\n",
      "\n",
      "Reasoning: Let's think step by step in order to ${produce the search_query}. We ...\n",
      "\n",
      "Search Query: ${search_query}\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «The Dancing Wu Li Masters | The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the book is mostly concerned with the work of pioneers in western physics down through the ages.»\n",
      "[2] «Gary Zukav | Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award.»\n",
      "[3] «Li Junfeng | Master Li Junfeng (born October 13, 1938 in Gaocheng, Hebei) is a qigong master, the founder of Sheng Zhen Qigong, and a world-renowned wushu coach. He has also starred-in and choreographed several Chinese martial arts films.»\n",
      "[4] «The Dancing Wu Li Masters | The Dancing Wu Li Masters is a 1979 book by Gary Zukav, a popular science work exploring modern physics, and quantum phenomena in particular. It was awarded a 1980 U.S. National Book Award in category of Science. Although it explores empirical topics in modern physics research, \"The Dancing Wu Li Masters\" gained attention for leveraging metaphors taken from eastern spiritual movements, in particular the Huayen school of Buddhism with the monk Fazang's treatise on The Golden Lion, to explain quantum phenomena and has been regarded by some reviewers as a New Age work, although the book is mostly concerned with the work of pioneers in western physics down through the ages.»\n",
      "[5] «Gary Zukav | Gary Zukav (born October 17, 1942) is an American spiritual teacher and the author of four consecutive New York Times Best Sellers. Beginning in 1998, he appeared more than 30 times on \"The Oprah Winfrey Show\" to discuss transformation in human consciousness concepts presented in his book \"The Seat of the Soul\". His first book, \"The Dancing Wu Li Masters\" (1979), won a U.S. National Book Award.»\n",
      "[6] «Wu Pao-chun | Wu Pao-chun (, born 5 September 1970), is a Taiwanese baker best known for winning the title of Master Baker in the bread category of the 2010 Bakery Masters competition held in Paris. Wu is also known for a rose-lychee bread he created which includes Taiwanese ingredients such as millet wine, rose petals and dried lychees.»\n",
      "\n",
      "Question: Which award did the first book of Gary Zukav receive?\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the search query. We know that the first book of Gary Zukav is \"The Dancing Wu Li Masters\". We also know that this book received an award. Therefore, we can start by searching for the name of the award that the book received.\n",
      "\n",
      "Search Query: \"The Dancing Wu Li Masters\" award\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «Benetton Group | Benetton Group S.r.l. (correct ] ; often mispronounced ] or ] ) is a global fashion brand, based in Ponzano Veneto, Italy. The name comes from the Benetton family who founded the company in 1965.»\n",
      "[2] «Benetton Rugby | Benetton Rugby (] or ] ) are an Italian professional rugby union team based in Treviso, Veneto competing in the Pro14 and the European Rugby Champions Cup.»\n",
      "[3] «Gilberto Benetton | Gilberto Benetton (born 19 June 1941) is an Italian billionaire businessman, one of the co-founders of Benetton Group, the Italian fashion brand.»\n",
      "\n",
      "Question: Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?\n",
      "\n",
      "Reasoning: Let's think step by step in order to produce the search query. We know that Michael Schumacher raced for a team in the 1995 Monaco Grand Prix. We also know that the team was purchased by someone in 2000. Therefore, we can start by searching for the name of the team that Michael Schumacher raced for in the 1995 Monaco Grand Prix.\n",
      "\n",
      "Search Query:\u001b[32m Michael Schumacher team Monaco Grand Prix 1995\n",
      "\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "multihop_compiled(question=\"Who purchased the team Michael Schumacher raced for in the 1995 Monaco Grand Prix in 2000?\")\n",
    "llama.inspect_history(n=1, skip=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py39_aug2023_dspy",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}