DSPy - Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2c03f790-bc38-4833-83c1-bce6b7d082f6",
   "metadata": {},
   "source": [
    "# DSPy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a68a0adf-97d1-4e46-94e8-32a616a58bfa",
   "metadata": {},
   "source": [
    "## What?\n",
    "Framework to algorithmically optimizing LM prompts and weights."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfb298f5-6290-49a1-a1ef-0e339f60d697",
   "metadata": {},
   "source": [
    "## Why?\n",
    "Using LMs to build a complex system we generally have to: \n",
    "- Break the problem down into steps\n",
    "- Prompt your LM well until each step works well in isolation\n",
    "- Tweak the steps to work well together\n",
    "- Generate synthetic examples to tune each step\n",
    "- Use these examples to finetune smaller LMs to cut costs.\n",
    "\n",
    "Currently, this is hard and messy: every time you change your pipeline, your LM, or your data, all prompts (or finetuning steps) may need to change."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea4b790f-3593-4856-8511-f3f708b75db5",
   "metadata": {},
   "source": [
    "## How?\n",
    "DSPy breaks this process into following three abstractions:\n",
    "- **Signatures:** Abstract the input and output behaviour\n",
    "- **Modules:** Defines the flow of your program and sets up a pipeline. E.g of modules: `Predict`, `ChainOfThought`, `ProgramOfThought`, `MultiChainComparison` and `React`.\n",
    "- **Optimizers:** Also known as `Teleprompters`, it takes the program, a training set and an evaluation metric and returns a new optimized program for the required use-case. Used to train smaller LMs (student) using larger LMs (teacher).\n",
    "\n",
    "> Training Set can be small or have incomplete examples or without labels unless needed to be used in metric.\n",
    "Metric can be `Exact Match (EM)` or `F1` or any custom defined metric"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87cd9813-2afd-4189-82ac-c8f8eb664737",
   "metadata": {},
   "source": [
    "## DSPy Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f45a43d-37a6-4e87-8916-00407f6e5099",
   "metadata": {},
   "source": [
    "![dspy_arch](./assets/dspy_arch.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efb6a062-aa54-4ecf-bdb5-9cfcfb49adf2",
   "metadata": {},
   "source": [
    "### Signature:\n",
    "Use signature to tell, `what to do`, instead of `how to do`. Need not write huge prompts.\n",
    "DSPy supports inline short strings as signatures, can always write custom classes for the same.\n",
    "\n",
    "Some signatures available in DSPy:\n",
    "\n",
    "| Task                        | Signature                      |\n",
    "|-----------------------------|--------------------------------|\n",
    "| Question-Answering         | \"question -> answer\"           |\n",
    "| Summarization               | \"document -> summary\"          |\n",
    "| Sentiment classification    | \"sentence -> sentiment\"        |\n",
    "| RAG                         | \"context, question -> answer\"  |\n",
    "| MCQs with Reasoning        | \"question, choices -> reasoning, selection\" |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f1cbedc-74ac-414f-b012-1c05832c9bdd",
   "metadata": {},
   "source": [
    "### Module:\n",
    "Takes the signature and converts it into a sophisticated prompt, based on a given technique and LLM used. Can be thought of as a model layer defined in Pytorch that learns from data (input/output)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ac58f58-0541-4329-9c03-88f31477b927",
   "metadata": {},
   "source": [
    "### Optimizer:\n",
    "DSPy optimizer can optimize 3 things\n",
    "- LM weights\n",
    "- Instructions (Prompt/Signature)\n",
    "- Demonstrations of Input-Ouput Behaviour\n",
    "\n",
    "Current available optimizers: [https://dspy-docs.vercel.app/docs/building-blocks/optimizers#what-dspy-optimizers-are-currently-available](https://dspy-docs.vercel.app/docs/building-blocks/optimizers#what-dspy-optimizers-are-currently-available)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e2fe867-8f1f-4b24-a509-f13bd84d1b4a",
   "metadata": {},
   "source": [
    "#### Which optimizer should I use?\n",
    "As a rule of thumb, if you don't know where to start, use `BootstrapFewShotWithRandomSearch`.\n",
    "\n",
    "Here's the general guidance on getting started:\n",
    "- If you have `very little data`, e.g. 10 examples of your task, use `BootstrapFewShot`.\n",
    "- If you have `slightly more data`, e.g. 50 examples of your task, use `BootstrapFewShotWithRandomSearch`.\n",
    "- If you have `more data than that`, e.g. 300 examples or more, use `MIPRO`.\n",
    "- If you have been able to use one of these with a `large LM` (e.g., 7B parameters or above) and need a very efficient program, `compile` that down to a `small LM` with `BootstrapFinetune`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "732c9b43-422e-4822-a342-d08fe125088c",
   "metadata": {},
   "source": [
    "## General Workflow\n",
    "Whatever the task, the general workflow is:\n",
    "\n",
    "- Collect a little bit of data.\n",
    "- Define examples of the inputs and outputs of your program (e.g., questions and their answers). This could just be a handful of quick examples you wrote down. If large datasets exist, the more the merrier!\n",
    "- Define the modules (i.e., sub-tasks) of your program and the way they should interact together to solve your task.\n",
    "- Define some validation logic. What makes for a good run of your program? Maybe the answers need to have a certain length or stick to a particular format? Specify the logic that checks that.\n",
    "- Compile! Ask DSPy to compile your program using your data. The compiler will use your data and validation logic to optimize your program (e.g., prompts and modules) so it's efficient and effective!Iterate.\n",
    "- Repeat the process by improving your data, program, validation, or by using more advanced features of the DSPy compiler."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae50aec8-78fe-4f21-8483-9b9abb9e2361",
   "metadata": {},
   "source": [
    "## Demo - RAG (Unoptimized)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53d845e2-deda-40cb-a65e-47ed9f4027b0",
   "metadata": {},
   "source": [
    "We shall try RAG on truefoundry docs that are ingested in local docker based Qdrant deployment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c694cf6d-c14c-49d6-96dd-175afa338890",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "from embeddings import MixBreadEmbeddings\n",
    "from vectordb.qdrant import CustomQdrantRetriever, QdrantClient\n",
    "from reranker import MxBaiReranker\n",
    "from dspy import OllamaLocal\n",
    "import dspy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "78b000fb-08dc-419c-963a-b8f13730efec",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First need the embedding\n",
    "embedding_model = MixBreadEmbeddings(\n",
    "    model_name=\"mixedbread-ai/mxbai-embed-large-v1\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "aad07409-8d31-4657-9744-d2e4735947e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up Retriever\n",
    "qdrant_client = QdrantClient(url=\"http://localhost:6333\")\n",
    "retriever = CustomQdrantRetriever(\n",
    "    qdrant_collection_name=\"tfdocs\", \n",
    "    qdrant_client=qdrant_client, \n",
    "    embedding_model=embedding_model,\n",
    "    k=5,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9153bd82-7329-469f-b18d-199a2162a8b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "220105c3d8ff49008c230f6d5ccad168",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'long_text': \"# Service\\n### Properties\\n| ports           | \\\\[[Port](doc:service-1#port)]            | true     | Specify the ports you want the service to be exposed to                                                                                                                      |\\n| liveness_probe  | [HealthProbe](doc:service-1#healthprobe) | false    | Describes the configuration for the Health Probe's<br>To learn more you can go [here](doc:add-health-checks-to-deployments)                                                  |\\n| readiness_probe | [HealthProbe](doc:service-1#healthprobe) | false    | Describes the configuration for the Health Probe's<br>To learn more you can go [here](doc:add-health-checks-to-deployments)                                                  |\\n| service_account | string                                   | false    | Service account that this workload should use                                                                                                                                |\"},\n",
       " {'long_text': '# Liveliness/Readiness Probe\\n## HttpProbe\\n### Schema\\n```json\\n{\\n\"type\": \"string\",\\n\"path\": \"string\",\\n\"port\": 65535,\\n\"host\": \"string\",\\n\"scheme\": \"HTTP\"\\n}\\n\\n```'}]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# also have other keys like metadata & score\n",
    "retriever(\"What is service\", k=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8487132f-a8de-40de-99ba-f9d267b490a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up LLM\n",
    "llm = OllamaLocal(\n",
    "    model=\"llama3:8b-instruct-q5_1\", \n",
    "    model_type=\"chat\", \n",
    "    max_tokens=1024, \n",
    "    top_p=1, \n",
    "    top_k=20, \n",
    "    base_url=\"http://localhost:11434\", \n",
    "    frequency_penalty=0.9,\n",
    "    presence_penalty=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "58b81693-a7b9-4af1-8cee-dd1564c66240",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the settings\n",
    "dspy.configure(lm=llm, rm=retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "32872036-8818-4f38-95af-20331821f0b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GenerateAnswer(dspy.Signature):\n",
    "    \"\"\"Answer the question in detail based on the given context.\"\"\"\n",
    "\n",
    "    context = dspy.InputField(desc=\"Contains relevant facts to answer the question\")\n",
    "    question = dspy.InputField()\n",
    "    answer = dspy.OutputField(desc=\"Detailed answer with respect to given question and context\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ce9d0fb9-7f9f-47d1-8485-4d25f7cd7318",
   "metadata": {},
   "outputs": [],
   "source": [
    "class RAG(dspy.Module):\n",
    "\n",
    "    def __init__(self, k: int = 15, reranker_model=\"mixedbread-ai/mxbai-rerank-xsmall-v1\", top_k: int = 5):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.k = k\n",
    "        self.top_k = top_k\n",
    "        self.retriever = dspy.Retrieve(k=self.k)\n",
    "        self.reranker = MxBaiReranker(model_name=reranker_model, k=self.top_k)\n",
    "        self.generate_answer = dspy.Predict(signature=GenerateAnswer)\n",
    "        # can also use CoT\n",
    "        # self.generate_answer = dspy.ChainOfThought(signature=GenerateAnswer)\n",
    "\n",
    "    def forward(self, question, k=None, top_k=None):\n",
    "        passages = self.retriever(question, k).passages\n",
    "        reranked_passages = self.reranker(question, top_k, documents=passages)\n",
    "        prediction = self.generate_answer(context=reranked_passages, question=question)\n",
    "        return dspy.Prediction(context=passages, answer=prediction.answer)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "87fcff52-9136-4f37-ae9a-dda0634ba1c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c89a3faa0fe44fba825a1b0fa971bcf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1fa0b94b71ac4dfd85bf3a0cba209f34",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====================================\n",
      "Answer: According to the provided text, a Service in Truefoundry refers to an application or function that can be deployed using their platform. It's described as \"a single unit of deployment\" which can contain multiple functions and dependencies.\n",
      "\n",
      "In other words, it represents a container for your code (functions) along with its dependencies (e.g., libraries), allowing you to deploy them together in one go.\n",
      "====================================\n"
     ]
    }
   ],
   "source": [
    "# Ask any question you like to this simple RAG program.\n",
    "uncompiled_rag = RAG()\n",
    "my_question = \"What is a service in Truefoundry?\"\n",
    "prediction = uncompiled_rag(my_question, k=10, top_k=5)\n",
    "print(\"====================================\")\n",
    "print(f\"Answer: {prediction.answer}\")\n",
    "print(\"====================================\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4fc56a58-ac40-4eee-a316-9f9b54b72619",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Answer the question in detail based on the given context.\n",
      "\n",
      "---\n",
      "\n",
      "Follow the following format.\n",
      "\n",
      "Context: Contains relevant facts to answer the question\n",
      "Question: ${question}\n",
      "Answer: Detailed answer with respect to given question and context\n",
      "\n",
      "---\n",
      "\n",
      "Context:\n",
      "[1] «{'long_text': '---\\ntitle: \"Introduction to a Service\"\\nslug: \"introduction-to-a-service\"\\nexcerpt: \"\"\\nhidden: false\\ncreatedAt: \"Thu Oct 26 2023 02:02:12 GMT+0000 (Coordinated Universal Time)\"\\nupdatedAt: \"Thu Dec 07 2023 20:15:11 GMT+0000 (Coordinated Universal Time)\"\\n---\\nA Truefoundry Service represents a continuously running application that typically provides a set of APIs for interaction. Services can be dynamically scaled based on incoming traffic or resource demands.\\nServices are perfect for scenarios where real-time responses are essential, such as:\\n- Hosting Real-time Model Inference (e.g., Flask, FastAPI)\\n- Fueling Dynamic Website Backends\\n- Creating Model Demos (e.g., Streamlit, Gradio)'}»\n",
      "[2] «{'long_text': '## Updating Services\\nTo update a service in TrueFoundry, simply deploy the service with the desired changes, maintaining the same service name and workspace. TrueFoundry will automatically detect the update and create a new version.\\nAlternatively, you can update your Machine Learning Service using the **`Edit`** button to modify your Service\\'s configuration, following the instructions below. This will also create and deploy a new version with your changes.\\n[block:embed]\\n{\\n\"html\": false,\\n\"url\": \"https://app.supademo.com/embed/uag-o124hmbQiKYqisyTL\",\\n\"provider\": \"app.supademo.com\",\\n\"href\": \"https://app.supademo.com/embed/uag-o124hmbQiKYqisyTL\",\\n\"typeOfEmbed\": \"iframe\",\\n\"height\": \"475px\",\\n\"width\": \"100%\",\\n\"iframe\": true\\n}\\n[/block]'}»\n",
      "[3] «{'long_text': '## Step 2: Deploying the functions as a service\\nYou can deploy services on TrueFoundry programmatically either using our Python SDK\\n**File Structure:**\\n```\\n.\\n├── module.py\\n├── requirements.txt\\n└── deploy.py\\n```\\n- Here we will use the `FunctionService` class from `servicefoundry` library to define and deploy the service.\\n- We can use the `register_function` method to register the functions we want to deploy.\\n- In the example below, the deployed service will have two different HTTP POST APIs. One for `normal` and another for the `uniform` function.\\n- While deploying, we automatically install the requirements defined in the `requirements.txt` file.\\n> 📘 Note:\\n>\\n> You can also run the service locally using **`service.run().join()`**'}»\n",
      "[4] «{'long_text': '# Interacting with the Application\\nWith the deployment now active, you can click on your specific service. Doing so will open up the dashboard dedicated to your service, allowing you to access various details.  \\nHere, you can see the Endpoint of your service at the top right corner. You can click on the Endpoint to open your application.  \\n[block:image]\\n{\\n\"images\": [\\n{\\n\"image\": [\\n\"https://files.readme.io/fc6d6eb-Screenshot_2023-09-07_at_11.34.02_AM.png\",\\n\"\",\\n\"\"\\n],\\n\"align\": \"center\"\\n}\\n]\\n}\\n[/block]  \\nNow you can click on one of the Images from the two options and see what predictions your model gives:  \\n[block:image]\\n{\\n\"images\": [\\n{\\n\"image\": [\\n\"https://files.readme.io/461a431-Screenshot_2023-09-07_at_11.35.10_AM.png\",\\n\"\",\\n\"\"\\n],\\n\"align\": \"center\"\\n}\\n]\\n}\\n[/block]  \\nCongratulations! You have successfully deployed the async service using Truefoundry.\\n# Sending requests\\n> 📘 The request body for sending a synchronous or an asynchronous process request is the same.'}»\n",
      "[5] «{'long_text': '---\\ntitle: \"Monitor your Service\"\\nslug: \"monitor-your-service\"\\nexcerpt: \"\"\\nhidden: false\\ncreatedAt: \"Mon Oct 30 2023 05:27:05 GMT+0000 (Coordinated Universal Time)\"\\nupdatedAt: \"Fri Dec 08 2023 15:00:45 GMT+0000 (Coordinated Universal Time)\"\\n---\\nTrueFoundry provides you with Logs, Metrics and Events to monitor your deployments to identify and debug issues.'}»\n",
      "Question: What is a service in Truefoundry?\n",
      "Answer:\u001b[32m According to the provided text, a Service in Truefoundry refers to an application or function that can be deployed using their platform. It's described as \"a single unit of deployment\" which can contain multiple functions and dependencies.\n",
      "\n",
      "In other words, it represents a container for your code (functions) along with its dependencies (e.g., libraries), allowing you to deploy them together in one go.\u001b[0m\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Track llm history\n",
    "llm.inspect_history(n=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91d435c6-d3da-46eb-87c0-289df3884f22",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}