transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/eherrador/NLP/blob/main/transformers.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is NLP?\n",
    "NLP is a field of linguistics and machine learning focused on understanding everything related to human language. The aim of NLP tasks is not only to understand single words individually, but to be able to understand the context of those words.\n",
    "\n",
    "The following is a list of common NLP tasks, with some examples of each:\n",
    "- **Classifying whole sentences:** Getting the sentiment of a review, detecting if an email is spam, determining if a sentence is grammatically correct or whether two sentences are logically related or not\n",
    "- **Classifying each word in a sentence:** Identifying the grammatical components of a sentence (noun, verb, adjective), or the named entities (person, location, organization)\n",
    "- **Generating text content:** Completing a prompt with auto-generated text, filling in the blanks in a text with masked words\n",
    "- **Extracting an answer from a text:** Given a question and a context, extracting the answer to the question based on the information provided in the context\n",
    "- **Generating a new sentence from an input text:** Translating a text into another language, summarizing a text\n",
    "\n",
    "In Machine Learning, there are different ways to process natural language:\n",
    "- **Classification and vector spaces.** Using logistic regression, naive Bayes and word vectors to apply sentiment analysis, complete analogies and translate words.\n",
    "- **Probabilistic Models** Using dynamic programming, hidden Markov models, and word embedding to implement autocorrection, autocompletion, and part-of-speech tag identification for words.\n",
    "- **Sequence models.** Using recurrent neural networks, LSTM, GRU and Siamese networks in Trax for sentiment analysis, text generation and named entity recognition.\n",
    "- **Attention models.** One of these deep learning models are `transformers`, which we will explain in detail later.\n",
    "\n",
    "NLP isn't limited to written text though. It also tackles complex challenges in speech recognition and computer vision, such as generating a transcript of an audio sample or a description of an image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why is it challenging?\n",
    "Computers don't process information in the same way as humans. For example, when we read the sentence \"I am hungry\", we can easily understand its meaning. Similarly, given two sentences such as \"I am hungry\" and \"I am sad\", we're able to easily determine how similar they are. For machine learning (ML) models, such tasks are more difficult. The text needs to be processed in a way that enables the model to learn from it. And because language is complex, we need to think carefully about how this processing must be done. There has been a lot of research done on how to represent text, and we will look at some methods in the next chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transformers\n",
    "In machine learning, a transformer is a specific type of deep learning model that excels at natural language processing (NLP) tasks. It was introduced in a 2017 research paper titled \"Attention is All You Need\" [arxiv.org].\n",
    "\n",
    "Here's what makes transformers unique:\n",
    "- **Attention mechanism:** Unlike traditional models that process data sequentially, transformers use an attention mechanism. This allows the model to focus on specific parts of the input data (like words in a sentence) that are relevant to the current task.\n",
    "- **Parallel processing:** By using the attention mechanism, transformers can consider all parts of the input data simultaneously. This makes them faster to train compared to sequential models.\n",
    "- **Encoder-decoder architecture (for some tasks):** Transformers can be used in an encoder-decoder architecture for tasks like machine translation. In this setup, the encoder processes the source language, and the decoder generates the target language translation.\n",
    "\n",
    "Transformers have become a foundational block for various NLP tasks, including:\n",
    "- Machine translation\n",
    "- Text summarization\n",
    "- Question answering\n",
    "- Text generation\n",
    "\n",
    "Their ability to understand complex relationships within sequences of data makes them powerful tools in the field of NLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with pipelines\n",
    "The most basic object in the 🤗 Transformers library is the pipeline() function. It connects a model with its necessary preprocessing and postprocessing steps, allowing us to directly input any text and get an intelligible answer.\n",
    "\n",
    "By default, this pipeline selects a particular pretrained model that has been fine-tuned for sentiment analysis in English. The model is downloaded and cached when you create the classifier object. If you rerun the command, the cached model will be used instead and there is no need to download the model again.\n",
    "\n",
    "There are three main steps involved when you pass some text to a pipeline:\n",
    "- The text is preprocessed into a format the model can understand.\n",
    "- The preprocessed inputs are passed to the model.\n",
    "- The predictions of the model are post-processed, so you can make sense of them.\n",
    "\n",
    "Some of the currently available pipelines are:\n",
    "- sentiment-analysis\n",
    "- zero-shot-classification\n",
    "- text-generation\n",
    "- fill-mask\n",
    "- ner (named entity recognition)\n",
    "- question-answering\n",
    "- summarization\n",
    "- translation\n",
    "- feature-extraction (get the vector representation of a text)\n",
    "\n",
    "Let’s have a look at a few of these!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the Transformers, Datasets, and Evaluate libraries to run this notebook.\n",
    "\n",
    "[SentencePiece](https://github.com/google/sentencepiece)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install datasets evaluate transformers[sentencepiece]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "classifier = pipeline(\"sentiment-analysis\")\n",
    "classifier(\"I've been waiting for a HuggingFace course my whole life.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'label': 'POSITIVE', 'score': 0.9598047137260437}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even pass several sentences!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier(\n",
    "    [\"I've been waiting for a HuggingFace course my whole life.\", \"I hate this so much!\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'label': 'POSITIVE', 'score': 0.9598047137260437},\n",
    " {'label': 'NEGATIVE', 'score': 0.9994558095932007}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zero-shot classification\n",
    "We’ll start by tackling a more challenging task where we need to classify texts that haven’t been labelled. This is a common scenario in real-world projects because annotating text is usually time-consuming and requires domain expertise. For this use case, the zero-shot-classification pipeline is very powerful: it allows you to specify which labels to use for the classification, so you don’t have to rely on the labels of the pretrained model. You’ve already seen how the model can classify a sentence as positive or negative using those two labels — but it can also classify the text using any other set of labels you like.\n",
    "\n",
    "This pipeline is called zero-shot because you don’t need to fine-tune the model on your data to use it. It can directly return probability scores for any list of labels you want!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "classifier = pipeline(\"zero-shot-classification\")\n",
    "classifier(\n",
    "    \"This is a course about the Transformers library\",\n",
    "    candidate_labels=[\"education\", \"politics\", \"business\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{'sequence': 'This is a course about the Transformers library',\n",
    " 'labels': ['education', 'business', 'politics'],\n",
    " 'scores': [0.8445963859558105, 0.111976258456707, 0.043427448719739914]}\n",
    " ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text generation\n",
    "Now let’s see how to use a pipeline to generate some text. The main idea here is that you provide a prompt and the model will auto-complete it by generating the remaining text. This is similar to the predictive text feature that is found on many phones. Text generation involves randomness, so it’s normal if you don’t get the same results as shown below.\n",
    "\n",
    "You can control how many different sequences are generated with the argument num_return_sequences and the total length of the output text with the argument max_length.\n",
    "\n",
    "Try it out! Use the num_return_sequences and max_length arguments to generate two sentences of 15 words each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "generator = pipeline(\"text-generation\")\n",
    "generator(\"In this course, we will teach you how to\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'generated_text': 'In this course, we will teach you how to understand and use '\n",
    "                    'data flow and data interchange when handling user data. We '\n",
    "                    'will be working with one or more of the most commonly used '\n",
    "                    'data flows — data flows of various types, as seen by the '\n",
    "                    'HTTP'}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Text generation using any model from 🤗 Hub in a pipelines\n",
    "The previous examples used the default model for the task at hand, but you can also choose a particular model from the Hub to use in a pipeline for a specific task — say, text generation. Go to the [Model Hub](https://huggingface.co/models) and click on the corresponding tag on the left to display only the supported models for that task.\n",
    "\n",
    "Let’s try the [distilgpt2](https://huggingface.co/distilbert/distilgpt2) model! Here’s how to load it in the same pipeline as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "generator = pipeline(\"text-generation\", model=\"distilgpt2\")\n",
    "generator(\n",
    "    \"In this course, we will teach you how to\",\n",
    "    max_length=30,\n",
    "    num_return_sequences=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'generated_text': 'In this course, we will teach you how to manipulate the world and '\n",
    "                    'move your mental and physical capabilities to your advantage.'},\n",
    " {'generated_text': 'In this course, we will teach you how to become an expert and '\n",
    "                    'practice realtime, and with a hands on experience on both real '\n",
    "                    'time and real'}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mask Filling\n",
    "The idea of this task is to fill in the blanks in a given text.\n",
    "\n",
    "The top_k argument controls how many possibilities you want to be displayed. Note that here the model fills in the special <mask> word, which is often referred to as a mask token. Other mask-filling models might have different mask tokens, so it’s always good to verify the proper mask word when exploring other models. One way to check it is by looking at the mask word used in the widget."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "unmasker = pipeline(\"fill-mask\")\n",
    "unmasker(\"This course will teach you all about <mask> models.\", top_k=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'sequence': 'This course will teach you all about mathematical models.',\n",
    "  'score': 0.19619831442832947,\n",
    "  'token': 30412,\n",
    "  'token_str': ' mathematical'},\n",
    " {'sequence': 'This course will teach you all about computational models.',\n",
    "  'score': 0.04052725434303284,\n",
    "  'token': 38163,\n",
    "  'token_str': ' computational'}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NER (Named Entity Recognition)\n",
    "Named entity recognition (NER) is a task where the model has to find which parts of the input text correspond to entities such as persons, locations, or organizations.\n",
    "\n",
    "In the next example, the model correctly identified that Sylvain is a person (PER), Hugging Face an organization (ORG), and Brooklyn a location (LOC).\n",
    "\n",
    "I pass the option grouped_entities=True in the pipeline creation function to tell the pipeline to regroup together the parts of the sentence that correspond to the same entity: here the model correctly grouped “Hugging” and “Face” as a single organization, even though the name consists of multiple words. In fact, as we will see in the next chapter, the preprocessing even splits some words into smaller parts. For instance, Sylvain is split into four pieces: S, ##yl, ##va, and ##in. In the post-processing step, the pipeline successfully regrouped those pieces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "ner = pipeline(\"ner\", grouped_entities=True)\n",
    "ner(\"My name is Sylvain and I work at Hugging Face in Brooklyn.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'entity_group': 'PER', 'score': 0.99816, 'word': 'Sylvain', 'start': 11, 'end': 18}, \n",
    " {'entity_group': 'ORG', 'score': 0.97960, 'word': 'Hugging Face', 'start': 33, 'end': 45}, \n",
    " {'entity_group': 'LOC', 'score': 0.99321, 'word': 'Brooklyn', 'start': 49, 'end': 57}\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question answering\n",
    "The question-answering pipeline answers questions using information from a given context.\n",
    "\n",
    "Note that this pipeline works by extracting information from the provided context; it does not generate the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "question_answerer = pipeline(\"question-answering\")\n",
    "question_answerer(\n",
    "    question=\"Where do I work?\",\n",
    "    context=\"My name is Sylvain and I work at Hugging Face in Brooklyn\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "{'score': 0.6385916471481323, 'start': 33, 'end': 45, 'answer': 'Hugging Face'}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarization\n",
    "Summarization is the task of reducing a text into a shorter text while keeping all (or most) of the important aspects referenced in the text.\n",
    "\n",
    "Like with text generation, you can specify a max_length or a min_length for the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "summarizer = pipeline(\"summarization\")\n",
    "summarizer(\n",
    "    \"\"\"\n",
    "    America has changed dramatically during recent years. Not only has the number of \n",
    "    graduates in traditional engineering disciplines such as mechanical, civil, \n",
    "    electrical, chemical, and aeronautical engineering declined, but in most of \n",
    "    the premier American universities engineering curricula now concentrate on \n",
    "    and encourage largely the study of engineering science. As a result, there \n",
    "    are declining offerings in engineering subjects dealing with infrastructure, \n",
    "    the environment, and related issues, and greater concentration on high \n",
    "    technology subjects, largely supporting increasingly complex scientific \n",
    "    developments. While the latter is important, it should not be at the expense \n",
    "    of more traditional engineering.\n",
    "\n",
    "    Rapidly developing economies such as China and India, as well as other \n",
    "    industrial countries in Europe and Asia, continue to encourage and advance \n",
    "    the teaching of engineering. Both China and India, respectively, graduate \n",
    "    six and eight times as many traditional engineers as does the United States. \n",
    "    Other industrial countries at minimum maintain their output, while America \n",
    "    suffers an increasingly serious decline in the number of engineering graduates \n",
    "    and a lack of well-educated engineers.\n",
    "\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'summary_text': ' America has changed dramatically during recent years . The '\n",
    "                  'number of engineering graduates in the U.S. has declined in '\n",
    "                  'traditional engineering disciplines such as mechanical, civil '\n",
    "                  ', electrical, chemical, and aeronautical engineering . Rapidly '\n",
    "                  'developing economies such as China and India, as well as other '\n",
    "                  'industrial countries in Europe and Asia, continue to encourage '\n",
    "                  'and advance engineering .'}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Translation\n",
    "For translation, you can use a default model if you provide a language pair in the task name (such as \"translation_en_to_fr\"), but the easiest way is to pick the model you want to use on the [Model Hub](https://huggingface.co/models)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import pipeline\n",
    "\n",
    "translator = pipeline(\"translation\", model=\"Helsinki-NLP/opus-mt-fr-en\")\n",
    "translator(\"Ce cours est produit par Hugging Face.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "[{'translation_text': 'This course is produced by Hugging Face.'}]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "Hugging Face. (2022). [The Hugging Face Course](https://huggingface.co/learn)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}