20309780_Mohan_Sameer_ex3_submission.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "20309780_Mohan_Sameer_ex3_submission.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mohansameer1983/DeepLearning/blob/main/20309780_Mohan_Sameer_ex3_submission.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# MMAI 894 - Exercise 3\n",
        "## Transfer learning with DistilBert\n",
        "The goal of this excercise is to build a text classifier using the pretrained DistilBert published by HuggingFace. You will be doing this using the Glue/CoLA dataset (https://nyu-mll.github.io/CoLA/).\n",
        "\n",
        "Submission instructions:\n",
        "\n",
        "- You cannot edit this notebook directly. Save a copy to your drive, and make sure to identify yourself in the title using name and student number\n",
        "- Do not insert new cells before the final one (titled \"Further exploration\") \n",
        "- Verify that your notebook can _restart and run all_. \n",
        "- Unlike previous assignments, please **submit all three formats: .py, .ipynb, and html** (see https://torbjornzetterlund.com/how-to-save-a-google-colab-notebook-as-html/)\n",
        " - The notebook and html submissions should show the completion of your best performing run\n",
        " - Submission files should be named: `studentID_lastname_firstname_ex3.py (or .html, .ipynb)`\n",
        "- The mark will be assessed on the implementation of the functions with #TODO\n",
        "- **Do not change anything outside the functions**  unless in the further exploration section\n",
        "- - As you are encouraged to explore the network configuration, 20% of the mark is based on final accuracy. \n",
        "- Note: You do not have to answer the questions in thie notebook as part of your submission. They are meant to guide you.\n",
        "\n",
        "- You should not need to use any additional libraries other than the ones listed below. You may want to import additional modules from those libraries, however."
      ],
      "metadata": {
        "id": "cQ25iAgAOJ5K"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hZj4c0xTeMwH"
      },
      "source": [
        "# This cell installs and sets up DistilBert import, as well as the dataset, which we will \n",
        "# use tf.datasets to load (https://www.tensorflow.org/datasets/catalog/overview)\n",
        "\n",
        "!pip install -q transformers tfds-nightly\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "import tensorflow.keras as keras\n",
        "import pandas as pd\n",
        "\n",
        "try: # this is only working on the 2nd try in colab :)\n",
        "  from transformers import DistilBertTokenizer, TFDistilBertModel\n",
        "except Exception as err: # so we catch the error and import it again\n",
        "  from transformers import DistilBertTokenizer, TFDistilBertModel\n",
        "\n",
        "import numpy as np\n",
        "import tensorflow.keras as keras\n",
        "from tensorflow.keras.layers import Dense, Input, Dropout\n",
        "from tensorflow.keras import regularizers\n",
        "\n",
        "import tensorflow_datasets as tfds\n",
        "\n",
        "dbert_tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')\n",
        "\n",
        "import re"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Preparation"
      ],
      "metadata": {
        "id": "jHvJJjnCRYF2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def load_data(save_dir=\"./\"):\n",
        "  dataset = tfds.load('glue/cola', shuffle_files=True)\n",
        "  train = tfds.as_dataframe(dataset[\"train\"])\n",
        "  val = tfds.as_dataframe(dataset[\"validation\"])\n",
        "  test = tfds.as_dataframe(dataset[\"test\"])\n",
        "  return train, val, test\n",
        "\n",
        "def prepare_raw_data(df):\n",
        "  raw_data = df.loc[:, [\"idx\", \"sentence\", \"label\"]]\n",
        "  raw_data[\"label\"] = raw_data[\"label\"].astype('category')\n",
        "  return raw_data\n",
        "\n",
        "train, val, test = load_data()\n",
        "train = prepare_raw_data(train)\n",
        "val = prepare_raw_data(val)\n",
        "test = prepare_raw_data(test)\n",
        "\n",
        "num_classes=len(train.label.unique())"
      ],
      "metadata": {
        "id": "q3gYLKfEd0Hb"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before using this data, we need to clean and QA it. Unlike MNIST, this is a text dataset, and we should be more caerful. For example:\n",
        "- Are there any duplicate entries? \n",
        "- What is the range of lengths for the sentences? Should we impose a minimum sentence length?\n",
        "- Are there \"non-sentence\" entries? For example, hashtags or other features we should remove? (luckily, this dataset is quite clean, but that might not always be the case!)\n",
        "\n",
        "NOTE! The sentences are encoded as binary strings. To do text manipulations, you might need to decode them using `s.decode(\"utf-8\")`\n",
        "\n",
        "You may notice that that test set has no labels. This is because Glue is a benchmark dataset, and only gets scored on submissions."
      ],
      "metadata": {
        "id": "IQC3J5brmZdy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def clean_data(df):\n",
        "#   # TODO: What data cleaning/filtering should you consider?\n",
        "#   # DO NOT CHANGE THE INPUTS OR OUTPUTS TO THIS FUNCTION\n",
        "\n",
        "  #df[\"sentence_string\"]=df[\"sentence\"].astype(str)\n",
        "  for ind in df.index:\n",
        "    \n",
        "    string = df[\"sentence\"][ind].decode(\"utf-8\")\n",
        "    punctuations=r'''!()-[]{};:'\"\\,<>./?@#$%^&*_~'''\n",
        "    stop_words=['the', 'a', 'and', 'is', 'be', 'will']\n",
        "    # Cleaning the urls\n",
        "    string = re.sub(r'https?://\\S+|www\\.\\S+', '', string)\n",
        "\n",
        "    # Cleaning the html elements\n",
        "    string = re.sub(r'<.*?>', '', string)\n",
        "\n",
        "    # Removing the punctuations\n",
        "    for x in string.lower(): \n",
        "        if x in punctuations: \n",
        "            string = string.replace(x, \"\") \n",
        "\n",
        "    # Converting the text to lower\n",
        "    string = string.lower()\n",
        "\n",
        "    # Removing stop words\n",
        "    string = ' '.join([word for word in string.split() if word not in stop_words])\n",
        "\n",
        "    # Cleaning the whitespaces\n",
        "    string = re.sub(r'\\s+', ' ', string).strip()\n",
        "\n",
        "    df[\"sentence\"][ind]=string.encode(\"utf-8\")\n",
        "  return df\n",
        "\n",
        "train = clean_data(train)\n",
        "val = clean_data(val)\n",
        "test = clean_data(test)\n"
      ],
      "metadata": {
        "id": "xHNJC2vZmTWl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7f287eaa-7ac9-4b2b-f7bb-0267ed2d0c3b"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:31: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we need to prepare the text for DistilBert. Instead of ingesting raw text, the model uses token IDs to map to internal embedding. Additionally, since the input is fixed size (due to our use of batches), we need to let the model know which tokens to use (i.e. are part of the sentence).\n",
        "\n",
        "Luckily, `dbert_tokenizer` takes care of all that for us - \n",
        "- Preprocessing: https://huggingface.co/transformers/preprocessing.html\n",
        "- Summary of tokenizers (DistilBert uses WordPiece): https://huggingface.co/transformers/tokenizer_summary.html#wordpiece"
      ],
      "metadata": {
        "id": "MAYstlfPQSvd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def extract_text_and_y(df):\n",
        "  text = [x.decode('utf-8') for x in  df.sentence.values]\n",
        "  # for multiclass problems, you can use sklearn.preprocessing.OneHotEncoder, but we only have two classes, so we'll use a single sigmoid output\n",
        "  y = np.array([x for x in df.label.values])\n",
        "  return text, y\n",
        "\n",
        "def encode_text(text):\n",
        "    # TODO: encode text using dbert_tokenizer\n",
        "    # DO NOT CHANGE THE INPUTS OR OUTPUTS TO THIS FUNCTION\n",
        "\n",
        "    # Encode the sentence\n",
        "    encodedText = dbert_tokenizer.batch_encode_plus(\n",
        "        batch_text_or_text_pairs=text,  # the sentence to be encoded\n",
        "        add_special_tokens=True,  # Add [CLS] and [SEP]\n",
        "        max_length = 24,\n",
        "        padding='max_length',\n",
        "        truncation=True\n",
        "        )\n",
        "\n",
        "    # Get the input IDs and attention mask in tensor format\n",
        "    input_ids = tf.convert_to_tensor(encodedText['input_ids'])\n",
        "    attention_mask = tf.convert_to_tensor(encodedText['attention_mask'])\n",
        "    return input_ids, attention_mask\n",
        "\n",
        "# the following prepares the input for running in DistilBert\n",
        "train_text, train_y = extract_text_and_y(clean_data(train))\n",
        "val_text, val_y = extract_text_and_y(clean_data(val))\n",
        "test_text, test_y = extract_text_and_y(clean_data(test))\n",
        "\n",
        "train_input, train_mask = encode_text(train_text)\n",
        "val_input, val_mask = encode_text(val_text)\n",
        "test_input, test_mask = encode_text(test_text)\n",
        "\n",
        "train_model_inputs_and_masks = {\n",
        "    'inputs' : train_input,\n",
        "    'masks' : train_mask\n",
        "}\n",
        "\n",
        "val_model_inputs_and_masks = {\n",
        "    'inputs' : val_input,\n",
        "    'masks' : val_mask\n",
        "}\n",
        "\n",
        "test_model_inputs_and_masks = {\n",
        "    'inputs' : test_input,\n",
        "    'masks' : test_mask\n",
        "}"
      ],
      "metadata": {
        "id": "_RBthPA0fcTA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e8bcfa07-7937-4bd0-e77e-efdb33c6648f"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:31: SettingWithCopyWarning: \n",
            "A value is trying to be set on a copy of a slice from a DataFrame\n",
            "\n",
            "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i2USajN2MWjn"
      },
      "source": [
        "# Modelling\n",
        "\n",
        "## Build and Train Model\n",
        "\n",
        "Resources:\n",
        "- BERT paper https://arxiv.org/pdf/1810.04805.pdf\n",
        "- DistilBert paper: https://arxiv.org/abs/1910.01108\n",
        "- DistilBert Tensorflow Documentation: https://huggingface.co/transformers/model_doc/distilbert.html#tfdistilbertmodel"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GZfFboF85rIe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b41c6ceb-7b56-4902-b93f-2bec3ebab15f"
      },
      "source": [
        "def build_model(base_model, trainable=False, params={}):\n",
        "    # TODO: build the model, with the option to freeze the parameters in distilBERT\n",
        "    # DO NOT CHANGE THE INPUTS OR OUTPUTS TO THIS FUNCTION\n",
        "    # Hint 1: the cls token (token for classification in bert / distilBert) corresponds to the first element in the sequence in DistilBert. Take a look at Figure 2 in BERT paper.\n",
        "    # Hint 2: this guide may be helpful for parameter freezing: https://keras.io/guides/transfer_learning/\n",
        "    # Hint 3: double check that your number of parameters make sense\n",
        "    # Hint 4: carefully consider your final layer activation and loss function\n",
        "\n",
        "    # Refer to https://keras.io/api/layers/core_layers/input/\n",
        "    max_seq_len = 24\n",
        "    inputs = Input(shape = (max_seq_len,), dtype='int64')\n",
        "    masks  = Input(shape = (max_seq_len,), dtype='int64')\n",
        "\n",
        "    base_model.trainable = trainable\n",
        "\n",
        "    dbert_output = base_model(inputs, attention_mask=masks)\n",
        "    # dbert_last_hidden_state gets you the output encoding for each of your tokens.ok \n",
        "    # Each such encoding is a vector with 768 values. The first token fed into the model is [cls]\n",
        "    # which can be used to build a sentence classification network\n",
        "    dbert_last_hidden_state = dbert_output.last_hidden_state\n",
        "\n",
        "    # Any additional layers should go here\n",
        "    # use the 'params' as a dictionary for hyper parameter to facilitate experimentation\n",
        "    dense_layer = Dense(params['dense_layer_1_units'],activation='relu')(dbert_last_hidden_state[:, 0, :])\n",
        "    dropout_layer = Dropout(params['dropout_rate_1'])(dense_layer)\n",
        "    dense_layer = Dense(params['dense_layer_2_units'],activation='relu')(dropout_layer)\n",
        "    probs = Dense(1, activation='sigmoid',kernel_regularizer=regularizers.l2(params['regularizer_l2_rate']), name=\"output\")(dense_layer)\n",
        "\n",
        "    model = keras.Model(inputs=[inputs, masks], outputs=probs)\n",
        "    model.summary()\n",
        "    return model\n",
        "\n",
        "dbert_model = TFDistilBertModel.from_pretrained('distilbert-base-uncased')\n",
        "params={# add parameters here\n",
        "        \"dense_layer_1_units\": 128,        \n",
        "        \"dropout_rate_1\": 0.1,\n",
        "\t      \"dense_layer_2_units\": 64,\n",
        "        \"regularizer_l2_rate\":0.01\n",
        "        }\n",
        "\n",
        "model = build_model(dbert_model, params=params)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_projector', 'vocab_layer_norm', 'vocab_transform', 'activation_13']\n",
            "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
            "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " input_1 (InputLayer)           [(None, 24)]         0           []                               \n",
            "                                                                                                  \n",
            " input_2 (InputLayer)           [(None, 24)]         0           []                               \n",
            "                                                                                                  \n",
            " tf_distil_bert_model (TFDistil  TFBaseModelOutput(l  66362880   ['input_1[0][0]',                \n",
            " BertModel)                     ast_hidden_state=(N               'input_2[0][0]']                \n",
            "                                one, 24, 768),                                                    \n",
            "                                 hidden_states=None                                               \n",
            "                                , attentions=None)                                                \n",
            "                                                                                                  \n",
            " tf.__operators__.getitem (Slic  (None, 768)         0           ['tf_distil_bert_model[0][0]']   \n",
            " ingOpLambda)                                                                                     \n",
            "                                                                                                  \n",
            " dense (Dense)                  (None, 128)          98432       ['tf.__operators__.getitem[0][0]'\n",
            "                                                                 ]                                \n",
            "                                                                                                  \n",
            " dropout_19 (Dropout)           (None, 128)          0           ['dense[0][0]']                  \n",
            "                                                                                                  \n",
            " dense_1 (Dense)                (None, 64)           8256        ['dropout_19[0][0]']             \n",
            "                                                                                                  \n",
            " output (Dense)                 (None, 1)            65          ['dense_1[0][0]']                \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 66,469,633\n",
            "Trainable params: 106,753\n",
            "Non-trainable params: 66,362,880\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def compile_model(model):\n",
        "    # TODO: compile the model, include relevant auc metrics when training\n",
        "    # DO NOT CHANGE THE INPUTS OR OUTPUTS TO THIS FUNCTION\n",
        "\n",
        "    METRICS = [\n",
        "          tf.keras.metrics.BinaryAccuracy(name='accuracy'),\n",
        "          tf.keras.metrics.Precision(name='precision'),\n",
        "          tf.keras.metrics.Recall(name='recall')\n",
        "    ]\n",
        "    optimizer = keras.optimizers.Adam(learning_rate=1e-5)\n",
        "    loss = keras.losses.binary_crossentropy\n",
        "    model.compile(optimizer=optimizer,\n",
        "              loss=loss,\n",
        "              metrics=METRICS)\n",
        "    return model\n",
        "\n",
        "model = compile_model(model)"
      ],
      "metadata": {
        "id": "Z3EyvQbSzu5m"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "def train_model(model, model_inputs_and_masks_train, model_inputs_and_masks_val,\n",
        "    y_train, y_val, batch_size, num_epochs):\n",
        "    # TODO: train the model\n",
        "    # DO NOT CHANGE THE INPUTS OR OUTPUTS TO THIS FUNCTION\n",
        "\n",
        "    # Training Data - Tensor Conversion\n",
        "    input_ids_train = model_inputs_and_masks_train['inputs']\n",
        "    attention_masks_train = model_inputs_and_masks_train['masks']\n",
        "    labels_train = tf.convert_to_tensor(y_train[:])\n",
        "\n",
        "    # Validation data - Tensor Conversion\n",
        "    input_ids_val = model_inputs_and_masks_val['inputs']\n",
        "    attention_masks_val = model_inputs_and_masks_val['masks']\n",
        "    labels_val = tf.convert_to_tensor(y_val[:])\n",
        "\n",
        "    # Preparing Training Dataset\n",
        "    dataset_train = tf.data.Dataset.from_tensors(( (input_ids_train, attention_masks_train), labels_train ))\n",
        "\n",
        "    history = model.fit(dataset_train,\n",
        "          epochs=num_epochs,\n",
        "          validation_data=([input_ids_val,attention_masks_val],labels_val),\n",
        "          batch_size=batch_size, verbose=1)\n",
        "    \n",
        "    return model, history\n",
        "\n",
        "model, history = train_model(model, train_model_inputs_and_masks, val_model_inputs_and_masks, train_y, val_y, batch_size=128, num_epochs=5)"
      ],
      "metadata": {
        "id": "Nz8kT3f8zykl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9b117764-1eda-45d1-f992-1a535a1b870d"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/5\n",
            "1/1 [==============================] - 383s 383s/step - loss: 0.6526 - accuracy: 0.6966 - precision: 0.7049 - recall: 0.9792 - val_loss: 0.6507 - val_accuracy: 0.6913 - val_precision: 0.6913 - val_recall: 1.0000\n",
            "Epoch 2/5\n",
            "1/1 [==============================] - 378s 378s/step - loss: 0.6511 - accuracy: 0.6980 - precision: 0.7052 - recall: 0.9816 - val_loss: 0.6499 - val_accuracy: 0.6913 - val_precision: 0.6913 - val_recall: 1.0000\n",
            "Epoch 3/5\n",
            "1/1 [==============================] - 374s 374s/step - loss: 0.6513 - accuracy: 0.7017 - precision: 0.7065 - recall: 0.9862 - val_loss: 0.6493 - val_accuracy: 0.6913 - val_precision: 0.6913 - val_recall: 1.0000\n",
            "Epoch 4/5\n",
            "1/1 [==============================] - 379s 379s/step - loss: 0.6499 - accuracy: 0.6985 - precision: 0.7050 - recall: 0.9836 - val_loss: 0.6486 - val_accuracy: 0.6913 - val_precision: 0.6913 - val_recall: 1.0000\n",
            "Epoch 5/5\n",
            "1/1 [==============================] - 380s 380s/step - loss: 0.6490 - accuracy: 0.6992 - precision: 0.7044 - recall: 0.9872 - val_loss: 0.6479 - val_accuracy: 0.6913 - val_precision: 0.6913 - val_recall: 1.0000\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xid8Xf2naNZW"
      },
      "source": [
        "# Further exploration (REMOVE ALL CODE AFTER THIS CELL BEFORE SUBMISSION)\n",
        "Any code after this is not evaluated, and must be removed before submission.\n",
        "Leaving code below will result in losing marks."
      ]
    }
  ]
}