National Language processing/Next_Word_Prediction.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Next Word Prediction.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyORSZkbG4V0B4X6TangNZUp",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hussain0048/Projects-/blob/master/Next_Word_Prediction.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZaI6Tg0J8T6i"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "olKf7idT8tNp"
      },
      "source": [
        "# **1-Introduction**\n",
        "\n",
        "Most of the keyboards in smartphones give next word prediction features; google also uses next word prediction based on our browsing history. So a preloaded data is also stored in the keyboard function of our smartphones to predict the next word correctly. In this article, I will train a Deep Learning model for next word prediction using Python. I will use the Tensorflow and Keras library in Python for next word prediction model[1].\n",
        "For making a Next Word Prediction model, I will train a Recurrent Neural Network (RNN). So let’s start with this task now without wasting any time.[1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ekrccN86BJx3"
      },
      "source": [
        "#**2- Import Library**\n",
        "To start with our next word prediction model, let’s import some all the libraries we need for this task:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ec4zrgo4BXtw"
      },
      "source": [
        "import numpy as np\n",
        "from nltk.tokenize import RegexpTokenizer\n",
        "from keras.models import Sequential, load_model\n",
        "from keras.layers import LSTM\n",
        "from keras.layers.core import Dense, Activation\n",
        "from keras.optimizers import RMSprop\n",
        "import matplotlib.pyplot as plt\n",
        "import pickle\n",
        "import heapq"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YI1pDtID9IYQ"
      },
      "source": [
        "# **3- Dataset**\n",
        "\n",
        "As I told earlier, Google uses our browsing history to make next word predictions, smartphones, and all the keyboards that are trained to predict the next word are trained using some data. So I will also use a dataset. You can download the dataset from [here](https://drive.google.com/file/d/1GeUzNVqiixXHnTl8oNiQ2W3CynX_lsu2/view)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JglqMAQTAFwy",
        "outputId": "a3636c23-d586-4428-c5ec-a25e171f9aa9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mCwasHeG-Uhy",
        "outputId": "179f86dc-1ad3-4925-f010-d61b9281bc41",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "path = \"/content/drive/My Drive/Datasets/Next Word Prediction/1661-0.txt\"\n",
        "text = open(path).read().lower()\n",
        "print('corpus length:', len(text))"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "corpus length: 581888\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mo6Lbqq9By3B"
      },
      "source": [
        "Now I will split the dataset into each word in order but without the presence of some special characters.[1]\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0DqYEwBtB97w"
      },
      "source": [
        "tokenizer = RegexpTokenizer(r'w+')\n",
        "words = tokenizer.tokenize(text)"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zVB6zOjkCOLn"
      },
      "source": [
        "Now the next process will be performing the feature engineering in our data. For this purpose, we will require a dictionary with each word in the data within the list of unique words as the key, and it’s significant portions as value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zO077DPMCZcY"
      },
      "source": [
        "unique_words = np.unique(words)\n",
        "unique_word_index = dict((c, i) for i, c in enumerate(unique_words))"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8w-AOzqsC0t4"
      },
      "source": [
        "# **4-Feature Engineering**\n",
        "\n",
        "Feature Engineering means taking whatever information we have about our problem and turning it into numbers that we can use to build our feature matrix. If you want a detailed tutorial of feature engineering, you can learn it from here.[1]\n",
        "\n",
        "Here I will define a Word length which will represent the number of previous words that will determine our next word. I will define prev words to keep five previous words and their corresponding next words in the list of next words[1]\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XLjbXJWeDeBn",
        "outputId": "66e28e2a-567b-47d3-ba82-6f3a482f71ac",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "WORD_LENGTH = 5\n",
        "prev_words = []\n",
        "next_words = []\n",
        "for i in range(len(words) - WORD_LENGTH):\n",
        "    prev_words.append(words[i:i + WORD_LENGTH])\n",
        "    next_words.append(words[i + WORD_LENGTH])\n",
        "print(prev_words[0])\n",
        "print(next_words[0])"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['w', 'w', 'w', 'w', 'w']\n",
            "www\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gM5ygqp0dyWn"
      },
      "source": [
        "Now I will create two numpy arrays x for storing the features and y for storing its corresponding label. I will iterate x and y if the word is available so that the corresponding position becomes 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ox38mcsud0TE"
      },
      "source": [
        "X = np.zeros((len(prev_words), WORD_LENGTH, len(unique_words)), dtype=bool)\n",
        "Y = np.zeros((len(next_words), len(unique_words)), dtype=bool)\n",
        "for i, each_words in enumerate(prev_words):\n",
        "    for j, each_word in enumerate(each_words):\n",
        "        X[i, j, unique_word_index[each_word]] = 1\n",
        "    Y[i, unique_word_index[next_words[i]]] = 1"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nmt7_Yt6d5Jl"
      },
      "source": [
        "Now before moving forward, have a look at a single sequence of words:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3NspIi8Ld9ld",
        "outputId": "6b1e963a-d975-43fb-cb8b-e3f60d3f7690",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(X[0][0])\n"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ True False]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "APxYmNJZeHO1"
      },
      "source": [
        "#**5-Building the Recurrent Neural network**\n",
        "As I stated earlier, I will use the Recurrent Neural networks for next word prediction model. Here I will use the LSTM model, which is a very powerful RNN.[1]\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pKgP1X4veqjM"
      },
      "source": [
        "model = Sequential()\n",
        "model.add(LSTM(128, input_shape=(WORD_LENGTH, len(unique_words))))\n",
        "model.add(Dense(len(unique_words)))\n",
        "model.add(Activation('softmax'))"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J1o_YDbSeuuU"
      },
      "source": [
        "#**6-Training the Next Word Prediction Model**\n",
        "I will be training the next word prediction model with 20 epochs[1]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-YaeRP-be6OY"
      },
      "source": [
        "optimizer = RMSprop(lr=0.01)\n",
        "model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
        "history = model.fit(X, Y, validation_split=0.05, batch_size=128, epochs=2, shuffle=True).history"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rfQGJjWlfK2m"
      },
      "source": [
        "Now we have successfully trained our model, before moving forward to evaluating our model, it will be better to save this model for our future use[1]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SmW_O_cSfNbV"
      },
      "source": [
        "model.save('keras_next_word_model.h5')\n",
        "pickle.dump(history, open(\"history.p\", \"wb\"))\n",
        "model = load_model('keras_next_word_model.h5')\n",
        "history = pickle.load(open(\"history.p\", \"rb\"))"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N6woIuxnf9PE",
        "outputId": "93b264a2-d51f-46a1-863b-e4098b4808ff",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "print(history)"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'loss': [0.009844760410487652, 0.002820393769070506], 'accuracy': [0.9999086856842041, 0.9999086856842041], 'val_loss': [0.2027292400598526, 0.23149938881397247], 'val_accuracy': [0.984402060508728, 0.984402060508728]}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lKf1l5bVfXk3"
      },
      "source": [
        "#**7-Evaluating the Next Word Prediction Model**\n",
        "\n",
        "Now let’s have a quick look at how our model is going to behave based on its accuracy and loss changes while training:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jeVQg0iFfie9"
      },
      "source": [
        "plt.plot(history['accuracy'])\n",
        "plt.plot(history['val_accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0XwYisZsgV6G"
      },
      "source": [
        "plt.plot(history['loss'])\n",
        "plt.plot(history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'test'], loc='upper left')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXAZaFzaghJz"
      },
      "source": [
        "#**8-Testing Next Word Prediction Model**\n",
        "\n",
        "Now let’s build a python program to predict the next word using our trained model. For this, I will define some essential functions that will be used in the process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ICqkTlN8gs-g"
      },
      "source": [
        "def prepare_input(text):\n",
        "    x = np.zeros((1, SEQUENCE_LENGTH, len(chars)))\n",
        "    for t, char in enumerate(text):\n",
        "        x[0, t, char_indices[char]] = 1.\n",
        "        \n",
        "    return x"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tCv0KCZWg04X"
      },
      "source": [
        "Now before moving forward, let’s test the function, make sure you use a lower() function while giving input :\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sWoj4Kp5g2id",
        "outputId": "7193d76c-4a1b-46db-9622-d39cccbd82c6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        }
      },
      "source": [
        "prepare_input(\"This is an example of input for our LSTM\".lower())\n"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-37-d4dddb46ad88>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprepare_input\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"This is an example of input for our LSTM\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-35-d3415c8d0738>\u001b[0m in \u001b[0;36mprepare_input\u001b[0;34m(text)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mprepare_input\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mSEQUENCE_LENGTH\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchars\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchar\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m         \u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mchar_indices\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mchar\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mNameError\u001b[0m: name 'SEQUENCE_LENGTH' is not defined"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UaIX4lIl8a0Y"
      },
      "source": [
        "# **References**\n",
        "[Next Word Prediction Model](https://thecleverprogrammer.com/2020/07/20/next-word-prediction-model/)"
      ]
    }
  ]
}