Natural_Language_Processing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "Natural Language Processing.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "38NLL-pgTHcz",
        "WWiJOALkTHc1"
      ],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/melisnv/Python-Data-Project/blob/main/Natural_Language_Processing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ir3jeGkTHcj"
      },
      "source": [
        "___\n",
        "\n",
        "\n",
        "# Text Generation with Neural Networks\n",
        "\n",
        "http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
        "\n",
        "#### The aim of this project is that the network can learn play writing structure and spacing, just from a character level.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3XvcAy_9THcn"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "%tensorflow_version 2.x\n",
        "import tensorflow as tf"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BbUApNuTHcp"
      },
      "source": [
        "## Step 1: The Data\n",
        "\n",
        "Grabbed any free text from here: https://www.gutenberg.org/\n",
        "\n",
        "Choosed all of shakespeare's works, mainly for two reasons:\n",
        "\n",
        "1. Its a large corpus of text, its usually recommended to have at least a source of 1 million characters total to get realistic text generation.\n",
        "\n",
        "2. It has a very distinctive style. Since the text data uses old style english and is formatted in the style of a stage play, it will be very obvious to us if the model is able to reproduce similar results.\n",
        "\n",
        "The aim of this mini learning project is to be able to see the recurrent neural network actually duplicate that style just on a character by character basis."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qiKrZrkUTHcp"
      },
      "source": [
        "path_to_file = \"shakespeare.txt\""
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jc0owq72THcq"
      },
      "source": [
        "text = open(path_to_file, 'r').read()"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        },
        "id": "9RmoJbZPTHcq",
        "outputId": "82d85a77-3f44-4828-880f-6a1b10dc00f8"
      },
      "source": [
        "text[:500]"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"\\n                     1\\n  From fairest creatures we desire increase,\\n  That thereby beauty's rose might never die,\\n  But as the riper should by time decease,\\n  His tender heir might bear his memory:\\n  But thou contracted to thine own bright eyes,\\n  Feed'st thy light's flame with self-substantial fuel,\\n  Making a famine where abundance lies,\\n  Thy self thy foe, to thy sweet self too cruel:\\n  Thou that art now the world's fresh ornament,\\n  And only herald to the gaudy spring,\\n  Within thine own bu\""
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o8buflx2THcs",
        "outputId": "595459ef-3905-4bc4-e46d-8b1df9e2fe2d"
      },
      "source": [
        "print(text[:500])"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "                     1\n",
            "  From fairest creatures we desire increase,\n",
            "  That thereby beauty's rose might never die,\n",
            "  But as the riper should by time decease,\n",
            "  His tender heir might bear his memory:\n",
            "  But thou contracted to thine own bright eyes,\n",
            "  Feed'st thy light's flame with self-substantial fuel,\n",
            "  Making a famine where abundance lies,\n",
            "  Thy self thy foe, to thy sweet self too cruel:\n",
            "  Thou that art now the world's fresh ornament,\n",
            "  And only herald to the gaudy spring,\n",
            "  Within thine own bu\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UuyiSWdZTHcs",
        "outputId": "ad31ffe5-8f56-4341-de1a-a86b8cb5b267"
      },
      "source": [
        "print(text[140500:141500])"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "reward.\n",
            "  HELENA. Inspired merit so by breath is barr'd.\n",
            "    It is not so with Him that all things knows,\n",
            "    As 'tis with us that square our guess by shows;\n",
            "    But most it is presumption in us when\n",
            "    The help of heaven we count the act of men.\n",
            "    Dear sir, to my endeavours give consent;\n",
            "    Of heaven, not me, make an experiment.\n",
            "    I am not an impostor, that proclaim  \n",
            "    Myself against the level of mine aim;\n",
            "    But know I think, and think I know most sure,\n",
            "    My art is not past power nor you past cure.\n",
            "  KING. Art thou so confident? Within what space\n",
            "    Hop'st thou my cure?\n",
            "  HELENA. The greatest Grace lending grace.\n",
            "    Ere twice the horses of the sun shall bring\n",
            "    Their fiery torcher his diurnal ring,\n",
            "    Ere twice in murk and occidental damp\n",
            "    Moist Hesperus hath quench'd his sleepy lamp,\n",
            "    Or four and twenty times the pilot's glass\n",
            "    Hath told the thievish minutes how they pass,\n",
            "    What is infirm from your sound parts shall fly,\n",
            "    Health shall live free, and s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-hoTLyn9THct"
      },
      "source": [
        "### Understanding unique characters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BOBd5YeNTHcu",
        "outputId": "2badd633-f2ab-46d2-a00a-0002f2f1dee2"
      },
      "source": [
        "vocab = sorted(set(text))\n",
        "vocab"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['\\n',\n",
              " ' ',\n",
              " '!',\n",
              " '\"',\n",
              " '&',\n",
              " \"'\",\n",
              " '(',\n",
              " ')',\n",
              " ',',\n",
              " '-',\n",
              " '.',\n",
              " '0',\n",
              " '1',\n",
              " '2',\n",
              " '3',\n",
              " '4',\n",
              " '5',\n",
              " '6',\n",
              " '7',\n",
              " '8',\n",
              " '9',\n",
              " ':',\n",
              " ';',\n",
              " '<',\n",
              " '>',\n",
              " '?',\n",
              " 'A',\n",
              " 'B',\n",
              " 'C',\n",
              " 'D',\n",
              " 'E',\n",
              " 'F',\n",
              " 'G',\n",
              " 'H',\n",
              " 'I',\n",
              " 'J',\n",
              " 'K',\n",
              " 'L',\n",
              " 'M',\n",
              " 'N',\n",
              " 'O',\n",
              " 'P',\n",
              " 'Q',\n",
              " 'R',\n",
              " 'S',\n",
              " 'T',\n",
              " 'U',\n",
              " 'V',\n",
              " 'W',\n",
              " 'X',\n",
              " 'Y',\n",
              " 'Z',\n",
              " '[',\n",
              " ']',\n",
              " '_',\n",
              " '`',\n",
              " 'a',\n",
              " 'b',\n",
              " 'c',\n",
              " 'd',\n",
              " 'e',\n",
              " 'f',\n",
              " 'g',\n",
              " 'h',\n",
              " 'i',\n",
              " 'j',\n",
              " 'k',\n",
              " 'l',\n",
              " 'm',\n",
              " 'n',\n",
              " 'o',\n",
              " 'p',\n",
              " 'q',\n",
              " 'r',\n",
              " 's',\n",
              " 't',\n",
              " 'u',\n",
              " 'v',\n",
              " 'w',\n",
              " 'x',\n",
              " 'y',\n",
              " 'z',\n",
              " '|',\n",
              " '}']"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8KpdA9d_THcv",
        "outputId": "05ab996d-40aa-429b-ef81-39a00d5643a8"
      },
      "source": [
        "len(vocab)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "84"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "shofHMx0THcv"
      },
      "source": [
        "## Step 2: Text Processing\n",
        "\n",
        "### Text Vectorization\n",
        "\n",
        "A neural network can't take in the raw string data, it needed to be assigned numbers to each character. Creating two dictionaries that can go from numeric index to character and character to numeric index."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eFw-HDJcTHcw",
        "outputId": "2b584d78-5ff0-4d92-be0a-5036b41f0427"
      },
      "source": [
        "char_to_index = {char:ind for ind,char in enumerate(vocab)}\n",
        "char_to_index"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'\\n': 0,\n",
              " ' ': 1,\n",
              " '!': 2,\n",
              " '\"': 3,\n",
              " '&': 4,\n",
              " \"'\": 5,\n",
              " '(': 6,\n",
              " ')': 7,\n",
              " ',': 8,\n",
              " '-': 9,\n",
              " '.': 10,\n",
              " '0': 11,\n",
              " '1': 12,\n",
              " '2': 13,\n",
              " '3': 14,\n",
              " '4': 15,\n",
              " '5': 16,\n",
              " '6': 17,\n",
              " '7': 18,\n",
              " '8': 19,\n",
              " '9': 20,\n",
              " ':': 21,\n",
              " ';': 22,\n",
              " '<': 23,\n",
              " '>': 24,\n",
              " '?': 25,\n",
              " 'A': 26,\n",
              " 'B': 27,\n",
              " 'C': 28,\n",
              " 'D': 29,\n",
              " 'E': 30,\n",
              " 'F': 31,\n",
              " 'G': 32,\n",
              " 'H': 33,\n",
              " 'I': 34,\n",
              " 'J': 35,\n",
              " 'K': 36,\n",
              " 'L': 37,\n",
              " 'M': 38,\n",
              " 'N': 39,\n",
              " 'O': 40,\n",
              " 'P': 41,\n",
              " 'Q': 42,\n",
              " 'R': 43,\n",
              " 'S': 44,\n",
              " 'T': 45,\n",
              " 'U': 46,\n",
              " 'V': 47,\n",
              " 'W': 48,\n",
              " 'X': 49,\n",
              " 'Y': 50,\n",
              " 'Z': 51,\n",
              " '[': 52,\n",
              " ']': 53,\n",
              " '_': 54,\n",
              " '`': 55,\n",
              " 'a': 56,\n",
              " 'b': 57,\n",
              " 'c': 58,\n",
              " 'd': 59,\n",
              " 'e': 60,\n",
              " 'f': 61,\n",
              " 'g': 62,\n",
              " 'h': 63,\n",
              " 'i': 64,\n",
              " 'j': 65,\n",
              " 'k': 66,\n",
              " 'l': 67,\n",
              " 'm': 68,\n",
              " 'n': 69,\n",
              " 'o': 70,\n",
              " 'p': 71,\n",
              " 'q': 72,\n",
              " 'r': 73,\n",
              " 's': 74,\n",
              " 't': 75,\n",
              " 'u': 76,\n",
              " 'v': 77,\n",
              " 'w': 78,\n",
              " 'x': 79,\n",
              " 'y': 80,\n",
              " 'z': 81,\n",
              " '|': 82,\n",
              " '}': 83}"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zjd8GLcqTHcw",
        "outputId": "1e477acd-c28e-4079-b81e-e9b48d83bc02"
      },
      "source": [
        "char_to_index['H']"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "33"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "9W89zCW0THcx",
        "outputId": "825a6ac0-d2b5-4b46-d12f-86e48c12f211"
      },
      "source": [
        "index_to_char = np.array(vocab)\n",
        "index_to_char[33]"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'H'"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ujYjbiDsTHcx",
        "outputId": "03157854-7669-44da-e641-986ca6016c27"
      },
      "source": [
        "encoded_text = np.array([char_to_index[c] for c in text])\n",
        "encoded_text"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 0,  1,  1, ..., 30, 39, 29])"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uyYcsM4bTHcx",
        "outputId": "c0255126-f301-41dc-c73d-feedea0fc8dd"
      },
      "source": [
        "encoded_text.shape"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5445609,)"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "msgZgFbxTHcy",
        "outputId": "2e45313e-54f6-4fd5-999e-0b52b427f744"
      },
      "source": [
        "sample = text[:500]\n",
        "sample"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"\\n                     1\\n  From fairest creatures we desire increase,\\n  That thereby beauty's rose might never die,\\n  But as the riper should by time decease,\\n  His tender heir might bear his memory:\\n  But thou contracted to thine own bright eyes,\\n  Feed'st thy light's flame with self-substantial fuel,\\n  Making a famine where abundance lies,\\n  Thy self thy foe, to thy sweet self too cruel:\\n  Thou that art now the world's fresh ornament,\\n  And only herald to the gaudy spring,\\n  Within thine own bu\""
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c1li-24TTHcy",
        "outputId": "6e671bac-310b-4d5a-e0f4-c00dfc219e5c"
      },
      "source": [
        "encoded_text[:500]"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 0,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
              "        1,  1,  1,  1,  1, 12,  0,  1,  1, 31, 73, 70, 68,  1, 61, 56, 64,\n",
              "       73, 60, 74, 75,  1, 58, 73, 60, 56, 75, 76, 73, 60, 74,  1, 78, 60,\n",
              "        1, 59, 60, 74, 64, 73, 60,  1, 64, 69, 58, 73, 60, 56, 74, 60,  8,\n",
              "        0,  1,  1, 45, 63, 56, 75,  1, 75, 63, 60, 73, 60, 57, 80,  1, 57,\n",
              "       60, 56, 76, 75, 80,  5, 74,  1, 73, 70, 74, 60,  1, 68, 64, 62, 63,\n",
              "       75,  1, 69, 60, 77, 60, 73,  1, 59, 64, 60,  8,  0,  1,  1, 27, 76,\n",
              "       75,  1, 56, 74,  1, 75, 63, 60,  1, 73, 64, 71, 60, 73,  1, 74, 63,\n",
              "       70, 76, 67, 59,  1, 57, 80,  1, 75, 64, 68, 60,  1, 59, 60, 58, 60,\n",
              "       56, 74, 60,  8,  0,  1,  1, 33, 64, 74,  1, 75, 60, 69, 59, 60, 73,\n",
              "        1, 63, 60, 64, 73,  1, 68, 64, 62, 63, 75,  1, 57, 60, 56, 73,  1,\n",
              "       63, 64, 74,  1, 68, 60, 68, 70, 73, 80, 21,  0,  1,  1, 27, 76, 75,\n",
              "        1, 75, 63, 70, 76,  1, 58, 70, 69, 75, 73, 56, 58, 75, 60, 59,  1,\n",
              "       75, 70,  1, 75, 63, 64, 69, 60,  1, 70, 78, 69,  1, 57, 73, 64, 62,\n",
              "       63, 75,  1, 60, 80, 60, 74,  8,  0,  1,  1, 31, 60, 60, 59,  5, 74,\n",
              "       75,  1, 75, 63, 80,  1, 67, 64, 62, 63, 75,  5, 74,  1, 61, 67, 56,\n",
              "       68, 60,  1, 78, 64, 75, 63,  1, 74, 60, 67, 61,  9, 74, 76, 57, 74,\n",
              "       75, 56, 69, 75, 64, 56, 67,  1, 61, 76, 60, 67,  8,  0,  1,  1, 38,\n",
              "       56, 66, 64, 69, 62,  1, 56,  1, 61, 56, 68, 64, 69, 60,  1, 78, 63,\n",
              "       60, 73, 60,  1, 56, 57, 76, 69, 59, 56, 69, 58, 60,  1, 67, 64, 60,\n",
              "       74,  8,  0,  1,  1, 45, 63, 80,  1, 74, 60, 67, 61,  1, 75, 63, 80,\n",
              "        1, 61, 70, 60,  8,  1, 75, 70,  1, 75, 63, 80,  1, 74, 78, 60, 60,\n",
              "       75,  1, 74, 60, 67, 61,  1, 75, 70, 70,  1, 58, 73, 76, 60, 67, 21,\n",
              "        0,  1,  1, 45, 63, 70, 76,  1, 75, 63, 56, 75,  1, 56, 73, 75,  1,\n",
              "       69, 70, 78,  1, 75, 63, 60,  1, 78, 70, 73, 67, 59,  5, 74,  1, 61,\n",
              "       73, 60, 74, 63,  1, 70, 73, 69, 56, 68, 60, 69, 75,  8,  0,  1,  1,\n",
              "       26, 69, 59,  1, 70, 69, 67, 80,  1, 63, 60, 73, 56, 67, 59,  1, 75,\n",
              "       70,  1, 75, 63, 60,  1, 62, 56, 76, 59, 80,  1, 74, 71, 73, 64, 69,\n",
              "       62,  8,  0,  1,  1, 48, 64, 75, 63, 64, 69,  1, 75, 63, 64, 69, 60,\n",
              "        1, 70, 78, 69,  1, 57, 76])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F6VeKCpdTHcy"
      },
      "source": [
        "-----\n",
        "## Step 3: Creating Batches\n",
        "\n",
        "Overall the aim to achieve is to have the model predict the next highest probability character given a historical sequence of characters. Its up to the user to choose how long that historic sequence. Too short a sequence and the result don't have enough information (e.g. given the letter \"a\" , what is the next character) , too long a sequence and the training will take too long and most likely overfit to sequence characters that are irrelevant to characters farther out. While there is no correct sequence length choice, it should be considered that the text itself, how long normal phrases are in it, and a reasonable idea of what characters/words are relevant to each other."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HfXPey2lTHcy"
      },
      "source": [
        "line = \"From fairest creatures we desire increase\""
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M1YK0wj4THcz",
        "outputId": "3c6f31a5-29ae-435f-c326-82ca7d7bf8f6"
      },
      "source": [
        "len(line)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "41"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EQo4ZxVYTHcz"
      },
      "source": [
        "lines = '''\n",
        "From fairest creatures we desire increase,\n",
        "  That thereby beauty's rose might never die,\n",
        "  But as the riper should by time decease,\n",
        "'''"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MU2302YFTHcz",
        "outputId": "a8c6e88d-c022-4e41-9736-054768fdaf38"
      },
      "source": [
        "len(lines)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "133"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "38NLL-pgTHcz"
      },
      "source": [
        "### Training Sequences\n",
        "\n",
        "The actual text data will be the text sequence shifted one character forward. For example:\n",
        "\n",
        "Sequence In: \"Hello my nam\"\n",
        "Sequence Out: \"ello my name\"\n",
        "\n",
        "\n",
        "Can use the `tf.data.Dataset.from_tensor_slices` function to convert a text vector into a stream of character indices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "06dFYujfTHcz"
      },
      "source": [
        "sequence_length = 120"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f4QnE_XcTHc0",
        "outputId": "f31b8aa4-734d-4ce7-f376-77609ecf5d37"
      },
      "source": [
        "total_num_seq = len(text) // (sequence_length)\n",
        "total_num_seq"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "45380"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YaYAkQKUTHc0",
        "outputId": "4c452d82-88c3-4932-a847-a5bd862cc003"
      },
      "source": [
        "# Create training sequences\n",
        "char_dataset = tf.data.Dataset.from_tensor_slices(encoded_text)\n",
        "\n",
        "for item in char_dataset.take(500):\n",
        "    print(item.numpy())"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "1\n",
            "12\n",
            "0\n",
            "1\n",
            "1\n",
            "31\n",
            "73\n",
            "70\n",
            "68\n",
            "1\n",
            "61\n",
            "56\n",
            "64\n",
            "73\n",
            "60\n",
            "74\n",
            "75\n",
            "1\n",
            "58\n",
            "73\n",
            "60\n",
            "56\n",
            "75\n",
            "76\n",
            "73\n",
            "60\n",
            "74\n",
            "1\n",
            "78\n",
            "60\n",
            "1\n",
            "59\n",
            "60\n",
            "74\n",
            "64\n",
            "73\n",
            "60\n",
            "1\n",
            "64\n",
            "69\n",
            "58\n",
            "73\n",
            "60\n",
            "56\n",
            "74\n",
            "60\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "45\n",
            "63\n",
            "56\n",
            "75\n",
            "1\n",
            "75\n",
            "63\n",
            "60\n",
            "73\n",
            "60\n",
            "57\n",
            "80\n",
            "1\n",
            "57\n",
            "60\n",
            "56\n",
            "76\n",
            "75\n",
            "80\n",
            "5\n",
            "74\n",
            "1\n",
            "73\n",
            "70\n",
            "74\n",
            "60\n",
            "1\n",
            "68\n",
            "64\n",
            "62\n",
            "63\n",
            "75\n",
            "1\n",
            "69\n",
            "60\n",
            "77\n",
            "60\n",
            "73\n",
            "1\n",
            "59\n",
            "64\n",
            "60\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "27\n",
            "76\n",
            "75\n",
            "1\n",
            "56\n",
            "74\n",
            "1\n",
            "75\n",
            "63\n",
            "60\n",
            "1\n",
            "73\n",
            "64\n",
            "71\n",
            "60\n",
            "73\n",
            "1\n",
            "74\n",
            "63\n",
            "70\n",
            "76\n",
            "67\n",
            "59\n",
            "1\n",
            "57\n",
            "80\n",
            "1\n",
            "75\n",
            "64\n",
            "68\n",
            "60\n",
            "1\n",
            "59\n",
            "60\n",
            "58\n",
            "60\n",
            "56\n",
            "74\n",
            "60\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "33\n",
            "64\n",
            "74\n",
            "1\n",
            "75\n",
            "60\n",
            "69\n",
            "59\n",
            "60\n",
            "73\n",
            "1\n",
            "63\n",
            "60\n",
            "64\n",
            "73\n",
            "1\n",
            "68\n",
            "64\n",
            "62\n",
            "63\n",
            "75\n",
            "1\n",
            "57\n",
            "60\n",
            "56\n",
            "73\n",
            "1\n",
            "63\n",
            "64\n",
            "74\n",
            "1\n",
            "68\n",
            "60\n",
            "68\n",
            "70\n",
            "73\n",
            "80\n",
            "21\n",
            "0\n",
            "1\n",
            "1\n",
            "27\n",
            "76\n",
            "75\n",
            "1\n",
            "75\n",
            "63\n",
            "70\n",
            "76\n",
            "1\n",
            "58\n",
            "70\n",
            "69\n",
            "75\n",
            "73\n",
            "56\n",
            "58\n",
            "75\n",
            "60\n",
            "59\n",
            "1\n",
            "75\n",
            "70\n",
            "1\n",
            "75\n",
            "63\n",
            "64\n",
            "69\n",
            "60\n",
            "1\n",
            "70\n",
            "78\n",
            "69\n",
            "1\n",
            "57\n",
            "73\n",
            "64\n",
            "62\n",
            "63\n",
            "75\n",
            "1\n",
            "60\n",
            "80\n",
            "60\n",
            "74\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "31\n",
            "60\n",
            "60\n",
            "59\n",
            "5\n",
            "74\n",
            "75\n",
            "1\n",
            "75\n",
            "63\n",
            "80\n",
            "1\n",
            "67\n",
            "64\n",
            "62\n",
            "63\n",
            "75\n",
            "5\n",
            "74\n",
            "1\n",
            "61\n",
            "67\n",
            "56\n",
            "68\n",
            "60\n",
            "1\n",
            "78\n",
            "64\n",
            "75\n",
            "63\n",
            "1\n",
            "74\n",
            "60\n",
            "67\n",
            "61\n",
            "9\n",
            "74\n",
            "76\n",
            "57\n",
            "74\n",
            "75\n",
            "56\n",
            "69\n",
            "75\n",
            "64\n",
            "56\n",
            "67\n",
            "1\n",
            "61\n",
            "76\n",
            "60\n",
            "67\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "38\n",
            "56\n",
            "66\n",
            "64\n",
            "69\n",
            "62\n",
            "1\n",
            "56\n",
            "1\n",
            "61\n",
            "56\n",
            "68\n",
            "64\n",
            "69\n",
            "60\n",
            "1\n",
            "78\n",
            "63\n",
            "60\n",
            "73\n",
            "60\n",
            "1\n",
            "56\n",
            "57\n",
            "76\n",
            "69\n",
            "59\n",
            "56\n",
            "69\n",
            "58\n",
            "60\n",
            "1\n",
            "67\n",
            "64\n",
            "60\n",
            "74\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "45\n",
            "63\n",
            "80\n",
            "1\n",
            "74\n",
            "60\n",
            "67\n",
            "61\n",
            "1\n",
            "75\n",
            "63\n",
            "80\n",
            "1\n",
            "61\n",
            "70\n",
            "60\n",
            "8\n",
            "1\n",
            "75\n",
            "70\n",
            "1\n",
            "75\n",
            "63\n",
            "80\n",
            "1\n",
            "74\n",
            "78\n",
            "60\n",
            "60\n",
            "75\n",
            "1\n",
            "74\n",
            "60\n",
            "67\n",
            "61\n",
            "1\n",
            "75\n",
            "70\n",
            "70\n",
            "1\n",
            "58\n",
            "73\n",
            "76\n",
            "60\n",
            "67\n",
            "21\n",
            "0\n",
            "1\n",
            "1\n",
            "45\n",
            "63\n",
            "70\n",
            "76\n",
            "1\n",
            "75\n",
            "63\n",
            "56\n",
            "75\n",
            "1\n",
            "56\n",
            "73\n",
            "75\n",
            "1\n",
            "69\n",
            "70\n",
            "78\n",
            "1\n",
            "75\n",
            "63\n",
            "60\n",
            "1\n",
            "78\n",
            "70\n",
            "73\n",
            "67\n",
            "59\n",
            "5\n",
            "74\n",
            "1\n",
            "61\n",
            "73\n",
            "60\n",
            "74\n",
            "63\n",
            "1\n",
            "70\n",
            "73\n",
            "69\n",
            "56\n",
            "68\n",
            "60\n",
            "69\n",
            "75\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "26\n",
            "69\n",
            "59\n",
            "1\n",
            "70\n",
            "69\n",
            "67\n",
            "80\n",
            "1\n",
            "63\n",
            "60\n",
            "73\n",
            "56\n",
            "67\n",
            "59\n",
            "1\n",
            "75\n",
            "70\n",
            "1\n",
            "75\n",
            "63\n",
            "60\n",
            "1\n",
            "62\n",
            "56\n",
            "76\n",
            "59\n",
            "80\n",
            "1\n",
            "74\n",
            "71\n",
            "73\n",
            "64\n",
            "69\n",
            "62\n",
            "8\n",
            "0\n",
            "1\n",
            "1\n",
            "48\n",
            "64\n",
            "75\n",
            "63\n",
            "64\n",
            "69\n",
            "1\n",
            "75\n",
            "63\n",
            "64\n",
            "69\n",
            "60\n",
            "1\n",
            "70\n",
            "78\n",
            "69\n",
            "1\n",
            "57\n",
            "76\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j-9fluYcTHc0"
      },
      "source": [
        "#for item in char_dataset.take(500):\n",
        "    #print(index_to_char[item.numpy()])"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OYRBGppVTHc0"
      },
      "source": [
        "The **batch** method converts these individual character calls into sequences that can be feeded in as a batch. Using seq_len+1 because of zero indexing. Here is what drop_remainder means:\n",
        "\n",
        "drop_remainder: (Optional.) A `tf.bool` scalar `tf.Tensor`, representing\n",
        "    whether the last batch should be dropped in the case it has fewer than\n",
        "    `batch_size` elements; the default behavior is not to drop the smaller\n",
        "    batch.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mdb1FlAyTHc0"
      },
      "source": [
        "sequences = char_dataset.batch(sequence_length+1, drop_remainder=True)"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6T9drMJ1THc0"
      },
      "source": [
        "----\n",
        "Now that there are sequences, will perform the following steps for each one to create the target text sequences:\n",
        "\n",
        "1. Grab the input text sequence\n",
        "2. Assign the target text sequence as the input text sequence shifted by one step forward\n",
        "3. Group them together as a tuple"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Y3QiI-7THc1"
      },
      "source": [
        "def create_sequence_target(seq):\n",
        "    \n",
        "    input_txt = seq[:-1] # Hello my nam\n",
        "    target_txt = seq[1:] # ello my name\n",
        "    \n",
        "    return input_txt, target_txt"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AraXArDyTHc1"
      },
      "source": [
        "dataset =sequences.map(create_sequence_target)"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4jwVhMYOTHc1",
        "outputId": "e77eb87b-9a29-4d2e-8112-35f7c9a3cf83"
      },
      "source": [
        "for input_txt,target_txt in dataset.take(1):\n",
        "    print(input_txt.numpy())\n",
        "    print(\"\".join(index_to_char[input_txt.numpy()]))\n",
        "    print('\\n')\n",
        "    print(target_txt.numpy())\n",
        "    print(\"\".join(index_to_char[target_txt.numpy()]))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[ 0  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1 12  0\n",
            "  1  1 31 73 70 68  1 61 56 64 73 60 74 75  1 58 73 60 56 75 76 73 60 74\n",
            "  1 78 60  1 59 60 74 64 73 60  1 64 69 58 73 60 56 74 60  8  0  1  1 45\n",
            " 63 56 75  1 75 63 60 73 60 57 80  1 57 60 56 76 75 80  5 74  1 73 70 74\n",
            " 60  1 68 64 62 63 75  1 69 60 77 60 73  1 59 64 60  8  0  1  1 27 76 75]\n",
            "\n",
            "                     1\n",
            "  From fairest creatures we desire increase,\n",
            "  That thereby beauty's rose might never die,\n",
            "  But\n",
            "\n",
            "\n",
            "[ 1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1 12  0  1\n",
            "  1 31 73 70 68  1 61 56 64 73 60 74 75  1 58 73 60 56 75 76 73 60 74  1\n",
            " 78 60  1 59 60 74 64 73 60  1 64 69 58 73 60 56 74 60  8  0  1  1 45 63\n",
            " 56 75  1 75 63 60 73 60 57 80  1 57 60 56 76 75 80  5 74  1 73 70 74 60\n",
            "  1 68 64 62 63 75  1 69 60 77 60 73  1 59 64 60  8  0  1  1 27 76 75  1]\n",
            "                     1\n",
            "  From fairest creatures we desire increase,\n",
            "  That thereby beauty's rose might never die,\n",
            "  But \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WWiJOALkTHc1"
      },
      "source": [
        "---\n",
        "### Generating training batches\n",
        "\n",
        "Now that there are the actual sequences, creating the batches, shuffle the sequences into a random order, so the model doesn't overfit to any section of the text, but can instead generate characters given any seed text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SPpX0dAETHc1"
      },
      "source": [
        "batch_size = 128"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D-6-_SsrTHc2"
      },
      "source": [
        "buffle_size = 10000\n",
        "\n",
        "dataset = dataset.shuffle(buffle_size).batch(batch_size, drop_remainder= True)"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TG-v_9Q9THc2",
        "outputId": "7fe4fdff-500a-4362-ef9f-6a8a19e452f4"
      },
      "source": [
        "dataset"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<BatchDataset shapes: ((128, 120), (128, 120)), types: (tf.int64, tf.int64)>"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RlYEUUGDTHc2"
      },
      "source": [
        "----\n",
        "## Step 4: Creating the Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MQnMT1j_THc2"
      },
      "source": [
        "Using an LSTM based model with a few extra features, including an embedding layer to start off with and **two** LSTM layers. This model architecture off the [DeepMoji](https://deepmoji.mit.edu/) and the original source code can be found [here](https://github.com/bfelbo/DeepMoji).\n",
        "\n",
        "The embedding layer will serve as the input layer, which essentially creates a lookup table that maps the numbers indices of each character to a vector with \"embedding dim\" number of dimensions. Could be imagined, the larger this embedding size, the more complex the training. This is similar to the idea behind word2vec, where words are mapped to some n-dimensional space. Embedding before feeding straight into the LSTM usually leads to more realisitic results."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tgj9astsTHc2"
      },
      "source": [
        "# Length of the vocabulary in chars\n",
        "vocab_size = len(vocab)\n",
        "\n",
        "# The embedding size\n",
        "embed_dim = 64\n",
        "\n",
        "# Number of RNN units\n",
        "rnn_neurons = 1026"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3oJNizVnTHc2"
      },
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense,LSTM,Embedding,Dropout, GRU"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h-3_2p1wTHc2"
      },
      "source": [
        "### Setting up Loss Function\n",
        "\n",
        "Sparse categorical crossentropy will be use for the loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8FcDnvwBTHc2"
      },
      "source": [
        "from tensorflow.keras.losses import sparse_categorical_crossentropy"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i3IVcqOSTHc3"
      },
      "source": [
        "sparse_categorical_crossentropy : Computes the sparse categorical crossentropy loss.\n",
        "\n",
        "    Args:\n",
        "      y_true: Ground truth values.\n",
        "      y_pred: The predicted values.\n",
        "      from_logits: Whether `y_pred` is expected to be a logits tensor. By default,\n",
        "        we assume that `y_pred` encodes a probability distribution.\n",
        "      axis: (Optional) Defaults to -1. The dimension along which the entropy is\n",
        "        computed.\n",
        "    \n",
        "    Returns:\n",
        "      Sparse categorical crossentropy loss value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IYYwjXr2THc3"
      },
      "source": [
        "def sparse_cat_loss(y_true, y_pred):\n",
        "    return sparse_categorical_crossentropy(y_true, y_pred, from_logits= True)"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WvlflVdwTHc3"
      },
      "source": [
        "     Embedding(*args, **kwargs) : Turns positive integers (indexes) into dense vectors of fixed size.\n",
        "\n",
        "    e.g. `[[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]`\n",
        "\n",
        "    This layer can only be used as the first layer in a model.\n",
        "\n",
        "    Args:\n",
        "      input_dim: Integer. Size of the vocabulary, i.e. maximum integer index + 1.\n",
        "      output_dim: Integer. Dimension of the dense embedding.\n",
        "      embeddings_initializer: Initializer for the `embeddings` matrix (see `keras.initializers`).\n",
        "      embeddings_regularizer: Regularizer function applied to the `embeddings` matrix (see `keras.regularizers`).\n",
        "      embeddings_constraint: Constraint function applied to the `embeddings` matrix (see `keras.constraints`)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZJSAjWSLTHc3"
      },
      "source": [
        "def create_model(vocab_size, embed_dim, rnn_neurons, batch_size):\n",
        "    \n",
        "    model = Sequential()\n",
        "    \n",
        "    model.add(Embedding(vocab_size, embed_dim,batch_input_shape = [batch_size, None]))\n",
        "    \n",
        "    model.add(GRU(rnn_neurons, \n",
        "                  return_sequences = True, \n",
        "                  stateful = True, \n",
        "                  recurrent_initializer = 'glorot_uniform'))\n",
        "    \n",
        "    # Final dense layer to predict\n",
        "    model.add(Dense(vocab_size))\n",
        "    \n",
        "    model.compile('adam', loss = sparse_cat_loss)\n",
        "    \n",
        "    return model"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vs74FSXwTHc3"
      },
      "source": [
        "model = create_model(vocab_size = vocab_size,\n",
        "                    embed_dim = embed_dim,\n",
        "                    rnn_neurons = rnn_neurons,\n",
        "                    batch_size = batch_size)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I3Ae-F6ITHc3",
        "outputId": "840a99bd-39be-4e82-d281-dfb9d1b9b110"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "embedding (Embedding)        (128, None, 64)           5376      \n",
            "_________________________________________________________________\n",
            "gru (GRU)                    (128, None, 1026)         3361176   \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (128, None, 84)           86268     \n",
            "=================================================================\n",
            "Total params: 3,452,820\n",
            "Trainable params: 3,452,820\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h6lId8wnTHc3"
      },
      "source": [
        "----\n",
        "## Step 5: Training the model\n",
        "\n",
        "Making sure everything is well with the model before the training! Passing in a batch to confirm the model currently predicts random characters without any training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AaiSUGnlTHc3"
      },
      "source": [
        "for input_example_batch, target_example_batch in dataset.take(1):\n",
        "  \n",
        "  example_batch_predictions = model(input_example_batch)"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kyT9hyv1VKkA",
        "outputId": "faa3d3f6-476f-44f8-9680-42bf9f2de237"
      },
      "source": [
        "example_batch_predictions.shape"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([128, 120, 84])"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LV5GISydVPxz"
      },
      "source": [
        " sampled_indices = tf.random.categorical(example_batch_predictions[0],num_samples=1)"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vVbcWmm_WMA6",
        "outputId": "8e023287-0296-4507-d921-a3af6fc28a3d"
      },
      "source": [
        "sampled_indices"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(120, 1), dtype=int64, numpy=\n",
              "array([[41],\n",
              "       [46],\n",
              "       [21],\n",
              "       [58],\n",
              "       [36],\n",
              "       [66],\n",
              "       [49],\n",
              "       [63],\n",
              "       [53],\n",
              "       [23],\n",
              "       [52],\n",
              "       [38],\n",
              "       [61],\n",
              "       [17],\n",
              "       [24],\n",
              "       [33],\n",
              "       [21],\n",
              "       [12],\n",
              "       [22],\n",
              "       [59],\n",
              "       [80],\n",
              "       [28],\n",
              "       [51],\n",
              "       [23],\n",
              "       [64],\n",
              "       [10],\n",
              "       [48],\n",
              "       [44],\n",
              "       [36],\n",
              "       [40],\n",
              "       [56],\n",
              "       [37],\n",
              "       [61],\n",
              "       [68],\n",
              "       [55],\n",
              "       [20],\n",
              "       [57],\n",
              "       [ 8],\n",
              "       [ 7],\n",
              "       [60],\n",
              "       [53],\n",
              "       [81],\n",
              "       [10],\n",
              "       [37],\n",
              "       [ 5],\n",
              "       [74],\n",
              "       [51],\n",
              "       [60],\n",
              "       [76],\n",
              "       [73],\n",
              "       [10],\n",
              "       [39],\n",
              "       [60],\n",
              "       [67],\n",
              "       [41],\n",
              "       [60],\n",
              "       [37],\n",
              "       [13],\n",
              "       [58],\n",
              "       [17],\n",
              "       [80],\n",
              "       [32],\n",
              "       [10],\n",
              "       [83],\n",
              "       [32],\n",
              "       [24],\n",
              "       [32],\n",
              "       [23],\n",
              "       [81],\n",
              "       [30],\n",
              "       [22],\n",
              "       [41],\n",
              "       [59],\n",
              "       [41],\n",
              "       [25],\n",
              "       [64],\n",
              "       [44],\n",
              "       [26],\n",
              "       [32],\n",
              "       [60],\n",
              "       [51],\n",
              "       [49],\n",
              "       [14],\n",
              "       [37],\n",
              "       [18],\n",
              "       [61],\n",
              "       [46],\n",
              "       [15],\n",
              "       [26],\n",
              "       [37],\n",
              "       [16],\n",
              "       [11],\n",
              "       [24],\n",
              "       [35],\n",
              "       [ 9],\n",
              "       [ 3],\n",
              "       [12],\n",
              "       [55],\n",
              "       [41],\n",
              "       [70],\n",
              "       [ 2],\n",
              "       [26],\n",
              "       [71],\n",
              "       [77],\n",
              "       [44],\n",
              "       [54],\n",
              "       [60],\n",
              "       [62],\n",
              "       [15],\n",
              "       [76],\n",
              "       [ 7],\n",
              "       [82],\n",
              "       [57],\n",
              "       [29],\n",
              "       [46],\n",
              "       [17],\n",
              "       [13],\n",
              "       [61],\n",
              "       [57],\n",
              "       [69]])>"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F-GVuksyW0Qx"
      },
      "source": [
        "    tf.squeeze : Removes dimensions of size 1 from the shape of a tensor.\n",
        "\n",
        "    Args\n",
        "    input :\tA Tensor. The input to squeeze.\n",
        "    axis:\tAn optional list of ints. Defaults to []. \n",
        "    If specified, only squeezes the dimensions listed. The dimension index starts at 0. It is an error to squeeze a dimension that is not 1.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aLnRP5tVXIvr",
        "outputId": "8d641461-a304-4829-b50f-89be071398d4"
      },
      "source": [
        "sampled_indices = tf.squeeze(sampled_indices, axis=-1).numpy()\n",
        "sampled_indices"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([41, 46, 21, 58, 36, 66, 49, 63, 53, 23, 52, 38, 61, 17, 24, 33, 21,\n",
              "       12, 22, 59, 80, 28, 51, 23, 64, 10, 48, 44, 36, 40, 56, 37, 61, 68,\n",
              "       55, 20, 57,  8,  7, 60, 53, 81, 10, 37,  5, 74, 51, 60, 76, 73, 10,\n",
              "       39, 60, 67, 41, 60, 37, 13, 58, 17, 80, 32, 10, 83, 32, 24, 32, 23,\n",
              "       81, 30, 22, 41, 59, 41, 25, 64, 44, 26, 32, 60, 51, 49, 14, 37, 18,\n",
              "       61, 46, 15, 26, 37, 16, 11, 24, 35,  9,  3, 12, 55, 41, 70,  2, 26,\n",
              "       71, 77, 44, 54, 60, 62, 15, 76,  7, 82, 57, 29, 46, 17, 13, 61, 57,\n",
              "       69])"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A4fJ0O_EXhRx",
        "outputId": "4533ac7b-6a99-429d-a484-dcf799924097"
      },
      "source": [
        "index_to_char[sampled_indices]"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['P', 'U', ':', 'c', 'K', 'k', 'X', 'h', ']', '<', '[', 'M', 'f',\n",
              "       '6', '>', 'H', ':', '1', ';', 'd', 'y', 'C', 'Z', '<', 'i', '.',\n",
              "       'W', 'S', 'K', 'O', 'a', 'L', 'f', 'm', '`', '9', 'b', ',', ')',\n",
              "       'e', ']', 'z', '.', 'L', \"'\", 's', 'Z', 'e', 'u', 'r', '.', 'N',\n",
              "       'e', 'l', 'P', 'e', 'L', '2', 'c', '6', 'y', 'G', '.', '}', 'G',\n",
              "       '>', 'G', '<', 'z', 'E', ';', 'P', 'd', 'P', '?', 'i', 'S', 'A',\n",
              "       'G', 'e', 'Z', 'X', '3', 'L', '7', 'f', 'U', '4', 'A', 'L', '5',\n",
              "       '0', '>', 'J', '-', '\"', '1', '`', 'P', 'o', '!', 'A', 'p', 'v',\n",
              "       'S', '_', 'e', 'g', '4', 'u', ')', '|', 'b', 'D', 'U', '6', '2',\n",
              "       'f', 'b', 'n'], dtype='<U1')"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2ZUYbwooYD38",
        "outputId": "ac807763-11b3-4df4-8d25-fea0e2353bbf"
      },
      "source": [
        "print(\"Given the input seq: \\n\")\n",
        "print(\"\".join(index_to_char[input_example_batch[0]]))\n",
        "print('\\n')\n",
        "print(\"Next Char Predictions: \\n\")\n",
        "print(\"\".join(index_to_char[sampled_indices ]))\n"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Given the input seq: \n",
            "\n",
            "ife to Hotspur, and sister to Mortimer.\n",
            "  Lady Mortimer, daughter to Glendower, and wife to Mortimer.\n",
            "  Mistress Quickly\n",
            "\n",
            "\n",
            "Next Char Predictions: \n",
            "\n",
            "PU:cKkXh]<[Mf6>H:1;dyCZ<i.WSKOaLfm`9b,)e]z.L'sZeur.NelPeL2c6yG.}G>G<zE;PdP?iSAGeZX3L7fU4AL50>J-\"1`Po!ApvS_eg4u)|bDU62fbn\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pyPDo9ukYO4R"
      },
      "source": [
        "epochs = 30"
      ],
      "execution_count": 48,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qere2YAoYkif"
      },
      "source": [
        "#model.fit(dataset,epochs= epochs)"
      ],
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j_HQ4R8VYtmo"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eF2f5PS7mBi3"
      },
      "source": [
        "def generate_text(model, start_seed, gen_size = 100, temp = 1.0):\n",
        "\n",
        "  '''\n",
        "  model : Trained model to generate text\n",
        "  start_seed : Initial seed text in string form\n",
        "  gen_size : Number of characters to generate\n",
        "\n",
        "  Basic idea of this function is to take in some seed text, format it so that it is \n",
        "  in the correct shape for our network, then loop the sequence as we keep adding our \n",
        "  predicted characters. Similar to the work in the RNN time series problems.\n",
        "  '''\n",
        "\n",
        "  # Number of characters to generate\n",
        "  num_generate = gen_size\n",
        "\n",
        "  # Vectorizing starting seed text\n",
        "  input_eval = [char_to_index[s] for s in start_seed]\n",
        "\n",
        "  # Expand to match batch format shape\n",
        "  input_eval = tf.expand_dims(input_eval,0)\n",
        "\n",
        "  # Empty list to hold the resulting generated text\n",
        "  text_generated = []\n",
        "\n",
        "  temperature = temp\n",
        "\n",
        "  # Batch size = 1\n",
        "  model.reset_states()\n",
        "\n",
        "  for i in range(num_generate):\n",
        "\n",
        "    # Generate predictions\n",
        "    predictions = model(input_eval)\n",
        "\n",
        "    # Remove the batch shape dimension\n",
        "    predictions = tf.squeeze(predictions, 0)\n",
        "\n",
        "    # Use a categorical distribution to select the next character\n",
        "    predictions = predictions / temperature\n",
        "    predicted_id = tf.random.categorical(predictions, num_samples= 1)[-1,0].numpy()\n",
        "\n",
        "    # Pass the predicted character for the next input\n",
        "    input_eval = tf.expand_dims([predicted_id],0)\n",
        "\n",
        "    # Transform back to character letter\n",
        "    text_generated.append(index_to_char[predicted_id])\n",
        "\n",
        "  return (start_seed + ''.join(text_generated))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L8fjhwslpgU9"
      },
      "source": [
        "print(generate_text(model,\"flower\",gen_size = 1000))"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}