day4/intro_ml_day4_basic_lstm_airlines.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
{
      "cell_type": "markdown",
      "source": [
        "<a href=\"https://colab.research.google.com/github/PrincetonUniversity/intro_machine_learning/blob/main/day4/intro_ml_day4_basic_lstm_airlines.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ],
      "metadata": {
        "id": "zpCkWYWdmgbk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Long Short-Term Memory for Sequential Predictions\n",
        "Introduction to Machine Learning (Day 4)\\\n",
        "Princeton University Wintersession\\\n",
        "Gage DeZoort\\\n",
        "\\\n",
        "Based on several helpful tutorials:\\\n",
        "[1] [LSTM for Time Series Prediction in PyTorch](https://machinelearningmastery.com/lstm-for-time-series-prediction-in-pytorch/)\\\n",
        "[2] [Predicting airline passengers using LSTM and Tensorflow](https://matthewmacfarquhar.medium.com/predicting-airline-passengers-using-lstm-and-tensorflow-ab86347cf318)\n"
      ],
      "metadata": {
        "id": "EcsMvCfajcyc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Temperature Predictions"
      ],
      "metadata": {
        "id": "6kgvYxPHRET4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This project is based on Long Short-Term Memory (LSTM) modules. LSTMs belong to the class of Recurent Neural Networks (RNNs), which operate on sequential data (ordered data, indexed by time or space). For example, the daily temperature is a time series we all experience:\n",
        "\n",
        "![Weather](https://www.influxdata.com/wp-content/uploads/time-series-data-weather-data.png \"weather\")\n",
        "(Image from [this article](https://www.influxdata.com/what-is-time-series-data/))\n",
        "\n",
        "Sentences are another example of sequential data:\n",
        "\n",
        "![Sentences](https://miro.medium.com/v2/resize:fit:1400/format:webp/1*phpgEszN4Q6n_Rtd24zpGw.png \"sentences\")\n",
        "(Image from [this article](https://bansalh944.medium.com/text-generation-using-lstm-b6ced8629b03))\n",
        "\n",
        "We see that sequential data is everywhere! RNNs have accordingly bene applied to a wide variety of domains, including:\n",
        "\n",
        "- Natural Language Processing (NLP): translation, word prediction, sentiment analysis\n",
        "- Time-Series Analysis: financial predictions, weather/climate forecasting\n",
        "- Music Generation: e.g. composition\n",
        "- Robotics: e.g. path predictions\n",
        "\n",
        "How do RNNs work? Here's a helpful diagram:\n",
        "\n",
        "![RNN](https://colah.github.io/posts/2015-08-Understanding-LSTMs/img/RNN-unrolled.png \"rnn\")\n",
        "(Image from [this article](https://colah.github.io/posts/2015-08-Understanding-LSTMs/))\n",
        "\n",
        "In this diagram, $A$ represents the NN. Here, we take $x_t$ to represent the sequence of inputs ($x_0$, $x_1$, $x_2$,...,$x_N$), and $h_t$ its sequence of outputs ($h_0$, $h_1$, $h_2$,...,$h_N$). The sequential nature of the predictions is highlight by the rightward arrows; the prediction at each timestep is informed by the prediction at the previous timestep. Unfortunately, it has been shown that simple RNNs *fail to learn long-term dependencies*. This was the motivation for developing LSTMs.\n",
        "\n",
        "Okay, let's switch to a bit of coding.\n"
      ],
      "metadata": {
        "id": "MqBfwwg3kDrf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import numpy as np\n",
        "import random\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# grab data\n",
        "!wget https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv"
      ],
      "metadata": {
        "id": "-E3OAoyMka0_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dataset Preparation"
      ],
      "metadata": {
        "id": "0MIJL6AZRLBk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the last line, we grabed a CSV (comma-separated value) file called `airline-passengers.csv`. Let's use Pandas to explore the data."
      ],
      "metadata": {
        "id": "V86GylF4r2RI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv(\"airline-passengers.csv\")\n",
        "df.head()"
      ],
      "metadata": {
        "id": "tC-c2ghnra2d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We see that this is a time-series, counting the number of passengers (in units of 1,000) between Jan. 1949 and Dec. 1960, corresponding to 12 years and 144 observations. Let's plot the trend:"
      ],
      "metadata": {
        "id": "VaZLbY4esK8P"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(df.Passengers)\n",
        "plt.xlabel(\"Months Since 01/1949\")\n",
        "plt.ylabel(\"Airline Passengers / 1,000\")\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "kqaIZNjirzTb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "What trends do you observe?"
      ],
      "metadata": {
        "id": "S9BF-m-PuRla"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The upward trend will be difficult for the ML model to capture given the limited size of the dataset. We can simply remove it before training the algorithm, then add it back! Our goal will be to fit a quadratic to the data:\n",
        "\n",
        "`P(m) = x_0 + x_1 * m + x_2 * m^2`\n",
        "\n",
        "Where `P(m)` is the number of passengers in a given month `m`. Let's grab the regression coefficients:"
      ],
      "metadata": {
        "id": "wyj9QoI8QdWU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "N = len(df)\n",
        "ones, xrange = np.ones(N), np.arange(N)\n",
        "X = np.stack((ones, xrange, xrange**2)).T\n",
        "y = df.Passengers.to_numpy().reshape(-1,1)\n",
        "beta = (np.linalg.inv(X.T @ X)@X.T@y)\n",
        "x0, x1, x2 = beta[0][0], beta[1][0], beta[2][0]\n",
        "x0, x1, x2"
      ],
      "metadata": {
        "id": "kZ883Os7Ako0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Exercise 1\n",
        "\n",
        "(3 mins) We've given you a plot of the original data (`passengers`) below; add to this plot 1) the regression line and 2) `passengers` with the regression line subtracted."
      ],
      "metadata": {
        "id": "M5wpPvQURCle"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "passengers = df.Passengers\n",
        "plt.plot(passengers, label=\"Raw Data\")\n",
        "\n",
        "# compute the regression line (\"trend\") as a function of xrange\n",
        "\n",
        "# compute passengers_c = passengers - trend\n",
        "\n",
        "# plot them both\n",
        "\n",
        "\n",
        "plt.xlabel(\"Months Since 01/1949\")\n",
        "plt.ylabel(\"Airline Passengers / 1,000\")\n",
        "plt.legend(loc=\"best\")\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "LNi3ywgiBf3a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Building Train/Test Sets"
      ],
      "metadata": {
        "id": "GmhrvNYJSpW7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# convert everything to plain arrays\n",
        "passengers = passengers.values.astype(\"float32\").reshape(-1,1)\n",
        "passengers_c = passengers_c.values.astype(\"float32\").reshape(-1,1)\n",
        "passengers_c.shape"
      ],
      "metadata": {
        "id": "2kek8arZC-B0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's turn this into an ML task. Given 2/3 of the time series, can we predict the remaining 1/3? Fundamentally, that means we're doing regression.\n"
      ],
      "metadata": {
        "id": "2SOsKDecu1gI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# split into train/test\n",
        "train_size = int(len(passengers_c)*0.67)\n",
        "train, test = passengers_c[:train_size], passengers_c[train_size:]\n",
        "train.shape, test.shape"
      ],
      "metadata": {
        "id": "bPpafstXuwlg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "To train a model, we need to show it data in the time interval $[t-w, t-1]$, where $w$ is the window or \"lookback\" size, and ask it to make predictions for the timestep $t$. To do this, we need to turn our training data into $(X,y)$ pairs,  $X,y\\in\\mathbb{R}^{w}$, where $X$ reprsents the inputs and $y$ represents the targets."
      ],
      "metadata": {
        "id": "MnJ6CLD8wnPC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def create_dataset(dataset, w=1):\n",
        "    X,Y = [], []\n",
        "    for i in range(len(dataset)-w-1):\n",
        "        X.append(dataset[i:(i+w), 0])\n",
        "        Y.append(dataset[i+w, 0])\n",
        "    X, Y = torch.Tensor(X), torch.Tensor(Y)\n",
        "    return X, Y.reshape(len(Y),1)"
      ],
      "metadata": {
        "id": "p4ce4gDWvCCb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "w = 3\n",
        "X_train, y_train = create_dataset(train, w=w)\n",
        "X_test, y_test = create_dataset(test, w=w)"
      ],
      "metadata": {
        "id": "r_e5FONTwNeJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Exercise 2\n",
        "\n",
        "Print the sizes of `X_train`, `y_train`, `X_test`, and `y_test`. Do they make sense? What happens if you increase/decrease `w`?"
      ],
      "metadata": {
        "id": "jlxOy4syS86Y"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "fdGRmXTMTNS0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Building an LSTM\n",
        "Now we're going to build a LSTM! Here's the PyTorch model:"
      ],
      "metadata": {
        "id": "NeSZ_2la3JRH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch.nn as nn\n",
        "\n",
        "class AirModel(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.lstm = nn.LSTM(\n",
        "            input_size=w,\n",
        "            hidden_size=64,\n",
        "            num_layers=1,\n",
        "            batch_first=False\n",
        "          )\n",
        "        self.linear1 = nn.Linear(\n",
        "            in_features=64,\n",
        "            out_features=1,\n",
        "        )\n",
        "        self.relu = torch.nn.ReLU()\n",
        "\n",
        "    def forward(self, x):\n",
        "        x, _ = self.lstm(x)\n",
        "        return self.linear1(x)"
      ],
      "metadata": {
        "id": "lNupbk4E0YKH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This model has several components. The main workhorse is the **LSTM Module**: see the [PyTorch docs](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html) for details of the implementation. Here's a diagram:\n",
        "\n",
        "![LSTM](https://miro.medium.com/v2/resize:fit:1400/format:webp/1*kT7TJdlJflJJSnEJ6XRKug.png \"lstm\")\\\n",
        "(Image from [this article](https://bansalh944.medium.com/text-generation-using-lstm-b6ced8629b03))\n",
        "\\\n",
        "\\\n",
        "This is a single LSTM \"block\" corresponding to the timestep $t$. There's a lot going on here, but here's the gist:\n",
        "\n",
        "- The LSTM block at timestep $t$ is fed by the input $x_t$ (# passengers), the output from the previous block $h_{t-1}$, and the memory from the previous block $c_{t-1}$.\n",
        "- The LSTM block at timestep $t$ is composed of several logical gates. These include an input gate, a forget gate, a cell gate, and an output gate. The full system of equations is\n",
        "\n",
        "$$\n",
        "\\begin{align*}\n",
        "  i_t &= \\sigma(W_{ii} x_t + b_{ii} + W_{hi}h_{t-1} + b_{hi})\\ &\\rightarrow \\ \\ \\text{input gate} \\\\\n",
        "  f_t &= \\sigma(W_{if}x_t + b_{if} + W_{hf}h_{t-1} + b_{hf})\\ &\\rightarrow \\ \\ \\text{forget gate}\\\\\n",
        "  g_t &= \\text{tanh}(W_{ig}x_t + b_{ig} + W_{hg}h_{t-1} + b_{hg})\\ &\\rightarrow \\ \\ \\text{cell features}\\\\\n",
        "  o_t &= \\sigma(W_{io}x_t + b_{io} + W_{ho}h_{t-1} + b_{ho})\\ &\\rightarrow \\ \\ \\text{output gate}\\\\\n",
        "  c_t &= f_t \\odot c_{t-1} + i_t \\odot g_t\\ &\\rightarrow \\ \\ \\text{cell state (memory)}\\\\\n",
        "  h_t &= o_t \\odot \\text{tanh}(c_t) \\ &\\rightarrow \\ \\ \\text{hidden state}\n",
        "\\end{align*}\n",
        "$$\n",
        "\n",
        "In practice, the PyTorch module `nn.LSTM()` has inputs `input_size` corresponding to the dimension of $x_t$, `hidden_size` corresponding to the size of the outputs $h_{t}$, and `num_layers` corresponding to the number of \"stacked\" LSTM modules. Let's train an our model:"
      ],
      "metadata": {
        "id": "2qkn_eY03ZA0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import torch.optim as optim\n",
        "import torch.utils.data as data\n",
        "\n",
        "model = AirModel()\n",
        "optimizer = optim.Adam(model.parameters(), lr=10**-3)\n",
        "loss_fn = nn.MSELoss()\n",
        "loader = data.DataLoader(\n",
        "    data.TensorDataset(X_train, y_train),\n",
        "    shuffle=True,\n",
        "    batch_size=8,\n",
        ")\n",
        "\n",
        "n_epochs = 1000\n",
        "for epoch in range(n_epochs):\n",
        "    model.train()\n",
        "    for X_batch, y_batch in loader:\n",
        "        y_pred = model(X_batch)\n",
        "        loss = loss_fn(y_pred, y_batch)\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "    # Validation\n",
        "    if epoch % 50 != 0:\n",
        "        continue\n",
        "    model.eval()\n",
        "    with torch.no_grad():\n",
        "        y_pred = model(X_train)\n",
        "        train_rmse = np.sqrt(loss_fn(y_pred, y_train))\n",
        "        y_pred = model(X_test)\n",
        "        test_rmse = np.sqrt(loss_fn(y_pred, y_test))\n",
        "    print(\"Epoch %d: train RMSE %.4f, test RMSE %.4f\" % (epoch, train_rmse, test_rmse))"
      ],
      "metadata": {
        "id": "iWoGxJQx4WRM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with torch.no_grad():\n",
        "    train_plot = np.ones(len(passengers_c)) * np.nan\n",
        "    train_plot[w:train_size-1] = model(X_train).flatten()\n",
        "    test_plot = np.ones(len(passengers_c)) * np.nan\n",
        "    test_plot[train_size+w:len(passengers_c)-1] = model(X_test).flatten()\n",
        "\n",
        "plt.plot(passengers_c.flatten() + trend, c='b', label=\"Truth\")\n",
        "plt.plot(train_plot + trend, c='r', label=\"Training\")\n",
        "plt.plot(test_plot + trend, c='g', label=\"Predicted\")\n",
        "plt.show()\n"
      ],
      "metadata": {
        "id": "C1DbmBP7_azt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Clearly, the LSTM understands how to model the trends in the data, but is having issues capturing the magnitude of the seasonal fluctuations."
      ],
      "metadata": {
        "id": "o8L0-OP5Ue3w"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Exercise 3\n",
        "Can you improve the performance of the model? The full code from the notebook is reproduced below for your convenience. You may want to explore:\n",
        "\n",
        "0) Add a second linear layer to the output of your network (see comments in the model below). \\\n",
        "1) Adjusting the learning rate.\\\n",
        "2) Increasing the size (# neurons, # layers) of the NN.\\\n",
        "3) Changing `w` (note that this alters the nature of the learning task, but may still be fun to explore).\\\n",
        "4) Changing the batch size."
      ],
      "metadata": {
        "id": "NvNO2WdrUmBS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import numpy as np\n",
        "import random\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "import torch.utils.data as data\n",
        "\n",
        "df = pd.read_csv(\"airline-passengers.csv\")\n",
        "\n",
        "# fit the trend\n",
        "N = len(df)\n",
        "ones, xrange = np.ones(N), np.arange(N)\n",
        "X = np.stack((ones, xrange, xrange**2)).T\n",
        "y = df.Passengers.to_numpy().reshape(-1,1)\n",
        "beta = (np.linalg.inv(X.T @ X)@X.T@y)\n",
        "x0, x1, x2 = beta[0][0], beta[1][0], beta[2][0]\n",
        "\n",
        "# convert everything to plain arrays\n",
        "passengers = df.Passengers\n",
        "trend = x0 + xrange*x1 + xrange**2 * x2\n",
        "passengers_c = passengers - trend\n",
        "passengers = passengers.values.astype(\"float32\").reshape(-1,1)\n",
        "passengers_c = passengers_c.values.astype(\"float32\").reshape(-1,1)\n",
        "passengers_c.shape\n",
        "\n",
        "# split into train/test\n",
        "train_size = int(len(passengers_c)*0.67)\n",
        "train, test = passengers_c[:train_size], passengers_c[train_size:]\n",
        "train.shape, test.shape\n",
        "\n",
        "# define the model\n",
        "class AirModel(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.lstm = nn.LSTM(\n",
        "            input_size=w,\n",
        "            hidden_size=64,\n",
        "            num_layers=1,\n",
        "            batch_first=False\n",
        "          )\n",
        "        self.linear1 = nn.Linear(\n",
        "            in_features=64,\n",
        "            out_features=1, #64,\n",
        "        )\n",
        "        #self.linear2 = nn.Linear(\n",
        "        #    in_features=64,\n",
        "        #    out_features=1,\n",
        "        #)\n",
        "        self.relu = torch.nn.ReLU()\n",
        "\n",
        "    def forward(self, x):\n",
        "        x, _ = self.lstm(x)\n",
        "        return self.linear1(x)\n",
        "\n",
        "model = AirModel()\n",
        "optimizer = optim.Adam(model.parameters(), lr=10**-3)\n",
        "loss_fn = nn.MSELoss()\n",
        "loader = data.DataLoader(\n",
        "    data.TensorDataset(X_train, y_train),\n",
        "    shuffle=True,\n",
        "    batch_size=8,\n",
        ")\n",
        "\n",
        "n_epochs = 1000\n",
        "for epoch in range(n_epochs):\n",
        "    model.train()\n",
        "    for X_batch, y_batch in loader:\n",
        "        y_pred = model(X_batch)\n",
        "        loss = loss_fn(y_pred, y_batch)\n",
        "        optimizer.zero_grad()\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "    # Validation\n",
        "    if epoch % 50 != 0:\n",
        "        continue\n",
        "    model.eval()\n",
        "    with torch.no_grad():\n",
        "        y_pred = model(X_train)\n",
        "        train_rmse = np.sqrt(loss_fn(y_pred, y_train))\n",
        "        y_pred = model(X_test)\n",
        "        test_rmse = np.sqrt(loss_fn(y_pred, y_test))\n",
        "    print(\"Epoch %d: train RMSE %.4f, test RMSE %.4f\" % (epoch, train_rmse, test_rmse))\n",
        "\n",
        "with torch.no_grad():\n",
        "  train_plot = np.ones(len(passengers_c)) * np.nan\n",
        "  train_plot[w:train_size-1] = model(X_train).flatten()\n",
        "  test_plot = np.ones(len(passengers_c)) * np.nan\n",
        "  test_plot[train_size+w:len(passengers_c)-1] = model(X_test).flatten()\n",
        "\n",
        "plt.plot(passengers_c.flatten() + trend, c='b', label=\"Truth\")\n",
        "plt.plot(train_plot + trend, c='r', label=\"Training\")\n",
        "plt.plot(test_plot + trend, c='g', label=\"Predicted\")\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "_Hc3-ZEE_z9q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise Solutions"
      ],
      "metadata": {
        "id": "h9e5ZEp0STqi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "Exercise 1\n",
        "\n",
        "passengers = df.Passengers\n",
        "plt.plot(passengers, label=\"Raw Data\")\n",
        "trend = x0 + xrange*x1 + xrange**2 * x2\n",
        "passengers_c = passengers - trend\n",
        "plt.plot(trend, \"r-\", label=\"Global Trend\")\n",
        "plt.plot(passengers_c, \"g--\", label=\"Corrected\")\n",
        "plt.xlabel(\"Months Since 01/1949\")\n",
        "plt.ylabel(\"Airline Passengers / 1,000\")\n",
        "plt.legend(loc=\"best\")\n",
        "plt.show()\n",
        "\n",
        "```"
      ],
      "metadata": {
        "id": "KpmyXNSfSO0s"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "Exercise 2\n",
        "\n",
        "In summary, the size is [window sample, time steps, features].\n",
        "print(X_train.shape)\n",
        "print(y_train.shape)\n",
        "print(X_test.shape)\n",
        "print(y_test.shape)\n",
        "print(X_train[0])\n",
        "print(y_train[0])\n",
        "\n",
        "What do the shapes of these tensors tell us?\n",
        "- `X_train` has 95 entries of the form [[t-5],[t-4],[t-3],[t-2],[t-1]].  `Y_train` has 95 entries of the form [[t]].\n",
        "The story is similar for the test set, which has 47 entries.\n",
        "```"
      ],
      "metadata": {
        "id": "w0_pRUrbTTeS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```\n",
        "Exercise 3\n",
        "Adding a second linear layer definitely helps. Slowing the learning rate helps a bit too.\n",
        "```"
      ],
      "metadata": {
        "id": "oqOqCtrnmYHq"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "kFMnCMd0AHe2"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}