1-neural_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "38250724",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1 style=\"text-align: center; vertical-align: middle;\">Introduction to neural networks</h1>\n",
    "\n",
    "<h3 style=\"text-align: center; vertical-align: middle; margin-top: 1em; margin-bottom: 1em;\"> Tutorial by Dr. Andrea Santamaria Garcia and Chenran Xu</h3>\n",
    "\n",
    "<div style=\"width: 50%; horizontal-align: middle; vertical-align: middle; \">\n",
    "<img src=\"./img/KIT_logo.svg\" style=\"width: 30%; float: right; vertical-align: center;\" />\n",
    "</div>\n",
    "<div style=\"clear: both; vertical-align: middle;\"></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a57cc074",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Get the repository with Git\n",
    "\n",
    "You will need to have Git previously installed in your computer.\n",
    "To check if you have it installed, open your terminal and type:\n",
    "\n",
    "``` bash\n",
    "git --version\n",
    "```\n",
    "\n",
    "## Git installation in mac\n",
    "\n",
    "``` bash\n",
    "brew update\n",
    "brew install git\n",
    "```\n",
    "\n",
    "## Git installation in linux\n",
    "\n",
    "In Ubuntu/Debian\n",
    "\n",
    "``` bash\n",
    "sudo apt install git\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d250bb9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Download the repository\n",
    "\n",
    "Once you have Git installed open your terminal, go to your desired directory, and type:\n",
    "\n",
    "``` bash\n",
    "git clone https://github.com/machine-learning-tutorial/neural-networks\n",
    "cd neural-networks\n",
    "```\n",
    "\n",
    "Or get the repository with direct download:\n",
    "\n",
    "``` bash\n",
    "wget https://github.com/machine-learning-tutorial/neural_networks/archive/refs/heads/main.zip\n",
    "unzip main.zip\n",
    "cd neural-networks\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61907598",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Install dependencies\n",
    "\n",
    "You need to install the dependencies before running the notebooks.\n",
    "\n",
    "## Using conda\n",
    "\n",
    "If you don't have conda installed already and want to use conda for environment management, you can install the miniconda as [described here](https://docs.conda.io/projects/miniconda/en/latest/miniconda-install.html).\n",
    "\n",
    "Then run the following commands:\n",
    "\n",
    "```bash\n",
    "conda create -n nn-tutorial python=3.10\n",
    "conda activate nn-tutorial\n",
    "pip install -r requirements.txt\n",
    "jupyter contrib nbextension install --user\n",
    "jupyter nbextension enable varInspector/main\n",
    "```\n",
    "\n",
    "- After the tutorial you can remove your environment with `conda remove -n nn-tutorial --all`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39f57d60",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Install dependencies\n",
    "\n",
    "You need to install the dependencies before running the notebooks.\n",
    "\n",
    "## Using venv only\n",
    "\n",
    "If you do not have conda installed:\n",
    "\n",
    "Alternatively, you can create the virtual env with `venv` in the standard library\n",
    "\n",
    "```bash\n",
    "python -m venv nn-tutorial\n",
    "```\n",
    "\n",
    "and activate the env with $ source <venv>/bin/activate (bash) or C:> <venv>/Scripts/activate.bat (Windows)\n",
    "\n",
    "Then, install the packages with pip within the activated environment\n",
    "\n",
    "\n",
    "```bash\n",
    "python -m pip install -r requirements.txt\n",
    "jupyter contrib nbextension install --user\n",
    "jupyter nbextension enable varInspector/main\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33befbca",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Running the tutorial\n",
    "\n",
    "You can start the notebook in the terminal, and it will start a browser automatically\n",
    "\n",
    "```bash\n",
    "jupyter notebook\n",
    "```\n",
    "\n",
    "Alternatively, you can use supported Editor to run the jupyter notebooks, e.g. with VS Code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb4a1435",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h1>Run this first!</h1>\n",
    "\n",
    "Imports and modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e538eebc",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import h5py\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import set_matplotlib_formats, display\n",
    "from torch.utils.data import DataLoader, TensorDataset\n",
    "\n",
    "plt.rcParams['figure.figsize'] = 6, 4\n",
    "plt.rcParams['savefig.dpi'] = 300\n",
    "plt.rcParams['image.cmap'] = \"viridis\"\n",
    "plt.rcParams['image.interpolation'] = \"none\"\n",
    "plt.rcParams['savefig.bbox'] = \"tight\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa5f101",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Reproducibility\n",
    "\n",
    "- We set the random seeds so that the training results are always the same\n",
    "- Feel free to change the seed number to see the effects of the random initialization of the network weights on the training results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "626bcc7b",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "SEED = 26\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.openmp.deterministic = True\n",
    "np.random.seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bff3655e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Accelerated computing\n",
    "\n",
    "- **Accelerated computing** = when we add extra hardware to accelerate computation, like GPUs (needed in deep machine learning).\n",
    "- **GPU**: many \"not-so-intelligent\" cores that are parallelizable. They can carry out specific operations in a very efficient way, e.g. tensor cores perform very efficient sparse tensor multiplication.\n",
    "\n",
    "We will be working with torch tensors in this notebook! instead of the usual numpy arrays. This means you could execute this code on a GPU if you have access to one with a simple command `torch.device(\"cuda\")`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acb09c91",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Use GPU if available\n",
    "device = ('cuda' if torch.cuda.is_available()\n",
    "          else 'cpu')\n",
    "print(f'Using {device} device')\n",
    "# device ='cpu'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a54b239f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conventions for this notebook\n",
    "\n",
    "### Jargon\n",
    "\n",
    "- Unit = activation = neuron\n",
    "- Model = neural network\n",
    "- Feature = dimension of input vector = number of independent variables\n",
    "- Hypothesis = prediction = output of the model\n",
    "\n",
    "### Indices\n",
    "\n",
    "- **Data points:** $i = 1,..., n$ \n",
    "- **Parameters of the model:** $k = 1,..., p$ \n",
    "- **Layers:** $j = 1,..., l$ \n",
    "- **Activation unit label:** $s$ \n",
    "\n",
    "### Scalars\n",
    "\n",
    "- $u^j$ = number of units in layer $j$\n",
    "- $z_s^j$ is the activation unit $s$ in layer $j$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e5a9069",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conventions for this notebook\n",
    "\n",
    "### Vectors and matrices\n",
    "\n",
    "- $\\pmb{X}$: input vector of dimension $[n \\times (p \\times 1)]$\n",
    "- $z^j$: activation vector of layer $j$ of dimension $[(u^j + 1) \\times 1]$\n",
    "- $\\pmb{w}^j$: weight matrix from layer $j$ to $j+1$, of dimension $[u^{j+1} \\times (u^j + 1)]$\n",
    "\n",
    "\n",
    "<span style='color:Blue'> where the $+1$ accounts for the bias unit </span>\n",
    "\n",
    "\n",
    "$$\n",
    "\\pmb{X} =\n",
    "\\begin{bmatrix}\n",
    "x_0 \\\\\n",
    "x_1 \\\\\n",
    "\\vdots \\\\\n",
    "x_p\n",
    "\\end{bmatrix} \\ \\ ; \\ \\\n",
    "\\pmb{w}^j =\n",
    "\\begin{bmatrix}\n",
    "w_{10} & \\dots & w_{1(u^j + 1)}\\\\\n",
    "w_{20} & \\ddots\\\\\n",
    "\\vdots \\\\\n",
    "w_{(u^{j+1}) 0} & & w_{(u^{j+1})(u^j + 1)}\\\\\n",
    "\\end{bmatrix} \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c56d4e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Universal Approximation Theorem\n",
    "- When the activation function is non-linear, then a two-layer neural network can be proven to be a **universal function approximator**.\n",
    "- <span style='color:red'> This is where the power of neural networks comes from! </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69df10fa",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Create a function to fit\n",
    "Let's create a simple non-linear function to fit with our neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b27267d5",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "sample_points = 1e3\n",
    "x_lim = 100\n",
    "x = np.linspace(0, x_lim, int(sample_points))\n",
    "y = np.sin(x * x_lim * 1e-4) * np.cos(x * x_lim * 1e-3) * 3\n",
    "plt.plot(x, y)\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')\n",
    "plt.grid()\n",
    "plt.title('Function to be fitted')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1457e577",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data shape\n",
    "- Our data is 1D, meaning it has only one feature.\n",
    "- We want a model that for a given $x$ it returns the correspondent $y$ value.\n",
    "- This means that a model with one neuron input and a one neuron output suffices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d964b9bd",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "n_input = 1\n",
    "n_out = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cda2d1f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(x))\n",
    "print(len(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac9c3d99",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "In order for the model to take each point of the data one by one we need to do some additional re-shaping, where we introduce an additional dimension for each entry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "436f80a6",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "x_reshape = x.reshape((int(len(x) / n_input), n_input))\n",
    "y_reshape = y.reshape((int(len(y) / n_out), n_out))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b669f0a9",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Uncomment to check the shape change\n",
    "print(x.shape, y.shape)\n",
    "print(x_reshape.shape, y_reshape.shape)\n",
    "print(x[10], x_reshape[10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2844eee1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(x_reshape)\n",
    "# print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa1ccdbe",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h2>PyTorch</h2>\n",
    "\n",
    "<a href=\"https://pytorch.org/\">PyTorch</a> is an optimized tensor library for deep learning using GPUs and CPUS.\n",
    "- A <span style='color:#b51f2a'> **tensor** </span> is an algebraic object that may map between different objects such as vectors, scalars, and even other tensors. It can be easily understood as a multidimensional matrix/array. \n",
    "    - These objects allow to easily carry out machine learning computations in problems with many features, weights, etc.\n",
    "    - In PyTorch, a <a href=\"https://pytorch.org/docs/stable/tensors.html#:~:text=A%20torch.,of%20a%20single%20data%20type.\">tensor</a> is a multi-dimensional matrix containing elements of a single data type.\n",
    "\n",
    "<img src=\"img/tensor_2.jpeg\" style=\"width:50%; margin:auto;\" />\n",
    "<p style=\"clear:both; font-size: small; text-align: center; margin-top:1em;\">image from <a href=\"https://towardsai.net/p/deep-learning/working-with-pytorch-tensors\">Working with PyTorch tensors</a></p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6664a613",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data type\n",
    "The data that we will input to the model needs to be of the type `torch.float32`\n",
    "\n",
    "_Side Remark_: The default dtype of torch tensors (also the layer parameters) is `torch.float32`, which is related to the GPU performance optimization. If one wants to use `torch.float64`/`torch.double` instead, one can set the tensors to double precision via `v = v.double()` or set the global precision via `torch.set_default_dtype(torch.float64)`. Just keep in mind, the NN parameters and the input tensors should have the same precision.\n",
    "\n",
    "Before starting, let's convert our data numpy arrays to torch tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "340bc8b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_torch = torch.from_numpy(x_reshape)\n",
    "y_torch = torch.from_numpy(y_reshape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95a39f64",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type checking:\n",
    "print(x.dtype, y.dtype)\n",
    "print(x_torch.dtype, y_torch.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b269357",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The type is still not correct, but we can easily convert it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2a41119",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_torch = x_torch.to(dtype=torch.float32)\n",
    "y_torch = y_torch.to(dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74760bf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type checking:\n",
    "print(x.dtype, y.dtype)\n",
    "print(x_torch.dtype, y_torch.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f74ae34d",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x_torch.numpy(), y_torch.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80ec2b92",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data normalization\n",
    "We will also need to normalize the data to make sure we are in the non-linear region of the activation functions:\n",
    "\n",
    "<img src=\"img/activation-functions.png\" width=\"85%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff97d515",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We are using min-max normalization to normalize the input tensors to [0,1] and output tensors to [-0.5,0.5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da256a5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_norm = (x_torch - x_torch.min()) / (x_torch.max() - x_torch.min())\n",
    "y_norm = (y_torch - y_torch.min()) / (y_torch.max() - y_torch.min()) - 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa4d6625",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(x_norm.detach().numpy(), y_norm.detach().numpy())\n",
    "plt.xlabel('x')\n",
    "plt.ylabel('y')\n",
    "plt.grid()\n",
    "plt.title('Normalized function')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc7f9018",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_norm.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306fc79d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Build your model\n",
    "- In PyTorch [`Sequential`](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html#torch.nn.Sequential) stands for *sequential container*, where modules can be added sequentially and are connected in a cascading way. The output for each module is forwarded sequentially to the next.\n",
    "- Now we will build a simple model with one hidden layer with `Sequential`\n",
    "- Remember that every layer in a neural network is followed by an **activation layer** that performs some additional operations on the neurons.\n",
    "\n",
    "### Let's build 3 different models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3da2221",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model 0\n",
    "\n",
    "A small model with small non-linearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bb6bff9",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_hidden_01 = 5\n",
    "\n",
    "model0 = nn.Sequential(nn.Linear(n_input, n_hidden_01),\n",
    "                      nn.LeakyReLU(),\n",
    "                      nn.Linear(n_hidden_01, n_out),\n",
    "                      )\n",
    "print(model0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f8a9b87",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model 1\n",
    "\n",
    "A small model with some non-linearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56967956",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_hidden_11 = 5\n",
    "\n",
    "model1 = nn.Sequential(nn.Linear(n_input, n_hidden_11),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(n_hidden_11, n_out),\n",
    "                      )\n",
    "print(model1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36734dbb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model 2\n",
    "\n",
    "A larger model with non-linearity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64f4f426",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_hidden_21 = 10\n",
    "n_hidden_22 = 5\n",
    "model2 = nn.Sequential(nn.Linear(n_input, n_hidden_21),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(n_hidden_21, n_hidden_22),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(n_hidden_22, n_out),\n",
    "                      )\n",
    "print(model2)\n",
    "\n",
    "# model2 = nn.Sequential(nn.Linear(n_input, n_hidden_21),\n",
    "#                       nn.LeakyReLU(),\n",
    "#                       nn.Linear(n_hidden_21, n_hidden_22),\n",
    "#                       nn.LeakyReLU(),\n",
    "#                       nn.Linear(n_hidden_22, n_out),\n",
    "#                       )\n",
    "# print(model2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f67c104",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3 style=\"color:#e6541a;\">How much do you think each hyperparameter will affect the quality of the model</h3>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ uncomment and execute the next line to explore the methods of the <code>model</code> object you created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02e4fbbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dir(model0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99dad8b",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Understanding the PyTorch model\n",
    "Try the `parameters` method (needs to be instantiated)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59410e54",
   "metadata": {},
   "outputs": [],
   "source": [
    "model0.parameters()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d3aa751",
   "metadata": {},
   "source": [
    "The `parameters` method gives back a *generator*, which means it needs to be iterated over to give back an output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95e9f71a",
   "metadata": {},
   "outputs": [],
   "source": [
    "for element in model0.parameters():\n",
    "    print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61096971",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3 style=\"color:#e6541a;\">Without taking into account any bias unit: can you identify the elements of the model by their dimensions?</h3>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ The first element corresponds to the weight matrix $\\theta^0$ from layer 0 to layer 1, of dimensions $u^{j+1} \\times u^j = u^2 \\times u^1$ (so, without bias)</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ The second element corresponds to the values of the activation units in layer 1</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ The third element corresponds to the weight matrix $\\theta^1$ from layer 1 to layer 2, of dimensions $u^{j+1} \\times u^j = u^3 \\times u^3 $ (without bias)</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ The fourth element is the output of the model</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abb37963",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's have a look at what the contents of those tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ef5f2e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "for element in model0.parameters():\n",
    "    print(element)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67e4fb22",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<h3 style=\"color:#e6541a;\">What are these values?</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a07f9f2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Define the loss function\n",
    "- Reminder: the **loss function** measures how distant the predictions made by the model are from the actual values\n",
    "- `torch.nn` provides many different types of [loss functions](https://pytorch.org/docs/stable/nn.html#loss-functions). One of the most popular ones in the [Mean Squared Error (MSE)](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss) since it can be applied to a wide variety of cases.\n",
    "- In general cost functions are chosen depending on desirable properties, such as convexity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc2ceaf9",
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_function = nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "085beddd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Define the optimizer\n",
    "[`torch.optim`](https://pytorch.org/docs/stable/optim.html) provides implementations of various optimization algorithms. The optimizer object will hold the current state and will update the parameters of the model based on computer gradients. It takes as an input an iterable containing the model parameters, that we explored before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85354d61",
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 1e-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90b75147",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer0 = torch.optim.Adam(model0.parameters(), lr=learning_rate)\n",
    "optimizer1 = torch.optim.Adam(model1.parameters(), lr=learning_rate)\n",
    "optimizer2 = torch.optim.Adam(model2.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f92a6ed6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# optimizer0 = torch.optim.SGD(model0.parameters(), lr=learning_rate)\n",
    "# optimizer1 = torch.optim.SGD(model1.parameters(), lr=learning_rate)\n",
    "# optimizer2 = torch.optim.SGD(model2.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b14ace05",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Train the models on a loop\n",
    "The model learns iteratively in a loop of a given number of epochs. Each loop consists of:\n",
    "- A **forward propagation**: compute $y$ given the input $x$ and current weights and calculate the loss\n",
    "- A **backward propagation**: compute the gradient of the loss function (error of the loss at each unit)\n",
    "- Gradient descent: update model weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43bdff9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64 # how many points to pass to the model at a time\n",
    "# batch_size = len(x_norm)  # uncomment to pass all data at once\n",
    "dataset = TensorDataset(x_norm, y_norm)\n",
    "dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True, pin_memory=False, drop_last=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84e1212f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the training loop NEW\n",
    "def training_loop(dataloader, model, optimizer, epochs):\n",
    "    losses = []\n",
    "    for _ in range(epochs):\n",
    "        for id_batch, (x_batch, y_batch) in enumerate(dataloader):\n",
    "            x_batch = x_batch.to(device)\n",
    "            y_batch = y_batch.to(device)\n",
    "            pred_y = model(x_batch)\n",
    "            optimizer.zero_grad()\n",
    "            loss = loss_function(pred_y, y_batch)\n",
    "            loss.backward()  # Back-prop\n",
    "            optimizer.step()\n",
    "            losses.append(loss.item())\n",
    "    return losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8455e259",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# Run the training for all the models\n",
    "# epochs = 2000\n",
    "epochs = 500\n",
    "\n",
    "losses0 = training_loop(dataloader, model0, optimizer0, epochs=epochs)\n",
    "losses1 = training_loop(dataloader, model1, optimizer1, epochs=epochs)\n",
    "losses2 = training_loop(dataloader, model2, optimizer2, epochs=epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a14805b5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(losses0, label='Model 0', color='green')\n",
    "plt.plot(losses1, label='Model 1', color='blue')\n",
    "plt.plot(losses2, label='Model 2', color='red')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.title(\"Learning rate %f\"%(learning_rate))\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0bac345",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3 style=\"color:#e6541a;\">Interpreting the loss curves</h3>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Have the NNs learned?</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Why is model 0 learning faster than model 1?</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Why is model 2 better than models 0 and 1?</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Train for more epochs. How does the loss curve change?</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Change the number of minibatches to pass all data at once. How does the loss curve change? which method is more effective</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "846d5b90",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Test the trained model\n",
    "- Let's create some random points in the x-axis within the model's interval that will serve as test data.\n",
    "- We will do the same data manipulations as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "436395cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_points = 50\n",
    "x_test = np.random.uniform(0, np.max(x_norm.detach().numpy()), test_points)\n",
    "x_test_reshape = x_test.reshape((int(len(x_test) / n_input), n_input))\n",
    "x_test_torch = torch.from_numpy(x_test_reshape)\n",
    "x_test_torch = x_test_torch.to(dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a28a45d3",
   "metadata": {},
   "source": [
    "Now we predict the y-value with our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38a68ec4",
   "metadata": {},
   "outputs": [],
   "source": [
    "y0_test_torch = model0(x_test_torch)\n",
    "y1_test_torch = model1(x_test_torch)\n",
    "y2_test_torch = model2(x_test_torch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "114f53af",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "plt.plot(x_norm.detach().numpy(), y_norm.detach().numpy())\n",
    "plt.scatter(x_test_torch.detach().numpy(), y0_test_torch.detach().numpy(), color='green', marker='*', label='Model 0')\n",
    "plt.scatter(x_test_torch.detach().numpy(), y1_test_torch.detach().numpy(), color='blue', marker='v', label='Model 1')\n",
    "plt.scatter(x_test_torch.detach().numpy(), y2_test_torch.detach().numpy(), color='red', label='Model 2')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "703024a9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3 style=\"color:#e6541a;\">Comment on the NN predictions</h3>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Why does the prediction of model 0 have that particular shape?</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Which activation function would be more appropriate to fit this function, the one from model 0 or model 1?</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Which NN gets the best prediction and why?</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "412c4eb3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<h3 style=\"color:#e6541a;\">Bonus</h3>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Change the seed at the top of the notebook. How do the predictions change?</p>\n",
    "<p style=\"color:#e6541a;\">$\\implies$ Change the optimizer in <code>Section 4</code> from <code>Adam</code> to <code>SGD</code> and re-train the models. What happens? How did the loss curves change? Did the NNs learn? Change the number of epochs and try to make it learn.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31ca70bb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Play with the notebook!\n",
    "Some ideas:\n",
    "- Change the number of epochs in `Section 5` to 5000 and re-train the models. What happens?\n",
    "- Change the random seed in the `Reproducibility` cell at the very top. How do the results change?\n",
    "- Change the optimizer in `Section 4` from `Adam` to `SGD` and re-train the models. What happens?\n",
    "- [**if time allows, takes several minutes**] Change the epochs in `Section 5` to 1000000. What happens?\n",
    "- Go back to 1000 epochs and the Adam optimizer. Change the learning rate in `Section 4` to 0.05. How do the results change? what does it tell us about our previous value?\n",
    "- Change the learning rate to 0.5. What happens now?\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "aefbd9db9e80b76479690376e0f6858b117f02d7b00937331627c3692732c1c7"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}