notebooks/8-optimization.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "8-optimization.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jonkrohn/ML-foundations/blob/master/notebooks/8-optimization.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aTOLgsbN69-P"
      },
      "source": [
        "# Optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sqs7yWgf_NQA"
      },
      "source": [
        "This class, *Optimization*, is the eighth of eight classes in the *Machine Learning Foundations* series. It builds upon the material from each of the other classes in the series -- on linear algebra, calculus, probability, statistics, and algorithms -- in order to provide a detailed introduction to training machine learning models. \n",
        "\n",
        "Through the measured exposition of theory paired with interactive examples, you’ll develop a working understanding of all of the essential theory behind the ubiquitous gradient descent approach to optimization as well as how to apply it yourself — both at a granular, matrix operations level and a quick, abstract level — with TensorFlow and PyTorch. You’ll also learn about the latest optimizers, such as Adam and Nadam, that are widely-used for training deep neural networks. \n",
        "\n",
        "Over the course of studying this topic, you'll:\n",
        "\n",
        "* Discover how the statistical and machine learning approaches to optimization differ, and why you would select one or the other for a given problem you’re solving.\n",
        "* Understand exactly how the extremely versatile (stochastic) gradient descent optimization algorithm works, including how to apply it\n",
        "* Get acquainted with the “fancy” optimizers that are available for advanced machine learning approaches (e.g., deep learning) and when you should consider using them.\n",
        "\n",
        "Note that this Jupyter notebook is not intended to stand alone. It is the companion code to a lecture or to videos from Jon Krohn's [Machine Learning Foundations](https://github.com/jonkrohn/ML-foundations) series, which offer detail on the following:\n",
        "\n",
        "*Segment 1: The Machine Learning Approach to Optimization*\n",
        "\n",
        "* The Statistical Approach to Regression: Ordinary Least Squares\n",
        "* When Statistical Approaches to Optimization Break Down\n",
        "* The Machine Learning Solution \n",
        "\n",
        "*Segment 2: Gradient Descent*\n",
        "\n",
        "* Objective Functions\n",
        "* Cost / Loss / Error Functions\n",
        "* Minimizing Cost with Gradient Descent\n",
        "* Learning Rate\n",
        "* Critical Points, incl. Saddle Points\n",
        "* Gradient Descent from Scratch with PyTorch\n",
        "* The Global Minimum and Local Minima\n",
        "* Mini-Batches and Stochastic Gradient Descent (SGD)\n",
        "* Learning Rate Scheduling\n",
        "* Maximizing Reward with Gradient Ascent\n",
        "\n",
        "*Segment 3: Fancy Deep Learning Optimizers*\n",
        "\n",
        "* A Layer of Artificial Neurons in PyTorch\n",
        "* Jacobian Matrices\n",
        "* Hessian Matrices and Second-Order Optimization\n",
        "* Momentum\n",
        "* Nesterov Momentum\n",
        "* AdaGrad\n",
        "* AdaDelta \n",
        "* RMSProp\n",
        "* Adam \n",
        "* Nadam\n",
        "* Training a Deep Neural Net\n",
        "* Resources for Further Study"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zBdzukZv5b1w"
      },
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VSe5VI38a1Ns"
      },
      "source": [
        "## Segment 1: Optimization Approaches"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ay3FqULN5b11"
      },
      "source": [
        "Refer to the slides and the *Ordinary Least Squares* section of the [*Intro to Stats* notebook](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/6-statistics.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D2y1HvVD5b12"
      },
      "source": [
        "## Segment 2: Gradient Descent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nD25jSYF5b12"
      },
      "source": [
        "### Cost Functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "grAFt27_5b13"
      },
      "source": [
        "Fundamentally, for some instance $i$, we'd like to quantify the difference between the \"correct\" target model output $y_i$ and the model's predicted output $\\hat{y}_i$. A first idea might be to take the simple difference: $$\\Delta y_i = \\hat{y}_i - y_i$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vf0Qh1mk5b13"
      },
      "source": [
        "For computational efficiency (we'll explore this later in the segment), in ML we seldom consider the cost associated with a single instance $i$. Instead, we typically consider several instances simultaneously, in which case calculating the simple difference is largely  ineffective because positive and negative $\\Delta y_i$ values cancel out. E.g., consider a situation where: \n",
        "\n",
        "* $\\Delta y_1 = \\hat{y}_1 - y_1$ = 7 - 2 = 5\n",
        "* $\\Delta y_2 = \\hat{y}_2 - y_2$ = 3 - 8 = -5\n",
        "\n",
        "On an individual-instance basis, there are differences between the predicted and target outputs, indicating the model could be improved. Despite this, the total cost ($\\Sigma \\Delta y_i = \\Delta y_1 + \\Delta y_2 = 5-5$) is zero and therefore the mean cost ($\\frac{\\Sigma{\\Delta y_i}}{n} = \\frac{0}{2}$) is also zero, erroneously implying a perfect model fit."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3PIhblP-5b14"
      },
      "source": [
        "#### Mean Absolute Error"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZmE1Zu255b14"
      },
      "source": [
        "A straightforward resolution to the simple-difference shortcoming is to take the absolute value of the difference. I.e., using the same contrived values as above: \n",
        "\n",
        "* $|\\Delta y_1| = |\\hat{y}_1 - y_1|$ = |7 - 2| = 5\n",
        "* $|\\Delta y_2| = |\\hat{y}_2 - y_2|$ = |3 - 8| = 5\n",
        "\n",
        "In this case:\n",
        "\n",
        "* The total error is ten: $\\Sigma |\\Delta y_i| = |\\Delta y_1| + |\\Delta y_2| = 5+5$ \n",
        "* The **mean absolute error** (MAE) is five: $\\frac{\\Sigma{|\\Delta y_i}|}{n} = \\frac{10}{2}$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PRngCTzq5b15"
      },
      "source": [
        "#### Mean Squared Error"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ICZZqWyP5b15"
      },
      "source": [
        "While MAE satisfactorily quantifies cost across multiple instances, for reasons we'll numerate in a moment, it is much more common in ML to use **mean *squared* error** (MSE). \n",
        "\n",
        "Let's use the contrived values again to show how MSE is calculated. As covered in [*Calculus II*](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/4-calculus-ii.ipynb), the individual-instance variant of MSE is *squared error* or *quadratic cost*, which -- as with absolute error -- dictates that the cost for each instance $i$ must be $\\geq 0$: \n",
        "\n",
        "* $(\\hat{y}_1 - y_1)^2 = (7 - 2)^2 = 5^2 = 25$\n",
        "* $(\\hat{y}_2 - y_2)^2 = (3 - 8)^2 = (-5)^2 = 25$\n",
        "\n",
        "As suggested by its name, MSE is the average: $$C = \\frac{1}{n} \\sum_{i=1}^n (\\hat{y_i}-y_i)^2 = \\frac{1}{2} (25 + 25) = \\frac{50}{2} = 25 $$\n",
        "\n",
        "Since cost for each instance $i$ must be $\\geq 0$, MSE (as with MAE) must therefore also be $\\geq 0$. In addition: \n",
        "\n",
        "1. The partial derivative of the MSE $C$ can be efficiently computed w.r.t. model parameters, providing a *gradient* of cost, $\\nabla C$ (the primary focus of [*Calculus II*](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/4-calculus-ii.ipynb) is deriving $\\nabla C$). Adjusting model parameters, we can *descend* $\\nabla C$ and thereby minimize $C$ (the mechanics of which are the primary focus of *Optimization*).\n",
        "2. Compared to MAE, MSE is relatively tolerant of small $\\Delta y_i$ and intolerant of large $\\Delta y_i$, a characteristic that tends to lead to better-fitting models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WnLkt4505b15"
      },
      "source": [
        "delta_y = np.linspace(-5, 5, 1000)"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eS8MQfFj5b18"
      },
      "source": [
        "abs_error = np.abs(delta_y)"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2VEm2rY75b1-"
      },
      "source": [
        "sq_error = delta_y**2"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iFDqR0Xr5b2B",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "ca63427c-ca25-4fff-9445-a77d6e172e58"
      },
      "source": [
        "fig, ax = plt.subplots()\n",
        "\n",
        "plt.plot(delta_y, abs_error)\n",
        "plt.plot(delta_y, sq_error)\n",
        "ax.axhline(c='lightgray')\n",
        "\n",
        "plt.xlabel('Delta y')\n",
        "plt.ylabel('Error')\n",
        "\n",
        "ax.set_xlim(-4.2, 4.2)\n",
        "ax.set_ylim(-1, 17)\n",
        "_ = ax.legend(['Absolute', 'Squared'])"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3zV5fXA8c/JIoywkwACJiAJCRCGYQnKCktciC24itWK/hS1VmsdrbNWa11VWykqxVUcuCcyZIisgCyT4GAGIQmEEUbIen5/PDchhEDmvd87zvv14kXuzffe70kgJ8/3+T7nPGKMQSmlVOAIcjoApZRSnqWJXymlAowmfqWUCjCa+JVSKsBo4ldKqQAT4nQA1dG6dWsTExMDuZuh8AhEdwPE6bCUUsp7FB6FnAxo1h4aRwKwevXqPcaYyIqH+kTij4mJITU1FTI+g7eugEkPQ9dxToellFLe45Pfw7q9cMdaaNgCABHZVtmhvjXV02U0RLSF1TOdjkQppbzHsTzY8C50n1CW9E/HtxJ/cAj0vhp+nAv7tzsdjVJKeYf170DBIUi+tlqH+1biB+jzG/v3mtedjUMppbyBMZA6A9okwRl9qvUSn5jjP0HzDtBlJHz3Ogz5k70KUH6tsLCQzMxM8vPznQ7Fp4SHh9O+fXtCQ0OdDkW5U+YqyNoIFzwLUr1FL76ZNc/+Lbx1OfzwJSRc4HQ0ys0yMzOJiIggJiYGqeZ/7EBnjGHv3r1kZmYSGxvrdDjKnVJnQFgE9PhVtV/ie1M9AF1GQUQ7vckbIPLz82nVqpUm/RoQEVq1aqVXSf7uSC5sfB96ToQGTar9Mt9M/MEh0Odq+Gke7Kt0tZLyM5r0a06/ZwFg7f+g+JidBakBtyV+EZkhItkisrHC87eISIaIfC8iT9T6BH0mgwTZyxyllAo0pTd1O/SHNt1r9FJ3jvhnAmPKPyEiw4CLgZ7GmG7Ak7V+92ZnQPxYWPMaFOrlrHK/Dz/8EBEhIyMDgIULF3LBBXW/x3TNNdcwe/bs0x6zcOFCvv322zqfS/mRLYsh9+dqL+Esz22J3xizGMit8PT/AY8bY465jsmu00n6XQ9HcyHtwzq9jVLVMWvWLAYPHsysWbM8fm5N/OokqTNssVbiJTV+qafn+OOAc0VkhYgsEpG+pzpQRKaISKqIpObk5FR+UOwQaNUFVr7kpnCVsg4dOsQ333zDK6+8wltvvVX2/MGDBxk3bhzx8fHceOONlJSUUFxczDXXXEP37t3p0aMHzzzzDABr165lwIABJCUlMX78ePbt23fSeWJiYtizZw8AqampDB06lK1btzJt2jSeeeYZevXqxZIlS8jJyWHChAn07duXvn37snTpUs98I5R3yNsNGZ9CryshNLzGL/f0cs4QoCUwAOgLvCMinUwl+z8aY6YD0wGSk5Mr3x9SBPr+Dr78E/zyHbTr7b7IlVd46JPvSfvlYL2+Z2K7pjxwYbfTHvPRRx8xZswY4uLiaNWqFatXrwZg5cqVpKWlceaZZzJmzBjef/99YmNj2blzJxs32ttb+/fvB+A3v/kNzz//PEOGDOH+++/noYce4tlnn60yvpiYGG688UaaNGnCnXfeCcAVV1zB7bffzuDBg9m+fTujR48mPT29Lt8G5Uu+ex1Kimp8U7eUp0f8mcD7xloJlACt6/SOvS6H0Maw6uX6iE+pSs2aNYtJkyYBMGnSpLLpnn79+tGpUyeCg4O5/PLL+eabb+jUqRObN2/mlltu4csvv6Rp06YcOHCA/fv3M2TIEAAmT57M4sWLax3PvHnzmDp1Kr169eKiiy7i4MGDHDp0qO5fqPJ+xUWw+lWIPQ9an1Wrt/D0iP9DYBjwtYjEAWHAnjq9Y3gzSPo1rJsFIx+BRi3rIUzlraoambtDbm4uCxYsYMOGDYgIxcXFiAjjxo07acmkiNCiRQvWrVvHnDlzmDZtGu+8807ZdE9VQkJCKCkpATjtGvySkhKWL19OeHjNL/OVj/vhSziwA8Y8Vuu3cOdyzlnAMiBeRDJF5DpgBtDJtcTzLWByZdM8Ndb3d1CUD2vfrPNbKVXR7Nmzufrqq9m2bRtbt25lx44dxMbGsmTJElauXMmWLVsoKSnh7bffZvDgwezZs4eSkhImTJjAX//6V9asWUOzZs1o0aIFS5YsAeD1118vG/2XFxMTUzaN9N5775U9HxERQV5eXtnjUaNG8fzzz5c9Xrt2rbu+fOVtVv4HmraHuLG1fgt3ruq53BjT1hgTaoxpb4x5xRhTYIy5yhjT3RjTxxizoF5O1qY7dBwIq14B12hJqfoya9Ysxo8ff8JzEyZMYNasWfTt25epU6eSkJBAbGws48ePZ+fOnQwdOpRevXpx1VVX8dhjdmT26quv8sc//pGkpCTWrl3L/ffff9K5HnjgAW677TaSk5MJDg4ue/7CCy/kgw8+KLu5+9xzz5GamkpSUhKJiYlMmzbNvd8E5R2yM+wyzr7X1qlPmdTHgNvdkpOTTWpq6ukP2jAb3rsOrnwPuqR4JjDlEenp6SQkJDgdhk/S752f+ewO25n4D2nQuOrboyKy2hiTXPF532zZUJmEi6BxlN7kVUr5p/wDsHaW3WylGkn/dPwn8YeEwdmT7Y0P7d+jlPI3696CwsO2cLWO/Cfxg13Tqv17lFL+pqQEVk6HM5KrvdnK6fhX4i/t3/Pd69q/RynlPzZ/DXt/gn5T6uXt/Cvxg70MOrIXvv/A6UiUUqp+rHwJGkdCt5r35amM/yX+2CHQOt6udfWBFUtKKXVa+7bae5dnXwMhDerlLf0v8YtA/ym2d8+OFU5Ho/zEo48+Srdu3UhKSqJXr16sWOH8/62ZM2cydepUp8NQ7rbqFXvvspZ9eSrjm3vuVqXn5TD/YVj+InQc4HQ0ysctW7aMTz/9lDVr1tCgQQP27NlDQUGB285XXFx8QvGWCmAFR+yeIwkX2HuY9cT/RvwAYY3tDl3pn8CBTKejUT5u165dtG7dmgYN7GV269atadeuHV9++SVdu3alT58+3HrrrWWbsjz44IM8+eTxPYa6d+/O1q1bAbjkkks4++yz6datG9OnTy87pkmTJtxxxx307NmTZcuW8cYbb9CvXz969erFDTfcQHFxMQD//e9/iYuLo1+/ftqKORBsnA35++vtpm4p/xzxg73Ju+wFe1Nk5ENOR6Pqyxd3w+4N9fuebXrA2MdP+elRo0bx8MMPExcXR0pKChMnTqR///5cf/31LFiwgLPOOouJEydW61QzZsygZcuWHD16lL59+zJhwgRatWrF4cOH6d+/P0899RTp6en8/e9/Z+nSpYSGhnLTTTfx5ptvMnLkSB544AFWr15Ns2bNGDZsGL17aytyv2UMrJgOUYlw5qB6fWv/HPEDNO8IXS+A1TPt5ZJStdSkSRNWr17N9OnTiYyMZOLEiUybNo3Y2Fi6dOmCiHDVVVdV672ee+45evbsyYABA9ixYwc//vgjAMHBwUyYMAGA+fPns3r1avr27UuvXr2YP38+mzdvZsWKFQwdOpTIyEjCwsKq/ctG+ait30DWBuh/o713WY/8d8QPMOD/IP1jWP82JNffjRHloNOMzN0pODiYoUOHMnToUHr06MGrr756ymPLt1aG4+2VFy5cyLx581i2bBmNGjVi6NChZZ8LDw8vm9c3xjB58uSy5m6lPvxQtxgNKMv/DY1a2bbz9cx/R/xgO3a2SYIVurRT1d6mTZvKRuZgWyBHR0ezdetWfv75Z4AT9uGNiYlhzZo1AKxZs4YtW7YAcODAAVq0aEGjRo3IyMhg+fLllZ5vxIgRzJ49m+xsuyV1bm4u27Zto3///ixatIi9e/dSWFjIu+++65avV3mBvT/Dpi/sRuqhDev97f078YvYUX9OOmxe6HQ0ykcdOnSIyZMnk5iYSFJSEmlpaTz++ONMnz6dcePG0adPH6KiosqOnzBhArm5uXTr1o0XXniBuLg4AMaMGUNRUREJCQncfffdDBhQ+YqzxMRE/vrXvzJq1CiSkpIYOXIku3btom3btjz44IMMHDiQQYMGaddNf7ZyOgSF2L1G3MBtbZlFZAZwAZBtjOle4XN3AE8CkcaYKnfgqlZb5lMpOgbPdIMzzoYr3q7deyhH+UJr4YULF/Lkk0/y6aefOh3KCXzhe6cqyD8ATyfae5SX/qdOb+VEW+aZwJhKAukAjAK2u/Hcx4U0sJdLP8yxl09KKeXN1rwOBYfsbIWbuHMHrsVAbiWfega4C/DcpHvydfayaeX0qo9VqhaGDh3qdaN95YOKi+w9yTMHQbtebjuNR+f4ReRiYKcxZp0nz0tENHS/FL57E/IPevTUqn74wk5x3ka/Zz5o02dwYLtbR/vgwcQvIo2Ae4GTNxqt/PgpIpIqIqk5OTl1D6D/jVCQB9+9Uff3Uh4VHh7O3r17NZHVgDGGvXv3Eh4e7nQoqiaW/Ruanwnx57v1NJ5cx98ZiAXWiS1GaA+sEZF+xpjdFQ82xkwHpoO9uVvns5/Rxy7vXPGiLX+uw0bFyrPat29PZmYm9TIACCDh4eG0b9/e6TBUde1cDTuWw+jHIMi9vZo8lv2MMRuAsjVvIrIVSK7Oqp56M3AqvH2lLerqfqnHTqvqJjQ0lNjYWKfDUMq9lr8IYRHQu3pV4HXhtqkeEZkFLAPiRSRTRK5z17mqLX4stOwM3z6vBV1KKe9x8Be7eVSf30B4U7efzp2rei43xrQ1xoQaY9obY16p8PkYj472wV4+DbwZflkD25d59NRKKXVKK18CU2L3EvEA/67crUzPy6FhSzvqV0oppx07BKkzoOs4aBHjkVMGXuIPa2RbNm/6Avb8WPXxSinlTt+9YXvun3Orx04ZeIkfoO/1EBwGy/7ldCRKqUBWXATL/wUdBkCHfh47bWAm/iaR0HMSrJsFhz17m0EppcqkfQj7t8Og2zx62sBM/GCXdhblw6qXnY5EKRWIjIGl/4TWcRB3UlsztwrcxB/p+mavnA6FR52ORikVaLYsgt3r7SA0yLOpOHATP8A5t8CRvXbKRymlPGnpc9A4CpI8v4VmYCf+MwdB2172Jm+5rfKUUsqtdm+En+dD/xsg1PP9lAI78YvYUf/en+CHL5yORikVKL59HkIbQ19nGhoEduIHSLwEmnWEb57VNg5KKfc7kAkbZ9v2DA1bOBKCJv7gEBh0K2SuhG3fOh2NUsrfLX/RDjIH3uRYCJr4AXpdCY1awzfPOB2JUsqfHd0Pq2fa7sDNOzoWhiZ+sG0cBvwf/DQXdq13OhqllL9a/V+7n64H2zNURhN/qb6/s72wddSvlHKHwnw7zdNpKLRNcjQUTfylGjaHvtfaEuq9PzsdjVLK36x9Aw5lwbl3OB2JJv4TDLgJgkLh2+ecjkQp5U+KC217hvZ9IeZcp6Nx6w5cM0QkW0Q2lnvuHyKSISLrReQDEWnurvPXSkQb6HUFrP0f5J20DbBSStXOxvdsM7Zz77D1Qw5z54h/JlCx89BcoLsxJgn4AbjHjeevnUG3QkmRtmxWStWPkhJY8jREdYMuo52OBnDv1ouLgdwKz31ljClyPVwOtHfX+WutZSfoNt7uiHN0n9PRKKV83abPYM8mOPcPHm/GdipORnEtcMo+CSIyRURSRSQ1JyfHg2EBg2+3S660ZbNSqi6MgcVPHh9QeglHEr+I3AcUAW+e6hhjzHRjTLIxJjkyMtJzwQG06QFdRtmlVwVHPHtupZT/+HkB7FoLg34PQcFOR1PG44lfRK4BLgCuNMaLm+MMvt22bF7zmtORKKV81ZKnIaKd3fHPi3g08YvIGOAu4CJjjHcPpc88x7ZtXvpPKDrmdDRKKV+zfTls+8Z2AA5p4HQ0J3Dncs5ZwDIgXkQyReQ64AUgApgrImtFZJq7zl8vzvsj5P0C373hdCRKKV+z5Glo1ArOnux0JCcJcdcbG2Mur+TpV9x1PrfoNBTa97NtHHpfDSFhTkeklPIFu9bDj3Ng+J8hrLHT0ZzEO9YWeSsRGPInOLBDt2dUSlXf4iegQTPoe73TkVRKE39VzhoB7frAkqds2bVSSp3O7o2Q/gkMuNH2APNCmvirUjrq378N1r/jdDRKKW+3+Alo0NS2evdSmvirI240tEmCJU9CcVHVxyulAlNWGqR9ZDdRd2hbxerQxF8dpaP+3M222ZJSSlVm0d/tvh4DnNtWsTo08VdX/PkQ3d2O+kuKnY5GKeVtstOPj/YbtXQ6mtPSxF9dQUF2Xf+eH+xmLUopVd6iJ+zSzYE3Ox1JlTTx10TCRRDZFRb9w7ZaVUopgOwM+P4D6DfF60f7oIm/ZkpH/TnpkP6x09EopbzF4icgtBEMnOp0JNWiib+muo2H1vGw8DGd61dKQc4m2Pg+9J8CjVs5HU21aOKvqaBgGHYP5GTYf2ylVGBb/A/XaP8WpyOpNk38tZFwsV3hs/Bvuq5fqUCWnQ4bZkO/3/nMaB808ddOUBAMu8+u69cePkoFrq8fhbAmdqMVH6KJv7bix9oePouegKICp6NRSnnaL9/ZnjwDb/aJlTzlaeKvLREYfh8c2A7f6S5dSgWcBX+1bRl8YN1+Re7ciGWGiGSLyMZyz7UUkbki8qPrb+9tZlEdnUdAx4F2M+XCo05Ho5TylG3L4Kd5doonvKnT0dSYO0f8M4ExFZ67G5hvjOkCzHc99l0idqOFvF2QOsPpaJRSnmAMLHgEmkTbgi0f5LbEb4xZDORWePpi4FXXx68Cl7jr/B4TMxhih9ht1o4dcjoapZS7bf4ati2Fc++EsEZOR1Mrnp7jjzbG7HJ9vBuIPtWBIjJFRFJFJDUnJ8cz0dXW8D/DkT2wcrrTkSil3MkYmP8INOvglXvpVpdjN3eNMQYwp/n8dGNMsjEmOTIy0oOR1UKHftBlNCz9Jxzd73Q0Sil32fQ5/LIGhtwFIQ2cjqbWPJ34s0SkLYDr72wPn999hv8Z8vfb5K+U8j8lJbDgUWjZGXpe4XQ0deLpxP8xUHp9NBn4yMPnd5+2SdDjV7D8RTi4q+rjlVK+5fv3Ift7GHYvBIc4HU2duHM55yxgGRAvIpkich3wODBSRH4EUlyP/cew+6CkCBb515elVMArKrAreaK6QbdLnY6mztz2a8sYc/kpPjXCXed0XMtY6HsdrHwJBtwMkXFOR6SUqg+pM2DfVrhytm3Z4uN8/yvwNufeCaENYcHDTkeilKoP+Qdtv/3Y8+CsFKejqRea+Otbk0g451bbw2PHKqejUUrV1dJ/wpG9MPJhW7TpBzTxu8PAm6FxJMx70K77VUr5poO7YNm/oPsEaNfb6WjqjSZ+d2jQBIb8CbZ9Y/t5KKV808LH7IKN4X9xOpJ6pYnfXfpMhhaxdtSvG7Mr5XtyNsF3r9sFGy1jnY6mXmnid5eQMBjxF8jaCBvecToapVRNzXsIQhvDeX90OpJ6p4nfnRLHQ9tetrdHwRGno1FKVdf25bDpMxh8GzRu7XQ09U4TvzsFBcHov8HBTFj+L6ejUUpVhzEw935o0gYG3OR0NG5RZeIXkSAROccTwfilmEGQcCEseQbyspyORilVlbSPYMcKGHYPhDV2Ohq3qDLxG2NKAB2u1kXKQ1DsKvlWSnmvwnyY+xeI7g69r3Y6Grep7lTPfBGZIOIn1Que1qoz9L8BvnsDdm9wOhql1Kks/zfs326naIOCnY7Gbaqb+G8A3gUKROSgiOSJyEE3xuV/zrsTGjaHOfdqUZdS3igvC5Y8BfHnQ6chTkfjVtVK/MaYCGNMkDEm1BjT1PXY93YYdlLDFjD0HtiyGH6Y43Q0SqmKFjwCRcdg1F+djsTtqr2qR0QuEpEnXX8ucGdQfiv5WmjVBb76MxQXOh2NUqrUrvV2Krb/DXZq1s9VK/GLyOPAbUCa689tIvKYOwPzS8GhdjSx90fb5lUp5Txj4Mt77FW5HxZrVaa6I/7zgZHGmBnGmBnAGGBcbU8qIreLyPcislFEZolIeG3fy+fEjYbYIbYHyJFcp6NRSmV8avtqDbvX3ocLADUp4Cr/HWlW2xOKyBnArUCyMaY7EAxMqu37+RwRu2Ig/4BN/kop5xQds1OvkV3h7N86HY3HVDfx/w34TkRmisirwGrg0TqcNwRoKCIhQCPglzq8l+9p0x2Sr4NVL+vyTqWctGKa3Vlr9N98fh/dmqhW5S5QAgwA3gfeAwYaY96uzQmNMTuBJ4HtwC7ggDHmq0rOO0VEUkUkNScnpzan8m7D7oXw5vD5Xbq8UyknHNwFi56ALqPhLP/dEbYy1a3cvcsYs8sY87Hrz+7anlBEWgAXA7FAO6CxiFxVyXmnG2OSjTHJkZGRtT2d92rUElIegO3fwobZTkejVOCZe7+tqB8TeFOu1Z3qmScid4pIBxFpWfqnludMAbYYY3KMMYXYq4jA7AXU+2rbvXPuX+BYntPRKBU4ti617dIH3RYQyzcrqm7inwjcDCzGzu+vBlJrec7twAARaeRqATECSK/le/m2oGA4/0nI2wWL/+F0NEoFhuIi+PyP0KwjDP6D09E4orpz/HcbY2Ir/OlUmxMaY1YAs4E1wAZXDNNr815+oUNf6HUlLPs37PnR6WiU8n+rXoLs72HM3yCskdPROKK6c/z1WtVgjHnAGNPVGNPdGHO1MeZYfb6/z0l5EEIbwhd/0hu9SrlTXhZ8/TfoPBy6Bm4DAifm+FVFTaJsH5+f58Omz52ORin/Ne9BKDwKY/9ha2oClBNz/Koy/a6HyAT44m4oOOx0NEr5n+0rYN3/4Jyp0Posp6NxVHW7c1ac36/1HL86heBQuOBpOLAdFv3d6WiU8i/FRfD5HdD0jIDpx3M6p038InJXuY9/VeFzf3NXUAHrzHOg91Ww7F+Q9b3T0SjlP1a8aKvkxzzmt9sp1kRVI/7yPXTuqfC5MfUciwIY+QiEN4NPboOSEqejUcr37d9ub+jGjYWEi5yOxitUlfjlFB9X9ljVh0YtbevmzFWwZqbT0Sjl24yBz+4EBM4P7Bu65VWV+M0pPq7ssaovPS+HmHPtCoRD2U5Ho5TvSvsIfpwDw++D5h2cjsZrVJX4e5busQskuT4ufdzDA/EFJhEY97RddjbnXqejUco35R+wtTFte0K/G5yOxqucNvEbY4LL7bEb4vq49HGop4IMSJFxMPh22PAu/LzA6WiU8j3zH4bD2XDhPwOq5XJ11GQjFuVpg/8ALTvDp3+AgiNOR6OU79ixCla9Ykf67Xo7HY3X0cTvzULD4cJnYd8WWKirZ5WqluJC+PT30LSdndtXJ9HE7+1iz4Ozr7Fr+zO1WFqpKn3zDGRttJ1vG0Q4HY1X0sTvC0Y+DE3awEc32z1ClVKVy/re7qrV/TLoer7T0XgtTfy+ILyZnfLJyYDFTzodjVLeqbgIPrzJ/ryMfcLpaLyaJn5fETcakibBN0/rBu1KVebb52DXWhj3JDRu5XQ0Xs2RxC8izUVktohkiEi6iAx0Ig6fM+YxaNjCjmqKC52ORinvkbMJFj5uWzJ0G+90NF7PqRH/P4EvjTFdgZ4E6taLNdWoJYx7Cnavt6MbpRSUFNv7X2GN7M+HqpLHE7+INAPOA14BMMYUGGP2ezoOn5V4sR3VLHwcsvX3pVIsf9H2thr7hN3USFXJiRF/LJAD/FdEvhORl0XkpD6pIjJFRFJFJDUnJ8fzUXqzcU/ZZWrvT4GiAqejUco5e36CBY/Yzps9flX18QpwJvGHAH2AF40xvYHDwN0VDzLGTDfGJBtjkiMjIz0do3drEmXL0Hevh8W6ekEFqOIi+GAKhITDBc9o580acCLxZwKZxpgVrsezsb8IVE0kXGi7eC55ypanKxVoljwFO1fbpN+0rdPR+BSPJ35jzG5gh4jEu54aAaR5Og6/MPbvENEOPrhB9+lVgWXnartFaY9fQ/dLnY7G5zi1qucW4E0RWQ/0ArQRTW2EN4NL/g25P8PcB5yORinPKDhi729FtLWbq6gac6RXqTFmLZDsxLn9TqchMOAmWP5viB8LZ41wOiKl3Gvu/bD3J5j8CTRs7nQ0Pkkrd/3BiPuhdbxdy3wk1+lolHKfH+fBqpdg4FTbwFDViiZ+fxDaEC79DxzOsZu0G90VU/mhI7l2cBOZAMP/4nQ0Pk0Tv79o19v+MKR/DKv/63Q0StUvY+CjqXBkL1w63e5VoWpNE78/OedW6DwcvrxHq3qVf1n5Emz6DEY+BG2TnI7G52ni9ydBQXDJNFvV++5v7WbtSvm6Xevhq/ugy2i7kEHVmSZ+fxMRDeOnQU46zNFt55SPO3YIZl8LjVrZpctanVsvNPH7o7NS4JxbIPUVSPvY6WiUqr0v7rJLNy99CRq3djoav6GJ318Nvx/a9YGPp8L+HU5Ho1TNrXsb1r4JQ+6C2HOdjsavaOL3VyFhcNkrUFICs3+rXTyVb9nzE3z2B+h4Dpx3l9PR+B1N/P6sZSe4+AXbq/yrPzsdjVLVU3AY3r4KgsNgwksQ7EiDAb+mid/fdbsEBtwMK/8DG2Y7HY1Sp2eMLULMybBXrM3aOx2RX9LEHwhGPgQdBsDHt0J2htPRKHVqq16GDe/C8PtsTYpyC038gSA4FH410+5J+s7VcCzP6YiUOtmOVbb4MG4MDL7D6Wj8mib+QNG0LVw2wy6N+/gW7eejvMvhPfDOb6BpO1uHEqSpyZ30uxtIYs+znTy//wCWveB0NEpZxUW2SOtoLkx8HRq2cDoiv+fY7XIRCQZSgZ3GmAtOd+yOfUf4bP0uzotrTUR4qGcC9FeDfg8719ie5pEJ0CXF6YhUoJv7F9iyCC7+F7Tt6XQ0Pm9H7hHmp2cxPyP7lMc4uU7qNiAdaFrVgXn5Rdz8vzWEBgsDOrUiJSGaEQlRtG/RyP1R+hsRuORFmLHZjrKunw+tuzgdlQpUa163mwj1/z/ofZXT0fikkhLD+p0HmJeWxbz0LDJ223t4nSIbn/I1YhyY6xWR9sCrwKPAH6oa8ScnJ5sXZ39V9oX9nGP3l+3aJoKUhGhSEqNJOqMZQUHax6Pa9m2Dl4ZBw5Y2+Yc3czoiFWi2L4eZF0DMYLhytq7Xr4GjBcUs/WkP81wj+5y8YwQHCZAh6A8AABdiSURBVMlntigbGHeKbIKIrDbGnLTboVOJfzbwGBAB3FlZ4heRKcAUgI4dO569bdu2ss9tzjnE/PRs5qVnsWprLiUGIiMaMKJrFCkJ0Qw6qzUNw4I99NX4sK1L4bWLoNMwuOJtCNLvmfKQ/TvswKNBBFy/QOf1qyH7YD7zM7KZn57Fkh/3cKyohCYNQhgSH8nIhGiGxkfSvFHYCa/xmsQvIhcA5xtjbhKRoZwi8ZeXnJxsUlNTK/3cvsMFLPwhm3np2SzalMOhY0U0CAni3C6tSUmIZnhCFFERumnDKa16xZbGn3MrjHrE6WhUICg4DDNG26vO382HyDinI/JKxhgydufZmY6MbNbt2A9A+xYN7UxHQjT9YlsSFnLqNTqnSvxOXFsNAi4SkfOBcKCpiLxhjKnVBF+LxmGM792e8b3bU1BUwsotucxLz3L9sTc3enZoTkrXKFISo+naJgLR1q7H9b0Osr6Hb5+zc/19fuN0RMqflZTABzfC7o1w5bua9CsoKCphxZa9rmntbHbut3tq9OrQnD+OjmdEQhTx0XXPYY5M9ZSdvB5G/KdijGFTVl7ZN3Ct67flGc0bkpJgfwn0j2112t+WAaO4EP73a9i8yP4wnjXC6YiUv5pzn11KPOpROGeq09F4hX2HC/h6Uzbz07NZ9IOdtQgPDWLwWZGMTIxiWNfaz1p4zVTPCSd3Y+KvKDsvn68zspmbls03P+WQX+iaH4uLZERCFMPio2jROKzqN/JX+Qfhv2Pt5fe1X0Kb7k5HpPzNiv/Y/vr9psDYJwJ6U5XNOYfsrERaNqnb7H3KqIgGjEg4fp8yPLTu99y8MvFXV30k/vLyC8vdEU/PJjvvGEECyTEt7dVAQjSdIpvU2/l8xoGd8HKK/YH83TxbRalUfcj4DN66EuLHwsQ3Am4hQVFxCau37WN+Rjbz0rLYvMeuTExo25SRCVGMSIimhxtWJmriP4WSEsOGnQfK7gmk7zoIQKfWjUlJtDdQ+nRsTkhwgEwJ7d4AM8ZAi1i49gu76kKpushcDTPHQXQiTP7U9owKAHn5hSz+wQ4wv96Uzf4jhWW1SCMToxne1f21SJr4qylz3xEWZGQzNy2L5Zv3UlhsaN4olOHx9rdyQFQP/zQP3vw1dBoKl79lN3VRqjZyt9iryLDGdgVPk0inI3KrzH1Hypaal+aPFo1CGRZv7yue28Wz+UMTfy3k5Rey5Mc9zEvLYkGF39h+Xz285nW7bWO3S2HCywF3aa7qQd5uu2wz/wBcN9cvK8RLq2bnp2cxN+3EqtmRCdGMcHjGQBN/HRUVl7Bm+377D5yexeZy1cMjXVNC7pijc9Q3z8K8ByD5Whj3dEDfjFM1dCTXTu/s2waTP4b2J+Uen1VZ1WzpPcKR5apmvYEm/npWWj08Nz2L1HLVwykJUYzo6kfVw3MfgKXPwrl32M6eSlXl2CF47WLYvd4uD+401OmI6iw7L58Frimcb37ac3xVYHwkKQlRDI3zzlWB3lTA5Rc6RTahU2QTrj+v0/Hq4bRsPlm3i1krd7jW4fpB9XDKg3B0Hyx5ypbVn3OL0xEpb1Z0DN66An5ZA79+zWeTfmnVrL3CP141e0bzhkzq25ERCVE+XQekib8eVKweXrFlr70aSDuxerh02ZZPVQ+LwAXP2Hnar/4MDZrC2ZOdjkp5o9K++lsW2Q6wCRc6HVGNnKpqtmeH5tw5Ko6UxOh6qZr1BjrV40blq4crjhpGJkb71qih6BjMuhx+XmD7pve+0umIlDcpKYYPbrD75Y55HAb8n9MRVUv5q/WKVbMpCVEM7xpFVFMfvVpH5/i9QvbBfBZkVDJPGBdJSqL3zhOWKTxqk//mhXDJv6HXFU5HpLxB+aQ/4n57P8iLlVXNpmefdH8uJSGaczr7yf05NPF7ndKVAfMz7H9Ab14ZcILCozBrku3rc8mL0OtypyNSTvKBpF+6Iq+0eWPpiryEtk3Lkr3frchz0cTvxcpXD1dcC1zaftWrqocLjtjkv2Wx3Ri75ySnI1JO8OKkH9A1OOVo4vch3lb9V6mCIzBrImxZonP+gai40LZX3jjba5J+ZT83AVd1X4Emfh9V2u9jfvrJIxd7gziaM5o3dCa4giPwlmvO34du6Kk6KjwK714DP3xpl/sOvt2RMHzuStkBmvj9wOk6/Dk2V1l0DN67DtI/gSF3w9C7tcLXn+UftDf4ty2FcU9C39959PSnuzeW4lou3dkb7405RBO/H/o55xDzT7M6ob56elepuAg+uRXWvgn9b4TRj0FQ4I6y/NbhvfDGpZC1Ecb/B3pc5pHTHq+aPXkvDZ9YDecgr0n8ItIBeA2IBgww3Rjzz9O9RhN/1apcj+zu6uGSEvjqPlj+b0iaBBc9r109/cn+7fDGZbB/m63IjRvttlOdrmpWd8+rGW9K/G2BtsaYNSISAawGLjHGpJ3qNZr4a+Z0+3aW/uC4pQLRGFj8D/j6UYgdYhNEw+b1ew7leb98B/+bCIX5cPn/IGZwvZ+iYsV7+apZn6x49xJek/hPCkDkI+AFY8zcUx2jib/2SkdP89KymJdxcvVwSkI0/WJb1u/oae3/4ONboFUXuPIdaN6x/t5bedamL2H2b6FRK9twLSqh3t56/xG71+y89GwWb8ohr+wq1dXjyserZr2BVyZ+EYkBFgPdjTEHT3WcJv76k30wn/kZ2cxPz2LJj3s4VlRCRIMQznN1GRwWH0XzRvUwRbN5Ebx9NYSGwxVvQ7vedX9P5VkrX7J75LZJgivegYjoOr9l+a62q7fto7jEEBnRgBFdj9+X8peqWW/gdYlfRJoAi4BHjTHvV/L5KcAUgI4dO569bds2D0fo/yrrKx4cJCSf2cIuh0uMJrZ149qfIDsd3vwVHNlrq3y7XVJ/wSv3KS60DflWTIO4sXDZK3YHrVo43T4Wpf/Hkvy0atYbeFXiF5FQ4FNgjjHm6aqO1xG/+5XuJGTvC5y8k1BKYjR9OrYguKY/oHlZ8PaVkLnKFvkMu0938/Jmh/fAO5Nh2zcw4CYY9dca/3sdOlbE4h9ymJdm95rdF6BVs97AaxK/2LszrwK5xpjfV+c1mvg9b0fukbKGcidUD7suyc+Li6RJg2p29S46Bp/fCWteg7NG2q0c9aav9/nlO3jrKjiyBy78Z41acezcf7Rs+8EVm3MpKC6heWm1eYBWzXoDb0r8g4ElwAagxPX0vcaYz0/1Gk38ziqtHp6XnsWCjGwOHC0kLDiI/p1aVr962BhInQFf/Amad4CJb0J0ome+AFW1dW/BJ7dBo9Yw6Y0q78mUVs2WLrlM32Vv0XVq3ZiUxGhGdI3i7DNbBHTVrDfwmsRfG5r4vUdp9XBpW9st5aqHS5fdnbZ6ePtye9P3WB6M/Tv0+Y1W+jqp4DB8fhesfQPOHAy/mglNIis9NL+w3D2h9GyyS6tmz2xJSqJWzXojTfzKLcqqh9OySd1mq4ejIhowIiGalISoyquH87Lggym2x0+38XZaIbyZI/EHtN0b7I5Ze36E8+60LTeCT5y+y87L5+uMbOamnVw1O8K1CkyrZr2XJn7ldvsOl67LzmLRphwOFxSXVQ+PTIxiWNdy1cMlJbD0GVjwKDRrD5fNgPYn/f9U7mAMrHoZ5txn77Vc+hJ0GuL61PFd4+alZ7O2QtXsiIRo+ndqSYMQvUHvCzTxK486VlTMis25ZdMC5auHS7edjI+OQHashPd+Bwd3wrl/gPPu0lYP7nTwF/j4VvhpLpyVApdMoyC8FSu35JZtVJK5z1U1275Z2ZJLrZr1TZr4lWOMMaTvynM1lMtiXeYBANq3aEhKQjSjO4XTb9MTBG94C6K62W0d2/VyOGo/Y4y9gfvFn6C4gCND/sycxhcxL2NPWdVsg5Agzu3SmhEJ9uasVs36Pk38ymtkle49nGb3Hi6tHr7pjB+5Zu+zhBfkIufeYeedQxo4Ha7vO7ATPrsDfviC3c1687ewW/hsZ0OKSwytmzQom8IZrFWzfkcTv/JKRwuK+eanPWXtpQsP7eXB0NcZH7yE/Y1iODrycdr2Hut0mD6puLCAnXOeoc2aZygpKeaJwonMLB5NXJtmWjUbIDTxK69XUmJYl7mf+enZ7F//Gb/Lm0ZMUBZfhwzi++5/on+vHrWrHg4gpVWzP6+ey5itT9CFHSwo6cOnZ9xGUveejEiIpkNLrZoNFJr4lc/ZkZ3Lni+foNvmVyg0wotFF/Fe2EUMTOjIyIRozq1J9bAfK62anZeezc6f0/h90CwuDF5Obkg0Pyc/QNehv9aq2QCliV/5rtwtFH5xL6E/fs6BkFY8W3QZr+UPJjg4lAGdW5XNUTu297CHVVY125w87ov4lEuLvoDgUGTgVILOvb3WzdWUf9DEr3zf9uXw1V8gcyVHm3Xmy1aTeSGrOz/vzQcgsXTv4cRourfzr7nr41WztqV2adXskA6h3NxoHr1/mUVw4SHofRUMvReatnU6ZOUFNPEr/2AMZHwK8x+BPZug1Vlk97yJj4oH8dWmXFZv23dC9fDIxCjO6eyhvYfrWU7eMRZkZJ1QNds4LJgh8ZGc3ymMEfvfpeF3M6AgD+LHwfA/a/8jdQJN/Mq/lBRD+iew+EnI2gDNOkL/G9gX/2sWbC1gfsbx6uGGocEM7tLa7j3cNZrICO9cIlpaNVu6/eC6zP0YY6tmRyTYLpcDmuYQlvoyrJsFhUch8WK77LVND6fDV15IE7/yT8bAj1/BN8/A9mUQ0hCSfgX9pnCsdSLLN+e6egll8cuBfERK9x62207GRTdxtCK1oKjklFWzI1wxJkQ3RH6aByv+A5u/huAw6H4ZDLoNoro6Frvyfpr4lf/btR5WvQTr34Wio9C2FyRNhB6XYRpHkr4rryzBrq9QPTwyMZq+MfW89/Ap7D9SwMJNOcxNzzqhanbwWa1JSbR7zUY3DYfdG+3IfsO7cCgLItpB3+vg7GugcWu3x6l8nyZ+FTiO7oO1s2D9W7BrHUgwdB4OCRdC3GiIaEPWwXzmu26Ulq8eHhIfSUpCNEPjI+tn72GXLXsOl21Ukuraa7Z1E9des4muqtnQIMjaCBmfQ/rH9uOgUBtzz0kQNwaCdVmmqj5N/CowZWfAhnfsqHn/dvtcu952L9nY8+CMPhwpCWbpT3uZl5bF/Iws9hwqKNt7uHSjmZruPVxcYlizfV/ZVpY/V9hrdkRCFD3bNycofx9s+xa2LIJNX8KB7YBA+76Q9Gvodik0blXP3xQVKLwq8YvIGOCfQDDwsjHm8dMdr4lf1ZkxkJ0Gm76AH76EzFTAQHADOONsOHMgtO1JSWQ31h1pwbyMHOanZ5ftPdw50u4slZJw6r2HDx0rYskPdgrn6wy712xIUOles1GM6NKMDkXb7BTOrrU24Wen2ReHNITOwyB+rB3ZN4ny4DdH+SuvSfwiEgz8AIwEMoFVwOXGmLRTvUYTv6p3R3LtzeBt39o/u9aBKbafC20EkV2hRQwHw9vy/eEIluU0ZMXuEvaXNCSkYTN6d+nIkLhIOkc1YdWWvSzalM2mrZmElxyiXYNjnNMuiP6tjnBWaC5hh3bCvi12w5OyczSGDv0gZpDd+eqMPtqQTtU7b0r8A4EHjTGjXY/vATDGPHaq1/To0cN89NFHHopQBSRTAoVH7FaEBUeg8LDdJL7oGMe3hq4FCbEJPSQcwhrZStqwJvaxUm7WuXPnShO/E41OzgB2lHucCfSveJCITAGmAMTHx3smMhW4JMgm5LBK9owtLoTiY1BSZOsHXH/nFxVTVGxoEBJEaEgQBIXYRB/k+hMSZv9Wyst47f9KY8x0YDrYqZ5OnTo5HJFSSvkH9y9aPtlOoEO5x+1dzymllPIAJxL/KqCLiMSKSBgwCfjYgTiUUiogeXyqxxhTJCJTgTnY5ZwzjDHfezoOpZQKVI7M8RtjPgc+d+LcSikV6JyY6lFKKeUgTfxKKRVgNPErpVSA0cSvlFIBRhO/UkoFGE38SikVYHyiH7+I5ADbTnNIa2CPh8KpDY2v9rw5NtD46krjq5uq4jvTGBNZ8UmfSPxVEZHUyjrQeQuNr/a8OTbQ+OpK46ub2sanUz1KKRVgNPErpVSA8ZfEP93pAKqg8dWeN8cGGl9daXx1U6v4/GKOXymlVPX5y4hfKaVUNWniV0qpAON3iV9E7hARIyKtnY6lPBF5RETWi8haEflKRNo5HVMpEfmHiGS44vtARJo7HVN5IvIrEfleREpExGuW1onIGBHZJCI/icjdTsdTnojMEJFsEdnodCyVEZEOIvK1iKS5/m1vczqm8kQkXERWisg6V3wPOR1TRSISLCLficinNX2tXyV+EekAjAK2Ox1LJf5hjEkyxvQCPgXudzqgcuYC3Y0xScAPwD0Ox1PRRuBSYLHTgZQSkWDgX8BYIBG4XEQSnY3qBDOBMU4HcRpFwB3GmERgAHCzl33/jgHDjTE9gV7AGBEZ4HBMFd0GpNfmhX6V+IFngLsAr7tjbYw5WO5hY7woRmPMV8aYItfD5dh9kL2GMSbdGLPJ6Tgq6Af8ZIzZbIwpAN4CLnY4pjLGmMVArtNxnIoxZpcxZo3r4zxsAjvD2aiOM9Yh18NQ1x+v+ZkVkfbAOODl2rzebxK/iFwM7DTGrHM6llMRkUdFZAdwJd414i/vWuALp4PwAWcAO8o9zsSLEpcvEZEYoDewwtlITuSaSlkLZANzjTHeFN+z2EFuSW1e7MjWi7UlIvOANpV86j7gXuw0j2NOF58x5iNjzH3AfSJyDzAVeMBbYnMdcx/2EvxNT8VVqjrxKf8jIk2A94DfV7gqdpwxphjo5brn9YGIdDfGOH7PREQuALKNMatFZGht3sOnEr8xJqWy50WkBxALrBMRsFMVa0SknzFmt9PxVeJN7J7DHkv8VcUmItcAFwAjjAPFHTX43nmLnUCHco/bu55T1SQiodik/6Yx5n2n4zkVY8x+Efkae8/E8cQPDAIuEpHzgXCgqYi8YYy5qrpv4BdTPcaYDcaYKGNMjDEmBnvZ3ceTSb8qItKl3MOLgQynYqlIRMZgLxsvMsYccToeH7EK6CIisSISBkwCPnY4Jp8hdoT2CpBujHna6XgqEpHI0tVtItIQGImX/MwaY+4xxrR35bpJwIKaJH3wk8TvIx4XkY0ish47JeVNy9deACKAua7lptOcDqg8ERkvIpnAQOAzEZnjdEyum+FTgTnYG5PvGGO+dzaq40RkFrAMiBeRTBG5zumYKhgEXA0Md/2fW+sawXqLtsDXrp/XVdg5/hovm/RW2rJBKaUCjI74lVIqwGjiV0qpAKOJXymlAowmfqWUCjCa+JVSKsBo4lcBS0SKXcsIv3d1YbxDRE77MyEiMaUdL0Wkl5ctQVSqWnyqclepenbU1S0VEYkC/gc0pfoV1b2AZGwVtlI+Q0f8SgHGmGxgCjBVrGDXPgWrXPsU3FD+eFe17sPARNdVw0QR6Sciy1w90r8VkfiK5xGR10TkknKP33Q1GFTKY3TEr5SLMWazq89+FLatxgFjTF8RaQAsFZGvcLXmNcYUiMj9QLIxZiqAiDQFzjXGFIlICvA3YEKF07wC3A58KCLNgHOAyZ74+pQqpYlfqcqNApJE5DLX42ZAF+xGNafSDHjV1ZfJYHu4n8AYs0hE/i0ikdhfCu+V2wtBKY/QxK+Ui4h0Aoqx/dcFuMUYM6fCMTGneYtHgK+NMeNdxy08xXGvAVdhG2z9ti4xK1UbOsevFLYbIzANeMHVlnoO8H+u1sGISJyINK7wsjxsc7tSzTjemvma05xuJvB7AGNMWp2DV6qGNPGrQNawdDknMA/4CijdVPtlIA27r8NG4D+cfIX8NZBYenMXeAJ4TES+q+TYMsaYLGxHz//W61ejVDVpd06lPExEGgEbsHtGHHA6HhV4dMSvlAe5VvukA89r0ldO0RG/UkoFGB3xK6VUgNHEr5RSAUYTv1JKBRhN/EopFWA08SulVID5f9j80h++rO7kAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3fpl9k75b2E"
      },
      "source": [
        "There are other cost functions out there (e.g., [cross-entropy cost](https://github.com/the-deep-learners/deep-learning-illustrated/blob/master/notebooks/cross_entropy_cost.ipynb) is the typical choice for a deep learning classifier), but since MSE is the most common, including being the default option for regression models, it will be our focus for the remainder of *ML Foundations*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5EDHWnw5b2E"
      },
      "source": [
        "**Return to slides here.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ocedC8el5b2F"
      },
      "source": [
        "### Minimizing Cost with Gradient Descent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CN3TlkVM5b2F"
      },
      "source": [
        "Refer to the slides and the [*Gradient Descent from Scratch* notebook](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/gradient-descent-from-scratch.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "43iljMvZ5b2G"
      },
      "source": [
        "### Critical Points"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-RO6uoRh5b2G"
      },
      "source": [
        "#### Minimum"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "52gWUjiW5b2H"
      },
      "source": [
        "x = np.linspace(-10, 10, 1000)"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nqbOSBOc5b2J"
      },
      "source": [
        "If $y = x^2 + 3x + 4$..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "suJYqaq_5b2J"
      },
      "source": [
        "y_min = x**2 + 3*x + 4"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Skkp7dSA5b2M"
      },
      "source": [
        "...then $y' = 2x + 3$. \n",
        "\n",
        "Critical point is located where $y' = 0$, so where $2x + 3 = 0$.\n",
        "\n",
        "Rearranging to solve for $x$:\n",
        "$$2x = -3$$\n",
        "$$x = \\frac{-3}{2} = -1.5$$\n",
        "\n",
        "At which point, $y = x^2 + 3x + 4 = (-1.5)^2 + 3(-1.5) + 4 = 1.75$\n",
        "\n",
        "Thus, the critical point is located at (-1.5, 1.75)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cXq47qEq5b2M",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "b5d76c63-d12c-4237-92b9-53509f1d9711"
      },
      "source": [
        "fig, ax = plt.subplots()\n",
        "\n",
        "plt.scatter(-1.5, 1.75, c='orange')\n",
        "plt.axhline(y=1.75, c='orange', linestyle='--')\n",
        "\n",
        "plt.axvline(x=0, c='lightgray')\n",
        "plt.axhline(y=0, c='lightgray')\n",
        "\n",
        "ax.set_xlim(-5, 2)\n",
        "ax.set_ylim(-1, 15)\n",
        "\n",
        "_ = ax.plot(x, y_min)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deVyVZf7/8dfnsG8CCm4sIiruuKGWWWlqaZlllqXV6NSMVtNUM+3Tvti+7zmpTZNttmpftUwzs0wFEzdQEVxAQdzYd67fH9D8TFMOcOA+5/B5Ph4+hHPuc+63qG9u7vu6r0uMMSillHJ9NqsDKKWUcgwtdKWUchNa6Eop5Sa00JVSyk1ooSullJvwbM6dhYWFmZiYmObcpVKqkcrKygDw8fGxOEnLlZSUdMgYE17Xds1a6DExMSQmJjbnLpVSjZSeng5AbGysxUlaLhHZY892espFKaXchBa6Ukq5CS10pZRyE1roSinlJrTQlVLKTWihK6WUm9BCV0opN6GFrpRSbkILXSml3ESdhS4ic0XkoIhs+YPnbhcRIyJhTRNPKaVatvLKaru3tecI/V1g7IkPikgUcD6w1+5gVfYHU0opBZ9tyLR72zoL3RizCjjyB0+9CNwF2L2GXW5Bmd3BlFKqpSuvrOa1FWl2b9+gc+gicgmQZYxJtmPbGSKSKCKJR4rKyTxa3JBdKqVUi/PZhkyyjpXYvX29C11E/IF/AQ/as70xZrYxJsEYkyDA69/b/91GKaVaqt+OzvtHhdj9moYcoXcBOgPJIrIbiAQ2iEj7ul7YOsCbBYmZ7DuiR+lKKXU6vx2d3zq6m92vqXehG2M2G2PaGmNijDExQCYw0BiTXddrw4N8sInoUbpSSp3Gb0fn/aJCGBFX57oW/2PPsMUPgTVAdxHJFJHrGxrSy8PGlCFRfJqkR+lKKXUqC5L2kXWshNtGdUNE7H6dPaNcphhjOhhjvIwxkcaYOSc8H2OMOWTvDm8c0RWbTXh1xU67QyqlVEtRWlHFq8vTGBgdwoju9h+dgwV3irYP9mXqkGg+25BFxqGi5t69Uko5tfd/2UN2fil3XNC9XkfnYNGt/zeN7IK3h40Xl+2wYvdKKeWUCssqeXPlLoZ3DWNYl/rfgG9JobcN8uXPZ8WwMHk/2/bnWxFBKaWczrzVGRwuKueOC7o36PWWTc4185wuBPl68sKy7VZFUEopp5FXXMHsH9MZ3bNdvcaeH8+yQg/29+KGc7vwXcpBkvYctSqGUko5hbdX7aKwrJLbz49r8HtYOn3u9GExhAV689w3epSulGq5cgvKmPfTbi6O70jPDq0a/D6WFnqAjyc3jejKmvTD/JRm98hHpZRyK69/n0Z5VTX/GNPwo3NwggUupg6NpmOwL898sx1j7J64USml3ELWsRI+WLuXywdG0jksoFHvZXmh+3p5cOvobiTvO8aybTlWx1FKqWb16vKamyxvqcecLadieaEDTKr9zvT8tzuoqtajdKVUy5BxqIgFSZlMHRpNRIhfo9/PKQrd08PGP8fEsT2ngEXJ+62Oo5RSzeLFZTvw9rDxt5FdHfJ+TlHoABf17UDPDq14YdkOKnSpOqWUm9u6P49Fm/Yz/awYwoN8HPKeTlPoNptw5wVx7D1SzMfr91kdRymlmtRTS1IJ9qu5H8dRnKbQAUZ2b0tCp1BeXr6T4vJKq+MopVSTWL3zED/uPMTNI7sS7OflsPd1qkIXEe69sCe5BWW882OG1XGUUsrhqqsNTy5JISLEj2vP7OTQ93aqQgcY1CmUsb3b8/YPu8gtKLM6jlJKOdSiTfvZuj+fOy6Iw8fTw6Hv7XSFDnDX2O6UVlbzynJdBEMp5T7KKqt49pvt9OrQikv6RTj8/Z2y0GPDA5k6JJoP1u0lPbfQ6jhKKeUQ83/ZS+bREu4Z1wObrX6LV9jDnjVF54rIQRHZctxjz4pIqohsEpEvRKRhcz2exi2juuHraeNZnbhLKeUG8ksreHXFToZ3DeOceiz8XB/2HKG/C4w94bFlQB9jTDywA7jXwbkID/JhxjldWLIlW6fXVUq5vLd/2MXR4gruHtujyfZhzyLRq4AjJzz2rTHmt3GFvwCRTZCNv5zdmfAgH55cnKITdymlXFZ2XilzVmcwoV9H+kYGN9l+HHEO/TpgyameFJEZIpIoIom5ubn1euMAH0/+MTqOxD1HdeIupZTLeum7mnmq7mzg0nL2alShi8h9QCUw/1TbGGNmG2MSjDEJ4eH1P280OSGSLuEBPLU0lUqdEkAp5WJ25hTwSeI+rjmjE1Gt/Zt0Xw0udBGZDowHrjZNeD7E08PG3WN7kJ5bxMeJOiWAUsq1PL00lQBvT/5+XuOnx61LgwpdRMYCdwETjDHFjo10sjG92jE4JpQXl+2kqEynBFBKuYaf0g7xXcpBbhzZhdYB3k2+P3uGLX4IrAG6i0imiFwPvAYEActEZKOIvNWUIX+bEuBQYRlv/7CrKXellFIOUVVteOzrbUSG+nHdWZ2bZZ+edW1gjJnyBw/PaYIspzUwOpQJ/Try9qp0rhzimMnglVKqqSxI3EdqdgGvTR2Ar5djb/E/Fae8U/RU7h5XM37z6SWpFidRSqlTKyyr5Llvd5DQKZSL+nZotv26VKFHhPgx45xYFibv15uNlFJO643v0zhUWMYD43sh4vhb/E/FpQod4IZzu9A2yIfHvt5Gta4/qpRyMplHi3lndQYTB0TQL8rhs6KclssVeoCPJ3de0J2N+46xaJOuP6qUci5PL92OTWjym4j+iMsVOsCkgZH0iWjFU0tSKSmvsjqOUkoBkLTnKIuS9zPj7Fg6WjBwwyUL3WYTHhzfmwN5pcxelW51HKWUorp2mGLbIB9mOnCd0PpwyUIHGNK5NRf2bc9bP+wiO6/U6jhKqRZu0ab9bNx3jDsv6E6AT50jwpuEyxY6wD1je1JVbXhmqQ5jVEpZp6iskicWp9A3IphJA5tk8lm7uHShR7fx57rhnfn81yw27NVhjEopa7z+fRo5+WU8PKF3k6xEZC+XLnSAm8/rStsgHx76aitVOoxRKdXMdh8q4p0fM7hsYASDOoVamsXlCz3Qx5P7LurJ5qw8PtHZGJVSzezRr7fh7WnjniZcicheLl/oABP6dWRITGueWZrKseJyq+MopVqIFak5rEg9yC2jutK2la/Vcdyj0EWEhyf0Jq+kgheW7bA6jlKqBSirrOLRRduIDQ9g+rDmmU2xLm5R6AC9OrbimjM68f4ve9i2P9/qOEopNzd39W52Hy7m4Yt74+3pHFXqHCkc5J9j4gjx9+bhhVt1UWmlVJPJzivl1RU7GdOrHefE1X9pzabiVoUe4u/NnRd0Z93uIyxM1nlelFJN48klKVRWGx64qJfVUX7HrQodYHJCFPGRwcz6vxQKdbk6pZSDrU0/zFcb9zPznFii2zTtos/15XaF7mETHpnQm4MFZby6fKfVcZRSbqS8spr7v9xCZKgfN43oanWck9izpuhcETkoIluOe6y1iCwTkZ21v1s7mv4EA6JDmZwQyZzVGWzPLrA6jlLKTcxZncHOg4U8MqE3ft7Ns6xcfdhzhP4uMPaEx+4BlhtjugHLaz93KveM60mQryf3f7lZF8JQSjVa5tFiXllecyF0VM92Vsf5Q3UWujFmFXDkhIcvAf5T+/F/gEsdnKvRWgd4c++4nqzffZRPkzKtjqOUcnGPLNoGwMMTeluc5NQaeg69nTHmQO3H2cApv12JyAwRSRSRxNzc3AburmEuHxTJ4JhQnlySwpEivYNUKdUw323LYdm2HG4d3Y0ICxausFejL4qamgHfpzynYYyZbYxJMMYkhIc373hNm014/NK+FJRW8tSSlGbdt1LKPRSXV/LQwq3EtQvk+uHOcUfoqTS00HNEpANA7e8HHRfJsbq3D+IvZ8fySWIm6zJOPHOklFKn9+qKNLKOlfD4pX3x8nDugYENTbcQmFb78TTgK8fEaRq3jOpKRIgf93+5mfLKaqvjKKVcxM6cAv69Kp3LB0UypHNrq+PUyZ5hix8Ca4DuIpIpItcDTwFjRGQnMLr2c6fl7+3JIxN6syOnkDmrM6yOo5RyAcYY7v9yCwE+ntw7zvqpce1R58J3xpgpp3hqlIOzNKnRvdpxfq92vLx8B+PjOxDV2rnu8FJKOZeP1+9jbcYRnrqsL20CfayOYxfnPiHkYA9N6I1NhAe+2qKTdymlTulgfilPLE5haOfWXDk4yuo4dmtRhR4R4scd53dn5fZcvtqok3cppf7Yw4u2UlpZzZOX9UXEujVC66tFFTrAtGEx9I8K4ZFFWzlcWGZ1HKWUk/lmazaLN2dz66huxIYHWh2nXlpcoXvYhGcuj6ewrJJHv95mdRyllBPJL63gwa+20KN9EDPOibU6Tr21uEIHiGsXxE0juvLVxv18n+q0Q+iVUs3s6SWp5BaU8dSkeKcfc/5HXC+xg9w0sgtd2wZy3xebdd50pRTrdx9h/tq9TB/Wmf5RIVbHaZAWW+g+nh48PSmeA/mlPLs01eo4SikLlVZUcc9nm4gI8eP28+OsjtNgLbbQAQZ1CmXamTG898sekvbotABKtVSvf5/GrtwiZk3sQ4BPnbfnOK0WXegAd1zQnY7Bftz92WbKKqusjqOUamZbsvJ4Y+UuJg6IYET3tlbHaZQWX+iBPp48PrEPaQcLefk7XbJOqZakvLKaOxYk0zrAm4cudq4FnxuixRc6wMjubZmcEMlbP+xi475jVsdRSjWTV1fsJDW7gCcn9iXE39vqOI2mhV7r/vG9aNfKl9s/2UhphZ56Ucrdbc6sOdVy2cAIRvdyziXl6ksLvVYrXy+enhTPrtwiXli2w+o4SqkmVFZZxR0LkgkL9Oah8c67pFx9aaEf55y4cKYOjebfP6aTuFtHvSjlrl5dnsb2nAKevKwvwf5eVsdxGC30E/zrwp50DPbjjgXJlJTrqRel3M2mzGO8+cMuJg2M5Lwe7nGq5Tda6CcI9PHk2Svi2X24mGe+0RuOlHInx59qedANRrWcSAv9DwzrEsa0Mzsx76fd/JJ+2Oo4SikHeWHZDnbkFPLUpHiC/dznVMtvtNBP4e5xPYhu7c8dC5IpKK2wOo5SqpF+ST/M7FXpTBkSxUgXv4HoVBpV6CLyDxHZKiJbRORDEfF1VDCr+Xt78sLkfuw/VsIji3SaXaVcWX5pBbd/kkyn1v7cf5H7nWr5TYMLXUQigFuABGNMH8ADuMpRwZxBQkxr/jayK58mZbJ48wGr4yilGuihr7aSnV/Ki1f2d+m5WurS2FMunoCfiHgC/oDbret2y6hu9IsM5t7PN5OdV2p1HKVUPS1K3s8Xv2bx9/O6MiA61Oo4TarBhW6MyQKeA/YCB4A8Y8y3J24nIjNEJFFEEnNzcxue1CJeHjZevLI/5ZXV3L5gI9XVuri0Uq7iQF4J932xmf5RIdw8sqvVcZpcY065hAKXAJ2BjkCAiFxz4nbGmNnGmARjTEJ4eHjDk1ooNjyQBy/uxU9ph5n7U4bVcZRSdqiuNtyxIJnKasOLV/bH0wVXIKqvxvwJRwMZxphcY0wF8DkwzDGxnM9Vg6MY06sdzyzdTsqBfKvjKKXqMPenDH5KO8wD43vROSzA6jjNojGFvhc4Q0T8RUSAUUCKY2I5HxHhqdrbhG/7SCfwUsqZpRzI55lvtjO6ZzuuGhxldZxm05hz6GuBT4ENwOba95rtoFxOqU2gD89eHs/2nAKeWqJ3kSrljIrLK/nbBxsI8fPi6Ul9qTnebBkadVLJGPOQMaaHMaaPMeZaY0yZo4I5qxHd2/Lns2J49+fdLNuWY3UcpdQJHvpqKxmHinjpyv60CfSxOk6zcv+rBE3gnnE96BPRijsWJJN1rMTqOEqpWl/+msWCpEz+PrIrw7qGWR2n2WmhN4CPpwevTRlIVbXhlg9/paKq2upISrV4GYeKuO+LzQyOCeWWUd2sjmMJLfQGigkL4InL+pK056guiKGUxcoqq/j7hxvw8rTx8lUDWsQQxT/SMv/UDjKhX0emDInizZW7+GGH6900pZS7eGpJKluy8nn28n50DPGzOo5ltNAb6cHxvYlrF8g/P97IwXydGkCp5rZsWw7zftrN9GExjHGTtUEbSgu9kfy8PXh96kCKyiu59aONVOnUAEo1m31Hirnz02R6d2zFvRf2sDqO5bTQHaBbuyAevaQPa9IP8/LynVbHUapFKK2o4qb5G6iqNrw+dSA+nh5WR7KcFrqDXDEokkkDI3ll+U5WpOr4dKWa2iOLtrE5K4/nr+hHTAu5tb8uWugOIiLMmtiHXh1acdtHG9l7uNjqSEq5rQWJ+/hw3V5uHNGF83u3tzqO09BCdyBfLw/eumYQADPfT6KkXOd7UcrRtu3P5/4vt3BmbBtuHxNndRynooXuYNFt/Hn5qgGkHMjnvi83Y4xeJFXKUfJKKrhxfhIh/l68OrXljjc/Ff1qNIGRPdpy66hufL4hi/lr91odRym3UF1tuP2TjWQdLeGNqwcS1sLmabGHFnoTuXVUN0Z0D+eRRVv5de9Rq+Mo5fLe/GEX36Uc5P6LejKoU2ur4zglLfQmYrMJL13Zn3atfLnx/Q3kFrj9RJRKNZnlKTk89+12JvTryLRhMVbHcVpa6E0oxN+bt64ZxLGScm54P4mySr1IqlR97cwp4NaPNtKnYzBPT4pvUfOb15cWehPrExHMc1f0I2nPUe7/YoteJFWqHo4Vl/PX9xLx9fJg9p8G4eetNw+djqfVAVqC8fEd2ZFdwCsr0ujePoi/nB1rdSSlnF5lVTU3f/Ar+4+V8uGMoXQIbrmTbtlLj9CbyW2j47igdzueWJyiMzMqZYdZi1NYnXaIxy/toxdB7aSF3kxsNuGFyf2JaxfEzR9sYFduodWRlHJan6zfx7yfdvPns2KY3IIWeW6sRhW6iISIyKcikioiKSJypqOCuaMAH0/emZaAt4eNv/wnkbziCqsjKeV0Encf4b4vNzO8axj3XdjT6jgupbFH6C8DS40xPYB+QErjI7m3yFB/3rp2EJlHi/nbBxt0+TqljpNxqIi/vpdIZKg/r+mdoPXW4K+WiAQD5wBzAIwx5caYY44K5s4Gx7Rm1sS+rE47xH1f6PQASgEcKSrnz/PWATBv+mBC/L0tTuR6GvPtrzOQC8wTkV9F5B0ROWkOSxGZISKJIpKYm6sXA38zOSGKW87ryieJmby2Is3qOEpZqrSiir++l8j+vFLemZag0+E2UGMK3RMYCLxpjBkAFAH3nLiRMWa2MSbBGJMQHh7eiN25n3+MieOyARE8v2wHnyVlWh1HKUtUVxvuWJBM0p6jvDi5v45oaYTGFHomkGmMWVv7+afUFLyyk4jw1KR4zoxtw92fbeLntENWR1Kq2T377Xa+3nSAe8b14KL4DlbHcWkNLnRjTDawT0S61z40CtjmkFQtiLenjbeuHURseAAz309ie3aB1ZGUajYfrN3Lmyt3MXVoNDPP0RvuGquxl5D/DswXkU1Af+CJxkdqeYL9vJj35yH4eXnw53nryMkvtTqSUk1ueUoOD3y1hXPjwnl0Qm+do8UBGlXoxpiNtefH440xlxpjdJ7YBooI8WPu9MHklVQwbe46HaOu3Nr63Ue4af4GendsxetXD9ThiQ6iX0Un0icimLevTSA9t4jr/rOe4vJKqyMp5XApB/K57t31RIT6MW/6YAJ9dEopR9FCdzLDu4Xx8lX9+XXvUW54fwPllXrjkXIf+44UM23uOgK8PXnvuiG00VWHHEoL3QmN69uBJy/ry6odufzjk41UVeuNR8r15RaUce2ctZRVVvPe9UOIDPW3OpLb0Z91nNSVg6PJK6ngicWptPL14omJffSikXJZeSUVTJ+3jpz8Mub/dShx7YKsjuSWtNCd2IxzunCsuII3Vu4ixN+Lu8f2sDqSUvVWUFpzoX9nTiH/npbAwOhQqyO5LS10J3fnBd05VlLBmyt3Eejjyd9GdrU6klJ2Ky6v5Lp317MlK483rh7IuXF6t3hT0kJ3ciLCY5f0oaiskme/2Y6HTbjh3C5Wx1KqTqUVVfzlP4kk7TnKq1MGcn7v9lZHcnta6C7AwyY8f0U/qg08tSQVT5voMnbKqZVVVnHD+0msST/M81f001v6m4kWuovw9LDx4uR+VFcbHv+/FGwiXDe8s9WxlDpJWWUVf5v/Kyu35/LkZX25bGCk1ZFaDC10F+LpYeOlq/pTVW149OtteNiEacNirI6l1P+UVlRx4/tJfL89l0cv6c2UIdFWR2pRdBy6i/HysPHKlAGM6dWOhxZuZc7qDKsjKQVASXnNnObfb8/liYl9+dOZMVZHanG00F2Qt6eN16cOZFyf9jz29TZe/14XyFDW+m00y+q0QzxzeTxTh+qRuRW00F2Ut6eNV6cMYOKACJ79ZjvPLE3VpeyUJQrLKpk+dz1rMw7zwuR+TE6IsjpSi6Xn0F2Yp4eN56/oh6+XB2+s3EVxeRUPXdxL7yhVzeZIUTl/rh1n/tJVA5jQr6PVkVo0LXQXZ7MJT0zsg5+XB3N/yqC0oopZE/viYdNSV01r/7ESrp2zln1HS3jzah1n7gy00N2AiPDA+J4E+njwyoo0jhSV88qUAfh6eVgdTbmptIOF/GnOWvJLK3nvuiGcEdvG6kgKPYfuNkSEf57fnYcv7sWylByunbNWF8lQTWJT5jEmv72G8qpqPppxhpa5E9FCdzPTz+rMq1MGkLwvjyve/pn9x0qsjqTcyI87c5ky+xf8vT1YcMMw+kQEWx1JHafRhS4iHiLyq4h87YhAqvHGx3fk3esGc+BYKZPe/JkdObrwtGq8j9btZfq89US19ufTG4bROSzA6kjqBI44Qr8VSHHA+ygHGtYljI9nnklltWHSmz/z485cqyMpF1VtDP9ed5B7Pt/M8K5hLLjhTNoH+1odS/2BRhW6iEQCFwHvOCaOcqReHVvxxU3DiAjxY/q89fz3lz1WR1IuprSiilnL9/PhxsNMHRrNnGkJBPl6WR1LnUJjj9BfAu4CTrnwpYjMEJFEEUnMzdWjxOYWGerPpzcO49y4cB74cgsPL9xKZZWuU6rqlltQxtXvrOX79HxmDG3LrEv74Omhl92cWYP/dkRkPHDQGJN0uu2MMbONMQnGmITwcJ3c3gqBPp78+08JXD+8M+/+vJvr/5NIQamOgFGntinzGBNeW83W/Xk8ODqCq/q10RvWXEBjvt2eBUwQkd3AR8B5IvK+Q1Iph/OwCQ+M78WsiX1YnXaIS17/ibSDerFUneyLXzO54q012ET49IZhjIhtZXUkZacGF7ox5l5jTKQxJga4ClhhjLnGYclUk7h6aCfev34o+SUVXPLaTyzZfMDqSMpJVFZV8/jX2/jHx8kMiA5h4c1n6bBEF6MnxFqgM7u0YdHfh9OtXRA3zt/Ak4tT9Lx6C5dbUMa0eet4Z3UG04fF8N/rh9Im0MfqWKqeHHLrvzFmJbDSEe+lmkeHYD8+nnkGjy7axtur0tmclccrUwYQpv+JW5yfdx3i1o82kl9SwbOXx3OFzpbosvQIvQXz8fRg1sS+PHt5PEl7jjL2pR91vHoLUlVtePm7nVzzzlpa+Xry1c1naZm7OC10xRUJUXx181mE+ntx7Zx1PLkkhQo9BePWcgvK+NPctbz43Q4u6R/BwpuH06O9Xvx0dVroCoAe7Vux8ObhTBkSzds/pHP5W2vYe7jY6liqCXy7NZuxL60icfdRnp7Ulxcm9yPARydedQda6Op//Lw9ePKyvrw+dSDpuYVc+MqPfLJ+n66E5CYKyyq569NkZvw3iXatfFn09+FcOThax5e7Ef22rE5yUXwH+kUFc8eCZO76bBOLtxzgqcvidf4OF7Z+9xH++clGso6WcNOILtw2Og5vTz2eczf6N6r+UGSoPx/85QwemdCbtelHGPPiD3yWlKlH6y6muLySx7/exuS31yAIn8w8k7vG9tAyd1N6hK5OyWYTpg2L4dy4cO5YkMztC5JZvPkAj17ah4gQP6vjqTqs3H6Q+77YQtaxEq4eGs29F/YkUM+VuzX9Nq3qFBMWwMczz+T+i3ry867DjH7+B976YZeOhHFShwrLuPWjX5k+bz2+XjYW3HAmsyb21TJvAfRvWNnFwyb85exYxvXtwMMLt/LUklS+2JDF4xP7MDimtdXxFDXjyj9ct5fnvt1OUVklt47qxk0ju+DjqWvLthRa6KpeIkL8+PefEli2LYeHF27lirfWMGlgJHde0F0vmlpoza7DPLJoK6nZBQzt3JrHL+1Dt3ZBVsdSzUwLXTXImF7tOKtrG15dkcacHzP4v837mXF2LDPP7aJjmpvRviPFPLkkhcWbs4kI8eONqwcyrk97HYrYQun/PNVg/t6e3D22B1OHRPP00lReWZHGh+v3cfuYOK5IiMLDpqXSVA4VlvHG97t4/5c92GzwzzFxzDgnFl8vPb3SkklzDkNLSEgwiYmJzbY/1bw27D3KrP9LIWnPUWLDA7h1VDfGx3fUYnegvJIK3vkxnTmrMyitqOLyQZHcNjqOjk046ig9PR2A2NjYJtuHOj0RSTLGJNS5nRa6ciRjDN9szeal73aSml2gxe4geSUVvP/LHmavSievpILx8R34x5g4uoQHNvm+tdCtZ2+h6ykX5VAiwtg+HTi/V3uWbs3m5e92cutHG3ll+U5mntOFCf076mmBesjJL2XO6gw+WLuXwrJKzuvRltvPj6N3R114Qp1Mj9BVk6quNizdms0ry2uO2MMCvbn2jBiuOSNaF1A4jZ05BbzzYwZf/JpFZXU14+M7MvPcWEuKXI/QradH6Mop2GzChX07MK5Pe37edZg5qzN48bsdvL4yjYn9I5gyNJp+kcE6KgMoq6xi6ZZs5q/dy7qMI/h42rhycBR/PTuW6Db+VsdTLkALXTULEeGsrmGc1TWMtIOFzPspg883ZPFx4j66twti8uAoJg6IoHWAt9VRm13awUI+TcpkQeI+DheVE93an3vH9eDyQZH6U4yqlwafchGRKOA9oB1ggNnGmJdP9xo95aKOV1BawaLkA3y8fi/JmXl4e9g4r0dbLorvwAaTRI4AAAnHSURBVHk92rr1ePYDeSV8nXyALzdmsXV/Ph42YVSPtlxzRieGdw3D5kQXkPWUi/Wa45RLJXC7MWaDiAQBSSKyzBizrRHvqVqQIF8vpg6NZurQaFKz8/l4/T4WJR9g6dZsfDxtjOzelnF923Nej7YE+XpZHbfR0nMLWZF6kGXbcli3+wjGQL/IYB4Y34uL4zvQtpXeaasax2EXRUXkK+A1Y8yyU22jR+iqLlXVhvW7j7B48wGWbMkmt6AMD5swMDqEs7uFc05cOH0jgl1iCGRRWSVJe46ycnsuK1Jz2F27AlRcu0Au6tuRCf070jkswOKUddMjdOs16zh0EYkBVgF9jDH5Jzw3A5gBEB0dPWjPnj2N3p9qGaqqDRv2HmXl9oOs2nGILfvzMAaC/bwY1CmUAVEhDIgOJT4qmFYNPYLPmA/J90HxXvCPhn6zoPPV9X4bYwy5BWVs2HuM9buPsH73Ebbuz6eq2uDtaWNYlzac16MtI7u3Jaq1a13g1EK3XrMVuogEAj8As4wxn59uWz1CV41xuLCMn3YdZvXOXDbsPUbawUIARKBLeCBx7QLpGh5Il7aBdG0bSGxYIH7epxnznjEf1s2AquPWTvXwhyGzT1nq1dWG3MIyMo8Wk55bRGp2AanZ+aQcKOBIUTkAPp42+keFMDimNYM7t2ZwTCj+3q57PUAL3XrNUugi4gV8DXxjjHmhru0T4oJM4huDfv9g9GSIuwkqi2HlhSe/KHZ6za/SQ7D68pOf73YjdLoSivbBmmtPfr7H7RB5MeRvh3UzT36+z/3QfjQc3QhJt538fL8nIHwY5P4Myf86+flBL0Fof8j+DrY8fvLzQ96GVt0hcxGkPn/y82f+FwKiYM/HsPPNk58f/in4hkH6uzW/TjRiMXj6w443YO8nJz8/emXN7ynPQdbXv3/Oww9GLqn5ePNjkLP898/7tIGzP6v5eOO9cGjN75/3j4Rh79d8nHRbzdfweEFxMHR2zcdrZ0DBjt8/H9q/5usH8PM1UJz5++fDzoT+T9Z8/OMkKDv8u6fzQs9nU6u/snHvMZI3LiWtMJS9JcFUHzfNfytfT9oH+9KuYjPtfApp412Cv0c5AR4V+JXtJsAcRqjGYKMKG1XGRpX4UOjbg2MBQ8jz7UleYRFH9m9gf2kQWSWtKDf/v5x9vWx0D/ehB2vpEZhLfKts+rTKwcdW5Tb/9tK3rYHSbGKPnDDmoQX/26PdKOj7QM3H34+DqpLfPx8xHnreUfPxdyM4ST17T8b80LQXRaVm4PAcIMWeMlfK0YK9qzi7WzhndwsH2y1QVUJplQe7S0JJK2rDHs8zyPEdTHZeKTmZPuwsCuNohS9l1fadnrGJIdjvAMG+NkIqvegVdJDzw3cS6ZdPpG8e0T3GERN/BR4lmbDm0Sb+0ypVt8YMWxwO/AhsBn5buuZfxpjFp3qNnnJRzqCq2lBcXknxwgSKig5RbWx4SDU2qrFJNR5+HQi6ZD2BPp56wxN6ysUZNPmwRWPMakD/tSuX42ETgny9CBp81ynOod8JbjBMUrU8uqaoark6X11zAdS/EyA1v5/mgqhSzs51L70r5Qidr9YCV25Dj9CVUspNaKErpZSb0EJXSik3oYWulFJuQgtdKaXchBa6Ukq5CS10pZRyE1roSinlJrTQlVLKTWihK6WUm9BCV0opN6GFrpRSbkILXSml3IQWulJKuQktdKWUchNa6Eop5SYaVegiMlZEtotImojc46hQSiml6q/BhS4iHsDrwDigFzBFRHo5KphSSqn6acwSdEOANGNMOoCIfARcAmw71QvKysr+t4K4Uso1lJaWAuj/XRfQmFMuEcC+4z7PrH3sd0RkhogkikhiRUVFI3anlFLqdJp8kWhjzGxgNkBCQoKJjY1t6l0qpRzotyNz/b/r/BpzhJ4FRB33eWTtY0oppSzQmEJfD3QTkc4i4g1cBSx0TCyllFL11eBTLsaYShG5GfgG8ADmGmO2OiyZUkqpemnUOXRjzGJgsYOyKKWUagS9U1QppdyEFrpSSrkJLXSllHITWuhKKeUmtNCVUspNaKErpZSb0EJXSik3oYWulFJuQowxzbczkQJge7Pt0PHCgENWh2gEV87vytlB81vN1fN3N8YE1bVRk8+2eILtxpiEZt6nw4hIoua3hitnB81vNXfIb892espFKaXchBa6Ukq5ieYu9NnNvD9H0/zWceXsoPmt1iLyN+tFUaWUUk1HT7kopZSb0EJXSik30eyFLiIPi0iWiGys/XVhc2dwBBG5XUSMiIRZncVeIvKYiGyq/bp/KyIdrc5UHyLyrIik1v4ZvhCREKsz1YeIXCEiW0WkWkRcZgidiIwVke0ikiYi91idpz5EZK6IHBSRLVZnqS8RiRKR70VkW+2/m1vreo1VR+gvGmP61/5yuRWPRCQKOB/Ya3WWenrWGBNvjOkPfA08aHWgeloG9DHGxAM7gHstzlNfW4DLgFVWB7GXiHgArwPjgF7AFBHpZW2qenkXGGt1iAaqBG43xvQCzgD+VtfXXk+5NMyLwF2AS11RNsbkH/dpAK6X/1tjTGXtp78AkVbmqS9jTIoxxtXulB4CpBlj0o0x5cBHwCUWZ7KbMWYVcMTqHA1hjDlgjNlQ+3EBkAJEnO41VhX6zbU/Ns8VkVCLMjSIiFwCZBljkq3O0hAiMktE9gFX43pH6Me7DlhidYgWIALYd9znmdRRKsrxRCQGGACsPd12TXLrv4h8B7T/g6fuA94EHqPm6PAx4Hlq/nM6jTry/4ua0y1O6XTZjTFfGWPuA+4TkXuBm4GHmjVgHerKX7vNfdT8ODq/ObPZw578StWHiAQCnwG3nfBT9kmapNCNMaPt2U5E/k3NuVyncqr8ItIX6AwkiwjU/Mi/QUSGGGOymzHiKdn7taemDBfjZIVeV34RmQ6MB0YZJ7yJoh5ff1eRBUQd93lk7WOqGYiIFzVlPt8Y83ld21sxyqXDcZ9OpOZCkUswxmw2xrQ1xsQYY2Ko+fFzoLOUeV1EpNtxn14CpFqVpSFEZCw11y4mGGOKrc7TQqwHuolIZxHxBq4CFlqcqUWQmqPGOUCKMeYFu17T3Ac5IvJfoD81p1x2AzONMQeaNYSDiMhuIMEY4xLTcorIZ0B3oBrYA9xgjHGZoy0RSQN8gMO1D/1ijLnBwkj1IiITgVeBcOAYsNEYc4G1qepWO7T4JcADmGuMmWVxJLuJyIfACGqmz80BHjLGzLE0lJ1EZDjwI7CZmv+zAP863chAvfVfKaXchA5bVEopN6GFrpRSbkILXSml3IQWulJKuQktdKWUchNa6Eop5Sa00JVSyk38PzL9EUMZKHYPAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EWFvP3Tr5b2P"
      },
      "source": [
        "#### Maximum"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bn6YmWLR5b2P"
      },
      "source": [
        "If $y = -x^2 + 3x + 4$..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XA2KkY9x5b2Q"
      },
      "source": [
        "y_max = -x**2 + 3*x + 4"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b8AUiYpt5b2S"
      },
      "source": [
        "...then $y' = -2x + 3$. \n",
        "\n",
        "Critical point is located where $y' = 0$, so where $-2x + 3 = 0$.\n",
        "\n",
        "Rearranging to solve for $x$:\n",
        "$$-2x = -3$$\n",
        "$$x = \\frac{-3}{-2} = 1.5$$\n",
        "\n",
        "At which point, $y = -x^2 + 3x + 4 = -(1.5)^2 + 3(1.5) + 4 = 6.25$\n",
        "\n",
        "Thus, the critical point is located at (1.5, 6.25)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M0S-2dX_5b2S",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        },
        "outputId": "5c77fa62-fa1d-4bfc-ac39-74eabc9781fa"
      },
      "source": [
        "fig, ax = plt.subplots()\n",
        "\n",
        "plt.scatter(1.5, 6.25, c='orange')\n",
        "plt.axhline(y=6.25, c='orange', linestyle='--')\n",
        "\n",
        "plt.axvline(x=0, c='lightgray')\n",
        "plt.axhline(y=0, c='lightgray')\n",
        "\n",
        "ax.set_xlim(-2, 5)\n",
        "ax.set_ylim(-5, 10)\n",
        "\n",
        "_ = ax.plot(x, y_max)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deVxVdf7H8deHfREQZFNRQVEQUVFwb7G00rJs35u2GZuaKVtmWqea1l/7NjWVOe1N5bSXabnlkqaC4YaICCq4AaKIIOv9/v6AZipTwXvg3Hv5PB8PHl7uvZzzjvDt4Zzv+X7FGINSSinP4WV3AKWUUtbSYldKKQ+jxa6UUh5Gi10ppTyMFrtSSnkYLXallPIwLS52EXldREpEZN3PnosQkTkisqn5z/C2iamUUqqlWnPE/iYw4VfP3QnMM8b0BeY1f66UUspG0poblEQkHvjKGJPa/PlGYKwxZqeIdAW+M8YktUVQpZRSLePj5NfHGGN2Nj/eBcQc7o0iMgWYAhAcHJyenJzs5K7VsaitrQXA39/f5iRKqdbKysoqM8ZEHe19zhb7fxljjIgc9vDfGDMNmAaQkZFhMjMzrdq1aoWCggIAevfubXMSpVRricjWlrzP2VExu5tPwdD8Z4mT21NKKeUkZ4v9C+DK5sdXAp87uT2llFJOas1wx/eBZUCSiBSLyLXAY8ApIrIJGN/8uVJKKRu1+By7MeaSw7w0zqIsSimlLKB3niqllIfRYldKKQ+jxa6UUh5Gi10ppTyMFrtSSnkYLXallPIwWuxKKeVhtNiVUsrDaLErpZSH0WJXSikPo8WulFIeRotdKaU8jBa7Ukp5GC12pZTyMFrsSinlYbTYlVLKw2ixK6WUh9FiV0opD6PFrpRSHkaLXSmlPIwlxS4it4jIehFZJyLvi0iAFdtVSinVek4Xu4h0B24CMowxqYA3cLGz21VKKXVsrDoV4wMEiogPEATssGi7SimlWsnpYjfGbAeeArYBO4EKY8y3v36fiEwRkUwRySwtLXV2t0oppQ7DilMx4cBkIAHoBgSLyOW/fp8xZpoxJsMYkxEVFeXsbpVSSh2GFadixgOFxphSY0w98Akw2oLtKqWUOgZWFPs2YKSIBImIAOOADRZsVyml1DGw4hz7cuAjYBWwtnmb05zdrlJKqWPjY8VGjDH3A/dbsS2llFLO0TtPlVLKw2ixK6WUh9FiV0opD6PFrpRSHkaLXSmlPIwWu1JKeRgtdqWU8jBa7Eop5WG02JVSysNosSullIfRYldKKQ+jxa6UUh5Gi10ppTyMFrtSSnkYLXallPIwWuxKKeVhtNiVUsrDaLErBVD4HnwWD//2avqz8D27Eyl1zCxZGk8pt1b4HqyYQkNDDQYvfKq2IiumNL2WcJm92ZQ6BvYU+/6NMHfsL5/reSH0uwEaquG70w/9mt5XNX3UlMGS8w99ve/10OsiqCqCZVcc+nrybRB3ZtO+V1x36Oupf4PY8bA3G7JuPvT1wY9C1GgoXQqr7z709fTnIDwNds2FdQ8f+vrwVyE0CYq/hNynD3191DsQ3AO2fgibXj709eM+goBIKHiz6ePXxn4NPkGQ90/YNuPQ18d/1/RndTHU7YG51/zvNe9AOGlW0+O1D8Hueb/8Wv8ucPzHTY+z74KyZb98PSgORr/b9Djr5qbv4c+F9IMRzeubL58ClXm/fD08ren7B7D08qaMPxc5CtL+r+nx4vOgds8vX48ZBwPvbXq8YCI0HgSgwSEU1YRRFHQaxSFnULS3ml25X1BeH8je+kDK6wKpbPCnzhFIreN9HHj/71tCIz7ZDXTymUFYkD9hYZF09ofY6oV0899P14BKugfsJzF4D5EDrkTi9WfvqD97G56C7V/98jUP/dn7r+6ToP9fmh7/uvPA+d47DEuKXUQ6A9OBVMAA1xhjlh35q5SyTk19I2u3V5C9JY0NFeHkHogkv7oLdY6ffsTX4uMlxPp1p4vfQSJ8D5IYVE6ITy3+NYX4e9XjJ/UANBhvGvCm3uHDAf8+VPgnU+HlQ0llNWvLEymrC/7FvjuvMPTtupSkLl4MOpjKkNCd9Aneg5e08zdBqWZijHF+IyJvAYuNMdNFxA8IMsbsO9z7MzIyTGZmptP7Va1XUFAAQO/evW1O4pzqugaWF5azZFMZmVvKydm5n/rGpp/lmFB/kmJDSYrpRL+YEHp1CSYuPJCY0AC8f6ttP4uH6q2HPh/UC87ecsjTNfWN7Kyooai8mvySA2wqOUB+SSW5OyuprG0AoJO/D0N6dmZ0n0jGJHZhQLew3963Uq0gIlnGmIyjvc/pI3YRCQNOAK4CMMbUAXXOblepXysqr2bWup18t7GUzC17qWt04OfjRVqPzlx7XG/Se4UzpGdnIjv5t27Dgx+BFVOgsfp/z3kHNT3/GwJ8vUmIDCYhMpgT+kX993mHw1BQdoDsogqyi/ayorCcx2fnAhAW6MtxfSM5NSWGsUnRhAX6tvq/X6mWsuJUTAJQCrwhIoOBLGCqMabq528SkSnAFICePXtasFvVERSUHmDWul3MWreTddv3A5AcG8JVY+I5vm8kw+IjCPD1PspWjuKnC6Sr74HqbRDUs6nUW3nh1MtLSIwOITE6hPPT4wAoqaxh2eY9LNlUxoKNpcxcsxMfL2Fk7y5MGtSV0wd1JTRAS15Zy+lTMSKSAfwAjDHGLBeR54H9xph7D/c1eirGPu5wKqbiYD1frN7BjJVFrN1eAcCQnp2ZmBrLxNSu9IgIsjnhsXE4DD8W7WNOzm6+Wb+LwrIq/Hy8OCUlhnOHdOeEflH4eusIZHV47XYqBigGio0xy5s//wi404Ltqg7EGMPywnJmrCzi63U7qal3kBwbwr2TUpiYGku3zoF2R3Sal5eQ3iuc9F7h3DEhiTXFFXyyqpgv1+xk5pqdRHby48KMHlw+spdH/Pcq+zhd7MaYXSJSJCJJxpiNwDggx/loqiOobWjki+wd/GtJIbm7Kgnx9+G8oXFcNKwHA7uHIeKZFxxFhME9OjO4R2f+NimFhRtL+TCziFcWbubVRQWcmhLDlaPjGZEQ4bHfA9V2rBrHfiPwXvOImALgaou2qzzU3qo6/r1iG28u3UJpZS3JsSE8cf4gzhzUjUA/J8+Zuxlfby/Gp8QwPiWG4r3VvPPDVj5cWcSsdbtIjg3hhpMSOWNgVx1Vo1rMkuGOraXn2O1j9zn28qo6pi0q4K2lWzhY38gJ/aL4w/EJHJcYqUemP1NT3/SbzLTFBeSXHKB3VDB/GpvIWWnd9Dx8B9bSc+xa7B2MXcW+r7qO6YsLeeP7QqrrG5k8uBvXj00kKTakXXO4G4fDMHv9Lv4xP58NO/fTIyKQP41N5Pz0OHy04Duc9rx4qtRhVdc18NqiQqYvLqCytoFJg7py8/i+JEZrobeEl5dw+sCuTEyNZX5uCS/Mz+fOT9YyfUkhd05IZlz/aP1NRx1Ci121CYfD8Fn2dp6YvZFd+2s4bUAMt5zSj+TYULujuSURYVz/GE5OjubbnN08PiuX37+dyfCECO6amMyQnuF2R1QuRItdWS5razkPfpnD6uIKBseF8eKlQ8iIj7A7lkcQEU4bEMvJydF8sLKI5+fmcc4/l3LW4G787Yz+RIcG2B1RuQAtdmWZ0spaHp6Zw+fZO4gJ9eeZCwdzdlp3vHQ0h+V8vb24YmQvzhnSnVebh0jOzy3h1lP68btRvfT8ewenxa6cZoxhRmYRj36dy8G6Rm46OZE/ju1DkJ/+eLW1Tv4+3HZqEucOjeP+L9bz4Fc5/CermIfPTiW9l56e6aj0b55ySn7JAe7+dC0rCssZnhDBo+cMJDG6k92xOpyEyGDeunoYs9bt4sEvczjv5aVcMbIXd05MJthf/5p3NPp/XB2ThkYHry4q4Pm5mwj08+bx8wZyQXoPPe1iI5GmETQn9IvimW/zeGNpId/llfD4eYMY3SfS7niqHemJONVqW8qquPDVZTz5zUZOGRDD3FtP5KJhPbXUXUQnfx/uOzOFGdeNwluES19bzr2fraOqea545fn0iF21mDGGf6/YxsNfbcDXW3j+4jQmp3W3O5Y6jGHxEcyaegJPfrPxv0fvz12URnovHaHk6fSIXbVIaWUt17y5kns+XUdGfDjf3nKilrobCPTz5r4zU/hwyigALnz1B16cv4lGR/vfca7ajxa7Oqqlm8s4/YXFLN28hwcnD+Dta4YTG6bjpd3J8IQIZt50PKcP7MpT3+Zx+fTl7KqosTuWaiNa7OqwGh2GF+Zt4vLpywkN8OGLPx/H70bF6y3sbio0wJcXLk7jifMGkV20j4nPL2Leht12x1JtQItd/abSylqufH0Fz8zJ46zB3fjiz8fphF0eQES4cFgPvrzxOGLDArn2rUye+majnprxMHrxVB0ia+tern83i4qD9Tx+3kAuzOihR+keJjG6E5/eMJr7P1/PiwvyWbu9gucvTqNzkJ/d0ZQF9Ihd/cKMzCIumfYDAb7efPanMVw0rKeWuocK8PXmsfMG8ug5A1m6uYwzX1xCzo79dsdSFtBiVwDUNzr4+xfruf2jNQxPiOCLP4+hf1edidHTiQiXjujJh9eNoq7Bwbkvf8/n2dvtjqWcpMWuKK+q43f/WsGbS7dw7XEJvHn1MP2VvIMZ2jOcr248nkHdOzP1g2ye+XYjdizCo6yhxd7BbS49wNkvfU/Wtr08fcFg7p2UojMDdlBRIf68+/sRXJAexwvz87npg2xq6hvtjqWOgV487cBWFJYz5Z1MvEX4cMpIXaxB4efjxRPnDyIhKpgnZm9k+95qpv0ug8hO/nZHU61g2aGZiHiLyI8i8pVV21Rt58vVO7h8+nIigv349IYxWurqv0SEG8Ym8s/LhrJ+x37Oful7Nu2utDuWagUrf+eeCmywcHuqDRhjeD+7jBvf/5HBPcL45PrR9OwSZHcs5YJOH9iVGdeNorbBwfmvLCNra7ndkVQLWVLsIhIHnAFMt2J7qm04HIYXvt/NaytKmTSoK+9cO0IvkqojGtyjM59cP5rwIF8um76c+bl6p6o7sOqI/TngdsBxuDeIyBQRyRSRzNLSUot2q1qqvtHBrTOy+TxnLxcMiuCFi4cQ4OttdyzlBnpEBPHR9aPpGx3CH97O4uOsYrsjqaNwuthFZBJQYozJOtL7jDHTjDEZxpiMqKgoZ3erWqGmvpHr383is+wd/H5YFH8cEa1zp6tWiezkz/tTRjKydwS3/Wc10xZttjuSOgIrjtjHAGeJyBbgA+BkEXnXgu0qC1TW1HPl6yuYl1vCQ2encumQSL2TVB2TTv4+vH7VMM4Y1JVHv87lyW9yday7i3K62I0xdxlj4owx8cDFwHxjzOVOJ1NOK6+q47Lpy8naupfnLkrjipG97I6k3Jy/jzcvXDyES4b34KUFm3n06w1a7i5Ix7F7qD0Harls+nIKy6qY9rt0Tk6OsTuS8hDeXsIjZw/E19uL1xYXUtfg4P4zB+jpPRdiabEbY74DvrNym6r1yg7Uctlry9myp4p/XTmM4/rqQsbKWl5ewgNnDcDP24vpSwqpa3TwyNkDtdxdhB6xe5iyA7Vc+toPbCuv5vWrhjEmUUtdtQ0R4Z4z+uPv68VLCzZT12B44vxBeGu5206L3YOUVjaVetHeal6/chijtdRVGxMR/nJqEn7e3jw7Nw8vgcfPG6RH7jbTYvcQPx2pF+89yBtXDWdUny52R1IdhIgwdXxfGk3TUor+vl48NDlVR1/ZSIvdA1RU13PFv1ZQtLdaS13Z5pbxfalrcPDKws34eXtz76T+Wu420WJ3cwdqG7jyjRVsLjnA9CsztNSVbUSEOyYkUdvQyOvfF+Lv68XtpyVpudtAi92N1dQ38oe3Mlm7vYKXLh3KCf30jl5lLxHhvkkp1DU4ePm7zQT4eDN1fF+7Y3U4Wuxuqq7BwQ3vreKHwj08c+FgJqTG2h1JKaCp3B+anEptg4Nn5+YRFujDVWMS7I7VoWixu6FGh+HWGdnMzy3hkXNSOWdInN2RlPoFLy/hsXMHsv9gPX//MofwYD8mp3W3O1aHoWuguRljDA9+uZ6v1uzkronJXDZCpwlQrsnH24sXLhnCiIQIbpuxmoV5Oqtre9FidzMvL9zMW8u28ofjE7juxD52x1HqiAJ8vXntygz6xoTwx3ey+HHbXrsjdQha7G7ko6xinpi9kclp3bhrYn+74yjVIqEBvrx1zTCiQ/25+s2V5JfoMnttTYvdTSzYWMIdH6/huMRInjx/sN7Zp9xKdEgA71wzAl9vL373rxWU7K+xO5JH02J3A6uL9nHDu6tIignh5cuH4uej/9uU++nZJYg3rhrGvoP1XPPWSqpqG+yO5LG0IVxcUXk11761ksgQP968ZhghAb52R1LqmKV2D+PFS4eQs2M/N73/I40Oncu9LWixu7D9NfVc+9ZK6hocvHHVcKJDAuyOpJTTTk6O4YGzBjAvt4QHvlyvC3W0AR3H7qIaGh386b1VFJRW8fY1w0mM7mR3JKUsc8WoeLaVV/Pa4kJ6RgTx++N72x3Jo2ixuyBjDH//cj2LN5Xx2LkDdfpd5ZHumtif4r0HeeTrDcSFB+nd0xbSUzEu6I3vt/DuD9u47oTeXDy8p91xlGoTXl7CsxelMTiuM7d8mE3Ojv12R/IYWuwuZn7ubh6emcOpKTHcMSHZ7jhKtakAX2+mXZFOWKAvf3g7k7IDtXZH8gha7C4kv+QAU9/PJqVbKM9dnKZj1VWHEB0awLTfpVN2oJYb3l1FXYPD7khuz+liF5EeIrJARHJEZL2ITLUiWEezv6aeKe9k4ufjxatXZBDkp5c/VMcxKK4zT5w/iBVbyrnv83U6UsZJVrRHA3CbMWaViIQAWSIyxxiTY8G2OwSHw3DLB9ls21PNe78fQffOgXZHUqrdTU7rTt7uSl5asJnk2BCd6tcJTh+xG2N2GmNWNT+uBDYAOj9nKzw3N495uSXcd2YKI3rrCkiq47rtlCROSYnhoZkbWJpfZncct2XpOXYRiQeGAMt/47UpIpIpIpmlpTp9509mr9vJC/PzuTAjjitG6hS8qmP7aaRM78hgbnz/R3bsO2h3JLdkWbGLSCfgY+BmY8wh45aMMdOMMRnGmIyoKF3CDWDjrkpunbGatB6deVBXdVcKgE7+Prx8eTq1zauE1TY02h3J7VhS7CLiS1Opv2eM+cSKbXq6iuqmi6XB/j68cnk6Ab7edkdSymUkRnfiqQsGkV20j4e+0st1rWXFqBgB/gVsMMY843wkz+dwGG77TzY79h3klcuHEhumc8Ao9WsTUrty3Ym9efeHbXycVWx3HLdixRH7GOAK4GQRyW7+ON2C7Xqs1xYXMHdDCfec3p/0XhF2x1HKZf311CRG9e7C3Z+uZf2OCrvjuA0rRsUsMcaIMWaQMSat+eNrK8J5ohWF5TzxzUbOGNiVK0fH2x1HKZf207qp4UF+XP/uKiqq6+2O5Bb0ztN2VHaglhvfX0WP8EAeO2+gXixVqgWiQvx56bKh7Kw4yC0zsnHoHO5HpcXeThodhps/yGZfdT3/vCxdF8xQqhXSe4Vz76QU5ueW8NriArvjuDwt9nbyj/mbWJJfxoOTB5DSLdTuOEq5nStG9mJiaixPfrORVdv22h3HpWmxt4PFm0p5ft4mzh3anQszetgdRym3JCI8dt4gYsMCuPHfP+r59iPQYm9jJftruPmDbPpGd+Lhs/UmJKWcERboyz8uGcLu/TXc8fEanSzsMLTY25DDYbh1xmqq6hr452VDdcZGpSwwpGc4fz0tidnrd/HuD1vtjuOStNjb0PQlBSzJL+O+SQNIjA6xO45SHuMPx/dmbFIUD83coOPbf4MWextZW1zBk99sZMKAWC4ZrufVlbKSl5fw9AWD6Rzoy43//pGq2ga7I7kULfY2UFXbwE0f/EhkJ38dr65UG+nSyZ/nLk6jcE8V936+zu44LkWLvQ088OV6tuyp4tmL0ugc5Gd3HKU81ug+kdx4UiKfrNrOzDU77Y7jMrTYLfbVmh3MyCzmT2MTGamLZijV5m4c15fBcWHc/eladlXU2B3HJWixW6h4bzV3fbKWtB6dmTq+r91xlOoQfL29ePaiNOoaHPzlP6t1ygG02C3T6DDc8mE2xsALFw/B11u/tUq1l95RnfjbpP4syS/jzaVb7I5jO20fi0xfXMDKLXt54KwB9OwSZHccpTqcS4f3ZFxyNI/NziVvd6XdcWylxW6BjbsqefrbPE4bEMO5Q3Udb6Xs8NOUAyH+Pkz9ILtDL6mnxe6kugYHt87IJiTAh0fP0aGNStkpKsSfx88bxIad+3lmTp7dcWyjxe6kf8zfxPod+/m/cwfSpZO/3XGU6vDGp8RwyfCeTFtUwA8Fe+yOYwstdidkF+3jn99t5ryhcZw6INbuOEqpZvdO6k+viCD++tHqDnlXqhb7Maqpb+TWGdnEhPhz/1kpdsdRSv1MkJ8PT14wmOK9B3l8dq7dcdqdFvsxenx2LgWlVTx5wWBCdTUkpVzOsPgIrh6dwNvLtrJ0c5ndcdqVJcUuIhNEZKOI5IvInVZs05Ut3VzGG99v4arR8YxJjLQ7jlLqMP56WhLxXYK44+M1HeqUjNPFLiLewEvARCAFuEREPPbcRHVdA3d8vIaEyGDumJBsdxyl1BEE+nl3yFMyVqz8MBzIN8YUAIjIB8BkIOdwX1BbW0tBgXsuSPvS0t0UlR/kuTN7sbPY/Sb5r6lpmkvDXb//SrVWF+DcARG8vWwrg7sYhnQLtjtSm7PiVEx3oOhnnxc3P/cLIjJFRDJFJLO+3j3XKly/u5pP1pUzOSWcQV317lKl3MW1w6PoHurLUwt3crDeYXecNtdua7UZY6YB0wAyMjJM796922vXlqhtaGTKZ0vo1jmQRy4aTid/91zm7qcjdXf7/ivlrOcujeDCV5fxYW4ND05OtTtOm7LiiH078PMlguKan/Mo/5iXT37JAR49d6DblrpSHdnPR8l4+o1LVhT7SqCviCSIiB9wMfCFBdt1Get3VPDywqYbkU7sF2V3HKXUMfrraUn0jAji7k/WUlPvuXPJOF3sxpgG4M/AN8AGYIYxZr2z23UV9Y0Obv9oDeFBftw7qb/dcZRSTgj08+aRc1IpKKvixfn5dsdpM5aMYzfGfG2M6WeM6WOMecSKbbqKaYsKWL9jPw+fPUCXuVPKAxzfN4pzh3bnlYWbyd213+44bULvPD2CzaUHeH7eJiamxjIhtavdcZRSFvnbGSmEBvpy58drafTAFZe02A/DGMPdn6wlwMeLByYPsDuOUspCEcF+3Dcpheyifbz7g/vdj3I0WuyH8fGq7SwvLOfOif2JDgmwO45SymKT07pxQr8onpidy459B+2OYykt9t9QXlXHIzNzSO8VzsXDehz9C5RSbkdEeOTsVBwG7vt8HcZ4zikZLfbf8H9fb6CypoFHzxmIl5euiKSUp+oREcStp/Rj7oYSvl67y+44ltFi/5XlBXv4T1Yxvz++N0mxIXbHUUq1savHxJPaPZT7v1hPxUH3nO7k17TYf6a2oZG7P11LXHggU8f1tTuOUqod+Hh78di5gyivquVZD1knVYv9Z6YtLGBzaRUPnZ1KoJ+33XGUUu0ktXsYl4/sxdvLtrBue4XdcZymxd5sS1kV/1iQzxkDu3JSUrTdcZRS7ey2U5OICPbjb5+tw+HmY9u12Gkas37v5+vw9/bivjM9do0QpdQRhAX6cs8Z/cku2seHmUVH/wIXpsUOfL12F4s3lfGX05KICdUx60p1VGendWdEQgSPzcplz4Fau+Mcsw5f7FW1DTw8M4eUrqFcPrKX3XGUUjYSER46O5Wq2ga3Xkqvwxf7Swvy2VlRw4OTB+CtY9aV6vD6xYRw7fEJzMgsJmtrud1xjkmHLvaC0gO8triAc4d2JyM+wu44SikXcdPJfekaFsA9n66jodH9ltLrsMVujOGBL3MI8PHmzonJdsdRSrmQYH8f7j8zhdxdlby1zP0mCeuwxT4nZzcL80q5+ZR+OsmXUuoQpw2I5cR+UTw3J4/SSve6kNohi72mvpEHv8ohKSaEK0fpBVOl1KFEhPvOTOFgfSNPfuNeF1I7ZLG//N1mivce5O9nDcDHu0N+C5RSLdAnqhNXj4nnP1nFrCneZ3ecFutwrbZtTzUvL9zMmYO7MapPF7vjKKVc3I3j+tIl2I+/f7Hebab27XDF/tDMHHy8hHtO14WplVJHFxrgy+2nJbNq2z4+y95ud5wWcarYReRJEckVkTUi8qmIdLYqWFtYsqmMOTm7+fPJicSG6QVTpVTLnJ8ex6C4MB6blUtVbYPdcY7K2SP2OUCqMWYQkAfc5XykttHQ6OChr3LoERHINWMS7I6jlHIjXl7C/WcOYPf+Wl5akG93nKNyqtiNMd8aY3765+sHIM75SG3jw8wiNu6u5O6J/Qnw1Sl5lVKtk94rnHOGdGf64kK27qmyO84RWXmO/Rpg1uFeFJEpIpIpIpmlpaUW7vbo9tfU8/S3eQxPiGBCamy77lsp5TnunJiMj7fw8MwNdkc5oqMWu4jMFZF1v/Ex+WfvuQdoAN473HaMMdOMMRnGmIyoqChr0rfQi/Pz2Vtdx32TUhDR+WCUUscmJjSAP52UyJyc3Sze1L4HqK3hc7Q3GGPGH+l1EbkKmASMMy44FmhLWRVvfF/IBelxpHYPszuOUsrNXXtcAh+uLOLhrzbw9dRIl5w80NlRMROA24GzjDHV1kSy1qNfb8DP24u/nJpkdxSllAcI8G2aX2rj7kr+46ILcjh7jv1FIASYIyLZIvKKBZksszS/jG9zdnPDSYlE6wIaSimLTEyNJb1XOE/PyXPJ4Y/OjopJNMb0MMakNX/80apgzmp0GB78KofunQO59jgd3qiUso6IcM8Z/SmtrOXVhZvtjnMIj73z9MOVReTuquTu03V4o1LKekN7hjNpUFemLS5gZ8VBu+P8gkcWe1VtA8/MySOjVzinD9ThjUqptnHHhGQcDnjqmzy7o/yCRxb7a4sLKDtQy91n9NfhjUqpNtMjIoirx8TzyY/FrNteYXec//K4YoWeuEMAAAkQSURBVC+prGHaogJOHxjL0J7hdsdRSnm4G05KpHOgL4/M3OAysz96XLE/P3cTdQ0O/nqaLnenlGp7YYG+3Dy+H8sK9jBvQ4ndcQAPK/b8kgN8sLKIy0b0JCEy2O44SqkO4tIRPekdFcyjszZQ7wKLX3tUsT8+O5dAX29uGtfX7ihKqQ7E19uLuyb2p6C0ig9X2n/TkscU+4rCcubk7Ob6sX3o0snf7jhKqQ5mfP9ohsWH8/y8TVTX2XvTkkcUuzGGR7/eQEyov861rpSyhYhw58RkSitreX1Joa1ZPKLYZ63bRXbRPm47JYlAP70ZSSllj/ReEZySEsOrCwvYW1VnWw63L/a6BgdPzM4lKSaE89Jddp0PpVQHcftpSVTVNdi60pLbF/sHK7exZU81d05MdsnpM5VSHUvfmBDOT4/j7WVbKd5rz6S3bl3s1XUNvDAvn+EJEYxNat/FO5RS6nBuHt8PBJ6ds8mW/bt1sb/x/RbKDtRyx4QknTpAKeUyunUO5OrRTVMN5O7a3+77d9tir6iu59WFmxmXHE16rwi74yil1C9cP7YPIf4+PDl7Y7vv222L/dVFm9lf08BfTtOVkZRSrqdzkB9/HNuHebklrNxS3q77dstiL6ms4Y3vt3DW4G707xpqdxyllPpNV49OICbUn8dm5bbrBGFuWewvzs+nvtHBraf0szuKUkodVqCfNzeP70fW1r0s2Nh+E4S5XbEXlVfz/optXDisB/E60ZdSysWdnx5Hry5BPP1tHg5H+xy1u12xPzsnDy8RbjpZJ/pSSrk+X28vbh7fl/U79vPN+l3tsk9Lil1EbhMRIyKRVmzvcPJ2V/Jp9nauGh1PbFhAW+5KKaUsc9bg7iRGd+LpOXk0tsNRu9PFLiI9gFOBbc7HObKnvtlIJz8f/nhin7belVJKWcbbS7j1lH7klxzgi9Xb23x/VhyxPwvcDrTpP0Nrivfxbc5u/nBCb8KD/dpyV0opZbkJA2JJ6RrKc3M3tfliHE4Vu4hMBrYbY1a34L1TRCRTRDJLS0tbva9n5+TROciXq8fEH0NSpZSyl5eXcNup/di6p5qPs4rbdl9He4OIzBWRdb/xMRm4G7ivJTsyxkwzxmQYYzKiolo3r8uqbXtZsLGUKSf0JiTAt1Vfq5RSruLk5GiG9OzMC/M2UdvQ2Gb7OWqxG2PGG2NSf/0BFAAJwGoR2QLEAatEJNbqkM/N3UREsB9Xjoq3etNKKdVuRIS/nJrEjooaPljRdkvoHfOpGGPMWmNMtDEm3hgTDxQDQ40xlo7nydpazqK8Uq47oTfB/j5Wbloppdrd6D5dGNk7ghcX5HOwrm2O2l1+HPuzczYR2cmPK0b1sjuKUko5TUS47dQkSitreWvZljbZh2XF3nzkXmbV9qBpgeol+WX88cQ+BPnp0bpSyjMMi4/gxH5RTFtUQFWt9Qtfu/QR+7Nz8ogK8eeyEXq0rpTyLDeP70t5VR3v/LDV8m27bLEv27yHZQV7uP7EPrpAtVLK4wzpGc4J/aJ4bVEB1XXWHrW7ZLEbY3h2bh4xof5cOqKn3XGUUqpNTB3Xlz1Vdbxr8VG7Sxb70s17WFFYzg1jEwnw1aN1pZRnSu8VzvF9I5m2qMDSETIuV+zGGJ6dk0fXsAAuGtbD7jhKKdWmpo7rS9mBOt5bbt1Ru8sV++JNZWRu3csNJ+nRulLK82XERzAmsQuvLLTuqN2lit0Yw3Nz8+gWFsCFGXF2x1FKqXYxdVw/yg7U8u8V1kyS61LFvmzzHlZt28cNJyXi76NH60qpjmF4QgSjenfhlYWbqal3/qjdpYr9H/PziQ7x5/x0PVpXSnUsU8f3pbSylvctOGqX9lw5+787FakENrb7jq0TCVh6l207c+f87pwdNL/d3D1/kjEm5Ghvsus+/Y3GmAyb9u00EcnU/PZw5+yg+e3mCflb8j6XOhWjlFLKeVrsSinlYewq9mk27dcqmt8+7pwdNL/dOkR+Wy6eKqWUajt6KkYppTyMFrtSSnkY24pdRJ4UkVwRWSMin4pIZ7uyHAsRuUBE1ouIQ0TcYviUiEwQkY0iki8id9qdpzVE5HURKRGRdXZnORYi0kNEFohITvPPzVS7M7WGiASIyAoRWd2c/wG7M7WWiHiLyI8i8pXdWVpLRLaIyFoRyW7JkEc7j9jnAKnGmEFAHnCXjVmOxTrgXGCR3UFaQkS8gZeAiUAKcImIpNibqlXeBCbYHcIJDcBtxpgUYCTwJzf7/tcCJxtjBgNpwAQRGWlzptaaCmywO4QTTjLGpLVkHL5txW6M+dYY89OyIT8AbjWPgDFmgzHGne6eHQ7kG2MKjDF1wAfAZJsztZgxZhFQbneOY2WM2WmMWdX8uJKmgulub6qWM00ONH/q2/zhNiMvRCQOOAOYbneW9uAq59ivAWbZHcLDdQeKfvZ5MW5ULJ5EROKBIcBye5O0TvOpjGygBJhjjHGn/M8BtwMOu4McIwN8KyJZIjLlaG9u0ykFRGQuEPsbL91jjPm8+T330PRr6nttmeVYtCS/Uq0hIp2Aj4GbjTH77c7TGsaYRiCt+XrYpyKSaoxx+WseIjIJKDHGZInIWLvzHKPjjDHbRSQamCMiuc2/xf6mNi12Y8z4I70uIlcBk4BxxgUH1B8tv5vZDvx8Saq45udUOxERX5pK/T1jzCd25zlWxph9IrKApmseLl/swBjgLBE5HQgAQkXkXWPM5TbnajFjzPbmP0tE5FOaTq0ettjtHBUzgaZfjc4yxlTblaMDWQn0FZEEEfEDLga+sDlThyEiAvwL2GCMecbuPK0lIlE/jVwTkUDgFCDX3lQtY4y5yxgTZ4yJp+nnfr47lbqIBItIyE+PgVM5yj+odp5jfxEIoenXimwRecXGLK0mIueISDEwCpgpIt/YnelImi9U/xn4hqYLdzOMMevtTdVyIvI+sAxIEpFiEbnW7kytNAa4Aji5+ec9u/kI0l10BRaIyBqaDhLmGGPcbtigm4oBlojIamAFMNMYM/tIX6BTCiillIdxlVExSimlLKLFrpRSHkaLXSmlPIwWu1JKeRgtdqWU8jBa7Eop5WG02JVSysP8P0a4nn/V1Q74AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "isgDTQNi5b2V"
      },
      "source": [
        "#### Saddle point"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EID56T7T5b2V"
      },
      "source": [
        "If $y = x^3 + 6$..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aoZghxgI5b2W"
      },
      "source": [
        "y_sp = x**3 + 6"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NzU1HTKQ5b2Y"
      },
      "source": [
        "...then $y' = 3x^2$. \n",
        "\n",
        "Critical point is located where $y' = 0$, so where $3x^2 = 0$.\n",
        "\n",
        "Rearranging to solve for $x$:\n",
        "$$x^2 = \\frac{0}{3} = 0$$\n",
        "$$x = \\sqrt{0} = 0$$\n",
        "\n",
        "At which point, $y = x^3 + 6 = (0)^3 + 6 = 6$\n",
        "\n",
        "Thus, the critical point is located at (0, 6)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7isUxig-5b2Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        },
        "outputId": "7680e12d-4d8a-4f8b-a371-d285c6b2b704"
      },
      "source": [
        "fig, ax = plt.subplots()\n",
        "\n",
        "plt.scatter(0, 6, c='orange')\n",
        "plt.axhline(y=6, c='orange', linestyle='--')\n",
        "\n",
        "plt.axvline(x=0, c='lightgray')\n",
        "plt.axhline(y=0, c='lightgray')\n",
        "\n",
        "ax.set_xlim(-2, 2)\n",
        "ax.set_ylim(-1, 10)\n",
        "\n",
        "_ = ax.plot(x, y_sp)"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhV5b328e8vMxkYQsIYpjCjoEJk0loU21rnVmudKlaRWtujtnawx/bYY0dt39a22sEqra2IVq1WqRbnEQWCEOZACFMSICGQkISMO8/7RzYeBAJJ9s5ee7g/15Ura2ft4WYluVl51lrPNuccIiISfeK8DiAiIt1DBS8iEqVU8CIiUUoFLyISpVTwIiJRSgUvIhKlTljwZjbfzMrNbO1hX8s0s1fMbLP/c5/ujSkiIp3VkT34vwLnHfG1O4HXnHOjgdf8t0VEJIxYRy50MrPhwCLn3Mn+24XALOfcLjMbCLzpnBvbnUFFRKRzErr4uP7OuV3+5d1A//buaGbzgHkAaWlpU8aNG9fFlxTpHo2NjQAkJyd7nETCgQM276kBYHS/DMy8zQOwYsWKvc657M4+rqsF/xHnnDOzdv8McM49BDwEkJeX5/Lz8wN9SZGgKi4uBiA3N9fjJBIO5r+7lXsWreeROXnMHt/uvmtImdn2rjyuq2fR7PEPzeD/XN7F5xERCRv76pq4/9VNfGJ0FueM6+d1nIB1teCfB+b4l+cA/wpOHBER79z/6iZqG1v4wYUTsHAYmwlQR06TXAi8D4w1sxIzuxH4OfApM9sMnOu/LSISsTbtqWHB0h1cM20YY/pneB0nKE44Bu+cu6qdVbODnEVExBPOOe55YT1pSfF841NjvI4TNLqSVURi3otrdvNu0V7u+PRYMtOSvI4TNCp4EYlpdY0t/GjReiYM7Mk104Z6HSeoAj5NUkQkkv329c3sPtDAg9dMJiE+uvZ5o+tfIyLSCUXlNTzyzlauyMthyrDom1JLBS8iMck5x93PryM1KZ7vnhedV9ir4EUkJj27spT3iir59nnj6JsendNUqOBFJObsq2viR4vWM2VYH66ZGl0HVg+ngheRmPPjReupbWzhZ5+fSFxc5F+x2h4VvIjElLc3VfDPlaXc/MmRUXPFantU8CISM+qbfNz13Bpys9L42tmjvI7T7XQevIjEjF++XMjOffUsvGk6KYnxXsfpdtqDF5GYsHzbPua/t5Vrpw9lxsi+XscJCRW8iES9+iYf336qgMG9e/C9z473Ok7IaIhGRKLefYs3sq3yIAtvmk5acuzUnvbgRSSqLS2u5K9LtjFnxrCYGZo5RAUvIlGrpqGZbz1dwJA+qXz3s9E5HcHxxM7fKiISc+5+fh2l++t56uYZpCbFXt1pD15EotILBWX888NSvn7OaKYMy/Q6jidU8CISdUqr6rnr2TWcNrQ3t54T/Rc0tUcFLyJRxdfq+OaTq/C1Ou7/4qlR9yYenRF7g1IiEtV+9/pmlm7dxy8un8Swvmlex/FU7P7XJiJRZ0nRXn7z2mY+P3kwl0/J8TqO51TwIhIVymsauPWJVYzMTufHl56MWfROA9xRGqIRkYjna3Xc/sQqahubWTB3WkyeEnks2goiEvF+/comlmyp5L7LJzF2QHTP8d4ZGqIRkYi2eN1uHnijiCvycviCxt0/RgUvIhGrqLyGO/5RwCk5vbjnEo27H0kFLyIR6UBDM/P+voKUxDj+cO2UmHgDj87SGLyIRJxDFzPtqDzIgrnTGNS7h9eRwpL24EUk4vz8pQ28uqGc/7loAtNyY2sK4M5QwYtIRFm4bAd/fmcrc2YM47oZw72OE9YCKngz+4aZrTOztWa20MxSghVMRORI7xXt5QfPrWXW2Gx+cOEEr+OEvS4XvJkNBm4F8pxzJwPxwJXBCiYicrhNe2r46mMrGJmdzu+uOi2mJxHrqEC3UALQw8wSgFSgLPBIIiIfV1ZVz5z5y0hJjOeR6/PISEn0OlJE6HLBO+dKgV8CO4BdQLVz7uUj72dm88ws38zyKyoqup5URGJS1cEm5sxfRm1DC4/eMJWcPqleR4oYgQzR9AEuAUYAg4A0M7v2yPs55x5yzuU55/Kys7O7nlREYk5Ds4+5j+azvfIgD12Xx/iBPb2OFFECGaI5F9jqnKtwzjUD/wRmBieWiMS6xhYfNz+2ghU79nP/lacyY6ROh+ysQAp+BzDdzFKt7frg2cCG4MQSkVjW7Gvl1oUrebOwgp9cOpHzJw70OlJECmQMfinwNPAhsMb/XA8FKZeIxChfq+OOfxSweN0e7r5oAldPG+p1pIgV0FQFzrm7gbuDlEVEYpyv1fHdZ1bzfEEZ3z1vHF8+Y4TXkSKa5qIRkbDQ4mvl20+v5tmVpdw2ezRfnTXS60gRTwUvIp5r9rVy+xOr+PeaXXzr02P4+jmjvY4UFVTwIuKpxhYf//X4Sl5ev4e7zh/PTWfleh0paqjgRcQzNQ3NzPvbCt4vruSHF03geo25B5UKXkQ8UV7TwPXzl7NpTw2//uIpfO40vd1esKngRSTktu2t47r5y6ioaeThOXnMGtvP60hRSQUvIiG1tLiSmx9bAcDjN03jtKF9PE4UvVTwIhIyT+Xv5L+fXcOQzFTmzzmd4VlpXkeKaip4Eel2vlbHLxYX8se3tnDGqL78/uop9ErVlL/dTQUvIt1qX10Tty5cybtFe7lm2lB+ePFJJOrNOkJCBS8i3aZgZxW3LPiQitpG7r1sIl88XfPKhJIKXkSCzjnHo0u28dMXN5KdkcwzN89kYk4vr2PFHBW8iARVZW0j33l6Na9tLOeccf34f184hT5pSV7HikkqeBEJmrc3VfCtpwqoOtjM3RdN4PqZw2l7uwjxggpeRAJW09DMT1/cwMJlOxmZncZfvzyVCYP09npeU8GLSEDe2VzBnc+sYVd1PV85K5dvfGoMKYnxXscSVPAi0kXlNQ385N8b+NeqMnKz03jq5plMGaarUsOJCl5EOsXX6nh86XbuW1xIY3Mrt84ezS2zRmqvPQyp4EWkQ5xzvFlYwc9e2sCmPbWcMaovP7rkZHKz072OJu1QwYvICa0treanL25gyZZKhvdN5Q/XTOa8kwfoDJkwp4IXkXZt3VvHb17dxHOryuiTmsgPL5rA1dOGkZSgqQYigQpeRI6yYdcBfv/mFv69uozE+Di+OmskX501kp4pmiAskqjgReQjH+7Yz4OvF/HaxnLSkxOYd9ZIbjhzOP0yUryOJl2ggheJcQ3NPl4oKOOxpTso2FlF79REvvmpMcyZMVxT+kY4FbxIjCquqGXB0h08vaKE6vpmRvVL5+6LJnBF3hDSklUN0UDfRZEYUlHTyL9Xl/F8QRkf7qgiIc74zMkDuHbaMKbnZuqsmCijgheJcvvqmnh1/R6eLyhjyZa9tDoYNyCD75w3lsun5Gh8PYqp4EWiTGurY/2uA7y+sZw3CstZtbMK52BoZiq3zBrFxacOYkz/DK9jSgio4EUiXGuro6iilmVb97Fs6z7eL66koqYRM5iU05vbZo/mnHH9mDi4l4ZgYowKXiTClB9oYF3ZAdaVVbNqZzX52/dRdbAZgH4ZyUzP7cusMdl8cmw2WenJHqcVLwVU8GbWG3gYOBlwwA3OufeDEUwkJLYugMo48DXA6nPglJ/AiGu8ToVzjoqaRor31lFcUUdxRS2by2tZV3aAvbWNH91vRFYan57Qn9OHZzJ1RCZDM1O1ly4fCXQP/jfAf5xzl5tZEpAahEwiobF1ASybB8MeaLt9cHvbbejWkq9v8rHvYBP765rYV9fE3tpGdlU3UFZVT1lVPbuqGyjZX09tY8tHj0lOiGNEVhqfHJPNSYN6cvLgXowfmEGGriyV4+hywZtZL+As4HoA51wT0HTcBx0ohFdnffxrQ6+AMbdAy0F48/yjH5N7fdtHw1549/Kj14/+Kgz7ItTthPe/dPT6cXdAzkVtr73sK0evP/n7MOBc2L8KVtx+9PpTfgrZM6FiCRT899Hrp9wPfU6F3a/C2h8fvX7qn6DnWCh5ATb+v6PXz/g7pA2B7U/C5j8cvf7MpyElC4r/2vZxpFkvQkIqbPo97PjH0evPfbPt84ZfQumij6+L7wFnv9S2vOZHsOe1j69P7gufeKZtedX3YO8Rf5yl5sDMx9qWV9zetg0PlzEGpj3Utrx0HtRs+vj6Pqe2bT+AJdfCwZKPr8+aAaf+rG35ncugsRIAnzPqfIk0ZJ5N85jbaW5ppfn9m2hqaaG5NY5mF09TazzNmTNwQy5r+9ty1XdxGM7R9hmgdjuuZSK7E5NwwKbqGQC4xX/GZbyFr9/ZtGafha/xAL7CB2h1cfic4SMO5wxf1ifw9T6V1sYqmrc+Sb0vkYO+ROpbE9uWU8dTH59NfX0tdVU7qGpOYV9zDxpbj13KfRIPMjC5hpyUGqZlHyB3zBmMGD6R3PiNDCr+AXGHdswbgC1A7/shRT97ofzZ+0j/2TDxB23Lb3wWfPUfXz/4Qhj/rbblIzsPAu+9DgpkD34EUAH8xcxOAVYAtznn6g6/k5nNA+YBTBqu8UD5uMYWH7uqGqiozKKypgcVTWlUNqWytymVSoawb+n71DW1UFd1GTXNCdS1JFLfevgbOL/u/9zeHvdy/+fj/JJsP7Rw1zFWHiqOY/wSFgJs8N84izhaSY1vpkd8Mz3iW0hthpTUFlLjjd4pBzgpYw+ZSfX0SawnM7GePqMuJHPw6WT6tjCw6Hv0iG/5+POfMhuys6FiM2jURbrAnHNde6BZHvABcIZzbqmZ/QY44Jz7QXuPycvLc/n5+V1LKhGrvsnHlopaNpfXUFReS8n+ev/HQfYcaDzmY3qnJtI3LYnMtCTSkxNIT0kkPTme9OQE0pITSE9OoEdSPInxcSTFx5EYH0dCvH20nBhvJMQbZobBR+PSbctgGPbWhVC/i9KcuzEgp/Tutn37lIHY7P8Qb0ZcnBFvRnzc/y3HxUGc2f+tjzMS4ozkhDiNf0u3MLMVzrm8zj4ukD34EqDEObfUf/tp4M4Ank+iQEVNI6tLqigoqWZtaTWby2so2V/Pof2I+DhjUO8UcnqnctbobHL6pDK4Tw/6ZSTTNz2JrPRkMtOSSIwPwXS0M2+GZfNI7d2255y7byvEp8LUH8MAvWG0RL4uF7xzbreZ7TSzsc65QmA2sD540STcOefYtKeWJVv2smzrPgp2VlFW3QBAnMGofumcOqQPX5gyhNH90hndP51hfdNCU94dcehA6l7/WTSpw8LmLBqRYAj0LJr/Ahb4z6ApBr4ceCQJZ2VV9bxRWM6SLZV8sKWSyrq24+o5fXowZXgmN+T0YlJOb04a1DMyJqwacQ244rbl6ds8jSISbAH9BjrnVgGdHheSyOGcY8OuGl5Zv4eX1+9mXdkBAAb0TOGTY7KZPrIvM3L7MiRTZ8iKhJsI2MUSLxSV1/LPD0t4vqCMkv31mMFpQ3pz52fHce74/ozMTtMBRZEwp4KXj+yva+KF1WU8s6KEgpJq4uOMM0dl8fWzRzF7fH+yM3Saq0gkUcELq3ZW8df3tvLvNbto9jkmDOzJ9y8Yz8WnDtJUsiIRTAUfo5p9rby4Zhd/eW8bq3ZWkZGcwLXTh3FF3hDGD9QpgiLRQAUfY+qbfCxYup0/v1PMngONjMhK438vPonLpuSQHglnvYhIh+k3OkYcKvY/vrWFvbVNzBzZl59/fhKfHJNNXJwOlopEIxV8lGto9vHYB9v541vF7K1t5MxRWdx27mhOH57pdTQR6WYq+CjlnOPFNbv56YsbKK2q54xRffn97MlMHaFiF4kVKvgotLa0mnsWrWfZ1n2MG5DB43OnMXNUltexRCTEVPBRpLK2kV8sLuTJ/J30SU3iJ587mStPH0q8xthFYpIKPgo453huVSn3vLCemoYWbjhjBLfOHk2vHnq3H5FYpoKPcCX7D3LXs2t5a1MFpw3tzb2XTWJM/wyvY4lIGFDBRyjnHE+tKOF/n18HwP9efBLXTh+m4RgR+YgKPgJV1jby38+uYfG6PUzPzeSXXziFnD6azVFEPk4FH2He3byX259cxYH6Zr5/wXhuOGOELlQSkWNSwUcIX6vjN69t5nevb2ZUdjqPzZ3KOL2tnIgchwo+AlTUNHLbEytZsqWSz08ezI8vPZnUJH3rROT41BJhbnVJFfP+toKq+ibuu3wSV+QN8TqSiEQIFXwYe25lKd99ZjVZ6ck889WZnDSol9eRRCSCqODDUGur497FG/nTW8VMHZHJH66ZTN90vZuSiHSOCj7MNDT7uOMfBfx7zS6umTaUH158EonxcV7HEpEIpIIPI/vrmrjpb/nkb9/PXeePZ+4nRuiNrUWky1TwYaJk/0Gue2QZJVX1PHj1ZC6YNNDrSCIS4VTwYaCovJYvPbKUusYWHrtxmuZsF5GgUMF7bG1pNdfNX0acwRPzZjBhkC5eEpHgUMF7KH/bPr78l+X07JHIY3OnMSIrzetIIhJFVPAeyd+2jznzl9G/ZwqPzZ3GoN49vI4kIlFGBe+Bw8v9iXnT6dczxetIIhKFdIJ1iK3Y3lbu/XqmsFDlLiLdSAUfQmtLq7l+/vK2cr9pOv1V7iLSjVTwIbKlopY585fRs0ciC+ZOY0AvlbuIdK+AC97M4s1spZktCkagaFRWVc+XHl6KGfz9xqk6oCoiIRGMPfjbgA1BeJ6otK+uiWsfWUpNQwuP3jCV3Ox0ryOJSIwIqODNLAe4AHg4OHGiS0Ozj7mPLqd0fz2PXH+6pvsVkZAKdA/+fuA7QGt7dzCzeWaWb2b5FRUVAb5c5GhtdXzjyVWs3FnF/V88VdMPiEjIdbngzexCoNw5t+J493POPeScy3PO5WVnZ3f15SLOT1/cwEtrd3PX+eP57ERNHCYioRfIHvwZwMVmtg14AjjHzB4LSqoI9+iSbTz87launzmcG88c4XUcEYlRXS5459z3nHM5zrnhwJXA6865a4OWLEK9u3kv9yxaz7nj+/ODCydoPncR8YzOgw+ibXvr+NrjHzIyO437rzyV+DiVu4h4Jyhz0Tjn3gTeDMZzRaqahmbm/i0fM3j4utNJT9Y0PyLiLbVQELS2Om5/YhVb99bx9xunMrRvqteRREQ0RBMMD7xRxGsby/mfCycwc2SW13FERAAVfMDe2VzBr1/dxOdOG8x1M4Z5HUdE5CMq+ACUVtVz68KVjOmXwU8+d7LOmBGRsKKC76LGFh+3LPiQZp/jD9dOJjVJhzNEJLyolbrovv8UUrCzij9cM1kTiIlIWNIefBe8sbGcR97dypwZwzQNgYiELRV8J+050MAdTxUwbkAG3zt/vNdxRETapYLvBJ9/hsj6Jh8PXH0aKYnxXkcSEWmXxuA74U9vb2HJlkruvWwio/pleB1HROS4tAffQevKqvn1K5s4f+IArsgb4nUcEZETUsF3QEOzj28+WUDv1CR+culEne8uIhFBQzQd8OtXNlG4p4a/fPl0+qQleR1HRKRDtAd/Asu27uOhd4q5etpQzh7bz+s4IiIdpoI/jrrGFu54ahVDM1O5S6dEikiE0RDNcfxicSEl++v5x1dmkKb53UUkwmgPvh352/bx6PvbuG76ME4fnul1HBGRTlPBH0NDs4/vPLOaQb168J3zxnkdR0SkSzTucAy/fW0zxRV1/O2GqRqaEZGIpT34I6wtreZPbxfzhSk5nDUm2+s4IiJdpoI/TGur467n1tInNZHvXzDB6zgiIgFRwR/mieU7KdhZxV0XjKdXaqLXcUREAqKC96usbeTe/2xk2ohMLj11sNdxREQCpoL3+/lLG6lrbOHHl+q9VUUkOqjggeXb9vHUihLmfiKX0f01DbCIRIeYL/hmXyvff3Ytg3qlcOvsUV7HEREJmpg/yfvRJdso3FPDn740hdSkmN8cIhJFYnoPfnd1A79+ZRPnjOvHpyf09zqOiEhQxXTB3/ufjTS3On540Uk6sCoiUSdmC75gZxXPrixl7pkjGNo31es4IiJB1+WCN7MhZvaGma03s3Vmdlswg3Un5xw/WrSerPQkbjlbB1ZFJDoFsgffAtzhnJsATAe+ZmYRcX3/S2t3k799P3d8eizpmkxMRKJUlwveObfLOfehf7kG2ACE/SWgDc0+fvbSBsYNyOCKvCFexxER6TZBGYM3s+HAacDSY6ybZ2b5ZpZfUVERjJcLyKNLtrFzXz3fv2AC8XE6sCoi0SvggjezdOAZ4Hbn3IEj1zvnHnLO5Tnn8rKzvZ1+d29tIw+8XsTscf04c3SWp1lERLpbQAVvZom0lfsC59w/gxOp+9z/6iYONvv4nt5AW0RiQCBn0RjwCLDBOfer4EXqHpv31PD40h1cO20oo/qlex1HRKTbBbIHfwbwJeAcM1vl/zg/SLmC7r7FhaQlJXDbuWO8jiIiEhJdPkfQOfcuEBFHKVfu2M8r6/dwx6fGkJmW5HUcEZGQiIkrWX/5ciF905L48pkjvI4iIhIyUV/w7xXt5b2iSm45e5QuahKRmBLVBe+c4xeLCxnUK4Vrpg31Oo6ISEhFdcG/uqGcVTuruO3c0aQkxnsdR0QkpKK24FtbHb9cXMiIrDQum5zjdRwRkZCL2oJ/YXUZhXtq+OanxpAQH7X/TBGRdkVl8zX7WvnVK5sYP7AnF0wc6HUcERFPRGXBP72ihO2VB/n2Z8YQpwnFRCRGRV3BN/taefCNIk4Z0puzx/bzOo6IiGeiruCfW1lKyf56bj1nlN5nVURiWlQVvK/V8fs3tzBhYE/OGae9dxGJbVFV8ItWl7F1bx3/pb13EZHoKfjWVseDbxQxpn86nzlpgNdxREQ8FzUFv3jdbjbtqeVrZ4/SmTMiIkRJwTvn+N3rRYzISuPCSYO8jiMiEhaiouBf31jO+l0HuGXWSL2RtoiIX8QXvHOO375eRE6fHlx62mCv44iIhI2IL/h3i/ZSsLOKW2aNIlFzzoiIfCTiG/GPb22hf89kLpuivXcRkcNFdMGvLa3mvaJKbjhjBMkJmu9dRORwEV3wf36nmPTkBK7SuzWJiBwlYgu+tKqeRat3ceXpQ+iZkuh1HBGRsBOxBT//3a0YcMOZI7yOIiISliKy4Kvrm3li2Q4unDSQQb17eB1HRCQsRWTBP750B3VNPm46K9frKCIiYSviCr6ppZW/vLeVM0dlcdKgXl7HEREJWxFX8M8XlFFe06i9dxGRE4iognfO8ee3ixk3IIOzRmd5HUdEJKxFVMG/tamCwj013PSJXL2hh4jICURUwc9/bxv9MpK56BRNCSwiciIRU/BbKmp5e1MF104fRlJCxMQWEfFMQE1pZueZWaGZFZnZncEKdSx/f387SfFxXDVV0xKIiHRElwvezOKBB4HPAhOAq8xsQrCCHa62sYWnV5RwwaSBZGckd8dLiIhEnYQAHjsVKHLOFQOY2RPAJcD69h7Q2NhIcXFxp1/ouXX7qG1s4dxhiV16vMjxNDQ0AOhnS6JOIEM0g4Gdh90u8X/tY8xsnpnlm1l+c3Nzp1/EOcdz6/YzNjuF8f00LYGISEcFsgffIc65h4CHAPLy8lxubucuUHpncwU7qpr41RWnkJub0x0RJcYd2nPv7M+mSLgLZA++FBhy2O0c/9eC6tEl28hKT+KCSQOD/dQiIlEtkIJfDow2sxFmlgRcCTwfnFhtdlQe5LWN5Vw1dajesUlEpJO6PETjnGsxs68Di4F4YL5zbl3QkgF//2Ab8WZcM21YMJ9WRCQmBDQG75x7EXgxSFk+5mBTC08u38lnTh7AgF4p3fESIiJRLWwvCf3XqjIONLQwZ8Zwr6OIiESksC34x5fuYNyADE4f3sfrKCIiESksC35NSTVrSqu5etpQzRopItJFYVnwC5fvICUxjktOPeq6KRER6aCwK/i6xhb+tbKUCycNolePRK/jiIhErLAr+BcKyqhr8mnWSBGRAIVdwS9ctoOx/TOYPLS311FERCJaWBX82tJqCkqquWrqEB1cFREJUFgV/BPLd5CcEMfnTtOkYiIigQqbgj/Y1MJzK8u4YNJAeqXq4KqISKDCpuAXFeyitrGFq3VwVUQkKMKm4B9ftoPR/dKZMkxXroqIBENYFPz6sgOs2lnFVVN15aqISLCERcE/uXwHSQlxfH6yrlwVEQkWc86F7sXMaoDCkL1g12UBe70O0QGRkDMSMoJyBptyBtdY51xGZx/U7e/JeoRC51xeiF+z08wsXzmDIxIygnIGm3IGl5nld+VxYTFEIyIiwaeCFxGJUqEu+IdC/HpdpZzBEwkZQTmDTTmDq0s5Q3qQVUREQkdDNCIiUUoFLyISpbq14M3sF2a20cxWm9mzZnbMSd7N7DwzKzSzIjO7szsztfP6XzCzdWbWambtnjJlZtvMbI2ZrerqaUuB6EROz7anmWWa2Stmttn/+ZhzT5iZz78dV5nZ8yHMd9xtY2bJZvakf/1SMxseqmxH5DhRzuvNrOKwbTjXg4zzzazczNa2s97M7Lf+f8NqM5sc6oz+HCfKOcvMqg/blv/jQcYhZvaGma33/47fdoz7dH57Oue67QP4NJDgX74XuPcY94kHtgC5QBJQAEzozlzHyDAeGAu8CeQd537bgKxQZutsTq+3J3AfcKd/+c5jfc/962o92H4n3DbALcAf/ctXAk+Gac7rgQdCne2IDGcBk4G17aw/H3gJMGA6sDRMc84CFnm8LQcCk/3LGcCmY3zPO709u3UP3jn3snOuxX/zA+BYE71PBYqcc8XOuSbgCeCS7sx1JOfcBudc2F9h28GcXm/PS4BH/cuPApeG8LVPpCPb5vD8TwOzLfQTJHn9PewQ59zbwL7j3OUS4G+uzQdAbzMbGJp0/6cDOT3nnNvlnPvQv1wDbACOnLul09szlGPwN9D2v8+RBgM7D7tdwtH/sHDhgJfNbIWZzfM6TDu83p79nXO7/Mu7gf7t3C/FzPLN7AMzC9V/Ah3ZNh/dx79zUg30DUm6Y2Twa+97eJn/T/WnzWxIaKJ1itc/i50xw8wKzOwlMzvJyyD+YcHTgKVHrOr09gx4qgIzexUYcIxVdznn/uW/z11AC7Ag0Nfrqo7k7IAznXOlZtYPeGTuXAMAAAJYSURBVMXMNvr3DoImSDm71fEyHn7DOefMrL3zcIf5t2Uu8LqZrXHObQl21ij2ArDQOddoZl+h7a+OczzOFKk+pO3nsdbMzgeeA0Z7EcTM0oFngNudcwcCfb6AC945d+7x1pvZ9cCFwGznH0g6Qilw+N5Hjv9rQXWinB18jlL/53Ize5a2P6WDWvBByNnt2/N4Gc1sj5kNdM7t8v/5WN7OcxzalsVm9iZteyzdXfAd2TaH7lNiZglAL6Cym3Md6YQ5nXOHZ3qYtmMf4SYkv9uBOrxInXMvmtnvzSzLORfSScjMLJG2cl/gnPvnMe7S6e3Z3WfRnAd8B7jYOXewnbstB0ab2QgzS6LtwFbIzqroKDNLM7OMQ8u0HUA+5lF5j3m9PZ8H5viX5wBH/dVhZn3MLNm/nAWcAawPQbaObJvD818OvN7Ojkl3OmHOI8ZeL6ZtzDbcPA9c5z/7YzpQfdjwXdgwswGHjrOY2VTaejGk/6n7X/8RYINz7lft3K3z27ObjwwX0TZmtMr/cejshEHAi0ccHd5E2x7cXd2ZqZ2cn6NtPKsR2AMsPjInbWc0FPg/1oVrTq+3J23j1a8Bm4FXgUz/1/OAh/3LM4E1/m25BrgxhPmO2jbAPbTthACkAE/5f3aXAbmh/j53MOfP/D+HBcAbwDgPMi4EdgHN/p/LG4GbgZv96w140P9vWMNxzlDzOOfXD9uWHwAzPch4Jm3H+FYf1pfnB7o9NVWBiEiU0pWsIiJRSgUvIhKlVPAiIlFKBS8iEqVU8CIiUUoFLyISpVTwIiJR6v8DJyffY5pTpjgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86XDUeDP5b2c"
      },
      "source": [
        "**Return to slides here.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GKUXnREM5b2c"
      },
      "source": [
        "### Stochastic Gradient Descent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7clTIe1z5b2c"
      },
      "source": [
        "Refer to the slides and the [*SGD from Scratch* notebook](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/SGD-from-scratch.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gqdFAfzB5b2d"
      },
      "source": [
        "### Learning Rate Scheduling"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KHoYnNJ55b2d"
      },
      "source": [
        "Refer to the slides and the [*Learning Rate Scheduling* notebook](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/learning-rate-scheduling.ipynb)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ob0c0k3d5b2e"
      },
      "source": [
        "## Segment 3: Fancy Deep Learning Optimizers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LymJg7aQ5b2f"
      },
      "source": [
        "### Jacobian and Hessian Matrices"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ihbu7w5H5b2f"
      },
      "source": [
        "A *gradient* holds the partial derivatives of a function with vector input and scalar output. That is, $\\boldsymbol{f}: \\mathbb{R}^n \\rightarrow \\mathbb{R}$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nvjLc4T_5b2g"
      },
      "source": [
        "A **Jacobian matrix** holds the partial derivatives of a function with vector input *and vector output*. That is, if $\\boldsymbol{f}: \\mathbb{R}^m \\rightarrow \\mathbb{R}^n$. \n",
        "\n",
        "E.g., in the [*Artificial Neurons* notebook](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/artificial-neurons.ipynb), $m = 784$ and $n = 128$. In this case: \n",
        "* The Jacobian matrix $\\boldsymbol{J}$ has 128 rows and 784 columns.\n",
        "* Each element of $\\boldsymbol{J}$ contains the partial derivative of a particular output element with respect to a particular input element: $J_{i,j} = \\frac{\\partial}{\\partial x_j} f(\\boldsymbol{x})_i$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q2GWdtGk5b2g"
      },
      "source": [
        "Note that a *generalized Jacobian* holds the partial derivatives of a function with an input tensor of any dimensionality and an output tensor of any dimensionality."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "02BNYUTF5b2g"
      },
      "source": [
        "**Return to slides here.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4PPuIRHT5b2h"
      },
      "source": [
        "A **Hessian matrix** is the Jacobian matrix of the gradient. Its elements hold all of the possible second derivatives of some function $f(\\boldsymbol{x})$: \n",
        "$$ \\boldsymbol{H}(f)(\\boldsymbol{x})_{i,j} = \\frac{\\partial^2}{\\partial x_i \\partial x_j} f(\\boldsymbol{x}) $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lQM3Ons65b2h"
      },
      "source": [
        "Otherwise, refer to the slides and the following notebooks: \n",
        "\n",
        "* [*Artificial Neurons*](https://github.com/jonkrohn/ML-foundations/blob/master/notebooks/artificial-neurons.ipynb)\n",
        "* [*Deep Net in TensorFlow* (1.x)](https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/deep_net_in_tensorflow.ipynb)\n",
        "* [*Deep Convolutional Net in TensorFlow* (1.x)](https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/lenet_in_tensorflow.ipynb)\n",
        "* [*Deep Convolutional Net in TensorFlow* (2.x, with Keras)](https://github.com/jonkrohn/DLTFpT/blob/master/notebooks/lenet_in_tensorflow.ipynb)\n",
        "* [*Deep Net in PyTorch*](https://github.com/jonkrohn/DLTFpT/blob/master/notebooks/deep_net_in_pytorch.ipynb)"
      ]
    }
  ]
}