DL_keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning \n",
    "\n",
    "NN account for interactions between features \n",
    "\n",
    "- Basics: Forward propagation, back propagation, gradient decent\n",
    "- Optimizing a neural network with backward propagation\n",
    "- Building deep learning models with keras\n",
    "- Fine-tuning keras models \n",
    "- How to choose a NN model ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import feather\n",
    "sys.path.append(r'C:\\Users\\Benny\\fastai\\old')\n",
    "sys.path.append(r'C:\\Users\\Benny\\Anaconda3\\pkgs\\cudatoolkit-9.0-1\\Library\\bin')\n",
    "from fastai.imports import *\n",
    "from fastai.structured import *\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Concept\n",
    "\n",
    "# Forward Propagation\n",
    "\n",
    "Coding the forward propagation algorithm\n",
    "\n",
    "Each data point is a customer. The first input is how many accounts they have, and the second input is how many children they have. The model will predict how many transactions the user makes in the next year. You will use this data throughout the first 2 chapters of this course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-39\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "input_data = [3,5]\n",
    "\n",
    "weights = {'node_0': np.array([2, 4]), 'node_1': np.array([ 4, -5]), \n",
    "           'output': np.array([2, 7])}\n",
    "\n",
    "# Calculate node 0 value: node_0_value\n",
    "node_0_value = (input_data * weights['node_0']).sum()\n",
    "\n",
    "# Calculate node 1 value: node_1_value\n",
    "node_1_value = (input_data* weights['node_1']).sum()\n",
    "\n",
    "# Put node values into array: hidden_layer_outputs\n",
    "hidden_layer_outputs = np.array([node_0_value, node_1_value])\n",
    "\n",
    "# Calculate output: output\n",
    "output = (hidden_layer_outputs*weights['output']).sum()\n",
    "\n",
    "# Print output\n",
    "print(output)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Rectified Linear Activation Function \n",
    "\n",
    "The rectified linear activation function (called ReLU) has been shown to lead to very high-performance networks. This function takes a single number as an input, returning 0 if the input is negative, and the input if the input is positive.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "52\n"
     ]
    }
   ],
   "source": [
    "def relu(input):\n",
    "    '''Define your relu activation function here'''\n",
    "    # Calculate the value for the output of the relu function: output\n",
    "    output = max(input, 0)\n",
    "    \n",
    "    # Return the value just calculated\n",
    "    return(output)\n",
    "\n",
    "# Calculate node 0 value: node_0_output\n",
    "node_0_input = (input_data * weights['node_0']).sum()\n",
    "node_0_output = relu(node_0_input)\n",
    "\n",
    "# Calculate node 1 value: node_1_output\n",
    "node_1_input = (input_data * weights['node_1']).sum()\n",
    "node_1_output = relu(node_1_input)\n",
    "\n",
    "# Put node values into array: hidden_layer_outputs\n",
    "hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "\n",
    "# Calculate model output (do not apply relu)\n",
    "model_output = (hidden_layer_outputs * weights['output']).sum()\n",
    "\n",
    "# Print model output\n",
    "print(model_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying the network to many observations/rows of data\n",
    "You'll now define a function called predict_with_network() which will generate predictions for multiple data observations, which are pre-loaded as input_data. As before, weights are also pre-loaded. In addition, the relu() function you defined in the previous exercise has been pre-loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define predict_with_network()\n",
    "def predict_with_network(input_data_row, weights):\n",
    "\n",
    "    # Calculate node 0 value\n",
    "    node_0_input = (input_data_row*weights['node_0']).sum()\n",
    "    node_0_output = relu(node_0_input)\n",
    "\n",
    "    # Calculate node 1 value\n",
    "    node_1_input = (input_data_row*weights['node_1']).sum()\n",
    "    node_1_output = relu(node_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_layer_outputs\n",
    "    hidden_layer_outputs = np.array([node_0_output, node_1_output])\n",
    "    \n",
    "    # Calculate model output\n",
    "    input_to_final_layer = (hidden_layer_outputs*weights['output']).sum()\n",
    "    model_output = relu(input_to_final_layer)\n",
    "    \n",
    "    # Return model output\n",
    "    return(model_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = {'node_0': np.array([2, 4]), 'node_1': np.array([ 4, -5]), 'output': np.array([2, 7])}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[52, 63, 0, 148]\n"
     ]
    }
   ],
   "source": [
    "input_data = [np.array([3, 5]), np.array([ 1, -1]), np.array([0, 0]), np.array([8, 4])]\n",
    "# Create empty list to store prediction results\n",
    "results = []\n",
    "for input_data_row in input_data:\n",
    "    # Append prediction to results\n",
    "    results.append(predict_with_network(input_data_row,weights))\n",
    "\n",
    "# Print results\n",
    "print(results)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multi-layer neural networks\n",
    "In this exercise, you'll write code to do forward propagation for a neural network with 2 hidden layers. Each hidden layer has two nodes. The input data has been preloaded as input_data. The nodes in the first hidden layer are called node_0_0 and node_0_1. Their weights are pre-loaded as weights['node_0_0'] and weights['node_0_1'] respectively.\n",
    "\n",
    "The nodes in the second hidden layer are called node_1_0 and node_1_1. Their weights are pre-loaded as weights['node_1_0'] and weights['node_1_1'] respectively.\n",
    "\n",
    "We then create a model output from the hidden nodes using weights pre-loaded as weights['output']. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_with_network(input_data):\n",
    "    # Calculate node 0 in the first hidden layer\n",
    "    node_0_0_input = (input_data * weights['node_0_0']).sum()\n",
    "    node_0_0_output = relu(node_0_0_input)\n",
    "\n",
    "    # Calculate node 1 in the first hidden layer\n",
    "    node_0_1_input = (input_data * weights['node_0_1']).sum()\n",
    "    node_0_1_output = relu(node_0_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_0_outputs\n",
    "    hidden_0_outputs = np.array([node_0_0_output, node_0_1_output])\n",
    "    \n",
    "    # Calculate node 0 in the second hidden layer\n",
    "    node_1_0_input = (hidden_0_outputs * weights['node_1_0']).sum()\n",
    "    node_1_0_output = relu(node_1_0_input)\n",
    "\n",
    "    # Calculate node 1 in the second hidden layer\n",
    "    node_1_1_input = (hidden_0_outputs * weights['node_1_1']).sum()\n",
    "    node_1_1_output = relu(node_1_1_input)\n",
    "\n",
    "    # Put node values into array: hidden_1_outputs\n",
    "    hidden_1_outputs = np.array([node_1_0_output, node_1_1_output])\n",
    "\n",
    "    # Calculate model output: model_output\n",
    "    model_output = (hidden_1_outputs*weights['output']).sum()\n",
    "    \n",
    "    # Return model_output\n",
    "    return(model_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "182\n"
     ]
    }
   ],
   "source": [
    "input_data = np.array([3,5])\n",
    "weights = {'node_0_0': np.array([2, 4]),\n",
    " 'node_0_1': np.array([ 4, -5]),\n",
    " 'node_1_0': np.array([-1,  2]),\n",
    " 'node_1_1': np.array([1, 2]),\n",
    " 'output': np.array([2, 7])}\n",
    "output = predict_with_network(input_data)\n",
    "print(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = {'node_0_0': np.array([2, 4]),\n",
    " 'node_0_1': np.array([ 4, -5]),\n",
    " 'node_1_0': np.array([-1,  2]),\n",
    " 'node_1_1': np.array([1, 2]),\n",
    " 'output': np.array([2, 7])}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How weight changes affect accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "# The data point you will make a prediction for\n",
    "input_data = np.array([0, 3])\n",
    "\n",
    "# Sample weights\n",
    "weights_0 = {'node_0': [2, 1],\n",
    "             'node_1': [1, 2],\n",
    "             'output': [1, 1]\n",
    "            }\n",
    "\n",
    "# The actual target value, used to calculate the error\n",
    "target_actual = 3\n",
    "\n",
    "# Make prediction using original weights\n",
    "model_output_0 = predict_with_network(input_data, weights_0)\n",
    "\n",
    "# Calculate error: error_0\n",
    "error_0 = model_output_0 - target_actual\n",
    "\n",
    "# Create weights that cause the network to make perfect prediction (3): weights_1\n",
    "weights_1 = {'node_0': [2, 1],\n",
    "             'node_1': [1, 0],\n",
    "             'output': [1, 1]\n",
    "            }\n",
    "\n",
    "# Make prediction using new weights: model_output_1\n",
    "model_output_1 = predict_with_network(input_data, weights_1)\n",
    "\n",
    "# Calculate error: error_1\n",
    "error_1 = model_output_1 - target_actual\n",
    "\n",
    "# Print error_0 and error_1\n",
    "print(error_0)\n",
    "print(error_1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scaling up to multiple data points\n",
    "\n",
    "you've seen how different weights will have different accuracies on a single prediction. But usually, you'll want to measure model accuracy on many points. You'll now write code to compare model accuracies for two different sets of weights, which have been stored as weights_0 and weights_1.\n",
    "\n",
    "input_data is a list of arrays. Each item in that list contains the data to make a single prediction. target_actuals is a list of numbers. Each item in that list is the actual value we are trying to predict.\n",
    "\n",
    "In this exercise, you'll use the mean_squared_error() function from sklearn.metrics. It takes the true values and the predicted values as arguments.\n",
    "\n",
    "You'll also use the preloaded predict_with_network() function, which takes an array of data as the first argument, and weights as the second argument.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "weight_0 = {'node_0': np.array([2, 1]), 'node_1': np.array([1, 2]), 'output': np.array([1, 1])}\n",
    "weight_1 = {'node_0': np.array([2, 1]), 'node_1': np.array([1, 2]), 'output': np.array([1, 1])}\n",
    "input_data = {'node_0': np.array([2, 1]), 'node_1': np.array([1, 2]), 'output': np.array([1, 1])}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "can't multiply sequence by non-int of type 'list'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-32-efcbff946327>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mrow\u001b[0m \u001b[1;32min\u001b[0m \u001b[0minput_data\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     10\u001b[0m     \u001b[1;31m# Append prediction to model_output_0\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 11\u001b[1;33m     \u001b[0mmodel_output_0\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpredict_with_network\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrow\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0mweights_0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     12\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     13\u001b[0m     \u001b[1;31m# Append prediction to model_output_1\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-29-6df6952abd6b>\u001b[0m in \u001b[0;36mpredict_with_network\u001b[1;34m(input_data_row, weights)\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m     \u001b[1;31m# Calculate node 0 value\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m     \u001b[0mnode_0_input\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0minput_data_row\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mweights\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'node_0'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msum\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m     \u001b[0mnode_0_output\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrelu\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnode_0_input\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: can't multiply sequence by non-int of type 'list'"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "# Create model_output_0 \n",
    "model_output_0 = []\n",
    "# Create model_output_1\n",
    "model_output_1 = []\n",
    "\n",
    "# Loop over input_data\n",
    "for row in input_data:\n",
    "    # Append prediction to model_output_0\n",
    "    model_output_0.append(predict_with_network(row,weights_0))\n",
    "    \n",
    "    # Append prediction to model_output_1\n",
    "    model_output_1.append(predict_with_network(row,weights_1))\n",
    "\n",
    "# Calculate the mean squared error for model_output_0: mse_0\n",
    "mse_0 = mean_squared_error(target_actuals,model_output_0)\n",
    "\n",
    "# Calculate the mean squared error for model_output_1: mse_1\n",
    "mse_1 = mean_squared_error(target_actuals,model_output_1)\n",
    "\n",
    "# Print mse_0 and mse_1\n",
    "print(\"Mean squared error with weights_0: %f\" %mse_0)\n",
    "print(\"Mean squared error with weights_1: %f\" %mse_1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Gradient descent\n",
    "\n",
    "Calculating slopes\n",
    "\n",
    "You're now going to practice calculating slopes. When plotting the mean-squared error loss function against predictions, the slope is 2 * x * (y-xb), or 2 * input_data * error. Note that x and b may have multiple numbers (x is a vector for each data point, and b is a vector). In this case, the output will also be a vector, which is exactly what you want.\n",
    "\n",
    "You're ready to write the code to calculate this slope while using a single data point. You'll use pre-defined weights called weights as well as data for a single point called input_data. The actual value of the target you want to predict is stored in target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "input_data = np.array([1, 2, 3])\n",
    "weights = np.array([0,2,1])\n",
    "target = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-14 -28 -42]\n"
     ]
    }
   ],
   "source": [
    "# Calculate the predictions: preds\n",
    "preds = (input_data*weights).sum()\n",
    "\n",
    "# Calculate the error: error\n",
    "error = target - preds\n",
    "\n",
    "# Calculate the slope: slope\n",
    "slope = 2 * input_data * error\n",
    "\n",
    "# Print the slope\n",
    "print(slope)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Improving model weights\n",
    "\n",
    "Hurray! You've just calculated the slopes you need. Now it's time to use those slopes to improve your model. If you add the slopes to your weights, you will move in the right direction. However, it's possible to move too far in that direction. So you will want to take a small step in that direction first, using a lower learning rate, and verify that the model is improving.\n",
    "\n",
    "The weights have been pre-loaded as weights, the actual value of the target as target, and the input data as input_data. The predictions from the initial weights are stored as preds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "weights = np.array([0, 2, 1])\n",
    "input_data = np.array([0, 2, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "4.5\n"
     ]
    }
   ],
   "source": [
    "# Set the learning rate: learning_rate\n",
    "learning_rate = 0.01\n",
    "\n",
    "# Calculate the predictions: preds\n",
    "preds = (weights * input_data).sum()\n",
    "\n",
    "# Calculate the error: error\n",
    "error = preds - target\n",
    "\n",
    "# Calculate the slope: slope\n",
    "slope = 2 * input_data * error\n",
    "\n",
    "# Update the weights: weights_updated\n",
    "weights_updated = weights - learning_rate*slope\n",
    "\n",
    "# Get updated predictions: preds_updated\n",
    "preds_updated = (weights_updated * input_data).sum()\n",
    "\n",
    "# Calculate updated error: error_updated\n",
    "error_updated = preds_updated - target\n",
    "\n",
    "# Print the original error\n",
    "print(error)\n",
    "\n",
    "# Print the updated error\n",
    "print(error_updated)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Making multiple updates to weights\n",
    "You're now going to make multiple updates so you can dramatically improve your model weights, and see how the predictions improve with each update.\n",
    "\n",
    "To keep your code clean, there is a pre-loaded get_slope() function that takes input_data, target, and weights as arguments. There is also a get_mse() function that takes the same arguments. The input_data, target, and weights have been pre-loaded.\n",
    "\n",
    "This network does not have any hidden layers, and it goes directly from the input (with 3 nodes) to an output node. Note that weights is a single array.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "weights = np.array([0, 2, 1])\n",
    "input_data = np.array([0, 2, 1])\n",
    "target=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'get_slope' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-38-e099feff2cab>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mn_updates\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m     \u001b[1;31m# Calculate the slope: slope\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 7\u001b[1;33m     \u001b[0mslope\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_slope\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0minput_data\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtarget\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mweights\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m     \u001b[1;31m# Update the weights: weights\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'get_slope' is not defined"
     ]
    }
   ],
   "source": [
    "n_updates = 20\n",
    "mse_hist = []\n",
    "\n",
    "# Iterate over the number of updates\n",
    "for i in range(n_updates):\n",
    "    # Calculate the slope: slope\n",
    "    slope = get_slope(input_data, target, weights)\n",
    "    \n",
    "    # Update the weights: weights\n",
    "    weights = weights - 0.01 * slope\n",
    "    \n",
    "    # Calculate mse with new weights: mse\n",
    "    mse = get_mse(input_data, target, weights)\n",
    "    \n",
    "    # Append the mse to mse_hist\n",
    "    mse_hist.append(mse)\n",
    "\n",
    "# Plot the mse history\n",
    "plt.plot(mse_hist)\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Mean Squared Error')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building deep learning models with keras\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>wage_per_hour</th>\n",
       "      <th>union</th>\n",
       "      <th>education_yrs</th>\n",
       "      <th>experience_yrs</th>\n",
       "      <th>age</th>\n",
       "      <th>female</th>\n",
       "      <th>marr</th>\n",
       "      <th>south</th>\n",
       "      <th>manufacturing</th>\n",
       "      <th>construction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.10</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>21</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.95</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>42</td>\n",
       "      <td>57</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6.67</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.00</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.50</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>17</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   wage_per_hour  union  education_yrs  experience_yrs  age  female  marr  \\\n",
       "0           5.10      0              8              21   35       1     1   \n",
       "1           4.95      0              9              42   57       1     1   \n",
       "2           6.67      0             12               1   19       0     0   \n",
       "3           4.00      0             12               4   22       0     0   \n",
       "4           7.50      0             12              17   35       0     1   \n",
       "\n",
       "   south  manufacturing  construction  \n",
       "0      0              1             0  \n",
       "1      0              1             0  \n",
       "2      0              1             0  \n",
       "3      0              0             0  \n",
       "4      0              0             0  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/hourly_wages.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(534, 10)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictors,wage_per_hour,nas, mappers = proc_df(df, y_fld='wage_per_hour',skip_flds=None,ignore_flds=None,do_scale=True)\n",
    "predictors = predictors.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Specifying a model**\n",
    "\n",
    "As a start, you'll predict workers wages based on characteristics like their industry, education and level of experience. You can find the dataset in a pandas dataframe called df. For convenience, everything in df except for the target has been converted to a NumPy matrix called predictors. The target, wage_per_hour, is available as a NumPy matrix called target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "ename": "ImportError",
     "evalue": "Could not find 'cudart64_90.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. Download and install CUDA 9.0 from this URL: https://developer.nvidia.com/cuda-toolkit",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\tensorflow\\python\\platform\\self_check.py\u001b[0m in \u001b[0;36mpreload_check\u001b[1;34m()\u001b[0m\n\u001b[0;32m     74\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 75\u001b[1;33m           \u001b[0mctypes\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mWinDLL\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbuild_info\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcudart_dll_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     76\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mOSError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\ctypes\\__init__.py\u001b[0m in \u001b[0;36m__init__\u001b[1;34m(self, name, mode, handle, use_errno, use_last_error)\u001b[0m\n\u001b[0;32m    347\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 348\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_handle\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_dlopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_name\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    349\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mOSError\u001b[0m: [WinError 126] The specified module could not be found",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-53-95f9c8973829>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# Import necessary modules\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mkeras\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      3\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlayers\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mDense\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mkeras\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmodels\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mSequential\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\keras\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0m__future__\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mabsolute_import\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 3\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mutils\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mactivations\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mapplications\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\keras\\utils\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mdata_utils\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mio_utils\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mconv_utils\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;31m# Globally-importable utils.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\keras\\utils\\conv_utils.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0msix\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmoves\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m\u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mbackend\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mK\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     10\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     11\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\keras\\backend\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     87\u001b[0m \u001b[1;32melif\u001b[0m \u001b[0m_BACKEND\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'tensorflow'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     88\u001b[0m     \u001b[0msys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mwrite\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'Using TensorFlow backend.\\n'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 89\u001b[1;33m     \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m\u001b[0mtensorflow_backend\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[1;33m*\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     90\u001b[0m \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     91\u001b[0m     \u001b[1;31m# Try and load external backend.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0m__future__\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mprint_function\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 5\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      6\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mframework\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mops\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mtf_ops\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      7\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtraining\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mmoving_averages\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\tensorflow\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     20\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     21\u001b[0m \u001b[1;31m# pylint: disable=g-bad-import-order\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 22\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpython\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mpywrap_tensorflow\u001b[0m  \u001b[1;31m# pylint: disable=unused-import\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     23\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mapp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     24\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;33m.\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mbitwise\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\tensorflow\\python\\__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     47\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     48\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 49\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpython\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mpywrap_tensorflow\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     50\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     51\u001b[0m \u001b[1;31m# Protocol buffers\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\tensorflow\\python\\pywrap_tensorflow.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     28\u001b[0m \u001b[1;31m# Perform pre-load sanity checks in order to produce a more actionable error\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     29\u001b[0m \u001b[1;31m# than we get from an error during SWIG import.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 30\u001b[1;33m \u001b[0mself_check\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpreload_check\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     31\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     32\u001b[0m \u001b[1;31m# pylint: disable=wildcard-import,g-import-not-at-top,unused-import,line-too-long\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\AppData\\Roaming\\Python\\Python36\\site-packages\\tensorflow\\python\\platform\\self_check.py\u001b[0m in \u001b[0;36mpreload_check\u001b[1;34m()\u001b[0m\n\u001b[0;32m     80\u001b[0m               \u001b[1;34m\"environment variable. Download and install CUDA %s from \"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     81\u001b[0m               \u001b[1;34m\"this URL: https://developer.nvidia.com/cuda-toolkit\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 82\u001b[1;33m               % (build_info.cudart_dll_name, build_info.cuda_version_number))\n\u001b[0m\u001b[0;32m     83\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     84\u001b[0m       if hasattr(build_info, \"cudnn_dll_name\") and hasattr(\n",
      "\u001b[1;31mImportError\u001b[0m: Could not find 'cudart64_90.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. Download and install CUDA 9.0 from this URL: https://developer.nvidia.com/cuda-toolkit"
     ]
    }
   ],
   "source": [
    "# Import necessary modules\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "\n",
    "# Set up the model: model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first layer\n",
    "model.add(Dense(50, activation='relu', input_shape=(n_cols,)))\n",
    "\n",
    "# Add the second layer\n",
    "model.add(Dense(32,activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "\n",
    "# Specify the model\n",
    "n_cols = predictors.shape[1]\n",
    "model = Sequential()\n",
    "model.add(Dense(50, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',loss='mean_squared_error')\n",
    "\n",
    "# Verify that model contains information from compiling\n",
    "print(\"Loss function: \" + model.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>male</th>\n",
       "      <th>age_was_missing</th>\n",
       "      <th>embarked_from_cherbourg</th>\n",
       "      <th>embarked_from_queenstown</th>\n",
       "      <th>embarked_from_southampton</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived  pclass   age  sibsp  parch     fare  male  age_was_missing  \\\n",
       "0         0       3  22.0      1      0   7.2500     1            False   \n",
       "1         1       1  38.0      1      0  71.2833     0            False   \n",
       "2         1       3  26.0      0      0   7.9250     0            False   \n",
       "3         1       1  35.0      1      0  53.1000     0            False   \n",
       "4         0       3  35.0      0      0   8.0500     1            False   \n",
       "\n",
       "   embarked_from_cherbourg  embarked_from_queenstown  \\\n",
       "0                        0                         0   \n",
       "1                        1                         0   \n",
       "2                        0                         0   \n",
       "3                        0                         0   \n",
       "4                        0                         0   \n",
       "\n",
       "   embarked_from_southampton  \n",
       "0                          1  \n",
       "1                          0  \n",
       "2                          1  \n",
       "3                          1  \n",
       "4                          1  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic = pd.read_csv('data/titanic_all_numeric.csv')\n",
    "titanic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(891, 11)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last steps in classification models\n",
    "\n",
    "You'll now create a classification model using the titanic dataset, which has been pre-loaded into a DataFrame called df. You'll take information about the passengers and predict which ones survived.\n",
    "\n",
    "The predictive variables are stored in a NumPy array predictors. The target to predict is in df.survived, though you'll have to manipulate it for keras. The number of predictive features is stored in n_cols."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import keras\n",
    "from keras.layers import Dense\n",
    "from keras.models import Sequential\n",
    "from keras.utils import to_categorical\n",
    "\n",
    "# Convert the target to categorical: target\n",
    "target = to_categorical(df.survived)\n",
    "\n",
    "# Set up the model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first layer\n",
    "model.add(Dense(32,activation='relu',input_shape=(n_cols,)))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(2,activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='sgd',loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(predictors,target)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making predictions\n",
    "The trained network from your previous coding exercise is now stored as model. New data to make predictions is stored in a NumPy array as pred_data. Use model to make predictions on your new data.\n",
    "\n",
    "In this exercise, your predictions will be probabilities, which is the most common way for data scientists to communicate their predictions to colleagues.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_data= np.array([[2, 34.0, 0, 0, 13.0, 1, False, 0, 0, 1],\n",
    "       [2, 31.0, 1, 1, 26.25, 0, False, 0, 0, 1],\n",
    "       [1, 11.0, 1, 2, 120.0, 1, False, 0, 0, 1],\n",
    "       [3, 0.42, 0, 1, 8.5167, 1, False, 1, 0, 0],\n",
    "       [3, 27.0, 0, 0, 6.975, 1, False, 0, 0, 1],\n",
    "       [3, 31.0, 0, 0, 7.775, 1, False, 0, 0, 1],\n",
    "       [1, 39.0, 0, 0, 0.0, 1, False, 0, 0, 1],\n",
    "       [3, 18.0, 0, 0, 7.775, 0, False, 0, 0, 1],\n",
    "       [2, 39.0, 0, 0, 13.0, 1, False, 0, 0, 1],\n",
    "       [1, 33.0, 1, 0, 53.1, 0, False, 0, 0, 1],\n",
    "       [3, 26.0, 0, 0, 7.8875, 1, False, 0, 0, 1],\n",
    "       [3, 39.0, 0, 0, 24.15, 1, False, 0, 0, 1],\n",
    "       [2, 35.0, 0, 0, 10.5, 1, False, 0, 0, 1],\n",
    "       [3, 6.0, 4, 2, 31.275, 0, False, 0, 0, 1],\n",
    "       [3, 30.5, 0, 0, 8.05, 1, False, 0, 0, 1],\n",
    "       [1, 29.69911764705882, 0, 0, 0.0, 1, True, 0, 0, 1],\n",
    "       [3, 23.0, 0, 0, 7.925, 0, False, 0, 0, 1],\n",
    "       [2, 31.0, 1, 1, 37.0042, 1, False, 1, 0, 0],\n",
    "       [3, 43.0, 0, 0, 6.45, 1, False, 0, 0, 1],\n",
    "       [3, 10.0, 3, 2, 27.9, 1, False, 0, 0, 1],\n",
    "       [1, 52.0, 1, 1, 93.5, 0, False, 0, 0, 1],\n",
    "       [3, 27.0, 0, 0, 8.6625, 1, False, 0, 0, 1],\n",
    "       [1, 38.0, 0, 0, 0.0, 1, False, 0, 0, 1],\n",
    "       [3, 27.0, 0, 1, 12.475, 0, False, 0, 0, 1],\n",
    "       [3, 2.0, 4, 1, 39.6875, 1, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 0, 0, 6.95, 1, True, 0, 1, 0],\n",
    "       [3, 29.69911764705882, 0, 0, 56.4958, 1, True, 0, 0, 1],\n",
    "       [2, 1.0, 0, 2, 37.0042, 1, False, 1, 0, 0],\n",
    "       [3, 29.69911764705882, 0, 0, 7.75, 1, True, 0, 1, 0],\n",
    "       [1, 62.0, 0, 0, 80.0, 0, False, 0, 0, 0],\n",
    "       [3, 15.0, 1, 0, 14.4542, 0, False, 1, 0, 0],\n",
    "       [2, 0.83, 1, 1, 18.75, 1, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 0, 0, 7.2292, 1, True, 1, 0, 0],\n",
    "       [3, 23.0, 0, 0, 7.8542, 1, False, 0, 0, 1],\n",
    "       [3, 18.0, 0, 0, 8.3, 1, False, 0, 0, 1],\n",
    "       [1, 39.0, 1, 1, 83.1583, 0, False, 1, 0, 0],\n",
    "       [3, 21.0, 0, 0, 8.6625, 1, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 0, 0, 8.05, 1, True, 0, 0, 1],\n",
    "       [3, 32.0, 0, 0, 56.4958, 1, False, 0, 0, 1],\n",
    "       [1, 29.69911764705882, 0, 0, 29.7, 1, True, 1, 0, 0],\n",
    "       [3, 20.0, 0, 0, 7.925, 1, False, 0, 0, 1],\n",
    "       [2, 16.0, 0, 0, 10.5, 1, False, 0, 0, 1],\n",
    "       [1, 30.0, 0, 0, 31.0, 0, False, 1, 0, 0],\n",
    "       [3, 34.5, 0, 0, 6.4375, 1, False, 1, 0, 0],\n",
    "       [3, 17.0, 0, 0, 8.6625, 1, False, 0, 0, 1],\n",
    "       [3, 42.0, 0, 0, 7.55, 1, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 8, 2, 69.55, 1, True, 0, 0, 1],\n",
    "       [3, 35.0, 0, 0, 7.8958, 1, False, 1, 0, 0],\n",
    "       [2, 28.0, 0, 1, 33.0, 1, False, 0, 0, 1],\n",
    "       [1, 29.69911764705882, 1, 0, 89.1042, 0, True, 1, 0, 0],\n",
    "       [3, 4.0, 4, 2, 31.275, 1, False, 0, 0, 1],\n",
    "       [3, 74.0, 0, 0, 7.775, 1, False, 0, 0, 1],\n",
    "       [3, 9.0, 1, 1, 15.2458, 0, False, 1, 0, 0],\n",
    "       [1, 16.0, 0, 1, 39.4, 0, False, 0, 0, 1],\n",
    "       [2, 44.0, 1, 0, 26.0, 0, False, 0, 0, 1],\n",
    "       [3, 18.0, 0, 1, 9.35, 0, False, 0, 0, 1],\n",
    "       [1, 45.0, 1, 1, 164.8667, 0, False, 0, 0, 1],\n",
    "       [1, 51.0, 0, 0, 26.55, 1, False, 0, 0, 1],\n",
    "       [3, 24.0, 0, 3, 19.2583, 0, False, 1, 0, 0],\n",
    "       [3, 29.69911764705882, 0, 0, 7.2292, 1, True, 1, 0, 0],\n",
    "       [3, 41.0, 2, 0, 14.1083, 1, False, 0, 0, 1],\n",
    "       [2, 21.0, 1, 0, 11.5, 1, False, 0, 0, 1],\n",
    "       [1, 48.0, 0, 0, 25.9292, 0, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 8, 2, 69.55, 0, True, 0, 0, 1],\n",
    "       [2, 24.0, 0, 0, 13.0, 1, False, 0, 0, 1],\n",
    "       [2, 42.0, 0, 0, 13.0, 0, False, 0, 0, 1],\n",
    "       [2, 27.0, 1, 0, 13.8583, 0, False, 1, 0, 0],\n",
    "       [1, 31.0, 0, 0, 50.4958, 1, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 0, 0, 9.5, 1, True, 0, 0, 1],\n",
    "       [3, 4.0, 1, 1, 11.1333, 1, False, 0, 0, 1],\n",
    "       [3, 26.0, 0, 0, 7.8958, 1, False, 0, 0, 1],\n",
    "       [1, 47.0, 1, 1, 52.5542, 0, False, 0, 0, 1],\n",
    "       [1, 33.0, 0, 0, 5.0, 1, False, 0, 0, 1],\n",
    "       [3, 47.0, 0, 0, 9.0, 1, False, 0, 0, 1],\n",
    "       [2, 28.0, 1, 0, 24.0, 0, False, 1, 0, 0],\n",
    "       [3, 15.0, 0, 0, 7.225, 0, False, 1, 0, 0],\n",
    "       [3, 20.0, 0, 0, 9.8458, 1, False, 0, 0, 1],\n",
    "       [3, 19.0, 0, 0, 7.8958, 1, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 0, 0, 7.8958, 1, True, 0, 0, 1],\n",
    "       [1, 56.0, 0, 1, 83.1583, 0, False, 1, 0, 0],\n",
    "       [2, 25.0, 0, 1, 26.0, 0, False, 0, 0, 1],\n",
    "       [3, 33.0, 0, 0, 7.8958, 1, False, 0, 0, 1],\n",
    "       [3, 22.0, 0, 0, 10.5167, 0, False, 0, 0, 1],\n",
    "       [2, 28.0, 0, 0, 10.5, 1, False, 0, 0, 1],\n",
    "       [3, 25.0, 0, 0, 7.05, 1, False, 0, 0, 1],\n",
    "       [3, 39.0, 0, 5, 29.125, 0, False, 0, 1, 0],\n",
    "       [2, 27.0, 0, 0, 13.0, 1, False, 0, 0, 1],\n",
    "       [1, 19.0, 0, 0, 30.0, 0, False, 0, 0, 1],\n",
    "       [3, 29.69911764705882, 1, 2, 23.45, 0, True, 0, 0, 1],\n",
    "       [1, 26.0, 0, 0, 30.0, 1, False, 1, 0, 0],\n",
    "       [3, 32.0, 0, 0, 7.75, 1, False, 0, 1, 0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify, compile, and fit the model\n",
    "model = Sequential()\n",
    "model.add(Dense(32, activation='relu', input_shape = (n_cols,)))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "model.compile(optimizer='sgd', \n",
    "              loss='categorical_crossentropy', \n",
    "              metrics=['accuracy'])\n",
    "model.fit(predictors, target)\n",
    "\n",
    "# Calculate predictions: predictions\n",
    "predictions = model.predict(pred_data)\n",
    "\n",
    "# Calculate predicted probability of survival: predicted_prob_true\n",
    "predicted_prob_true = predictions[:,1]\n",
    "\n",
    "# print predicted_prob_true\n",
    "print(predicted_prob_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing optimization parameters\n",
    "\n",
    "It's time to get your hands dirty with optimization. You'll now try optimizing a model at a very low learning rate, a very high learning rate, and a \"just right\" learning rate. You'll want to look at the results after running this exercise, remembering that a low value for the loss function is good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the SGD optimizer\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "# Create list of learning rates: lr_to_test\n",
    "lr_to_test = [0.000001, 0.01, 1]\n",
    "\n",
    "# Loop over learning rates\n",
    "for lr in lr_to_test:\n",
    "    print('\\n\\nTesting model with learning rate: %f\\n'%lr )\n",
    "    \n",
    "    # Build new model to test, unaffected by previous models\n",
    "    model = get_new_model()\n",
    "    \n",
    "    # Create SGD optimizer with specified learning rate: my_optimizer\n",
    "    my_optimizer = SGD(lr)\n",
    "    \n",
    "    # Compile the model\n",
    "    model.compile(optimizer=my_optimizer,loss='categorical_crossentropy')\n",
    "    \n",
    "    # Fit the model\n",
    "    model.fit(predictors,target)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluating model accuracy on validation dataset\n",
    "Now it's your turn to monitor model accuracy with a validation data set. A model definition has been provided as model. Your job is to add the code to compile it and then fit it. You'll check the validation score in each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Specify the model\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape = input_shape))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "hist = model.fit(predictors,target,validation_split=0.3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Early stopping: Optimizing the optimization\n",
    "\n",
    "Now that you know how to monitor your model performance throughout optimization, you can use early stopping to stop optimization when it isn't helping any more. Since the optimization stops automatically when it isn't helping, you can also set a high value for epochs in your call to .fit(), as Dan showed in the video.\n",
    "\n",
    "The model you'll optimize has been specified as model. As before, the data is pre-loaded as predictors and target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import EarlyStopping\n",
    "from keras.callbacks import EarlyStopping\n",
    "\n",
    "# Save the number of columns in predictors: n_cols\n",
    "n_cols = predictors.shape[1]\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Specify the model\n",
    "model = Sequential()\n",
    "model.add(Dense(100, activation='relu', input_shape = input_shape))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(2, activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "\n",
    "# Define early_stopping_monitor\n",
    "early_stopping_monitor = EarlyStopping(patience=2)\n",
    "\n",
    "# Fit the model\n",
    "model.fit(predictors,target,validation_split=0.3,epochs=30,callbacks=[early_stopping_monitor])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Experimenting with wider \n",
    "\n",
    "Now you know everything you need to begin experimenting with different models!\n",
    "\n",
    "A model called model_1 has been pre-loaded. You can see a summary of this model printed in the IPython Shell. This is a relatively small network, with only 10 units in each hidden layer.\n",
    "\n",
    "In this exercise you'll create a new model called model_2 which is similar to model_1, except it has 100 units in each hidden layer.\n",
    "\n",
    "After you create model_2, both models will be fitted, and a graph showing both models loss score at each epoch will be shown. We added the argument verbose=False in the fitting commands to print out fewer updates, since you will look at these graphically instead of as text.\n",
    "\n",
    "Because you are fitting two models, it will take a moment to see the outputs after you hit run, so be patien"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define early_stopping_monitor\n",
    "early_stopping_monitor = EarlyStopping(patience=2)\n",
    "\n",
    "# Create the new model: model_2\n",
    "model_2 = Sequential()\n",
    "\n",
    "# Add the first and second layers\n",
    "model_2.add(Dense(100, activation='relu', input_shape=input_shape))\n",
    "model_2.add(Dense(100, activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model_2.add(Dense(2,activation='softmax'))\n",
    "\n",
    "# Compile model_2\n",
    "model_2.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "\n",
    "# Fit model_1\n",
    "model_1_training = model_1.fit(predictors, target, epochs=15, validation_split=0.2, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Fit model_2\n",
    "model_2_training = model_2.fit(predictors, target, epochs=15, validation_split=0.2, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Create the plot\n",
    "plt.plot(model_1_training.history['val_loss'], 'r', model_2_training.history['val_loss'], 'b')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation score')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding layers to a network\n",
    "\n",
    "You've seen how to experiment with wider networks. In this exercise, you'll try a deeper network (more hidden layers).\n",
    "\n",
    "Once again, you have a baseline model called model_1 as a starting point. It has 1 hidden layer, with 50 units. You can see a summary of that model's structure printed out. You will create a similar network with 3 hidden layers (still keeping 50 units in each layer).\n",
    "\n",
    "This will again take a moment to fit both models, so you'll need to wait a few seconds to see the results after you run your code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The input shape to use in the first hidden layer\n",
    "input_shape = (n_cols,)\n",
    "\n",
    "# Create the new model: model_2\n",
    "model_2 = Sequential()\n",
    "\n",
    "# Add the first, second, and third hidden layers\n",
    "model_2.add(Dense(50,activation='relu',input_shape=input_shape))\n",
    "model_2.add(Dense(50,activation='relu'))\n",
    "model_2.add(Dense(50,activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model_2.add(Dense(2,activation='softmax'))\n",
    "\n",
    "# Compile model_2\n",
    "model_2.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "\n",
    "# Fit model 1\n",
    "model_1_training = model_1.fit(predictors, target, epochs=20, validation_split=0.4, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Fit model 2\n",
    "model_2_training = model_2.fit(predictors, target, epochs=20, validation_split=0.4, callbacks=[early_stopping_monitor], verbose=False)\n",
    "\n",
    "# Create the plot\n",
    "plt.plot(model_1_training.history['val_loss'], 'r', model_2_training.history['val_loss'], 'b')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation score')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to choose a NN model ?\n",
    "\n",
    "Model capacity is the ability to capture predictive patterns in the data. Choosing model capacity concerns underfit and overfit issue. Increasing nodes and layer increase the capacity, but could overfit. \n",
    "\n",
    "Workflow for optimizing model capacity\n",
    "- Start with a small network \n",
    "- Get the validation score\n",
    "- Keep increasing capacity until validation score is no longer improve\n",
    "- Once it stop improving, you can decrease the capacity slightly, but you are probably near the ideal \n",
    "- No general consent if increase number of nodes/layer or number of layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building your own digit recognition model\n",
    "You've reached the final exercise of the course - you now know everything you need to build an accurate model to recognize handwritten digits!\n",
    "\n",
    "We've already done the basic manipulation of the MNIST dataset shown in the video, so you have X and y loaded and ready to model with. Sequential and Dense from keras are also pre-imported.\n",
    "\n",
    "To add an extra challenge, we've loaded only 2500 images, rather than 60000 which you will see in some published results. Deep learning models perform better with more data, however, they also take longer to train, especially when they start becoming more complex.\n",
    "\n",
    "If you have a computer with a CUDA compatible GPU, you can take advantage of it to improve computation time. If you don't have a GPU, no problem! You can set up a deep learning environment in the cloud that can run your models on a GPU. Here is a blog post by Dan that explains how to do this - check it out after completing this exercise! It is a great next step as you continue your deep learning journey.\n",
    "\n",
    "Ready to take your deep learning to the next level? Check out Advanced Deep Learning with Keras in Python to see how the Keras functional API lets you build domain knowledge to solve new types of problems. Once you know how to use the functional API, take a look at \"Convolutional Neural Networks for Image Processing\" to learn image-specific applications of Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model: model\n",
    "model = Sequential()\n",
    "\n",
    "# Add the first hidden layer\n",
    "model.add(Dense(50,activation='relu', input_shape=(784,)))\n",
    "\n",
    "# Add the second hidden layer\n",
    "model.add(Dense(50,activation='relu'))\n",
    "\n",
    "# Add the output layer\n",
    "model.add(Dense(10,activation='softmax'))\n",
    "\n",
    "# Compile the model\n",
    "model.compile(optimizer='adam',loss='categorical_crossentropy',metrics=['accuracy'])\n",
    "\n",
    "# Fit the model\n",
    "model.fit(X,y,validation_split=0.3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}