Assignment_3_Deep_Learning_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t3rYNW8SS0AK"
   },
   "source": [
    "# Abstract\n",
    "\n",
    "The problem chosen is from Kaggle and its a **Multi class image classification problem**. \n",
    "\n",
    "The Dataset for this problem has been taken from kaggle - [Intel Image Classification\n",
    "Image Scene Classification of Multiclass](https://www.kaggle.com/puneet6060/intel-image-classification)\n",
    "\n",
    "This Data contains around **25000** **images** of size 150x150 distributed under 6 categories.\n",
    "\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "{'buildings' -> 0,\n",
    "'forest' -> 1,\n",
    "'glacier' -> 2,\n",
    "'mountain' -> 3,\n",
    "'sea' -> 4,\n",
    "'street' -> 5 }\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "The Train, Test and Prediction data is separated in each zip files. There are around **14000** images in **Train**, **3000** in **Test** and **7000** in **Prediction**.\n",
    "\n",
    "For this problem the Convolution Neural Network has been built from scratch using Pytorch 🔦 and computations for all the model runs have been performed on the Discovery Cluster.\n",
    "\n",
    "\n",
    "In due course of the tweaking different parameters of the model like activation function, cost function, gradient estimation, network architecture and weight initializations it was observed that these changes have an impact on the accuracy and loss related to the model. The plots for the same taken from tensorboard corroborate the same.\n",
    "\n",
    "**Acknowledgement**\n",
    "\n",
    "This data was initially published on https://datahack.analyticsvidhya.com by Intel to host a Image classification Challenge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0pOaZNYjh23p"
   },
   "source": [
    "### 1.1 Importing necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "u8cLmb5wSqY-"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import torch\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix\n",
    "# from resources.plotcm import plot_confusion_matrix\n",
    "import pandas as pd\n",
    "import glob\n",
    "from PIL import Image\n",
    "from io import open\n",
    "from tqdm import tqdm\n",
    "import torch.nn as nn\n",
    "from torchvision.transforms import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "from torchsummary import summary\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "from torch.optim import SGD\n",
    "from torch.optim import Adam\n",
    "from torch.autograd import Variable\n",
    "import torchvision\n",
    "from itertools import product\n",
    "import pathlib\n",
    "from sklearn.metrics import classification_report, confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a8maO5JMvVix",
    "outputId": "92e810dc-5a68-4069-cf3e-f635cf5138df"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.10.0+cu102\n",
      "0.11.1+cu102\n"
     ]
    }
   ],
   "source": [
    "print(torch.__version__)\n",
    "print(torchvision.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H_jD5JjIiAfC"
   },
   "source": [
    "### 1.2 Loading the dataset source 📁"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WDlVXaImywLF"
   },
   "source": [
    "We can get a glance at the GPU in use and Torch support with the following snippets of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "udgiMbPVy6_n",
    "outputId": "348c9451-02db-43ba-b399-1caf6c95f3ff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fri Dec 10 23:31:05 2021       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  Tesla K80           Off  | 00000000:06:00.0 Off |                    0 |\n",
      "| N/A   54C    P0    75W / 149W |      0MiB / 11441MiB |      2%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "|  No running processes found                                                 |\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a7nFMvELy-Sz",
    "outputId": "b7cea53a-5f8b-4a5c-91ed-c90e86021262"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "torch.cuda.is_available()\n",
    "# Output would be True if Pytorch is using GPU otherwise it would be False.\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oHaV_MdZiHxS"
   },
   "source": [
    "### 1.3 Dataset: Pre-Processing Images 📸"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "reuQBNAE07lc"
   },
   "source": [
    "Reference - [Pytorch - TorchVision Transforms](https://pytorch.org/vision/stable/transforms.html)\n",
    "\n",
    "In order to get a better performing model we will employ standard image pre-processing techniques such as - \n",
    "\n",
    " \n",
    "\n",
    "*   Resize the Image ✅\n",
    "*   Use Random Horizontal Flip ✅\n",
    "*   Use ToTensor to convert image/numpy nd array to torch float tensor ✅\n",
    "*   Use Normalize function in Pytorch transforms(Image Standardization) - Normalize each channel of Input Tensor ✅\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "fJHef86ku5nh"
   },
   "outputs": [],
   "source": [
    "# Image Pre-processing/Augmentation using Pytorch inbuilt transforms\n",
    "# Normalize -> 2x3 matrix -> col is RGB channel and row is mean, deviation\n",
    "# Normalize - Convert 0-1 to  [-1,1], mean and std for 3 channel is 0.5 -> formula = (x-mean)/std\n",
    "transformer = transforms.Compose([\n",
    "                                  transforms.Resize((150,150)),\n",
    "                                  transforms.RandomHorizontalFlip(),\n",
    "                                  transforms.ToTensor(), # 0-255 to 0-1 , numpy to tensor\n",
    "                                  transforms.Normalize([0.5,0.5,0.5],\n",
    "                                                       [0.5,0.5,0.5]) \n",
    "]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1-dLucuu9XD7"
   },
   "source": [
    "### 1.4 Loading the Dataset ⚒\n",
    "\n",
    "In dealing with CNN problems and datasets, pytorch dataloader helps us to load the data in batches so that the GPU is not overloaded.\n",
    "\n",
    "Reference - [torchvision ImageFolder](https://pytorch.org/vision/0.8/datasets.html#imagefolder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "pTFz3C_S9wgx"
   },
   "outputs": [],
   "source": [
    "# Paths for train and test\n",
    "train_path = '/home/zutshi.k/datasets/seg_train/seg_train'\n",
    "test_path = '/home/zutshi.k/datasets/seg_test/seg_test'\n",
    "\n",
    "# train_loader = DataLoader(\n",
    "#     torchvision.datasets.ImageFolder(train_path, transform=transformer),\n",
    "#     batch_size=256, shuffle = True\n",
    "# )\n",
    "train_set = torchvision.datasets.ImageFolder(train_path, transform=transformer)\n",
    "#Dataloader\n",
    "test_loader = DataLoader(\n",
    "    torchvision.datasets.ImageFolder(test_path, transform=transformer),\n",
    "    batch_size=256, shuffle = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "l2cwPbclxNCm",
    "outputId": "5a822966-af3f-4239-feab-3cf75a13cc21"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "initial_count = 0\n",
    "for path in pathlib.Path(\".\").iterdir():\n",
    "    if path.is_file():\n",
    "        initial_count += 1\n",
    "\n",
    "print(initial_count)\n",
    "\n",
    "len(os.listdir(train_path))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OioZTXLw_nhW",
    "outputId": "3a490b54-f081-4345-e5fa-6bd569b90ba9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<generator object Path.iterdir at 0x2b2b7fefb650>\n",
      "['buildings', 'forest', 'glacier', 'mountain', 'sea', 'street']\n"
     ]
    }
   ],
   "source": [
    "#categories\n",
    "root = pathlib.Path(train_path)\n",
    "print(root.iterdir())\n",
    "classes = sorted(j.name.split('/')[-1] for j in root.iterdir())\n",
    "print(classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_94Tla5tzjR0"
   },
   "source": [
    "### 1.5 Peek at the Dataset and Visualize 📊\n",
    "\n",
    "What we can see from the below EDAs - \n",
    "\n",
    "- The Train and Test Datasets have almost same counts of images per category.\n",
    "- From the Pie plots we can see that the dataset is fairly distributed among the Six classes of - 'buildings', 'forest', 'glacier', 'mountain', 'sea', 'street'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "p8nbw_7G9R2Z",
    "outputId": "cdef3452-fe94-40ee-df70-c4081fd6f9a5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'buildings': 2191, 'forest': 2271, 'glacier': 2404, 'mountain': 2512, 'sea': 2274, 'street': 2382}\n",
      "{'buildings': 437, 'forest': 474, 'glacier': 553, 'mountain': 525, 'sea': 510, 'street': 501}\n"
     ]
    }
   ],
   "source": [
    "dictData_train = {}\n",
    "dictData_test = {}\n",
    "for i in classes:\n",
    "  if i not in dictData_train:\n",
    "    dictData_train[i] = len(os.listdir(train_path+'/'+i))\n",
    "  else:\n",
    "    dictData_train[i]+=1\n",
    "\n",
    "for i in classes:\n",
    "  if i not in dictData_test:\n",
    "    dictData_test[i] = len(os.listdir(test_path+'/'+i))\n",
    "  else:\n",
    "    dictData_test[i]+=1\n",
    "\n",
    "print(dictData_train)\n",
    "print(dictData_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 299
    },
    "id": "5-1IgSdI2Wau",
    "outputId": "62d3f058-cc38-4672-a613-002fca32e7c4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BarContainer object of 6 artists>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEICAYAAACuxNj9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfvElEQVR4nO3de7wVdb3/8ddbRLwhiBJHEcQLeY6XJCMv4a+jViqWYb/Ko1miXUjTY5601C5iph37mWaeyoTigPefZSalZmiaaXkBD4FoJikmiII3wMQbfs4f3+/WYbv2mrU3e13Y+/18POaxZ77znZnPzFp7PjPfmTWjiMDMzKyadZodgJmZtT4nCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSTha2xiTdKGl8s+OwNSPpx5K+0ew4rDU5WfRSkl4odK9LWlkYPqIz84qIsRExrYtxLMjLXiHpeUl/lHSMpJq+m5JGSApJ63Zl+Z2Is6blSHq7pJ9JelrSMklzJH1JUp8aljFV0lndF3XnRMQxEfGt7p6vpN0l3ZA/32cl3SPp6O5eToXl3ibps/VeTm/hZNFLRcTGbR3wd+DgQtnlbfXqvRPODo6I/sDWwDnAKcBPG7DcbiVpO+Bu4HFgl4gYAHwcGA30b2ZsZWpJZl2c717A74DfA9sDmwHHAmPrsTyro4hw18s7YAHw/ty/D7CQtMN+ErgU2BT4NbAUeC73b1WY/jbgs7n/KOAO4Lu57qPA2FqWXSjbHXgd2DkPfxD4H2A5aUd8RqHu34EAXsjdXsB2pB3UM8DTwOXAwMI0pwCLgBXAQ8D7cvk6wKnA3/K0VwODOlpOhXW5DLi+ZFv/LG/XZcDtwE65fALwKvBKnv+vcvmWwDV52z8KnFCY1wbAtLydHwS+AiwsjP+X/Nk8D8wDPlwYNxW4CLgB+Afw/lx2VqHOh4DZefo/Au8o24YV1vcO4Icl2+RzwHzgWWA6sGUuH5G3+bqd/a4BZwOrgJfy9vxBs//P1vau6QG4a37HW5PFa8B3gH55h7QZ8FFgQ9IR8s+AXxamb/8P/GreAfQhHUU+Aahs2e3K/w4cW4hpF9LO/B3AU8AheVylHcr2wAdy/INJO+UL8rgdSAmnuEPaLvd/EbgL2CpPezFwZUfLqRDzk8DRJdv603kb9gMuAGYXxk1l9Z31OsAs4HRgPWBb4BHggDz+HNIR+6Y55jnkZAH0Je2Av5qn3Y+0Y9+hsKxlwJi8nPWLywfeCSwB9sif4/j8WfWrtg3breuGpB32vlW2x36khL5bnvd/AbdX+Wxvo8bvWrGuuzXv3AxllbwOTIyIlyNiZUQ8ExHXRMSLEbGCdNT2r1WmfywiJkfEKtKR7xbAkE7G8AQwCCAibouIuRHxekTMAa6stvyImB8RM3L8S4HzC/VXkXZKO0rqGxELIuJvedwxwNciYmFEvAycAXysE01xmwGLq1WIiCkRsaIw/10lDeig+ruBwRFxZkS8EhGPAJOBw/L4Q4FvR8RzEbEQuLAw7Z7AxsA5edrfkc4IDy/UuS4i7szb9aV2y54AXBwRd0fEqkjXpF7O8622DYs2JSWiatvkCGBKRNyXt8lpwF6SRlSZpqg7vmtWAycLq2RpcechaUNJF0t6TNJy0pH6wCrt3E+29UTEi7l3407GMJTULIGkPSTdKmmppGWknfrmHU0oaYikqyQtyvFe1lY/IuYDJ5J21EtyvS3zpFsD1+YLsc+TmnZWUfvO5xnSzqqjuPpIOkfS33JcC/KojtZla2DLtnhyTF8txLMl6Qi/TbF/S+DxiHi9UPYYabtWql9p2Se1W/Yw0tlEtW1Y9BzpwKPDbZLjfKxtICJeIG3HoR1Osbru+K5ZDZwsrJL2jyI+idT0sEdEbAK8N5erHguX9G7SzuKOXHQFqS17WKSLxj8uLLvSY5O/nct3yfF+shhrRFwREXuTdohBanKDtPMcGxEDC936EbGog+W0dzOpua4jnwDGka4PDCA1s1BlXR4HHm0XT/+IOCiPX0xqfmozrND/BDCs3V1lw0nXGdpUW6fHgbPbLXvDiLgSqm7DN2eedt5/ovo2eSLPAwBJG5HO0BaRrqVAas5q809V5vWWEDpR10o4WVgt+gMrgeclDQIm1mMhkjaR9CHgKuCyiJhbWP6zEfGSpN1JO902S0lHr9u2i/cFYJmkocCXC8vYQdJ+kvqRLn6uzNNDSkJnS9o61x0saVyV5bQ3EXiPpHMl/VOex/aSLpM0MMf1MunIeUNSUit6qt387wFWSDpF0gb5zGTnnEwhXYA/TdKmeT2PL0x7N/Ai8BVJfSXtAxxM2ra1mAwck8/qJGkjSR+U1L9kG7b3FeAoSV+WtFneJrtKaovjSuBoSaPy/L4N3J2btpaSksYn87p/mnTzQq3ab09bA04WVosLSBe6nyZdAP5NN8//V5JWkI5mv0a6xlC8D/8LwJm5zumknSTwxtHr2cCdublkT+CbpAumy4DrgV8U5tWPdGH4aVITxttI7eQA3yedwfw2L+su0gXejpazmtxuvxfpjGFebjK7BphJurh8CanJZRHwQJ5/0U9J1wGel/TL3A7/IWAU6U6fp4GfkM5KAM4k3bn2KOms5uekZEREvEJKDmPzdD8CjoyIv7SPu5KImEm6cPwDUnPSfNIF5bJt2H4+fyRdxN4PeETSs8Ak0l1YRMTNwDfydlpMSgaHFWbxOVKyfwbYiXRXVq2+T7rm9JykC0trW1Vtdw2Y2VpO0rHAYRFR7eYDsy7xmYXZWkrSFpLGSFpH0g6ka0vXNjsu65ka8etcM6uP9Ui/BdmG9MO5q0jNTWbdzs1QZmZWys1QZmZWqkc2Q22++eYxYsSIZodhZrZWmTVr1tMRMbjSuB6ZLEaMGMHMmTObHYaZ2VpF0mMdjXMzlJmZlXKyMDOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlZGFmZqXqliwkDctvN3tA0jxJX8zlZ+Q3mM3O3UGFaU6TNF/SQ5IOKJQfmMvmSzq1XjGbmVll9fxR3mvASRFxn6T+wCxJM/K470XEd4uVJe1Ieo79TqRXLd4s6e159A+BD5Ce3X+vpOkR8UAdYzczs4K6JYuIWEx+UXtErJD0INXfqzsOuCq/tP1RSfOB3fO4+fll9eQ3bI0jvTzGrEf43I+ebXYINZn8hUHNDsGapCHXLCSNAN5JetUjwPGS5kiaImnTXDaU1V8gvzCXdVTefhkTJM2UNHPp0qXdvQpmZr1a3ZOFpI1Jr0w8MSKWAxeRXp04inTmcV53LCciJkXE6IgYPXhwxedgmZlZF9X1QYKS+pISxeUR8QuAiHiqMH4y8Os8uAgYVph8q1xGlXIzM2uAet4NJdIL6B+MiPML5VsUqn0EuD/3TwcOk9RP0jbASOAe4F5gpKRtJK1Hugg+vV5xm5nZW9XzzGIM8ClgrqTZueyrwOGSRgEBLAA+DxAR8yRdTbpw/RpwXESsApB0PHAT0AeYEhHz6hi3mZm1U8+7oe4AVGHUDVWmORs4u0L5DdWmMzOz+vIvuM3MrJSThZmZleqRr1W1ns8/YjNrLCcLM7Ma9PYDFDdDmZlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSvlHeWZWF739R2w9jZNFL+B/WjNbU26GMjOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlu6Eq8N1DZmar85mFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzErVLVlIGibpVkkPSJon6Yu5fJCkGZIezn83zeWSdKGk+ZLmSNqtMK/xuf7DksbXK2YzM6usnmcWrwEnRcSOwJ7AcZJ2BE4FbomIkcAteRhgLDAydxOAiyAlF2AisAewOzCxLcGYmVlj1C1ZRMTiiLgv968AHgSGAuOAabnaNOCQ3D8OuCSSu4CBkrYADgBmRMSzEfEcMAM4sF5xm5nZWzXkmoWkEcA7gbuBIRGxOI96EhiS+4cCjxcmW5jLOipvv4wJkmZKmrl06dLuXQEzs16u7slC0sbANcCJEbG8OC4iAojuWE5ETIqI0RExevDgwd0xSzMzy+qaLCT1JSWKyyPiF7n4qdy8RP67JJcvAoYVJt8ql3VUbmZmDVLPu6EE/BR4MCLOL4yaDrTd0TQeuK5QfmS+K2pPYFlurroJ2F/SpvnC9v65zMzMGqSe7+AeA3wKmCtpdi77KnAOcLWkzwCPAYfmcTcABwHzgReBowEi4llJ3wLuzfXOjIi14yXZZmY9RN2SRUTcAaiD0e+rUD+A4zqY1xRgSvdFZ2ZmneFfcJuZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVcrIwM7NSThZmZlbKycLMzEo5WZiZWSknCzMzK+VkYWZmpZwszMyslJOFmZmVKk0Wki6tpczMzHquWs4sdioOSOoDvKs+4ZiZWSvqMFlIOk3SCuAdkpbnbgXpNajXdTSdmZn1PB0mi4j4z4joD5wbEZvkrn9EbBYRpzUwRjMza7LSN+VFxGmShgJbF+tHxO31DMzMzFpHabKQdA5wGPAAsCoXB+BkYWbWS9TyDu6PADtExMv1DsbMzFpTLXdDPQL0rXcgZmbWumo5s3gRmC3pFuCNs4uIOKFuUZmZWUupJVlMz52ZmfVStdwNNa0RgZiZWeuq5W6oR0l3P60mIratS0RmZtZyammGGl3oXx/4ODCoPuGYmVkrKr0bKiKeKXSLIuIC4IP1D83MzFpFLc1QuxUG1yGdadRyRmJmZj1ELTv98wr9rwELgEPrEo2ZmbWkWu6G2rcRgZiZWeuq5eVHAySdL2lm7s6TNKARwZmZWWuo5XEfU4AVpKanQ4HlwH/XMygzM2sttSSL7SJiYkQ8krtvAqW/sZA0RdISSfcXys6QtEjS7NwdVBh3mqT5kh6SdECh/MBcNl/SqZ1dQTMzW3O1JIuVkvZuG5A0BlhZw3RTgQMrlH8vIkbl7oY8zx1Jj0HfKU/zI0l98itcfwiMBXYEDs91zcysgWq5G+oY4JLCdYrngKPKJoqI2yWNqDGOccBV+THoj0qaD+yex82PiEcAJF2V6z5Q43zNzKwb1HI31J+BXSVtkoeXr+Eyj5d0JDATOCkingOGAncV6izMZQCPtyvfo9JMJU0AJgAMHz58DUM0M7OiDpuhJH1J0mfahiNieUQsl/QZSSd2cXkXAdsBo4DFrP4bjjUSEZMiYnREjB48eHB3zdbMzKh+zeII4JIK5ZcCn+7KwiLiqYhYFRGvA5N5s6lpETCsUHWrXNZRuZmZNVC1ZLFuRLzavjAiXgHUlYVJ2qIw+BGg7U6p6cBhkvpJ2gYYCdwD3AuMlLSNpPVIF8H9bg0zswards1iHUlDIuKpYqGkIbXMWNKVwD7A5pIWAhOBfSSNIj3yfAHweYCImCfpatKF69eA4yJiVZ7P8cBNQB9gSkTMq3ntzMysW1RLFucC10s6Cbgvl70rl3+3bMYRcXiF4p9WqX82cHaF8huAG8qWZ2Zm9dNhsoiISyQtBc4EdiadDcwDTo+IGxsUn5mZtYCqt87mpODEYGbWy9XyC24zM+vlnCzMzKxULY8o79OIQMzMrHXVcmbxsKRz/QA/M7Peq5ZksSvwV+Anku6SNKHtOVFmZtY7lCaLiFgREZMj4j3AKaQf1y2WNE3S9nWP0MzMmq6maxaSPizpWuAC0sP/tgV+hX8sZ2bWK9TyPouHgVuBcyPij4Xyn0t6b33CMjOzVlI1WeQ7oaZGxJmVxkfECXWJyszMWkrVZqj8ML8PNSgWMzNrUbU0Q90p6QfA/wf+0VYYEfd1PImZmfUktSSLUflvsSkqgP26PRozM2tJtbyDe99GBGJmZq2rlltnB0g6X9LM3J0naUAjgjMzs9ZQyy+4pwArgENztxz473oGZWZmraWWaxbbRcRHC8PflDS7TvGYmVkLquXMYqWkvdsGJI0BVtYvJDMzazW1nFkcC0zL1ykEPAscVc+gzMystdRyN9RsYNe2J81GxPJ6B2VmZq2lNFlI+lK7YYBlwKycSMzMrIer5ZrFaOAYYGjuPg8cCEyW9JU6xmZmZi2ilmsWWwG7RcQLAJImAtcD7wVmAf+vfuGZmVkrqOXM4m3Ay4XhV4EhEbGyXbmZmfVQtZxZXA7cLem6PHwwcIWkjYAH6haZmZm1jFruhvqWpBuBMbnomIiYmfuPqFtkZmbWMmpphgJYH1geEd8HHpO0TR1jMjOzFlPLgwQnAqcAp+WivsBl9QzKzMxaSy1nFh8BPkx+8VFEPAH0r2dQZmbWWmpJFq9ERJBeeES+sG1mZr1ILcniakkXAwMlfQ64GfhJfcMyM7NWUposIuK7wM+Ba4AdgNMj4sKy6SRNkbRE0v2FskGSZkh6OP/dNJdL0oWS5kuaI2m3wjTjc/2HJY3vykqamdmaqeUC93ciYkZEfDkiTo6IGZK+U8O8p5IeC1J0KnBLRIwEbsnDAGOBkbmbAFyUlz0ImAjsAewOTGxLMGZm1ji1NEN9oELZ2LKJIuJ20uPMi8YB03L/NOCQQvklkdxFavLaAjgAmBERz0bEc8AM3pqAzMyszjr8UZ6kY4EvANtKmlMY1R+4s4vLGxIRi3P/k8CQ3D8UeLxQbyFvPriwUnmleCeQzkoYPnx4F8MzM7NKqv2C+wrgRuA/ebO5CGBFRLQ/Y+i0iAhJsabzKcxvEjAJYPTo0d02XzMzq9IMFRHLImJBRBweEY+RXqUawMaSunro/lRuXiL/XZLLFwHDCvW2ymUdlZuZWQPVcoH7YEkPA48CvwcWkM44umI60HZH03jgukL5kfmuqD2BZbm56iZgf0mb5gvb++cyMzNroFqeOnsWsCdwc0S8U9K+wCfLJpJ0JbAPsLmkhaS7ms4h/W7jM8BjwKG5+g3AQcB84EXgaICIeFbSt4B7c70zu6MJzMzMOqeWZPFqRDwjaR1J60TErZIuKJsoIg7vYNT7KtQN4LgO5jMFmFJDnGZmVie1JIvnJW0M3A5cLmkJ+TlRZmbWO9TyO4txpKah/wB+A/yN9AIkMzPrJTpMFpK2lzQmIv4REa9HxGsRMQ24DxjYsAjNzKzpqp1ZXAAsr1C+LI8zM7NeolqyGBIRc9sX5rIRdYvIzMxaTrVkMbDKuA26OQ4zM2th1ZLFzPz+itVI+iwwq34hmZlZq6l26+yJwLWSjuDN5DAaWI/0qlUzM+slOkwWEfEU8J78i+2dc/H1EfG7hkRmZmYto/RHeRFxK3BrA2IxM7MWVcuP8szMrJdzsjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1JOFmZmVsrJwszMSjlZmJlZKScLMzMr1ZRkIWmBpLmSZkuamcsGSZoh6eH8d9NcLkkXSpovaY6k3ZoRs5lZb9bMM4t9I2JURIzOw6cCt0TESOCWPAwwFhiZuwnARQ2P1Mysl2ulZqhxwLTcPw04pFB+SSR3AQMlbdGE+MzMeq1mJYsAfitplqQJuWxIRCzO/U8CQ3L/UODxwrQLc5mZmTXIuk1a7t4RsUjS24AZkv5SHBkRISk6M8OcdCYADB8+vPsiNTOz5pxZRMSi/HcJcC2wO/BUW/NS/rskV18EDCtMvlUuaz/PSRExOiJGDx48uJ7hm5n1Og1PFpI2ktS/rR/YH7gfmA6Mz9XGA9fl/unAkfmuqD2BZYXmKjMza4BmNEMNAa6V1Lb8KyLiN5LuBa6W9BngMeDQXP8G4CBgPvAicHTjQzYz690aniwi4hFg1wrlzwDvq1AewHENCM3MzDrQSrfOmplZi3KyMDOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlZGFmZqWcLMzMrJSThZmZlXKyMDOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlZGFmZqWcLMzMrJSThZmZlXKyMDOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlZGFmZqWcLMzMrJSThZmZlXKyMDOzUk4WZmZWysnCzMxKOVmYmVkpJwszMyvlZGFmZqWcLMzMrJSThZmZlVprkoWkAyU9JGm+pFObHY+ZWW+yViQLSX2AHwJjgR2BwyXt2NyozMx6j7UiWQC7A/Mj4pGIeAW4ChjX5JjMzHoNRUSzYygl6WPAgRHx2Tz8KWCPiDi+UGcCMCEP7gA81PBAq9sceLrZQXSjnrY+0PPWqaetD/S8dWq19dk6IgZXGrFuoyOpl4iYBExqdhwdkTQzIkY3O47u0tPWB3reOvW09YGet05r0/qsLc1Qi4BhheGtcpmZmTXA2pIs7gVGStpG0nrAYcD0JsdkZtZrrBXNUBHxmqTjgZuAPsCUiJjX5LA6q2WbyLqop60P9Lx16mnrAz1vndaa9VkrLnCbmVlzrS3NUGZm1kROFmZmVsrJoh1JIyTd34n6H257/IikMySdXG2ekkZLurD7Il5zkk6Q9KCky+sw7xGSPtHd8+3E8qfm3+l0drotJf28HjE1k6SjJG1ZQ70zJb2/ETH1VJJOlLRhq86vs5ws1lBETI+IczpRf2ZEnFDPmLrgC8AHIuKIsoqSOntTxAigacmiqyLiiYioOcl0Ybs0y1FAabKIiNMj4ub6h9OjnQhU3LnnRxh12/wawcmisnUlXZ6Ptn8uaUNJCyRtDm+cHdyW+4+S9IP2M5D0Lkl/lvRn4LhC+T6Sfp37z5A0RdJtkh6RdEKh3jfygxPvkHRl2xlLPgt4QNIcSVet6YpK+jGwLXCjpJMk/TLP+y5J7yjEeamkO4FLJQ2WdI2ke3M3Jtf7V0mzc/c/kvoD5wD/J5f9x5rGW7IuFbdZYfzpOd77JU2SpFy+vaSb8+d1n6Tt2p0N9pF0bp52jqTP5/J9JP1B0nTggS7GPELSX/IZ0F/z9+79ku6U9LCk3SUNqvK5nFyY1/15fiPyd3eypHmSfitpg3yGNRq4PH8eG1TZJm+ckeXv/jfztpkr6Z+7sq41bIuNJF2fP4f7Jf1b/j/6vaRZkm6StEWu+7kc95/zd7FpO9EOYp9ISsq3Sro113lB0nlK+4S9JH1S0j35s7hYOYFI2l/Sn/L2/pmkjZX2DavNr+Eiwl2hIx0JBzAmD08BTgYWAJvnstHAbbn/KOAHuf8M4OTcPwd4b+4/F7g/9+8D/LpQ/49AP9LP/p8B+gLvBmYD6wP9gYcL830C6Jf7B3bTOi/Iy/8vYGIu2w+YXYhzFrBBHr4C2Dv3DwcezP2/Kmy3jUm3Zr+xvnX+3CpuM2Aq8LFcZ1Ch/qXAwbn/buAjuX990tHbiMJnNgH4eu7vB8wEtsnr9g9gmzX8vr0G7EI6eJuVv3MiPf/slyWfy8mFed2f59c2z1G5/Grgk7n/NmB0YZqOtklxuy0A/j33fwH4SZ0+w48CkwvDA0j/H4Pz8L+RbpsH2KxQ76y2+JrVdRD7AvI+I5cFcGju/5f8/9I3D/8IOJL0f3g7sFEuPwU4vfA5bF7vdemoW1tOnRvt8Yi4M/dfBnSq2UjSQNKO/PZcdCnpibmVXB8RLwMvS1oCDAHGANdFxEvAS5J+Vag/h3Rk+EvSjqQ77U360hMRv5O0maRN8rjpEbEy978f2DEfhAJsImlj4E7gfKVrH7+IiIWFOvVWbZu12VfSV0jJYBAwT+kMcWhEXAuQp6dd3PsD79Cb1z4GACOBV4B7IuLRNYz90YiYm5c7D7glIkLSXNKOf2s6/lyqzXN27p+V51PJW7YJaSfW3i8K8/q/taxUF8wFzpP0HeDXwHPAzsCM/Hn0ARbnujtLOgsYSDowualOMdVqtdgj4g8VvvurgGty//uAdwH35nobAEuAPUlP1r4zl68H/Knu0dfAyaKy9j8+CdKRWluz3frduKyXC/2rKP9MPgi8FzgY+JqkXSLitW6MpyP/KPSvA+zZtmMtOEfS9cBBpC/7AQ2IqyaS1icdvY2OiMclnUHtn6NIR66r7ZAk7cPq26Writ+B1wvDr5O+D692MF3xOwmrr0/779UG7Sfu5DZpm18t39EuiYi/StqN9P05C/gdMC8i9qpQfSpwSET8WdJRpLO8pmkfu6RbKlR7KSJW5X4B0yLitGIFSQcDMyLi8PpG3Hm+ZlHZcEltX9BPAHeQTgHflcs+Wm3iiHgeeF7S3rmo9MJxO3cCB0taPx+xfwhA0jrAsIi4lXR6OoB0VNVd/tAWa94RPh0RyyvU+y3w720Dkkblv9tFxNyI+A7pES3/DKwgNQvVW8VtVtC2E3w6j/8YQESsABZKOiSvQ78K7d83AcdK6pvrvF3SRnVaj0o6+lwWALvl8t1ITWNlip9HxW3SLEp3ab0YEZeRmm73AAa3/S9K6itpp1y9P7A4fyad/f/qdhVi343q3/1bgI9JeluefpCkrYG7gDGSts/lG0l6e56mUf9LFfnMorKHgOMkTSFduLwIuAf4qaRvkdp9yxwNTJEUpJ1rzSLiXqWLpnOAp0inuMtIp+GXSRpAOjK5MCem7nJGjnkO8CIwvoN6JwA/zPXWJbWxHgOcKGlf0hHxPODG3L8qX9SbGhHf68Z431Blm7WNf17SZFK7/pOkZNbmU8DFks4kHcV/PMfd5iekZpz7lNoGlgKH1GM9OnAGlT+Xa4Ajc9PV3cBfa5jXVODHklYCewEdbZNm2AU4V9LrpM/hWNLZ04X5O78ucAHpu/UN0jovzX+bthPNKsW+F/AbSU9ExL7FyhHxgKSvA7/NB4GvAsdFxF35TOlKSf1y9a+TPttJHc2vEfy4jxYlaeOIeCEf5d4OTIiI+5odVyvzNjOrH59ZtK5JSq+OXZ/UtumdXjlvM7M68ZmFmZmV8gVuMzMr5WRhZmalnCzMzKyUk4WZmZVysjAzs1L/C837HsWd1YvEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "keys = dictData_train.keys()\n",
    "values = dictData_train.values()\n",
    "plt.title('Train Dataset Categories Count')\n",
    "plt.ylabel('Category Count')\n",
    "plt.bar(keys, values, color='cornflowerblue')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 299
    },
    "id": "Q80w2tzS3zKx",
    "outputId": "5aaece35-72c2-4ad5-b2aa-48e4da54ed37"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BarContainer object of 6 artists>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEICAYAAACwDehOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAeRElEQVR4nO3deZwdVZ338c83CwmyJBPSZDAJhM0FRBiMCMJoEHAABcIj4IJCfHAiioMMMgIvHTbRwUEEeXTUIEiAiOCCRAJqDEQEZUkQAgGFiGGSEElYsrBKkt/zxzldVNrbt28nXfcm3d/361Wvrjp16tSvqm7X79apuvcqIjAzMwPo1+oAzMxsw+GkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMOsjJM2VNK7VcdiGzUmhD5H0fGlYI+ml0vSx69DeTEmfqDN/jKQoreMpSTdJOqgb65gg6Y7uxtZdja5H0r9Iul3SSklLJf1G0uENrmO+pAPXP9p1ExG7RsTMnm5X0kckzcrHeLGkWyTt19PrqbHekLRT1evpa5wU+pCI2Lx9AP4XOKxUNqXCVQ/N69wdmA7cIGlCheurhKSjgB8BVwGjgBHAWcBhrYyrK5IGVNj2qcAlwFdI+2Nb4H+AI6pap1UsIjz0wQGYDxyYx/sBZwB/Bp4BrgeG5XmDgWty+TLgXtI//5eB1cDLwPPAN2usYwwQwIAO5acBTwH98nT7ulcCDwNH5vI35/ZX53Usy+XvA/4ArAAWAOeU2q4Zb543BLgcWAwsAs4H+ne2ng4xi5RI/6POPt0RuDWv+2lgCikhAlwNrAFeyuv4fC7fG/hdjvUBYFypve2B2/N++TXwLeCa0vzDgbl52ZnAmzsc39OBOcArwID1PeY1tndI3paj6+yTQaSk8WQeLgEG5XkTgDs61A9gpzx+Zd7maXkf3A3smOfdnuu+kGP4YKv/p3rL0PIAPLTowK99gvgscBfp3e8g4LvAtXneJ4GfA6/LJ9C3AVvmeTOBT9RZxxhqJ4Udcvmb8/TRwOvzieqD+R99mzyv1oljHLBbrv9WUoIZ30C8N+Rt2wzYGrgH+GRn6+mwzjflmLevU2cn4KC8D9vyieuSWvs8T4/MJ95D87YclKfb8vzfA18DNgH2IyXBa/K8N+T9dBAwEPg8MA/YpLSu+4HRwKY9dcw7bO/BwKqOx7dDnfPyerbO++R3wJfqHNuOSeEZYC9SUpsC/LBWXQ89N7j7yABOBL4QEQsj4hXgHOCo3O3wKrAV6Z9vdUTMjogV67m+J/PfYQAR8aOIeDIi1kTEdcBjpBNBTRExMyIezPXnANcC786za8YraQTp5HtKRLwQEUuAi4EPNRjzVvnv4jpxzYuI6RHxSkQsBb5eiquWjwI3R8TNeVumA7OAQyVtC7wdOCsi/hYRdwBTS8t+EJiW1/cqKXlsCryzVOfSiFgQES/VWHdPHPOtgKcjYlWdbTwWOC8iluR9ci7wsTr1O7ohIu7J65gC7NGNZW0dVNbXaBuV7Uj9/GtKZatJ3URXk95t/lDSUFK3whfyiWhdjcx/nwWQdBxwKunKAmBzYHhnC0t6B3AB8BbSu+hBpL5+Oos3b+NAYLGk9qb6kbqfGvFM/rsN8JdO4hoBfAP4Z2CL3P5zddrcDjhaUvmexEDgNtKV07MR8WJp3oK8beT5T7TPiIg1khbw2r5tr19v3et7zJ8BhksaUCcxrBVnHn99nbg6+mtp/EXSa8Mq5CsFg3TyOCQihpaGwRGxKCJejYhzI2IX0rvQ9wPH5eXW9St2jwSWAH+StB1wGfAZYKuIGAo8ROrD72wdPyC9ax4dEUOA77TXrxPvAlLf+vDSNm4ZEbs2uC1/ym18oE6dr+R2douILUlXAirN77iOBcDVHfb7ZhFxAemKZJik15Xqjy6NP0k6sQOglOlGk+6VdLa+jutel2Ne9nvSPh1fZz1rxUm6Ed1+pfgCqYuqfRv+sU471iROCgbppPrlfIJGUpukI/L4/pJ2k9Sf1Kf9KumGKaS+/B0aXYmkEZI+A5wNnBkRa0j9+wEszXU+TroCaPcUMErSJqWyLUjvol+WtBfwkdI6asYbEYuBXwEXSdpSUj9JO0p6d531FCIiSFcz/ynp46U29pM0qRTX88BySSOB/+jQTMf9dQ1wWH7Mtb+kwZLGSRoVEU+QupLOkbSJpH1Y+ymn64H3STpA0kDgc6QT9O9qxV/Duh7z8j5ZTnr66luSxkt6naSBkg6R9N+52rXAF3P7w3P9a/K8B4BdJe0haTCpC6s7uvX6swa1+qaGh9YM/P2TKKeS3g2vJD2R8pU878O5/AXSP+Gl5BuLwD7Ao6QukktrrGMM6YT/fF5+CXAzcHCHel8mdSU9TeqH/w35Bjape2ha+/xcdhSpG2IlcBPwTV67AVsv3iHAt4GFwHLSE0wf6mw9ney3g4Hf5m1aSrrZ/r48b1dgdp53P+lEvbC07BGkJ5iWAaflsnfk7X02tzcN2DbP2zGvayUwA5gEXF5q70jS01rLcxu71jq+PXnMO9knx5IS2Auk7p5pwDvzvMF5+cV5uBQYXFr2C/m4LyBdWXW80Xx+qe64DvvzxNzmMuCYVv9P9ZZBeeea2QZO0nXAHyPi7FbHYr2Xu4/MNlCS3p67uPpJOph0pfGzFodlvZyfPjLbcP0j8FPSo58LgU9FxB9aG5L1du4+MjOzgruPzMyssFF3Hw0fPjzGjBnT6jDMzDYqs2fPfjoi2mrN26iTwpgxY5g1a1arwzAz26hIeqKzee4+MjOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs8JG/Ylm691emPLxVofQkM2O/X6rQzDrMb5SMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs4I/0WzWRP6Utm3ofKVgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWqPTpI0nzgZXAamBVRIyVNAy4DhgDzAeOiYjnJAn4BnAo8CIwISLuqzI+M1s/fpqq92nGlcL+EbFHRIzN02cAMyJiZ2BGngY4BNg5DxOBbzchNjMzK2lF99ERwOQ8PhkYXyq/KpK7gKGStmlBfGZmfVbVH14L4FeSAvhuREwCRkTE4jz/r8CIPD4SWFBadmEuW4yZWZP09S6xqpPCfhGxSNLWwHRJfyzPjIjICaNhkiaSupfYdtttey5SMzOrtvsoIhblv0uAG4C9gKfau4Xy3yW5+iJgdGnxUbmsY5uTImJsRIxta2urMnwzsz6nsqQgaTNJW7SPA+8FHgKmAsfnascDN+bxqcBxSvYGlpe6mczMrAmq7D4aAdyQnjRlAPCDiPiFpHuB6yWdADwBHJPr30x6HHUe6ZHUjaNjz8ysF6ksKUTE48DuNcqfAQ6oUR7ASVXFY2ZmXfMnms3MrOCkYGZmBScFMzMrOCmYmVnBP8fZi/T1T2Ka2frzlYKZmRWcFMzMrOCkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZoc9+S6q/UdTM7O/5SsHMzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlaoPClI6i/pD5JuytPbS7pb0jxJ10naJJcPytPz8vwxVcdmZmZra8aVwmeBR0rTXwUujoidgOeAE3L5CcBzufziXM/MzJqo0qQgaRTwPuB7eVrAe4Af5yqTgfF5/Ig8TZ5/QK5vZmZNUvWVwiXA54E1eXorYFlErMrTC4GReXwksAAgz1+e669F0kRJsyTNWrp0aYWhm5n1PZUlBUnvB5ZExOyebDciJkXE2IgY29bW1pNNm5n1eVX+nsK+wOGSDgUGA1sC3wCGShqQrwZGAYty/UXAaGChpAHAEOCZCuMzM7MOKrtSiIgzI2JURIwBPgTcGhHHArcBR+VqxwM35vGpeZo8/9aIiKriMzOzv9dlUpB0dSNl3XA6cKqkeaR7Bpfn8suBrXL5qcAZ67EOMzNbB410H+1anpDUH3hbd1YSETOBmXn8cWCvGnVeBo7uTrtmZtazOr1SkHSmpJXAWyWtyMNKYAmvdfmYmVkv0mlSiIj/iogtgAsjYss8bBERW0XEmU2M0czMmqTL7qOIOFPSSGC7cv2IuL3KwMzMrPm6TAqSLiA9PfQwsDoXB+CkYGbWyzRyo/lI4I0R8UrVwZiZWWs18jmFx4GBVQdiZmat18iVwovA/ZJmAMXVQkScXFlUZmbWEo0khal5MDOzXq6Rp48md1XHzMx6h0aePvoL6WmjtUTEDpVEZGZmLdNI99HY0vhg0ldRDKsmHDMza6Uunz6KiGdKw6KIuIT0a2pmZtbLNNJ9tGdpsh/pyqHK32EwM7MWaeTkflFpfBUwHzimkmjMzKylGnn6aP9mBGJmZq3XyI/sDJH0dUmz8nCRpCHNCM7MzJqrka+5uAJYSeoyOgZYAXy/yqDMzKw1GrmnsGNEfKA0fa6k+yuKx8zMWqiRK4WXJO3XPiFpX+Cl6kIyM7NWaeRK4UTgqtJ9hOeACZVFZGZmLdPI00cPALtL2jJPr6g8KjMza4lOu48knSrphPbpiFgRESsknSDplKZEZ2ZmTVXvnsKxwFU1yq8G/m814ZiZWSvVSwoDIuLVjoUR8TdA1YVkZmatUi8p9JM0omNhrTIzM+sd6iWFC4Fpkt4taYs8jANuAr7WjODMzKy5On36KCKukrQUOA94C+mHduYCZ0XELU2Kz8zMmqjuI6n55O8EYGbWRzTyieZ1ImmwpHskPSBprqRzc/n2ku6WNE/SdZI2yeWD8vS8PH9MVbGZmVltlSUF4BXgPRGxO7AHcLCkvYGvAhdHxE6kT0e3fxbiBOC5XH5xrmdmZk3UyFdn91+XhiN5Pk8OzEMA7wF+nMsnA+Pz+BF5mjz/AEl+9NXMrIkauVJ4TNKFknbpbuOS+udvVF0CTAf+DCyLiFW5ykJgZB4fCSwAyPOXA1vVaHNi+287LF26tLshmZlZHY0khd2BR4HvSborn5S3bKTxiFgdEXsAo4C9gDetc6SvtTkpIsZGxNi2trb1bc7MzEq6TAoRsTIiLouIdwKnA2cDiyVNlrRTIyuJiGXAbcA+wFBJ7U89jQIW5fFFwGiAPH8I8Ew3tsXMzNZTQ/cUJB0u6QbgEuAiYAfg58DNdZZrkzQ0j28KHAQ8QkoOR+VqxwM35vGpeZo8/9aIiG5uj5mZrYdGfk/hMdKJ/MKI+F2p/MeS3lVnuW2AyflGdT/g+oi4SdLDwA8lnQ/8Abg8178cuFrSPOBZ4EPd3BYzM1tPdZNCPqFfGRHn1ZofESd3tmxEzAH+qUb546T7Cx3LXwaO7ipgMzOrTt3uo4hYDby/SbGYmVmLNdJ9dKekbwLXAS+0F0bEfZVFZWZmLdFIUtgj/y13IbV/CM3MzHqRRn6jef9mBGJmZq3XyCOpQyR9vf1TxJIukjSkGcGZmVlzNfKJ5iuAlcAxeVgBfL/KoMzMrDUauaewY0R8oDR9bv4+IzMz62UauVJ4SdJ+7ROS9gVeqi4kMzNrlUauFD5F+mTyEECkTxtPqDIoMzNrjUaeProf2L39m1EjYkXVQZmZWWt0mRQkndphGtJvHczOCcPMzHqJRu4pjAVOJP0Izkjgk8DBwGWSPl9hbGZm1mSN3FMYBezZ/tOaks4GpgHvAmYD/11deGZm1kyNXClsDbxSmn4VGBERL3UoNzOzjVwjVwpTgLsltf8YzmHADyRtBjxcWWRmZtZ0jTx99CVJtwD75qITI2JWHj+2ssjMzKzpGuk+AhgMrIiIbwBPSNq+wpjMzKxFGvlCvLOB04Ezc9FA4JoqgzIzs9Zo5ErhSOBw8g/sRMSTwBZVBmVmZq3RSFL4W0QE6Yd1yDeYzcysF2okKVwv6bvAUEn/Cvwa+F61YZmZWSs08vTR1yQdRPodhTcCZ0XE9MojMzOzpmvku4++GhGnA9NrlJmZWS/SSPfRQTXKDunpQMzMrPU6vVKQ9Cng08AOkuaUZm0B3Fl1YGZm1nz1uo9+ANwC/BdwRql8ZUQ8W2lUZmbWEp0mhYhYTvrdhA8DSNqa9MnmzSVtHhH/25wQzcysWRr5RPNhkh4D/gL8BphPuoIwM7NeppEbzecDewOPRsT2wAHAXZVGZWZmLdFIUng1Ip4B+knqFxG3kX6NrS5JoyXdJulhSXMlfTaXD5M0XdJj+e8/5HJJulTSPElzJO25XltmZmbd1khSWCZpc+B2YIqkb5C/B6kLq4DPRcQupCuNkyTtQrppPSMidgZm8NpN7EOAnfMwEfh2t7bEzMzWWyNJ4QjgReDfgV8Afyb90E5dEbE4Iu7L4yuBR0i/8XwEMDlXmwyML63nqkjuIn2txjaNb4qZma2vTpOCpJ0k7RsRL0TEmohYFRGTgfuAod1ZiaQxwD8Bd5N+ynNxnvVXYEQeHwksKC22MJd1bGuipFmSZi1durQ7YZiZWRfqXSlcQvq+o46W53kNyV1PPwFOiYi12it/+2qjImJSRIyNiLFtbW3dWdTMzLpQLymMiIgHOxbmsjGNNC5pICkhTImIn+bip9q7hfLfJbl8ETC6tPioXGZmZk1SLykMrTNv064aliTgcuCRiPh6adZU4Pg8fjxwY6n8uPwU0t7A8lI3k5mZNUG9pDAr/37CWiR9ApjdQNv7Ah8D3iPp/jwcClwAHJQ/EHdgnga4GXgcmAdcRvreJTMza6J63310CnCDpGN5LQmMBTYh/URnXRFxB6BOZh9Qo34AJ3XVrpmZVafedx89BbxT0v7AW3LxtIi4tSmRmZlZ0zXyy2u3Abc1IRYzM2uxRj68ZmZmfYSTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7OCk4KZmRWcFMzMrOCkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs4KTgpmZFSpLCpKukLRE0kOlsmGSpkt6LP/9h1wuSZdKmidpjqQ9q4rLzMw6V+WVwpXAwR3KzgBmRMTOwIw8DXAIsHMeJgLfrjAuMzPrRGVJISJuB57tUHwEMDmPTwbGl8qviuQuYKikbaqKzczMamv2PYUREbE4j/8VGJHHRwILSvUW5rK/I2mipFmSZi1durS6SM3M+qCW3WiOiABiHZabFBFjI2JsW1tbBZGZmfVdzU4KT7V3C+W/S3L5ImB0qd6oXGZmZk3U7KQwFTg+jx8P3FgqPy4/hbQ3sLzUzWRmZk0yoKqGJV0LjAOGS1oInA1cAFwv6QTgCeCYXP1m4FBgHvAi8PGq4jIzs85VlhQi4sOdzDqgRt0ATqoqFjMza4w/0WxmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlZwUjAzs4KTgpmZFZwUzMys4KRgZmYFJwUzMys4KZiZWcFJwczMCk4KZmZWcFIwM7OCk4KZmRWcFMzMrOCkYGZmBScFMzMrOCmYmVnBScHMzApOCmZmVnBSMDOzgpOCmZkVnBTMzKzgpGBmZgUnBTMzKzgpmJlZwUnBzMwKTgpmZlbYoJKCpIMl/UnSPElntDoeM7O+ZoNJCpL6A98CDgF2AT4saZfWRmVm1rdsMEkB2AuYFxGPR8TfgB8CR7Q4JjOzPkUR0eoYAJB0FHBwRHwiT38MeEdEfKZDvYnAxDz5RuBPTQ20vuHA060Ooof1tm3qbdsDvW+betv2wIa3TdtFRFutGQOaHcn6iohJwKRWx1GLpFkRMbbVcfSk3rZNvW17oPdtU2/bHti4tmlD6j5aBIwuTY/KZWZm1iQbUlK4F9hZ0vaSNgE+BExtcUxmZn3KBtN9FBGrJH0G+CXQH7giIua2OKzu2iC7tdZTb9um3rY90Pu2qbdtD2xE27TB3Gg2M7PW25C6j8zMrMWcFMzMrNAnk4KkMZIe6kb9w9u/dkPSOZJOq9empLGSLu25iHuGpJMlPSJpSgVtj5H0kZ5utxvrvzJ/1qW7y71e0o+riKmVJE2Q9PoG6p0n6cBmxNRbSTpF0us21Pa6q08mhe6KiKkRcUE36s+KiJOrjGkdfRo4KCKO7aqipO4+hDAGaFlSWFcR8WRENJxM1mG/tMoEoMukEBFnRcSvqw+nVzsFqHkSz1/f02PtNUNfTgoDJE3J75x/LOl1kuZLGg7Fu/2ZeXyCpG92bEDS2yQ9IOkB4KRS+ThJN+XxcyRdIWmmpMclnVyq95/5CwDvkHRt+xVIfkf/sKQ5kn7YExsr6TvADsAtkj4n6We5/bskvbUU69WS7gSultQm6SeS7s3DvrneuyXdn4c/SNoCuAD451z27z0Rc51tqbnfSvPPyvE+JGmSJOXynST9Oh+z+yTt2OEKr7+kC/OycyR9MpePk/RbSVOBh9cx5jGS/pivaB7Nr70DJd0p6TFJe0kaVue4nFZq66Hc3pj8+r1M0lxJv5K0ab5iGgtMycdj0zr7pLjCyq//c/O+eVDSm9ZlWxvYF5tJmpaPw0OSPpj/l34jabakX0raJtf91xz3A/m12LKTZSexn01KvrdJui3XeV7SRUrnhX0kfVTSPflYfFc5UUh6r6Tf5/39I0mbK50f1mqv6SKizw2kd7UB7JunrwBOA+YDw3PZWGBmHp8AfDOPnwOclsfnAO/K4xcCD+XxccBNpfq/AwaRPur+DDAQeDtwPzAY2AJ4rNTuk8CgPD60B7d7fo7h/wFn57L3APeXYp0NbJqnfwDsl8e3BR7J4z8v7bvNSY82F9tc8bGrud+AK4Gjcp1hpfpXA4fl8buBI/P4YNK7sTGl4zYR+GIeHwTMArbP2/YCsP16vuZWAbuR3ozNzq87kb7j62ddHJfTSm09lNtrb3OPXH498NE8PhMYW1qms31S3m/zgX/L458GvlfRMfwAcFlpegjpf6QtT3+Q9Eg6wFaleue3x9eqoZPY55PPG7ksgGPy+Jvz/8vAPP0/wHGk/8Pbgc1y+enAWaXjMLzqbels2FguhauwICLuzOPXAN3q7pE0lHTCvj0XXU36htdapkXEK8ArkpYAI4B9gRsj4mXgZUk/L9WfQ3qX9zPSyaKn7Ud6cRMRt0raStKWed7UiHgpjx8I7JLfVAJsKWlz4E7g60r3Jn4aEQtLdapWb7+121/S50kn/WHAXKWrvpERcQNAXp4Ocb8XeKteuzcxBNgZ+BtwT0T8ZT1j/0tEPJjXOxeYEREh6UHSCX47Oj8u9dq8P4/Pzu3U8nf7hHSy6uinpbb+TyMbtQ4eBC6S9FXgJuA54C3A9Hw8+gOLc923SDofGEp6A/LLimJq1FqxR8Rva7z2VwM/yeMHAG8D7s31NgWWAHuTvg36zly+CfD7yqNvQF9OCh0/oBGkd13tXWqDe3Bdr5TGV9P1fn8f8C7gMOALknaLiFU9GE89L5TG+wF7t59ASy6QNA04lPSi/pcmxdYlSYNJ78bGRsQCSefQ+LEU6Z3oWiceSeNYe7+sq/LrYE1peg3pNfFqJ8uVX5ew9vZ0fG1t2nHhbu6T9vYaeZ2uk4h4VNKepNfP+cCtwNyI2KdG9SuB8RHxgKQJpKu2lukYu6QZNaq9HBGr87iAyRFxZrmCpMOA6RHx4Woj7r6+fE9hW0ntL8KPAHeQLtvelss+UG/hiFgGLJO0Xy7q8uZtB3cCh0kanN99vx9AUj9gdETcRrqkHEJ6h9STfkuON5/wno6IFTXq/Qr4t/YJSXvkvztGxIMR8VXS15O8CVhJ6s6pWs39VtJ+sns6zz8KICJWAgsljc/bMKhG//QvgU9JGpjrvEHSZhVtRy2dHZf5wJ65fE9Sl1ZXysej5j5pFaWnol6MiGtI3a7vANra/x8lDZS0a66+BbA4H5Pu/o/1uBqx70n91/4M4ChJW+flh0naDrgL2FfSTrl8M0lvyMs063+ppr58pfAn4CRJV5BuHn4buAe4XNKXSH2yXfk4cIWkIJ1AGxYR9yrduJwDPEW6LF1OunS+RtIQ0ruMS3MC6knnkOKeA7wIHN9JvZOBb+V6A0h9oCcCp0jan/QOdy5wSx5fnW+uXRkRF/dwzEDd/dY+f5mky0j97n8lJa12HwO+K+k80rvyo3Pc7b5H6n65T+mafikwvort6MQ51D4uPwGOy11OdwOPNtDWlcB3JL0E7AN0tk9aYTfgQklrSMfhU6SroUvz634AcAnptfWfpG1emv+27GSZ1Yp9H+AXkp6MiP3LlSPiYUlfBH6V3/C9CpwUEXflK59rJQ3K1b9IOraTOmuvGfw1Fy0kafOIeD6/Y70dmBgR97U6rg2d95tZdfrylcKGYJLST44OJvU7+sTWGO83s4r4SsHMzAp9+UazmZl14KRgZmYFJwUzMys4KZiZWcFJwczMCv8fWbF0vgP4JjwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "keys = dictData_test.keys()\n",
    "values = dictData_test.values()\n",
    "plt.title('Test Dataset Categories Count')\n",
    "plt.ylabel('Category Count')\n",
    "plt.bar(keys, values, color = 'sandybrown')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "Ra3QEhwZ5Sx5",
    "outputId": "51df98be-0cfb-4d63-a79d-7a952f8841d7"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAD8CAYAAADUv3dIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABBaklEQVR4nO2dd3gc1fm273eLerPkXmVcsLDX2NjYYLohQEwzLYQSQg0GB5KgFCc0UfKLaQkfYCDUCEyIKKFEpndTXHAvEjbuvchWL6vdPd8fMzJrFavN7uyuzn1durQ7c+acZ2Znnj37niZKKTQajUYTHhx2C9BoNJquhDZdjUajCSPadDUajSaMaNPVaDSaMKJNV6PRaMKINl2NRqMJI9p0m0FELheRD20o9zgRWSsilSIyNUxl5onIbIvy+lxErrMir3AhIv8Skfvs1tEVMO/rwyzKa6OInGZFXuEmbKZrXqQa88LvMm/2lHCVfwhd2SKiRMTVsE0p9bJS6nQb5NwDPK6USlFKvWVD+RoLEJGrROQru3V0FhE5wXxeK0WkynxOKoP+BrYnP/O+Xh8qvS1h6h4a7nJbItw13XOUUinAUcB44PbGCYLNL9SEs6w2MghYZbeISCYCP7OIwsrro5SaaxplCjDS3JzRsE0ptTkU5cY6toQXlFLbgPeAUXDgm2i6iKwF1prbrheRH0Rkn4i8IyJ9G443098iIutFZK+IPCgiDnOfQ0RuF5FNIrJbRF4UkXRzX0Ot9loR2Qx8CnxpZltqfnsf27imIiKTRGShiJSZ/ycF7ftcRO4Vka9FpEJEPhSR7i2de0vnJSLrgMOA/5k64ps5tq+IvCEie0Rkg4jcErRvgoh8KyKlIrJDRB4Xkbig/SNF5COz3F0i8pegrOPM61QhIqtEZPwh9Ld4LUyGiMgCESkXkbdFJNM8LkFEZotIialxoYj0Mveli8hzpu5tInKfiDjNfVeZ1/YfIlIC3GsePypIUw/zV1RP8/3ZIrLUTPeNiIwOSjtWRBab51oAJLR0rkGfV5GZfrWIHGVunyEi64K2n29uzwGeAo41P8dSc3u8iDwkIpvN6/+UiCQGlfNH8/y3i8h1ElQ7M6/Pi+bnvsm8vxvu98bX5x7zM/YE5d1TRKpFpMehzrU9iBGWet38TMuBq9pwDwaf079EZJaIzDGv4XwRGXKI8n5hnnuJiNzWaF+L5YpIw/O9zPw8LhGRbiJSaF7P/ebr/lZdm1ZRSoXlD9gInGa+HoBRo7vXfK+Aj4BMIBGYDOzFqBHHA48BXwblpYDPzPQDgTXAdea+a4AfMAwsBfgv8JK5L9s89kUg2SyrYZsrKP+rgK/M15nAfuAXgAu41HyfZe7/HFgHDDfz+xyY2cI1aO28DlyjZo51AIuAO4E48/zWA2eY+8cBx5gas4Ei4LfmvlRgB5CLYTKpwERzXx5QC0wBnMDfgHktaGjLtdiG8WWaDLwBzDb33QD8D0gyyxkHpJn73gT+aR7TE1gA3BD0WfiAm80yE4Hngb8G6ZoOvG++HgvsBiaa5fzSvK7x5nXbBPwOcAMXAfXAfS2c78Xm+RwNCDAUGBS0r6/5uVwCVAF9Gt8/QXn9A3jHvIap5rX4m7nvTGAnRm0yCZiNcU8ONfe/CLxtHpeNcb9fe4jr8wRwf1DZvwH+18nnN5ug58S8b+qBqeY1SOQQ92DQc9twTv8CSoAJZvqXgf+0UPYRQCVwovk5/t085wY/aXO55vss4ELzWqcCrwFvhc0Lw1aQceNXAqXmjf8EkBh0USYHpX0OeCDofYr5AWcHpT8zaP9NwCfm60+Am4L2HW4e2/CBKOCwlm6mxg8NhsEsaHQu3wJXBRnN7Y20vN/CNWjtvDbSsulOBDY32vZn4IUW0v8WeNN8fSmwpIV0ecDHjW7wmhbStuVazGyUlxfD/K4BvgFGNzq+F1DXcC8E6f0s6LNofN6nAeuC3n8NXGm+fhLzyzxo//fASRgP7XZAgvZ9Q8um+wHwmzbe30uB8xrfP+Z7wTDlIUHbjgU2mK+fxzRg8/1Q854cal47L3BE0P4bgM8PcX0mApsbzhP4DvhZR57blp4T8775spVjDtyD6sfnNth0nw3aNwUobiGfOwkyZIwvZy8tPystlttC+jHA/s5cn/b8hTsOM1Up9XEL+7YEve4LLG54o5SqNH869cMwpsbpN5nHNBy7qdE+F8bD3VxZrdE4v4Y8+wW93xn0uhrDTFvKq7XzaolBQN+Gn6smTmAugIgMx6gBjMf4Bndh1IzB+GWx7hB5N9afICIupZSvGf2tXYvGn4sb6A68ZOr4j4hkYNTmbjPPyw3sEJGG4xyN8mn8eX0GJInIRGAXxkPzprlvEPBLEbk5KH2cqV0B25T5pAVpbIkWr5uIXAncimFGYHzmLYWVemB8JouCzlEwPj9Mbd8FpQ8+3+4Y16fxPd3SNUcpNV9EqoGTRWQHhnm/08J5rMK4ZgA/VUrNbeEcmuOgclu5B5ujPc/NgbKUUlXmc9OhckUkCeOXx5lAN3Nzqog4lVL+Q+i1hEjqMhb8IGznxxsBEUnG+EmwLSjNgKDXA81jmhxr7vNhPJzNlRX8ujka59eQ57Zm0rZGW86rJbZg1Iwygv5SlVJTzP1PAsXAMKVUGvAXjAe74Vgruuq05Vo0/lzqgb1KqXql1N1KqSOAScDZwJWmtjqge9B5pSmlRgblc9BnZD4Yr2LUiC8FCpVSFebuLRihh+DrlKSUegUjxNJPgpzP1NgSW4AmcUYRGQQ8A/waI7SSAazkx+vd+J7aC9QAI4M0pSujgQpTV3BMcUCjY+tpek8HX/Pm7uF84AqMXyevK6VqmztBpdRI9WPDWHsMt7lyD3UPdoYdBF0T0zSzOlFuLsYv4Ilm+hMbsrZAa6tEkukG8wpwtYiMEaNB6f+A+UqpjUFp/mAGxAdgxKwKgo79nYgMFqNL2v8BBc3U2hrYAwRo2ZTeBYaLyGUi4hKRSzB+NheG6LxaYgFQISJ/EpFEEXGKyCgROdrcnwqUA5UiMgK4MejYQqCPiPzWbNBJNWuJ7aUt1+IKETnCfDDuwXjg/SJyioh4xGggK8cwkoBSagfwIfCwiKSJ0RA6REROakXLvzFiqZebrxt4BpgmIhPFIFlEzhKRVIxQiA+4RUTcInIBRkyxJZ4Ffi8i48y8hpqGm4xhOHsARORqzEZhk11A/4bGHKVUwNT1D/mxsa+fiJxhpn8V477IMa/bHQ0ZBX3B/NX83AZh1LBb61s9Gzgfw3hfbCWtVRzqHuwMrwNni8jx5jW9h4O9q7Vyd3Hw852K8SVYKkZD710W6WwTEWm6ZgjiDoyGmB0YtY2fN0r2NsZPiKXAHIx4KRjxsZcweiVswGgkupkWUEpVA38FvjZbP49ptL8Eo1aWixH4/yNwtlJqb4jOq6Vj/aaOMRjntRfDFNLNJL8HLgMqMB7wgqBjK4CfAOdg/KRbC5zSAf1tuRYvYcTrdmI02jX0sOiN8fCUYzR0fGGmBaPGGwesxmiYex3o04qW+Rhx0r4YPWEatn8HXA88bub1A0bcE6WUF7jAfL8Pw7T/e4gyXsO4N/6NcV3fAjKVUquBhzFMfBfgwYgrN/ApRkPxThFpuDZ/MrXME6O1/2OM2hZKqfeARzHCJj8A88xj6sz/N5vnuh74ytTzfCvXZwtGKEthhqDCQIv3YGdQSq3CaCz9N8Zzsx/Y2o5y84B88/n+GfAIRsPfXoxr/b4VOttKQ6A9qhARhfFT4ge7tWg0ViNGt7OVQPwhfqG1JZ/nge1KqSb94TX2EZE1XY2mqyEi55uhn27A/RhdvDpjuNkYtfrnWkmqCTPadDWayOAGjP7F6wA/nYiHisi9GDXlB5VSG6yRp7GKqAwvaDTtQYyRezcCi5VSl1ucdzYwSSn179bSajSga7qarsFNwE/aYrjS/jkEsjEacTSaNqFNVxPTiMhTGN2F3hORXBF5S0SWi8g8MedkEGMegZdE5GvgJTHmcnhDjPkhForIcWa6k8SY02GpiCwxu6HNBE4wt/3OthPVRA06vKCJeURkI8ZopbswBmrcLSKTgb8rpcaISB5Gd7rjlVI1IvJv4Aml1FdiTF/4gVIqR0T+hzHM+WuzD3gtcDzwe6XU2Xacmyb60NOxaboSx2NMdIJS6lMRyRKRNHPfO0qpGvP1acARQQPX0kyT/Rr4u4i8DPxXKbX14MFtGk3raNPVaAyqgl47gGOaGTo7U0TmYEzO8nXQiDKNps3omK6mKzEXY9gwInIyRqihvJl0HxI0ilFExpj/hyilViil7gcWAiMwRkGlhlS1JqbQpqvpSuQB40RkOUYD2C9bSHcLMN5scFsNTDO3/1ZEVprH12MMP14O+EVkmW5I07QF3ZCm0Wg0YUTXdDUajSaMaNPVaDSaMKJNV6PRaMKINl2NRqMJI7qfriayyEsXjBWB+2NMZJ6GsXZWCsaKDSkY62AJRg8CX6P/VRgTlO/DmLVrF7CTvLIaNJoIQPde0IQfw1gPw1hxYRSQg7EGVn+MlSDiQ1DqfowVM9ZiLGHe8Pp78soqDnWgRmMl2nQ1oScvfQRwAnAMhtEegVFrjQQUxhLtC4D55v9l5JXV26pKE7No09VYi1GLHYOxwuqJGPMd9LRTUgeow1h/7yPgA2ABeWUhX5pb0zXQpqvpPHnpCRiTxJyHMVtXL3sFWc5+jIUk3wfeJ69su816NFGMNl1Nx8hLTwOmmn+nEznhglCjMFbkfQV4jbyydq8KrenaaNPVtB0jdHAycA3GoodJtuqxHx9GCOIV4C3dIKdpC9p0Na2Tlz4IuApjgpjB9oqJWCqB2cDj5JWtsluMJnLRpqtpmbz0E4FbMeK0eiBN2/kceByj9qsb4DQHoU1XcxCefI8DuGhKZdU59+8pucJuPVHOFuD/AU+SV1ZttxhNZKBNVwOAJ9/jwggh/AkYilLl32zaSqpSaYc+UtMGdgMPAk9o89Vo09XgyfdcCPwVODx4+0XlFV/cVbL/JHtUxSS7gYeAWdp8uy7adLswnnzPScD9wMTm9juU2vHdxi3d3eAOr7KYZzdwB/AseWUBu8Vowos23S6IJ98zEngAY4HFQ/LbfaXfXFtWPin0qrokS4DfkFc2124hmvChTbcL4cn3JAJ3Abm0cYa5hEDg+4Wbth7eekpNJygA/kBe2Ra7hWhCjzbdLoIn33M68CTG7F7t4qHdexefUVV9lPWqNEFUA/cCD+puZrGNNt0Yx5Pv6QH8A3Pp8Y7Q3edf9NmWbeOsU6U5BIuAa8grW263EE1o0B3eYxhPvudioJhOGC7AXpdz3Iq4uDXWqNK0wrhXfSf9PXvGnD9nz5ijn88YRNd0YxBPvicJeBS41qo8h3q9X7+5bedxVuWnaZ5ylbjyyLpnjlA4HBgT6/xi48yzNtosS2Mh+ps0xvDke47E+IlqmeEC/OB2T9jucu6wMk/NwShF7cXeu5JNwwVjLuLl2TPmXGWjLI3FaNONITz5npsxVj8YYXnmIu67szJ1iCGEvOKfPP97NbDxhEKpwAvZM+a8lj1jTleZPjOm0eGFGMAMJ7wIXBjSgvTQ4JBRppJWjKl7emRQLbc5lgBnb5x5lp5EPYrRNd0ox5Pv6Y8R+wut4QKIpD2c2W1JyMvpYihFzcXeu1JaMVyAscC87BlzPOHQpQkN2nSjGE++ZwKwEONhDAtvpiYPrzeWOtdYxGz/aQvWqAFtnad4APBV9ow5p4dSkyZ0aNONUjz5np8DXwC9w1luQKRPfnragnCWGcuUquTld/quOqGdh6UBc7JnzLk+FJo0oUXHdKMQT77nNuA+u8pPCATWLNy0dbhd5ccKSlF9uveB3WtV/+xOZHM/8OeNM8/SD3KUoGu6UYYn33M/NhouQK3DMfz95KTFdmqIBV70n76wk4YLxvzH/8meMSfBAkmaMKBrulGCJ98jwGPAdLu1AGT5/Is+10ODO0ypSl42pu7p0SBiUZbfAOdtnHmWXp04wtE13SjAXELnOSLEcAFKXM5xy+PjvrdbRzSiFNUXeO/OsNBwASYB32TPmNPHwjw1IUCbboRjLqPzb+Bqu7U05q7umbpW1QFe8J+5cL3qOygEWQ8D3sueMSc9BHlrLEKHFyIYM6QwG7jMbi3NolT9B1u37+3r8+vaVRvZr1KWjq3755EW13Ib8wVwxsaZZ9WFsAxNB9E13cjmESLVcAFE3Hl6aHCbUYqqC7x5mSE2XICTgJf1LGWRif5QIhSzW9gtdutojW8TE8ZWiJTbrSMaeM4/ZdEG1XdgmIq7EJgVprI07aBLma6I/EtELurAcX1F5PVQaGoOT77nemzuFtZmRNIeytJDg1ujRKUuuc93eXsHQXSWadkz5twV5jI1rdClTLejKKW2K6XabNYi0qb1x5rDk+85H2NZnajhrRQ9NPhQKEXlBd67e4QhrNAcedkz5txgQ7maFohZ0xWRO0TkexH5SkReEZHfN9p/p4gsFJGVIvK0iPFAiMhQEflYRJaJyGIRGSIi2SKy0tzvFJEHzWOXi8gN5vaTRWSuiLwDrO6IZk++ZwzwMuDszLmHm4BIn3/pocEt8rT/rMWbVO/+Nkp4InvGnPNtLF8TREyarogcjRHTOhL4KTC+mWSPK6WOVkqNAhKBs83tLwOzlFJHYvR9bDxx97VAmVLqaOBo4HoRaZis5CjgN0qpdg+R9eR7soA3TS1Rx9MZaT2syuuat2vo+WAFo56oPGj7Y/O9jHi8kpFPVPLHj2qbHLelLMAp+VUcMctI8//m/dh4/6ePahn9ZCVXvllzYNvs5V4emRfaBv69Km3x33yXhTus0BgH8O/sGXNOtFmHhhg1XeA44G2lVK1SqgL4XzNpThGR+SKyApgMjBSRVKCfUupNAPP46kbHnQ5cKSJLMSYMz8LoHwmwQCm1ob1iPfkeJ/AfILu9x0YKtQ7H8HeTkxZZkddVY9y8f0XSQds+2+Dj7e/rWTYtmVU3pfD7SXFNjnM54OHTE1g9PYV51yYza2E9q/f4KatVLN7pZ/mNKcQ5YcUuPzX1iheW1jP96Kb5WIVSVFzgvbunTWGFxiQAb2fPmNPW2cw0ISJWTfeQiEgC8ARwkVLKAzyDcVO26XDgZqXUGPNvsFLqQ3NfVQcl/Q04rYPHRgwPZHWzxFxOHOQiM/HgrJ78zsuM4+OJdxnbeyY3vXX7pDo4qo8RmUmNF3J6ONhWrnAI1PtBKUV1vcLthIe+8XLzhDjcztD54VP+c5ZsVr3sDCs0JgN4MXvGnKgKX8UasWq6XwPniEiCiKTwY+iggQaD3WvuvwjArBVvFZGpACISLyJJjY79ALhRRNxmmuEi0uFlVDz5np8Bf+jo8ZFEidN51LIQDQ1eUxJg7iYfE5+t5KR/VbFwm/+Q6TeWBliyw8/E/k5S44Upw1yM/WcVfVIcpMcL87f5mTrCHQqpAOxR6Yvu910aiT/njwdm2C2iKxOTpquUWgi8AywH3gNWAGVB+0sxarcrMUx0YdDhvwBuEZHlGJOINJ6v9lmMhrLFZuPaP4EO9Vbw5HuGYsypEDPc1T2zJBT5+gKwr0Yx79pkHvxJAj97vZqWRlNWehUXvlrNI2cmkBZv1GT/eFw8S6el8PAZCdzxWR33nBLPs4u9/Oy1au770tq4rlKUX+C9O6zzHLeTu7JnzGmunUMTBmLSdE0eMhu0zgAGAYuUUlcppV4HUErdrpQaopQ6Til1tVIqz9y+Vik1WSk1Wik1Tim1Xim10WxwQykVUEr9RSnlUUqNUkqdopQqU0p9rpRqXKNuEXNOhZeBFKtP3E7Wud0Ttrmclq/h1T9NuCDHjYgwoZ8Th8De6qamW+83DPdyj5sLcprWZJfs8KMUHJ7l4LXV9bx6cRLr9gdYW3LomnN7mOU/b9kW1bOfZRlajxuYnT1jTuNfcZowEMum+7TZ2LUYeEMpFWnzv94FTLBbhOWIuPK6Z661OtupI9x8ttEHwJoSP14/dE86OB6rlOLad2rJ6e7k1mPjm83njs/quHdyPPUB8AeMbQ6g2qJexrtV+qKHfJfY3VuhLRwOPGy3iK5IzJquUuoys6FrhFLqb3brCcZc2+zPdusIFfMSEo4qd0hZ6ymb59I3qjn2uSq+LwnQ/+8VPLfYyzVj3azfrxj1RCU/f72G/KmJiAjbKwJMednoYPL1Fj8vLa/n0w0+xjxVyZinKnl37Y9u+lZxPeP7Ouib6iAjQRjT24nnyUpq/Yoje3e+bckIK9wTTZP/TMueMafNv84aE9x/vY3pzxWRGebrvMZ95xvnKSLjReTRjuqLVPQsY2HGk+9JwFhKe4TdWkLJ+RWVX9yzd99JdusIJ//Pd/7cf/gujoZabjC7Ac/GmWftbu+BIpINFDaE3tp5bB5QqZR6yKo8o4WYrelGMHcT44YL8HYXGxq8S2V8F4WGC9CTzjXmukTkZREpEpHXRSRJRDaKSHc4UFv93Hx9lYg83jgDERlnjgBdRtBE/eYoz0LzdZ6IPC8in4vIehG5JShds6NPReQWEVltjhz9TyfO0VK06YYRT77nCOBWu3WEg4BIn+e7yNBgpSg7v+6eSG44a42zs2fMmdbBYw8HnlBK5QDlwE0dyOMFjL7vR7aSbgRGw/gE4C4Rcbcy+nQGMFYpNRro6PlZjjbd8PIYHexeFo08k5HW024N4eAfvgtXbKd7NMVym+Oh7Blz+nbguC1Kqa/N17Mx+gG3GRHJADKUUl+am146RPI5Sqk6pdRejLBILw49+nQ58LKIXAH42qMrlGjTDRPmIIjJdusIJ3UOx7B3k5O+s1tHKNmhui181H9hu4wmQkmmY9OJNm4UUhgG1+AtVq5SHNyh2k/rFZizMOYUPgpY2JnZ/6xEm24Y8OR7kumi3XPuz+oWs0NOlaLsgrp7Btitw0J+mT1jTms/8RszUESONV9fBnwFbAQaVoq+8FAHmwOVSkWk4Yvr8naW3+zoUxFxAAOUUp9hLFOfToT0idemGx7uACJpDH7Y2Od0jl0aH1dst45Q8LDv4pU7yIrkkWftxUH7KwffA9NFpAjohjEX9N3A/xOR7zBqpK1xNTDL7FffrskwDjH61AnMNie0WgI8ahq87eguYyHGk+8ZBKwBQjedVYRzmLf+m7e37Zhktw4r2a4yF0yqezz2BrcYnLtx5lnNzcwXkYhIilKq0pwn5UvgVxE4GOoAuqYbeu6gCxsuwHq3a8JWl3Ob3TqsIqAovaDunlAsoR4pPBBlM5FF+ujTg9CmG0I8+Z4hwC/t1mE7Iq687lk/2C3DKh70XbJqJ5m97NYRQkYQyatQNyKSR582hzbd0HInXaiL2KGYnxDfqaHBkcI2lbXgSf95x9mtIwzcmT1jjr53Q4A23RDhyfccTvtbYmMXkdQHM7sttVtGZwgo9l9Qd3e23TrCxFCMaU41FqNNN3TcRZQtMBlq3klJPjyahwbf77u0aBeZXWLAh8kdurZrPdp0Q4An3zMYuMRuHZFGQKT38xlp8+3W0RG2BLrP/6f/nJjqgdEGBgNX2S0i1tCmGxqmo69tszyTnhZ1DVABJfsu9N59mN06bOJmuwXEGtoYLMYcfXat3ToilTqHY9icKBsa/H++y4p3082yJeajjNF6aR9r0aZrPVdgrLqqaYFoGhq8OdBz3rP+s7paWKEx19gtIJbQI9IsxpPvWQGEbQLmrc9tpWJpBa40F8P+OgyAzU9sxrvDC4C/2o8zycnQe4e26ViAna/upGJ5BYkDE+n/K2P0cuk3pfgqfHQ/o7slul/avrN4TJ03oucVDigpmVA3K7CXjK5ay22gDOizceZZNXYLiQV0TddCPPmeUwij4QJ0O74b2bnZB20beNNAht47lKH3DiVtfBpp49PafKy/2k/NphqG3TcMcQm1W2oJeAPsn7ufrFOzLNN9Z/esfZZlFiLu9V2xRhsuYEwWc4HdImIFbbrW8qtwF5h8eDLO5OZ/rSulKFtYRvrE9LYfK6B8CqUUAW8AcQp739tL1mlZiKtdc5Eckg0RPjR4Y6DXty/4f3ps6ym7DLqdwiK06VqE2YB2rt06gqleU40rzUV87+ZXxm0OZ6KT1CNTWXfnOlzpLhxJDmrW15A2rvnacocRcd0VoUODA0r2XujNG9Z6yi7Fydkz5nTVHhyWojs+W8d5QJLdIoIpm1dGxsSMdh/XY0oPekwxflVve34bPc/vyb4v9lG5spKEAQn0PNea8QELEuLHlTmkLD2gmq+K20Se78q1JaTrWu7BCMYUjHfYLSTa0TVd64ioCUKUX1G2qOXQQluo2VSDUor4PvGULyxn4PSBeHd7qdtZ1/rBbUEk5aHMbkusycwa1gd6f/ui/wxtuM1zVfaMOdozOomu6VqAJ9+TBZxut45gKldVEt8nHnemu8N57P7vbvpe1deI8QbMXi4CAW/AIpXwTkryiDv27vPGRcD0lwEley7y3jW8M3nsffcRatYtxJmUTt9rnwCg9KuXqVz2AY4k4wuw24lXkjjk6Kbl11ZS8t6jePduBqD7lN8Q3y+H/Z+/QM36RcT1HEz3s3MBqFz1GYHqctKOPq8zcttLf4z7/P1wFhpr6G8ta7gI6Li7dYItT25h/X3rqdtZR/Hvitn3hdEpoGx+09BC/f56Nv59Y6vHApQvKichOwF3NzfOZCcJAxNYe/taVL0icWCiZfrNocERsWrwnb6r1u0jvVNdNFI8p9Hz4rubbE8dP5W+Vz9G36sfa9ZwAfZ98jQJh42j3/VP0feax3BnDSBQV4V35zr6XvM44nTj3bORQH0dVSs+IvWoszojtaPoBrVOomu61nCxXQUPuLH5Jbr6X990dSB3NzfZt2a3eixA2ri0gxrP+vw8dIvdPpue1ntaaXnI8m8L6wJ9vpnt/0mnB0EkDBiFr2xXu48L1FVRu2UVWVN+B4A43YjTTaCuGhXwGb1J6usQh5PyBf8l9ahzEKctj+9Z2TPmxG+ceZZFMaauh67pdhKz18IJduuIZuocjqGFNg4N9hthhZAO1KhYXMj253/N3ncfwV9b2WS/r3QXzqQ0St59hO0v3ELJe48S8NbiiE8icch4dvzrFpwp3ZD4ZLw71pA03LawcyIw0a7CYwFtup3nZCIgHhntPJDVzbZfXbf7rlm/n7TMUOWfOnYK/W54hj5XP4ozJZP9nz7bJI0K+PHuXEfq2Cn0vfpRxB1P+bzXAEifeBF9r36MzMnXUTZ3NunHX07Fsg/Y89ZMSr/5T6hkH4qT7Sg0VtCm23nOsFtALLDf6RyzOD6uKNzlrg30/foV/6khrbk5k7shDiciDlKPPAPvjjVN0rhSu+NM7U5838MBSDr8OLy71h2UxrtrHUop3Jn9qS7+ih5TZ+Dbv5P6fWEfY3JyuAuMJbTpdh5tuhaR1z1rfzjL8yvZfbH3riNCXY6v8scGyuo13+Lu3nRNS2dKN1xp3akv2QpA7aZluLsPPChN6dzZZJxwBQR8oMweJCIoX9jDq8dkz5jT9hE3moPQDWmdwJPvyQY61cVI8yMb3K4JW1yurQN8vqatgCHgz77rNpaSauky6nveeYC6zSvw15SzddYvST/+cuq2rMC7az2I4ErvSeYZvwbAV1FCyfuP0svs7ZB52jT2Fj6E8vtwZfQma8pvD+RbveZb4noPxZVqdK6I63kY25+bjrtnNnE9wz5QrCGu+2W4C44F9CxjncCT77keeNpuHbHEhJraL57bufukUJfzfaD/12d4H+gKC0yGirs2zjzrHrtFRCM6vNA5uvo8q5bTMDQ4lGX4lez6mffOkaEsowtwst0CohVtup1Dz6hvNSIpD4R4aPCffL/aVEZKRijL6ALouG4H0abbQcz+uTl264hFClOSc7zgDUXeRYEBX7/uP8nSOG4XRffX7SDadDvOWPQS6yEhINLruRAMDfYrx85LvHeEdZL5GOdkuwVEI9p0O07zA+g1lvBsenofBZa28v6+/oYt5aRE1DSSUc6JdguIRrTpdhxtuiHE65AhhSlJi6zKb1Vg0FdvBk7Qn5m1RPQad5GKNt2OM9puAbHOg5nWrBrsU44dP/fe7rEiL81B9MmeMceW2fWiGW26HcCT7xFAL10SYvY7nWMXxcd3emhwbv2N2ypI1mEF63EAA1tNpTkIbbodow9G660mxOR1z+zU0OCVgeyv3g4cp7v2hY5suwVEG9p0O8YQuwV0FTa6XRO3uFxbO3KsTzm2/9x7uw4DhZamE0loDok23Y6hTTdciDjv6p65rvWETflt/fQdlSRZvIyxphHZdguINrTpdgxtumFkYUL8uDKHo7Q9xywPDJ5bGDh2XIgkaX4k224B0YY23Y6hG9HCiUjK/ZkZS9ua3Kcc2y713j4mdII0QejwQjvRptsxQrdgmKZZ5qQkH9HWocG31N+8q4rE1FBr0gC6pttutOl2jO52C+hqBER6PpuR3urQ4CWBIV++G5h4VDg0aQDolz1jjp6Xux1o0+0Y2nRt4Ln0tEMODfYpx9YrvH8ZG05NGpxAWCadjxX0N1QH+M9MX8DvYEO9k+raOGqq46mrSMRfniSB0mSkNEVc+1JxlyYTX5YsyWXJpFQkku51i+7b2wm8Dhnyv5TkhedWVjUZzqsU6tf1N++uIlEbQPgZCGy0W0S0oE23nRSNyIlzQD+HH9x+SPJC5oEVtVWj/wejoFYJZT4HlV63YdhV8XgrEiVQlkygLBnZnyLu/Sm496eQUJYsKWVJpFQmkuF3ih5uCTyYmeE+t7KqyfbFatjc9wMT9QQs9pBst4BoQptu++lwv0+BBFEkxPnpFeeHlNqGOEWwSbdo2FUBocznpLLOTU1NHDVVCdRXJEmgLAlVmoLDNOy40mRJLEsmpSyJ1KpE0pVIzISRSp3OMd8lxK8eX1t3YEHJeuXceqV3ho7j2oeuELQDbbrtJ8WOQgWSnYpkpw/ifZBWA5RBa4ZtxkDL/Y4Dhl1bHU9tVQI+MxyiSlNwmoYdbxp2ankiqTUJEpEDC/K6Z5YWbt0BGGGF6fW37NFhBVvRptsOtOm2H7FbQHsQQ2+aK0CaKwAJ9ZBe3bC3VcP2KyjzOyivd1FV56a2OoG6ygR8ZUmiSlOgNFmc+1OI259CfFmKJJUlkVqeRJrXLUmhOqdNLtfEzS7X1oE+X//v1PC5HwaO1mEFe4mzW0A0oU1X0yICToFMR4BMt9eIX3c7EE5tNX7tNePX5fUuqmvjqK1KwFuRiL8sSRri1679qcTtTybBbHBMrUwk3eeUQz/E5tDgp3eUBH7pnaFHndmPrum2A2267UevWd8GBOJE0SPOT484PyTXQVZFw95WDbu6IX7tdVFdY4RD6isSxV+WjCpNxlGR7OZex8+WZrOhdzjOR9MyXkecz24N0YQ23fajTTfECCQ5FUkN8evUWprErzf3n/xtUd+M9FPrPteLTNpPAtxit4aoQZuuJurwO1y16w6bOsAl9PTXrdgMAT2Rtr347RYQTcRMV6Iwomu6NvP98EvnKYezv4gzzpV4Qofm2tVYig4vtANtuu1H32A2UudO3bOz18QDjWeuhHGTIG6VnZo0VLeeRNOANt3206nlYzSdY7lnWjEiB80g5k45S/+8tRf9TLQDbbrtJKe4qAaotVtHV6Q8deDaitRBkxpvd7oHjxZH2nw7NGkAbbrtQptuxyixW0BXZLnnxnJEml2W3Z0ytSc69GMX2nTbgTbdjqFNN8zs6DVxoTcurcWBEA5n98Hi7PtNODVpDqBNtx1o0+0Y2nTDSEAcvuLDL81qLV1cyjlHAOVhkKT5EV9uQWFF68k0DWjT7Rj77BbQlVh32PnfKIe71XXpxJHc3RGXsyQcmjQH0M9CO9Gm2zF22S2gq1DvSizb0v+UUW1N7046bQLIjlBq0hzERrsFRBvadDvGD3YL6CqsPOK6pYhktjW9iDvRlXDsulBq0hzEersFRBvadDvGWrsFdAWqEntt2t/t8GPbe5wzYeIkcH0fCk2aJugvuHaiTbdjrLFbQFdg2eibdiCtTPPYDCLicCef2XRNH00o0DXddqJNt2NsQE/yEVL2Zo1aVpvY/ZiOHu+MG34UkvydlZo0zaJNt51o0+0AOcVF9egGhJChQK3KubrTqxHEpZyXDgQskKRpGR1eaCfadDuOjuuGiE0Dz/zG70rI6Ww+DlfvYeLsqQdMhI5yQM/y1k606XaclXYLiEX8jrjq9YPParVPbluJSzlvKHoWrFCxOLegUE912k606XYcPcFKCCga8YuFiKOPVfmJI7W3wz10gVX5aQ5Cx8w7gDbdjqNN12Jq47vt2N1j7NFW5+tOPnM8yB6r89Vo0+0I2nQ7SE5x0RZgu906YollnhvXI9Yv3S4Sl+KMH19kdb4abbodQZtu59C1XYsoTR9SVJXct8lcuVbhSjzuOHDqlnbr2J9bUKivZwfQpts5tOlaxPJRN3gRkVDlL+JwupJO1bPDWYe+9zuIXg24c3wb7gJv27GDL6oqyXQ6eWew0ch/6/ZtbPB6Aajw+0l1Onkze3CTY1/av4/XSktRwMXpGVyZaUxp8PCe3cytrGJEQjwz+/QF4J2yMkr9/gNpQsnWvifM97mTJ4a6HFf8qAm+mi+XomrHhLqsLsDHdguIVnRNt3PMB8I63PT89HSe7j/goG1/79uPN7MH82b2YH6SmspPUlKbHLe2ro7XSkspGJTNm9mD+byqkk1eLxV+P6tra3lr8GDcIqypq6U2EODN8jIu7dYt5OcTEKd37dCLeoe8IJO45HPj0Ss6W8GHdguIVnRNtxPkFBfVFY3I+QQ4N1xljk9KYlu9t9l9Sik+qKjg+QEDm+xb561jdGIiiQ7je/boxCQ+rqjg590y8CmFUoragMKF8MK+fVye0Q136H7tH2DNsIu/VQ7XSSEvyMTh7p8jjsyvVWDfcVbnXbBgGat37CYlPo4/nGmc0kvfLmZPhfG9XOOtJzHOza2nn9Dk2OIdu3l76WoCSjFx8AAm5wwF4OV5S9hZVkFOn55MGT0CgI9Xr6V3eiqj+oXtu6oxO3ILClfYVXi0o2u6nWeO3QIaWFRTQ5bTRXZc0xG0w+LiWVRdTanfT00gwJdVlezw1ZPscHJicgoXbNpId5eTVKeT5bU1nJbatLZsNV53Ssn2PsePCXlBjXCnTB0E1Fmd7/jB/bn+xAkHbfvFsUdx6+kncOvpJ+Dp37tZowwEFG8uXsV1J0zgD2ecxJLN29lZVsH20nLcTie5Z5zIlv1l1HjrKa+pZVNJqZ2GC/CRnYVHO9p0O8+7dgtoYE55OVPSmjfLIfHxXJeZxXVbNvOrrVsYEZ+A06zJXpuVxZvZg/lTz148uncPN3fvweulpfxu+zaeKtkbMr0rRl6/CpH0kBXQAg5nRn+HK9vyePyQHlkkxbmb3aeUYtmWHYwd2LfJvs37SslKSSIrJQmX08GYgX1ZtX0XTodQ7/cTUAp/IIBDhA9WruGMUcOtlt5edGihE2jT7SQ5xUVbgeV26/ApxceVFfw0Na3FNBdmZPB69mBeGjiINKeTbPfBNeLVtbUoIDsujg8qKvhH335s9taz0dt8OKMzVCT3W1eWPiRkXcRaw508ZSxhXFBx/d59pCbE0yM1ucm+sppaMpISD7zPSEygrKaWXmmppMTH8chHX3FE317sraxCoejfLezfU8EE0DXdTqFjutbwLjDaTgHfVlcxOC6O3u7ma1oAJT4fWS4X2+vr+biyglcGDjpo/2N793B37974lMJvtjU5BGoD1k/UtXz0jfsQGWJ5xm1EHAnpzvgjv/TXLTsxHOUt3bydMc3UclvjvLEjD7x+bu5CLhrv4ePVa9leWsHwXt05ZkjT+H2I+Sq3oHB3uAuNJXRN1xreCVdBv9++jUs3bWKj18sp637gjdJSAN4rL2dK2sG13N2+em7YuuXA+99s38bZG9YzfdtWbu/ZizSn88C+jysqGJmQQE+XmzSnkxHxCZy3YQN1AcWIhARLz2FXj6MW1cV3s3y4b3txJZ58LDg2hbocfyDAiq07GTOg+Skl0hMTKK2uOfC+tKaW9MSDr/nKbTvp3y0dr89HSWU1V046iuVbd+D1hX1a51fCXWCsoWu6FpBTXPRt0YicdUDIa24P9e3X7Pb/69O0FtXT5eafQd3LZjeq2QZzWmrqQY1nf+zZsxMqW0Yh/qIRv2g5BhJGRJxuV+JJ2301n7V8YSxg7a699ExLOSiEEMyAzHT2VlZRUllNemICSzdv5/Jjxh7Y7w8EmLtmA9eeMIG9lVU0dCpRZqwXnM3mGwJ8wOvhKixW0TVd63jJbgHRwPrB53wTcMYNs1tHA66EsccicZZM0zn72yU89sk37Kmo4t7/fcL89ZsBWLplB2MGHPylWFZTy7NfGpOfOR0Ozj9qFM98uYAH3/+CIwf0oXf6j1+AX/+wifHZ/YlzOemTnorXF+ChD76kX7d0EltouAsRH+cWFIauZbWLIErpfuJWUDQiZzDGLPqh79wapfic8RVfHv9QLeLoYbeWYPz1G1fUV/7XY7eOKOCXuQWFL9otItrRNV2LyCku2gB8ZbeOSGZVztWLI81wAZzubI840ufZrSPCqQXesltELKBN11ry7RYQqdQkdN9akjUq5PMrdBR3ytTeQL3dOiKY13ILCsvtFhELaNO1lteAmlZTdUGWjb5pCyLWdoOwEIczK1tc/cI+gVEU8bjdAmIFbboWklNcVA68bLeOSGNftxErqpN6HWu3jtaISz5nJFBmt44IZGFuQaFe8sgitOlaz8PoWawOoECtGHldVNxn4kjKcsQdscRuHRGIruVaSFQ8DNFETnFRMRE0H4PdbOl/6rd+V+LI1lNGBu6k044B0csw/cgeoMBuEbGENt3Q8JDdAiIBv8NVu+6w80I68MBqRFwJroRJG+zWEUE8m1tQaPmMbF0ZbbohIKe46HNgkd067KZ4+GXzlMPZ/BC6CMaZMGESuIvt1hEB1ACP2i0i1tCmGzoetluAndTFpe3Z1WvCeLt1dAQREXfymboXCjyZW1C4024RsYY23dDxGsYItS7Jcs+0YkRS7NbRUZxxw8YiKQvt1mEj1cD9douIRbTphoic4iIfcJfdOuygPHXQmoqUgZYvhxNu4lLO6waEfRqvCOEJPYVjaNCmG1peAbrcWlLLPDdWIhL195bD1WuoOHt9Y7cOG6gCHrBbRKwS9Q9GJJNTXBQAbrNbRzjZ3vuYhfVxqUfZrcMq4lLOHU6YV3yOAGblFhTusVtErNIlTVdErhKRVqfxF5F7ROS0zpSVU1z0P+DzzuQRLQTE4ft++M+7263DSsSR2svhHtaVYrt7gb/ZLSKW6ZKmC1wFtGq6Sqk7lVIfW1De7+kCo9R+GHLB18rhHmy3DqtxJ59xNEhXiW/emVtQWGq3iFgmIkxXRLJFpFhE/iUia0TkZRE5TUS+FpG1IjJBRDJF5C0RWS4i80RktHlsnoj8PiivlWZ+2SJSJCLPiMgqEflQRBJF5CJgPPCyiCw1t90pIgvNY58WMebmN/VcZL7eKCJ3i8hiEVkhIiPaen45xUWLiPEZyOpdSaVb+51s6zpxoUIkLtmZcPT3dusIAyuAp+0WEetEhOmaDMXo2zrC/LsMOB6jlvgX4G5giVJqtPm+LZMpDwNmKaVGAqXAhUqp14HvgMuVUmOUUjXA40qpo5VSo4BE4OwW8turlDoKeNLU1R5+j/HTLSZZOfK6ZYh0s1tHqHAlTJoEzljuAqiAm3ILCjvUW0NEfisiSVaJsTq/SCKSTHeDUmqFUioArAI+UcayFiuAbAwDfglAKfUpkCUira21tUEptdR8vcjMpzlOEZH5IrICmAy0NFfAf9uQV7PkFBeVAL9rzzHRQlVS7437M4bbtpx6OBBxON1Jp+2zW0cIeTG3oLAzk/D/FmjWJEWkI4u4tZhftBNJphs8vjsQ9D7AoRfQ9HHweQTP2Rqcp7+5fMSY4/UJ4CKllAd4plEezWlsNq/WyCkumg182N7jIp1lnpt2IRLWxbrswBk/8mgkMRZnISsB/tjWxCKSLCJzRGSZGZK7C6ON5DMR+cxMUykiD4vIMuBYEblCRBaYIb1/NhixiJwuIt+aYbvXRCRFRG5pnF8sEUmm2xpzgcsBRORkjJ/65cBG4Chz+1FAWxpyKoCGlf8aDHavGCOoLrJMcfNMwxjtExPsyRq9tDYxK2JXhLCauJRzE4i9RtEb2zkQ4kxgu1LqSDMk9wiwHThFKXWKmSYZmK+UOhLD1C8BjlNKjcGotFwuIt2B24HTzLDdd8CtSqlHm8kvZoimJdjzgOdFZDmGaf3S3P4GcKWIrALmA2vakNe/gKdEpAY4FqN2uxLYCYS0e1BOcdGGohE5ecRA53OFBFYdcVWnVoOY/fmDrNw0j9TEDG772XMHtn++8k3mrnobEQejBk5k6jE3HHTc/srdvPjZTCqq94MIx+WcxSmeCwF4a97TrN6ygP5ZQ7ly8gwAFqz5iKrack4ZfWFn5OJw9csRR9bXKlAS9SPuTF7JLSh8rZ3HrAAeFpH7gUKl1FyRJuux+jGeTYBTgXHAQjNdIrAbOAY4Avja3B4HxPzqHXo1YBsoGpHjxFjE8hi7tXSGDYN++tWGwWcf35k8fti+nHh3Ai9+dv8B012zbQkfLPk30376V9zOOCpq9pOaeHAbXVlVCeXVJQzoMZxabzX3/3cavzrjHjKSuvPsR3dz89kP8vIXD3HyqAvokd6Pp967jelTZuJ0dr6eEfCXbfOWP5dFy2GoaGE7MCq3oHB/ew8UkUxgCnA98AlwDTBeKbXX3F+plEoxX98M9FVK/blRHucAlymlLm0m/43B+cUS0RReiBlyiov8wM8xelREJX5HXNWG7CnDOpvP0L6jSUo4uD107ur/8ZMxP8ftjANoYrgA6clZDOgxHICEuCR6ZwyitGovIg78AT9KKby+OpwOF58se5WTRk21xHABHM70fg5X9nxLMrOXaztouH2BaqXUbOBBjPBecMiuMZ8AF4lIT/P4TBEZBMwDjhORoeb2ZBEZbh5zqPyiGm26NpFTXLQJuNZuHR1ldc6V3yGOXqHIe3fZVtbtWMGDb07nkXd+x6bdh57atqRiJ1tLfiC7Zw4JcUmMHDiBmW/cQHpSJolxyWzcXcSRgztVIW+CO3nKGIxYZbTyz9yCwvc7eKwHWCAiSzEmdboPo3/v+801fCmlVmPEbj80w4MfAX2UUnswBiq9Ym7/FqO7KIfKL9rR4QWbKRqRMwu4yW4d7aEmPnPHt8fck45F/ShLKnby1Hu3HQgv/PXVaxnWdwwXH/drNu35nhc+vpe8S2fTTNyQuvoaHnnnd5wx9nLGHHZCk/0vf/EQJx5xLlv2rqVo6yL6ZR3GmUddYYVs6qs//cJft/QkSzILL0uBSbkFhXrOYBvQNV37uRVYZreI9rDcc+N6qwy3OTKSezBm8PGICNk9RyAiVNY2XaTX7/fxzId5jB92arOGu2XvWlCKnhkDWLz+S679yZ3sKdvO7rKtluh0JZ40CRybLMksfOwHLtCGax/adG0mp7ioDqM7TaXdWtrC/vShq6uS+4R0IMTowcexZvtSAHaVbsHn95GSkH5QGqUUL3/xEL0zBnLq6Iubzadw4QucdfTVZow3AIBDBK/PmiW/RJxuV+LJOyzJLDwEgMtyCwr1GnA2ok03AsgpLvoeuBTjoYhoVoz6la/Z3/kd5IWP7+Pht25mV9kWbp99Cd8Uv8uxh59JSfkO/vrqtbzwyX384pQ/ISKUVu3liXeNBvD1O1eyYO1HrNm+hL+9/iv+9vqvWLX5x7atZRu+YmCPw8lI7k5SfAr9sobw19euo95fT/+sIVbJx5Uw5hgkPlrmTM7rRBxXYxE6phtBFI3I+S3wD7t1tMTWvifOWzP8kqju5hYK/PWbVtRXvuGxW0crvANMzS0o1A+8zeiabgSRU1z0CMaQ5IgjIK66tUMvbHU6zK6I0z3II470SO7UvwS4QhtuZKBNN/K4BfjAbhGN+X7Yz+Yph2ug3ToiFXfK1L5Avd06mmE98NPcgsIKu4VoDLTpRhjmwImfYQxLjgi87tS9O/pMGmO3jkjG4cwaJK7+kbae2h7gjNyCwl12C9H8iDbdCCSnuKgcY1KRiJi/dfmoX61GJL31lF2buOSzPUDTvm32UAlMyS0o/MFuIZqD0aYboeQUF23DmNt3o506KlL6rytPGxwrk7uEFHEkZTrjRkbC1I/1wEW5BYXf2S1E0xRtuhFMTnHRZuAUYLNdGpZ5btxPxyah7pK4kk49BmSbjRLqgAtzCwojrl1AY6BNN8LJKS7aiGG81gyjage7eo77zhufMT7c5UYzIq4EV+Jxdo1SqwXOyy0o/J9N5WvagDbdKCCnuGg9Rqhhe7jKVIi/6PArMsJVXizhjD/6WHAXhbnYauAsXcONfLTpRgk5xUVrMdaJa8sk7Z1m3WHnfhNwxg0NR1mxhoiIO/mntWEssgI4M7eg8NMwlqnpINp0o4ic4qINwCRCPLu+z5lQvnnAaW1eYl7TFGfc0LFISkhXITHZC5yeW1A4NwxlaSxAm26UYa4qfCrwVqjKWHnENUsQR49Q5d9ViEuZmomxbE2oKAIm5hYUzgthGRqL0aYbheQUF9UAFxKCIcPViT227Ms8Qs+vYAEOV88h4uwdqgETn2DMibs+RPlrQoQ23Sglp7gokFNcNB34AxbWppZ5btqKSLxV+XV14lLOPRzrp+18BiOGW2pxvpowoGcZiwGKRuScArwCdGr5nJJuOcuXHfnr0dao0jTgrSz8IlC/xooVJvzAn3ILCh+2IC+NTeiabgyQU1z0GcbigB1uTFGgVo68Vg+CCAHu5NPHg3R2/oOtwMnacKMfbboxQk5x0XaMvrwPdeT4zQNO+8bvShxprSoNgEhcsjNhQme6+hUCY3ILCr+ySpPGPnR4IQYpGpFzPvAc0HTt8mbwO9w1X5zw8H7EqefLDRFKBfx1pY+vB197lq2vB2bkFhT+PVS6NOFH13RjkJziojeBkcDbbUlffPjlC7ThhhYRh9Od9JP2zEC2BjhOG27soU03RskpLtqRU1w0FbgcKGkpXW1cxq5dPcePC5uwLowzPmc8kri4lWQ+4P+AI3MLCsMxuEITZnR4oQtQNCKnF0af3gsa71sw7k9fVaYOPD78qromAd/2770V/xlG8xWe74BrcwsKl4dZliaM6JpuFyCnuGhXTnHRhRhLvR+YNKcsbfD3lSkDQrqcuuZgHK6+h4uze+MBE9VALnCMNtzYR9d0uxhFI3JSgL8At86dNHN1fVzqWLs1dTUC/rLt3vLnMoEE4FWMvrcb7VWlCRfadLsocyddOGj56OkPYKzHpgkz3qr38wPe1c/kFhR+bbcWTXjRptvFmTXt04nAA8CJdmvpIqwBbpv+1OTX7RaisQdtuhoAZk379GTgNuA0m6XEKt9jfLm9OP2pyT67xWjsQ5uu5iDMmu9twDl2a4kRvgNmAm9Of2pywG4xGvvRpqtpllnTPh2NMYPZxYCedaz9fAr8bfpTkz+2W4gmstCmqzkks6Z9mgVcDdwA6OV7Dk0JkA88O/2pyeFeI00TJWjT1bSJWdM+FYx47zSM0IPbXkURgwI+B57GCCHU2StHE+lo09W0m1nTPu0GTMXobnYqXdOAFwFvAK9Of2ryOrvFaKIHbbqaTmEa8PkYsd/JQJy9ikKGwlgQ9A3gv9OfmrzRXjmaaEWbrsYyZk37NAk4ASMMcRpwJCC2iuoc64HPMBrFPp3+1OSdNuvRxADadDUhY9a0T7tj1H4nAeOAsUCyraJaxgcUY3Tx+hz4bPpTkzfbqkgTk2jT1YSNWdM+dQCHYxjwUcAo4DBgIOGNC+8CfgCWmn9LgJW6EUwTDrTpamxn1rRPncAAYAiGCQ8AMs2/bkH/MzBixi6gYT23gPlXBZQ1+isFdgMbgU3m/83Tn5pcE+pz0mhaQpuuRqPRhBE9n65Go9GEEW26Go1GE0a06Wo0Gk0Y0aar0Wg0YUSbrkaj0YQRbboajUYTRrTpajQaTRjRpquxBRFJFpE5IrJMRFaKyCUiMk5EvhCRRSLygYj0MdNeLyILzbRviEiS3fo1mo6iB0dobEFELgTOVEpdb75PB94DzlNK7RGRS4AzlFLXiEiWUqrETHcfsEsp9Zht4jWaTuCyW4Cmy7ICeFhE7gcKgf0YczF8JCJgDPPdYaYdZZptBpACfBB2tRqNReiarsY2RCQTmAJcjzF94plKqWObSbcBmKqUWiYiVwEnK6WuCqdWjcYqdExXYwsi0heoVkrNBh4EJgI9RORYc79bREaayVOBHSLiBi63RbBGYxE6vKCxCw/woIgEgHrgRow5bR8147su4BFgFXAHMB/YY/5PtUOwRmMFOryg0Wg0YUSHFzQajSaMaNPVaDSaMKJNV6PRaMKINl2NRqMJI9p0NRqNJoxo09VoNJowok1Xo9Fowsj/B461yzn5KVwZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.pie(dictData_train.values(),\n",
    "        explode=(0, 0, 0, 0, 0, 0) , \n",
    "        labels=dictData_train.keys(),\n",
    "        autopct='%1.1f%%')\n",
    "plt.axis('equal')\n",
    "plt.title('Proportion of each observed category - Train data')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "xktuxwbt5eJJ",
    "outputId": "0984cc64-4fa1-4986-b501-ad0ad1951571"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAD8CAYAAADUv3dIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABA40lEQVR4nO2dd3xb1dnHv4+WlzyyN5iQhJhEJCSBsAuUsnfZUMpugJIO85a0FJoW+hZKKX2BQCiFYggFs4dDmQESwkggm9iQ5Sw70/Geks77x71OFMeObelKV5LP9/Pxx9K9Zzx3/XTuGc8jSik0Go1GExscdhug0Wg0PQktuhqNRhNDtOhqNBpNDNGiq9FoNDFEi65Go9HEEC26Go1GE0O06HaCiFwpIu/bUO+xIrJKRGpF5PwY1TldRGZZVNYnInKDFWXFChF5RkTutdsOTfuISK6IKBFx2W1LJNgiuiJSKiINpqBsNW92rx22tLFrn4uqlHpeKXWqDeb8CXhUKeVVSr1hQ/0aCxCRa0TkM7vtiBQROd58XmtFpM58TmpD/g4Io0wlIiOiZO+JIrIpGmVHip0t3XOUUl5gAjAJ+H3bBLH8RYvDX88DgW/tNiKeicNrFldYeX6UUvPMBoAXGGNuzmndppTaYFVdyY7t3QtKqc3Af4GxsPvX71YRWQWsMrfdKCKrRaRCRN4SkcGt+c30U0VkrYjsEJEHRMRh7nOIyO9FZL2IbBORZ0Uk29zX2qq9XkQ2AHOAuWaxleav99FtWyoicoyILBSRKvP/MSH7PhGRe0RkvojUiMj7ItK3o2Pv6LhEZA0wHHjbtCOlnbyDReRVEdkuIutEZGrIviNF5AsRqRSRchF5VEQ8IfvHiMgHZr1bReR3IUV7zPNUIyLfisik/djf4bkwOVhEFohItYi8KSK9zXypIjJLRHaaNi4UkQHmvmwRecq0e7OI3CsiTnPfNea5fUhEdgL3mPnHhtjUz3yL6m9+P1tElpjpPheRw0LSHi4ii8xjLQRSOzrWkOtVbKZfKSITzO3TRGRNyPYLzO15wEzgaPM6VprbU0TkbyKywTz/M0UkLaSe35jHXyYiN0hIi9A8P8+a1329eX+33u9tz8+fzGvsCym7v4jUi0i//R1rd+jkmo0QkU/Ne2SHeZ4RkdZnbal5bi5tp1yneZ52iMha4Kw2+68NuR5rReRn5vYMDE0ZLHta4oOlk+ciZiilYv4HlAKnmJ+HYbTo7jG/K+ADoDeQBpwM7MBoEacAjwBzQ8pSwMdm+gOA74EbzH3XAasxBMwLvAY8Z+7LNfM+C2SYdbVuc4WUfw3wmfm5N7AL+AngAi43v/cx938CrAFGmeV9AtzXwTno7Lh2n6N28jqAb4C7AY95fGuB08z9E4GjTBtzgWLgl+a+TKAcyMcQmUxgsrlvOtAInAk4gb8AX3ZgQ1fOxWaMH9MM4FVglrnvZ8DbQLpZz0Qgy9z3OvCEmac/sAD4Wci18AO3mXWmAU8Dfw6x61bgXfPz4cA2YLJZz0/N85pinrf1wK8AN3AR0ALc28HxXmwezxGAACOAA0P2DTavy6VAHTCo7f0TUtZDwFvmOcw0z8VfzH2nA1swWpPpwCyMe3KEuf9Z4E0zXy7G/X79fs7PY8D9IXX/Ang7wuc3l5DnpJNr9gJwp3luUoHj2jy7I/ZTzxSgBEMjemM856H1ngUcbF6PHwD1wARz34nApjbldfhcxFT/Yl1hiKDUApXmjf8YkBZyIU4OSfsU8NeQ717z4cgNSX96yP5bgI/Mzx8Bt4TsO8TM23rSFTC8o5up7UODITAL2hzLF8A1ao/Q/L6NLe92cA46O65SOhbdycCGNtt+C/y7g/S/BF43P18OLO4g3XTgw5DvhwINHaTtyrm4r01ZzRjidx3wOXBYm/wDgKbWeyHE3o9DrkXb4z4FWBPyfT5wtfn5ccwf85D932E8oCcAZYCE7PucjkX3PeAXXby/lwDntb1/zO+CIcoHh2w7Glhnfn4aU4DN7yPMe3KEee6agUND9v8M+GQ/52cysKH1OIGvgUvCeW7be066cM2eBf4JDG2nnM5Edw4wJeT7qbR5Ptukf6P1GtGO6O7vuYjln519YucrpT7sYN/GkM+DgUWtX5RStear0xAMYWqbfr2ZpzXv+jb7Wm+U9urqjLbltZY5JOT7lpDP9Rhi2lFZnR1XRxyI8epUGbLNCcwDEJFRwN8x+srTMY75GzPdMIzWeEe0tT9VRFxKKX879nd2LtpeFzfQF3jOtONFEcnBaM3daR6XGygXkdZ8jjbltL1eHwPpIjIZ2AqMx2h5YZb3UxG5LSS9x7RdAZuV+fSF2NgRHZ43Ebka+DWGGIFxzTvqVuqHcU2+CTlGwbh+mLZ9HZI+9Hj7Ypyftvd0R+ccpdRXIlIPnCgi5Rji/VYHx/EtxjkDOEMpNa+DYwils2v2G+AeYIGI7AIeVEo93YVywTgXbe+hUHvPAP6A8WbpwDivyzsqrJPnImbY3qfbAaEPQhl7boTW/po+GK96rQwL+XyAmWefvOY+P8bD2V5doZ/bo215rWVubidtZ3TluDpiI0bLKCfkL1Mpdaa5/3GM17KRSqks4HcYD3Zr3uFh2Ltf+03anou216UF2KGUalFK/VEpdShwDHA2cLVpWxPQN+S4spRSY0LK2esaKaUCwEsYravLgSKlVI25eyNG10PoeUpXSr2A0cUyREKUwrSxIzZivMruhYgcCDwJ/ByjayUHWMGe8932ntoBNABjQmzKVsYAFaZdQ0PSD2uTt4V97+nQc97ePVwAXIXxdvKKUqqxvQNUSo1RewbGuiK40Mk1U0ptUUrdqJQajNEqf0y6PmOhnH3vIcDoF8fosvobMMA87+/Q8XmH/T8XMSNeRTeUF4BrRWS8eaL/F/hKKVUakuZ/RKSXiAzD6LMqDMn7KxE5SIwpaf8LFLbTamtlOxCkY1F6BxglIleIiMvs/D8UKIrScXXEAqBGRO4QkTRzwGGsiBxh7s8EqoFaERkN3ByStwgYJCK/NAd0Ms1WYnfpyrm4SkQOFZF0jClwryilAiJykoj4zMGWagwhCSqlyoH3gQdFJEuMgdCDReQHndjyH4y+1CvNz608CUwRkclikCEiZ4lIJkZXiB+YKiJuEbkQOHI/dfwLuF1EJppljTAFNwPjAd8OxuAO5qCwyVZgaOuAjVIqaNr1kOwZ7BsiIqeZ6V/CuC/yzPN2V2tBIT8wfzav24EYLezO5lbPAi7AEN5nO0nbLTq7ZiJysYi0/ojswjhXQfP7VvbfAHgJ4/oMFZFewLSQfR6MvvntgN9s9YZO7dwK9BFz4Nxkf89F7Ih1f4b5NldKx/2V+/TzYHSorwEqMB7qoW3ST8UYSNoJPAg4zX0OjMGmjRgXZxbQS7Xpl2pT15/MtJUYne7XsHef3HEYryRV5v/QgYFPMAfx1J4+ts/2cx72d1wdniNz/2AM4d6CcTN/yZ7ByRMwftFrMboc/tTmGMZi9HfvMvNPM7dPxxzs2t856sa5+AvGD0Q1xmBRX3Pf5Rh9q3UYD8fD7BkcycZokWwyy10MXNbZ+cQYMK0APG22nw4sNK9nOfAykGnum2SWX4PxQ11IB326IdfrO/O8rgAON7f/2ax7B8br66fsGcz1ALNb95vbUjF+ZNea56YYmBpSz2/N61KGIQwKGGbu64VxH2/HuK/vBhxdOD8fmveUdHR83Xh+97ovOrlmf8Voiddi3Os3tTmf5ea12aefGeP1/yGM53odxiBpaL23mvdPJUaX1Yuh1w+jf3ynuX8wnTwXsfpr7VxPWEREYbwurLbbFo3GasSYdrYCSFEdv6F1pZyngTKl1D7z4TWxJRG6FzSaHoWIXGB2/fQC7seY4hWJ4OYCF2LMmNHYjBZdjSb++BnG/OI1QIAI+h5F5B6MlvIDSql11piniYSE717QaDSaREK3dDUaQIyl5MUi8nwUys4VkSusLleTmGjR1WgMbgF+pJS6srOE0n1HMrmAFl0NoEVXo0FEZmLMF/2viOSLyBsiskxEvhTTQY4YvoafE5H5wHNiONZ5VQxnPQtF5Fgz3Q/EcLCzREQWm3OC7wOON7f9yrYD1cQFuk9Xo8Hw8Ywxb/cPGPNp/ygiJwN/V0qNF5HpwDkYc5EbROQ/wGNKqc/E8CX7nlIqT0TexvA5Md9ckNOIMZ/5dqXU2XYcmya+0P5INZq9OQ74MYBSao6I9BGRLHPfW0qpBvPzKcChIauIs0yRnQ/83ewbfk0ptUkk5itNNXGMFl2NpuvUhXx2AEepff0Y3CciszHcY84PWd6r0QC6T1ejacs8DB8OiMiJGF0N1e2kex/Dby1m2vHm/4OVUsuVUvdjLD8ejbHMODOqVmsSBi26Gs3eTAcmisgyjAGwn3aQbiowyRxwW4nhRwDglyKywszfghHBYBkQEJGleiBNowfSNBqNJobolq5Go9HEEC26Go1GE0O06Go0Gk0M0VPGNInD9GwXRrSA1j83RqiYBqCR6VXNNlqn0XQJPZCmsZfp2Q4M3wSHYARNHIgROLT1rz9GQMY0On8zC2CsAKvDiKxQ3uavDMNd4vdMr6rpqBCNJppo0dXEjunZgzFCgk8C8jCE9mCMVmusKQdWYviaXY4RVuhbplcF95tLo4kQLbqa6GB0BRyJEe13Mka8uaH7zWM/1cBXwOcYgSu/ZHpVlb0maZINLboa65iePRQ4AyMY5A8xAhYmMgEMEZ4NzGZ61VKb7dEkAVp0NZExPXsScBFwNjDGZmuizWaM0PNvA+/pgTtNOGjR1XQbX4HvkJHNzee9tnnLTRh9sj2RCuAlYBbTq+bbbYwmcdCiq+kSvgJfFvATDF8ER6BUYM7GzRX9AsF+NpsWD6wF/gP8m+lVa+02RhPfaNHV7BdfgW8Ehjeta2njKeuKqpq5v63YdYIthsUnQYzuh/9jetWHdhujiU+06GraxVfg+yHwSwy/sO3Oj/UGg99+sX5Tsvfjhsu3wCPAc0yvqrfbGE38oEVXsxtfgc+B4Uv2N8DYruR5e2PZhly//4CoGpbYVAAPYbR+9YIMjfa9oDHwFfguxPD7+ixdFFyAmb2y10XNqOSgN3APsI7p2dOYnp1ht0Eae9Et3R6Or8B3KnAvcEQ4+T1Bteab9Rt76gyGcNgOPAA8yvSqhs4Sa5IPLbo9FF+BbzJGZIQTIy3r+bIt3x/W1DwqYqN6FhuA25le9bLdhmhiixbdHoavwNcLuB+4AbAkTO3x9Q2fPLZ1+4lWlNUD+RiYyvSqFXYbookNWnR7EL4C30+Av2F47rIMp1KbFpduHCIWiXgPJAA8DtzN9KpddhujiS5adHsAvgLfKIyH+uRo1fHYlm3Lj29o9EWr/B7CFuAmple9bbchmuihRTeJ8RX4nMDvgDuJsvvEcY1Nc2eVb9ULJazhWeAXTK+qtNsQjfVo0U1SfAW+AzCWph4bi/pEqZ2LSjdmu3Q0EqvYDNzA9Kp37TZEYy16nm4S4ivwXQwsJUaCC6BE+rybkb4kVvX1AIYA/2V69hNMz0612xiNdeiWbhLhK/ClAw8D19tR/4jm5vmvb94SM6HvQSwDLmZ61fd2G6KJHN3STRJ8Bb6xwDfYJLgAq93ucY0iesK/9RxWGhzwz9xps39styGayNEt3STAV+A7G6P/NrOztNHmjp27vriquuZou+1IJlqUc8PEpsezq/FmAX8B7iq97ywdyy1B0S3dBMdX4Ps18CZxILgAz2VlOu22IZlQisZLmu9uqMabjTEP+nfA27nTZufYa5kmXHRLN0HxFfjcwAzgRrtt2Qulmj/bsKkhO6gSPT5aXPB//gvmPeS/+Ph2dq0Gzi2976ziWNukiQzd0k1AzKW87xFvggsg4nk2K2u53WYkA98GD/ysA8EFGAHMzZ02e3wMTdJYgBbdBMNX4BuMESL8JLtt6YiXs7zafWGE1CvPdz9unj6xk2R9gTm502ZPioVNGmvQoptAmAse5gKj7bZlf+xyOMZtdTq32m1HoqIUVec0/zm1kZS0LiTvBXyUO222HrxMELToJgi+At9wDMGNf9+1Io4nc7JK7DYjUbnTf13xGjXkwG5kyQLez502Wy/DTgB6rOiKyDMiclEY+QaLyCvRsKkjTIc1nwLdeRBt5W1vho4SHAZzA75P/xM45agwsnqB/+ZOm32K1TZprKXHim64KKXKlFJdFmsRicgXga/AdyiG4A6NpJxYU+9wHLrW7Vpvtx2JRKXKWHpty28iWdGXjjGd7EyrbNJYT48QXRG5S0S+E5HPROQFEbm9zf67RWShiKwQkX+KiJjbR4jIhyKyVEQWicjBIpIrIivM/U4RecDMu0xEfmZuP1FE5onIW8DKcO02uxTmAAPDPngbeTwnu9RuGxKFgJJtpzfdPzCAM1KHQanA67nTZp9vgVmaKJD0oisiRwA/BsYBZwDtjfQ+qpQ6Qik1FkgDzja3Pw/MUEqNA44Bytvkux6oUkodgRFj7EYROcjcNwH4hVIqrDA2vgJfX+BdYEA4+eOBORnpw+y2IRFQisBNLb8u20Jvq661B3g5d9rsCywqT2MhSS+6GJ623lRKNSqlaoD2HESfJCJfichyDEffY0QkExiilHodwMxf3ybfqcDVIrIE+AroA4w09y1QSoUVKdd0XFMUUlZC0iwyfHGKRw+odcLLgRPmfRScON7iYl3ALD2PN/7oCaK7X0QkFXgMuEgp5QOexHhF61J24Dal1Hjz7yCl1Pvmvrpw7DEdjxcCk8PJH2/MzMnWU8f2w2bVZ8Fv/D/7QZSKTwfezJ0229LwTJrI6AmiOx84R0RSRcTLnq6DVloFdoe5/yIAs1W8SUTOBxCRFBFJb5P3PeBmEXGbaUaJSKQLAx5vx8aE5cu01FEK9FrzdmhRzg1nNP3lEJBoxpY7AHg1d9psdxTr0HSDpBddpdRC4C0Mn6T/BZYDVSH7KzFatyswRHRhSPafAFNFZBnGKrC2A1r/whgoW2QOrj1BBJETfAW+3xGPS3sjICgy6NO0tGWRlnPdmw30f6CGsY/V7t42/ZNGhvy9hvEzaxk/s5Z3VrW0m/ehL5oY81gtYx+r5fJX62n0G78BV75Wz2GP1/K7jxp3p713bhNvlLRfjpUoReOlzXfVm45sos1xGG9zmjigRzi8ERGvUqrWbKnOBW5SSi2y265QfAW+MzD6cZPuh3BsY9O8F8q3duRDoEvMXe/H6xGufr2BFbd4AUN0vR7h9mM6Dv+2uTrIcf+uY+UtXtLcwiUv13PmSBcTBjl5+Ktm/nVuGj96ro5XLk6nvkVxU1Ejb1/e9oXGeh7xnz/vQf8lEZ2TMLit9L6zHo1xnZo2JN0D3gH/NAe7FgGvxqHgDseYKZGU1+PbFM/YFoio+XjCgS56p4X3Fu4PQoMf/EFFfQsMznTgdkCDXxFUipYAOB1w98dN/PHEqMbvBGBl8IDPbBBcgIdyp82OWkRoTddIyoe8LUqpK8yBrtFKqb/YbU8ovgJfCvAKxhr6pESJ9JrtzVgSjbIfXdDMYY/Xct2bDexq2PetbUiWg9uP9nDAQzUMerCW7FQ49WAXef2c9Et3MOGJOs4Z5WJ1RZCgggmDousOuF55vruw+Y+dObKJFi6MqWTDbapfQw8R3TjnH8DhdhsRbZ7OzrK8o/TmSR7WTPWyZEoGg7xC/vuN+6TZ1aB48zs/637hpezXXuqaYdayZgD+cXoqS6Z4yT8mhbs+buKek1P489wmLnm5nie/abba3O46sokWvYG3cqfNjgun9z0RLbo24ivwXQpMsduOWLDO7RpXLxLWNLqOGOB14HQIDhFunOhhwebAPmk+XOvnoBwH/TIcuJ3ChXkuPt+4d7o3S1qYOMhBbbNiza4gL12czivFLdS3WDveEYYjm2gxBphptxE9FS26NuEr8A3FmO3QMxDJeCnTu9TKIstr9oQJe724hbH9972dD8gWvtwcoL5FoZTio3UB8vru6UJoCSj+8VUzvzk2hYYWY+I1QCAIzftqeNhE4MgmWlyhfTTYgxZd+3gC6FEhbZ7Pzgx7rujlr9Zz9FN1fLczyNC/1/DUomZ+82ETvsdrOezxWj4uDfDQacaU67KaIGc+bywenDzUxUV5LiY8UYfv8TqCCm6auMeMGQub+ek4N+lu4bABDur9Ct/jtUwc5CQn1ZrpsxY4sokWM3U3Q+zpEVPG4g1fge9qoMBuO2KOUi3zNmyuzQkGk3bQsC0BJduPbXokaKFfBauZUXrfWT+324iehG7pxhhfgW8gxuBZz0PE/Ux25gq7zYgVShGY0vKrzXEsuAC35E6bHY+t8KRFi27seYwknh7WGa9kenvM6+yrwRPmfRCcNN5uOzpBgH/lTpsd/QnKGkCLbkzxFfguAXq0u70qh+OwMpezrYvMpGOz6rPg9paoObKxmtHA78PNHOpjuovpzxWRaebn6W39W7ctU0QmicjD4doXb2jRjRG+Ap+XntqtEIqI45/Z2d/bbUY0iZEjG6u5I3fabF8sKlJKvaWUuq8b6b9WSk2Npk2xRItu7PgNMMhuI+KBd7zp8dzHGRExdmRjJW7gqdxps8NdkucSkedFpFhEXhGRdBEpFZG+sLu1+on5+RoR2ccHhIhMNKO0LAVuDdl+oogUmZ+ni8jTIvKJiKwVkakh6dqNECMiU0VkpRnd5cUwj88ytOjGAF+BbwiQb7cd8UKDwzH6e7c7LAfv8c6MwHkLF6lRo+22I0yOAG4LM+8hwGNKqTygGrgljDL+jeGfelwn6UYDpwFHAn8QEXcnEWKmAYcrpQ4jDhYjadGNDX/GcCitMZnZK3uD3TZYTXHwgM/+5r/UDkc2VnJX7rTZWWHk26iUmm9+noXhTrLLiEgOkKOUmmtuem4/yWcrpZqUUjuAbRghrfYXIWYZ8LyIXAX4u2NXNNCiG2V8Bb7DgavttiPe+CQ9LR6Ww1pGg/J8f4F9jmyspDfwqzDytZ3wrzAErlVjuhqNpSs0hXwO0LkP67OAGRhxCxdGGqE7UrToRp8H2bO6VGPSIpK7MDUl7EjJ8YRSVJ/bfK/HZkc2VvKr3Gmze3czzwEicrT5+QrgM6AUaP0h+vH+MpvBBCpFpLWFfGU36283QoyIOIBhSqmPgTswVoF6u1m2pWjRjSK+At+pwEl22xGvPJGTvd1uG6zgTv91K1epobl222Eh2cD/dDPPd8CtIlKMMQ/9ceCPwP+JyNcYLdLOuBaYYfq+7lZDZT8RYpzALDPo7GLgYVPgbUMvA44ivgLfx8CJdtsRrziU2rK4dGN/RwL/+M8L+D79SctvE2U+bneoA4aX3nfWNrsN6SqJECEGEvhmj3d8Bb7JaMHdL0GRgXPSI4+fZheVKmPpNfHpyMYKMoBf2G1EN4nrCDGtaNGNHnfYbUAi8GROVm3nqeKPgJLtpzfdPzCA09ZBmShzSyJ5IYvnCDGhaNGNAr4C32jgfLvtSARWejy+ZrA+TEMUUYrAzS2/3BTnjmysIIc4mNeabGjRjQ6/IcYzFjY9tYni24pZdeeq3dsa1jew5k9rWH3XalZPX0392voO8wcaApT8qoSy58oACLYEKf1bKavuXMXOj3buTrf535tpKG2wznCR7LejFD8tWrwWPH7e+8Ejkj7EksmvtDMca9GiazG+At9guj/dJWJ6HdeL3PzcvbZteWkL/c/vz4h7RjDgggFsKdzSYf5tr20j45CM3d9rV9SSPiqdEfeMoPLzSgAaNjSggoq0XGtnRv07J8vCGA3RZbPqsyC/ZUoyDpx1xCDgJ3YbkUxo0bWeawBPrCvNOCQDZ8bey+ZFhGCDEdIm0BDA3av9wA0NpQ34q/x4x+yZvihOIdgcRAXU7mnv217bxoALrX+jXu9yja8Tifu+3Rbl3JiAjmys4Hq7DUgmtOhaz7V2G9DKwCsGsqVwCyW/LmHLi1sYcNG+gqmCivIXyhl42cC9tnvHeGnZ0cLae9bS50d9qF5cTeqBqR0Kd0SIpL2YlWlp/DSrUYqmy5t/X5uAjmys4KjcabMPttuIZEGLroX4CnwnACPstqOVijkVDLx8IKP/PppBVwxi89Ob202TOS4Td++9xVScwrApwxjxpxFkH5HNzvd30vf0vpS/UM6GRzdQvbjaUlv/k+WN637DGYHzFnytDsmz2w4bucJuA5IFLbrWcp3dBoRSOb+SrEmG75KsI7JoWLvvAFj96noqPqzgu/zv2FK4hcr5lWx5ae++351zdpJzTA4NaxpwpjkZdsswdry7w1Jbtzmd4yscjp2dp4w9SeLIJlJiPk6RrGjRtQjTSflFdtsRijvHTV1JHQB1xXV4Buzb1TxsyjAO+fshHPLgIQy8dCA5x+Yw8JI9XQ2BugA1S2vIOTaHYHNw95wM1WzxSkYR19M5Wd9aW2jkJJEjm0g5JHfa7EmdJ9N0RjJP7I41l2Ks4rGFjY9vpK6kDn+tn5JfldD//P4MvnYw5c+XQxDELQy5dggADesaqPi4giHXDem03G1vbqPf2f0Qh+Ad62XnRzup+n0VvU/qrj+Uznndm5Fze0Wl5eWGSxI6somUq4Cv7TYi0dG+FyxC+1mwAKXUfzeVlQ31Bzr/NYgBd7Vc8+VzgVOPstuOOGIrMKT0vrMSZopfPKK7FyzAV+DrRTedNmvaQUSeyMlebbcZAJ8Fxn4aruDueOcfbHzkSsqe2jd4QvWC11h//9kE6qvazeuv3sbWwrvY/OQUyv51M/6qrQBsf/sByp7+Obs+LdidtvLzF6n//otwTAyXAcAPY1lhMqJF1xrOQHfVWMK7Gem2x5GrVBnLftpyR9iObLy+U+h/8R/32e6v3k7DusU4s/p1mHdH0d/JOvJChtw4k4FX/x1HejbN29bhcKUw+LpHaS5fRbCpDn9tBc1l35E+6ugOy4oSV8W6wmRDi641nGO3AclCo8MxqtjjXmNX/QEl289ouq9/JI5sUoeNxZm2r5+YXR89Sa+TrqWjFeLNOzZAMEjaQcYKY4cnDYc7FXG4CPqbUCqICvpBHFTNm0X2cbZMKLggd9psHXoqArToRoivwOcGTrfbjmTi8ZzsTXbU2+rIppw+AztP3T3qV32JM7MPnv7DO0zjr9iMIzWDba//mbJ/T2XXx0+jggHcfYfhTMum/JlfkD7iSPy7ylFKkTLQlinhXuBUOypOFvQrceQcj+GNSWMR89LTDrKjXtORzYlWlxtsaaTqi5cYcOk9+02nggEaN37LoGsfxpXVjx1v3k/t8o/IHHcqvU+5aXe6ba/8kd6n/Zyqzwtp3raO1NzxZI6P6e/+8cAbsawwmdAt3cjRXQsW4xc54MvUlJjO2S1TvaPmyMZfuQV/1VbKnr6NTY9fR6BmB+XP/JJA7a690rky++IZMBx3zkDE4SRt5FE0b927p6V+1Zd4Bo5AtTTSUllOv/OnUf/dfIItjdEwvSP0oHEEaNGNnFPsNiAZeSIn29olb/vBcGRz36hoObLx9Mtl2G3PM/Tmpxl689M4M/sy6Jp/4PT22jvdoJEEG2t3z2xoXL8MT99hu/ergJ/qr98ka/KPUf4m9qxUCUIgppHFD8+dNlvPXQ4TLboR4CvwZQOH2m1HMrIoNSUv0LVghhGhFE1XNN9ZW4U3x6oyt7/1V7Y8dzstFZvZNOOn1Cx9v8O0TeWr2PnfhwEQh5NeJ13P1hfvpOypWwGFd9xpu9PWLJqNd+wPcbhTcfc7COVvouypW/EMHIEjNaYBbt3A5FhWmEzoxRERYEb7fc9uO5KVv23bsei0uvoJ0axjhv/ceQ/4L+vpfhXC4a7S+866124jEhHd0o2MmE+S7Ek8mZ3VcagLCygJDvtMC27YJGtAzqijRTcyjrHbgGTmO4/b1wxN0Si7QXlWnd/8J+3IJnyOzp02W+tHGOiTFia+Ap+g+7Wii0j265neJVYXazqycWtHNhGRDfjsNiIR0aIbPodi3HiaKFKQnWn5oMPd/mtWrlJDc60utweiuxjCQItu+Iy324CewEaXa3yNiGVhKuYHxoTtyEazD3q+bhho0Q2fUXYb0CMQSf1PduYyK4qqUunLrm6Zpltn1tFTwtBbihbd8NGiGyNezMyMuO81qGT76U33R+TIRrMPB9htQCKiRTd8tOjGiB1Ox/gdTsf2cPMbjmx+ERVHNj2c9Nxps/vYbUSioUU3fLToxgoR51PZWcXhZn89eNy894JH6lfh6DCs8ySaULTohoGvwDcYw8WdJka84W3jqKCLlKneC37dcnNUHNloAC263UaLbnjoVm6MqXU6fOtdro3dyRNtRzYaQPfrdhstuuGhbzQbeCIne21X00bDkY2mXXRLt5to0Q2PAXYb0BN5PyOty1GCHw+cu2ChGp0XTXs0gG6AdBstuuHR324DeiJNDseIFR7Pqs7SfRccOv+v2pFNrNAt3W6i5yyGwSVzA/5Gj8zfmUlqRaak7fKSWZVBTkOK7BuNUGMpj/fKLpuxdfvIjvY3Kveq85vv0TMVYodu6XYT7U83DIpH570NnN12u4IWJezyO6luclNbn0JjTRotlRkS2OVFKjLFtTML985M0nZ5JbMyg+zaNHIQPdDTVZxKbVpcunGItBNSVymqT2u+f+f3apgtMdZ6KH4gpfS+s4J2G5Io6JZueOS0t1HALYr+Hj/9PX7IbIABlQCtP2z7/sApCCrYFXBS1eyitj6Fhto0mqvTJbDLCzszcezMFHdFFukVXsmo9JJVnU6voEN65LULiAydn5a6/LiGxn08XN3tv2bl92qY9qsQW1xAX2Cb3YYkCj3ywbUAy7yLCTgEejsC9HYHIKMJ+lXD3gLdrlhXBRyGUDd6qK9LpakqXQKVXoIVplDvzCK1wivplV6yqzLIaXFJqlV228k/c7J2HdewdyBG05GNno9rDx67DUgktOiGh+1+WAWyXUGyXc2Q3gy9a2FY50JdHxQqW1xUN7qpr0ulsSYdf2WGqIpM2Gl0f6RWZEr6rgwyq41+6rhbBLIkJSXPD36Xef9qRza247bbgERCi254JOSsD4F0pyLd2cLg1BbIqQcqoJPuj2Yl7Gox+qnrdvdTe/fqp/ZUeEnblWn2U6eSHc1+aiXS772M9G/OqqufqB3ZxAVadLuBvlHDo8cMfAl4RDEgxc+AFD9kNcDASuhEqAMKKgNOqppc1DYYQt1UnS6BXZmI2U/tqcgkrSJTMiozyKrpZj/1UzlZjWfW1gdNRzZ6toK9aNHtBlp0wyMhW7qxQsAp0McRoI87AN4u9FMrY0NVwEFls4vaBg/1dcaAon+XF2X2U3t2ZpJSkSkZVemuQS+0HPPhx/5xR3toqonh4WnaEMThtNuGREKLbnho0bUYcwrYXv3UfWqhI6HelT1y5Vcjhnl+5n9az422Hyecb7cNCYMWj/DoMd0L8YhCgst8U8TtPXsCELafXY1lBOw2IJHQohseWnRtZPXB538WcKXmiaRkOVMnl9htj0aLbnfQohsedXYb0FNpcmdu3zj0h+Nav7tSjzkWXN/ZaZOGJrsNSCS06IZHpd0G9FSWjLvte0R2L04REYc74+zG/eXRRJ1ddhuQSGjRDY9Kuw3oiezoM3ZpnXfIPosgnJ7h48SR/YUdNmlQ6OehW2jRDY9Kuw3oaQTF4V9x6PUZHe13ey8cBugWb+ypyS8s0n263UCLbnhU2m1AT+O7kZfNDzo9Izra73D2GupwD/8yljZpAN210G206IZHpd0G9CQaUnqXlw86ZlJn6dwZZ04C2RoLmzS70aLbTbTohscOuw3oSSweP3U9Ih12LbQi4vG6Uo/uNLKExlIq7DYg0dCiGx7r7Tagp7BlwBFfN6b167KPXGfq5GPBvTKaNmn2Qj8L3USLbnh0OSqtJnwCDldj8SFX9etOHhERt/dcPbATO/Sz0E206IaHvtFiQPHon36lHK4Du5vP6T7QJ47e86Nhk2Yf1tltQKKhRTcM8kqKK4Aqu+1IZurSB6zf1u/wyeHm93gvHA7UW2iSpn10A6SbaNENH/0LH0UWj5u6HQk/vJA4swY53KMWWGmTpl206HYTLbrho2+2KLFp8AlfNqfkdDpFrDPcGadNBimzwiZNu9TnFxbpKXrdRItu+OgR8ijgd6bUfj/yom7347aHiDvNlXZ8qRVladrlW7sNSES06IbPN3YbkIysGHPDN4hzkFXluVInHYOkLLeqPM1e6GcgDLTohs/XdhuQbNR4h62u6JVneVRfT8Z5TtoL5qaJFC26YaBFN0zySoo3AVvstiOZWDzutjqk68Epu4rDPfRQcfbVU8isR4tuGGjRjQx901lE6QGnzfe7M8Z1njI8PN4LRgK10Sq/B9IErLDbiEREi25kaNG1gBZXWtXag84eFc06xJE5wOHJ011C1rEsv7CoxW4jEhEdDTgyYj4P9M7ycj6tq6W308lbBw0H4Ndlm1nX3AxATSBAptPJ67kH7ZP3lDWryXA4cQi4EF7OzQXgwe3bmFdbx+jUFO4bNBiAt6qqqAwEuLp376gf01LfLUsRxwnRrsed/qOjmpq/2wjBYdGuqwfwld0GJCpadCNjHuAnhufxguxsruzVi2nle6af/n3wkN2f79+2lUyHs8P8zwwbRi/XHnNrAgFWNjbyxkEHcdeWcr5vauQAt4fXq6v459Doa9Ou7BErq7MOOi7qFQEirlRX2omb/Q1ztOhGzod2G5Co6O6FCMgrKa4GYuo4e1J6OtnO9i+bUor3amo4Myury+U5BPxKoZSiMahwIfy7ooIrc3rhlugGPTZCqd8MIjG7D12p449CUpfEqr4kxQ98bLcRiYpu6UbO+0BMWmqd8U1DA32cLnI9nnb3iwg3bNqIAJfk9OKSnBwyHE5OyPBy4fpSjkpPJ9PpZFljAzf37Rt1e9cMP29+wJV6fNQraoPHe35ac82LQSxudBQuWMrK8m14Uzz8z+k/2L39s1XrmL96PQ4R8gb15+xxeXvl21Zdy6wvF+/+vrO2ntPGjuKEUQdRtLSY77ZsZ3BOFpdPHg/AN+s3UdfUwgmj9u1CihEL8guLqu2qPNHRohs57wN/stsIgNnV1ZyZldnh/lnDDmCA281Ov58bNm1kuMfDpPR0ru/Th+v79AHgri3l3Na3H69UVjK/vo5DUlKY0sd6AW5yZ27fMOyUwywvuAs4XIMPEeeAeSqw1VLBn3TQUI4dmcsLXy3ZvW31th18u3kr+acej8vppKZx32jl/bO8/PpUw5RgUHFP0UeMHTKAhuYWNldWk3/aCby0cBnlldX09WawcN0mbjzhSCtN7y66ayECdPdC5CwkDkKW+JXiw9oazsjsuGthgNsNQB+Xix96vSxrbNhr/8rGRhSQ6/HwXk0NDw0ewobmFkrNQToraRtKPdZ4vOePBixtrR3crw/pHvde2z5fvYGT8kbgchr97JmpKfstY9W2HfTJSKd3RjoiQiAYRClFSyCA0+Hgk+/WctzIXJwOWx/dD+ysPNHRohsheSXFQeAju+34or6OgzweBrrd7e6vDwapCwZ2f/68rp6RKXsLwCM7tjO1b1/8ShEwF3A5BBqDQUtt3dF7TLuh1GOJODL6OT2+xZ2njIwdtXWs217B/304n8c+/oINFZX7Tb9kQxnjDzBmkKS6XeQN6s9DH3xGZmoKqW4XGyoqGTtkYLTN3h9VxHgcI9nQomsNb8WqotvLNnP5+vWUNjdz0prVvFpZCcB/q6v3GUDb5m/hZ5s2ArDT7+eqDRu4oHQdl64v5QRvBsdneHen/bCmhjGpqfR3uclyOhmdksp569bRFFSMTg3bw+I+BMXhXzHmhnTLCowAV/rJR4MjquFmAsEg9c3NTP3hMZx9WB7PfbEIpdpfkewPBPm2bCvjhu1xPXHS6IP59anHc+74Q3l3xfecNnYUX63dwLOfL+LDlbaEg3sjv7DIb0fFyYLu07WGN4AGIC3aFf0tZHpYKP9rzq8Npb/LzRPmtK9hHk+7c3dbOSUzk1My9/QH/6Z//wgtbZ/vR146P+j0/KDzlNFHxOlxpf9wq7/+A0u8mrVHTnoavqEDEREO6JODA6GuqRlvO90MJVu2MbRXdrtdEJt3VQGKfpkZvLOshJt+MJkXFyxle00d/TI7jdlpJS/GsrJkRLd0LSCvpLgGKLLbjninIaV3edmgYyfabUcorhTfkUj6omiVP2bwAFZv2wnA9ppa/MEgGSntzy4J7Vpoi9HKPYRgUO1uKTsEWgIxDQe3Ez2IFjFadK3jP3YbEO8sMUKpeztPGVs83gsygYjVa9YXi3nko8/ZXlPHPW9/xFdrN3DkQcOoqK3ngXc/ZdYXi7nsyHGICFUNjfxr7p4FjU1+P99v3YGvnf7aFZu3MLRXNtlpqaR53AzOyeJv782lJRBkcE7X52RbwGu6ayFypKP+JU33KB6dl4LhdSzHZlPikq39J3797aHXRRwNIlo0Vb84VwXKor4UOcE5Jb+wyPZB40RHt3QtIq+kuAl4zW474pGguJpWjr66W6HUY43He+5YoNJuO+KYLcAndhuRDGjRtZbn7DYgHlmZd/WX4YRSjyXiSO/tTBm/1G474ph/5RcWxbQDOVnRomsheSXFn6Bjp+1FXdqA9dv6TQg7lHoscaWdeCw419htRxwSAJ6w24hkQYuu9TxitwHxxOLxkYVSjyUiDpc7/VTbVxfGIW/nFxZtstuIZEGLrvU8i+4bBGDT4OO/siKUeixxpuRNQrwL7bYjznjMbgOSCS26FpNXUlwPPGW3HXYTcHjqvh95cUL6rfVkXtAX0FERDL5Hz821FC260WEGYK3DggRj+Zgbvkac7c/0j3Mczn4HOVzDPrfbjjhhRn5hkZ5XaiFadKNAXknxOuBtu+2wixrv0DUVvQ89xm47IsGdcc54jBVYPZktwJN2G5FsaNGNHn+22wC7WDLuthpE2nd3liCIIzXbmTLpW7vtsJm/5RcWNXSeTNMdtOhGibyS4oXAO3bbEWtKDzh1fovbO95uO6zAlXbcseCyxZVXHLANeNxuI5IRLbrRZbrdBsQSI5T6OVENpR5LRBxOd8YZtXbbYRMP5hcW1dttRDLS40RXRK4RkU4HeETkTyJySiR1ma3dNyIpI5FY5rt5KeKI6+W+3cXpGXm4OLJ6WrjxHRiDwZoo0ONEF7gG6FR0lVJ3K6WsmCrze3rATIbK7IOLq7KGx0WATqtxey8cCFgfsyh+uT+/sKjObiOSFdtFV0RyRaRERJ4Rke9F5HkROUVE5ovIKhE5UkR6i8gbIrJMRL4UkcPMvNNF5PaQslaY5eWKSLGIPCki34rI+yKSJiIXAZOA50VkibntbhFZaOb9p4gRd9y05yLzc6mI/FFEFonIchEZ3dXjyysp/pYk98mgkOBS383BWIZSjyUOZ+8DHa7cnjKFbC3wsN1GJDPx8pCMAB4ERpt/V2CENb8d+B3wR2CxUuow8/uzXShzJDBDKTUGY4XYj5VSrwBfA1cqpcYrpRqAR5VSRyilxmJEfji7g/J2KKUmYAwu3N5Bmo74LRYHQYwn1gw/d37AlTbGbjuiidt71kSQ7XbbEQPuyC8s6kmt+pgTL6K7Tim1XCkVBL4FPlKGo9/lQC6GAD8HoJSaA/QRkc68N69TSi0xP39jltMeJ4nIVyKyHDgZ6Eg8Wt027q+sdskrKS4H/tCdPIlCsztzx4ZhP/LZbUe0EUnJdKZOLrHbjijzSX5h0St2G5HsxIvoNoV8DoZ8D7L/OG5+9j6GUMcqoWUG2itHDEcsjwEXKaV8GBPBO3LO0lpeu2V1gUeAZWHki2uWjPt5CSI5dtsRC1ypRx8L7mQVXj9wW7iZReSXImJZwFGry4sn4kV0O2MecCWAiJyI8apfDZQCE8ztE4COIy/uoQZojcDYKrA7xAgjc5FlFrchr6Q4ANwKJM2Syp29D11Wm2FvKPVYIiIOt/esps5TJiSP5hcWrYgg/y+BdkVSRJxWlpfoJIroTgcmisgy4D7gp+b2V4HeIvIt8HMM5xyd8QwwU0SWYLRenwRWAO8BUfUulVdS/BlJMqgWFId/+Zgb0zAHHnsKTvfwceLI+cJuOyxmHXBXVxOLSIaIzBaRpeYA9B8wZgR9LCIfm2lqReRBEVkKHC0iV4nIAnMA+4lWIRaRU0XkC3OQ+mUR8YrI1LblJRM6RlqMKR6d1x/D0Xkfu22JhJJRl31aNvj4uAilHmuCgcpNzdVP96XjrqhEQgEn5xcWfdLVDCLyY+B0pdSN5vdsYCkwSSm1w9ymgEuVUi+JSB7wV+BCpVSLiDwGfImxYvM14AylVJ2I3AGkKKX+JCKloeUlE+H0TWoiIK+keFvx6LyfAQk7YNGY0qu8bNBxloRSn/XJA6xY/yWZaTnceYnhEfPpD+5ha9VGABqaaklL8fLbi/65T96VGxbwyuczCKogx4w+k1MPvxyAZz76X8oq1jL2gKM4d/INALy7aBaDeuUy7qDIpxI7nDlDHe6DPwm2rDkx4sLs57HuCK7JcuBBEbkfKFJKzWvnhSeA8SYK8ENgIrDQTJeGscz4KOBQYL653QMk21vEPmjRtYG8kuJXi0fnPYOxUCPhWDxuaikiR1tR1lGjTuMHY87j2Y/v373tuh/tedN97YvHSfNk7JMvGAzw0vyH+flZfyUnox8PvHYLvtyjCQYDuF0efnfxv3ik6H9oaKql2d9E6dZiTp9wlRUmA+DOOOOIpsoZW0DtGzM9cVgL3NHdTEqp780xlDOBe0WkvQjBjUqp1phqAhQopX4bmkBEzgE+UEpd3l0bEplE6dNNRqZi3PQJxdZ+E79pSO9vieACjBh8GOmp7c/+U0qxaM2nTBxx8j77SreV0DdrCH2zBuNyupkw4iSWlX6O0+Gixd9MUAUJBAM4HE5mf/0MZ036aTs1hI+IJ8OVekwix1NTwLXhrDwzl9HXK6VmAQ9gDGaHDlC35SPgIhHpb+bvLSIHYnQxHCsiI8ztGSLS6rtjf+UlNFp0bSKvpLgG+AnGa1hCEBRX08q8q2PWF72mfDmZab3onz10n31V9Tvo5d3j5qFXRj+q6nYwsNeBeFOzuf/VKfgOPIrtVZtRKsiwftb74XGmHnkMeBLV/eMD+YVFc8PM6wMWmIPRfwDuBf4JvNvewJdSaiXGcvj3zcHwD4BBSqntGG97L5jbv8BYHMX+ykt0dPeCjeSVFH9ePDrvPuBOu23pCitH/+RL5XDFbPDs6zVzmDTipG7nu+jYW3d/nvnfO7nshF/x7qLn2bxzDaOHTuTYvLMssU9ExO09R7XUvqowXqEThU8xVnaGhVLqPYzZPqF8TUhQVqWUt02eQqCwnbLmAEe0s/0RkjTIq27p2s90YI7dRnRGfVr/Ddv6T4xZKPVAMMDSdfOYcHD7opud3pddtXtW5e6q2052Rt+90iwrnc+wfqNoamlgR3UZ1//obhavnUtzS6NldjrdB44VR+9E8suwBbgsv7AoYd6wkg0tujaTV1LsBy7BWOgRtyweN3VrLEOpf7fpGwbkHLBXF0IoB/YfzfaqzeyoLscfaGHR6o857MA9EYICAT8fL3+VH427lJZA8+5mqFJB/EG/pbZ6vBcOBxLB96wfuDS/sGiL3Yb0ZLToxgF5JcU7gfOBuHSnt3nQcV81pfba5xXQCv794b08+MZtbK3ayO9nXcrnJUawjW/WfLzPAFpl3Q4ee8cYAHc6nFxy3G3MeOcO7n3pWg4/+EQG9c7dnXbut28yedSpeNypDOk9nGZ/E39++QaG9R1Jespeb74RI86sQQ73qAWWFhodfhdBP67GIvTiiDiieHTexcBLdtsRSsDhqZt73N8qlcM5xG5b4hmlWhqaKh+tABWv56kwv7DoMruN0OiWblyRV1L8MvC/dtsRyoox1y/Ugts5Iu40V9oJ6+22owPmsmfpvMZmtOjGH3cBL9ttBEBNxpA1O3uP6TEObSLFlTrxGCQl3jzJFQPn5xcWJaujnoRDi26ckVdSHASuAt6325Yl46YmfCj1WOPJOM9F/HiSKwfOyC8s2mW3IZo9aNGNQ/JKipuBC7FxHfr6YT/6vMWTHKHUY4nDPfRQcfabb7cdQC1wVn5hUbx2efRYtOjGKXklxXXAWRjORWJKizOtas3wc0fGut5kweO9YBTGMla7qAfOzS8sWmyjDZoO0KIbx+SVFO8CTgViusZ/mW/KkmQLpR5LxOHt7/Ac+o1N1dcDZ+cXFiXd8tlkQYtunJNXUrwFwzXe6ljUV5k1vLgq++CkDKUeS9zppxwNjg0xrrYeo0tBC24co0U3AcgrKV4PHE+UuxoUElx62C1BwguvoglBxJXiSj+pLIZV1gFnhuEbVxNjtOgmCGaL9wcY7vCiwtqDzkn6UOqxxJUy7igkdUkMqqrBaOF+GoO6NBGiRTeBMPt4T8HwT2opzW7vzvUHnDrW6nJ7Oh7vBWkYUa2jxWbgeC24iYMW3QQjZFbD61aWu+Swnxcj0svKMjXgcA06RJwDozWFbBlwVH5h0dIola+JAlp0E5C8kuImjHDxf7GivJ298pbVeofqlWdRwuM9Lw+osrjYDzBauJssLlcTZbToJih5JcXBvJLi3wGXAw3hlhMUh3/52Jt6XCj1WCKOjL5Oj2+JhUU+g9GHW21hmZoYob2MJQHFo/MmAG8Aw7qbt2TkZZ+WDemZodRjiVKBlqbKRzZDMDeCYpqBX+cXFs2wyCyNDeiWbhKQV1K8CJgEdKvvsDElZ0vZ4OMmRMcqTSgiTrcr/ZRtERSxHjhOC27io0U3ScgrKd4GnITRz9ul0fLF46auQyQpI67GI66UsUci6eGsVHsHmJBfWLTQaps0sUeLbhKRV1LcYvbzngTsdzXU1n4TvmlIH2BZKHVN1/B4L8zBCJvTFfwYUXTPzi8sqoiaUZqYovt0k5Ti0XnZwGPAFW33BcXV9OnxD5Yrhys35oZpaKop/FT5N3fWj74cuDa/sMguHw6aKKFbuklKXklxVV5J8ZXAlbSZrlQ8+qovtODahyfj3MOAjnzc+oF7gUlacJMT3dLtARSPzhsMPAz8uD6t38Yvj/xDX0TS7LarJ9NS//GngabFbVu7y4Fr8guLFtlhkyY26JZuDyCvpLgsr6T4IuCcpb5blmvBtR9X2g+OBWery84G4G6M1q0W3CRHt3R7GDOmzEkD7jD/Um02p0cTaC75uqXunVIgP7+wKNZuIDU2oUW3hzJjypyDgD8DlwF6NVrs+RrIv3XmyXPtNkQTW7To9nBmTJnjA+4BzrPblh7CGmA68PytM0/WD18PRIuuBoAZU+YciTFq/iO7bUlSSjDeLF64debJAbuN0diHFl3NXsyYMudEjP7e09DdDlawHOPH7JVbZ54cTb+6mgRBi66mXWZMmZMH/AL4CZBuszmJyCcY0/Te0N0ImlC06Gr2y4wpc3oDNwG3AkNtNifeqQQKgJm3zjy5xGZbNHGKFl1Nl5gxZY4LIxz8TzAG3fRc3z0sBB4HXrx15slh+zbW9Ay06Gq6zYwpczKBCzEE+CR65iKbpcBLwMu3zjx5ld3GaBIHLbqaiJgxZc5g4HzgDOBkkrv/VwutJmK06GosY8aUOSkYYeLPxBDhUfZaFDGbMSIvfwTMuXXmyToemSZitOhqosaMKXOGAUcBk83/E4jfvmAFrAIWA58CH9068+Tv7TVJk4xo0dXEDHMwbhxwJJCH0RIeCRwIOGNoShWwGliCIbJLgKW3zjy5NoY2aHooWnQ1tjNjyhwPcDB7BLgv0K/N/74YrWSP+Rcq0n6gup2/KqAMI75Yqfl//a0zT7Y6HLpG02W06Go0Gk0M6YlTfTQajcY2tOhqNBpNDNGiq9FoNDFEi65Go9HEEC26Go1GE0O06Go0Gk0M0aKriTkikiEis0VkqYisEJFLRWSiiHwqIt+IyHsiMshMe6OILDTTvioiyezbQdMD0PN0NTFHRH4MnK6UutH8ng38FzhPKbVdRC4FTlNKXScifZRSO8109wJblVKP2Ga8RhMhLrsN0PRIlgMPisj9QBGwCxgLfCAiYKw2KzfTjjXFNgfwAu/F3FqNxkJ0S1djCyLSG8Mb2Y3AHIyW79HtpFsHnK+UWioi1wAnKqWuiaWtGo2V6D5dTcwRkcFAvVJqFvAAhheyfiJytLnfLSJjzOSZQLmIuIErbTFYo7EQ3b2gsQMf8ICIBIEW4GYMpzUPm/27LuAfwLfAXcBXwHbzf6YdBms0VqG7FzQajSaG6O4FjUajiSFadDUajSaGaNHVaDSaGKJFV6PRaGKIFl2NRqOJIVp0NRqNJoZo0dVoNJoYokVXo9FoYsj/A5o+gnzPBFNDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.pie(dictData_test.values(),\n",
    "        explode=(0, 0, 0, 0, 0, 0) , \n",
    "        labels=dictData_test.keys(),\n",
    "        autopct='%1.1f%%')\n",
    "plt.axis('equal')\n",
    "plt.title('Proportion of each observed category - Test data')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lqzxbxVeu8WO"
   },
   "source": [
    "### 2.1 Create Convolution Neural Network(CNN) using different layers & ReLU ⚙\n",
    "\n",
    "Here the Activation function used is Rectified Linear (ReLU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "3WGvs0MovIL_"
   },
   "outputs": [],
   "source": [
    "#CNN network\n",
    "\n",
    "class ConvNet_RELU(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "        super(ConvNet_RELU,self).__init__()\n",
    "\n",
    "        # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "        # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "        self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.relu1 = nn.ReLU()\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "        # Reduce the image size by factor of 2\n",
    "        #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "        # 2nd Layer\n",
    "\n",
    "\n",
    "        self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,12,75,75)\n",
    "        self.relu2 = nn.ReLU()\n",
    "        #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "        # 3rd Layer\n",
    "\n",
    "\n",
    "        self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,32,75,75)\n",
    "        self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "        #Shape -> (256,32,75,75)\n",
    "        self.relu3 = nn.ReLU()\n",
    "        #Shape -> (256,32,75,75) \n",
    "\n",
    "        self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "\n",
    "# Feed forward function\n",
    "  def forward(self, input):\n",
    "    output = self.conv1(input)\n",
    "    output = self.bn1(output)\n",
    "    output = self.relu1(output)\n",
    "\n",
    "    output = self.pool(output)\n",
    "\n",
    "    output = self.conv2(output)\n",
    "    output = self.relu2(output)\n",
    "\n",
    "    output = self.conv3(output)\n",
    "    output = self.bn2(output)\n",
    "    output = self.relu3(output)\n",
    "\n",
    "    # the output will be in matric form of shape (256,32,75,75)\n",
    "    # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "    output = output.view(-1,32*75*75)\n",
    "    output = self.fullyConnected(output)\n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YbG9S4htAUk9"
   },
   "source": [
    "### 2.2 Create Convolution Neural Network(CNN) using different layers & Exponential Linear Unit(ELU) ⚙\n",
    "\n",
    "Here the Activation function used is Exponential Linear Unit(ELU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "Vk34fQb9ATPm"
   },
   "outputs": [],
   "source": [
    "#CNN network\n",
    "class ConvNet_ELU(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "        super(ConvNet_ELU,self).__init__()\n",
    "\n",
    "        # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "        # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "        self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.elu1 = nn.ELU()\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "        # Reduce the image size by factor of 2\n",
    "        #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "        # 2nd Layer\n",
    "\n",
    "\n",
    "        self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,12,75,75)\n",
    "        self.elu2 = nn.ELU()\n",
    "        #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "        # 3rd Layer\n",
    "\n",
    "\n",
    "        self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,32,75,75)\n",
    "        self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "        #Shape -> (256,32,75,75)\n",
    "        self.elu3 = nn.ELU()\n",
    "        #Shape -> (256,32,75,75) \n",
    "\n",
    "        self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "\n",
    "# Feed forward function\n",
    "  def forward(self, input):\n",
    "    output = self.conv1(input)\n",
    "    output = self.bn1(output)\n",
    "    output = self.elu1(output)\n",
    "\n",
    "    output = self.pool(output)\n",
    "\n",
    "    output = self.conv2(output)\n",
    "    output = self.elu2(output)\n",
    "\n",
    "    output = self.conv3(output)\n",
    "    output = self.bn2(output)\n",
    "    output = self.elu3(output)\n",
    "\n",
    "    # the output will be in matric form of shape (256,32,75,75)\n",
    "    # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "    output = output.view(-1,32*75*75)\n",
    "    output = self.fullyConnected(output)\n",
    "\n",
    "    return output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "fPo7MMPAvbqX"
   },
   "outputs": [],
   "source": [
    "# model using RELU activation \n",
    "model_RELU = ConvNet_RELU(num_classes=6).to(device)\n",
    "#model using ELU activation\n",
    "model_ELU = ConvNet_ELU(num_classes=6).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "z_qUgjwrTHwB",
    "outputId": "8f27a774-f5e8-4448-aaf9-3d3a3a29e9e4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1         [-1, 12, 150, 150]             336\n",
      "       BatchNorm2d-2         [-1, 12, 150, 150]              24\n",
      "              ReLU-3         [-1, 12, 150, 150]               0\n",
      "         MaxPool2d-4           [-1, 12, 75, 75]               0\n",
      "            Conv2d-5           [-1, 20, 75, 75]           2,180\n",
      "              ReLU-6           [-1, 20, 75, 75]               0\n",
      "            Conv2d-7           [-1, 32, 75, 75]           5,792\n",
      "       BatchNorm2d-8           [-1, 32, 75, 75]              64\n",
      "              ReLU-9           [-1, 32, 75, 75]               0\n",
      "           Linear-10                    [-1, 6]       1,080,006\n",
      "================================================================\n",
      "Total params: 1,088,402\n",
      "Trainable params: 1,088,402\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.26\n",
      "Forward/backward pass size (MB): 12.53\n",
      "Params size (MB): 4.15\n",
      "Estimated Total Size (MB): 16.94\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Get model summary model_RELU\n",
    "# Reference - https://stackoverflow.com/questions/42480111/model-summary-in-pytorch\n",
    "summary(model_RELU, (3,150,150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HxllixXRBfdi",
    "outputId": "42b90d3a-7b68-4eaa-a430-e395eac0837b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1         [-1, 12, 150, 150]             336\n",
      "       BatchNorm2d-2         [-1, 12, 150, 150]              24\n",
      "               ELU-3         [-1, 12, 150, 150]               0\n",
      "         MaxPool2d-4           [-1, 12, 75, 75]               0\n",
      "            Conv2d-5           [-1, 20, 75, 75]           2,180\n",
      "               ELU-6           [-1, 20, 75, 75]               0\n",
      "            Conv2d-7           [-1, 32, 75, 75]           5,792\n",
      "       BatchNorm2d-8           [-1, 32, 75, 75]              64\n",
      "               ELU-9           [-1, 32, 75, 75]               0\n",
      "           Linear-10                    [-1, 6]       1,080,006\n",
      "================================================================\n",
      "Total params: 1,088,402\n",
      "Trainable params: 1,088,402\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.26\n",
      "Forward/backward pass size (MB): 12.53\n",
      "Params size (MB): 4.15\n",
      "Estimated Total Size (MB): 16.94\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Get model summary model_LRELU\n",
    "# Reference - https://stackoverflow.com/questions/42480111/model-summary-in-pytorch\n",
    "summary(model_ELU, (3,150,150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "bbjTk_iW-G_s"
   },
   "outputs": [],
   "source": [
    "# loss functions\n",
    "\n",
    "'''\n",
    "Things to try:\n",
    "•Changing the batch size\n",
    "•Changing the weight decay parameter\n",
    "•Reduce the size of the training set (+ weight decay)\n",
    "•Changing the learning rate\n",
    "\n",
    "'''\n",
    "# Types of loss fns for multi class image classification\n",
    "# Reference - https://neptune.ai/blog/pytorch-loss-functions\n",
    "CrossEntropyLoss = nn.CrossEntropyLoss()\n",
    "KLDivLoss = nn.KLDivLoss()\n",
    "NLLLoss = nn.NLLLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "-7QWDH7P-WHX"
   },
   "outputs": [],
   "source": [
    "num_epochs = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "Z9nJXIz0-ZJT"
   },
   "outputs": [],
   "source": [
    "#calculating the size of training and testing images\n",
    "train_count=len(glob.glob(train_path+'/**/*.jpg'))\n",
    "test_count=len(glob.glob(test_path+'/**/*.jpg'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eectNE2m-fX8",
    "outputId": "910b91a3-e506-43ce-848d-970ef5bda9f4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14034 3000\n"
     ]
    }
   ],
   "source": [
    "print(train_count,test_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "qfMefk58nLut"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import seaborn as sn\n",
    "import pandas as pd\n",
    "\n",
    "def createConfusionMatrix(loader):\n",
    "    y_pred = [] # save predction\n",
    "    y_true = [] # save ground truth\n",
    "\n",
    "    # iterate over data\n",
    "    for inputs, labels in loader:\n",
    "        output = model(Variable(inputs.cuda()))  # Feed Network\n",
    "\n",
    "        output = (torch.max(torch.exp(output), 1)[1]).data.cpu().numpy()\n",
    "        y_pred.extend(output)  # save prediction\n",
    "\n",
    "        labels = labels.data.cpu().numpy()\n",
    "        y_true.extend(labels)  # save ground truth\n",
    "\n",
    "    # constant for classes\n",
    "    classes = ('buildings', 'forest', 'glacier', 'mountain', 'sea', 'street')\n",
    "\n",
    "    # Build confusion matrix\n",
    "    cf_matrix = confusion_matrix(y_true, y_pred)\n",
    "    df_cm = pd.DataFrame(cf_matrix/np.sum(cf_matrix) * 10, index=[i for i in classes],\n",
    "                         columns=[i for i in classes])\n",
    "    plt.figure(figsize=(12, 7))    \n",
    "    return sn.heatmap(df_cm, annot=True).get_figure()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GQzBCMhf8L9D"
   },
   "source": [
    "### 3.1  HyperParameters visualization - Tensorboard\n",
    "\n",
    "To Answer - ***How do different Activation Function, Gradient estimation, loss function, Number of Epochs affect Model  Accuracy and Loss***\n",
    "\n",
    "\n",
    "The model will be trained on different runs and at end of it we can visualize the plots of all **run combinations** in **tensorboard** scalars tab.\n",
    "\n",
    "**After getting the best parameters we use the model on Test dataset.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hGrPsgRaitMH"
   },
   "source": [
    "### 3.2 Affect of Different Activation Functions 1️⃣\n",
    "\n",
    "Here we will compare 2 Activation Functions in ConvNet namely ELU and RELU and Visualize the Accuracy and loss in Tensorboard\n",
    "\n",
    "Change the activation function. How does it affect accuracy? How does it affect how quickly the network plateaus?\n",
    "\n",
    "Various activation functions:  \n",
    "\n",
    "- Rectified linear unit (ReLU) ✅\n",
    "- TanH\n",
    "- Leaky rectified linear unit (Leaky ReLU)\n",
    "- Parameteric rectified linear unit (PReLU) Randomized leaky rectified linear unit (RReLU)\n",
    "- Exponential linear unit (ELU) ✅\n",
    "- Scaled exponential linear unit (SELU)\n",
    "- S-shaped rectified linear activation unit (SReLU)\n",
    "- Adaptive piecewise linear (APL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "3nrf7etzqoSL"
   },
   "outputs": [],
   "source": [
    "#CNN network\n",
    "class ConvNet(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "            \n",
    "            super(ConvNet,self).__init__()\n",
    "\n",
    "            # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "            # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "            self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.relu1 = nn.ReLU()\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "            # Reduce the image size by factor of 2\n",
    "            #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "            # 2nd Layer\n",
    "\n",
    "\n",
    "            self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,75,75)\n",
    "            self.relu2 = nn.ReLU()\n",
    "            #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "            # 3rd Layer\n",
    "\n",
    "\n",
    "            self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.relu3 = nn.ReLU()\n",
    "            #Shape -> (256,32,75,75) \n",
    "\n",
    "            self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "\n",
    "      # Feed forward function\n",
    "      def forward(self, input):\n",
    "            \n",
    "            output = self.conv1(input)\n",
    "            output = self.bn1(output)\n",
    "            output = self.relu1(output)\n",
    "\n",
    "            output = self.pool(output)\n",
    "\n",
    "            output = self.conv2(output)\n",
    "            output = self.relu2(output)\n",
    "\n",
    "            output = self.conv3(output)\n",
    "            output = self.bn2(output)\n",
    "            output = self.relu3(output)\n",
    "\n",
    "            # the output will be in matric form of shape (256,32,75,75)\n",
    "            # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "            output = output.view(-1,32*75*75)\n",
    "            output = self.fullyConnected(output)\n",
    "\n",
    "            return output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "B1dOYrUIqpjC"
   },
   "outputs": [],
   "source": [
    "model = ConvNet(num_classes=6).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "A6LY6d8Es2cc"
   },
   "outputs": [],
   "source": [
    "#CNN network\n",
    "class ConvNetELU(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "        super(ConvNetELU,self).__init__()\n",
    "\n",
    "        # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "        # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "        self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.elu1 = nn.ELU()\n",
    "        #Shape -> (256,12,150,150)\n",
    "        self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "        # Reduce the image size by factor of 2\n",
    "        #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "        # 2nd Layer\n",
    "\n",
    "\n",
    "        self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,12,75,75)\n",
    "        self.elu2 = nn.ELU()\n",
    "        #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "        # 3rd Layer\n",
    "\n",
    "\n",
    "        self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "        #Shape -> (256,32,75,75)\n",
    "        self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "        #Shape -> (256,32,75,75)\n",
    "        self.elu3 = nn.ELU()\n",
    "        #Shape -> (256,32,75,75) \n",
    "\n",
    "        self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "\n",
    "# Feed forward function\n",
    "  def forward(self, input):\n",
    "    output = self.conv1(input)\n",
    "    output = self.bn1(output)\n",
    "    output = self.elu1(output)\n",
    "\n",
    "    output = self.pool(output)\n",
    "\n",
    "    output = self.conv2(output)\n",
    "    output = self.elu2(output)\n",
    "\n",
    "    output = self.conv3(output)\n",
    "    output = self.bn2(output)\n",
    "    output = self.elu3(output)\n",
    "\n",
    "    # the output will be in matric form of shape (256,32,75,75)\n",
    "    # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "    output = output.view(-1,32*75*75)\n",
    "    output = self.fullyConnected(output)\n",
    "\n",
    "    return output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "yWEDjibds3ui"
   },
   "outputs": [],
   "source": [
    "#model using ELU activation\n",
    "model_ELU = ConvNet(num_classes=6).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HjGWF9BoxwSe"
   },
   "source": [
    "First, Lets run the model having RELU activation and saving the Accuracy/Loss to Tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "dCu1uQHq9ajR"
   },
   "outputs": [],
   "source": [
    "parameters = dict(\n",
    "    modelTypes = [model, model_ELU]\n",
    ")\n",
    "param_values = [v for v in parameters.values()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WMsODq5S--30",
    "outputId": "ed373117-bdcb-45e3-e87c-42de6fd48ce3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[ConvNet(\n",
       "    (conv1): Conv2d(3, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (bn1): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (relu1): ReLU()\n",
       "    (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "    (conv2): Conv2d(12, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (relu2): ReLU()\n",
       "    (conv3): Conv2d(20, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (relu3): ReLU()\n",
       "    (fullyConnected): Linear(in_features=180000, out_features=6, bias=True)\n",
       "  ),\n",
       "  ConvNet(\n",
       "    (conv1): Conv2d(3, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (bn1): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (relu1): ReLU()\n",
       "    (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "    (conv2): Conv2d(12, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (relu2): ReLU()\n",
       "    (conv3): Conv2d(20, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "    (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (relu3): ReLU()\n",
       "    (fullyConnected): Linear(in_features=180000, out_features=6, bias=True)\n",
       "  )]]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "param_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sZHvrR7dKcbw",
    "outputId": "2c08a9d3-a7c7-46bc-abe4-b2cef62b6e0a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ConvNet(\n",
       "  (conv1): Conv2d(3, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn1): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu1): ReLU()\n",
       "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "  (conv2): Conv2d(12, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (relu2): ReLU()\n",
       "  (conv3): Conv2d(20, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu3): ReLU()\n",
       "  (fullyConnected): Linear(in_features=180000, out_features=6, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 448
    },
    "id": "7JPb7JDei48Z",
    "outputId": "4deea8ba-c78b-44d0-9fdd-7749fd13615e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0 Accuracy 0.5617072823143794 loss: tensor(7.4354)\n",
      "epoch: 1 Accuracy 0.7126977340743907 loss: tensor(1.5485)\n",
      "epoch: 2 Accuracy 0.771626051018954 loss: tensor(1.1773)\n",
      "epoch: 3 Accuracy 0.8156619638021947 loss: tensor(0.7832)\n",
      "epoch: 4 Accuracy 0.8457317942140515 loss: tensor(0.6330)\n",
      "epoch: 5 Accuracy 0.8730226592560923 loss: tensor(0.5813)\n",
      "epoch: 6 Accuracy 0.9052301553370387 loss: tensor(0.3643)\n",
      "epoch: 7 Accuracy 0.9397178281316803 loss: tensor(0.2355)\n",
      "epoch: 8 Accuracy 0.9481972352857346 loss: tensor(0.1720)\n",
      "epoch: 9 Accuracy 0.9590993301980903 loss: tensor(0.1327)\n",
      "___________________________________________________________________\n",
      "epoch: 0 Accuracy 0.5302835969787658 loss: tensor(12.9690)\n",
      "epoch: 1 Accuracy 0.7359270343451617 loss: tensor(1.2118)\n",
      "epoch: 2 Accuracy 0.8016958814308109 loss: tensor(0.7880)\n",
      "epoch: 3 Accuracy 0.8563488670371954 loss: tensor(0.5124)\n",
      "epoch: 4 Accuracy 0.8860624198375374 loss: tensor(0.4238)\n",
      "epoch: 5 Accuracy 0.9036625338463731 loss: tensor(0.3463)\n",
      "epoch: 6 Accuracy 0.9110731081658828 loss: tensor(0.3315)\n",
      "epoch: 7 Accuracy 0.9467008693173721 loss: tensor(0.1807)\n",
      "epoch: 8 Accuracy 0.9578167307966368 loss: tensor(0.1452)\n",
      "epoch: 9 Accuracy 0.9556078096052444 loss: tensor(0.1482)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training Loop\n",
    "modelRuns = [model, model_ELU]\n",
    "for index, model in enumerate(modelRuns):\n",
    "  \n",
    "  train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "  # Adam_RELU = Adam(model_RELU.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "  optimizer = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "  criterion = CrossEntropyLoss\n",
    "  modelTypes = [\"model_RELU\", \"model_ELU\"]\n",
    "  comment = f' model={model} Adam_0_001 Cross_EntropyLoss'\n",
    "  tb = SummaryWriter(log_dir = 'runs/Exp_'+modelTypes[index],comment=comment)\n",
    "  for epoch in range(10):\n",
    "      total_loss = 0\n",
    "      total_correct = 0\n",
    "      train_accuracy=0.0\n",
    "      train_loss=0.0\n",
    "      model.train()\n",
    "      # loop = tqdm(enumerate(train_loader), total = len(train_loader), leave = False)\n",
    "      for images, labels in train_loader:\n",
    "          # images, labels = images.to(device), labels.to(device)\n",
    "          # preds = model(images)\n",
    "          if torch.cuda.is_available():\n",
    "              images=Variable(images.cuda())\n",
    "              labels=Variable(labels.cuda())\n",
    "          # at start of new batch the gradient is made 0    \n",
    "          optimizer.zero_grad()\n",
    "          # This gives us the prediction\n",
    "          outputs=model(images)\n",
    "          # cal loss using pred and actual value\n",
    "          loss=criterion(outputs,labels)\n",
    "          #backpropogation\n",
    "          loss.backward()\n",
    "          # Update weights\n",
    "          optimizer.step()\n",
    "          \n",
    "          # Multiply loss with image size\n",
    "          train_loss+= loss.cpu().data*images.size(0)\n",
    "          # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "          # the more the network thinks that the image is of the particular class.\n",
    "          # So, let’s get the index of the highest energy:\n",
    "          _,prediction=torch.max(outputs.data,1)\n",
    "          \n",
    "          train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "          # loss = criterion(preds, labels)\n",
    "          # total_loss+= loss.item()\n",
    "          # total_correct+= get_num_correct(preds, labels)\n",
    "\n",
    "          # optimizer.zero_grad()\n",
    "          # loss.backward()\n",
    "          # optimizer.step()\n",
    "\n",
    "      # tb.add_scalar(\"Loss\", total_loss, epoch)\n",
    "      # tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "      # tb.add_scalar(\"Accuracy\", total_correct/train_count, epoch)\n",
    "      total_loss_in_epoch = train_loss/train_count\n",
    "      tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "      #  tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "      tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "      # model_index+=1\n",
    "\n",
    "      # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "      # print(\"model:\",model)\n",
    "      print(\"epoch:\", epoch, \"Accuracy\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "  print(\"___________________________________________________________________\")\n",
    "\n",
    "  # tb.add_hparams(\n",
    "  #         {\"lr\": lr, \"bsize\": batch_size, \"shuffle\":shuffle},\n",
    "  #         {\n",
    "  #             \"accuracy\": total_correct/train_count,\n",
    "  #             \"loss\": total_loss,\n",
    "  #         },\n",
    "  #     )\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy and Loss Plots as from Tensorboard\n",
    "\n",
    "NOTE : **For running the tensorboard in discovery cluster, we need to launch separate instance of tensorboard from Interactive Apps**\n",
    "\n",
    "#### Inference 🔎\n",
    "\n",
    "It is observed that the changes in activation function - RELU and ELU didnt affect the output accuracy and loss by much, given the no. of epochs is 10.\n",
    "If this model is tested on a test set or validation set, we can know more about its performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![activation_accuracy](https://user-images.githubusercontent.com/13203059/145697917-96b60774-7f47-4818-8440-72ad0384fb25.jpg)\n",
    "![activation_loss](https://user-images.githubusercontent.com/13203059/145697931-3138166d-80a2-4de0-80e6-47c77ad30d69.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1ZWx3LG-oA2D"
   },
   "source": [
    "### 3.3 Affect of Different Cost Functions 2️⃣\n",
    "\n",
    "Here I will try with 2 cost functions and observe the affects of these on accuracy, loss in Tensorboard - \n",
    "\n",
    "- Cross-Entropy ✅\n",
    "- Negative Log likelihood(NLL) loss ✅"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CNN network for NLL loss\n",
    "class ConvNet(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "            \n",
    "            super(ConvNet,self).__init__()\n",
    "\n",
    "            # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "            # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "            self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.relu1 = nn.ReLU()\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "            # Reduce the image size by factor of 2\n",
    "            #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "            # 2nd Layer\n",
    "\n",
    "\n",
    "            self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,75,75)\n",
    "            self.relu2 = nn.ReLU()\n",
    "            #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "            # 3rd Layer\n",
    "\n",
    "\n",
    "            self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.relu3 = nn.ReLU()\n",
    "            #Shape -> (256,32,75,75) \n",
    "\n",
    "            self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "            self.logSoftmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "      # Feed forward function\n",
    "      def forward(self, input):\n",
    "            \n",
    "            output = self.conv1(input)\n",
    "            output = self.bn1(output)\n",
    "            output = self.relu1(output)\n",
    "\n",
    "            output = self.pool(output)\n",
    "\n",
    "            output = self.conv2(output)\n",
    "            output = self.relu2(output)\n",
    "\n",
    "            output = self.conv3(output)\n",
    "            output = self.bn2(output)\n",
    "            output = self.relu3(output)\n",
    "\n",
    "            # the output will be in matric form of shape (256,32,75,75)\n",
    "            # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "            output = output.view(-1,32*75*75)\n",
    "            output = self.fullyConnected(output)\n",
    "            output = self.logSoftmax(output)\n",
    "\n",
    "            return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ConvNet(num_classes=6).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1         [-1, 12, 150, 150]             336\n",
      "       BatchNorm2d-2         [-1, 12, 150, 150]              24\n",
      "              ReLU-3         [-1, 12, 150, 150]               0\n",
      "         MaxPool2d-4           [-1, 12, 75, 75]               0\n",
      "            Conv2d-5           [-1, 20, 75, 75]           2,180\n",
      "              ReLU-6           [-1, 20, 75, 75]               0\n",
      "            Conv2d-7           [-1, 32, 75, 75]           5,792\n",
      "       BatchNorm2d-8           [-1, 32, 75, 75]              64\n",
      "              ReLU-9           [-1, 32, 75, 75]               0\n",
      "           Linear-10                    [-1, 6]       1,080,006\n",
      "       LogSoftmax-11                    [-1, 6]               0\n",
      "================================================================\n",
      "Total params: 1,088,402\n",
      "Trainable params: 1,088,402\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.26\n",
      "Forward/backward pass size (MB): 12.53\n",
      "Params size (MB): 4.15\n",
      "Estimated Total Size (MB): 16.94\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "summary(model, (3,150,150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "fauPJeegogPv"
   },
   "outputs": [],
   "source": [
    "# Training Loop\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "criterion = NLLLoss\n",
    "\n",
    "comment = f' ConvNet_RELU Adam_0_001 loss = NLLLoss'\n",
    "\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_Loss_NLLLoss_Exp',comment=comment)\n",
    "for epoch in range(10):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,torch.unique(labels))\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "     # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "    print(\"epoch:\", epoch, \"Accuracy:\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ConvNet for CrossEntropy loss\n",
    "class ConvNet(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "            \n",
    "            super(ConvNet,self).__init__()\n",
    "\n",
    "            # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "            # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "            self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.relu1 = nn.ReLU()\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "            # Reduce the image size by factor of 2\n",
    "            #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "            # 2nd Layer\n",
    "\n",
    "\n",
    "            self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,75,75)\n",
    "            self.relu2 = nn.ReLU()\n",
    "            #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "            # 3rd Layer\n",
    "\n",
    "\n",
    "            self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.relu3 = nn.ReLU()\n",
    "            #Shape -> (256,32,75,75) \n",
    "\n",
    "            self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "\n",
    "      # Feed forward function\n",
    "      def forward(self, input):\n",
    "            \n",
    "            output = self.conv1(input)\n",
    "            output = self.bn1(output)\n",
    "            output = self.relu1(output)\n",
    "\n",
    "            output = self.pool(output)\n",
    "\n",
    "            output = self.conv2(output)\n",
    "            output = self.relu2(output)\n",
    "\n",
    "            output = self.conv3(output)\n",
    "            output = self.bn2(output)\n",
    "            output = self.relu3(output)\n",
    "\n",
    "            # the output will be in matric form of shape (256,32,75,75)\n",
    "            # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "            output = output.view(-1,32*75*75)\n",
    "            output = self.fullyConnected(output)\n",
    "\n",
    "            return output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ConvNet(num_classes=6).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training Loop\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "criterion = CrossEntropyLoss\n",
    "\n",
    "comment = f' ConvNet_RELU Adam_0_001 loss = CrossEntropyLoss'\n",
    "\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_Loss_CrossEntropyLoss_Exp',comment=comment)\n",
    "for epoch in range(10):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,torch.unique(labels))\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "     # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "    print(\"epoch:\", epoch, \"Accuracy:\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy and Loss Plots as from Tensorboard\n",
    "\n",
    "NOTE : **For running the tensorboard in discovery cluster, we need to launch separate instance of tensorboard from Interactive Apps**\n",
    "\n",
    "#### Inference 🔎\n",
    "\n",
    "It is observed that the changes in Cost function didnt affect the output accuracy and loss by much, given the no. of epochs is 10. One reason being that cross entropy and NLL have similar calculation algorithm.\n",
    "If this model is tested on a test set or validation set, we can know more about its performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![loss_accuracy](https://user-images.githubusercontent.com/13203059/145697939-75c3f767-2495-4ff7-a955-926840eb6735.jpg)\n",
    "\n",
    "![lossfn_loss](https://user-images.githubusercontent.com/13203059/145697951-a1b80695-cd9b-4da7-8e9c-3f12fe2e3dbb.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qR2Bm2uYpdQj"
   },
   "source": [
    "### 3.4 Affect of Different Epochs 3️⃣\n",
    "\n",
    "Here I will record the training over **15 Epochs**, 5 more than usual Epochs run in previous experiments and observe the **affects** of these **on accuracy, loss** in Tensorboard - \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "id": "UQ444WZgptcl"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0 Accuracy 0.5484537551660253 loss: tensor(7.9433)\n",
      "epoch: 1 Accuracy 0.7314379364400742 loss: tensor(1.1803)\n",
      "epoch: 2 Accuracy 0.7855208778680348 loss: tensor(0.8853)\n",
      "epoch: 3 Accuracy 0.827704147071398 loss: tensor(0.7189)\n",
      "epoch: 4 Accuracy 0.8503634031637451 loss: tensor(0.6395)\n",
      "epoch: 5 Accuracy 0.8740914920906371 loss: tensor(0.5730)\n",
      "epoch: 6 Accuracy 0.9225452472566624 loss: tensor(0.2965)\n",
      "epoch: 7 Accuracy 0.9430668376799202 loss: tensor(0.1907)\n",
      "epoch: 8 Accuracy 0.9293145218754454 loss: tensor(0.2818)\n",
      "epoch: 9 Accuracy 0.9288869887416275 loss: tensor(0.2906)\n",
      "epoch: 10 Accuracy 0.9590993301980903 loss: tensor(0.1419)\n",
      "epoch: 11 Accuracy 0.9619495510902095 loss: tensor(0.1366)\n",
      "epoch: 12 Accuracy 0.9687188257089925 loss: tensor(0.1142)\n",
      "epoch: 13 Accuracy 0.973421690180989 loss: tensor(0.0962)\n",
      "epoch: 14 Accuracy 0.9883140943423115 loss: tensor(0.0461)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training Loop\n",
    "\n",
    "# model = ConvNet().to(device)\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "loss = CrossEntropyLoss\n",
    "comment = f' ConvNet_RELU Adam_0_001_CrossEntropy'\n",
    "# print('LOSS HERE---',loss[run_id])\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_EPOCH_Exp',comment=comment)\n",
    "for epoch in range(15):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=loss(outputs,labels)\n",
    "        #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "     #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "    # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "    print(\"epoch:\", epoch,\"Accuracy\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy and Loss Plots as from Tensorboard\n",
    "\n",
    "NOTE : **For running the tensorboard in discovery cluster, we need to launch separate instance of tensorboard from Interactive Apps**\n",
    "\n",
    "#### Inference 🔎\n",
    "\n",
    "It is observed that the changes in Epoch runs affect the output accuracy and loss since the model reduces the subsequent loss by adjusting weights. We can see that perhaps the model may plateau after 15 epochs.\n",
    "If this model is tested on a test set or validation set, we can know more about its performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![epoch_accuracy](https://user-images.githubusercontent.com/13203059/145697967-a4501de0-f0c0-478a-995a-2f0db58c03cd.jpg)\n",
    "\n",
    "![epoch_loss](https://user-images.githubusercontent.com/13203059/145698004-2e1c0ceb-4ca1-48e7-a6e7-8c29b211bf84.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9wec6qMHrs6m"
   },
   "source": [
    "### 3.5 Affect of Different Gradient Estimation 4️⃣\n",
    "\n",
    "Here I will compare two gradient estimations on ConvNet - \n",
    "\n",
    "Change the gradient estimation. How does it affect accuracy? How does it affect how quickly the network plateaus?\n",
    "Various forms of gradient estimation:\n",
    "\n",
    "- Adam ✅\n",
    "- Stochastic Gradient Descent(SGD) ✅\n",
    "- AGAdadelta\n",
    "- Momentum\n",
    "- Adagrad\n",
    "- RMSProp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W7RWtpX8-hHl",
    "outputId": "965439f3-cf33-4427-87ce-224ff95023ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr: 0.001 loss: tensor(2.6736)\n",
      "epoch: 0 Accuracy 0.4343736639589568 loss: tensor(2.6736)\n",
      "lr: 0.001 loss: tensor(1.4788)\n",
      "epoch: 1 Accuracy 0.5658401026079521 loss: tensor(1.4788)\n",
      "lr: 0.001 loss: tensor(1.2393)\n",
      "epoch: 2 Accuracy 0.6136525580732507 loss: tensor(1.2393)\n",
      "lr: 0.001 loss: tensor(0.9904)\n",
      "epoch: 3 Accuracy 0.6710845090494514 loss: tensor(0.9904)\n",
      "lr: 0.001 loss: tensor(0.8806)\n",
      "epoch: 4 Accuracy 0.7035770272196096 loss: tensor(0.8806)\n",
      "lr: 0.001 loss: tensor(0.7800)\n",
      "epoch: 5 Accuracy 0.725167450477412 loss: tensor(0.7800)\n",
      "lr: 0.001 loss: tensor(0.7159)\n",
      "epoch: 6 Accuracy 0.7504631608949693 loss: tensor(0.7159)\n",
      "lr: 0.001 loss: tensor(0.6421)\n",
      "epoch: 7 Accuracy 0.7719110731081659 loss: tensor(0.6421)\n",
      "lr: 0.001 loss: tensor(0.5737)\n",
      "epoch: 8 Accuracy 0.794712840245119 loss: tensor(0.5737)\n",
      "lr: 0.001 loss: tensor(0.5364)\n",
      "epoch: 9 Accuracy 0.8167307966367393 loss: tensor(0.5364)\n",
      "lr: 0.001 loss: tensor(0.5096)\n",
      "epoch: 10 Accuracy 0.8249964372238848 loss: tensor(0.5096)\n",
      "lr: 0.001 loss: tensor(0.4851)\n",
      "epoch: 11 Accuracy 0.83133817870885 loss: tensor(0.4851)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "#  Training Loop on convnet RELU SGD with different Lr\n",
    "# Learning Rate = 0.001\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "optimizer = SGD(model.parameters(), lr=0.001)\n",
    "loss_criterion = CrossEntropyLoss\n",
    "# model_names= ['ConvNet_RELU', 'ConvNet_ELU']\n",
    "comment = f'model_RELU batch_size=256 lr = 0.001 grad_est = SGD CrossEntropyLoss'\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_GradientEstim_SGD_0.001_Exp',comment=comment)\n",
    "for epoch in range(12):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=loss_criterion(outputs,labels)\n",
    "        #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "#             preds = model_type(images)\n",
    "\n",
    "#             lossObserved = loss_criterion(preds, labels)\n",
    "#             total_loss+= lossObserved.item()\n",
    "#             total_correct+= get_num_correct(preds, labels)\n",
    "\n",
    "#             optimizer.zero_grad()\n",
    "#             lossObserved.backward()\n",
    "#             optimizer.step()\n",
    "\n",
    "\n",
    "    tb.add_scalar(\"Loss\", train_loss/train_count, epoch)\n",
    "#         tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "    print(\"lr:\", \"0.001\", \"loss:\", train_loss/train_count)\n",
    "    print(\"epoch:\", epoch,\"Accuracy\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "# tb.add_hparams(\n",
    "#         {\"model:\",model_type, \"batch_size:\", batch_size,\"gradient_est:\",gradient_est, \"loss:\", loss},\n",
    "#         {\n",
    "#             \"accuracy\": total_correct/train_count,\n",
    "#             \"loss\": total_loss,\n",
    "#         },\n",
    "#     )\n",
    "\n",
    "tb.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr: 0.01 loss: tensor(102.5951)\n",
      "epoch: 0 Accuracy 0.36447199657973495 loss: tensor(102.5951)\n",
      "lr: 0.01 loss: tensor(3.1781)\n",
      "epoch: 1 Accuracy 0.2089924469146359 loss: tensor(3.1781)\n",
      "lr: 0.01 loss: tensor(1.8011)\n",
      "epoch: 2 Accuracy 0.2003705287159755 loss: tensor(1.8011)\n",
      "lr: 0.01 loss: tensor(1.7554)\n",
      "epoch: 3 Accuracy 0.2341456462875873 loss: tensor(1.7554)\n",
      "lr: 0.01 loss: tensor(1.6868)\n",
      "epoch: 4 Accuracy 0.28395325637736923 loss: tensor(1.6868)\n",
      "lr: 0.01 loss: tensor(1.5210)\n",
      "epoch: 5 Accuracy 0.4112156192104888 loss: tensor(1.5210)\n",
      "lr: 0.01 loss: tensor(1.3514)\n",
      "epoch: 6 Accuracy 0.48304118569189114 loss: tensor(1.3514)\n",
      "lr: 0.01 loss: tensor(1.2058)\n",
      "epoch: 7 Accuracy 0.533205073393188 loss: tensor(1.2058)\n",
      "lr: 0.01 loss: tensor(1.1178)\n",
      "epoch: 8 Accuracy 0.5669089354424968 loss: tensor(1.1178)\n",
      "lr: 0.01 loss: tensor(1.2785)\n",
      "epoch: 9 Accuracy 0.48959669374376513 loss: tensor(1.2785)\n",
      "lr: 0.01 loss: tensor(1.1189)\n",
      "epoch: 10 Accuracy 0.5684053014108593 loss: tensor(1.1189)\n",
      "lr: 0.01 loss: tensor(1.0139)\n",
      "epoch: 11 Accuracy 0.6218469431380932 loss: tensor(1.0139)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training Loop on convnet RELU Adam with different Lr\n",
    "# for run_id, lrate in enumerate([0.01, 0.001]):\n",
    "#     print(\"run id:\", run_id + 1)\n",
    "# #     model_type = model_RELU\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "optimizer = Adam(model.parameters(), lr=0.01)\n",
    "loss_criterion = CrossEntropyLoss\n",
    "# model_names= ['ConvNet_RELU', 'ConvNet_ELU']\n",
    "comment = f'model_RELU batch_size=256 lr = 0.01 grad_est = ADAM CrossEntropyLoss'\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_GradientEstim_ADAM_0_01_Exp',comment=comment)\n",
    "for epoch in range(12):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=loss_criterion(outputs,labels)\n",
    "        #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "#             preds = model_type(images)\n",
    "\n",
    "#             lossObserved = loss_criterion(preds, labels)\n",
    "#             total_loss+= lossObserved.item()\n",
    "#             total_correct+= get_num_correct(preds, labels)\n",
    "\n",
    "#             optimizer.zero_grad()\n",
    "#             lossObserved.backward()\n",
    "#             optimizer.step()\n",
    "\n",
    "\n",
    "    tb.add_scalar(\"Loss\", train_loss/train_count, epoch)\n",
    "#         tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "    print(\"lr:\", \"0.01\", \"loss:\", train_loss/train_count)\n",
    "    print(\"epoch:\", epoch,\"Accuracy\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "# tb.add_hparams(\n",
    "#         {\"model:\",model_type, \"batch_size:\", batch_size,\"gradient_est:\",gradient_est, \"loss:\", loss},\n",
    "#         {\n",
    "#             \"accuracy\": total_correct/train_count,\n",
    "#             \"loss\": total_loss,\n",
    "#         },\n",
    "#     )\n",
    "\n",
    "tb.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy and Loss Plots as from Tensorboard\n",
    "\n",
    "NOTE : **For running the tensorboard in discovery cluster, we need to launch separate instance of tensorboard from Interactive Apps**\n",
    "\n",
    "#### Inference 🔎\n",
    "\n",
    "It is observed that the changes in Gradient Estimatations affect the output accuracy and loss by a significant factor, given the no. of epochs is 12. The model using RELU activation ADAM gradient with 0.01 learning rate performs best.\n",
    "If this model is tested on a test set or validation set, we can know more about its performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![gradient_accuracy](https://user-images.githubusercontent.com/13203059/145698014-ad291d8e-14af-4646-a152-8fb41a074ae1.jpg)\n",
    "![gradient_loss](https://user-images.githubusercontent.com/13203059/145698018-7ee189e1-1213-4d72-a4df-31c7b9acc713.jpg)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dKiyW7oytoUd"
   },
   "source": [
    "### 3.6 Affect of Different Network Architectures 5️⃣\n",
    "\n",
    "On the Deep Learning model changing the network architecture. How does it affect accuracy? How does it affect how quickly the network plateaus?\n",
    "\n",
    "Various forms of network architecture:\n",
    "\n",
    "- Number of layers\n",
    "\n",
    "- Size of each layer\n",
    "\n",
    "- Connection type\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "GHIESbNHwnSk"
   },
   "outputs": [],
   "source": [
    "#CNN network extra layer\n",
    "class ConvNet_RELU(nn.Module):\n",
    "  def __init__(self, num_classes=6):\n",
    "    super(ConvNet_RELU,self).__init__()\n",
    "\n",
    "    # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "    # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "    self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "    #Shape -> (256,12,150,150)\n",
    "    self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "    #Shape -> (256,12,150,150)\n",
    "    self.relu1 = nn.ReLU()\n",
    "    #Shape -> (256,12,150,150)\n",
    "    self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "    # Reduce the image size by factor of 2\n",
    "    #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "    # 2nd Layer\n",
    "\n",
    "    \n",
    "    self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "    #Shape -> (256,12,75,75)\n",
    "    self.relu2 = nn.ReLU()\n",
    "    #Shape -> (256,20,75,75)\n",
    "\n",
    "    \n",
    "    # 3rd Layer\n",
    "\n",
    "    \n",
    "    self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "    #Shape -> (256,32,75,75)\n",
    "    self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "    #Shape -> (256,32,75,75)\n",
    "    self.relu3 = nn.ReLU()\n",
    "    #Shape -> (256,32,75,75) \n",
    "    \n",
    "    # 4th layer Added Extra Layer to check the affect on Accuracy&Loss ✔\n",
    "    \n",
    "    self.conv4 = nn.Conv2d(in_channels=32, out_channels=44, kernel_size=3, stride=1, padding=1)\n",
    "    #Shape -> (256,32,75,75)\n",
    "    self.bn3 = nn.BatchNorm2d(num_features=44)\n",
    "    #Shape -> (256,32,75,75)\n",
    "    self.relu4 = nn.ReLU()\n",
    "    #Shape -> (256,32,75,75) \n",
    "\n",
    "    self.fullyConnected = nn.Linear(in_features=44*75*75, out_features=num_classes)\n",
    "\n",
    "# Feed forward function\n",
    "  def forward(self, input):\n",
    "    output = self.conv1(input)\n",
    "    output = self.bn1(output)\n",
    "    output = self.relu1(output)\n",
    "\n",
    "    output = self.pool(output)\n",
    "\n",
    "    output = self.conv2(output)\n",
    "    output = self.relu2(output)\n",
    "\n",
    "    output = self.conv3(output)\n",
    "    output = self.bn2(output)\n",
    "    output = self.relu3(output)\n",
    "    \n",
    "    output = self.conv4(output)\n",
    "    output = self.bn3(output)\n",
    "    output = self.relu4(output)\n",
    "\n",
    "    # the output will be in matric form of shape (256,32,75,75)\n",
    "    # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "    output = output.view(-1,44*75*75)\n",
    "    output = self.fullyConnected(output)\n",
    "\n",
    "    return output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "XJCoBozwwqA7"
   },
   "outputs": [],
   "source": [
    "model_RELU = ConvNet_RELU(num_classes=6).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1         [-1, 12, 150, 150]             336\n",
      "       BatchNorm2d-2         [-1, 12, 150, 150]              24\n",
      "              ReLU-3         [-1, 12, 150, 150]               0\n",
      "         MaxPool2d-4           [-1, 12, 75, 75]               0\n",
      "            Conv2d-5           [-1, 20, 75, 75]           2,180\n",
      "              ReLU-6           [-1, 20, 75, 75]               0\n",
      "            Conv2d-7           [-1, 32, 75, 75]           5,792\n",
      "       BatchNorm2d-8           [-1, 32, 75, 75]              64\n",
      "              ReLU-9           [-1, 32, 75, 75]               0\n",
      "           Conv2d-10           [-1, 44, 75, 75]          12,716\n",
      "      BatchNorm2d-11           [-1, 44, 75, 75]              88\n",
      "             ReLU-12           [-1, 44, 75, 75]               0\n",
      "           Linear-13                    [-1, 6]       1,485,006\n",
      "================================================================\n",
      "Total params: 1,506,206\n",
      "Trainable params: 1,506,206\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.26\n",
      "Forward/backward pass size (MB): 18.20\n",
      "Params size (MB): 5.75\n",
      "Estimated Total Size (MB): 24.20\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "summary(model_RELU, (3,150,150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0 Accuracy: 0.5078381074533276 loss: tensor(13.9754)\n",
      "epoch: 1 Accuracy: 0.6901809890266496 loss: tensor(1.7204)\n",
      "epoch: 2 Accuracy: 0.7696308963944706 loss: tensor(0.9795)\n",
      "epoch: 3 Accuracy: 0.8134530426108023 loss: tensor(0.7053)\n",
      "epoch: 4 Accuracy: 0.8349009548239988 loss: tensor(0.6299)\n",
      "epoch: 5 Accuracy: 0.8448767279464159 loss: tensor(0.6283)\n",
      "epoch: 6 Accuracy: 0.8886276186404446 loss: tensor(0.4116)\n",
      "epoch: 7 Accuracy: 0.927319367250962 loss: tensor(0.2586)\n",
      "epoch: 8 Accuracy: 0.9113581302550948 loss: tensor(0.3433)\n",
      "epoch: 9 Accuracy: 0.9234715690466011 loss: tensor(0.3014)\n",
      "epoch: 10 Accuracy: 0.956747897962092 loss: tensor(0.1614)\n",
      "epoch: 11 Accuracy: 0.9429955821576173 loss: tensor(0.2208)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model_RELU.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "criterion = CrossEntropyLoss\n",
    "\n",
    "comment = f' ConvNet_RELU Adam_0_001 loss = CrossEntropyLoss'\n",
    "\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_Netw_Arch_Exp',comment=comment)\n",
    "for epoch in range(12):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model_RELU(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,labels)\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "     # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "    print(\"epoch:\", epoch, \"Accuracy:\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CNN network no extra layer, additional layer removed\n",
    "class ConvNet_RELU(nn.Module):\n",
    "  def __init__(self, num_classes=6):\n",
    "    super(ConvNet_RELU,self).__init__()\n",
    "\n",
    "    # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "    # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "    self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "    #Shape -> (256,12,150,150)\n",
    "    self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "    #Shape -> (256,12,150,150)\n",
    "    self.relu1 = nn.ReLU()\n",
    "    #Shape -> (256,12,150,150)\n",
    "    self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "    # Reduce the image size by factor of 2\n",
    "    #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "    # 2nd Layer\n",
    "\n",
    "    \n",
    "    self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "    #Shape -> (256,12,75,75)\n",
    "    self.bn2 = nn.BatchNorm2d(num_features=20)\n",
    "    self.relu2 = nn.ReLU()\n",
    "    #Shape -> (256,20,75,75)\n",
    "\n",
    "    \n",
    "    # 3rd Layer\n",
    "\n",
    "    \n",
    "#     self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "#     #Shape -> (256,32,75,75)\n",
    "#     self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "#     #Shape -> (256,32,75,75)\n",
    "#     self.relu3 = nn.ReLU()\n",
    "    #Shape -> (256,32,75,75) \n",
    "    \n",
    "    # 4th layer Added Extra Layer to check the affect on Accuracy&Loss ✔\n",
    "    \n",
    "#     self.conv4 = nn.Conv2d(in_channels=32, out_channels=44, kernel_size=3, stride=1, padding=1)\n",
    "    #Shape -> (256,32,75,75)\n",
    "#     self.bn3 = nn.BatchNorm2d(num_features=44)\n",
    "    #Shape -> (256,32,75,75)\n",
    "#     self.relu4 = nn.ReLU()\n",
    "    #Shape -> (256,32,75,75) \n",
    "\n",
    "    self.fullyConnected = nn.Linear(in_features=20*75*75, out_features=num_classes)\n",
    "\n",
    "# Feed forward function\n",
    "  def forward(self, input):\n",
    "    output = self.conv1(input)\n",
    "    output = self.bn1(output)\n",
    "    output = self.relu1(output)\n",
    "\n",
    "    output = self.pool(output)\n",
    "\n",
    "    output = self.conv2(output)\n",
    "    output = self.bn2(output)\n",
    "    output = self.relu2(output)\n",
    "\n",
    "#     output = self.conv3(output)\n",
    "    \n",
    "#     output = self.relu3(output)\n",
    "    \n",
    "#     output = self.conv4(output)\n",
    "#     output = self.bn3(output)\n",
    "#     output = self.relu4(output)\n",
    "\n",
    "    # the output will be in matric form of shape (256,32,75,75)\n",
    "    # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "    output = output.view(-1,20*75*75)\n",
    "    output = self.fullyConnected(output)\n",
    "\n",
    "    return output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_RELU = ConvNet_RELU(num_classes=6).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1         [-1, 12, 150, 150]             336\n",
      "       BatchNorm2d-2         [-1, 12, 150, 150]              24\n",
      "              ReLU-3         [-1, 12, 150, 150]               0\n",
      "         MaxPool2d-4           [-1, 12, 75, 75]               0\n",
      "            Conv2d-5           [-1, 20, 75, 75]           2,180\n",
      "       BatchNorm2d-6           [-1, 20, 75, 75]              40\n",
      "              ReLU-7           [-1, 20, 75, 75]               0\n",
      "            Linear-8                    [-1, 6]         675,006\n",
      "================================================================\n",
      "Total params: 677,586\n",
      "Trainable params: 677,586\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.26\n",
      "Forward/backward pass size (MB): 9.27\n",
      "Params size (MB): 2.58\n",
      "Estimated Total Size (MB): 12.11\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "summary(model_RELU, (3,150,150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0 Accuracy: 0.5389055151774262 loss: tensor(7.6321)\n",
      "epoch: 1 Accuracy: 0.725238705999715 loss: tensor(1.0242)\n",
      "epoch: 2 Accuracy: 0.8063274903805044 loss: tensor(0.6265)\n",
      "epoch: 3 Accuracy: 0.844449194812598 loss: tensor(0.4784)\n",
      "epoch: 4 Accuracy: 0.8666096622488243 loss: tensor(0.4380)\n",
      "epoch: 5 Accuracy: 0.8916203505771697 loss: tensor(0.3297)\n",
      "epoch: 6 Accuracy: 0.9386489952971355 loss: tensor(0.1906)\n",
      "epoch: 7 Accuracy: 0.9526863331908223 loss: tensor(0.1559)\n",
      "epoch: 8 Accuracy: 0.9616645290009975 loss: tensor(0.1277)\n",
      "epoch: 9 Accuracy: 0.9731366680917771 loss: tensor(0.0948)\n",
      "epoch: 10 Accuracy: 0.956961664529001 loss: tensor(0.1420)\n",
      "epoch: 11 Accuracy: 0.9759868889838963 loss: tensor(0.0826)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model_RELU.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "criterion = CrossEntropyLoss\n",
    "\n",
    "comment = f' ConvNet_RELU Adam_0_001_2_layer loss = CrossEntropyLoss'\n",
    "\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_Netw_Arch_2_layer_Exp',comment=comment)\n",
    "for epoch in range(12):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model_RELU(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,labels)\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "     # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "    print(\"epoch:\", epoch, \"Accuracy:\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy and Loss Plots as from Tensorboard\n",
    "\n",
    "NOTE : **For running the tensorboard in discovery cluster, we need to launch separate instance of tensorboard from Interactive Apps**\n",
    "\n",
    "#### Inference 🔎\n",
    "\n",
    "It is observed that the changes in network architecture by adding another hidden layer and also having just 2 layers in the network, given the no. of epochs is 12 -> the one with lesser number of layers performed slightly better, suggesting that adding layers beyond a point perhaps dosent affect performance.\n",
    "If this model is tested on a test set or validation set, we can know more about its performance and whether the high accuracy and low loss in 2 layer model is actually good or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![compare_netw_arch_accuracy](https://user-images.githubusercontent.com/13203059/145698028-7962a775-1b45-4d24-b4a9-64211b6cde60.png)\n",
    "![compare_netw_arch_loss](https://user-images.githubusercontent.com/13203059/145698030-67c4f5b3-51ea-400c-a1f1-8d08414085b8.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.7 Affect of Different Network Initialization 6️⃣\n",
    "Change the network initialization. How does it affect accuracy? How does it affect how quickly the network plateaus?\n",
    "\n",
    "Various forms of network initialization:\n",
    "\n",
    "- 0\n",
    "- UniformGaussian\n",
    "- Xavier Uniform ✅\n",
    "- Glorot Initialization ->  http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initializationXavier\n",
    "- Uniform\n",
    "- Xavier Gaussian\n",
    "- Kaiming He Uniform ✅\n",
    "\n",
    "Reference - https://www.youtube.com/watch?v=xWQ-p_o0Uik"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CNN network \n",
    "class ConvNet(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "            \n",
    "            super(ConvNet,self).__init__()\n",
    "\n",
    "            # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "            # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "            self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            \n",
    "#             nn.init.xavier_normal_(self.conv1.weight)\n",
    "            \n",
    "            self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.relu1 = nn.ReLU()\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "            # Reduce the image size by factor of 2\n",
    "            #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "            # 2nd Layer\n",
    "\n",
    "\n",
    "            self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,75,75)\n",
    "            \n",
    "#             nn.init.xavier_normal_(self.conv2.weight)\n",
    "            \n",
    "            self.relu2 = nn.ReLU()\n",
    "            #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "            # 3rd Layer\n",
    "\n",
    "\n",
    "            self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            \n",
    "#             nn.init.xavier_normal_(self.conv3.weight)\n",
    "            \n",
    "            self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.relu3 = nn.ReLU()\n",
    "            #Shape -> (256,32,75,75) \n",
    "\n",
    "            self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "            \n",
    "#             self.initialize_weights()\n",
    "\n",
    "      \n",
    "                    \n",
    "      # Feed forward function\n",
    "      def forward(self, input):\n",
    "            \n",
    "            output = self.conv1(input)\n",
    "            output = self.bn1(output)\n",
    "            output = self.relu1(output)\n",
    "\n",
    "            output = self.pool(output)\n",
    "\n",
    "            output = self.conv2(output)\n",
    "            output = self.relu2(output)\n",
    "\n",
    "            output = self.conv3(output)\n",
    "            output = self.bn2(output)\n",
    "            output = self.relu3(output)\n",
    "\n",
    "            # the output will be in matric form of shape (256,32,75,75)\n",
    "            # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "            output = output.view(-1,32*75*75)\n",
    "            output = self.fullyConnected(output)\n",
    "\n",
    "            return output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with Kaiming uniform weight init\n",
    "def initialize_weights(m):\n",
    "        if isinstance(m, nn.Conv2d):\n",
    "                \n",
    "            nn.init.kaiming_uniform_(m.weight.data,nonlinearity='relu')\n",
    "            if m.bias is not None:\n",
    "                nn.init.constant_(m.bias.data, 0)\n",
    "                    \n",
    "        elif isinstance(m, nn.BatchNorm2d):\n",
    "            nn.init.constant_(m.weight.data, 1)\n",
    "            nn.init.constant_(m.bias.data, 0)\n",
    "            \n",
    "        elif isinstance(m, nn.Linear):\n",
    "            nn.init.kaiming_uniform_(m.weight.data)\n",
    "            nn.init.constant_(m.bias.data, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ConvNet(\n",
       "  (conv1): Conv2d(3, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn1): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu1): ReLU()\n",
       "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "  (conv2): Conv2d(12, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (relu2): ReLU()\n",
       "  (conv3): Conv2d(20, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu3): ReLU()\n",
       "  (fullyConnected): Linear(in_features=180000, out_features=6, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ConvNet(num_classes=6).to(device)\n",
    "model.apply(initialize_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0 Accuracy: 0.5225880005700442 loss: tensor(11.5776)\n",
      "epoch: 1 Accuracy: 0.7240273621205643 loss: tensor(1.4212)\n",
      "epoch: 2 Accuracy: 0.7963517172580875 loss: tensor(0.8307)\n",
      "epoch: 3 Accuracy: 0.834188399600969 loss: tensor(0.6760)\n",
      "epoch: 4 Accuracy: 0.8694598831409435 loss: tensor(0.5562)\n",
      "epoch: 5 Accuracy: 0.920478837109876 loss: tensor(0.2864)\n",
      "epoch: 6 Accuracy: 0.9488385349864614 loss: tensor(0.1777)\n",
      "epoch: 7 Accuracy: 0.9281031779962947 loss: tensor(0.2400)\n",
      "epoch: 8 Accuracy: 0.9383639732079236 loss: tensor(0.2105)\n",
      "epoch: 9 Accuracy: 0.926891834117144 loss: tensor(0.2979)\n",
      "epoch: 10 Accuracy: 0.9432093487245261 loss: tensor(0.1978)\n",
      "epoch: 11 Accuracy: 0.9677212483967508 loss: tensor(0.1071)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training with Kaiming wt init\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "criterion = CrossEntropyLoss\n",
    "\n",
    "comment = f' ConvNet_RELU Adam_0_001_Wt_init_kaiming_uniform_ loss = CrossEntropyLoss'\n",
    "\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_Wt_init_kaiming_uniform__Exp',comment=comment)\n",
    "for epoch in range(12):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,labels)\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "     # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "    print(\"epoch:\", epoch, \"Accuracy:\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with Xavier uniform weight init\n",
    "def initialize_weights(m):\n",
    "        if isinstance(m, nn.Conv2d):\n",
    "                \n",
    "            nn.init.xavier_uniform_(m.weight.data)\n",
    "            if m.bias is not None:\n",
    "                nn.init.constant_(m.bias.data, 0)\n",
    "                    \n",
    "        elif isinstance(m, nn.BatchNorm2d):\n",
    "            nn.init.constant_(m.weight.data, 1)\n",
    "            nn.init.constant_(m.bias.data, 0)\n",
    "            \n",
    "        elif isinstance(m, nn.Linear):\n",
    "            nn.init.xavier_uniform_(m.weight.data)\n",
    "            nn.init.constant_(m.bias.data, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ConvNet(\n",
       "  (conv1): Conv2d(3, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn1): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu1): ReLU()\n",
       "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "  (conv2): Conv2d(12, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (relu2): ReLU()\n",
       "  (conv3): Conv2d(20, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu3): ReLU()\n",
       "  (fullyConnected): Linear(in_features=180000, out_features=6, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ConvNet(num_classes=6).to(device)\n",
    "model.apply(initialize_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0 Accuracy: 0.5323500071255522 loss: tensor(8.2226)\n",
      "epoch: 1 Accuracy: 0.7506056719395753 loss: tensor(1.1656)\n",
      "epoch: 2 Accuracy: 0.788086076670942 loss: tensor(0.9962)\n",
      "epoch: 3 Accuracy: 0.8656833404588855 loss: tensor(0.5359)\n",
      "epoch: 4 Accuracy: 0.8932592275901382 loss: tensor(0.4032)\n",
      "epoch: 5 Accuracy: 0.9226165027789653 loss: tensor(0.2834)\n",
      "epoch: 6 Accuracy: 0.9422830269345874 loss: tensor(0.2018)\n",
      "epoch: 7 Accuracy: 0.9588855636311814 loss: tensor(0.1436)\n",
      "epoch: 8 Accuracy: 0.9484110018526436 loss: tensor(0.1930)\n",
      "epoch: 9 Accuracy: 0.950334900954824 loss: tensor(0.1743)\n",
      "epoch: 10 Accuracy: 0.9551090209491235 loss: tensor(0.1629)\n",
      "epoch: 11 Accuracy: 0.9611657403448767 loss: tensor(0.1377)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training with Xavier wt init\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "criterion = CrossEntropyLoss\n",
    "\n",
    "comment = f' ConvNet_RELU Adam_0_001_Wt_init_xavier_uniform_ loss = CrossEntropyLoss'\n",
    "\n",
    "tb = SummaryWriter(log_dir = 'runs/Model_RELU_Wt_init_xavier_uniform__Exp',comment=comment)\n",
    "for epoch in range(12):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,labels)\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Accuracy\", train_accuracy/train_count, epoch)\n",
    "\n",
    "     # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "    print(\"epoch:\", epoch, \"Accuracy:\", train_accuracy/train_count,\"loss:\",train_loss/train_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy and Loss Plots as from Tensorboard\n",
    "\n",
    "NOTE : **For running the tensorboard in discovery cluster, we need to launch separate instance of tensorboard from Interactive Apps**\n",
    "\n",
    "#### Inference 🔎\n",
    "\n",
    "It is observed that the changes in network weight initialization, given the no. of epochs is 12 -> both the techniques Xavier uniform and Kaiming He Uniform gave similar results in accuracy. Although the loss plot shows that xavier weights initialization brought the initial loss very much lower than Kaiming He.\n",
    "If this model is tested on a test set or validation set, we can know more about its performance and whether the high accuracy and low loss in both of these models is actually good or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![wt_init_accuracy](https://user-images.githubusercontent.com/13203059/145698037-51fec8b4-5f10-4c6a-bb03-9575b3dadbcf.png)\n",
    "![wt_init_loss](https://user-images.githubusercontent.com/13203059/145698038-ff20c267-9035-4e20-85c7-f179069bb44e.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A7EE5R4nvK02"
   },
   "source": [
    "### 3.8 Training the CNN with the dataset & model saving 📩\n",
    "\n",
    "Training the CNN with the dataset and saving the best model based on best test accuracy\n",
    "\n",
    "Based on the above permutations of activation functions/loss/Epochs/gradient estimate/network arch and weight init., We can move ahead with using a good model choice among these and save the model for Validation data preditions(unlabelled data).\n",
    "\n",
    "Model to use - **Model_RELU_Wt_init_kaiming_uniform_Exp**\n",
    "\n",
    "- Activation Fn -> ReLU ✔\n",
    "- Cost fn -> CrossEntropy Loss ✔\n",
    "- Gradient Est. -> Adam with lr=0.01 ✔\n",
    "- Network Arch -> 3 layer ReLU CNN ✔\n",
    "- Weight Init -> Kaiming he Uniform ✔"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CNN network \n",
    "class ConvNet(nn.Module):\n",
    "      def __init__(self, num_classes=6):\n",
    "            \n",
    "            super(ConvNet,self).__init__()\n",
    "\n",
    "            # Input Shape = (256,3,150,150) -> (Batch size, no. of channels, 150, 150 -> Image dim)\n",
    "            # Stride means scan X pixels, padding is how many pixels to keep around before passing -> 3x3\n",
    "            self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            \n",
    "#             nn.init.xavier_normal_(self.conv1.weight)\n",
    "            \n",
    "            self.bn1 = nn.BatchNorm2d(num_features=12)\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.relu1 = nn.ReLU()\n",
    "            #Shape -> (256,12,150,150)\n",
    "            self.pool = nn.MaxPool2d(kernel_size=2)\n",
    "            # Reduce the image size by factor of 2\n",
    "            #Shape -> (256,12, 75,75)\n",
    "\n",
    "\n",
    "            # 2nd Layer\n",
    "\n",
    "\n",
    "            self.conv2 = nn.Conv2d(in_channels=12, out_channels=20, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,12,75,75)\n",
    "            \n",
    "#             nn.init.xavier_normal_(self.conv2.weight)\n",
    "            \n",
    "            self.relu2 = nn.ReLU()\n",
    "            #Shape -> (256,20,75,75)\n",
    "\n",
    "\n",
    "            # 3rd Layer\n",
    "\n",
    "\n",
    "            self.conv3 = nn.Conv2d(in_channels=20, out_channels=32, kernel_size=3, stride=1, padding=1)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            \n",
    "#             nn.init.xavier_normal_(self.conv3.weight)\n",
    "            \n",
    "            self.bn2 = nn.BatchNorm2d(num_features=32)\n",
    "            #Shape -> (256,32,75,75)\n",
    "            self.relu3 = nn.ReLU()\n",
    "            #Shape -> (256,32,75,75) \n",
    "\n",
    "            self.fullyConnected = nn.Linear(in_features=32*75*75, out_features=num_classes)\n",
    "            \n",
    "#             self.initialize_weights()\n",
    "\n",
    "      \n",
    "                    \n",
    "      # Feed forward function\n",
    "      def forward(self, input):\n",
    "            \n",
    "            output = self.conv1(input)\n",
    "            output = self.bn1(output)\n",
    "            output = self.relu1(output)\n",
    "\n",
    "            output = self.pool(output)\n",
    "\n",
    "            output = self.conv2(output)\n",
    "            output = self.relu2(output)\n",
    "\n",
    "            output = self.conv3(output)\n",
    "            output = self.bn2(output)\n",
    "            output = self.relu3(output)\n",
    "\n",
    "            # the output will be in matric form of shape (256,32,75,75)\n",
    "            # in order to feed it to fully connected layer output needs to be reshaped\n",
    "\n",
    "            output = output.view(-1,32*75*75)\n",
    "            output = self.fullyConnected(output)\n",
    "\n",
    "            return output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with Kaiming uniform weight init\n",
    "def initialize_weights(m):\n",
    "        if isinstance(m, nn.Conv2d):\n",
    "                \n",
    "            nn.init.kaiming_uniform_(m.weight.data,nonlinearity='relu')\n",
    "            if m.bias is not None:\n",
    "                nn.init.constant_(m.bias.data, 0)\n",
    "                    \n",
    "        elif isinstance(m, nn.BatchNorm2d):\n",
    "            nn.init.constant_(m.weight.data, 1)\n",
    "            nn.init.constant_(m.bias.data, 0)\n",
    "            \n",
    "        elif isinstance(m, nn.Linear):\n",
    "            nn.init.kaiming_uniform_(m.weight.data)\n",
    "            nn.init.constant_(m.bias.data, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ConvNet(\n",
       "  (conv1): Conv2d(3, 12, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn1): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu1): ReLU()\n",
       "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
       "  (conv2): Conv2d(12, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (relu2): ReLU()\n",
       "  (conv3): Conv2d(20, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (bn2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu3): ReLU()\n",
       "  (fullyConnected): Linear(in_features=180000, out_features=6, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ConvNet(num_classes=6).to(device)\n",
    "model.apply(initialize_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 0 Train Accuracy: 0.5091207068547813 Train loss: tensor(9.2487) Test Accuracy: 0.6 Test loss: tensor(1.9792)\n",
      "epoch: 1 Train Accuracy: 0.7108450904945133 Train loss: tensor(1.3979) Test Accuracy: 0.6966666666666667 Test loss: tensor(1.3644)\n",
      "epoch: 2 Train Accuracy: 0.7556648140230868 Train loss: tensor(1.1911) Test Accuracy: 0.7546666666666667 Test loss: tensor(1.1337)\n",
      "epoch: 3 Train Accuracy: 0.8417414849650848 Train loss: tensor(0.6241) Test Accuracy: 0.826 Test loss: tensor(0.6616)\n",
      "epoch: 4 Train Accuracy: 0.8215049166310389 Train loss: tensor(0.7600) Test Accuracy: 0.8553333333333333 Test loss: tensor(0.5674)\n",
      "epoch: 5 Train Accuracy: 0.8744477697021519 Train loss: tensor(0.4768) Test Accuracy: 0.9286666666666666 Test loss: tensor(0.2847)\n",
      "epoch: 6 Train Accuracy: 0.9393615505201653 Train loss: tensor(0.2061) Test Accuracy: 0.9493333333333334 Test loss: tensor(0.1906)\n",
      "epoch: 7 Train Accuracy: 0.9181274048738777 Train loss: tensor(0.2789) Test Accuracy: 0.9606666666666667 Test loss: tensor(0.1394)\n",
      "epoch: 8 Train Accuracy: 0.9540401881145789 Train loss: tensor(0.1481) Test Accuracy: 0.96 Test loss: tensor(0.1283)\n",
      "epoch: 9 Train Accuracy: 0.9419980048453755 Train loss: tensor(0.2134) Test Accuracy: 0.9703333333333334 Test loss: tensor(0.0931)\n",
      "epoch: 10 Train Accuracy: 0.976913210773835 Train loss: tensor(0.0870) Test Accuracy: 0.9696666666666667 Test loss: tensor(0.1152)\n",
      "epoch: 11 Train Accuracy: 0.9818298418127405 Train loss: tensor(0.0671) Test Accuracy: 0.9863333333333333 Test loss: tensor(0.0643)\n",
      "___________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Training with Kaiming uniform weight init\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "Adam_RELU = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)\n",
    "optimizer = Adam_RELU\n",
    "criterion = CrossEntropyLoss\n",
    "\n",
    "# comment = f' ConvNet_RELU Adam_0_001_Wt_init_xavier_uniform_ loss = CrossEntropyLoss'\n",
    "\n",
    "writer = SummaryWriter('runs/Image-classification')\n",
    "# train_loader = torch.utils.data.DataLoader(train_set,batch_size = 256, shuffle = True)\n",
    "tb = SummaryWriter('runs/Final_Model_Train_Test_acc_loss')\n",
    "for epoch in range(12):\n",
    "    total_loss = 0\n",
    "    total_correct = 0\n",
    "    train_accuracy=0.0\n",
    "    train_loss=0.0\n",
    "    model.train()\n",
    "    for images, labels in train_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,labels)\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        train_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        train_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = train_loss/train_count\n",
    "    tb.add_scalar(\"Train Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Train Accuracy\", train_accuracy/train_count, epoch)\n",
    "    tb.add_histogram(\"conv1.bias\", model.conv1.bias, epoch)\n",
    "    tb.add_histogram(\"conv1.weight\", model.conv1.weight, epoch)\n",
    "    tb.add_histogram(\"conv2.bias\", model.conv2.bias, epoch)\n",
    "    tb.add_histogram(\"conv2.weight\", model.conv2.weight, epoch)\n",
    "    tb.add_histogram(\"conv3.bias\", model.conv3.bias, epoch)\n",
    "    tb.add_histogram(\"conv3.weight\", model.conv3.weight, epoch)\n",
    "    writer.add_figure(\"Confusion matrix - Training Data\", createConfusionMatrix(train_loader), epoch)\n",
    "     # print(\"batch_size:\",batch_size, \"lr:\",lr,\"shuffle:\",shuffle)\n",
    "        \n",
    "    # Evaluation on testing dataset\n",
    "        \n",
    "    test_accuracy=0.0\n",
    "    test_loss=0.0\n",
    "    model.eval()\n",
    "    for images, labels in test_loader:\n",
    "        images, labels = images.to(device), labels.to(device)\n",
    "\n",
    "\n",
    "        if torch.cuda.is_available():\n",
    "          images=Variable(images.cuda())\n",
    "          labels=Variable(labels.cuda())\n",
    "        # at start of new batch the gradient is made 0    \n",
    "        optimizer.zero_grad()\n",
    "        # This gives us the prediction\n",
    "        outputs=model(images)\n",
    "        # cal loss using pred and actual value\n",
    "        lossEpoch=criterion(outputs,labels)\n",
    "        \n",
    "         #backpropogation\n",
    "        lossEpoch.backward()\n",
    "        # Update weights\n",
    "        optimizer.step()\n",
    "\n",
    "        # Multiply loss with image size\n",
    "        test_loss+= lossEpoch.cpu().data*images.size(0)\n",
    "        # The outputs are energies for the 6 classes. The higher the energy for a class, \n",
    "        # the more the network thinks that the image is of the particular class.\n",
    "        # So, let’s get the index of the highest energy:\n",
    "        _,prediction=torch.max(outputs.data,1)\n",
    "\n",
    "        test_accuracy+=int(torch.sum(prediction==labels.data))\n",
    "\n",
    "\n",
    "\n",
    "    total_loss_in_epoch = test_loss/test_count\n",
    "    tb.add_scalar(\"Test Loss\", total_loss_in_epoch.cpu().data.numpy(), epoch)\n",
    "         #tb.add_scalar(\"Correct\", total_correct, epoch)\n",
    "    tb.add_scalar(\"Test Accuracy\", test_accuracy/test_count, epoch)    \n",
    "    writer.add_figure(\"Confusion matrix - Test Data\", createConfusionMatrix(test_loader), epoch)    \n",
    "        \n",
    "    print(\"epoch:\", epoch, \"Train Accuracy:\", train_accuracy/train_count,\"Train loss:\",train_loss/train_count, \"Test Accuracy:\", test_accuracy/test_count,\"Test loss:\",test_loss/test_count)\n",
    "print(\"___________________________________________________________________\")\n",
    "\n",
    "\n",
    "\n",
    "tb.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy and Loss Plots as from Tensorboard\n",
    "\n",
    "NOTE : **For running the tensorboard in discovery cluster, we need to launch separate instance of tensorboard from Interactive Apps**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Accuracy and Training Loss\n",
    "![final_train_acc](https://user-images.githubusercontent.com/13203059/145698054-d2fb66e8-0fff-405d-8b3b-da2d25b703e5.png)\n",
    "![final_train_loss](https://user-images.githubusercontent.com/13203059/145698055-b535b4a2-27f9-4748-8729-1712d7d507fd.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test Accuracy and Test Loss\n",
    "![final_test_acc](https://user-images.githubusercontent.com/13203059/145698066-b02c559a-3cfe-43ab-a801-072a72295220.png)\n",
    "![final_test_loss](https://user-images.githubusercontent.com/13203059/145698067-c9fc89ae-2397-42d0-96e5-5a3d58d9a988.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save the best model\n",
    "torch.save(model.state_dict(),'best_checkpoint.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion 📖\n",
    "\n",
    "The initial Exploratory Data analysis of the dataset revealed a balanced ratio of all 6 image classes, both in train and test dataset. In approaching the Image classification, various parameters of the CNN model were changed in order to see the changes in accuracy and loss. \n",
    "\n",
    "The following affects were observed and ploted in Tensorbaord - \n",
    "\n",
    "- Affect of Activation Function - RELU and ELU\n",
    "- Affect of Cost functions - CrossEntropy and Negative Log Loss(NLL)\n",
    "- Affect of Epochs - Ranging from 0 to 15\n",
    "- Affect of Gradient estimation and their corresponding learning Rates - Adam and SGD\n",
    "- Affect of Network Architecture - Adding and remove Hidden layers from the CNN network\n",
    "- Affect of Network Weight Initializations - Using Kaiming He uniform and Xavier Uniform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Licensing & Citation 📰\n",
    "\n",
    "Copyright 2021 Kshitij Zutshi\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "  http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "#### Citation Reference -\n",
    "\n",
    "\n",
    "https://pytorch.org/vision/stable/models.html\n",
    "\n",
    "Image classification using CNN from scratch in Pytorch Part I and II\n",
    "\n",
    "https://www.youtube.com/watch?v=9OHlgDjaE2I\n",
    "\n",
    "https://www.youtube.com/watch?v=zwHSQrNVNNE\n",
    "\n",
    "Image Classification with Convolutional Neural Networks | Deep Learning with PyTorch: Zero to GANs\n",
    "\n",
    "https://www.youtube.com/watch?v=d9QHNkD_Pos&t=234s\n",
    "\n",
    "https://thevatsalsaglani.medium.com/multi-class-image-classification-using-cnn-over-pytorch-and-the-basics-of-cnn-fdf425a11dc0\n",
    "\n",
    "TENSORBOARD PYTORCH Reference - \n",
    "\n",
    "https://towardsdatascience.com/a-complete-guide-to-using-tensorboard-with-pytorch-53cb2301e8c3\n",
    "https://github.com/ajinkya98/TensorBoard_PyTorch/blob/master/model.py\n",
    "https://towardsdatascience.com/pytorch-performance-analysis-with-tensorboard-7c61f91071aa\n",
    "https://cnvrg.io/tensorboard-guide/"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Assignment-3-Deep-Learning_CNN.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "pytorch_kz_env",
   "language": "python",
   "name": "pytorch_kz_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  },
  "toc-showcode": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}