04.07. Binary classification evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Оценка бинарной классификации\n",
    "Бинарная классификация одна из наиболее важных и часто встречаемых классов задач. Причем не только модели машинного обучения но и другие тесты также могут выдавать результат в виде бинарных классов. Например, медицинский тест на наличие заболевания (больной/здоровый), радиолокатор для определения вражеских объектов (свой/чужой) также являются бинарными классификаторами.\n",
    "\n",
    "По этой причине важно оценить модель правильно и иногда несколькими разными способами. В этом материале мы рассмотрим несколько таких методов оценки. Для этого мы сначала обучим модель логистической регрессии, чтобы потом её оценить разными методами. В качестве такой задачи будем использовать набор данных <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html\">breast cancer</a> (рак груди) доступный в sklearn. Данный набор содержит данные по 569 пациентам, у которых выявлена опухоль груди, из которых 212 являются злокачественными (malignant) и 357 доброкачественными (benign). Данные содержат 30 признаков. Необходимо создать модель бинарной классификации, которая будет различать злокачественную опухоль от доброкачественной. Злокачественная опухоль соответствует отрицательным (0), а доброкачественна к положительным (1) классам.\n",
    "\n",
    "Для начала симпортируем все необходимые модули."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:09.828238Z",
     "start_time": "2018-02-02T16:34:08.905738Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy import stats\n",
    "\n",
    "# use seaborn plotting defaults\n",
    "import seaborn as sns; sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "С помощью функции `datasets.load_breast_cancer` загрузим набор данных и посмотрим их количество."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:09.958239Z",
     "start_time": "2018-02-02T16:34:09.830238Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X shape                : (569, 30)\n",
      "y shape                : (569,)\n",
      "target names           : ['malignant' 'benign']\n",
      "positve (benign) cancer: 357\n"
     ]
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "breast_cancer = datasets.load_breast_cancer()\n",
    "X = breast_cancer.data\n",
    "y = breast_cancer.target\n",
    "print('X shape                :', X.shape)\n",
    "print('y shape                :', y.shape)\n",
    "print('target names           :', breast_cancer.target_names)\n",
    "print('positve (benign) cancer:', np.sum(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Как обычно разобьем данные на тренировочный и тестовый наборы в пропорции 75% на 25%. При этом зафиксируем генератор случайных чисел, чтобы получать воспроизводимые результаты."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:09.966739Z",
     "start_time": "2018-02-02T16:34:09.959738Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "random_state = np.random.RandomState(0)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.25, random_state=random_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь можно создать модель логистической регрессии и обучить её."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:09.993239Z",
     "start_time": "2018-02-02T16:34:09.968739Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2',\n",
       "          random_state=<mtrand.RandomState object at 0x0000025A9FD26B88>,\n",
       "          solver='liblinear', tol=0.0001, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier = LogisticRegression(random_state=random_state)\n",
    "classifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "С помощью этой модели можно предсказать тип опухоли"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.002239Z",
     "start_time": "2018-02-02T16:34:09.994739Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1,\n",
       "       1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0,\n",
       "       1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0,\n",
       "       1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0,\n",
       "       0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0,\n",
       "       0, 1, 1, 1, 0])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_pred = classifier.predict(X_test)\n",
    "y_test_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Введем несколько определений, которые будем использовать дальше\n",
    "\n",
    "|      Основные показатели                                           |\n",
    "|----------------|---------------------------------------------------|\n",
    "|  $T_p$         | True positive - истинно положительные результаты  |\n",
    "|  $T_n$         | True negative - истинно отрицательные результаты  |\n",
    "|  $F_p$         | False positive - ложно положительные результаты   |\n",
    "|  $F_n$         | False negative - ложно отрицательные результаты   |\n",
    "\n",
    "\n",
    "## Матрица заблуждений (confusion matrix)\n",
    "Для начала будем оценивать результаты модели классификации с помощью так называемой матрицы заблуждений (confusion matrix). Матрица заблуждений показывает количество верных и ошибочных предсказаний для каждого класса. В задаче бинарной классификации матрица имеет размер 2x2. В sklearn для получения этой матрицы используется функция `confusion_matrix` из модуля `sklearn.metrics`. Для модели бинарной классификации она возвращает матрицу, у которой ячейка [0, 0] содержит количество истинно отрицательных (true negative, $T_n$), ячейка [1, 1] содержит количество истинно положительных (true positive, $T_p$) результатов. Ячейки [1, 0] и [0, 1] содержат количество ложноотрицательных (false negative, $F_n$) и ложноположительных (false positive, $F_p$) результатов соответственно."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.014739Z",
     "start_time": "2018-02-02T16:34:10.004239Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[52  1]\n",
      " [ 5 85]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cmat = confusion_matrix(y_test, y_test_pred)\n",
    "print(cmat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Матрица заблуждений лучше всего визуализируется с помощью тепловой карты (heatmap). Для создания тепловой карты воспользуемся функцией `heatmap` из библиотеки `seaborn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.130239Z",
     "start_time": "2018-02-02T16:34:10.016739Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(90, 86)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAAFJCAYAAAAWit+oAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFEFJREFUeJzt3Xt0lIWZx/HfDAkECEouIJdESSAhZQtbKERYLq6Wk7aR\nUBEKOdXYupUW3RjWSCUgSLAiWkCLAsvNCoTjEmJDLZc9Cr24xIpQuVpaLuEiIcFEQgiBEHKZ/cND\ndjnC4wgM70zy/ZzDOcm887484cz58l4m77g8Ho9HAICrcjs9AAD4MyIJAAYiCQAGIgkABiIJAAYi\nCQCGIF9u/M0fz/Hl5tEMjZs9xukR0AS16RJzzWXsSQKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKA\ngUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCB\nSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFI\nAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgC\ngIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKAgUgCgIFIAoCBSAKA\ngUgCgIFIAoCBSAKAgUgCgIFIAoAhyOkBmqMBqf+qbok9VVN1UZJUeapcW5f9twY+MlwdYjpJbpfK\nCku0bdUW1dfWOTwtAtG8RUu15f2tuq1dO0lSt+govTxjqsNTBSYi6YCOcV31/qL1Kj1c3PhYv9FD\n5G7h1u+mr5BLLg2bcL/6jLhbu9Z94OCkCFR7/rZfs6dP0be+2cvpUQIekbzF3EEtFH5nR33z+wPU\nrmN7VX5Woe1v/VGnDhSpquys5JE88uj08VKFdY1welwEoEuXLunAoULlrH1bL75aouiuXTTp33+u\nznd0dHq0gOTVOclFixZd8f28efN8Mkxz0KZ9qE79/VP9Ne9/9M70lSorLNZ3/mOUij85psrPzkiS\n2kbcpn9K+raObj/g8LQIRGWnyzWg37f05GOPKnf5IvXplaCnps2Ux+NxerSA5PIY/3J5eXl6++23\nVVhYqB49ekiS6uvrVVdXp3Xr1n3lxt/88ZybN2kT9tDiDL0zbaWqPj+riG536L6MB3Tgj7u1d8NH\nTo/md8bNHuP0CAHH4/Fo6IjRyl2+SF07d3J6HL/UpkvMNZeZh9s/+MEPNGjQIC1ZskQTJkyQJLnd\nbkVEcBh4vcKiOyg8uoMK/7K/8TGXXGqor1fM3Qka9Mhwbcv5g45s+7uDUyKQHSw8ooOFRzQiaXjj\nYx6PR0FBLRycKnCZh9stW7ZUVFSUZs6cqdOnT6u4uFhFRUXas2fPrZqvyfE0eHT3w99RaOTtkqSE\n+76l8qIydejeRXc/fJ/enZNHIHFD3G63fvX6Yp0sOSVJyntng+JiY3RHhw4OTxaYvLpwk5GRodOn\nT6tz586SJJfLpQEDBvh0sKaq4uTn2rb6Dxr+1Ci53G6dLz+n9xet1/cmj5NLLg3+t+81Prf00Elt\ny9ni4LQIRD1iumlyxuOaOHWGGhoa1LFDpGZPz3J6rIBlnpO8LDU1VWvWrPnaG+ecJG42zknCF6xz\nkl5d3Y6JidFnn3120wYCgEDh1eH2zp07de+99yo8PLzxsYKCAp8NBQD+wqtIvvvuu76eAwD8kleR\n3L17t/Lz81VbWytJKi0t1RtvvOHTwQDAH3h1TjI7O1uJiYmqqqpSly5d1L59e1/PBQB+watIhoWF\nacSIEQoNDdWTTz7JRRwAzYZXkXS73Tp06JCqq6t15MgRnT171tdzAYBf8CqSWVlZOnTokNLS0jRp\n0iSNHj3a13MBgF/w6sJNXFyc4uLiJEn5+fk+HQgA/IlXkVy8eLGWL1+ukJCQxsd4nySA5sCrSG7a\ntElbt25V69atfT0PAPgVr85JRkVFXbEXCQDNhVd7krW1tUpJSVF8fLykL+4CxN3JATQHXkVy/Pjx\nvp4DAPySV5Hcvn37Fd8HBwerpKREycnJCg4O9slgAOAPvDoneeDAAR07dkyRkZE6efKkPvzwQxUU\nFGjqVD7HF0DT5lUkKysrNXfuXKWmpmr27Nlyu92aM2eOioqKfD0fADjKq0ieO3dO5eXlkqQzZ87o\n3Llzqq2t1cWLF306HAA4zatzkk8++aTGjh2r0NBQXbhwQdOmTdObb76pMWO4lT6Aps2rSN577726\n5557VF5eroiICLlcLg0bNszXswGA48xIPv/883ruuec0btw4uVyuK5ZdzweDAUCgMSP5xBNPSJJe\neeWVWzIMAPgbM5KrVq265rLMzMybPgwA+BszkjEx1/4sWgBoDsxIjho1SpJUV1enffv2qa6uTh6P\nR6WlpbdkOABwmldXt9PT01VbW6vS0lLV19erY8eOGjFihK9nAwDHefVm8jNnzuiNN95Qnz59lJ+f\nr5qaGl/PBQB+watIXr6XZHV1NfeVBNCseBXJpKQkLVy4UAkJCRo3bpxatWrl67kAwC+Y5ySnTJnS\n+HVDQ4Pcbrc6duyooCCvTmUCQMAza/fJJ5/o4sWLGjlypPr27SuPx3Or5gIAv2Aebq9fv14LFy5U\nTU2Nli5dqt27d+vOO+/U0KFDb9V8AOCorzxujo+P16RJkyRJO3bs0Lx583Tq1CmtXbvW58MBgNO8\nOrlYVVWlzZs3a8OGDaqurtbIkSN9PRcA+AUzkps2bdKmTZtUXFyspKQkzZw5U1FRUbdqNgBwnBnJ\nzMxMxcbGKiEhQQcPHtSrr77auIyPlAXQHFz3XYAAoDkwI5mYmHir5gAAv+TVb9wAQHNFJAHAQCQB\nwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHA\nQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBA\nJAHAQCQBwODyeDweX238UuVpX20azVT/3g86PQKaoL3H37/mMvYkAcBAJAHAQCQBwEAkAcBAJAHA\nQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBA\nJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAk\nAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQB\nwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHA\nQCQBwEAkAcBAJAHAQCQBwEAkAcBAJAHAEOT0AJA8Ho+mzZyluO6x+knaj5weBwHovu8O1RNPPaqG\nhgZVVlYp+5lfqejTYv155zsqPVXW+LwVS9do0++2ODhp4CGSDjty9Jhm/Wqu9u77m+K6xzo9DgJQ\nq1YtNfvXz2rM936qE8dP6uGf/lBZMzM095cLVXn2nMYmP+b0iAGNSDrsv/J+qwdS7lenOzo5PQoC\nlLtFC8nlUmi7tpKkNm1b61LNJf3zt7+phoYGLV/za7Vvf5s2b3pfyxbkqKGhweGJAwuRdNizzzwt\nSdq2/WOHJ0Ggqr5QrRemvqKc/IWqqKhUC7dbj4xOV+Kgvvpw61/1yov/qZCQVlrw5ks6X3Veq3/z\nttMjBxSvIllSUqINGzaopqam8bH09HSfDQXAe3E9Y/XziY/ogeE/VtGnxfrRT0brlcXP64ff/2nj\nc2ov1Spn+Vr96CejieTX5NXV7YkTJ6qqqkqRkZGNfwD4h3+5Z4B2//UTFX1aLElas2qdevSM0YhR\nSYpL+L/z3C6XS3V1dU6NGbC82pNs27atnnrqKV/PAuA6/P2Tg0p9ZJTCI8NU/vkZ3ffdITp5okQ9\nesZo+PeHKXPCcwoODlLqI6O4sn0dvIpkXFycNm7cqG984xtyuVySpJiYGJ8OBsA72/+ySyuWrtFv\ncuer9lKtzp49p4mPPauTJ0o05Zf/od++96aCgoK0eeOf9ds1G5weN+C4PB6P56uelJaWduVKLpdW\nrVr1lRu/VHn6+icDrqJ/7wedHgFN0N7j719zmVd7kjk5OTdtGAAIJF5FcujQoSovL1dYWJgqKirU\nsmVLRUZGasaMGRo8eLCvZwQAx3h1dXvAgAFav369CgoKtGnTJg0fPlzLli3T/PnzfT0fADjKq0ie\nOnVKsbFfvJXgzjvvVElJie666y61aNHCp8MBgNO8Otzu0KGD5s6dq759+2rXrl2KjIzUBx98oODg\nYF/PBwCO8urqdk1NjXJzc1VYWKj4+HiNGTNG+/fvV3R0tPnGcq5u42bj6jZ84bqvbu/bt0+9e/fW\njh07FBsb23jIvWPHDg0ZMuTmTgkAfsiM5IcffqjevXtr48aNX1pGJAE0B14dbl8vDrdxs3G4DV+4\n4TeTL1myRMuWLVNISEjjYwUFBTc+GQD4Oa8iuXHjRm3dulWtW7f29TwA4Fe8ep9kVFTUFXuRANBc\neLUnWVtbq5SUFMXHxzfeBWjevHk+HQwA/IFXkRw/fryv5wAAv+TV4XavXr30wQcfaN26daqoqNAd\nd9zh67kAwC94FcmpU6cqOjpax48fV2RkpJ599llfzwUAfsGrSFZUVGjMmDEKCgpSv379+EhKAM2G\nV5GUpMLCQklf3BGIu/8AaC68iuS0adM0depU7d69WxkZGZoyZYqv5wIAv2BG8h//+IcmTJiglStX\nKiMjQyEhITp+/LgOHjx4q+YDAEeZkczOzlZaWpoGDx6s9PR0rV27Vps3b9Zbb711q+YDAEeZ75MM\nDg5u/AybVatWqVu3bpKkNm3a+HwwAPAH5p7k5d+ukaSWLVs2fs3VbQDNhbknefjwYT399NPyeDxX\nfH35SjcANHXm/SS3b99+zRUTExO/cuPcTxI3G/eThC9c9/0kvQkhADRlXr+ZHACaIyIJAAYiCQAG\nIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYi\nCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJ\nAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGIgkABiIJAAYiCQAGl8fj\n8Tg9BAD4K/YkAcBAJAHAQCQBwEAkAcBAJAHAQCQBwEAkr8NHH32knj17auPGjVc8npKSoqysrKuu\nk5+fr7lz56qsrEzZ2dk+na+mpkZ5eXk+/Ttw63z00UcaNGiQ0tLS9PDDD2vs2LHav3//19rG0qVL\ntXfvXh9N2LQRyesUGxt7RSQPHDig6urqr1yvQ4cOPo9kWVkZkWxiBg4cqJycHK1evVoZGRmaP3/+\n11r/Zz/7mfr06eOj6Zq2IKcHCFQJCQk6evSozp07p3bt2un3v/+9UlJSVFJSotWrV+u9995TdXW1\nwsLCtGDBgsb1ioqKlJmZqbVr1+pPf/qTXnvtNYWGhur2229Xz549lZiYqGXLlik4OFhFRUVKTk7W\n448/roMHD+qll15SfX29zpw5o+zsbPXr109JSUnq16+fjh49qoiICL3++utavHixDh8+rAULFig9\nPd3BfyX4QmVlpcLDw3XgwAG98MILkqT27dvrxRdf1P79+6/6+snKylJycrISExP1zDPPqLS0VJ07\nd9aOHTtUUFCgtLQ0JSQk6NChQ6qqqtL8+fPVtWtXh39S/8Ce5A1ISkrSe++9J4/Ho71796pv375q\naGhQRUWFVqxYoby8PNXX12vfvn1fWre+vl4vvPCCli1bppycHLVq1apxWXFxsV5//XXl5uZq+fLl\nkqTDhw9r8uTJWrlypcaPH6/8/HxJ0okTJzRx4kTl5uaqvLxc+/bt04QJE9SjRw8C2YRs27ZNaWlp\nGjdunKZMmaL7779f06dP14wZM5STk6Nhw4Y1vlau9vq5LDc3V1FRUVqzZo3S09N1+vTpxmV9+vTR\nihUrNHjw4C+dSmrO2JO8ASkpKcrOzlZ0dLT69+8vSXK73QoODlZmZqbatGmjU6dOqa6u7kvrlpeX\nKzQ0VJGRkZKk/v376/PPP5ckxcfHKygoSEFBQQoJCZEkdezYUYsWLVJISIjOnz+v0NBQSVJYWJg6\nd+4sSercubNqamp8/nPj1hs4cKBeffVVSdKRI0eUmpqqCxcuaObMmZKk2tpadevWTdLVXz+XFRYW\natiwYZKk7t27Kzw8vHFZr169JEmdOnVqfC2CSN6Q6OhoXbhwQTk5OcrMzNSJEydUVVWlLVu2KC8v\nT9XV1XrwwQd1tV+Pj4iI0Pnz51VeXq7w8HDt2bOn8fDG5XJ96fmzZs3S3Llz1b17d7322ms6efLk\nNZ/rdrvV0NBwk39a+IvL/7H27NlTL7/8srp06aKPP/5YZWVlkq7+mrgsPj5eu3bt0vDhw/Xpp5/q\nzJkzt2TmQEYkb1BycrLeeecdxcTE6MSJE2rRooVat26t1NRUSV9cqCktLf3Sem63W9OnT9f48ePV\nrl07NTQ06K677rrm3zNy5EhNnDhRt912mzp16mS+uCMiIlRbW6s5c+boF7/4xY3/kHDc5cNtt9ut\n8+fPKysrS/Hx8Zo8ebLq6urkcrk0a9asq77W/r8xY8YoKytLDz30kLp06XLFaR5cHXcBctCSJUv0\n6KOPqmXLlpo0aZKGDBmiBx54wOmx0ITt3LlTFy5c0JAhQ3Ts2DE99thj2rJli9Nj+TX2JB3Utm1b\njR07ViEhIeratauSk5OdHglNXHR0tDIzM7VgwQLV1dXpueeec3okv8eeJAAYeAsQABiIJAAYiCQA\nGIgkABiIJAAYiCQAGP4XOaZW2NqXxuoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25a9e868c88>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "target_names = ['Malignant', 'Benign']\n",
    "sns.heatmap(cmat.T, square=True, annot=True, fmt='d', cbar=False,\n",
    "            xticklabels=target_names, yticklabels=target_names)\n",
    "y_test.sum(), y_test_pred.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Точность, полнота и f1-score (precision, recall & f1-score)\n",
    "В информационном поиске точность это мера релевантности результатов поиска. Например, если из всех результатов поиска только 60% релевантны запросу, то точность составляет 60%. В информационном поиске полнота это мера доли релевантных ответов в результате поиска относительно всех релевантных ответов. Например, если в целом в базе есть 100 релевантных запросу документов, из которых в результате поиска вернулось только 40, то полнота равна 40%. \n",
    "\n",
    "Модель, у которой высокая полнота, но низкая точность, возвращает очень много результатов, среди которых много неверных ответов. Например, если модель может определить всех истинно больных но при этом очень часто определяет здорового человека больным, то у нее будет высокая полнота но низкая точность. И наоборот, если у модели высокая точность но низкая полнота, то она будет редко определять положительные классы, но эти кассы будут в большинстве случаев корректно классифицированы. Например, если модель не может определить всех больных, но при этом никогда не определяет здорового человека больным, то у этой модели высокая точность но низкая полнота.\n",
    "\n",
    "Точность (P) и полнота (R) определяются следующими формулами\n",
    "$$\n",
    "\\Large \n",
    "\\begin{align}\n",
    "P &= \\frac{T_p}{T_p + F_p} \\\\\n",
    "R &= \\frac{T_p}{T_p + F_n}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "Иногда межет быть желательно найти баланс между этими показателями. В таком случае можно использовать меру F1-score, которая является <a href=\"https://ru.wikipedia.org/wiki/%D0%A1%D1%80%D0%B5%D0%B4%D0%BD%D0%B5%D0%B5_%D0%B3%D0%B0%D1%80%D0%BC%D0%BE%D0%BD%D0%B8%D1%87%D0%B5%D1%81%D0%BA%D0%BE%D0%B5\">гармоническим средним</a> (harmonic mean) значением между точность и полнотой:\n",
    "$$\n",
    "\\Large F1 = 2 \\frac{PR}{P + R}\n",
    "$$\n",
    "\n",
    "Функция `classification_report` возвращает точность (precision), полноту (recall) и f1-score для каждого класса. Колонка support отображает количество данных соответсующему классу в данных."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.138739Z",
     "start_time": "2018-02-02T16:34:10.132239Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "  Malignant       0.91      0.98      0.95        53\n",
      "     Benign       0.99      0.94      0.97        90\n",
      "\n",
      "avg / total       0.96      0.96      0.96       143\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test, y_test_pred, target_names=target_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ROC-кривая (ROC curve)\n",
    "Модель бинарной классификации может возвращать вероятность принадлежности одному из двух классов. Чтобы отнести пример к тому или другому классу необходимо подобрать порог вероятности, выше которого все предсказания будут считаться положительными, а ниже этого порога отрицательными. Выбор этого порогового значения позволяет балансировать точность и полноту.\n",
    "\n",
    "ROC-кривая (англ. receiver operating characteristic, рабочая характеристика приёмника) - график, позволяющий оценить качество бинарной классификации при изменении порога решений (decision threshold). Использование этой кривой очень распространено. Изначально кривая была создана для оценки работы военных радаров (отсюда и название). Вскоре её начали использовать для оценки моделей в психологии, медицине, радиологии, метереологии и т.д. \n",
    "\n",
    "ROC-кривая показывает зависимость изменения полноты по отношению к ложноположительным результатам при изменении порога верятности. Численно ROC-кривая характеризуется размером площади под кривой (ROC area under curve, ROC-AUC). В идеале эта площадь должна быть равна единице, тогда модель будет находить все истинно положительные результаты (полнота 100%). Для сравнения вместе ROC-кривой всегда рисуется диагональная линия, которая соответствует модели, которая возвращает ответы случайно в пропорции 50/50. ROC-AUC значение этой кривой равно 0.5. ROC-кривая любой другой модели должна быть выше этой линии иначе никакой ценности эта модель не будет из себя представлять. Можно попробовать интерактивные примеры ROC-кривой по ссылкам <a href=\"http://www.navan.name/roc/\">Understanding ROC curves</a> и <a href=\"http://arogozhnikov.github.io/2015/10/05/roc-curve.html\">ROC curve demonstration</a>.\n",
    "\n",
    "Для получения ROC-кривой в sklearn используется функция `roc_curve` из модуля `sklearn.metrics`. ROC-AUC можно вычислить с помощью функции `roc_auc_score` из модуля `sklearn.metrics`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.154739Z",
     "start_time": "2018-02-02T16:34:10.141239Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_test_prob = classifier.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.331239Z",
     "start_time": "2018-02-02T16:34:10.156739Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x25aa01f19e8>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAFlCAYAAAAZGcpRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FPXCxfHvpveEEpDeAygoVWqkBglVRAFBiuJFxUoP\nShMCxBACAoLYqIJ4EVQQEELvIr2D9E4gJCEJqTvvH7zuFSGGttmU83keH7Mls2d/M+TszE4xGYZh\nICIiIjmCna0DiIiIyOOjYhcREclBVOwiIiI5iIpdREQkB1Gxi4iI5CAqdhERkRxExS6Zpnz58rRu\n3Zq2bdvywgsv8Pzzz9O+fXv2799vlddr27YtsbGxVpm2rezbt49hw4YBsH//ft5//32rv2b58uWJ\nioqy+uv805AhQzhw4MAD/15G8/3mzZt069btvp+f1dlq/kjW5WDrAJK7zJo1i7x581puf/PNNwQH\nB7NgwYLH/lo///zzY5+mrf35559cuXIFgMqVKzNp0iQbJ7KeLVu20LFjxwf+vYzme0xMzB0fJnPi\nciK5m4pdbCY1NZVLly7h7e1tuW/atGmsXLkSs9lMkSJFGD58OAULFiQyMpLhw4dz8uRJ7Ozs6NSp\nE926dePmzZuMHj2aY8eOkZKSQp06dRg4cCAODg6UL1+erVu30rt3b3r06EHz5s0BCAsLwzAMBgwY\nwH//+1/mz5+P2WzGx8eHoUOHUqZMGYKCgoiOjubcuXM0bNiQAQMG3JF9wYIFzJkzBzs7O/Lnz8/Q\noUMpVaoUQUFBmEwmTpw4QVRUFPXq1WPIkCE4Ojpy4sQJRo8eTXR0NGlpaXTt2pWXXnqJ7du3M3r0\naNzc3EhISGDhwoWEhoayd+9e4uPjMQyD4OBgChcuzKRJk7h58yaDBw/mhRdeYNSoUSxdupSgoCA8\nPDw4evQoly9fpnTp0oSHh+Pu7s769esJCwvDzs6OihUrsmXLFubNm0fRokXveE979+4lODiYW7du\n4ejoyMCBA6lTpw4AkydPZu/evURHR9OzZ0+6dOlCQkICI0aM4PTp08TExODu7k5YWBilS5ema9eu\neHt7c/LkSV555RUqV67MuHHjSE5OJjIykrp16zJmzBgA1q5dy8SJEzGbzbi5ufHJJ5+wfPlyrl69\nSv/+/QkNDaV06dLpzudKlSrRpEkTjhw5QlhYGC+99BJbt24lLS2NQYMGcePGDQAaNGjAhx9+yODB\ng0lMTKRt27YsWrSIJ598kq1bt5I3b16mT5/O4sWLcXBwoESJEoSEhODp6XnHOKU3HxcvXsyUKVP4\n5ZdfMJlMtG/fnjfffJM2bdowZsyYu+Zn9erVCQoKwtnZmf3793Pt2jUCAwPJmzcva9euJTIykuDg\nYOrUqfOvy9Xfpbc8Sy5jiGQSPz8/o1WrVkbr1q2NevXqGY0bNzZGjRplXLt2zTAMw1i8eLHx4Ycf\nGikpKYZhGMb3339vvPHGG4ZhGMY777xjfPrpp4ZhGEZsbKzRsmVL4/Tp00ZQUJAxe/ZswzAMIzU1\n1ejfv7/x5ZdfWl7v+vXrxsKFC41evXpZnuPv72+cOnXK2L59u9G5c2cjISHBMAzD2LhxoxEYGGgY\nhmEMGjTI6N69+z3fx5YtW4ymTZsa169fNwzDMH788UcjMDDQMJvNxqBBg4wXXnjBiIuLM5KSkowu\nXboYc+bMMVJSUowWLVoYBw4csLyHwMBAY/fu3ca2bduMChUqGOfPnzcMwzB27dplvPfee0ZaWpph\nGIYxffp0480337S81l/vZdu2bUbLli0teTt27GgkJSUZycnJxgsvvGAsXLjQiIqKMp599lnj8OHD\nhmEYxqJFiww/Pz/j3Llzd7yn5ORko169esbatWsNwzCM/fv3G61atTLS0tIMPz8/45tvvjEMwzAO\nHjxoVKpUyUhOTjaWL19ujBo1yjKNoUOHGiNHjjQMwzBeffVVY/DgwZbH+vTpY2zbts0wDMOIi4sz\natWqZezfv9+IjIw0qlevbhw6dMgwDMP47bffjJ49exqGYRiNGjUy9u3bZxiGkeF8Xrx4seW1/prv\nU6ZMMYYOHWoYhmHEx8cbH374oREbG2ucO3fOqFKlyl3Pj4iIMJo1a2ZER0cbhmEYY8aMMaZOnXrH\nOP3bfDQMw+jbt68xfPhwY/DgwcaQIUMynJ+DBg0yXn75ZSM5Odm4evWq4efnZ3mfM2fONF577TXL\n8+61XP09/78tz5K7aI1dMtVfm+IPHTrEf/7zH6pWrUq+fPmA22tu+/fvp3379gCYzWZu3boF3N4s\n+9das6enJ0uXLgVg3bp17N+/n4ULFwKQmJh412sGBgYSGhpKZGQkhw4dokSJEpQsWZIffviBM2fO\n0KlTJ8tzY2JiiI6OBqB69er3fA8bN26kRYsWlq8UXnzxRUaPHs358+cBaNeuHe7u7sDt729Xr15N\n7dq1OXv2LB999JFlOomJiRw6dIgyZcpQqFAhihQpAkDVqlXx9vbm+++/59y5c2zfvt0yvX/j7++P\nk5MTAH5+fsTExPDHH39QpkwZKlSoYMkWHBx81+8eO3YMOzs7GjZsCEClSpVYsmSJ5fFWrVoBULFi\nRZKTk4mLi6N58+YUK1aMOXPmcObMGX7//XeqVq1q+Z0aNWpYfg4JCWHDhg188cUXnDx5ksTERBIS\nEti1axflypWjYsWKADRr1oxmzZrdlS+j+fz31/r7ePTq1YtLly5Rt25d+vXrh6enJzExMfccv61b\nt9K8eXPLFqTBgwff9ZzTp0+nOx+rVKnCJ598Qtu2bXFxcWHRokVAxvOzUaNGODo64uvri5ubG/7+\n/gAUL17csizCvZerV1999Y4xSm959vHxued7lpxJxS428eSTTzJ48GCGDBnCM888Q9GiRTGbzbzx\nxht07twZgOTkZMsfYQcHB0wmk+X3z507R548eTCbzXz22WeWzY2xsbF3PA/Azc2N559/nqVLl7J7\n925efvll4PYHh7Zt21o+MJjNZq5evWr5w+7m5nbP7MY9Lq9gGAapqakA2Nvb33G/nZ0daWlpeHl5\n3fF97rVr1/D09GTPnj13vNa6desYPXo0r732Gk2aNKF06dL88ssvGY6pi4uL5WeTyYRhGNjb29+V\n187u7n1m7e3t7xq3Y8eOUbp0aeD2+P813b/e17x58/jhhx/o0qULrVu3xsfHx/LhBu4cvy5dulCh\nQgX8/f0JDAxk7969lnx/f13DMDh69Kjlg8hfMprP95pXTz/9NKtXr2br1q1s27aNl19+mc8//5wC\nBQrca/juyhIbG0tsbOwdX1n823wEuH79OklJSSQnJ3P16lWKFSuW4fz868PYX/4a63vl+/s4/XM+\nZrQ8S+6hveLFZlq1akWVKlUs37XWr1+fhQsXEhcXB8Bnn33GwIEDAahTpw4//vgjcHuv5u7du3P6\n9Gnq16/PzJkzMQyD5ORk3n77bebOnXvXa3Xo0IFFixaxe/dunn/+eQDq1avHr7/+ytWrVwGYP38+\n3bt3zzB3/fr1WbZsmWVP5B9//BEfHx9KlCgBwPLly0lOTiYpKYnFixfTqFEjSpUqhbOzs6UQLl26\nRKtWre651/fmzZtp1KgRnTt3pnLlykRERJCWlgbc/uP+1weI+1GtWjVOnz7NkSNHAPjtt9/u+eGn\ndOnSmEwmNm/eDMDBgwfp3r07ZrM53Wlv2rSJdu3a8fLLL1OqVCnWrFljyfl3MTExHDhwgP79+9Os\nWTOuXLnC2bNnMZvNPPPMM5w4cYLjx48DsHr1aksx/f293u98/ruwsDCmTp1K06ZN+fjjjylbtiyn\nT5/GwcGBtLS0uz7w1K1bl1WrVlmWv8mTJzNz5sw7nvNv8zElJYW+ffvywQcf8O6779K3b19SUlL+\ndX4+iHstV3/3sMuz5DxaYxebGjp0KG3atGHjxo28/PLLXLlyhQ4dOmAymShUqBAhISEADBs2jBEj\nRtC6dWsMw+DNN9+kUqVKfPzxx4wePZrWrVuTkpJC3bp1eeONN+56nUqVKuHg4MDzzz+Ps7MzcHtT\n7X/+8x9ef/11TCYTHh4eTJky5a7S+6d69erRo0cPS/H9tdPVX2tQLi4udO7cmdjYWMshfXZ2dkyd\nOpXRo0fz9ddfk5qaygcffED16tXZvn37HdPv1KkT/fv3p3Xr1tjb21OjRg3LDoVVq1Zl4sSJvPPO\nO3ccspUeHx8fwsPDGTRoEHZ2dpZxcHV1veN5Tk5OTJ48mTFjxhAaGoqjoyOTJ0++a23y715//XWG\nDRvGokWLsLe356mnnuLYsWN3Pc/b25tevXrRrl07fHx8yJMnD9WqVePMmTPUqVOHsLAwBg0aRFpa\nGh4eHkyYMAGApk2b0qdPH4KDg+97Pv9d9+7dCQoKolWrVjg5OVG+fHlatWqFvb09Tz75JIGBgcyf\nP9/y/AYNGvDnn3/yyiuvAFC2bFlGjRp11zilNx8//fRTfH19LVuEIiIimDBhwr/Ozwdxr+Xq7x52\neZacx2Tca7uiiDyUoKAgypUrR8+ePW0dBYC4uDimTp3Ke++9h6urKwcPHuTNN99k48aN+oOfjWS1\n5UqyNq2xi+RgHh4eODo68tJLL+Hg4ICDgwMTJ05UqYvkYFpjFxERyUG085yIiEgOomIXERHJQVTs\nIiIiOUi22XkuNTWNGzcSbB0jR8uTx01jnAk0ztanMbY+jXHm8PX1zPhJ/5Bt1tgdHOwzfpI8Eo1x\n5tA4W5/G2Po0xllXtil2ERERyZiKXUREJAdRsYuIiOQgKnYREZEcRMUuIiKSg6jYRUREchAVu4iI\nSA6iYhcREclBVOwiIiI5iFWLfe/evXTt2vWu+9esWUP79u3p2LEjP/zwgzUjiIiI5CpWO1f8V199\nxS+//IKrq+sd96ekpDB27FgWLlyIq6srr7zyCo0bNyZ//vzWiiIiIpJrWK3YixcvzuTJkxk4cOAd\n9584cYLixYvj7e0NQPXq1dmxYweBgYHWiiIPwWv1SzhfWGnrGDmWr60D5AIaY+vTGGeCfsYD/4rV\niv3555/n/Pnzd90fFxeHp+f/rlbj7u5OXFzcfU3zYa5yIw/GMsYqdRGRTHfyeh5K5b2ByfTw08j0\ny7Z6eHgQHx9vuR0fH39H0f+byMib1ool3C71v8b4r0/ikd1ibRcoh/r7OIt1aIytT2P8eCUmphIe\nvo0pU/5g2rRA2rYtDzzcVpFML/YyZcpw5swZoqOjcXNz448//qBnz55Wf11tWr4/2rQmIpK5tmw5\nR9++qzh5MhqTCQ4fvmYp9oeRacW+ZMkSEhIS6NixI0FBQfTs2RPDMGjfvj0FCxa0+uur1B9cUpFm\nto4gIpJjRUcnMnLkBubOPQBA+fL5GD8+gGefLfxI0zUZhvHg38zbyKNs9vGd7XV7Gtq0nC5tWssc\nGmfr0xhbn8b40Rw8GEnHjou4ejUeJyd7PvzwWd57rybOzneubz/MvmWZvineWrSpXUREsovSpX1w\nd3fk2WcLEx4egJ9fvsc27RxT7PdT6tq0LCIitmA2G3z33X7atPHD29sFV1dHFi9+mSee8MDO7hF2\ngb+HHFPsf9GmdhERyUqOHr1O376r2LHjInv3XiUsrCkAhQtb5xDuHFfsIiIiWUFSUioTJ/7OpEm/\nk5JipkABdxo0KG7111Wxi4iIPGbbtl2gX79VHD8eBUDXrpUZNswfb28Xq7+2il1EROQxOn48irZt\nF2AYUKZMHsLDA6hTp2imvb6KXURE5DEqVy4vr7zyFE884cGHH9bCxSVzq1bFLiIi8gguX47j44/X\n8vbb1alR4/bJZSZMaIbpUU74/ghU7CIiIg/BbDaYO3c/I0duJDY2iYsX41i2rBMmk8lmpQ4qdhER\nkQd2/HgU/fqtYtu2CwA0a1aaTz9tYtNC/4uKXURE5D4lJ6cxefIOJkzYTnJyGvnzuzFmTCPatvXL\nEqUOKnYREZH7Fh2dyPTpO0lOTqNz56cYPvw58uRxtXWsO6jYRURE/kVcXDLOzvY4OtpToIA74eHN\n8PZ2xt/f+iebeRh2tg4gIiKSVa1ceRJ//1lMm7bTcl+rVuWybKmDil1EROQuV6/G06vXr7z66k9c\nuHCTiIhTmM3Z4yrn2hQvIiLy/wzDYP78g4wYsZ7o6CTc3BwICqrHG29UfexXYbMWFbuIiAi3d4zr\n2XMJGzeeA6BRoxKMG9eU4sW9bZzswajYRUREAC8vZ5KTzeTL58qoUQ1p375CljmE7UGo2EVEJNfa\nvfsy+fK5Ury4N3Z2JqZODcTNzZF8+bLWIWwPQjvPiYhIrhMXl8zQoesIDJzPgAERGMbtHeOKFfPK\n1qUOWmMXEZFcZs2aUwwYsJpz52KxszNRsWJ+UlPNODra2zraY6FiFxGRXOHatQSGDl3Hjz8eAaBy\n5QKEhwfwzDMFbZzs8VKxi4hIjpeQkEKjRnO4ciUeFxd7Bg6sy1tvVcfBIed9I61iFxGRHM/NzZEu\nXSqxY8clwsKaUqqUj60jWY2KXUREcpzUVDNffrmL4sW9adWqHAD9+9fB3t6210rPDCp2ERHJUfbv\nv0qfPivZt+8qvr5uNGpUEnd3xxy52f1eVOwiIpIjJCSkMG7cVr74YidpaQZFi3oyblxT3N0dbR0t\nU6nYRUQk21u//gz9+0dw5kwMdnYm3nyzGoMG1cXDw8nW0TKdil1ERLK1lJQ0Bg5czZkzMVSsmJ8J\nEwKoVq2QrWPZjIpdRESyHcMwSEkx4+Rkj6OjPWFhTdm16zK9e1fPMSeaeVgqdhERyVbOnYtl4MAI\nihb1Yty4pgD4+xfH37+4jZNlDbljF0EREcn20tJuH8Lm7z+L1atP8/PPR4mKumXrWFmO1thFRCTL\nO3gwkn79VrFr12UA2rTxY/ToRuTNm70v2GINKnYREcmyDMNg7NjNTJnyB6mpZgoV8uDTT5vQvHkZ\nW0fLslTsIiKSZZlMJs6diyUtzczrrz/Dxx/Xx9PT2daxsjQVu4iIZCnR0Ylcu5ZA2bJ5ARg1qiGv\nv16FmjUL2zhZ9qCd50REJEswDIOffz5KvXoz6dlzCcnJaQDkz++mUn8AWmMXERGbu3DhJkFBq/nt\nt5MAlCrlw40biRQs6G7jZNlP9in28SZ8bZ1BREQeK7PZYMaMvQQHbyQ+PgVPTyeGDvWnW7ensbPL\n2Vdhs5bsU+z3IalIM1tHEBGRB9C160+sWnUKgMDAMoSENKZQIU8bp8reslWxR3aLtXUEERF5jFq0\nKMu+fVcZO7ax5brp8miyVbGLiEj2tm3bBU6fjqZTp6cA6Ny5Em3a+OkQtsdIxS4iIlYXG5vEqFEb\nmTVrHy4u9tSqVYRSpXwwmUwq9cdMxS4iIlb166/HGTx4DZcvx+PoaEfv3jUoVMjD1rFyLBW7iIhY\nxeXLcQQFrWHZsj8BqF69EBMmBFChQn4bJ8vZVOwiImIVffuuIiLiFO7ujgwZUp8ePZ7B3l7nRbM2\nFbuIiDw2hmFgMt0+/nzEiOdwdrYnOLgRRYroELbMoo9OIiLyyJKT0xg/fhs9evyCYRgA+PnlY8aM\nNir1TKY1dhEReSQ7dlykX79VHDlyHYCdOy9Ro4bO7W4rVit2s9nMiBEjOHr0KE5OTgQHB1OiRAnL\n47/88gszZszAzs6O9u3b07lzZ2tFERERK4iLS2b06E18++0eDOP2+d3Hj2+qUrcxqxV7REQEycnJ\nLFiwgD179hASEsK0adMsj4eGhrJ06VLc3Nxo2bIlLVu2xNvb21pxRETkMfr112P06rWEixfjcHCw\n4513atC3by1cXR1tHS3Xs1qx79y5E39/fwCqVKnCgQMH7ni8fPny3Lx5EwcHhzt2thARkaxv69bz\nXLwYR9WqBRk/vhmVKukyXVmF1Yo9Li4OD4//nYDA3t6e1NRUHBxuv2S5cuVo3749rq6uBAQE4OXl\nleE0fX21A4a1aYwzh8bZ+jTGj5dhGJw5E0PJkj4ADBnyHCVL+vDaa1V0CFsWY7Vi9/DwID4+3nLb\nbDZbSv3IkSOsW7eO1atX4+bmxoABA1i+fDmBgYH/Os3IyJvWiivc/kOoMbY+jbP1aYwfr5Mnb9Cv\n3yqOHYti8+Ye+Pi44OvrSdu25YiKis94AvLQHuYDqtU+ZlWrVo0NGzYAsGfPHvz8/CyPeXp64uLi\ngrOzM/b29uTNm5fYWF25TUQkK0lJSeOzz36nQYPZbN58HrPZ4PjxKFvHkgxYbY09ICCAzZs306lT\nJwzDYMyYMSxZsoSEhAQ6duxIx44d6dy5M46OjhQvXpx27dpZK4qIiDyg3bsv06fPSg4dugZAhw5P\n8sknDciXz9XGySQjJuOvMwlkdeNNuh67lWnzZebQOFufxvjRTJmyg+DgTZjNBsWLexMW1pSGDUvc\n8RyNceZ4mE3xOkGNiIjcoXLlAphM8M47NRgwoA5ubjqELTtRsYuI5HKRkQmsXn2KTp2eAqBBgxL8\n/ntPihXL+GglyXpU7CIiuZRhGCxYcIjhw9dz40YipUr5UKtWEQCVejamYhcRyYVOnYpmwIAINmw4\nC8BzzxWnYEF3G6eSx0HFLiKSi6Smmvnii52MG7eVW7dSyZPHhZEjG9KhQ0WdATSHULGLiOQi48Zt\nZcKE7QC0b1+BUaMakj+/m41TyeOkYhcRyUXeeKMqa9acJiioLk2alLJ1HLECneBXRCQHW7fuDN26\n/UxKShoAvr5urFzZWaWeg6nYRURyoKioW7z77go6dPiRFStOMHfu/66wqe/SczZtihcRyUEMw2DR\noiMMGbKO69dv4eJiT//+dXj11Uq2jiaZRMUuIpJDnDsXy4ABEaxZcxqA+vWLERbWlNKl89g2mGQq\nFbuISA6xYcMZ1qw5jbe3M5980oBXXnlKm91zIRW7iEg2FhOTiLe3CwCdO1fi8uV4Xn21sk42k4tp\n5zkRkWzo1q0UgoM3Uq3a15w+HQ3c3imuX7/aKvVcTsUuIpLNbNp0loYN5zBp0g7i4pJZt+6MrSNJ\nFqJN8SIi2cSNG7f45JMNzJt3EIAKFfIxfnwANWsWtnEyyUpU7CIi2cCGDWd5661lXLuWgJOTPX36\n1OK992ri5GRv62iSxajYRUSygYIF3YmJSaRWrSKEhwdQrlxeW0eSLErFLiKSBaWlmVmx4gQtWpTF\nZDJRvnw+li17hcqVC2Bnp0PYJH3aeU5EJIs5fPgarVot4LXXlrB48VHL/c88U1ClLhnSGruISBaR\nmJjKxInbmTx5BykpZgoWdMfT08nWsSSbUbGLiGQB27adp2/fVfz55w0Aund/mqFD/fHycrZxMslu\nVOwiIja2fPmfdO/+CwDlyuVl/Pim1K5d1MapJLtSsYuI2FijRiWpWDEfLVqU48MPn8XZWX+a5eFp\n5zkRkUx2+XIcffqsJDo6EQAXFwciIl5l0KC6KnV5ZFqCREQyidlsMHv2PkaN2sjNm8k4O9sTEtIE\nAEdHnWhGHg8Vu4hIJjh27Dr9+kWwffsFAJo3L8P77z9r41SSE6nYRUSsKDk5jUmTfmfixN9JTk6j\nQAF3xo5tRKtW5XStdLEKFbuIiBXt3HmJ0NCtALz6aiWGDXsOHx8XG6eSnOy+ij0hIYGzZ89Svnx5\nbt26hZubm7VziYhkW8nJaZaLs9SpU5S+fWvh71+cevWK2TiZ5AYZ7hW/detW2rZtS+/evYmMjKRx\n48Zs2rQpM7KJiGQ7K1acoHbtb/n994uW+4KC6qnUJdNkWOzh4eHMmzcPLy8vChQowNy5cwkNDc2M\nbCIi2caVK/H07LmEbt1+5vz5m8ycudfWkSSXynBTvNlsxtfX13K7bNmyVg0kIpKdGIbBd98d4JNP\nNhATk4SbmyMffVSPnj2r2Dqa5FIZFvsTTzzB2rVrMZlMxMbG8t1331G4cOHMyCYikqWdPx/Le++t\nYPPm8wA0bVqKTz9tQrFiXjZOJrlZhpviR44cyZIlS7h06RIBAQEcPnyYUaNGZUY2EZEszdnZgUOH\nrpE/vytffNGC7757QaUuNpfhGvuRI0cIDw+/476VK1fSrFkzq4USEcmq9u27QsWK+XF0tMfX141Z\ns9rg55ePvHldbR1NBPiXYl+2bBnJyclMmjSJ999/33J/amoq06dPV7GLSK4SF5dMSMhmvvpqNx99\nVJ8PPrh91jhdhU2ymnSLPS4ujt27dxMfH8/27dst99vb29OnT59MCScikhVERJxk4MDVnD9/E3t7\nE7dupdg6kki60i32Dh060KFDB7Zu3UqdOnUyM5OISJYQGZnAkCFrWbz4KABPP12ACROaUblyARsn\nE0lfht+xOzo68vbbb5OQkIBhGJjNZi5evMiaNWsyI5+IiE2cPHmDwMD53LiRiKurA4MG1aVXr2o4\nOOhq15K1ZbiEDhkyhKZNm5KWlkaXLl0oUaIETZs2zYxsIiI2U7KkDxUr5qdBgxKsX9+N3r1rqNQl\nW8hwjd3FxYX27dtz4cIFvLy8CA4O5sUXX8yMbCIimSY11cz06bto2bIsJUv6YGdnYvbstnh6Oukq\nbJKtZPjx09nZmejoaEqVKsXevXsxmUwkJCRkRjYRkUyxb98Vnn9+Hp98soEBA1ZjGAYAXl7OKnXJ\ndjIs9h49etCnTx8aNWrETz/9RMuWLalUqVJmZBMRsar4+BSGD19Ps2bz2L//KsWKefH229VV5pKt\nZbgpPjAwkObNm2MymVi0aBGnT5+mePHimZFNRMRq1q49zYABqzl7NgY7OxNvvlmNQYPq4uHhZOto\nIo8k3TX2qKgoxo8fz9dff01aWhpw+/v23bt36+Q0IpKtXb4cR9euP3P2bAxPPeXL8uWvMGpUQ5W6\n5AjprrH3798fd3d3bty4QUpKCg0aNGDgwIHcunWLwYMHZ2ZGEZFH9tf35iaTiSee8GDgwNvn53j7\n7eo4OtrbMprIY5VusZ89e5aIiAji4uLo1KkT8+bNo2vXrvTo0QMnJ32qFZHs4+zZGAYMiKBTp6do\n164CAO+//6yNU4lYR7rF7uHhYfl/dHQ0kydPpmrVqpkWTETkUaWlmfnqq92EhGwmISGVc+diadu2\nPHZ22jnpNpZLAAAgAElEQVROcq50i/3ve4Xmz5//gUvdbDYzYsQIjh49ipOTE8HBwZQoUcLy+L59\n+wgJCcEwDHx9fRk3bhzOzs4P8RZERO524EAkffuuZM+eKwC88EJ5goMbqtQlx0u32OPj4/njjz8w\nm83cunWLP/74w/IdFUDNmjX/dcIREREkJyezYMEC9uzZQ0hICNOmTQNuf9c1dOhQJk2aRIkSJfjv\nf//LhQsXKF269GN6WyKSWyUmpjJ4cATjxm0hLc2gcGEPQkOb0qyZ/r5I7pBusRcsWJDPPvsMgAIF\nClh+httr87Nnz/7XCe/cuRN/f38AqlSpwoEDByyPnTp1Ch8fH2bOnMnx48dp0KCBSl1EHgvDMPjx\nx8OYzQZvvFGFjz6qr73dJVdJt9jnzJnzSBOOi4uzfE8Pty/3mpqaioODAzdu3GD37t0MGzaM4sWL\n89Zbb1GpUqUMryLn6+v5SJkkYxrjzKFxfryiom5hb2/C29sFgFmzXsBkMula6Vam5ThryvAENQ/L\nw8OD+Ph4y22z2YyDw+2X8/HxoUSJEpQpUwYAf39/Dhw4kGGxR0betFZc4fY/Uo2x9WmcHx/DMPj5\n52N89NFaAgPLMH58AAB16hQjMvKmxtmKtBxnjof58GS1SxVVq1aNDRs2ALBnzx78/PwsjxUrVoz4\n+HjOnDkDwB9//EG5cuWsFUVEcqDz52N59dWf6NXrV65dS+DEiRskJ6fZOpaIzVltjT0gIIDNmzfT\nqVMnDMNgzJgxLFmyhISEBDp27Mjo0aPp168fhmFQtWpVGjZsaK0oIpKDpKWZmTFjL6NHbyI+PgUv\nL2eGD/enS5fK2uNdBDAZf9/V/R5iYmIYN24cZ8+e5bPPPiM0NJSgoCC8vb0zK+Nt401EdovN3NfM\nZbRpLXNonB/erVspvPjiQnbuvARAq1blGDu2EQULetzxPI2x9WmMM4dVNsUPHTqUypUrEx0djbu7\nOwUKFGDAgAEPFVBE5FG4ujpSurQPTzzhzsyZbfj229Z3lbpIbpdhsZ8/f56OHTtiZ2eHk5MTffr0\n4fLly5mRTUSErVvPs3v3//7mjB7diE2betCiRVkbphLJujIsdnt7e27evGk5E93p06exs7PaPnci\nIgDExCTSr98q2rb9gQ8++M2yY5yPjwteXjpLpUh6Mtx57r333qNr165cunSJ3r17s2fPHsaMGZMZ\n2UQkFzIMg6VLj/PRR2u5ciUeR0c7WrXSUTMi9yvDYq9Xrx6VKlVi3759pKWlMXLkSPLnz58Z2UQk\nl7l06SaDBq1hxYoTANSsWZjw8ADKl89n42Qi2UeGxd6wYUMCAgJo06YNVapUyYxMIpILpaaaad16\nAWfPxuLh4cTQof507/60DmETeUAZFvvSpUtZuXIlEyZM4MqVK7Rs2ZI2bdrccaU2EZFH5eBgR79+\ntVm+/AQhIY0pXFinKxV5GBkex/53+/fvZ/jw4Rw5coRDhw5ZM9fddBy71em41Myhcb4tKSmVSZN2\n4OHhxNtvVwewXEHy75eNfhgaY+vTGGeOhzmOPcM19qioKJYvX86yZcuIiYmhVatWTJky5aECiogA\n/P77Rfr2XcmxY1G4ujrw8ssVyZ/f7ZELXUTuo9jbtm1LYGAggwcPplKlSpmRSURyqJs3kwgO3sTM\nmXsxDChd2ofx4wPIn9/N1tFEcowMi339+vU6bl1EHtny5X8SFLSGS5ficHCw4733atKnTy1cXKx2\nyQqRXCndf1Ht2rVj8eLFPPnkk3dsHjMMA5PJxOHDhzMloIhkf4Zh8MUXO7l0KY5q1Z4gPDyAJ5/0\ntXUskRwp3WJfvHgxAEeOHLnrseTkZOslEpEcwWw2iItLxsvLGZPJxPjxAaxde5rXX6+Cvb22AopY\nS4b/ujp27HjHbbPZTPv27a0WSESyvz//jKJdux/o2XOpZU/3smXz8p//VFOpi1hZumvs3bp14/ff\nfwegQoUK//sFBwcaN25s/WQiku0kJ6fx+ed/EB6+jaSkNPLnd+X8+ZsUK+Zl62giuUa6xT579mwA\ngoODGTJkSKYFEpHsaefOS/Ttu5LDh68D0KnTU4wY8Rx587raOJlI7pJusa9du5ZGjRrx1FNP8dNP\nP931+AsvvGDVYCKSfYwYsZ5p03ZiGFCihDdhYU1p0EBnpxSxhXSLff/+/TRq1MiyOf6fVOwi8hdX\nV0fs7Ey8/XZ1+vevg5ubo60jieRaD3RK2bi4OC5dukS5cja4hKJOKWt1OkVk5sgJ43z1ajynTkVT\nq1YR4PbpYY8fv0GlSlnjELacMMZZncY4czzMKWUz3D31v//9L4MHDyYqKooWLVrw/vvvM2HChIcK\nKCLZm2EYzJ9/gPr1Z/Laa0u4ceMWAM7ODlmm1EVyuwyLff78+QwaNIilS5fSpEkTlixZwsaNGzMj\nm4hkISdP3uCll37kgw9WEh2dRKVKviQmpto6loj8w32dy9HHx4f169fTrVs3HBwcSEpKsnYuEcki\nUlLSmDZtJ2FhW0lMTCNvXhdGjWrISy9V1EVbRLKgDIu9bNmyvPnmm5w/f546derwwQcfULly5czI\nJiJZwFtvLWPJkuMAvPRSRUaObKCLtohkYRkW+5gxY9i9ezd+fn44OTnRtm1bnnvuuczIJiJZwGuv\nPcPevVcJDW1C48YlbR1HRDKQYbGnpKSwdu1axo4dS1paGrVq1aJ27do4OOiKTCI50dq1p9m58xL9\n+9cBoH794mzd2gNHR3sbJxOR+5HhznMjR44kMTGRMWPG8Omnn5Kamsrw4cMzI5uIZKLr12/Ru/dy\nOnZcRGjoVnbuvGR5TKUukn1kuNp98OBBfvnlF8vtYcOG0aJFC6uGEpHMYxgGCxceZujQdURFJeLi\nYs+AAXV5+ukCto4mIg8hw2I3DIPY2Fi8vG5fxCE2NhZ7e316F8kJzpyJYeDACNauPQOAv38xxo1r\nSunSeWycTEQeVobF3qNHD1566SXLFd3WrFlDr169rB5MRKxv4sTtrF17Bh8fZ0aObEjHjk/qEDaR\nbC7DYm/fvj2VK1dmx44dmM1mJk+eTPny5TMjm4hYQUpKmuU78yFD/LGzMzFoUF0KFHC3cTIReRzS\nLXaz2cx3333H6dOnqV69Ol26dMnMXCLymCUkpBAWtpV1686wYkVnnJzsyZfPlfHjA2wdTUQeo3T3\nih8xYgQrVqzA1dWVL774gilTpmRmLhF5jDZsOEvDhrOZMuUPDh6MZNOmc7aOJCJWku4a+44dO1i2\nbBkmk4mePXvSvXt33n333czMJiKP6MaNW4wYsYH58w8CULFiPsLDm1G9eiEbJxMRa0m32J2dnS07\n0eTJk0c71IhkM7/+epwBA1Zz7VoCzs729O1bm3feqYGTk45qEcnJ0i32fxa5nV2G57IRkSwkJiaJ\na9cSqFu3KOPHB1CmjA5hE8kN0i32ixcvMnjw4HRvjx071rrJROSBpKWZOXAgkmeeKQjAK688RZ48\nLjz/fBns7LTFTSS3SLfYg4KC7rj97LPPWj2MiDycQ4ci6dt3FYcORbJuXTdKl7799VlgYFlbRxOR\nTJZusbdr1y4zc4jIQ0hMTCU8fBtTpvxBaqqZQoU8uHo1XmeOE8nFdIk2kWxqy5Zz9O27ipMno4Hb\nl1cdMqQ+np7ONk4mIrakYhfJhqZP38XQoesA8PPLy/jxAdSqVcS2oUQkS7ivXd0TEhI4cuQIhmGQ\nkJBg7UwikoHGjUvi4eHEgAF1WL36VZW6iFhkWOxbt26lbdu29O7dm8jISBo3bsymTZsyI5uI/L9L\nl24SHr4NwzAAKFcuL7t3v8GAAXVwdtaGNxH5nwyLPTw8nHnz5uHl5UWBAgWYO3cuoaGhmZFNJNcz\nmw2+/XYP9erNIiRkCwsXHrY85u3tYsNkIpJVZfhR32w24+vra7ldtqwOnxHJDEePXqdv31Xs2HER\ngMDAMtSvX8zGqUQkq8uw2J944gnWrl2LyWQiNjaW7777jsKFC2dGNpFcKSkplYkTf2fSpN9JSTFT\noIA7ISGNadWqnK2jiUg2kOGm+JEjR7JkyRIuXbpE06ZNOXz4MCNHjsyMbCK50owZexk/fhspKWa6\ndq3M5s3dVeoict8yXGPPly8f4eHhmZFFJNcyDMNyfYbXXnuGLVvO8/bb1alTp6iNk4lIdpNhsTdu\n3PieV3ZbvXq1VQKJ5DbLl//J+PHb+O9/25MnjyvOzg7Mnt3W1rFEJJvKsNjnzJlj+Tk1NZVVq1aR\nnJxs1VAiucGVK3EMHryWpUuPAzBz5j769Kll41Qikt1l+B17kSJFLP+VKFGCN954g4iIiAwnbDab\nGTZsGB07dqRr166cOXPmns8bOnQoYWFhD55cJJsymw1mz95HvXqzWLr0OO7ujowZ04j3369p62gi\nkgNkuMa+Y8cOy8+GYXD8+HGSkpIynHBERATJycksWLCAPXv2EBISwrRp0+54zvfff8+xY8eoWVN/\n0CR3OHHiBi+99CMbNtz+oBsQUIpPP21C0aJeNk4mIjlFhsU+adIky88mk4k8efIQEhKS4YR37tyJ\nv78/AFWqVOHAgQN3PL5r1y727t1Lx44dOXny5IPmFsmWLl68yYYNZ8if340xYxrRtq3fPfdhERF5\nWBkWe2BgIJ07d37gCcfFxeHh4WG5bW9vT2pqKg4ODly9epXPP/+cKVOmsHz58vuepq+v5wPnkAej\nMX78Tp+OpmRJHwBefPEpZsxIoU2b8uTN62rjZDmblmXr0xhnTRkW+7x58x6q2D08PIiPj7fcNpvN\nODjcfrkVK1Zw48YNevXqRWRkJImJiZQuXZoXX3zxX6cZGXnzgXPI/fP19dQYP0ZxccmMGbOJGTP2\nsnjxy9SuffvQtR49qhAZeVNjbUValq1PY5w5HubD032dea5bt24888wzODv/7zrP77777r/+XrVq\n1Vi7di0tWrRgz549+Pn5WR7r1q0b3bp1A2DRokWcPHkyw1IXyU5WrjzJoEGruXDhJvb2Jg4ciLQU\nu4iINWVY7FWqVHmoCQcEBLB582Y6deqEYRiMGTOGJUuWkJCQQMeOHR9qmiJZ3dWr8QwZso6ffjoK\nQJUqBRk/PoDKlQvYOJmI5BYm46/rQP7D4sWLadeuXWbnSd94E5HdYm2dIkfTprVHs2XLOXr0+IXo\n6CTc3BwICqrHG29UxcHhzqNKNc7WpzG2Po1x5niYTfHpHsc+e/bsRwojktv4+eXDzs5Eo0Yl2LCh\nO2+9Vf2uUhcRsbYMN8WLyL2lpKQxb95BXnnlKZyc7Mmf342VK7tQrJiXDmETEZtJt9iPHz9OkyZN\n7rr/r4tV6Fzxkpvt2XOZPn1WcfBgJFFRtyyngi1e3NvGyUQkt0u32EuUKMGXX36ZmVlEsrz4+BRC\nQjbz1Ve7MZsNihf3pmrVJ2wdS0TEIt1id3R0pEiRIpmZRSRLW7PmFAMGrObcuVjs7Ez07l2dAQPq\n4u7uaOtoIiIW6RZ7tWrVMjOHSJa2efM5OnVaDEDlygUIDw/gmWcK2jiViMjd0i32YcOGZWYOkSyt\nbt2iBASUok6dotrbXUSyNP11ErmHM2di6NbtZ06evAHcvgDS3Lkv8O67NVXqIpKl6XA3kb9JTTUz\nffouQkO3cOtWKvb2JmbMaAOgQ9hEJFtQsYv8v/37r9Knz0r27bsKwIsvlmfUqEY2TiUi8mBU7JLr\nJSSkEBa2lWnTdpKWZlC0qCehoU1o2rS0raOJiDwwFbvkehcu3OTLL28fl96rV1WCgurh4eFk61gi\nIg9FxS65UmxsEp6eTphMJsqVy8unnzamQoX8VK9eyNbRREQeiXbvlVzFMAwWLTpC7drf8uOPRyz3\nd+lSWaUuIjmCil1yjXPnYunceTFvvbWMa9dusXz5n7aOJCLy2GlTvOR4aWlmvvlmD2PGbCYhIQVv\nb2eGD3+Ozp0r2TqaiMhjp2KXHO3ixZu8/voSdu26DECbNn6MHt2IggXdbZxMRMQ6VOySo+XN60pM\nTBKFCnnw6adNaN68jK0jiYhYlYpdcpytW89TsWJ+fHxccHFxYNasNhQq5IGnp7Oto4mIWJ12npMc\nIzo6kb59V9K27Q988skGy/1+fvlU6iKSa2iNXbI9wzBYuvQ4QUFriIxMwMnJnqJFvTAMQ+d3F5Fc\nR8Uu2drFizcJClrDihUnAKhVqwjjxzfFzy+fjZOJiNiGil2yrStX4vH3n8XNm8l4ejoxdKg/3bo9\njZ2d1tJFJPdSsUu2VbCgOy1bliM2NomxYxtRqJCnrSOJiNicil2yjaSkVCZO/J2GDUtQq1YRAMLC\nmuLkZG/jZCIiWYeKXbKFbdsu0K/fKo4fj2Lp0uOsW9cVe3s7lbqIyD+o2CVLi41NYtSojcyatQ+A\nsmXzEBraBHt7HakpInIvKnbJspYt+5OgoNVcvhyPo6Md771Xkw8/rIWLixZbEZH06C+kZEmxsUn0\n7buSqKhEqlcvRHh4ABUr5rd1LBGRLE/FLlmG2WxgNhs4ONjh5eXM2LGNiYq6RY8ez2jTu4jIfVKx\nS5Zw/HgU/fqtonHjknz4YS0A2rWrYONUIiLZj1aDxKaSk9MYP34bjRrNYdu2C8yZs5+kpFRbxxIR\nyba0xi42s2PHRfr1W8WRI9cB6NKlEsOHP4ezsxZLEZGHpb+gkumSklIZMWID3367B8OAUqV8GD++\nKfXrF7d1NBGRbE/FLpnOycmeI0euYWdn4t13a9K3by1cXR1tHUtEJEdQsUumuHIlntRUM0WKeGIy\nmQgPb0Z8fAqVKvnaOpqISI6inefEqgzD4Lvv9lO//kzef/83DMMAbm9+V6mLiDx+WmMXqzl58gb9\n+q1i8+bzADg62hEfn4KHh5ONk4mI5FwqdnnsUlLSmDp1J2FhW0lKSiNfPleCgxvy4osVMJl0rXQR\nEWtSsctjlZZmplWr79m9+woAHTo8ySefNCBfPlcbJxMRyR1U7PJY2dvbERBQmuvXEwkLa0rDhiVs\nHUlEJFdRscsjW7PmFMnJZpo3LwPA++8/y9tv18DdXYewiYhkNhW7PLRr1xIYMmQdixYdIV8+V559\ntgd587ri5GSPk5O9reOJiORKKnZ5YIZh8MMPhxk+fB1RUYm4ujrw7rs18fJytnU0EZFcT8UuD+T0\n6WgGDFjN+vVnAHjuueKEhTWlZEkfGycTERFQscsDMAyD119fwoEDkeTJ48LIkQ3p0KGiDmETEclC\nVOySIcMwMJlMmEwmgoMbMnv2fkaNaoivr5uto4mIyD+o2CVdCQkphIZuIT4+hXHjmgJQt24x6tYt\nZuNkIiKSHhW73NP69Wfo3z+CM2disLc38c47NfQ9uohINmC1YjebzYwYMYKjR4/i5OREcHAwJUr8\n72QlS5cuZdasWdjb2+Pn58eIESOws9M1aWwtKuoWw4evZ8GCQwBUrJifCRMCVOoiItmE1Zo0IiKC\n5ORkFixYQL9+/QgJCbE8lpiYyMSJE5k9ezbff/89cXFxrF271lpR5D7Nn3/7KmwLFhzC2dmejz+u\nT0REF6pVK2TraCIicp+stsa+c+dO/P39AahSpQoHDhywPObk5MT333+Pq+vt84enpqbi7KxjoG3t\nt99OcO3aLerVK0pYWABlyuSxdSQREXlAViv2uLg4PDw8LLft7e1JTU3FwcEBOzs78ufPD8CcOXNI\nSEigXr16GU7T19fTWnFzpbQ0M5cuxVG0qBcA48c3o2HDknTv/owOYbMyLcvWpzG2Po1x1mS1Yvfw\n8CA+Pt5y22w24+DgcMftcePGcerUKSZPnnxfRRIZedMqWXOjAwci6ddvJfHxKaxe/SrOzg74+nrS\nsmUZrl2Ls3W8HM3X11PLspVpjK1PY5w5HubDk9W+Y69WrRobNmwAYM+ePfj5+d3x+LBhw0hKSmLq\n1KmWTfJifbdupRAcvJFmzb5j9+4rxMUlc+ZMjK1jiYjIY2K1NfaAgAA2b95Mp06dMAyDMWPGsGTJ\nEhISEqhUqRILFy6kRo0adO/eHYBu3boREBBgrTgCbNp0ln79Ijh1KhqTCXr2rMLHH9fHw8PJ1tFE\nROQxsVqx29nZMXLkyDvuK1OmjOXnI0eOWOul5R5GjFjP1Kk7AahQIR/jxwdQs2ZhG6cSEZHHTSeo\nySUqVMiPk5M9ffrU4r33auqyqiIiOZSKPYe6cOEmu3ZdonXr2/s2dOz4JHXqFKVECW8bJxMREWtS\nsecwaWlmZs7cS3DwJlJTzTz5pC9lyuTBZDKp1EVEcgEVew5y+PA1+vZdxc6dlwBo2bIsHh6ONk4l\nIiKZScWeAyQmpjJx4nYmT95BSoqZggXdCQlpTMuW5WwdTUREMpmKPQcYNGg18+cfBKB796cZOtQf\nLy+doldEJDdSsecA77//LPv3X2XMmMbUrl3E1nFERMSGdJ3UbGjp0uO8++4KDMMAoEyZPKxe/apK\nXUREtMaenVy+HEdQ0BqWLfsTgFatytG8+e2T/uiiLSIiAir2bMFsNpg9ex+jRm3k5s1k3N0dGTKk\nPgEBpWwdTUREshgVexZ3/HgUffuuYvv2CwA8/3xpQkKaUKSILpcoIiJ3U7FncT/9dJTt2y/g6+vG\n2LGNad26nDa7i4hIulTsWVBsbJLlcLX3369JWpqZt96qjo+Pi42TiYhIVqe94rOQmzeTGDRoNXXq\nzCAq6hYAzs4OBAXVU6mLiMh9UbFnEStWnKB+/VnMmLGXGzcS2bLlvK0jiYhINqRN8TZ25Uo8H320\nhiVLjgNQtWpBwsOb8dRTvjZOJiIi2ZGK3YaWLDlG376riIlJws3NkcGD6/HGG1Wwt9eGFBEReTgq\ndhvKm9eVmJgkmjQpSWhoU4oV87J1JBERyeZU7JkoJSWNDRvO0qTJ7RPL1KtXjOXLX6FatSd0CJuI\niDwW2uabSXbtukRAwHe88spitmw5Z7m/evVCKnUREXlstMZuZXFxyYSEbObrr/dgNhuUKOGNnZ2K\nXERErEPFbkUREScZOHA158/fxN7exDvv1GDAgDq4uTnaOpqIiORQKnYrmTVrHwMGRADw9NMFmDCh\nGZUrF7BxKhERyen0HbuVtG5djmLFvBgx4jlWrOisUhcRkUyhYn9MTp2Kpl+/VSQlpQK3D2XbuvU1\neveugYODhllERDKHNsU/otRUM198sZNx47Zy61YqxYt788EHzwLg5GRv43QiIpLbqNgfwb59V+jT\nZxX7918FoH37CnTpUsnGqUREJDdTsT+EhIQUQkO38MUXuzCbDYoV82LcuCY0blzK1tFERCSXU7E/\nhBUrTjB16k7s7Ey8+WY1Bg2qi4eHk61jiYiIqNjvV0pKGo6Ot78zb9euPDt2XKRDhyepWvUJGycT\nERH5H+2unQHDMFi48DDPPvstJ0/eAMBkMjF2bGOVuoiIZDkq9n9x9mwMnTotonfv5Vy4cJO5c/fb\nOpKIiMi/0qb4e0hLM/PVV7sJCdlMQkIqPj7OfPJJAzp1esrW0URERP6Viv0fjh27zrvvrmDPnisA\nvPBCeYKDG1KggLuNk4mIiGRMxf4PdnYmDh++RuHCHoSGNqVZs9K2jiQi8q927fqDYcMGU7JkKUwm\nE/Hx8RQuXIThw4NxdHTkxo0bfP75RC5fvoTZbKZAgYK8914f8uXLD8DevbuZMeMrUlNTSUxMpEWL\n1rz44ss2fU8xMdFMn/45Awd+bNMcSUmJjBw5lBs3buDm5sbHH39Cnjx57njO3LkziYhYibu7O507\nd6NePX9iY2MYOXIo8fHxeHt7M2jQEPLkycs330ynceMASpWyXrfoO3Zg794rGIYBQNmyeZk1qy2b\nNvVQqYtItlG9eg2mTPmSyZOn8+23c3FwcGDTpvUYhsHHHw+gQYNGTJnyJVOnfk3Llm0YOLAPaWlp\nXLhwnokTxzFs2CimTPmSzz//khUrfmXbti02fT9ffTWNF1/sYNMMAIsXL6R06bJMnfo1zZu3ZNas\nb+54/MSJP1m16jemT59BePgUvvnmCxITE5k9ewZPP12FadO+oX37jkyf/jkAHTp05vPPJ1o1c65e\nY79x4xYjRmxg/vyDTJnSnA4dngSgceOStg0mItmW1+qXcL6w8rFOM6lIM2KbLLzv56ekpHD9+jU8\nPb04evQwHh4e+Ps3tDxes2Ytliz5ib17d7Nnzy6aN29J3rz5AHB2diE8fAqurq53TPPcubN8+mkw\nKSkpuLi4MGXKJEaPHk2TJs2oXbsu27ZtYfXqlXz88Qjat29FiRIlKVmyFJs3b2TmzPm4uroyb94c\n7O3taNiwCaGhY0hKSsTZ2YWBAz+iYMH/HWUUHx/H4cOH6N+/HAA//riA9evXcuvWLXx8fBgzJoxV\nq1bw66+/YDab6dnzTWJjY1mw4Dvs7Ox4+ukqvP32e1y9eoWwsBCSk5O4fv0a//lPb5577n/jcP78\nOUJCRt3xPgMCmtO27YuW2/v27aVz524A1K5dj5kz7yz206dPUbVqdZydnQEoWrQ4f/55nNOnT9Kr\nV28Ann76GSZMCAXA09MTZ2dn/vzzOGXLlrvvefogcmWxG4bBzz8f46OP1nLtWgJOTvZERyfaOpaI\nyEPbufMP3n23F9HRNzCZTLRp8yI1ajzL6tWrKFy46F3PL1y4CJcvX+LatUjKlfO74zEPD4+7nv/5\n5xN59dUe1K5dl02b1nPo0KF0s1y9eoVvv52Lt7cPDg6OrFu3msDAVkRErGDChM8ZP/5TXnqpI3Xq\n1OOPP37niy+mMHx4sOX3Dx48QPHiJQAwm83ExMQwceJU7Ozs6Nv3XQ4fPgjcLsmQkHBiY2Po3fsN\nvv56Di4uLowaNZQdO7YBJjp16kK1ajXYv38v33wz/Y5iL1q0GFOmfPmv4xofH28ZDzc3N+Lj4+54\nvEyZssydO4OEhHhSUlI4cGAfbdq0o1y58mzatAE/vwps2rSB/2vvzqOiLvcHjr/ZRUXARH5qgMtF\nNAq9vskAABAQSURBVMGFFBWXEKVMWUJU1MA0cykT8WhH7eJoibjgbip585KR5koucN3J7BJeN9ST\n3twCi3KXUWRkkOb7+4PLJIkgKI4zfV7ncA7zfeb7fT7zOXP48DzznecpKCh44Bx3MjOPSWF/WnJy\n7jB58n727s0CoHPnRixYEMDf/lbXwJEJIUxBZUbWT9PLL7fno49mc/u2mgkTxtKgQUMAnJycuHLl\nt4een5PzMx06dOTGjetcu3a1VNv58+dQFB3Nm7fQH/v550t4erYGoGvXV3BysmPz5q/17SUfZwLY\n2ztgb+8AQFDQG8yfPwc3t8a4uLhhb+/ATz9dICkpkbVr1wBgYVG6FKnVaurWLf6bbG5ujpWVFTNm\n/B1bW1uuXbtGUVHxLpolxT8n5xfU6lwmTYoCQKPR8OuvObRu3Y41a1aTmroNMNOf90cOKh6x16pV\nC40mX3/dP//T07hxE8LCBjJx4jjq1/8/XnqpFfb2DkRGDmPx4vmMHTuSzp274OzsrD/nhRfqcePG\ndarLX6qwHzt2mf79N5Off586dWxQqboREeGFubmZoUMTQoinwt7egWnTZhIVNYYWLdbh5dWGmzdv\n8u9/H6Rr1+4AHDr0PTk5ObRt603Dho2YOnUS/v6v4ujoiEajIT4+juHD3yl1XTe3Jvz3v6fp0KEj\ne/bsRKfTYm1tzc2bNwA4d+5H/XPNzf+4fcvFxRVQWLcuidDQ/gC4ujZm8OAIvLzacOlSNpmZx0r1\n5ejoSF5eHgAXLpzn4MED/OMfaygoKGDEiAj988zMivtp0KAR9es7s3jxCiwtLfnXv3bg7t6czz5L\nICjoDTp37kJq6nZ27kwp1c/jjNi9vNqQkZHOSy95cuhQOm3atCvVnpubi0ajYeXKf3L37l0mTBhL\n06bN+M9/MggKegMvrzYcOLAfL682+nPy8u7g4OD4566emr9UYff0dKJhQzs8PF5g9uweODs/PN0k\nhBDGrkmTpvTvH87ixfHExs5l3rxFLFmygKSkRADq13cmPn4xFhYWNGjQkPfei+Lvf/8Ac3NzNBrN\n/4ph11LXHDt2PPHxcaxZs5oaNWqwZMkiGjduzuzZH7Nnz67/FfCy9e0bwurVCXh7t9dfa8GCORQW\nFqLVFjB+/KRSz2/VyouVK5cBxcXX1taWd999Gyh7tOvo6Eh4+Ju8//4ofv/9dxo0aIi/fwA9evRk\n+fIlfPnl5zg51UetVlc6l6Gh/YmNnc67747AyspK/5HB+vVf8uKLLnTp0p3s7CzeeWcoVlaWjB07\nHgsLC1xd3YiNnQ5AvXpOTJ06TX/NM2dOM3r02ErH8rjMlAfnT55nC8y4PvROpU4pKChi5cpjDBvW\nGkfH4htB1OoCHBxqVEeERs/JyY7r1/MMHYbJkzxXP8lx9avuHMfHxxES0q/UxwGm4M6d28TGzmDe\nvEWP9XwnJ7tK92GyX3fLyMjB3z+J2bPTmTHjoP64FHUhhHj+vfPOGL7+2jD3K1SnDRvWVetoHUxw\nKv727QI+/vg7kpKK13V3d6/L4MGeBo5KCCFEZTg61mXy5BhDh/HUjRz5brX3YVKFPSXlPFOnpnH1\naj5WVuZERfkQHe2DjY1JvUwhhBDikUym4p0+fZ23394BQPv2DVi4MIAWLeoZOCohhBDi2TLqwq4o\nCmZmxV9Va9XKidGjvWnSxIFhw9rIV9iEEEL8JRntzXPnzt0kJGQjGRk5+mMzZ/rx9tttpagLIYT4\nyzK6wq7VFhEfn4G//5ccOvQrc+cadqMCIYQQ4nlSbVPxOp2OGTNmcPbsWaytrYmNjcXNzU3fnpaW\nxvLly7G0tCQsLIyBAyvYxadJHw4f/o2JE/dy9uxNACIiPFGpulfXSxBCCCGMTrUV9n379lFYWMiG\nDRs4ceIEc+bMYeXKlUDxzkOzZ89m8+bN2NraMnjwYPz9/alX79E3u72fNpYVK9ajKNC0qQMLFwbg\n6+tSXeELIYQQRqnapuKPHTtGt27dAGjbti0//PCDvu3ixYu4urpib2+PtbU1L7/8MkeOHCn3ehs2\nnMbCwpzoaB8OHBgqRV0IIYQoQ7WN2O/evVtqFxwLCwuKioqwtLTk7t272Nn9sUxerVq1uHv3blmX\n0bt+/YPqClU8oCrLF4rKkzxXP8lx9ZMcP5+qbcReu3Zt8vPz9Y91Oh2WlpZltuXn55cq9EIIIYSo\nmmor7N7e3hw8WLxG+4kTJ2jevLm+rVmzZly6dAm1Wk1hYSFHjx6lXbt2j7qUEEIIIR5Tte3uVnJX\n/Llz51AUhbi4OM6cOYNGoyE8PFx/V7yiKISFhfHmm29WRxhCCCHEX4rxbNsqhBBCiAoZ3QI1Qggh\nhHg0KexCCCGECXnuCrtOp0OlUhEeHk5kZCSXLl0q1Z6WlkZYWBjh4eFs3LjRQFEat4pynJKSwoAB\nAxg0aBAqlQqdTmegSI1XRTkuMW3aNObPn/+MozMNFeX41KlTDBkyhMGDBxMVFYVWqzVQpMatojxv\n376d0NBQwsLCWLdunYGiNA0nT54kMjLyoeOVrnvKc2b37t3K5MmTFUVRlMzMTGXMmDH6tsLCQqVX\nr16KWq1WtFqt0q9fP+X69euGCtVolZfje/fuKT179lQ0Go2iKIoyYcIEZd++fQaJ05iVl+MSX331\nlTJw4EAlPj7+WYdnEsrLsU6nU4KDg5Xs7GxFURRl48aNysWLFw0Sp7Gr6L3cpUsXJTc3V9Fqtfq/\nz6LyVq1apQQGBioDBgwodbwqde+5G7E/7RXrxMPKy7G1tTXr16/H1tYWgKKiImxsbAwSpzErL8cA\nx48f5+TJk4SHhxsiPJNQXo6zsrJwcHDg888/JyIiArVaTdOmTQ0VqlGr6L3s4eFBXl4ehYWFpbbS\nFpXj6urKsmXLHjpelbr33BX2R61YV9JW2RXrxMPKy7G5ubl+zf6kpCQ0Gg1dunQxSJzGrLwcX7t2\njeXLl6NSqQwVnkkoL8e5ublkZmYSERFBYmIihw4dIiMjw1ChGrXy8gzg7u5OWFgYffv2xc/Pjzp1\n6hgiTKP32muv6Rdxe1BV6t5zV9hlxbrqV16OSx7PnTuX9PR0li1bJv+BV0F5Od61axe5ubmMGjWK\nVatWkZKSQnJysqFCNVrl5djBwQE3NzeaNWuGlZUV3bp1e2ikKR5PeXn+8ccfOXDgAPv37yctLY1b\nt26xc+dOQ4VqkqpS9567wi4r1lW/8nIMoFKp0Gq1rFixQj8lLyqnvBwPHTqU5ORkkpKSGDVqFIGB\ngfTr189QoRqt8nLs4uJCfn6+/kavo0eP4u7ubpA4jV15ebazs6NGjRrY2NhgYWFB3bp1uXPnjqFC\nNUlVqXvVtglMVQUEBJCens6gQYP0K9bt2LFDv2LdlClTGDFihH7FOmdnZ0OHbHTKy7GnpyebN2+m\nffv2vPXWW0BxIQoICDBw1MalovexeHIV5XjWrFlMnDgRRVFo164dfn5+hg7ZKFWU5/DwcIYMGYKV\nlRWurq6EhoYaOmST8CR1T1aeE0IIIUzIczcVL4QQQoiqk8IuhBBCmBAp7EIIIYQJkcIuhBBCmBAp\n7EIIIYQJee6+7iaEKcrJyaF37940a9as1PGEhAQaNGhQ5jkly0uOGzeuyv0mJyczZ84cfR8FBQX4\n+Pgwffr0Mle5Ks+SJUvw9PSkZ8+eREZGkpSUBEBISAjbtm2rcowAkZGRXLlyhZo1awLFq225uLgw\nf/58/UqIZdmwYQO1atUiMDDwifoXwpRIYRfiGalfv/4TF8Cq8Pf3Z86cOQD8/vvvREZGsnbtWv06\nBY9r/Pjx+t8PHz6s//1pvabY2Fg6duwIFK9uFhUVRWJiIh988MEjz8nMzMTHx+ep9C+EqZDCLoSB\nnTt3jpkzZ6LRaLh16xbDhw9n6NCh+vb79+/z4Ycfcv78eQCGDBnCwIEDuXHjBiqViitXrmBmZsbE\niRPx9fUtty8LCwvatWtHdnY2AFu2bCExMREzMzNatWrFtGnTsLa2LrO/KVOm4OPjw5kzZwAYMGAA\nmzZtwsPDg9OnT+Pn58fWrVupV68earWawMBAvvnmGzIyMli6dClFRUW8+OKLzJw5E0dHx3Lj1Gg0\n5Obm0rp1awB27txJYmIiBQUFaLVaYmNjuX//PmlpaRw6dAgnJydatmxZ6XwIYYrkM3YhnpFr164R\nEhKi//nss88A2LRpE++99x5btmzhiy++YNGiRaXOy8zM5Pbt22zdupXExESOHz8OwKxZswgLCyM5\nOZmVK1eiUqkq3BwiNzeXgwcP4u3tzdmzZ0lISCApKYkdO3Zga2vLJ5988sj+SsTExOjjLmFpaUnv\n3r3ZtWsXAHv27KFXr17k5eWxYMECVq9ezdatW+natesj95+PiYkhODiYrl27Eh4ejq+vL8OGDUOn\n07F+/XoSEhLYvn07I0eOZPXq1fj6+uLv709UVBTdunWrUj6EMEUyYhfiGXnUVPyUKVP47rvv+PTT\nTzl79iwajaZUu7u7O1lZWYwYMYLu3bszadIkAL7//nt++uknli5dChRvsfvLL7/QsmXLUuenpaUR\nEhKCoigoikJAQACBgYGsXbuWHj166EfP4eHhTJ06lVGjRpXZX0VCQkKIi4sjIiKClJQUoqOjOXny\nJJcvX9bPQOh0Ouzt7cs8v2Qq/vjx40RFRfHKK69gbW0NwPLly0lLSyMrK4vDhw9jbv7wmORx8yGE\nqZPCLoSBRUdHU6dOHXr06EGfPn1ITU0t1e7o6Ehqairp6el8++23hIaGkpqaik6nY82aNTg4OABw\n9erVMm80e/Az9gfpdLpSjxVFoaio6JH9VcTLy4vbt29z6tQprl69ire3N/v27cPb25uEhAQAtFpt\nqZ2qyuLt7U1kZCSTJ09m27ZtaLVawsLCCAkJoUOHDnh4eLB27doyX8/j5EMIUydT8UIYWHp6OlFR\nUfTq1YsjR44AxTe5ldi/fz+TJk3Cz8+PmJgYatasyeXLl+nUqRPr1q0D4MKFCwQHB3Pv3r3H7tfH\nx4e0tDTUajUAGzdupGPHjo/s70F/3pO7RFBQENOnT6dPnz4AtGnThhMnTpCVlQXAihUrmDdvXoWx\nDR8+nHv37rF+/Xqys7MxNzdnzJgxdOrUiYMHD+rzY2Fhof/9SfMhhKmQEbsQBjZu3DiGDBlCnTp1\naNKkCY0aNSInJ0ff3r17d3bv3k3fvn2xsbHh1VdfxcPDg5iYGFQqFUFBQQDMmzeP2rVrP3a/LVq0\nYPTo0URGRnL//n1atWrFRx99hI2NTZn9Pahnz56EhIQ8tI98cHAwS5YsYeHChQA4OTkRFxdHdHQ0\nOp0OZ2dn4uPjK4zN2tqa6Oho4uLi2Lt3Ly1btuT111+nRo0adOjQgd9++w0AX19fFi5ciJ2d3RPn\nQwhTIbu7CSGEECZEpuKFEEIIEyKFXQghhDAhUtiFEEIIEyKFXQghhDAhUtiFEEIIEyKFXQghhDAh\nUtiFEEIIEyKFXQghhDAh/w+bAB8xZvtcyAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25a9fef0400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve, roc_auc_score\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_test_prob, pos_label=1)\n",
    "score = roc_auc_score(y_test, y_test_prob)\n",
    "plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % score)\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic example')\n",
    "plt.legend(loc=\"lower right\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Само значение порога вероятности в графике не отображается. Можно выбрать нужный порог и отобразить точку на графике самостоятельно. Например, следующий код вычисляет f1-score для каждого порога, который вернула функция `roc_curve`, и выбирает тот порог, при котором f1-score достигает максимума."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.492239Z",
     "start_time": "2018-02-02T16:34:10.333239Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best f1 score          : 0.983240223464\n",
      "best f1 score threshold: 0.311596391406\n",
      "false positive rate    : 0.0377358490566\n",
      "true positive rate     : 0.977777777778\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Rustam\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAFlCAYAAAAZGcpRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FPXCxfHvpveEEpDeAygoVWqkBglVBAFBiuJFxUoP\nShNCCBACCoLoVSkK4kVQQUAIvYv0DtI7gZCEJKTuvH/wulcuYGi7m3I+z+NjtmT27G+GnJ3ZKSbD\nMAxEREQkR3CwdwARERF5fFTsIiIiOYiKXUREJAdRsYuIiOQgKnYREZEcRMUuIiKSg6jYxWbKly9P\n69atadu2LS+88ALPP/887du3Z9++fVZ5vbZt2xIfH2+VadvL3r17GT58OAD79u3jvffes/prli9f\nnpiYGKu/zv8aOnQo+/fvf+Dfy2y+37hxg+7du9/387M6e80fybqc7B1AcpdZs2aRN29ey+2vvvqK\n0NBQ5s+f/9hf6+eff37s07S3P//8k8uXLwNQuXJlPv30Uzsnsp7NmzfTqVOnB/69zOZ7XFzcbR8m\nc+JyIrmbil3sJj09nYsXL+Lr62u5b/r06axYsQKz2UyRIkUYMWIEBQsWJDo6mhEjRnDixAkcHBzo\n3Lkz3bt358aNG4wZM4ajR4+SlpZGnTp1GDRoEE5OTpQvX54tW7bQp08fevbsSfPmzQGIiIjAMAwG\nDhzIf/7zH+bNm4fZbMbPz49hw4ZRpkwZQkJCiI2N5ezZszRs2JCBAwfeln3+/PnMmTMHBwcH8ufP\nz7BhwyhVqhQhISGYTCaOHz9OTEwM9erVY+jQoTg7O3P8+HHGjBlDbGwsGRkZdOvWjQ4dOrBt2zbG\njBmDh4cHSUlJLFiwgPHjx7Nnzx4SExMxDIPQ0FAKFy7Mp59+yo0bNxgyZAgvvPACo0ePZsmSJYSE\nhODl5cWRI0e4dOkSpUuXJjIyEk9PT9atW0dERAQODg5UrFiRzZs3M3fuXIoWLXrbe9qzZw+hoaHc\nvHkTZ2dnBg0aRJ06dQCYMmUKe/bsITY2ll69etG1a1eSkpIYOXIkp06dIi4uDk9PTyIiIihdujTd\nunXD19eXEydO8PLLL1O5cmUmTJhAamoq0dHR1K1bl7CwMADWrFnD5MmTMZvNeHh48PHHH7Ns2TKu\nXLnCgAEDGD9+PKVLl77nfK5UqRJNmjTh8OHDRERE0KFDB7Zs2UJGRgaDBw/m+vXrADRo0IAPPviA\nIUOGkJycTNu2bVm4cCFPPvkkW7ZsIW/evMyYMYNFixbh5OREiRIlCA8Px9vb+7Zxutd8XLRoEVOn\nTuWXX37BZDLRvn173njjDdq0aUNYWNgd87N69eqEhITg6urKvn37uHr1KsHBweTNm5c1a9YQHR1N\naGgoderU+cfl6u/utTxLLmOI2EhAQIDRqlUro3Xr1ka9evWMxo0bG6NHjzauXr1qGIZhLFq0yPjg\ngw+MtLQ0wzAM4/vvvzdef/11wzAM4+233zbGjRtnGIZhxMfHGy1btjROnTplhISEGLNnzzYMwzDS\n09ONAQMGGF988YXl9a5du2YsWLDA6N27t+U5gYGBxsmTJ41t27YZXbp0MZKSkgzDMIwNGzYYwcHB\nhmEYxuDBg40ePXrc9X1s3rzZaNq0qXHt2jXDMAzjxx9/NIKDgw2z2WwMHjzYeOGFF4yEhAQjJSXF\n6Nq1qzFnzhwjLS3NaNGihbF//37LewgODjZ27dplbN261ahQoYJx7tw5wzAMY+fOnca7775rZGRk\nGIZhGDNmzDDeeOMNy2v99V62bt1qtGzZ0pK3U6dORkpKipGammq88MILxoIFC4yYmBjj2WefNQ4d\nOmQYhmEsXLjQCAgIMM6ePXvbe0pNTTXq1atnrFmzxjAMw9i3b5/RqlUrIyMjwwgICDC++uorwzAM\n48CBA0alSpWM1NRUY9myZcbo0aMt0xg2bJgxatQowzAM45VXXjGGDBlieaxv377G1q1bDcMwjISE\nBKNWrVrGvn37jOjoaKN69erGwYMHDcMwjN9++83o1auXYRiG0ahRI2Pv3r2GYRiZzudFixZZXuuv\n+T516lRj2LBhhmEYRmJiovHBBx8Y8fHxxtmzZ40qVarc8fyoqCijWbNmRmxsrGEYhhEWFmZMmzbt\ntnH6p/loGIbRr18/Y8SIEcaQIUOMoUOHZjo/Bw8ebLz00ktGamqqceXKFSMgIMDyPmfOnGm8+uqr\nlufdbbn6e/5/Wp4ld9Eau9jUX5viDx48yL/+9S+qVq1Kvnz5gFtrbvv27aN9+/YAmM1mbt68Cdza\nLPvXWrO3tzdLliwBYO3atezbt48FCxYAkJycfMdrBgcHM378eKKjozl48CAlSpSgZMmS/PDDD5w+\nfZrOnTtbnhsXF0dsbCwA1atXv+t72LBhAy1atLB8pfDiiy8yZswYzp07B0C7du3w9PQEbn1/u2rV\nKmrXrs2ZM2f48MMPLdNJTk7m4MGDlClThkKFClGkSBEAqlatiq+vL99//z1nz55l27Ztlun9k8DA\nQFxcXAAICAggLi6OP/74gzJlylChQgVLttDQ0Dt+9+jRozg4ONCwYUMAKlWqxOLFiy2Pt2rVCoCK\nFSuSmppKQkICzZs3p1ixYsyZM4fTp0/z+++/U7VqVcvv1KhRw/JzeHg469ev5/PPP+fEiRMkJyeT\nlJTEzp07KVeuHBUrVgSgWbNmNGvW7I58mc3nv7/W38ejd+/eXLx4kbp169K/f3+8vb2Ji4u76/ht\n2bKF5s2bW7YgDRky5I7nnDp16p7zsUqVKnz88ce0bdsWNzc3Fi5cCGQ+Pxs1aoSzszP+/v54eHgQ\nGBgIQPHixS3LItx9uXrllVduG6N7Lc9+fn53fc+SM6nYxS6efPJJhgwZwtChQ3nmmWcoWrQoZrOZ\n119/nS5dugCQmppq+SPs5OSEyWSy/P7Zs2fJkycPZrOZTz75xLK5MT4+/rbnAXh4ePD888+zZMkS\ndu3axUsvvQTc+uDQtm1bywcGs9nMlStXLH/YPTw87prduMvlFQzDID09HQBHR8fb7ndwcCAjIwMf\nH5/bvs+9evUq3t7e7N69+7bXWrt2LWPGjOHVV1+lSZMmlC5dml9++SXTMXVzc7P8bDKZMAwDR0fH\nO/I6ONy5z6yjo+Md43b06FFKly4N3Br/v6b71/uaO3cuP/zwA127dqV169b4+flZPtzA7ePXtWtX\nKlSoQGBgIMHBwezZs8eS7++vaxgGR44csXwQ+Utm8/lu8+rpp59m1apVbNmyha1bt/LSSy/x2Wef\nUaBAgbsN3x1Z4uPjiY+Pv+0ri3+ajwDXrl0jJSWF1NRUrly5QrFixTKdn399GPvLX2N9t3x/H6f/\nnY+ZLc+Se2iveLGbVq1aUaVKFct3rfXr12fBggUkJCQA8MknnzBo0CAA6tSpw48//gjc2qu5R48e\nnDp1ivr16zNz5kwMwyA1NZW33nqLb7/99o7X6tixIwsXLmTXrl08//zzANSrV49ff/2VK1euADBv\n3jx69OiRae769euzdOlSy57IP/74I35+fpQoUQKAZcuWkZqaSkpKCosWLaJRo0aUKlUKV1dXSyFc\nvHiRVq1a3XWv702bNtGoUSO6dOlC5cqViYqKIiMjA7j1x/2vDxD3o1q1apw6dYrDhw8D8Ntvv931\nw0/p0qUxmUxs2rQJgAMHDtCjRw/MZvM9p71x40batWvHSy+9RKlSpVi9erUl59/FxcWxf/9+BgwY\nQLNmzbh8+TJnzpzBbDbzzDPPcPz4cY4dOwbAqlWrLMX09/d6v/P57yIiIpg2bRpNmzblo48+omzZ\nspw6dQonJycyMjLu+MBTt25dVq5caVn+pkyZwsyZM297zj/Nx7S0NPr168f777/PO++8Q79+/UhL\nS/vH+fkg7rZc/d3DLs+S82iNXexq2LBhtGnThg0bNvDSSy9x+fJlOnbsiMlkolChQoSHhwMwfPhw\nRo4cSevWrTEMgzfeeINKlSrx0UcfMWbMGFq3bk1aWhp169bl9ddfv+N1KlWqhJOTE88//zyurq7A\nrU21//rXv3jttdcwmUx4eXkxderUO0rvf9WrV4+ePXtaiu+vna7+WoNyc3OjS5cuxMfHWw7pc3Bw\nYNq0aYwZM4Z///vfpKen8/7771O9enW2bdt22/Q7d+7MgAEDaN26NY6OjtSoUcOyQ2HVqlWZPHky\nb7/99m2HbN2Ln58fkZGRDB48GAcHB8s4uLu73/Y8FxcXpkyZQlhYGOPHj8fZ2ZkpU6bcsTb5d6+9\n9hrDhw9n4cKFODo68tRTT3H06NE7nufr60vv3r1p164dfn5+5MmTh2rVqnH69Gnq1KlDREQEgwcP\nJiMjAy8vLyZNmgRA06ZN6du3L6Ghofc9n/+uR48ehISE0KpVK1xcXChfvjytWrXC0dGRJ598kuDg\nYObNm2d5foMGDfjzzz95+eWXAShbtiyjR4++Y5zuNR/HjRuHv7+/ZYtQVFQUkyZN+sf5+SDutlz9\n3cMuz5LzmIy7bVcUkYcSEhJCuXLl6NWrl72jAJCQkMC0adN49913cXd358CBA7zxxhts2LBBf/Cz\nkay2XEnWpjV2kRzMy8sLZ2dnOnTogJOTE05OTkyePFmlLpKDaY1dREQkB9HOcyIiIjmIil1ERCQH\nUbGLiIjkINlm57n09AyuX0+yd4wcLU8eD42xDWicrU9jbH0aY9vw9/fO/En/I9ussTs5OWb+JHkk\nGmPb0Dhbn8bY+jTGWVe2KXYRERHJnIpdREQkB1Gxi4iI5CAqdhERkRxExS4iIpKDqNhFRERyEBW7\niIhIDqJiFxERyUFU7CIiIjmIVYt9z549dOvW7Y77V69eTfv27enUqRM//PCDNSOIiIjkKlY7V/yX\nX37JL7/8gru7+233p6WlMXbsWBYsWIC7uzsvv/wyjRs3Jn/+/NaKIiIikmtYrdiLFy/OlClTGDRo\n0G33Hz9+nOLFi+Pr6wtA9erV2b59O8HBwdaKIg/BZ1UHXM+vsHeMHMvf3gFyAY2x9WmMbaC/8cC/\nYrVif/755zl37twd9yckJODt/d+r1Xh6epKQkHBf03yYq9zIg7GMsUpdRMTmTlzLQ6m81zGZHn4a\nNr9sq5eXF4mJiZbbiYmJtxX9P4mOvmGtWMKtUv9rjP/6JB7dPd5+gXKov4+zWIfG2Po0xo9XcnI6\nkZFbmTr1D6ZPD6Zt2/LAw20VsXmxlylThtOnTxMbG4uHhwd//PEHvXr1svrrevzWAc/LWgvNjDat\niYjY1ubNZ+nXbyUnTsRiMsGhQ1ctxf4wbFbsixcvJikpiU6dOhESEkKvXr0wDIP27dtTsGBBq71u\naip8+KErs55WqT+olCLN7B1BRCTHio1NZtSo9Xz77X4AypfPx8SJQTz7bOFHmq7JMIwH/2beTh5m\ns8+AAa7Mnu2CEXHrCwvTgFtvt3v3VCIiUh5rvuxOm9ZsQ+NsfRpj69MYP5oDB6Lp1GkhV64k4uLi\nyAcfPMu779bE1fX29e2H2bfM5pvireVee3HPehpmRdz5/BUrnIiPT8HHxwbhRERE/qZ0aT88PZ15\n9tnCREYGERCQ77FNO8ecee5+Ds369VALy8+XLjlw9myOefsiIpKFmc0Gc+bsJS4uGQB3d2cWLXqJ\nX37p9FhLHXLQGvtf/r4Xd3w81K/vyaVLdxb4E0+YKVbMbMtoIiKSCx05co1+/VayffsF9uy5QkRE\nUwAKF7bOIdw5epXVxweaNUu/62PNmqVrM7yIiFhNSko648ZtpnHjOWzffoECBTxp0KC41V83x62x\n/6+wsFs7yK1Y4cSlSw488YSZZs3SLfeLiIg8blu3nqd//5UcOxYDQLdulRk+PBBfXzerv3aOL3YX\nF4iISCE+PoWzZx0oVsysNXUREbGaY8diaNt2PoYBZcrkITIyiDp1itrs9XN8sf/FxweeekrfqYuI\niHWVK5eXl19+iiee8OKDD2rh5mbbqs01xS4iImINly4l8NFHa3jrrerUqHHr5DKTJjXD9CgnfH8E\nKnYREZGHYDYbfPvtPkaN2kB8fAoXLiSwdGlnTCaT3UodVOwiIiIP7NixGPr3X8nWrecBaNasNOPG\nNbFrof9FxS4iInKfUlMzmDJlO5MmbSM1NYP8+T0IC2tE27YBWaLUQcUuIiJy32Jjk5kxYwepqRl0\n6fIUI0Y8R5487vaOdRsVu4iIyD9ISEjF1dURZ2dHChTwJDKyGb6+rgQGWv9kMw8jR595TkRE5FGs\nWHGCwMBZTJ++w3Jfq1blsmypg4pdRETkDleuJNK796+88spPnD9/g6iok5jN2eMq59oULyIi8v8M\nw2DevAOMHLmO2NgUPDycCAmpx+uvV8XBIWvsHJcZFbuIiAi3dozr1WsxGzacBaBRoxJMmNCU4sV9\n7ZzswajYRUREAB8fV1JTzeTL587o0Q1p375CljmE7UGo2EVEJNfatesS+fK5U7y4Lw4OJqZNC8bD\nw5l8+bLWIWwPQjvPiYhIrpOQkMqwYWsJDp7HwIFRGMatHeOKFfPJ1qUOWmMXEZFcZvXqkwwcuIqz\nZ+NxcDBRsWJ+0tPNODs72jvaY6FiFxGRXOHq1SSGDVvLjz8eBqBy5QJERgbxzDMF7Zzs8VKxi4hI\njpeUlEajRnO4fDkRNzdHBg2qy5tvVsfJKed9I61iFxGRHM/Dw5muXSuxfftFIiKaUqqUn70jWY2K\nXUREcpz0dDNffLGT4sV9adWqHAADBtTB0dG+10q3BRW7iIjkKPv2XaFv3xXs3XsFf38PGjUqiaen\nc47c7H43KnYREckRkpLSmDBhC59/voOMDIOiRb2ZMKEpnp7O9o5mUyp2ERHJ9tatO82AAVGcPh2H\ng4OJN96oxuDBdfHycrF3NJtTsYuISLaWlpbBoEGrOH06jooV8zNpUhDVqhWydyy7UbGLiEi2YxgG\naWlmXFwccXZ2JCKiKTt3XqJPn+o55kQzD0vFLiIi2crZs/EMGhRF0aI+TJjQFIDAwOIEBha3c7Ks\nIXfsIigiItleRsatQ9gCA2exatUpfv75CDExN+0dK8vRGruIiGR5Bw5E07//SnbuvARAmzYBjBnT\niLx5s/cFW6xBxS4iIlmWYRiMHbuJqVP/ID3dTKFCXowb14TmzcvYO1qWpWIXEZEsy2QycfZsPBkZ\nZl577Rk++qg+3t6u9o6VpanYRUQkS4mNTebq1STKls0LwOjRDXnttSrUrFnYzsmyB+08JyIiWYJh\nGPz88xHq1ZtJr16LSU3NACB/fg+V+gPQGruIiNjd+fM3CAlZxW+/nQCgVCk/rl9PpmBBTzsny36y\nT7FPNOFv7wwiIvJYmc0G33yzh9DQDSQmpuHt7cKwYYF07/40Dg45+yps1pJ9iv0+pBRpZu8IIiLy\nALp1+4mVK08CEBxchvDwxhQq5G3nVNlbtir26O7x9o4gIiKPUYsWZdm79wpjxza2XDddHk22KnYR\nEcnetm49z6lTsXTu/BQAXbpUok2bAB3C9hip2EVExOri41MYPXoDs2btxc3NkVq1ilCqlB8mk0ml\n/pip2EVExKp+/fUYQ4as5tKlRJydHejTpwaFCnnZO1aOpWIXERGruHQpgZCQ1Sxd+icA1asXYtKk\nICpUyG/nZDmbil1ERKyiX7+VREWdxNPTmaFD69Oz5zM4Ouq8aNamYhcRkcfGMAxMplvHn48c+Ryu\nro6EhjaiSBEdwmYr+ugkIiKPLDU1g4kTt9Kz5y8YhgFAQEA+vvmmjUrdxrTGLiIij2T79gv077+S\nw4evAbBjx0Vq1NC53e3FasVuNpsZOXIkR44cwcXFhdDQUEqUKGF5/JdffuGbb77BwcGB9u3b06VL\nF2tFERERK0hISGXMmI18/fVuDOPW+d0nTmyqUrczqxV7VFQUqampzJ8/n927dxMeHs706dMtj48f\nP54lS5bg4eFBy5YtadmyJb6+vtaKIyIij9Gvvx6ld+/FXLiQgJOTA2+/XYN+/Wrh7u5s72i5ntWK\nfceOHQQGBgJQpUoV9u/ff9vj5cuX58aNGzg5Od22s4WIiGR9W7ac48KFBKpWLcjEic2oVEmX6coq\nrFbsCQkJeHn99wQEjo6OpKen4+R06yXLlStH+/btcXd3JygoCB8fn0yn6e+vHTCsTWNsGxpn69MY\nP16GYXD6dBwlS/oBMHToc5Qs6cerr1bRIWxZjNWK3cvLi8TERMtts9lsKfXDhw+zdu1aVq1ahYeH\nBwMHDmTZsmUEBwf/4zSjo29YK65w6w+hxtj6NM7WpzF+vE6cuE7//is5ejSGTZt64ufnhr+/N23b\nliMmJjHzCchDe5gPqFb7mFWtWjXWr18PwO7duwkICLA85u3tjZubG66urjg6OpI3b17i43XlNhGR\nrCQtLYNPPvmdBg1ms2nTOcxmg2PHYuwdSzJhtTX2oKAgNm3aROfOnTEMg7CwMBYvXkxSUhKdOnWi\nU6dOdOnSBWdnZ4oXL067du2sFUVERB7Qrl2X6Nt3BQcPXgWgY8cn+fjjBuTL527nZJIZk/HXmQSy\nuokmXY/dyrT50jY0ztanMX40U6duJzR0I2azQfHivkRENKVhwxK3PUdjbBsPsyleJ6gREZHbVK5c\nAJMJ3n67BgMH1sHDQ4ewZScqdhGRXC46OolVq07SufNTADRoUILff+9FsWKZH60kWY+KXUQklzIM\ng/nzDzJixDquX0+mVCk/atUqAqBSz8ZU7CIiudDJk7EMHBjF+vVnAHjuueIULOhp51TyOKjYRURy\nkfR0M59/voMJE7Zw82Y6efK4MWpUQzp2rKgzgOYQKnYRkVxkwoQtTJq0DYD27SswenRD8uf3sHMq\neZxU7CIiucjrr1dl9epThITUpUmTUvaOI1agE/yKiORga9eepnv3n0lLywDA39+DFSu6qNRzMBW7\niEgOFBNzk3feWU7Hjj+yfPlxvv32v1fY1HfpOZs2xYuI5CCGYbBw4WGGDl3LtWs3cXNzZMCAOrzy\nSiV7RxMbUbGLiOQQZ8/GM3BgFKtXnwKgfv1iREQ0pXTpPPYNJjalYhcRySHWrz/N6tWn8PV15eOP\nG/Dyy09ps3supGIXEcnG4uKS8fV1A6BLl0pcupTIK69U1slmcjHtPCcikg3dvJlGaOgGqlX7N6dO\nxQK3dorr37+2Sj2XU7GLiGQzGzeeoWHDOXz66XYSElJZu/a0vSNJFqJN8SIi2cT16zf5+OP1zJ17\nAIAKFfIxcWIQNWsWtnMyyUpU7CIi2cD69Wd4882lXL2ahIuLI3371uLdd2vi4uJo72iSxajYRUSy\ngYIFPYmLS6ZWrSJERgZRrlxee0eSLErFLiKSBWVkmFm+/DgtWpTFZDJRvnw+li59mcqVC+DgoEPY\n5N6085yISBZz6NBVWrWaz6uvLmbRoiOW+595pqBKXTKlNXYRkSwiOTmdyZO3MWXKdtLSzBQs6Im3\nt4u9Y0k2o2IXEckCtm49R79+K/nzz+sA9OjxNMOGBeLj42rnZJLdqNhFROxs2bI/6dHjFwDKlcvL\nxIlNqV27qJ1TSXalYhcRsbNGjUpSsWI+WrQoxwcfPIurq/40y8PTznMiIjZ26VICffuuIDY2GQA3\nNyeiol5h8OC6KnV5ZFqCRERsxGw2mD17L6NHb+DGjVRcXR0JD28CgLOzTjQjj4eKXUTEBo4evUb/\n/lFs23YegObNy/Dee8/aOZXkRCp2ERErSk3N4NNPf2fy5N9JTc2gQAFPxo5tRKtW5XStdLEKFbuI\niBXt2HGR8eO3APDKK5UYPvw5/Pzc7JxKcrL7KvakpCTOnDlD+fLluXnzJh4eHtbOJSKSbaWmZlgu\nzlKnTlH69atFYGBx6tUrZudkkhtkulf8li1baNu2LX369CE6OprGjRuzceNGW2QTEcl2li8/Tu3a\nX/P77xcs94WE1FOpi81kWuyRkZHMnTsXHx8fChQowLfffsv48eNtkU1EJNu4fDmRXr0W0737z5w7\nd4OZM/fYO5LkUpluijebzfj7+1tuly1b1qqBRESyE8Mw+O67/Xz88Xri4lLw8HDmww/r0atXFXtH\nk1wq02J/4oknWLNmDSaTifj4eL777jsKFy5si2wiIlnauXPxvPvucjZtOgdA06alGDeuCcWK+dg5\nmeRmmW6KHzVqFIsXL+bixYsEBQVx6NAhRo8ebYtsIiJZmqurEwcPXiV/fnc+/7wF3333gkpd7C7T\nNfbDhw8TGRl5230rVqygWbNmVgslIpJV7d17mYoV8+Ps7Ii/vwezZrUhICAfefO62zuaCPAPxb50\n6VJSU1P59NNPee+99yz3p6enM2PGDBW7iOQqCQmphIdv4ssvd/Hhh/V5//1bZ43TVdgkq7lnsSck\nJLBr1y4SExPZtm2b5X5HR0f69u1rk3AiIllBVNQJBg1axblzN3B0NHHzZpq9I4nc0z2LvWPHjnTs\n2JEtW7ZQp04dW2YSEckSoqOTGDp0DYsWHQHg6acLMGlSMypXLmDnZCL3lul37M7Ozrz11lskJSVh\nGAZms5kLFy6wevVqW+QTEbGLEyeuExw8j+vXk3F3d2Lw4Lr07l0NJydd7VqytkyX0KFDh9K0aVMy\nMjLo2rUrJUqUoGnTprbIJiJiNyVL+lGxYn4aNCjBunXd6dOnhkpdsoVM19jd3Nxo374958+fx8fH\nh9DQUF588UVbZBMRsZn0dDMzZuykZcuylCzph4ODidmz2+Lt7aKrsEm2kunHT1dXV2JjYylVqhR7\n9uzBZDKRlJRki2wiIjaxd+9lnn9+Lh9/vJ6BA1dhGAYAPj6uKnXJdjIt9p49e9K3b18aNWrETz/9\nRMuWLalUqZItsomIWFViYhojRqyjWbO57Nt3hWLFfHjrreoqc8nWMt0UHxwcTPPmzTGZTCxcuJBT\np05RvHhxW2QTEbGaNWtOMXDgKs6cicPBwcQbb1Rj8OC6eHm52DuayCO55xp7TEwMEydO5N///jcZ\nGRnAre/bd+3apZPTiEi2dulSAt26/cyZM3E89ZQ/y5a9zOjRDVXqkiPcc419wIABeHp6cv36ddLS\n0mjQoAFsSl7mAAAgAElEQVSDBg3i5s2bDBkyxJYZRUQe2V/fm5tMJp54wotBg26dn+Ott6rj7Oxo\nz2gij9U9i/3MmTNERUWRkJBA586dmTt3Lt26daNnz564uOhTrYhkH2fOxDFwYBSdOz9Fu3YVAHjv\nvWftnErEOu5Z7F5eXpb/x8bGMmXKFKpWrWqzYCIijyojw8yXX+4iPHwTSUnpnD0bT9u25XFw0M5x\nknPds9j/vldo/vz5H7jUzWYzI0eO5MiRI7i4uBAaGkqJEiUsj+/du5fw8HAMw8Df358JEybg6ur6\nEG9BRORO+/dH06/fCnbvvgzACy+UJzS0oUpdcrx7FntiYiJ//PEHZrOZmzdv8scff1i+owKoWbPm\nP044KiqK1NRU5s+fz+7duwkPD2f69OnAre+6hg0bxqeffkqJEiX4z3/+w/nz5ylduvRjelsiklsl\nJ6czZEgUEyZsJiPDoHBhL8aPb0qzZvr7IrnDPYu9YMGCfPLJJwAUKFDA8jPcWpufPXv2P054x44d\nBAYGAlClShX2799veezkyZP4+fkxc+ZMjh07RoMGDVTqIvJYGIbBjz8ewmw2eP31Knz4YX3t7S65\nyj2Lfc6cOY804YSEBMv39HDrcq/p6ek4OTlx/fp1du3axfDhwylevDhvvvkmlSpVyvQqcv7+3o+U\nSTKnMbYNjfPjFRNzE0dHE76+bgDMmvUCJpNJ10q3Mi3HWVOmJ6h5WF5eXiQmJlpum81mnJxuvZyf\nnx8lSpSgTJkyAAQGBrJ///5Miz06+oa14gq3/pFqjK1P4/z4GIbBzz8f5cMP1xAcXIaJE4MAqFOn\nGNHRNzTOVqTl2DYe5sOT1S5VVK1aNdavXw/A7t27CQgIsDxWrFgxEhMTOX36NAB//PEH5cqVs1YU\nEcmBzp2L55VXfqJ371+5ejWJ48evk5qaYe9YInZntTX2oKAgNm3aROfOnTEMg7CwMBYvXkxSUhKd\nOnVizJgx9O/fH8MwqFq1Kg0bNrRWFBHJQTIyzHzzzR7GjNlIYmIaPj6ujBgRSNeulbXHuwhgMv6+\nq/tdxMXFMWHCBM6cOcMnn3zC+PHjCQkJwdfX11YZb5loIrp7vG1fM5fRpjXb0Dg/vJs303jxxQXs\n2HERgFatyjF2bCMKFvS67XkaY+vTGNuGVTbFDxs2jMqVKxMbG4unpycFChRg4MCBDxVQRORRuLs7\nU7q0H0884cnMmW34+uvWd5S6SG6XabGfO3eOTp064eDggIuLC3379uXSpUu2yCYiwpYt59i1679/\nc8aMacTGjT1p0aKsHVOJZF2ZFrujoyM3btywnInu1KlTODhYbZ87EREA4uKS6d9/JW3b/sD77/9m\n2THOz88NHx+dpVLkXjLdee7dd9+lW7duXLx4kT59+rB7927CwsJskU1EciHDMFiy5BgffriGy5cT\ncXZ2oFUrHTUjcr8yLfZ69epRqVIl9u7dS0ZGBqNGjSJ//vy2yCYiuczFizcYPHg1y5cfB6BmzcJE\nRgZRvnw+OycTyT4yLfaGDRsSFBREmzZtqFKlii0yiUgulJ5upnXr+Zw5E4+XlwvDhgXSo8fTOoRN\n5AFlWuxLlixhxYoVTJo0icuXL9OyZUvatGlz25XaREQelZOTA/3712bZsuOEhzemcGGdrlTkYWR6\nHPvf7du3jxEjRnD48GEOHjxozVx30nHsVqfjUm1D43xLSko6n366HS8vF956qzqA5QqSf79s9MPQ\nGFufxtg2HuY49kzX2GNiYli2bBlLly4lLi6OVq1aMXXq1IcKKCIC8PvvF+jXbwVHj8bg7u7ESy9V\nJH9+j0cudBG5j2Jv27YtwcHBDBkyhEqVKtkik4jkUDdupBAaupGZM/dgGFC6tB8TJwaRP7+HvaOJ\n5BiZFvu6det03LqIPLJly/4kJGQ1Fy8m4OTkwLvv1qRv31q4uVntkhUiudI9/0W1a9eORYsW8eST\nT962ecwwDEwmE4cOHbJJQBHJ/gzD4PPPd3DxYgLVqj1BZGQQTz7pb+9YIjnSPYt90aJFABw+fPiO\nx1JTU62XSERyBLPZICEhFR8fV0wmExMnBrFmzSlee60Kjo7aCihiLZn+6+rUqdNtt81mM+3bt7da\nIBHJ/v78M4Z27X6gV68llj3dy5bNy7/+VU2lLmJl91xj7969O7///jsAFSpU+O8vODnRuHFj6ycT\nkWwnNTWDzz77g8jIraSkZJA/vzvnzt2gWDEfe0cTyTXuWeyzZ88GIDQ0lKFDh9oskIhkTzt2XKRf\nvxUcOnQNgM6dn2LkyOfIm9fdzslEcpd7FvuaNWto1KgRTz31FD/99NMdj7/wwgtWDSYi2cfIkeuY\nPn0HhgElSvgSEdGUBg10dkoRe7hnse/bt49GjRpZNsf/LxW7iPzF3d0ZBwcTb71VnQED6uDh4Wzv\nSCK51gOdUjYhIYGLFy9SrpwdLqGoU8panU4RaRs5YZyvXEnk5MlYatUqAtw6PeyxY9epVClrHMKW\nE8Y4q9MY28bDnFI2091T//Of/zBkyBBiYmJo0aIF7733HpMmTXqogCKSvRmGwbx5+6lffyavvrqY\n69dvAuDq6pRlSl0kt8u02OfNm8fgwYNZsmQJTZo0YfHixWzYsMEW2UQkCzlx4jodOvzI+++vIDY2\nhUqV/ElOTrd3LBH5H/d1Lkc/Pz/WrVtH9+7dcXJyIiUlxdq5RCSLSEvLYPr0HUREbCE5OYO8ed0Y\nPbohHTpU1EVbRLKgTIu9bNmyvPHGG5w7d446derw/vvvU7lyZVtkE5Es4M03l7J48TEAOnSoyKhR\nDXTRFpEsLNNiDwsLY9euXQQEBODi4kLbtm157rnnbJFNRLKAV199hj17rjB+fBMaNy5p7zgikolM\niz0tLY01a9YwduxYMjIyqFWrFrVr18bJSVdkEsmJ1qw5xY4dFxkwoA4A9esXZ8uWnjg7O9o5mYjc\nj0x3nhs1ahTJycmEhYUxbtw40tPTGTFihC2yiYgNXbt2kz59ltGp00LGj9/Cjh0XLY+p1EWyj0xX\nuw8cOMAvv/xiuT18+HBatGhh1VAiYjuGYbBgwSGGDVtLTEwybm6ODBxYl6efLmDvaCLyEDItdsMw\niI+Px8fn1kUc4uPjcXTUp3eRnOD06TgGDYpizZrTAAQGFmPChKaULp3HzslE5GFlWuw9e/akQ4cO\nliu6rV69mt69e1s9mIhY3+TJ21iz5jR+fq6MGtWQTp2e1CFsItlcpsXevn17KleuzPbt2zGbzUyZ\nMoXy5cvbIpuIWEFaWoblO/OhQwNxcDAxeHBdChTwtHMyEXkc7lnsZrOZ7777jlOnTlG9enW6du1q\ny1wi8pglJaUREbGFtWtPs3x5F1xcHMmXz52JE4PsHU1EHqN77hU/cuRIli9fjru7O59//jlTp061\nZS4ReYzWrz9Dw4azmTr1Dw4ciGbjxrP2jiQiVnLPNfbt27ezdOlSTCYTvXr1okePHrzzzju2zCYi\nj+j69ZuMHLmeefMOAFCxYj4iI5tRvXohOycTEWu5Z7G7urpadqLJkyePdqgRyWZ+/fUYAweu4urV\nJFxdHenXrzZvv10DFxcd1SKSk92z2P+3yB0cMj2XjYhkIXFxKVy9mkTdukWZODGIMmV0CJtIbnDP\nYr9w4QJDhgy55+2xY8daN5mIPJCMDDP790fzzDMFAXj55afIk8eN558vg4ODtriJ5Bb3LPaQkJDb\nbj/77LNWDyMiD+fgwWj69VvJwYPRrF3bndKlb319Fhxc1t7RRMTG7lns7dq1s2UOEXkIycnpREZu\nZerUP0hPN1OokBdXriTqzHEiuZgu0SaSTW3efJZ+/VZy4kQscOvyqkOH1sfb29XOyUTEnlTsItnQ\njBk7GTZsLQABAXmZODGIWrWK2DeUiGQJ97Wre1JSEocPH8YwDJKSkqydSUQy0bhxSby8XBg4sA6r\nVr2iUhcRi0yLfcuWLbRt25Y+ffoQHR1N48aN2bhxoy2yicj/u3jxBpGRWzEMA4By5fKya9frDBxY\nB1dXbXgTkf/KtNgjIyOZO3cuPj4+FChQgG+//Zbx48fbIptIrmc2G3z99W7q1ZtFePhmFiw4ZHnM\n19fNjslEJKvK9KO+2WzG39/fcrtsWR0+I2ILR45co1+/lWzffgGA4OAy1K9fzM6pRCSry7TYn3ji\nCdasWYPJZCI+Pp7vvvuOwoUL2yKbSK6UkpLO5Mm/8+mnv5OWZqZAAU/CwxvTqlU5e0cTkWwg003x\no0aNYvHixVy8eJGmTZty6NAhRo0aZYtsIrnSN9/sYeLEraSlmenWrTKbNvVQqYvIfct0jT1fvnxE\nRkbaIotIrmUYhuX6DK+++gybN5/jrbeqU6dOUTsnE5HsJtNib9y48V2v7LZq1SqrBBLJbZYt+5OJ\nE7fyn/+0J08ed1xdnZg9u629Y4lINpVpsc+ZM8fyc3p6OitXriQ1NdWqoURyg8uXExgyZA1LlhwD\nYObMvfTtW8vOqUQku8v0O/YiRYpY/itRogSvv/46UVFRmU7YbDYzfPhwOnXqRLdu3Th9+vRdnzds\n2DAiIiIePLlINmU2G8yevZd69WaxZMkxPD2dCQtrxHvv1bR3NBHJATJdY9++fbvlZ8MwOHbsGCkp\nKZlOOCoqitTUVObPn8/u3bsJDw9n+vTptz3n+++/5+jRo9SsqT9okjscP36dDh1+ZP36Wx90g4JK\nMW5cE4oW9bFzMhHJKTIt9k8//dTys8lkIk+ePISHh2c64R07dhAYGAhAlSpV2L9//22P79y5kz17\n9tCpUydOnDjxoLlFsqULF26wfv1p8uf3ICysEW3bBtx1HxYRkYeVabEHBwfTpUuXB55wQkICXl5e\nltuOjo6kp6fj5OTElStX+Oyzz5g6dSrLli2772n6+3s/cA55MBrjx+/UqVhKlvQD4MUXn+Kbb9Jo\n06Y8efO62zlZzqZl2fo0xllTpsU+d+7chyp2Ly8vEhMTLbfNZjNOTrdebvny5Vy/fp3evXsTHR1N\ncnIypUuX5sUXX/zHaUZH33jgHHL//P29NcaPUUJCKmFhG/nmmz0sWvQStWvfOnStZ88qREff0Fhb\nkZZl69MY28bDfHi6rzPPde/enWeeeQZX1/9e5/mdd975x9+rVq0aa9asoUWLFuzevZuAgADLY927\nd6d79+4ALFy4kBMnTmRa6iLZyYoVJxg8eBXnz9/A0dHE/v3RlmIXEbGmTIu9SpUqDzXhoKAgNm3a\nROfOnTEMg7CwMBYvXkxSUhKdOnV6qGmKZHVXriQydOhafvrpCABVqhRk4sQgKlcuYOdkIpJbmIy/\nrgP5PxYtWkS7du1snefeJpqI7h5v7xQ5mjatPZrNm8/Ss+cvxMam4OHhREhIPV5/vSpOTrcfVapx\ntj6NsfVpjG3jYTbF3/M49tmzZz9SGJHcJiAgHw4OJho1KsH69T14883qd5S6iIi1ZbopXkTuLi0t\ng7lzD/Dyy0/h4uJI/vwerFjRlWLFfHQIm4jYzT2L/dixYzRp0uSO+/+6WIXOFS+52e7dl+jbdyUH\nDkQTE3PTcirY4sV97ZxMRHK7exZ7iRIl+OKLL2yZRSTLS0xMIzx8E19+uQuz2aB4cV+qVn3C3rFE\nRCzuWezOzs4UKVLElllEsrTVq08ycOAqzp6Nx8HBRJ8+1Rk4sC6ens72jiYiYnHPYq9WrZotc4hk\naZs2naVz50UAVK5cgMjIIJ55pqCdU4mI3OmexT58+HBb5hDJ0urWLUpQUCnq1Cmqvd1FJEvTXyeR\nuzh9Oo7u3X/mxInrwK0LIH377Qu8805NlbqIZGk63E3kb9LTzcyYsZPx4zdz82Y6jo4mvvmmDYAO\nYRORbEHFLvL/9u27Qt++K9i79woAL75YntGjG9k5lYjIg1GxS66XlJRGRMQWpk/fQUaGQdGi3owf\n34SmTUvbO5qIyANTsUuud/78Db744tZx6b17VyUkpB5eXi72jiUi8lBU7JIrxcen4O3tgslkoly5\nvIwb15gKFfJTvXohe0cTEXkk2r1XchXDMFi48DC1a3/Njz8ettzftWtllbqI5Agqdsk1zp6Np0uX\nRbz55lKuXr3JsmV/2juSiMhjp03xkuNlZJj56qvdhIVtIikpDV9fV0aMeI4uXSrZO5qIyGOnYpcc\n7cKFG7z22mJ27rwEQJs2AYwZ04iCBT3tnExExDpU7JKj5c3rTlxcCoUKeTFuXBOaNy9j70giIlal\nYpccZ8uWc1SsmB8/Pzfc3JyYNasNhQp54e3tau9oIiJWp53nJMeIjU2mX78VtG37Ax9/vN5yf0BA\nPpW6iOQaWmOXbM8wDJYsOUZIyGqio5NwcXGkaFEfDMPQ+d1FJNdRsUu2duHCDUJCVrN8+XEAatUq\nwsSJTQkIyGfnZCIi9qFil2zr8uVEAgNnceNGKt7eLgwbFkj37k/j4KC1dBHJvVTskm0VLOhJy5bl\niI9PYezYRhQq5G3vSCIidqdil2wjJSWdyZN/p2HDEtSqVQSAiIimuLg42jmZiEjWoWKXbGHr1vP0\n77+SY8diWLLkGGvXdsPR0UGlLiLyP1TskqXFx6cwevQGZs3aC0DZsnkYP74Jjo46UlNE5G5U7JJl\nLV36JyEhq7h0KRFnZwfefbcmH3xQCzc3LbYiIveiv5CSJcXHp9Cv3wpiYpKpXr0QkZFBVKyY396x\nRESyPBW7ZBlms4HZbODk5ICPjytjxzYmJuYmPXs+o03vIiL3ScUuWcKxYzH077+Sxo1L8sEHtQBo\n166CnVOJiGQ/Wg0Su0pNzWDixK00ajSHrVvPM2fOPlJS0u0dS0Qk29Iau9jN9u0X6N9/JYcPXwOg\na9dKjBjxHK6uWixFRB6W/oKKzaWkpDNy5Hq+/no3hgGlSvkxcWJT6tcvbu9oIiLZnopdbM7FxZHD\nh6/i4GDinXdq0q9fLdzdne0dS0QkR1Cxi01cvpxIerqZIkW8MZlMREY2IzExjUqV/O0dTUQkR9HO\nc2JVhmHw3Xf7qF9/Ju+99xuGYQC3Nr+r1EVEHj+tsYvVnDhxnf79V7Jp0zkAnJ0dSExMw8vLxc7J\nRERyLhW7PHZpaRlMm7aDiIgtpKRkkC+fO6GhDXnxxQqYTLpWuoiINanY5bHKyDDTqtX37Np1GYCO\nHZ/k448bkC+fu52TiYjkDip2eawcHR0ICirNtWvJREQ0pWHDEvaOJCKSq6jY5ZGtXn2S1FQzzZuX\nAeC9957lrbdq4OmpQ9hERGxNxS4P7erVJIYOXcvChYfJl8+dZ5/tSd687ri4OOLi4mjveCIiuZKK\nXR6YYRj88MMhRoxYS0xMMu7uTrzzTk18fFztHU1EJNdTscsDOXUqloEDV7Fu3WkAnnuuOBERTSlZ\n0s/OyUREBFTs8gAMw+C11xazf380efK4MWpUQzp2rKhD2EREshAVu2TKMAxMJhMmk4nQ0IbMnr2P\n0aMb4u/vYe9oIiLyP1Tsck9JSWmMH7+ZxMQ0JkxoCkDdusWoW7eYnZOJiMi9qNjlrtatO82AAVGc\nPh2Ho6OJt9+uoe/RRUSyAasVu9lsZuTIkRw5cgQXFxdCQ0MpUeK/JytZsmQJs2bNwtHRkYCAAEaO\nHImDg65JY28xMTcZMWId8+cfBKBixfxMmhSkUhcRySas1qRRUVGkpqYyf/58+vfvT3h4uOWx5ORk\nJk+ezOzZs/n+++9JSEhgzZo11ooi92nevFtXYZs//yCuro589FF9oqK6Uq1aIXtHExGR+2S1NfYd\nO3YQGBgIQJUqVdi/f7/lMRcXF77//nvc3W+dPzw9PR1XVx0DbW+//Xacq1dvUq9eUSIigihTJo+9\nI4mIyAOyWrEnJCTg5eVlue3o6Eh6ejpOTk44ODiQP39+AObMmUNSUhL16tXLdJr+/t7WipsrZWSY\nuXgxgaJFfQCYOLEZDRuWpEePZ3QIm5VpWbY+jbH1aYyzJqsVu5eXF4mJiZbbZrMZJyen225PmDCB\nkydPMmXKlPsqkujoG1bJmhvt3x9N//4rSExMY9WqV3B1dcLf35uWLctw9WqCvePlaP7+3lqWrUxj\nbH0aY9t4mA9PVvuOvVq1aqxfvx6A3bt3ExAQcNvjw4cPJyUlhWnTplk2yYv13byZRmjoBpo1+45d\nuy6TkJDK6dNx9o4lIiKPidXW2IOCgti0aROdO3fGMAzCwsJYvHgxSUlJVKpUiQULFlCjRg169OgB\nQPfu3QkKCrJWHAE2bjxD//5RnDwZi8kEvXpV4aOP6uPl5WLvaCIi8phYrdgdHBwYNWrUbfeVKVPG\n8vPhw4et9dJyFyNHrmPatB0AVKiQj4kTg6hZs7CdU4mIyOOmE9TkEhUq5MfFxZG+fWvx7rs1dVlV\nEZEcSsWeQ50/f4OdOy/SuvWtfRs6dXqSOnWKUqKEr52TiYiINanYc5iMDDMzZ+4hNHQj6elmnnzS\nnzJl8mAymVTqIiK5gIo9Bzl06Cr9+q1kx46LALRsWRYvL2c7pxIREVtSsecAycnpTJ68jSlTtpOW\nZqZgQU/CwxvTsmU5e0cTEREbU7HnAIMHr2LevAMA9OjxNMOGBeLjo1P0iojkRir2HOC9955l374r\nhIU1pnbtIvaOIyIidqTrpGZDS5Yc4513lmMYBgBlyuRh1apXVOoiIqI19uzk0qUEQkJWs3TpnwC0\nalWO5s1vnfRHF20RERFQsWcLZrPB7Nl7GT16AzdupOLp6czQofUJCipl72giIpLFqNizuGPHYujX\nbyXbtp0H4PnnSxMe3oQiRXS5RBERuZOKPYv76acjbNt2Hn9/D8aObUzr1uW02V1ERO5JxZ4Fxcen\nWA5Xe++9mmRkmHnzzer4+bnZOZmIiGR12is+C7lxI4XBg1dRp843xMTcBMDV1YmQkHoqdRERuS8q\n9ixi+fLj1K8/i2++2cP168ls3nzO3pFERCQb0qZ4O7t8OZEPP1zN4sXHAKhatSCRkc146il/OycT\nEZHsSMVuR4sXH6Vfv5XExaXg4eHMkCH1eP31Kjg6akOKiIg8HBW7HeXN605cXApNmpRk/PimFCvm\nY+9IIiKSzanYbSgtLYP168/QpMmtE8vUq1eMZcteplq1J3QIm4iIPBba5msjO3deJCjoO15+eRGb\nN5+13F+9eiGVuoiIPDZaY7eyhIRUwsM38e9/78ZsNihRwhcHBxW5iIhYh4rdiqKiTjBo0CrOnbuB\no6OJt9+uwcCBdfDwcLZ3NBERyaFU7FYya9ZeBg6MAuDppwswaVIzKlcuYOdUIiKS0+k7ditp3boc\nxYr5MHLkcyxf3kWlLiIiNqFif0xOnoylf/+VpKSkA7cOZduy5VX69KmBk5OGWUREbEOb4h9RerqZ\nzz/fwYQJW7h5M53ixX15//1nAXBxcbRzOhERyW1U7I9g797L9O27kn37rgDQvn0FunatZOdUIiKS\nm6nYH0JSUhrjx2/m8893YjYbFCvmw4QJTWjcuJS9o4mISC6nYn8Iy5cfZ9q0HTg4mHjjjWoMHlwX\nLy8Xe8cSERFRsd+vtLQMnJ1vfWferl15tm+/QMeOT1K16hN2TiYiIvJf2l07E4ZhsGDBIZ599mtO\nnLgOgMlkYuzYxip1ERHJclTs/+DMmTg6d15Inz7LOH/+Bt9+u8/ekURERP6RNsXfRUaGmS+/3EV4\n+CaSktLx83Pl448b0LnzU/aOJiIi8o9U7P/j6NFrvPPOcnbvvgzACy+UJzS0IQUKeNo5mYiISOZU\n7P/DwcHEoUNXKVzYi/Hjm9KsWWl7RxIR+Uc7d/7B8OFDKFmyFCaTicTERAoXLsKIEaE4Oztz/fp1\nPvtsMpcuXcRsNlOgQEHefbcv+fLlB2DPnl18882XpKenk5ycTIsWrXnxxZfs+p7i4mKZMeMzBg36\nyK45UlKSGTVqGNevX8fDw4OPPvqYPHny3Pacb7+dSVTUCjw9PenSpTv16gUSHx/HqFHDSExMxNfX\nl8GDh5InT16++moGjRsHUaqU9bpF37EDe/ZcxjAMAMqWzcusWW3ZuLGnSl1Eso3q1WswdeoXTJky\ng6+//hYnJyc2blyHYRh89NFAGjRoxNSpXzBt2r9p2bINgwb1JSMjg/PnzzF58gSGDx/N1Klf8Nln\nX7B8+a9s3brZru/nyy+n8+KLHe2aAWDRogWULl2WadP+TfPmLZk166vbHj9+/E9WrvyNGTO+ITJy\nKl999TnJycnMnv0NTz9dhenTv6J9+07MmPEZAB07duGzzyZbNXOuXmO/fv0mI0euZ968A0yd2pyO\nHZ8EoHHjkvYNJiLZls+qDrieX/FYp5lSpBnxTRbc9/PT0tK4du0q3t4+HDlyCC8vLwIDG1oer1mz\nFosX/8SePbvYvXsnzZu3JG/efAC4uroRGTkVd3f326Z59uwZxo0LJS0tDTc3N6ZO/ZQxY8bQpEkz\nateuy9atm1m1agUffTSS9u1bUaJESUqWLMWmTRuYOXMe7u7uzJ07B0dHBxo2bML48WGkpCTj6urG\noEEfUrDgf48ySkxM4NChgwwYUA6AH3+cz7p1a7h58yZ+fn6EhUWwcuVyfv31F8xmM716vUF8fDzz\n53+Hg4MDTz9dhbfeepcrVy4TERFOamoK165d5V//6sNzz/13HM6dO0t4+Ojb3mdQUHPatn3Rcnvv\n3j106dIdgNq16zFz5u3FfurUSapWrY6rqysARYsW588/j3Hq1Al69+4DwNNPP8OkSeMB8Pb2xtXV\nlT//PEbZsuXue54+iFxZ7IZh8PPPR/nwwzVcvZqEi4sjsbHJ9o4lIvLQduz4g3fe6U1s7HVMJhNt\n2rxIjRrPsmrVSgoXLnrH8wsXLsKlSxe5ejWacuUCbnvMy8vrjud/9tlkXnmlJ7Vr12XjxnUcPHjw\nnlmuXLnM119/i6+vH05Ozqxdu4rg4FZERS1n0qTPmDhxHB06dKJOnXr88cfvfP75VEaMCLX8/oED\n+yIGqeQAABCrSURBVClevAT8X3v3HhVltT5w/MsdvAEm8tMEU0I0QZMUCy8hSpkihKiogWmWmibi\n0ZZaOFoiong3lTz6I0PNW+aNo3khIwmOpqgnLS+FFuU9BoWRAZr3/MFhkkQQEoeZns9arMW8e953\nP/OsWTzsPe/sDeh0OnJzc1m8eAXm5ub84x9v8d13p4GSIhkXt5Bbt3IZO/Z1Vq9OwtbWllmzpnP0\naAZgxuDBr+Dt3ZH//Ocka9Z8WKawN2vmwgcfrKowr/n5+fp81KlTh/z8vDLtbm5Psm5dIhpNPkVF\nRXz77SmCgkJwd/fg8OFUWrVqzeHDqRQUFNx1jjuZmceksD8s2dm3mDLlIPv3ZwHw3HOPs2BBAE8+\n2dDAkQkhTEFVRtYP0zPPdOS99+aQm6tm4sRxNGnSFAAnJyeuXPn1nudnZ/9Ep06duXHjOteuXS3T\ndv78ORRFR6tWrfXHfvrpEp6e7QDo2vV5nJzqs3XrZ/r20o8zAeztHbC3dwCgX7+XmT8/jubNn8DF\npTn29g78+OMFkpISWb9+LQAWFmVLkVqtpmHDkr/J5ubmWFlZMXPmu9jZ2XHt2jWKi0t20Swt/tnZ\nP6NW5zB5ciQAGo2GX37Jpl27Dqxdu4bk5B2Amf68P3JQ+Yi9bt26aDT5+uv++Z+eJ55oQWjoICZN\nGk/jxv/HU0+1xd7egYiI4SxePJ9x497guee64OzsrD/nsccacePGdWrK36qwHzt2mQEDtpKfX0SD\nBjaoVN0ID/fC3NzM0KEJIcRDYW/vwPTps4iMHEPr1hvw8mrPzZs3OXw4la5duwOQkfE12dnZPP20\nN02bPs60aZPx938BR0dHNBoN8fGxjBjxepnrNm/egu++O02nTp3Zt28POp0Wa2trbt68AcC5c9/r\nn2tu/sftWy4uroDChg1JhIQMAMDV9QmGDAnHy6s9ly5dJDPzWJm+HB0duX37NgAXLpwnNfUQ//zn\nWgoKChg5Mlz/PDOzkn6aNHmcxo2dWbx4BZaWlvzrX7twd2/F6tUJ9Ov3Ms8914Xk5J3s2bO7TD8P\nMmL38mpPenoaTz3lSUZGGu3bdyjTnpOTg0ajYeXK/ycvL4+JE8fRsqUb//53Ov36vYyXV3sOHTqI\nl1d7/Tm3b9/CwcHxz109NH+rwu7p6UTTpvXx8HiMOXN64Ox873STEEIYuxYtWjJgQBiLF8cTEzOX\nefMWsWTJApKSEgFo3NiZ+PjFWFhY0KRJU8aOjeTdd9/G3NwcjUbzv2LYtcw1x42bQHx8LGvXrsHW\n1pYlSxbxxBOtmDPnffbt2/u/Al6+vn2DWbMmAW/vjvprLVgQR2FhIVptARMmTC7z/LZtvVi5chlQ\nUnzt7Ox4883XgPJHu46OjoSFvcJbb43i999/p0mTpvj7B9CjR0+WL1/CunUf4eTUGLVaXeVchoQM\nICZmBm++ORIrKyv9RwYbN66jWTMXunTpzsWLWbz++jCsrCwZN24CFhYWuLo2JyZmBgCNGjkxbdp0\n/TXPnDnN6NHjqhzLgzJT7p4/qc0WmHF92K0qnVJQUMzKlccYPrwdjo4lN4Ko1QU4ONjWRIRGz8mp\nPtev3zZ0GCZP8lzzJMc1r6ZzHB8fS3Bw/zIfB5iCW7dyiYmZybx5ix7o+U5O9avch8l+3S09PRt/\n/yTmzElj5sxU/XEp6kIIUfu9/voYPvvMMPcr1KRNmzbU6GgdTHAqPje3gPff/4qkpJJ13d3dGzJk\niKeBoxJCCFEVjo4NmTIl2tBhPHRvvPFmjfdhUoV99+7zTJuWwtWr+VhZmRMZ6UNUlA82Nib1MoUQ\nQoj7MpmKd/r0dV57bRcAHTs2YeHCAFq3bmTgqIQQQohHy6gLu6IomJmVfFWtbVsnRo/2pkULB4YP\nby9fYRNCCPG3ZLQ3z507d5Pg4M2kp2frj82a5cdrrz0tRV0IIcTfltEVdq22mPj4dPz915GR8Qtz\n5xp2owIhhBCiNqmxqXidTsfMmTM5e/Ys1tbWxMTE0Lx5c317SkoKy5cvx9LSktDQUAYNqmQXnxZ9\nOHLkVyZN2s/ZszcBCA/3RKXqXlMvQQghhDA6NVbYDxw4QGFhIZs2beLEiRPExcWxcuVKoGTnoTlz\n5rB161bs7OwYMmQI/v7+NGp0/5vd3koZx4oVG1EUaNnSgYULA/D1damp8IUQQgijVGNT8ceOHaNb\nt24APP3003z77bf6th9++AFXV1fs7e2xtrbmmWee4ejRoxVeb9Om01hYmBMV5cOhQ8OkqAshhBDl\nqLERe15eXpldcCwsLCguLsbS0pK8vDzq1/9jmby6deuSl5dX3mX0rl9/u6ZCFXepzvKFouokzzVP\nclzzJMe1U42N2OvVq0d+fr7+sU6nw9LSsty2/Pz8MoVeCCGEENVTY4Xd29ub1NSSNdpPnDhBq1at\n9G1ubm5cunQJtVpNYWEh33zzDR06dLjfpYQQQgjxgGpsd7fSu+LPnTuHoijExsZy5swZNBoNYWFh\n+rviFUUhNDSUV155pSbCEEIIIf5WjGfbViGEEEJUyugWqBFCCCHE/UlhF0IIIUxIrSvsOp0OlUpF\nWFgYERERXLp0qUx7SkoKoaGhhIWFsXnzZgNFadwqy/Hu3bsZOHAggwcPRqVSodPpDBSp8aosx6Wm\nT5/O/PnzH3F0pqGyHJ86dYqhQ4cyZMgQIiMj0Wq1BorUuFWW5507dxISEkJoaCgbNmwwUJSm4eTJ\nk0RERNxzvMp1T6llPv/8c2XKlCmKoihKZmamMmbMGH1bYWGh0qtXL0WtVitarVbp37+/cv36dUOF\narQqyvGdO3eUnj17KhqNRlEURZk4caJy4MABg8RpzCrKcalPPvlEGTRokBIfH/+owzMJFeVYp9Mp\nQUFBysWLFxVFUZTNmzcrP/zwg0HiNHaVvZe7dOmi5OTkKFqtVv/3WVTdqlWrlMDAQGXgwIFljlen\n7tW6EfvDXrFO3KuiHFtbW7Nx40bs7OwAKC4uxsbGxiBxGrOKcgxw/PhxTp48SVhYmCHCMwkV5Tgr\nKwsHBwc++ugjwsPDUavVtGzZ0lChGrXK3sseHh7cvn2bwsLCMltpi6pxdXVl2bJl9xyvTt2rdYX9\nfivWlbZVdcU6ca+Kcmxubq5fsz8pKQmNRkOXLl0MEqcxqyjH165dY/ny5ahUKkOFZxIqynFOTg6Z\nmZmEh4eTmJhIRkYG6enphgrVqFWUZwB3d3dCQ0Pp27cvfn5+NGjQwBBhGr0XX3xRv4jb3apT92pd\nYZcV62peRTkufTx37lzS0tJYtmyZ/AdeDRXleO/eveTk5DBq1ChWrVrF7t272bZtm6FCNVoV5djB\nwYHmzZvj5uaGlZUV3bp1u2ekKR5MRXn+/vvvOXToEAcPHiQlJYXffvuNPXv2GCpUk1SdulfrCrus\nWFfzKsoxgEqlQqvVsmLFCv2UvKiainI8bNgwtm3bRlJSEqNGjSIwMJD+/fsbKlSjVVGOXVxcyM/P\n19/o9c033+Du7m6QOI1dRXmuX78+tra22NjYYGFhQcOGDbl165ahQjVJ1al7NbYJTHUFBASQlpbG\n4MGD9SvW7dq1S79i3dSpUxk5cqR+xTpnZ2dDh2x0Ksqxp6cnW7dupWPHjrz66qtASSEKCAgwcNTG\npbL3sfjrKsvx7NmzmTRpEoqi0KFDB/z8/AwdslGqLM9hYWEMHToUKysrXF1dCQkJMXTIJuGv1D1Z\neU4IIYQwIbVuKl4IIYQQ1SeFXQghhDAhUtiFEEIIEyKFXQghhDAhUtiFEEIIE1Lrvu4mhCnKzs6m\nd+/euLm5lTmekJBAkyZNyj2ndHnJ8ePHV7vfbdu2ERcXp++joKAAHx8fZsyYUe4qVxVZsmQJnp6e\n9OzZk4iICJKSkgAIDg5mx44d1Y4RICIigitXrlCnTh2gZLUtFxcX5s+fr18JsTybNm2ibt26BAYG\n/qX+hTAlUtiFeEQaN278lwtgdfj7+xMXFwfA77//TkREBOvXr9evU/CgJkyYoP/9yJEj+t8f1muK\niYmhc+fOQMnqZpGRkSQmJvL222/f95zMzEx8fHweSv9CmAop7EIY2Llz55g1axYajYbffvuNESNG\nMGzYMH17UVER77zzDufPnwdg6NChDBo0iBs3bqBSqbhy5QpmZmZMmjQJX1/fCvuysLCgQ4cOXLx4\nEYBPP/2UxMREzMzMaNu2LdOnT8fa2rrc/qZOnYqPjw9nzpwBYODAgWzZsgUPDw9Onz6Nn58f27dv\np1GjRqjVagIDA/niiy9IT09n6dKlFBcX06xZM2bNmoWjo2OFcWo0GnJycmjXrh0Ae/bsITExkYKC\nArRaLTExMRQVFZGSkkJGRgZOTk60adOmyvkQwhTJZ+xCPCLXrl0jODhY/7N69WoAtmzZwtixY/n0\n00/5+OOPWbRoUZnzMjMzyc3NZfv27SQmJnL8+HEAZs+eTWhoKNu2bWPlypWoVKpKN4fIyckhNTUV\nb29vzp49S0JCAklJSezatQs7Ozs++OCD+/ZXKjo6Wh93KUtLS3r37s3evXsB2LdvH7169eL27dss\nWLCANWvWsH37drp27Xrf/eejo6MJCgqia9euhIWF4evry/Dhw9HpdGzcuJGEhAR27tzJG2+8wZo1\na/D19cXf35/IyEi6detWrXwIYYpkxC7EI3K/qfipU6fy1Vdf8eGHH3L27Fk0Gk2Zdnd3d7Kyshg5\nciTdu3dn8uTJAHz99df8+OOPLF26FCjZYvfnn3+mTZs2Zc5PSUkhODgYRVFQFIWAgAACAwNZv349\nPXr00I+ew8LCmDZtGqNGjSq3v8oEBwcTGxtLeHg4u3fvJioqipMnT3L58mX9DIROp8Pe3r7c80un\n4o8fP05kZCTPP/881tbWACxfvpyUlBSysrI4cuQI5ub3jkkeNB9CmDop7EIYWFRUFA0aNKBHjx70\n6dOH5OTkMu2Ojo4kJyeTlpbGl19+SUhICMnJyeh0OtauXYuDgwMAV69eLfdGs7s/Y7+bTqcr81hR\nFIqLi+/bX2W8vLzIzc3l1KlTXL16FW9vbw4cOIC3tzcJCQkAaLXaMjtVlcfb25uIiAimTJnCjh07\n0Gq1hIaGEhwcTKdOnfDw8GD9+vXlvp4HyYcQpk6m4oUwsLS0NCIjI+nVqxdHjx4FSm5yK3Xw4EEm\nT56Mn58f0dHR1KlTh8uXL/Pss8+yYcMGAC5cuEBQUBB37tx54H59fHxISUlBrVYDsHnzZjp37nzf\n/u725z25S/Xr148ZM2bQp08fANq3b8+JEyfIysoCYMWKFcybN6/S2EaMGMGdO3fYuHEjFy9exNzc\nnDFjxvDss8+Smpqqz4+FhYX+97+aDyFMhYzYhTCw8ePHM3ToUBo0aECLFi14/PHHyc7O1rd3796d\nzz//nL59+2JjY8MLL7yAh4cH0dHRqFQq+vXrB8C8efOoV6/eA/fbunVrRo8eTUREBEVFRbRt25b3\n3nsPGxubcvu7W8+ePQkODr5nH/mgoCCWLFnCwoULAXByciI2NpaoqCh0Oh3Ozs7Ex8dXGpu1tTVR\nUVHExsayf/9+2rRpw0svvYStrS2dOnXi119/BcDX15eFCxdSv379v5wPIUyF7O4mhBBCmBCZihdC\nCCFMiBR2IYQQwoRIYRdCCCFMiBR2IYQQwoRIYRdCCCFMiBR2IYQQwoRIYRdCCCFMiBR2IYQQwoT8\nF/QOUm6/OlXlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25a9ae0f710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "f1 = [f1_score(y_test, (y_test_prob > t).astype(int)) for t in thresholds]\n",
    "best_threshold_idx = np.argmax(f1)\n",
    "best_threshold = thresholds[best_threshold_idx]\n",
    "\n",
    "plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % score)\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic example')\n",
    "plt.legend(loc=\"lower right\")\n",
    "\n",
    "plt.scatter(fpr[best_threshold_idx], tpr[best_threshold_idx], s=50, c='b')\n",
    "\n",
    "print('best f1 score          :', np.max(f1))\n",
    "print('best f1 score threshold:', best_threshold)\n",
    "print('false positive rate    :', fpr[best_threshold_idx])\n",
    "print('true positive rate     :', tpr[best_threshold_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precision-recall кривая\n",
    "Иногда в задачах классификации классы могут быть сильно несбалансированы. Например, допустим, что при определении мошеннических транзакций (fraud detection) лишь одна из тысячи транзакций являются мошенническими (т.е. 0.1%). Если создать модель, которая всегда будет возвращать 0 (т.е. транзакция не мошенническая), то она будет работать с точностью 99.9%. Однако очевидно, что эта модель бесполезная. Для оценки моделей с такими сильно несбалансированными значениями классов можно использовать precision-recall кривую. Кривая показывает зависимость изменения точности по отношению к полноте при изменении порога верятности. Чтобы увеличить полноту необходимо понижать порог вероятности. Однако это ведет к сниженгию точности, так как при этом будут появляться все больше ложно положительных результатов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.504239Z",
     "start_time": "2018-02-02T16:34:10.494739Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average precision-recall score: 1.00\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import average_precision_score\n",
    "average_precision = average_precision_score(y_test, y_test_prob)\n",
    "\n",
    "print('Average precision-recall score: {0:0.2f}'.format(average_precision))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-02T16:34:10.665240Z",
     "start_time": "2018-02-02T16:34:10.507739Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "precision [ 0.89108911  0.89        0.8989899   0.90816327  0.91752577]\n",
      "recall [ 1.          0.98888889  0.98888889  0.98888889  0.98888889]\n",
      "thresholds [ 0.04947608  0.05042727  0.0564132   0.05862503  0.07422455]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfYAAAFlCAYAAAAZGcpRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtYVXW+x/HPZm8RBBR4SjuZOomXmqNJ6pSXIROvSaXG\nKHivcfI4mc445i2TnDIl5YxzRk3zTI0nnRR1vJddEIs0K29o5C0tOeFRxMILbGUDe50/HPeJkyKg\nC/TH+/U8Po9rrb1++7u+Ip+9fnvttR2WZVkCAABG8KvqAgAAwI1DsAMAYBCCHQAAgxDsAAAYhGAH\nAMAgBDsAAAZxVXUBqN7WrVunN954Qw6HQ4GBgZoyZYpatmxZ5v0nTZqkpk2bavjw4bbVGB0drRo1\naiggIEAOh0OFhYXq2LGjJk2aJD+/639tvHnzZm3fvl0vvPDCVR8zZcoUxcTEqEOHDtf9fNKlvm3b\ntk3h4eGSJK/XK7fbrfj4eD399NM35Dl+bMiQIRo0aJBatGihxx57THv27Lnhz2GXxMRELV26VCkp\nKbrjjjt864cMGaLjx48rJCTE93PRokULTZs2TYGBgWUae9WqVUpJSdHChQuvuP3YsWN6/vnndebM\nGdWqVUuvvvqqIiIifPu+8cYbKi4uVvv27fXCCy+oRo0a13/AuPVZQBU5evSo1bFjRys7O9uyLMv6\n6KOPrE6dOpVrjIkTJ1p//etfbaju/3Tu3Nnat2+fb7mgoMDq37+/tWTJEluf105X6tvx48etVq1a\nWUeOHLnhzzd48GBr06ZN1nfffWdFRkbe8PHtcvHiRevBBx+0/vCHP1izZ88use3yMV3m9Xqt0aNH\nW4mJidccNzc315o6darVqlUra8SIEVd9XGxsrLV+/XrLsi79/+jVq5fl9XqtQ4cOWQ899JD1/fff\nW8XFxdbYsWOtRYsWVfAoYRrO2FFl/P39NX36dNWtW1eS1KJFC50+fVoej0f+/v4lHpufn6/p06dr\n9+7dcjqd6tq1q8aOHVviMatWrVJycrIKCwt19uxZPf300xo4cKBycnI0ceJE5ebmSpI6deqk3//+\n91ddX5a627Rpo2+++UZZWVkaNGiQIiIidPz4cS1ZskRZWVlKSkrShQsX5HA4NHr0aHXu3FmS9Prr\nr2vNmjVyuVxq1KiREhMT9eGHH+r999/X66+/rg8++EALFiyQw+GQ0+nUhAkT9Itf/MJ3xtuzZ0+l\npKRo3rx5Ki4uVnBwsCZPnqz77rtPc+fO1fHjx5WTk6Pjx48rPDxcc+bMUb169cr073Hy5ElJUnBw\nsCRp9+7d5ToOp9OpadOm6dixYzp79qyCgoKUlJSkxo0bl+n5t2zZoj//+c/yer2qVauW/vjHPyo4\nOLjEGX5WVpZvefXq1Vq1apUuXLig4OBgFRYW6sknn1TPnj0lSUlJSbIsS+PHj9fKlSu1bNkyeb1e\nhYaGaurUqYqIiNCXX36pF154QevWrbtiTe+8844aNmyoJ598UsOHD9eoUaOuejbucDj04IMPKi0t\nTefOndOQIUN+8piePXvqt7/9rTZt2qS6detqwoQJ+vjjj684XnZ2tr755hvFxMRIuvTz+cc//lH7\n9+9XWlqaoqOjfTMucXFxmj59ui2zLbj1EOyoMnfddZfuuusuSZJlWZo5c6aio6N/EuqS9Je//EUF\nBQV69913VVxcrF//+tf64osvfNvz8/O1cuVKLVq0SGFhYUpPT9dTTz2lgQMHasWKFbrrrrv05ptv\nyu12a8qUKTp//vxV14eEhJRad3Z2trZs2eJ7EXDy5En9+7//u9q2bauzZ89q8uTJeuONN3TXXXcp\nOztb/fv3V/PmzXXgwAGtXr1aK1asUJ06dTRz5kwtXbq0RPDOmjVLSUlJioyM1NatW/X555/rF7/4\nhW/70aNH9eKLL2r58uVq0KCBtm/frmeeeUbvvfeeJGnnzp1au3atgoODNXLkSCUnJ2vMmDFXPI7F\nixdr/fr1ys/P1/nz59WmTRstXLhQ9erVq9Bx3H333apdu7ZWrFghSUpISNDf//53TZ069Zo/C6dP\nn9b48eO1ZMkS3Xvvvfrggw+UlJSkadOmlbrfkSNHlJqaquDgYP3jH//QmjVr1LNnTxUXF2v9+vV6\n66239MUXX2jt2rX6+9//rsDAQG3dulWjR4/Wu+++q5YtW1411CVp2bJl6t27t1q2bKnbb79da9as\n0cCBA6/42LNnz2rTpk2Kjo5W7dq1Sx13wIABkqTVq1df9TEnTpxQ3bp1S7zdU69ePZ08eVInTpzw\n/d+RpDvuuEPZ2dlXHQvVC8GOKud2uzVp0iSdPHlSf/3rX6/4mE8//VSTJ0+W0+mU0+nU0qVLJUlr\n1qyRJAUFBWnhwoX6+OOPdezYMR08eFBut1uSFBUVpREjRujEiRPq0KGDxo0bp5CQkKuuv5LnnntO\nAQEB8nq9qlGjhvr166cePXooKytLLpdLkZGRkqT09HTl5ORo1KhRvn0dDocOHTqk7du3q2fPnqpT\np44kafLkyZJK/nKPiYnRs88+q06dOqljx44/OQP77LPP1K5dOzVo0ECS1L59e4WHhysjI0OS9MAD\nD/jOuH/+85/r7NmzV+375bNQt9utsWPHys/Pz/cioiLHIUkNGjTQkiVLlJmZqS+++EL333//VZ//\nx3bv3q2mTZvq3nvvlSR1795d3bt3V1ZWVqn7NW/e3He8jzzyiGbNmqWcnBzt379fjRo10s9+9jOt\nWLFCmZmZio+P9+139uxZnTlzRqGhoVcd+6uvvtKBAwe0aNEiSVKfPn301ltvacCAAXI4HJIuvRBb\nsGCBrH/embtz584aOnToNc/Yy8Lr9V5xvdPp9D3fj92I6z1gBoIdVep//ud/NHLkSEVEROitt95S\nQECAJKl3796+x0yfPl0ul8v3y1S6dDZz+bHSpbPmuLg49e/fX23atFHPnj21ZcsWSdJ9993nu0Dt\ns88+U79+/TR//ny1bt36quv/v6SkpKte1Ofv7y+X69J/peLiYkVERGjlypW+7dnZ2QoPD9dnn31W\n4hjOnTunc+fOlRhr7Nix+tWvfqWtW7dq9erVWrRoUYngv9IvdMuyVFRUJEkleuJwOGRZlm+6+bL/\nfyZZq1YtzZo1S7169dLf/vY3/eY3v6nQcaSlpWnFihUaNGiQHnvsMYWGhl4zmC9zOp0lxrQsS4cO\nHVJISEiJYy4sLPxJ7T/+e48ePbRx40bt2bNH/fr1k3QpIHv37q3x48f7lk+dOuV7YXI1b7/9tlwu\nl2JjYyVJRUVFOnXqlNLS0tSpUydJ0oQJE3xT/z92rTP2srjzzjt1+vRpWZbl6012drbuuOMO/cu/\n/ItOnTrle+zl9YDEx91Qhc6cOaPBgwere/fumjNnTolQWrdune9Py5Yt1b59e61Zs0Zer1cej0dj\nxozRjh07fI/PyMhQeHi4nnnmGUVFRflCvbi4WElJSXrttdfUtWtXTZkyRU2aNNGxY8euuv56REZG\nKjMz01fbgQMH1KNHD506dUodOnTQhx9+qLy8PEnS3LlztXjxYt++RUVFio6Oltvt1oABA/Tiiy/q\n6NGjvtCWpHbt2mnbtm367rvvJEnbt2/XiRMn1KpVq6vWdHm6+fKfK6lTp44mTpyo+fPnKzs7u0LH\nsXXrVvXt21f9+vXT3XffrdTUVBUXF5epb61atdLRo0f19ddfS7r0SYHx48erdu3aKiws1JEjRyRJ\nH374Yanj9O/fX6tXr9aePXvUo0cPSVLHjh31zjvv+IJw2bJlGjZsWKnjnDt3Tu+8844WLlyo1NRU\npaamKi0tTY8//niJfzM73XHHHWrYsKHeffddSdInn3wiPz8/NWvWTNHR0UpNTdX3338vy7KUnJys\nrl27VkpduPlxxo4qs2zZMp04cUIffvhhiV/YixcvVlhYWInHPvvss3rllVfUu3dvFRcXq1evXure\nvbtSU1MlXfrlvWrVKvXs2VOBgYG67777FB4erszMTA0bNkyTJk3So48+Kn9/fzVv3lyPPvqozp49\ne8X11yM8PFx/+ctfNGvWLBUUFMiyLM2aNUv169dX/fr1deTIEd/7q02aNNHLL7+sDz74QJLkcrn0\n/PPP67nnnvPNUMyYMaPENQdNmjTRiy++qGeffVbFxcUKCAjQwoULr3ldQFk8/vjjWrlypRITEzVn\nzpxyH8fBgweVkJCg1atXy+l06l//9V91+PDhMj33bbfdpqSkJE2cONF3UeCcOXMUEhKi8ePH6+mn\nn1Z4ePgVz45/rEWLFnK5XOrRo4dq1qwp6dJbMU8//bR+/etfy+FwKDg4WPPmzZPD4bjqxXNr1qxR\nRESE2rVrV2L9b3/7W8XExJT5uCqid+/emj59ulq2bKk//elPmjp1qhYsWCB/f3/9x3/8h/z8/HTP\nPfdo1KhRGjZsmAoLC9WqVSsunIOPw7rS3B4AALglMRUPAIBBCHYAAAxCsAMAYBCCHQAAgxDsAAAY\n5Jb5uFtRUbFyc91VXYbRwsJq0eNKQJ/tR4/tR48rx+23l/+jrLfMGbvL5azqEoxHjysHfbYfPbYf\nPb553TLBDgAAro1gBwDAIAQ7AAAGIdgBADAIwQ4AgEEIdgAADEKwAwBgEIIdAACDEOwAABjE1mDf\nu3evhgwZ8pP1qampio2NVVxcnFasWGFnCQAAVCu23Sv+P//zP7V+/XoFBgaWWF9YWKiZM2dq1apV\nCgwM1IABAxQdHa3bbrvNrlIAAKg2bAv2hg0bau7cuZowYUKJ9UePHlXDhg1Vp04dSVKbNm20Y8cO\nPfLII6WOt3OnlJvLOwd2Cgujx5WBPtuPHtvvRvf49tst1a9v3bDxqjPbgr1Hjx7Kysr6yfq8vDyF\nhPzft9UEBQUpLy/vmuN9+aXkcgXd0BpR0smTkkSP7Uaf7UeP7Xcje3zx4qUXCrfffkOGq/Yq/Wtb\ng4ODlZ+f71vOz88vEfRX07y55HbnX/NxqLiwsCDl5tJju9Fn+9Fj+93IHh8+7FBurqWcHO8NGc8k\nFfna1koP9oiICGVmZurMmTOqVauWdu7cqeHDh1d2GQCAauT4cYdychxVXYaPnW89VFqwb9iwQW63\nW3FxcZo0aZKGDx8uy7IUGxurevXqVVYZAICb0IULUnq6fddFuN3SkSN+Cgys+vfxL150qGVLr23B\n7rAsq+qPsgw+/ZSpeLsxfVk56LP96LH9bmSPT51y6MyZGzJUqUJDpbp1qz7yDh92qGlTS5GR137r\n4ZaYigcA4Mfq1rVUt25VV1G5rjRDcaOm5wl2AAAqUWiolJVV8v3+Gzk9T7ADAFCJrjRDcfjwjRuf\nOzgAAGAQgh0AAIMQ7AAAGIRgBwDAIAQ7AAAGIdgBADAIwQ4AgEEIdgAADEKwAwBgEIIdAACDEOwA\nABiEYAcAwCAEOwAABiHYAQAwCMEOAIBBCHYAAAxCsAMAYBCCHQAAgxDsAAAYhGAHAMAgBDsAAAYh\n2AEAMIirqgsAAADShQtSenrJ8+2uXSWHo3zjEOwAAFSx0FApK6tkghcWSlFRUkBA+cYi2AEAqGJ1\n61qqW7fkOo+nYmPxHjsAAAYh2AEAMAjBDgCAQQh2AAAMQrADAGAQgh0AAIMQ7AAAGIRgBwDAIAQ7\nAAAGIdgBADAIwQ4AgEEIdgAADEKwAwBgEIIdAACDEOwAABiEYAcAwCAEOwAABiHYAQAwCMEOAIBB\nCHYAAAxCsAMAYBDbgt3r9SohIUFxcXEaMmSIMjMzS2xfv369+vbtq9jYWL399tt2lQEAQLXismvg\nlJQUeTweJScnKz09XYmJiVqwYIFv+6xZs7Rx40bVqlVLMTExiomJUZ06dewqBwCAasG2YN+1a5ei\noqIkSZGRkcrIyCixvXnz5jp//rxcLpcsy5LD4bCrFAAAqg3bgj0vL0/BwcG+ZafTqaKiIrlcl56y\nadOmio2NVWBgoLp166batWtfc8ywsCC7ysU/0ePKQZ/tR4/tR4/tVVBQsf1sC/bg4GDl5+f7lr1e\nry/UDx48qI8++kibN29WrVq1NH78eG3atEmPPPJIqWPm5uaXuh3XJywsiB5XAvpsP3psP3psP49H\nksr/4sm2i+dat26ttLQ0SVJ6erqaNWvm2xYSEqKAgADVrFlTTqdT4eHhOnfunF2lAABQbdh2xt6t\nWzdt27ZN8fHxsixLM2bM0IYNG+R2uxUXF6e4uDgNHDhQNWrUUMOGDdW3b1+7SgEAoNpwWJZlVXUR\nZfHpp5LbzbSPnZhaqxz02X702H702H4ejxQdHaSAgPLtxw1qAAAwCMEOAIBBCHYAAAxCsAMAYBCC\nHQAAgxDsAAAYhGAHAMAgBDsAAAYh2AEAMAjBDgCAQQh2AAAMQrADAGAQgh0AAIMQ7AAAGIRgBwDA\nIAQ7AAAGIdgBADAIwQ4AgEEIdgAADEKwAwBgEIIdAACDEOwAABiEYAcAwCAEOwAABiHYAQAwCMEO\nAIBBCHYAAAxCsAMAYBCCHQAAgxDsAAAYhGAHAMAgBDsAAAYh2AEAMAjBDgCAQQh2AAAMQrADAGAQ\ngh0AAIMQ7AAAGIRgBwDAIAQ7AAAGIdgBADAIwQ4AgEEIdgAADEKwAwBgEIIdAACDEOwAABiEYAcA\nwCAEOwAABiHYAQAwiMuugb1er6ZNm6ZDhw7J399f06dPV6NGjXzb9+3bp8TERFmWpdtvv12zZ89W\nzZo17SoHAIBqwbYz9pSUFHk8HiUnJ2vcuHFKTEz0bbMsS1OnTtXMmTO1bNkyRUVF6fjx43aVAgBA\ntWHbGfuuXbsUFRUlSYqMjFRGRoZv27fffqvQ0FAtXrxYX3/9tTp16qTGjRvbVQoAANWGbcGel5en\n4OBg37LT6VRRUZFcLpdyc3O1Z88eJSQkqGHDhho5cqRatGih9u3blzpmWFiQXeXin+hx5aDP9qPH\n9qPH9iooqNh+ZQr248ePa+nSpTp79qwsy/Ktnzlz5lX3CQ4OVn5+vm/Z6/XK5br0dKGhoWrUqJEi\nIiIkSVFRUcrIyLhmsOfm5pe6HdcnLCyIHlcC+mw/emw/emw/j0eSyv/iqUzvsf/+97+XJLVt21YP\nPPCA709pWrdurbS0NElSenq6mjVr5tvWoEED5efnKzMzU5K0c+dONW3atNzFAwCAksp0xl5UVKSJ\nEyeWa+Bu3bpp27Ztio+Pl2VZmjFjhjZs2CC32624uDi98sorGjdunCzL0v3336+HH364IvUDAIAf\ncVg/nlu/iunTp6tDhw765S9/KX9//8qo6yc+/VRyu5n2sRNTa5WDPtuPHtuPHtvP45Gio4MUEFC+\n/cp0xv7ee+9p6dKlJdY5HA4dOHCgfM8GAABsVaZg37p1q911AACAG6BMwX7hwgXNmzdP27dvV3Fx\nsdq1a6ff/e53qlWrlt31AQCAcijTVfEvvfSSLly4oBkzZujVV19VYWGhXnzxRbtrAwAA5VSmM/av\nvvpK69ev9y0nJCSoV69ethUFAAAqpkxn7JZl6dy5c77lc+fOyel02lYUAAComDKdsT/55JP61a9+\npejoaFmWpS1btmjEiBF21wYAAMqpTMEeGxurli1baseOHfJ6vZo7d66aN29ud20AAKCcSp2K37Jl\niyRp7dq12r9/v4KCghQSEqIDBw5o7dq1lVIgAAAou1LP2L/88kt17txZn3/++RW39+nTx5aiAABA\nxZTplrI/dv78eZ08ebLSv7SFW8raj1tEVg76bD96bD96bL+K3lK2TFfFr1y5UpMnT9YPP/ygmJgY\njRkzRnPmzKlInQAAwEZlCvZly5Zp4sSJ2rhxo7p06aINGzbok08+sbs2AABQTmUKdkkKDQ3Vxx9/\nrIcfflgul0sFBQV21gUAACqgTMHepEkT/du//ZuysrLUvn17/e53v1OLFi3srg0AAJRTmT7HPmPG\nDO3Zs0dNmzaVv7+/evfurU6dOtldGwAAKKdSgz05OVlxcXFauHChJJX42Nv+/fv17LPP2lsdAAAo\nl1Kn4sv5STgAAFDFSj1jj4+PlySNHDlSH3/8sbp06aIffvhBqampio2NrZQCAQBA2ZXp4rmpU6fq\ngw8+8C1//vnnfB87AAA3oTJdPJeRkaENGzZIksLDwzV79mw99thjthYGAADKr0xn7F6vV6dOnfIt\nf//99/LzK/NH4AEAQCUp0xn7yJEj1bdvX7Vp00aWZWnfvn2aMmWK3bUBAIByKlOwP/bYY3rggQeU\nnp4ul8ulqVOnqm7dunbXBgAAyqlM8+kej0dr1qzR5s2b9cADD2jFihXyeDx21wYAAMqpTMH+0ksv\nye12a//+/XK5XPrv//5vpuIBALgJlSnYv/rqK/3hD3+Qy+VSYGCgXn31VR04cMDu2gAAQDmVKdgd\nDoc8Ho8cDockKTc31/d3AABw8yjTxXNDhw7VU089pZycHL3yyitKSUnRqFGj7K4NAACUU5mC/aGH\nHlKLFi30+eefq7i4WAsWLNA999xjd20AAKCcyhTsgwYN0qZNm9SkSRO76wEAANehTMF+zz33aO3a\ntbrvvvsUEBDgW3/nnXfaVhgAACi/MgX73r17tW/fvhJf4+pwOLR582bbCgMAAOVXarBnZ2fr5Zdf\nVq1atdS6dWs999xzql27dmXVBgAAyqnUj7s9//zzaty4sSZMmKDCwkLNnDmzsuoCAAAVcM0z9jfe\neEOS1L59e/Xp06dSigIAABVT6hl7jRo1Svz9x8sAAODmU64vVeducwAA3NxKnYr/+uuv1aVLF99y\ndna2unTpIsuyuCoeAICbUKnB/v7771dWHQAA4AYoNdjr169fWXUAAIAboFzvsQMAgJsbwQ4AgEEI\ndgAADEKwAwBgEIIdAACDEOwAABiEYAcAwCAEOwAABrEt2L1erxISEhQXF6chQ4YoMzPzio+bOnWq\nkpKS7CoDAIBqxbZgT0lJkcfjUXJyssaNG6fExMSfPGb58uU6fPiwXSUAAFDt2Bbsu3btUlRUlCQp\nMjJSGRkZJbbv3r1be/fuVVxcnF0lAABQ7ZR6r/jrkZeXp+DgYN+y0+lUUVGRXC6XTp06pfnz52ve\nvHnatGlTmccMCwuyo1T8CD2uHPTZfvTYfvTYXgUFFdvPtmAPDg5Wfn6+b9nr9crluvR07733nnJz\nczVixAjl5OTo4sWLaty4sZ544olSx8zNzS91O65PWFgQPa4E9Nl+9Nh+9Nh+Ho8klf/Fk23B3rp1\na23ZskW9evVSenq6mjVr5ts2dOhQDR06VJK0evVqffPNN9cMdQAAcG22BXu3bt20bds2xcfHy7Is\nzZgxQxs2bJDb7eZ9dQAAbOKwLMuq6iLK4tNPJbebaR87MbVWOeiz/eix/eix/TweKTo6SAEB5duP\nG9QAAGAQgh0AAIMQ7AAAGIRgBwDAIAQ7AAAGIdgBADAIwQ4AgEEIdgAADEKwAwBgEIIdAACDEOwA\nABiEYAcAwCAEOwAABiHYAQAwCMEOAIBBCHYAAAxCsAMAYBCCHQAAgxDsAAAYhGAHAMAgBDsAAAYh\n2AEAMAjBDgCAQQh2AAAMQrADAGAQgh0AAIMQ7AAAGIRgBwDAIAQ7AAAGIdgBADAIwQ4AgEEIdgAA\nDEKwAwBgEIIdAACDEOwAABiEYAcAwCAEOwAABiHYAQAwCMEOAIBBCHYAAAxCsAMAYBCCHQAAgxDs\nAAAYhGAHAMAgBDsAAAYh2AEAMAjBDgCAQQh2AAAMQrADAGAQl10De71eTZs2TYcOHZK/v7+mT5+u\nRo0a+bZv3LhR//Vf/yWn06lmzZpp2rRp8vPjdQYAANfDtiRNSUmRx+NRcnKyxo0bp8TERN+2ixcv\n6s9//rPeeustLV++XHl5edqyZYtdpQAAUG3YFuy7du1SVFSUJCkyMlIZGRm+bf7+/lq+fLkCAwMl\nSUVFRapZs6ZdpQAAUG3YNhWfl5en4OBg37LT6VRRUZFcLpf8/Px02223SZKWLFkit9utjh07XnPM\nsLAgu8rFP9HjykGf7UeP7UeP7VVQULH9bAv24OBg5efn+5a9Xq9cLleJ5dmzZ+vbb7/V3Llz5XA4\nrjlmbm7+NR+DigsLC6LHlYA+248e248e28/jkaTyv3iybSq+devWSktLkySlp6erWbNmJbYnJCSo\noKBAr732mm9KHgAAXB/bzti7deumbdu2KT4+XpZlacaMGdqwYYPcbrdatGihVatWqW3btho2bJgk\naejQoerWrZtd5QAAUC3YFux+fn566aWXSqyLiIjw/f3gwYN2PTUAANUWHxwHAMAgBDsAAAYh2AEA\nMAjBDgCAQQh2AAAMQrADAGAQgh0AAIMQ7AAAGIRgBwDAIAQ7AAAGIdgBADAIwQ4AgEEIdgAADEKw\nAwBgEIIdAACDEOwAABiEYAcAwCAEOwAABiHYAQAwCMEOAIBBCHYAAAxCsAMAYBCCHQAAgxDsAAAY\nhGAHAMAgBDsAAAYh2AEAMAjBDgCAQQh2AAAMQrADAGAQgh0AAIMQ7AAAGIRgBwDAIAQ7AAAGIdgB\nADAIwQ4AgEEIdgAADEKwAwBgEIIdAACDEOwAABiEYAcAwCAEOwAABiHYAQAwCMEOAIBBCHYAAAxC\nsAMAYBCCHQAAgxDsAAAYhGAHAMAgtgW71+tVQkKC4uLiNGTIEGVmZpbYnpqaqtjYWMXFxWnFihV2\nlQEAQLViW7CnpKTI4/EoOTlZ48aNU2Jiom9bYWGhZs6cqTfffFNLlixRcnKyTp8+bVcpAABUGy67\nBt61a5eioqIkSZGRkcrIyPBtO3r0qBo2bKg6depIktq0aaMdO3bokUceuep4hYWSx2NXtZCkggJ6\nXBnos/3osf3osf0KCyu2n23BnpeXp+DgYN+y0+lUUVGRXC6X8vLyFBIS4tsWFBSkvLy8Usd78EFJ\nCrKpWvwfelw56LP96LH96LHdatYs/z62BXtwcLDy8/N9y16vVy6X64rb8vPzSwT9lQQE2FMnAAAm\nse099tatWystLU2SlJ6ermbNmvm2RUREKDMzU2fOnJHH49HOnTt1//3321UKAADVhsOyLMuOgb1e\nr6ZNm6aXhC1wAAAGSklEQVTDhw/LsizNmDFD+/fvl9vtVlxcnFJTUzV//nxZlqXY2FgNGjTIjjIA\nAKhWbAt2AABQ+bhBDQAABiHYAQAwyE0X7Nyxzn7X6vHGjRvVr18/xcfHKyEhQV6vt4oqvXVdq8eX\nTZ06VUlJSZVcnRmu1eN9+/Zp4MCBGjBggMaMGaOCgoIqqvTWdq0+r1+/Xn379lVsbKzefvvtKqrS\nDHv37tWQIUN+sr7cuWfdZN5//31r4sSJlmVZ1p49e6yRI0f6tnk8Hqtr167WmTNnrIKCAuuJJ56w\ncnJyqqrUW1ZpPb5w4YLVpUsXy+12W5ZlWWPHjrVSUlKqpM5bWWk9vmzZsmVW//79rdmzZ1d2eUYo\nrcder9d6/PHHrWPHjlmWZVkrVqywjh49WiV13uqu9bPcsWNHKzc31yooKPD9fkb5LVq0yHr00Uet\nfv36lVhfkdy76c7Yy3rHOn9/f98d61A+pfXY399fy5cvV2BgoCSpqKhINStyh4RqrrQeS9Lu3bu1\nd+9excXFVUV5Riitx99++61CQ0O1ePFiDR48WGfOnFHjxo2rqtRb2rV+lps3b67z58/L4/HIsiw5\nHI6qKPOW17BhQ82dO/cn6yuSezddsF/tjnWXt5X3jnX4qdJ67Ofnp9tuu02StGTJErndbnXs2LFK\n6ryVldbjU6dOaf78+UpISKiq8oxQWo9zc3O1Z88eDR48WH/729/02Wefafv27VVV6i2ttD5LUtOm\nTRUbG6uYmBg9/PDDql27dlWUecvr0aOH7yZuP1aR3Lvpgv1G37EOP1Vajy8vv/rqq9q2bZvmzp3L\nK/AKKK3H7733nnJzczVixAgtWrRIGzdu1OrVq6uq1FtWaT0ODQ1Vo0aNFBERoRo1aigqKuonZ5oo\nm9L6fPDgQX300UfavHmzUlNT9cMPP2jTpk1VVaqRKpJ7N12wc8c6+5XWY0lKSEhQQUGBXnvtNd+U\nPMqntB4PHTpUq1ev1pIlSzRixAg9+uijeuKJJ6qq1FtWaT1u0KCB8vPzfRd67dy5U02bNq2SOm91\npfU5JCREAQEBqlmzppxOp8LDw3Xu3LmqKtVIFck92+4VX1HdunXTtm3bFB8f77tj3YYNG3x3rJs0\naZKGDx/uu2NdvXr1qrrkW05pPW7RooVWrVqltm3batiwYZIuBVG3bt2quOpby7V+jnH9rtXjV155\nRePGjZNlWbr//vv18MMPV3XJt6Rr9TkuLk4DBw5UjRo11LBhQ/Xt27eqSzbC9eQed54DAMAgN91U\nPAAAqDiCHQAAgxDsAAAYhGAHAMAgBDsAAAa56T7uBsAeWVlZ6tmzpyIiIiRdutFIfn6++vTpozFj\nxtyQ57h8S8zRo0erefPmOnTo0A0ZF0DZEexANVK3bl2tW7fOt5ydna0ePXooJibGF/gAbm0EO1CN\n5eTkyLIsBQUFadGiRdq0aZOKi4v1y1/+UuPHj5fD4dDixYu1bNkyOZ1Ode7cWePHj9fhw4f18ssv\ny+1264cfftBTTz2loUOHVvXhABDBDlQrp06dUu/evVVQUKDc3Fy1bNlS8+bN0+HDh5WRkaFVq1bJ\n4XBo/PjxWr9+ve6++269/fbb+sc//qHAwED95je/UUZGhtatW6dnnnlG7du313fffafHH3+cYAdu\nEgQ7UI1cnor3er1KTEzUoUOH1K5dO/3pT3/Svn37fPesv3jxou68806dPn1anTt39n3pxOLFiyVJ\n9957rz755BO9/vrrOnTokNxud1UdEoD/h2AHqiE/Pz9NmDBBffr00Ztvvqni4mINGzZMTz31lCTp\n3LlzcjqdWrVqVYn9srOzFRgYqClTpqh27drq3LmzevXqpXfeeacqDgPAFfBxN6CacrlcmjBhghYu\nXKif//znWrdunfLz81VUVKRRo0bp/fffV9u2bZWWluZbP27cOGVkZGjbtm0aM2aMunbtqh07dkiS\niouLq/iIAEicsQPV2kMPPaTIyEjt2LFD3bt3V//+/VVcXKyoqCj17dtXDodDgwcPVnx8vLxer7p1\n66YOHTpo9OjRGjhwoGrXrq27775b9evXV1ZWVlUfDgDx7W4AABiFqXgAAAxCsAMAYBCCHQAAgxDs\nAAAYhGAHAMAgBDsAAAYh2AEAMAjBDgCAQf4XFBIC/1mHbIgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x25a9ff2e6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "precision, recall, thresholds = precision_recall_curve(y_test, y_test_prob)\n",
    "\n",
    "plt.step(recall, precision, color='b', alpha=0.2, where='post')\n",
    "plt.fill_between(recall, precision, step='post', alpha=0.2, color='b')\n",
    "\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.title('2-class Precision-Recall curve: AP={0:0.2f}'.format(average_precision))\n",
    "print('precision', precision[:5])\n",
    "print('recall', recall[:5])\n",
    "print('thresholds', thresholds[:5])"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}