CRISP-DM Python Template (3).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/ASR_LOGO.png\"=\"130\" align = \"center\">\n",
    "<img src=\"img/revhub_logo.png\" width=\"110\" align = \"center\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CRISP-DM Python Template\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<img src=\"img/crisp-dm-4.png\" width = \"350\" align=\"right\"/> ![](desktop/crisp-dm-4.png)\n",
    "\n",
    "***\n",
    "CRISP-DM stands for cross-industry process for data mining. It provides a structured <br> approach to planning a data mining project. It is a robust and well-proven methodology. \n",
    "\n",
    "The CRISP-DM model consists of six steps:\n",
    "\n",
    "1. [Business Understanding](#Businessunderstanding)\n",
    "2. [Data Understanding](#Dataunderstanding)\n",
    "3. [Data Preparation](#Datapreparation)\n",
    "4. [Modelling](#Modelling)\n",
    "5. [Evaluation](#Evaluation)\n",
    "6. [Deploymnet](#Deployment)\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 1. Business Understanding <a class=\"anchor\" id=\"Businessunderstanding\"></a>\n",
    "<img src=\"img/1..png\" width=\"35\" height=\"30\" align=\"left\"/>\n",
    "\n",
    "Determine business objectives and assess the situation. Include requirements, data security, and constraints. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tax Type (Sales Tax, Personal Income Tax, etc.)\n",
    "- <font color=red>User input required</font>\n",
    "\n",
    "#### Model Type (Schedule Analaysis, Preparer Analysis, Restaurant Sales Tax)\n",
    "- <font color=red>User input required</font>\n",
    "\n",
    "#### Requirements\n",
    "- <font color=red>User input required</font>\n",
    "\n",
    "#### Security (FTI)\n",
    "- <font color=red>User input required</font>\n",
    "\n",
    "#### Constraints\n",
    "- <font color=red>User input required</font>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 2. Data Understanding <a class = \"ancor\" id = \"Dataunderstanding\"></a>  \n",
    "<img src=\"img/2..png\" width=\"30\" height=\"30\" align=\"left\"/>\n",
    "\n",
    "The initial collection of data required in the project resources includes data loading and data integration if data from multiple sources. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Required Libraries\n",
    "\n",
    ">```python\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Initial Data\n",
    "\n",
    ">```python\n",
    ">#Importing data from .csv\n",
    ">example = pd.read_csv(\"example.csv\")\n",
    ">```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initial Data Analysis\n",
    "\n",
    ">```python\n",
    "example.head()\n",
    "example.columns()\n",
    "example.info()\n",
    "example.describe()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 3. Data Preparation <a class = \"anchor\" id = \"Datapreparation\"></a>\n",
    "<img src=\"img/3..png\" width=\"30\" height=\"30\" align=\"left\"/>\n",
    "\n",
    "Data preparation includes additional data acquisition, data cleaning, data integration, and data transformation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Acquisition \n",
    "From database, files, etc.\n",
    ">```python\n",
    ">#Inputting data from .csv\n",
    "example2 = pd.read_csv(\"example.csv\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Cleaning \n",
    "Identify and fix errors in data, deal with missing data, etc.\n",
    "\n",
    ">```python\n",
    ">#Replacing 0s\n",
    "cols = ['col1', 'col2']\n",
    "df[cols] = df[cols].replace(0, np.nan)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Integration\n",
    "Merge data from different sources\n",
    ">```python\n",
    ">#Combining two data\n",
    "df = pd.merge(example, example2)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data Transformation and Enrichment\n",
    "Create new features from existing features\n",
    ">```python\n",
    ">#Ratio Variable\n",
    "Deduction_Ratio = taxdata.AZDeductions/taxdata.AZAGI\n",
    "taxdata[\"Deduction_Ratio\"] = Deduction_Ratio\n",
    ">#Indicator Variable\n",
    "taxdata[\"Sus_PTIN\"] = taxdata.PTIN\n",
    "taxdata[\"Sus_PTIN\"] = np.where(taxdata[\"Sus_PTIN\"] == 'P00749774', '1', taxdata[\"Sus_PTIN\"])\n",
    "taxdata[\"Sus_PTIN\"] = np.where(taxdata[\"Sus_PTIN\"] != '1', '0', taxdata[\"Sus_PTIN\"])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 4. Modelling <a class = \"anchor\" id = \"Modelling\"></a>\n",
    "<img src=\"img/4..png\" width=\"30\" height=\"30\" align=\"left\"/>\n",
    "\n",
    "Creation of Model and Model Criteria:\n",
    "- Parameter Settings: <font color=red>User input required</font>\n",
    "- Model: <font color=red>User input required</font>\n",
    "- Model Descrition:<font color=red>User input required</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating Training and Testing Sets\n",
    ">```python\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1) \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Creating the Model\n",
    ">```python\n",
    "># Create Decision Tree classifer object \n",
    "clf = DecisionTreeClassifier() \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training the Model\n",
    ">```python\n",
    "clf = clf.fit(X_train,y_train) \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing the Model\n",
    ">```python\n",
    ">#Predict the response for test dataset \n",
    "y_pred = clf.predict(X_test) \n",
    "X_TEST = [[86556]] \n",
    "outcome = clf.predict(X=X_TEST) \n",
    "print (outcome) \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 5. Evaluation <a class = \"anchor\" id = \"Evaluation\"></a>\n",
    "<img src=\"img/5..png\" width=\"30\" height=\"30\" align=\"left\"/>\n",
    "\n",
    "Interpret the models according to your domain knowledge, your data mining success criteria and your desired test design."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Analysis\n",
    ">```python\n",
    ">#Confusion Matrix Analysis\n",
    "from sklearn.metrics import confusion_matrix\n",
    "matrix = confusion_matrix(Y_test, y_pred)\n",
    "print (matrix)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## 6. Deployment <a class = \"anchor\" id = \"Deployment\"></a>\n",
    "<img src=\"img/6..png\" width=\"30\" height=\"30\" align=\"left\"/>\n",
    "\n",
    "Prepare the model for deployment within MarkLogic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Converting to .onnx File\n",
    ">```python\n",
    "from skl2onnx import convert_sklearn \n",
    "from skl2onnx.common.data_types import FloatTensorType \n",
    "initial_type = [('float_input', FloatTensorType([None, 4]))] \n",
    "onx = convert_sklearn(clf, initial_types=initial_type) \n",
    "with open(\"ZIP_PREDICT.onnx\", \"wb\") as f: \n",
    "    f.write(onx.SerializeToString())\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}