ABtesting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b4b01ea1-e51a-4959-9429-3e3b09341f41",
   "metadata": {},
   "source": [
    "# A/B Testing\n",
    "## by Sindi Ingraham"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c087cdba-ebb3-42cb-9ecd-a6b5254d6adf",
   "metadata": {},
   "source": [
    "### Project Background\n",
    "\n",
    "Globox is an online marketplace that specializes in sourcing unique and high-quality products from around the world. Their logo **\" We believe that shopping should be an adventure, and we want to bring the world to your doorstep. From exotic spices and rare teas to handmade jewelry and textiles, we have a curated selection of products that you won't find anywhere else\"**.\n",
    "\n",
    "GloBox is primarily known amongst its customer base for boutique fashion items and high-end decor products. However, their food and drink offerings have grown tremendously in the last few months, and the company wants to bring awareness to this product category to increase revenue.\n",
    "\n",
    "The Growth team decides to run an A/B test that highlights key products in the food and drink category as a banner at the top of the website. The control group does not see the banner, and the test group sees it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65de0f4e-4167-4ff1-ae2e-076c98dffc62",
   "metadata": {},
   "source": [
    "### The setup of the A/B test is as follows:\n",
    "\n",
    "- The experiment is only being run on the mobile website.\n",
    "- A user visits the GloBox main page and is randomly assigned to either the control or test group. This is the join date for the user.\n",
    "- The page loads the banner if the user is assigned to the test group, and does not load the banner if the user is assigned to the control group.\n",
    "- The user subsequently may or may not purchase products from the website. It could be on the same day they join the experiment, or days later. If they do make one or more purchases, this is considered a “conversion”."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af51aaf4-7b9e-4e04-a1a1-c09a7fadbdcc",
   "metadata": {},
   "source": [
    "### Task\n",
    "\n",
    "Analyze the results of the A/B test and provide a recommendation to the stakeholders about whether Globox should launch the experience to all users."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b435d766-9efb-461b-912a-58607d67d495",
   "metadata": {},
   "source": [
    "### Stakeholders\n",
    "\n",
    "- Growth Product & Engineering Team: This is the team that you work with at GloBox. The team is made up of a product manager, a user experience designer, an engineering manager and several software engineers, and you, the data analyst. The team develops features for the GloBox website that drive growth in users and revenue.\n",
    "- Leila Al-Farsi, Product Manager, Growth: Leila is the product manager for the Growth product and engineering team. Alongside Alejandro, she leads the Growth team by deciding their goals and projects, measuring their success against defined KPIs, and communicating results to other company leaders like Mei.\n",
    "- Alejandro Gonzalez, User Experience Designer, Growth: Alejandro is the designer for the Growth product and engineering team. He conducts user research and designed the experience that the A/B test is evaluating.\n",
    "- Mei Kim, Head of Marketing: Mei oversees the Marketing team, which works on targeting audiences with effective marketing campaigns to drive customers to the GloBox website. She collaborates frequently with Leila and Alejandro to design website experiences that will align well with the current marketing efforts.\n",
    "\n",
    "Together, Leila, Alejandro, and Mei will decide whether or not to launch the experiment based on the results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e23c516-746b-4be4-b03c-51560c1588d2",
   "metadata": {},
   "source": [
    "### The Dataset\n",
    "\n",
    "GloBox stores its data in a relational database, which I extracted using SQL queries. \n",
    "\n",
    "To consider:\n",
    "\n",
    "- All users should be assigned to one A/B test group\n",
    "- Not all users make a purchase.\n",
    "\n",
    "![data description](data_image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c1c1545-4ea6-43af-940d-b478f70a5f3f",
   "metadata": {},
   "source": [
    "# Metric\n",
    "\n",
    "- Revenue\n",
    "- Conversion Rate = #converted / #converted + # notconverted\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c32897dd-a16b-4595-b3d7-08d82967700b",
   "metadata": {},
   "source": [
    "## Statistical Hypothesis\n",
    "\n",
    "- Ho : CRcon = CRexp\n",
    "- Ha : CRcon &ne; CRexp\n",
    "\n",
    "- Ho : &mu;con = &mu;exp\n",
    "- Ha: &mu;con &ne; &mu;exp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7119fb5b-a544-4674-a623-40e5a0c2f8b7",
   "metadata": {},
   "source": [
    "## Power Analysis\n",
    "\n",
    "In this instance, power analysis was already conducted and the test data was readily available and extracted from the database. Nonetheless, I have established the confidence level and the significance level for the study.\n",
    "\n",
    "- **Confidence Level**: 95%\n",
    "- **Significance Level**: 5% or 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ef38e8c-53b5-400e-8fdd-028d8402feaa",
   "metadata": {},
   "source": [
    "## Minimum Detectable Effect\n",
    "\n",
    "The targeted objective is to discern the magnitude of change between the new and old versions, as a basis for deciding whether to launch the new feature. Although the stakeholders did not provide a specific percentage, we have set a benchmark of a 2% change in conversion rate and revenue for this analysis, which will be assessed based on the average amount spent per user."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5221fc5-6bab-4d3e-a58f-dfa947e0b613",
   "metadata": {},
   "source": [
    "# Analyzing A/B test results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "d8a48687-5fd0-4b18-b82b-577ef6f83adb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Importing the necessary packages\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sb\n",
    "from scipy.stats import ttest_ind\n",
    "import scipy.stats as stats\n",
    "from math import ceil\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "eef27268-0ca9-4f2d-943d-c6905e6041e1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>uid</th>\n",
       "      <th>group</th>\n",
       "      <th>join_date</th>\n",
       "      <th>date_of_purchase</th>\n",
       "      <th>amount_spent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1000000</td>\n",
       "      <td>B</td>\n",
       "      <td>2023-01-28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1000001</td>\n",
       "      <td>A</td>\n",
       "      <td>2023-01-27</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1000002</td>\n",
       "      <td>A</td>\n",
       "      <td>2023-02-01</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1000003</td>\n",
       "      <td>B</td>\n",
       "      <td>2023-01-25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1000004</td>\n",
       "      <td>A</td>\n",
       "      <td>2023-02-04</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       uid group   join_date date_of_purchase  amount_spent\n",
       "0  1000000     B  2023-01-28              NaN           0.0\n",
       "1  1000001     A  2023-01-27              NaN           0.0\n",
       "2  1000002     A  2023-02-01              NaN           0.0\n",
       "3  1000003     B  2023-01-25              NaN           0.0\n",
       "4  1000004     A  2023-02-04              NaN           0.0"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reading the data extrated from the database into pandas dataframe\n",
    "ab = pd.read_csv('testingdata.csv')\n",
    "ab.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "9c1f7b9d-5bb5-4bc6-aa24-f20f663ed105",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49082 entries, 0 to 49081\n",
      "Data columns (total 5 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   uid               49082 non-null  int64  \n",
      " 1   group             49082 non-null  object \n",
      " 2   join_date         49082 non-null  object \n",
      " 3   date_of_purchase  2233 non-null   object \n",
      " 4   amount_spent      49082 non-null  float64\n",
      "dtypes: float64(1), int64(1), object(3)\n",
      "memory usage: 1.9+ MB\n"
     ]
    }
   ],
   "source": [
    "#Cheking the dataset\n",
    "ab.info()   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "f8ef063a-cad5-4b90-80bb-1e2b6d77273c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 139 users that appear multiple times in the dataset\n"
     ]
    }
   ],
   "source": [
    "# Searching for duplicates\n",
    "\n",
    "clean_dat = ab.copy()\n",
    "data_count = clean_dat['uid'].value_counts(ascending = False)\n",
    "duplicate = data_count[data_count > 1].count()\n",
    "print(f'There are {duplicate} users that appear multiple times in the dataset')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3a68758-04ae-4422-a9c8-58342594e5a1",
   "metadata": {},
   "source": [
    "After identifying 139 duplicates in the dataset, it was determined that the impact of their removal would not significantly affect the results. Therefore, I proceed with deleting the duplicates from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "d1f0c130-f397-4d37-967e-3249021d40ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 48943 entries, 0 to 49081\n",
      "Data columns (total 5 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   uid               48943 non-null  int64  \n",
      " 1   group             48943 non-null  object \n",
      " 2   join_date         48943 non-null  object \n",
      " 3   date_of_purchase  2094 non-null   object \n",
      " 4   amount_spent      48943 non-null  float64\n",
      "dtypes: float64(1), int64(1), object(3)\n",
      "memory usage: 2.2+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Deleteing duplicates\n",
    "\n",
    "clean_ab = df = clean_dat.drop(clean_dat[clean_dat.duplicated(subset='uid')].index)\n",
    "print(clean_ab.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2f09ee5-851c-4c19-b79c-edf9cff3ae3e",
   "metadata": {},
   "source": [
    "## Convertion Rate\n",
    "\n",
    "In order to gain insight from the same, I'll run some basic statistics and know what the sample looks like.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "7d8cd4b5-ef4d-47ec-980b-a0836f2fc2e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "group\n",
      "A    0.039231\n",
      "B    0.046301\n",
      "Name: amount_spent, dtype: float64\n",
      "group\n",
      "A    24.758693\n",
      "B    24.222659\n",
      "Name: amount_spent, dtype: float64\n",
      "group\n",
      "A    0.158687\n",
      "B    0.154438\n",
      "Name: amount_spent, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Calculating convertion rate.\n",
    "conversion_rates = clean_ab.groupby('group')['amount_spent'].apply(lambda x: (x > 0).mean())\n",
    "print(conversion_rates)\n",
    "\n",
    "# Calculating standard deviation.\n",
    "std_deviation = clean_ab.groupby('group')['amount_spent'].std()\n",
    "print(std_deviation)\n",
    "\n",
    "# Calculating standard of error.\n",
    "std_error = clean_ab.groupby('group')['amount_spent'].sem()\n",
    "print(std_error)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "05d98c59-440f-44cc-84aa-7cf863958e29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfcAAAGACAYAAACwUiteAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZxklEQVR4nO3de7hddX3n8ffHRG6DgIZ4IVyCgpfodLxEUEqrlacK3qIWFPTxNmimrXS0XsZ4GYpUW3GqWAXHpgMOTRW0KDYFFPUBrTKIBLFKUGyKKIGqISAIiBD4zh97BTe7Jyebc7JyTn68X89znqz1W7+91neHJ3z277d+Z+1UFZIkqR0PmOkCJEnSlmW4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJU1Lklck+dJM1yHpNwx3aZZI8vIkq5LckuTfk3whycEzXdewJAuTVJK5G9uq6pNV9ewervXMJHd3fx+/THJlktfeh9d/NcnrtnRd0rbAcJdmgSRvBj4M/AXwMGBv4GPAkhks616GA30ruq6qdgZ2Af4U+Nskj5mBOqRtiuEuzbAkuwLHA2+oqs9V1a1VdWdV/VNVva3rs32SDye5rvv5cJLtu2PPTLI2yVuS/Lwb9b+2O3Zgkp8mmTN0vRcn+W63/YAky5L8W5L1ST6T5CHdsY2j9KOT/AQ4H/jn7jS/6EbUT0/ymiTfGDr/QUkuSXJT9+dBQ8e+muTPk1zYjca/lGT3zf0d1cC5wA3Ab3XnenCSs5OsS3Jjt71nd+x9wO8AJ3V1ntS1PzbJl5Pc0M0EvHRK/9GkWc5wl2be04EdgLMm6fMu4GnAE4H/AhwAvHvo+MOBXYEFwNHAyUkeXFUXA7cCzxrq+3LgU932nwAvAp4B7AHcCJw8cu1nAI8DngP8bte2W1XtXFUXDXfsPhicA3wEmAd8CDgnybyR678WeCiwHfDWSd73xvM+IMkLgd2BNV3zA4BPAPswmOn4FXASQFW9C/g6cExX5zFJ/hPw5e69PxQ4EvhYkkWbu760rTHcpZk3D7i+qjZM0ucVwPFV9fOqWge8B3jl0PE7u+N3diPcW4CN09enA0cBJHkQ8NyuDeAPgXdV1dqq+jVwHHD4yBT8cd1swq/GeC/PA/61qlZU1YaqOh34AfCCoT6fqKofduf7DIMPLJuyR5JfMAjus4A3V9VlAFW1vqo+W1W3VdUvgfcx+CCyKc8Hrq6qT3S1XQZ8FjhijPclbVMMd2nmrQd238w97T2AHw/t/7hru+ccIx8ObgN27rY/Bbykm8Z/CfDtqtp4rn2As5L8ogvR7wN3Mbjvv9E19+G9jNa5sdYFQ/s/3USdE7muqnZjcM/9IwzNQCTZKcnfJPlxkpsZ3DLYbfgWxIh9gAM3vtfu/b6CwayH1BTDXZp5FwG/ZjA9vinXMQinjfbu2jarqq5gELCHce8peRgE92FVtdvQzw5Vde3wKTaxPU6dG2u9doK+Y+tmFd4O/OckL+qa38JgduLAqtqF39wyyCZqvQb42sh73bmq/mg6tUmzkeEuzbCqugk4lsF98hd1I9IHJjksyQe6bqcD704yv1uAdizw9/fhMp8C3sggAP9hqP3jwPuS7APQnX+yFfrrgLuBR27i+LnAo7tf65ub5GXAIuDs+1DrhKrqDuCDDN47wIMYTNf/orvX/2cjL/nZSJ1nd7W9svv7fWCSpyZ53HRrk2Ybw12aBarqg8CbGSySW8dglHkM8Pmuy3uBVcB3ge8B3+7axnU6g/vR51fV9UPtfw2sBL6U5JfAN4EDJ6nzNgb3ti/sprafNnJ8PYN7229hcLvhfwDPH7nmdJwK7J3kBQx+dXBH4Pqu7i+O9P1rBusHbkzyke6+/LMZLKS7jsHtgROA7bdQbdKskarNzbJJkqRtiSN3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMTPxLU+92H333WvhwoUzXYYkSVvNpZdeen1VzR9tbybcFy5cyKpVq2a6DEmStpoko497BpyWlySpOYa7JEmNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhrTzLfCSVIfFi47Z6ZL0DRc/f7nzXQJM8KRuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMYa7JEmNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMb0Gu5JDk1yZZI1SZZNcHz7JJ/ujl+cZOHI8b2T3JLkrX3WKUlSS3oL9yRzgJOBw4BFwFFJFo10Oxq4sar2A04EThg5/iHgC33VKElSi/ocuR8ArKmqq6rqDuAMYMlInyXAad32mcAhSQKQ5EXAj4DVPdYoSVJz+gz3BcA1Q/tru7YJ+1TVBuAmYF6SnYG3A+/psT5Jkpo0WxfUHQecWFW3TNYpydIkq5KsWrdu3dapTJKkWW5uj+e+FthraH/Prm2iPmuTzAV2BdYDBwKHJ/kAsBtwd5Lbq+qk4RdX1XJgOcDixYurjzchSdK2ps9wvwTYP8m+DEL8SODlI31WAq8GLgIOB86vqgJ+Z2OHJMcBt4wGuyRJmlhv4V5VG5IcA5wHzAFOrarVSY4HVlXVSuAUYEWSNcANDD4ASJKkaehz5E5VnQucO9J27ND27cARmznHcb0UJ0lSo3oNd205C5edM9MlaBqufv/zZroESfcjs3W1vCRJmiLDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMYa7JEmNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMYa7JEmNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktSYXsM9yaFJrkyyJsmyCY5vn+TT3fGLkyzs2g9I8p3u51+SvLjPOiVJaklv4Z5kDnAycBiwCDgqyaKRbkcDN1bVfsCJwAld++XA4qp6InAo8DdJ5vZVqyRJLelz5H4AsKaqrqqqO4AzgCUjfZYAp3XbZwKHJElV3VZVG7r2HYDqsU5Jkpqy2XBP8rAkpyT5Qre/KMnRY5x7AXDN0P7arm3CPl2Y3wTM665zYJLVwPeAPxwKe0mSNIlxRu7/FzgP2KPb/yHwpp7quUdVXVxVjweeCrwjyQ6jfZIsTbIqyap169b1XZIkSduEccJ996r6DHA33DPCvmuM110L7DW0v2fXNmGf7p76rsD64Q5V9X3gFuAJoxeoquVVtbiqFs+fP3+MkiRJat844X5rknl0972TPI3B9PnmXALsn2TfJNsBRwIrR/qsBF7dbR8OnF9V1b1mbne9fYDHAlePcU1Jku73xlmB/mYGIfyoJBcC84EjNveiqtqQ5BgGU/pzgFOranWS44FVVbUSOAVYkWQNcAODDwAABwPLktzJYMbgj6vq+vv43iRJul8aJ9xXA88AHgMEuJIxV9lX1bnAuSNtxw5t384EHxSqagWwYpxrSJKkexsnpC+qqg1VtbqqLq+qO4GL+i5MkiRNzSZH7kkezuBX1XZM8iQGo3aAXYCdtkJtkiRpCiabln8O8BoGq9w/NNT+S+CdPdYkSZKmYZPhXlWnAacl+YOq+uxWrEmSJE3DZhfUVdVnkzwPeDyDR8FubD++z8IkSdLUjPP42Y8DLwP+hMF99yOAfXquS5IkTdE4q+UPqqpXMfj2tvcATwce3W9ZkiRpqsYJ9191f96WZA/gTuAR/ZUkSZKmY5yH2JydZDfgfwHfZvAY2v/TZ1GSJGnqxllQ9+fd5meTnM1gUZ1fvypJ0iw16bR8kgVJFndf/AKDb217O/CvvVcmSZKmZJPhnuRNwHeAjwLfTPI64PvAjsBTtkZxkiTpvptsWn4p8JiquiHJ3sAPgd+uqku3TmmSJGkqJpuWv72qbgCoqp8AVxrskiTNfpON3PdM8pGh/UcM71fVf++vLEmSNFWThfvbRvYdtUuStA3Y3BfHSJKkbcw4T6iTJEnbEMNdkqTGGO6SJDVms4+fTTIfeD2wcLh/Vf3X/sqSJElTNc4Xx/wj8HXgK8Bd/ZYjSZKma5xw36mq3t57JZIkaYsY55772Ume23slkiRpixgn3N/IIOBvT/LL7ufmvguTJElTM873uT9oaxQiSZK2jHHuuZPkhcDvdrtfraqz+ytJkiRNx2an5ZO8n8HU/BXdzxuT/GXfhUmSpKkZZ+T+XOCJVXU3QJLTgMuAd/RZmCRJmppxn1C329D2rj3UIUmStpBxRu5/CVyW5AIgDO69L+u1KkmSNGXjrJY/PclXgad2TW+vqp/2WpUkSZqyTU7LJ3ls9+eTgUcAa7ufPbo2SZI0C002cn8zsBT44ATHCnhWLxVJkqRp2WS4V9XSbvOwqrp9+FiSHXqtSpIkTdk4q+X/35htkiRpFtjkyD3Jw4EFwI5JnsRgpTzALsBOW6E2SZI0BZPdc38O8BpgTwb33TeG+83AO/stS5IkTdVk99xPS7ICOKqqPrkVa5IkSdMw6T337pGzf7qVapEkSVvAOAvqvpLkrUn2SvKQjT+9VyZJkqZknMfPvqz78w1DbQU8csuXI0mSpmucx8/uuzUKkSRJW8Y43+e+U5J3J1ne7e+f5Pn9lyZJkqZinHvunwDuAA7q9q8F3ttbRZIkaVrGCfdHVdUHgDsBquo2fvM775IkaZYZJ9zvSLIjg0V0JHkU8Oteq5IkSVM2zmr544AvAnsl+STw2wyeXCdJkmahcVbLfynJpcDTGEzHv7Gqru+9MkmSNCWbDfck/wR8ClhZVbf2X5IkSZqOce65/xXwO8AVSc5Mcrjf5y5J0uw1zrT814CvJZkDPAt4PXAqg69+lSRJs8w4C+roVsu/gMGjaJ8MnNZnUZIkaerGuef+GeAABivmTwK+1n1bnCRJmoXGGbmfwuA73e/quxhJkjR949xzPy/JQUkWDvevqr/rszBJkjQ140zLrwAeBXwH2Dh6L8BwlyRpFhpnWn4xsKiqqu9iJEnS9I3ze+6XAw+fysmTHJrkyiRrkiyb4Pj2ST7dHb+4m/onye8nuTTJ97o/nzWV60uSdH80zsh9dwYPsPkWQ18YU1UvnOxF3e/Fnwz8PrAWuCTJyqq6Yqjb0cCNVbVfkiOBExj8ut31wAuq6rokTwDOAxbch/clSdL91rhfHDMVBwBrquoqgCRnAEuA4XBfMnT+M4GTkqSqLhvqsxrYMcn2VeW30UmStBmbnZbvnlD3A+BB3c/3u7bNWQBcM7S/lv84+r6nT1VtAG4C5o30+QPg2wa7JEnj2Wy4J3kp8C3gCOClwMVJDu+7sO7aj2cwVf/fNnF8aZJVSVatW7dua5QkSdKsN860/LuAp1bVzwGSzAe+wmAafTLXAnsN7e/ZtU3UZ22SucCuwPruOnsCZwGvqqp/m+gCVbUcWA6wePFiV/NLksR4q+UfsDHYO+vHfN0lwP5J9k2yHXAksHKkz0rg1d324cD5VVVJdgPOAZZV1YVjXEuSJHXGGbl/Mcl5wOnd/suAL2zuRVW1IckxDFa6zwFOrarVSY4HVlXVSgaPtl2RZA1wA4MPAADHAPsBxyY5tmt79siHDEmSNIFxHj/7tiQvAQ7umpZX1VnjnLyqzgXOHWk7dmj7dgb38kdf917gveNcQ5Ik3dsmwz3JfsDDqurCqvoc8Lmu/eAkj9rUfXBJkjSzJrt3/mHg5gnab+qOSZKkWWiycH9YVX1vtLFrW9hbRZIkaVomC/fdJjm24xauQ5IkbSGThfuqJK8fbUzyOuDS/kqSJEnTMdlq+TcBZyV5Bb8J88XAdsCLe65LkiRN0SbDvap+BhyU5PeAJ3TN51TV+VulMkmSNCXj/J77BcAFW6EWSZK0BYzzGFlJkrQNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMYa7JEmNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMYa7JEmNMdwlSWpMr+Ge5NAkVyZZk2TZBMe3T/Lp7vjFSRZ27fOSXJDkliQn9VmjJEmt6S3ck8wBTgYOAxYBRyVZNNLtaODGqtoPOBE4oWu/HfifwFv7qk+SpFb1OXI/AFhTVVdV1R3AGcCSkT5LgNO67TOBQ5Kkqm6tqm8wCHlJknQf9BnuC4BrhvbXdm0T9qmqDcBNwLwea5IkqXnb9IK6JEuTrEqyat26dTNdjiRJs0Kf4X4tsNfQ/p5d24R9kswFdgXWj3uBqlpeVYuravH8+fOnWa4kSW3oM9wvAfZPsm+S7YAjgZUjfVYCr+62DwfOr6rqsSZJkpo3t68TV9WGJMcA5wFzgFOranWS44FVVbUSOAVYkWQNcAODDwAAJLka2AXYLsmLgGdX1RV91StJUit6C3eAqjoXOHek7dih7duBIzbx2oV91iZJUqu26QV1kiTpPzLcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMYa7JEmNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJaozhLklSYwx3SZIaY7hLktQYw12SpMYY7pIkNcZwlySpMYa7JEmNMdwlSWqM4S5JUmMMd0mSGmO4S5LUGMNdkqTGGO6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhpjuEuS1BjDXZKkxhjukiQ1xnCXJKkxhrskSY0x3CVJakyv4Z7k0CRXJlmTZNkEx7dP8unu+MVJFg4de0fXfmWS5/RZpyRJLekt3JPMAU4GDgMWAUclWTTS7WjgxqraDzgROKF77SLgSODxwKHAx7rzSZKkzehz5H4AsKaqrqqqO4AzgCUjfZYAp3XbZwKHJEnXfkZV/bqqfgSs6c4nSZI2o89wXwBcM7S/tmubsE9VbQBuAuaN+VpJkjSBuTNdwHQkWQos7XZvSXLlTNajadkduH6mi+hLTpjpCqRN8t/etm2fiRr7DPdrgb2G9vfs2ibqszbJXGBXYP2Yr6WqlgPLt2DNmiFJVlXV4pmuQ7q/8d9em/qclr8E2D/Jvkm2Y7BAbuVIn5XAq7vtw4Hzq6q69iO71fT7AvsD3+qxVkmSmtHbyL2qNiQ5BjgPmAOcWlWrkxwPrKqqlcApwIoka4AbGHwAoOv3GeAKYAPwhqq6q69aJUlqSQYDZWlmJVna3WaRtBX5b69NhrskSY3x8bOSJDXGcJckqTGGu2aFJAcnOXmm65CkFmzTD7HRti3Jk4CXA0cAPwI+N7MVSfc/SXYH1pcLsJriyF1bVZJHJ/mzJD8APgr8hMHCzt+rqo/OcHlS05I8LclXk3wuyZOSXA5cDvwsyaEzXZ+2HFfLa6tKcjfwdeDoqlrTtV1VVY+c2cqk9iVZBbyTwdNAlwOHVdU3kzwWOL2qnjSjBWqLceSure0lwL8DFyT52ySHAJnhmqT7i7lV9aWq+gfgp1X1TYCq+sEM16UtzHDXVlVVn6+qI4HHAhcAbwIemuR/J3n2jBYnte/uoe1fjRxzGrchTstrxiV5MINFdS+rqkNmuh6pVUnuAm5lMFu2I3DbxkPADlX1wJmqTVuW4S5JUmOclpckqTGGuyRJjTHcJd0jycOSfCrJVUkuTXJRkhfPdF2S7hvDXRIASQJ8HvjnqnpkVT0FOBLYc6SfT7aUZjkX1EkCoHvmwLFV9YwJjr2GwTMKdgbmAC8GTgUeyWDF9dKq+m6S44BbquqvutddDjy/O80XgUuBJwOrgVdV1W1I2uIcuUva6PHAtyc5/mTg8C783wNcVlW/xeCJZ383xvkfA3ysqh4H3Az88TTrlbQJhrukCSU5Ocm/JLmka/pyVd3QbR8MrACoqvOBeUl22cwpr6mqC7vtv+/OIakHhrukjVYzGJ0DUFVvAA4B5ndNt45xjg3c+/8rOwxtj94D9J6g1BPDXdJG5wM7JPmjobadNtH368ArAJI8E7i+qm4Grqb7gJDkycC+Q6/ZO8nTu+2XA9/YUoVLujcX1Em6R5JHACcCBwLrGIzWP87gUaWLq+qYrt9DmHhB3Y7APwILgIuBpwOHdaf/IrAKeApwBfBKF9RJ/TDcJfUuyULg7Kp6wkzXIt0fOC0vSVJjHLlLktQYR+6SJDXGcJckqTGGuyRJjTHcJUlqjOEuSVJjDHdJkhrz/wHcuWRsPTN+9wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,6))\n",
    "conversion_rates.plot(kind='bar')\n",
    "plt.title('Convertion Rate')\n",
    "plt.xlabel('Group')\n",
    "plt.ylabel('Convertion Rate')\n",
    "plt.savefig(\"convertionrate.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2e0b826-679f-40ca-922d-371b0ab44026",
   "metadata": {},
   "source": [
    "We can observe from the stats above that the treatment group perform slightly better with approximately 4.63% conversion rate that the control group with 3.92%. However, We have to identify if this results or difference is statistical significant."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f14cbb1-7b24-4866-8305-8eee111668cd",
   "metadata": {},
   "source": [
    "- In order to test for a statistically significant difference between the control and treatment groups, I will utilize a two-sided Z-test for comparing proportions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0af2338-0372-49ac-ba09-d6733c4680ba",
   "metadata": {},
   "source": [
    "## Hypothesis\n",
    "\n",
    "- Ho : CRcon = CRexp\n",
    "- Ha : CRcon &ne; CRexp\n",
    "\n",
    "- Ho: CRcon - CRexp = 0\n",
    "- Ha : CRcon - CRexp &ne; 0\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "ac110874-b1bd-4f15-aff4-a72bcab201fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "group\n",
      "A     955\n",
      "B    1139\n",
      "Name: uid, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "group\n",
       "A    24343\n",
       "B    24600\n",
       "Name: uid, dtype: int64"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Identifying how many users convert in each group\n",
    "\n",
    "converted = clean_ab[clean_ab['amount_spent'] > 0]\n",
    "num_converted = converted.groupby('group')['uid'].count()\n",
    "print(num_converted)\n",
    "\n",
    "# Number of sample in each group\n",
    "clean_ab.groupby('group')['uid'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "fd3aa7c9-8c50-4c67-ae7f-aa484d379cbd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Convertion rate control group: 0.0392\n",
      "Control group confidence Interval: (0.0368, 0.0417)\n"
     ]
    }
   ],
   "source": [
    "# Importing the neccesary modules to work with the normal distribution.\n",
    "import math\n",
    "\n",
    "# assigning each value to a variable to make the calculations easier.\n",
    "x_control = 955 # number of users that made a purchase\n",
    "N_control = 24343 # sample size for control group\n",
    "p_hat_control = x_control / N_control # This is the proportion or convertion rate for the control group\n",
    "z_score_A = 1.96\n",
    "\n",
    "#Standard error\n",
    "SE_A = math.sqrt(p_hat_control * (1- p_hat_control) / N_control)\n",
    "\n",
    "#Confidence level for control group\n",
    "\n",
    "ci_A = (p_hat_control - z_score_A * SE_A, p_hat_control + z_score_A *  SE_A)\n",
    "\n",
    "print(f\"Convertion rate control group:\", np.round(p_hat_control,4))\n",
    "print(\"Control group confidence Interval: ({:.4f}, {:.4f})\".format(ci_A[0], ci_A[1]))\n",
    "                 \n",
    "                 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9a1b24-345d-427b-9bb2-8fd7a97a8f7b",
   "metadata": {},
   "source": [
    "We can observe that the true convertion rate in the control group falls between (0.0368, 0.0417). The convertion rate for the control group is 0.0392, which falls within the confident interval indicating that the sample is representative of the population.\n",
    "\n",
    "Now, let's calculate the confidence interval for the treatment group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "c18d7098-3e84-4942-b56f-646173ac0dff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Convertion rate treatment group: 0.0463\n",
      "Treatment group confidence Interval: (0.0437, 0.0489)\n"
     ]
    }
   ],
   "source": [
    "x_treat = 1139 # number of users that made a purchase in the treatment group\n",
    "N_treat = 24600 # sample size fro treatment group\n",
    "p_hat_treatment = x_treat / N_treat  # This is the proportion or convertion rate for the treatment group\n",
    "z_score_B = 1.96\n",
    "\n",
    "#Standard error\n",
    "SE_B = math.sqrt(p_hat_treatment * (1- p_hat_treatment) / N_treat)\n",
    "\n",
    "#Confidence level for control group\n",
    "\n",
    "ci_B = (p_hat_treatment - z_score_B * SE_B, p_hat_treatment + z_score_B *  SE_B)\n",
    "\n",
    "print(f\"Convertion rate treatment group:\", np.round(p_hat_treatment,4))\n",
    "print(\"Treatment group confidence Interval: ({:.4f}, {:.4f})\".format(ci_B[0], ci_B[1]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ad1bab-2f91-4767-ae7e-5113ac45714e",
   "metadata": {},
   "source": [
    "The convertion rate for the treatment group is 0.0463 which falls within the confidence interval (0.0437, 0.0489) indicating that the sample is representative of the population. However, we still have to calculate the proportion difference to stablish if there is a statistical significance between the proportions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "7f875d10-43c2-4f51-9962-1154960dccd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reject the null\n",
      "0.0001\n",
      "Test statistics stat:  3.8643\n",
      "Z-Critical score:  1.96\n",
      "P_value;  0.0001\n",
      "Confidence Interval of 2 sample for proportion:  [0.0035 0.0107]\n"
     ]
    }
   ],
   "source": [
    "# Importing the neccesary modules to work with the normal distribution.\n",
    "from scipy.stats import norm\n",
    "\n",
    "#Significance level\n",
    "alpha = 0.05\n",
    "\n",
    "# Now, I am going to calculate the proportion of the difference.\n",
    "p_pooled = (x_control + x_treat) / (N_control + N_treat)\n",
    "pooled_variance = p_pooled * (1-p_pooled) * (1/N_control + 1/N_treat) # Variance of the difference of the two groups\n",
    "\n",
    "#Standard of error\n",
    "SE = np.sqrt(pooled_variance)\n",
    "\n",
    "#Test statistists\n",
    "test_stat = (p_hat_treatment - p_hat_control)/ SE\n",
    "\n",
    "# Critical value or z-score using the normal distribution\n",
    "z_score = norm.ppf(1-alpha /2)\n",
    "\n",
    "# Calculating the margin of error\n",
    "ME = SE * z_score\n",
    "\n",
    "#Calculating p-value\n",
    "p_value = norm.sf(test_stat)*2 # We multiply here for 2 because we are using two sided test with normal distribution.\n",
    "\n",
    "# Calculating confidence intervals\n",
    "CI = [(p_hat_treatment - p_hat_control) - SE * z_score, (p_hat_treatment - p_hat_control) + SE * z_score]\n",
    "\n",
    "if np.abs(test_stat) >= z_score:\n",
    "    print(\"reject the null\")\n",
    "    print(np.round(p_value,4))\n",
    "    \n",
    "print(\"Test statistics stat: \", np.round(test_stat,4))\n",
    "print(\"Z-Critical score: \", np.round(z_score,2))\n",
    "print(\"P_value; \", np.round(p_value,4))\n",
    "print(\"Confidence Interval of 2 sample for proportion: \" , np.round(CI,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df3a92ff-1d4d-4534-8b02-4e46a6f5f642",
   "metadata": {},
   "source": [
    "The test statistics (3.864) is greater than the z-critical score (1.96) and the p-value(0.0001) is less that the significance level of 0.05. This means that we have rejected the null hypothesis (that there is no difference between the proportions of the two groups) and we can conclude that there is a statistically significant difference between the two sample proportions.\n",
    "\n",
    "Additionally, the confidence interval for the difference in proportions (0.0035, 0.0107) does not include zero, which further supports the conclusion that there is a significant difference between the two groups.\n",
    "\n",
    "Now, let's visualize the convertion rate with the standard error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "a532fa67-ebc6-4689-979d-28e5ad158d12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEWCAYAAABFSLFOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAAsTAAALEwEAmpwYAAAaZklEQVR4nO3debhddX3v8feXJAyZQBkkzMggApYgKIrgDQ4oaJgpekEFrV7aSq9FikO5No51aK9cFIu09ZGKAmKr4lCtFoLKTBgFQYkEA2EMU8JM+N4/fr8Dm+M5JznnrHNWsvJ+Pc9+zt5r+K3vb+1z1uesYa8dmYkkSU1Zo+0CJEndYrBIkhplsEiSGmWwSJIaZbBIkhplsEiSGmWwqJMi4siI+K+26+gTEetExA8i4qGIOLftesbCyrbO1R6DRUOKiP8ZEVdGxNKIuDMi/jMi9mq7ruXJzG9m5r5t19HjMOBFwPqZefhAE0TE9hFxbkTcVwPouog4PiImjG+pIzNW6zwiZkXEM/V3cElE3BwRxwxj/rkR8WdN16XBGSwaVEQcD5wMfIayUdwC+ApwYItlLVdETGy7hgFsCfw2M58eaGREbANcBiwEXpaZ6wKHA7sD08atyiFE0dY2Y1FmTgWmA38N/HNEvKSlWrQ8menDxx89gHWBpcDhQ0yzFiV4FtXHycBaddws4HbgROAe4E7gIGB/4LfA/cBHe9qaA3wHOAdYAlwF7NIz/sPA/DruRuDgnnFHAxcBXwQWA5+qw35Vx0cddw/wMHA9sHNPP/8NuBe4DTgJWKOn3V8B/wA8ANwK7DfE+ngpMBd4ELgBOKAO/zjwJPBUXafvGWDeM4EfLec9OaC2+2Bdzkt7xi0ATgCuAx6q63HtOu43wFt7pp1Y+/vy+vpVwMW13WuBWT3TzgU+XdfvY8C2db38vr4XtwJH9q6vnnn3BK6o9VwB7Nmv3U/WdpcA/wVsMEi/ZwG39xt2D/V3E3gB8MPapwfq883quE8Dy4DH67r/ch2+A/Azyu/hzcCf9rS9P+V3bAlwB3BC23+Pq9qj9QJ8rJwP4M3A08DEIab5BHApsBGwYd04fbKOm1Xn/xgwCXhv/cP/FuU/8J3qhmrrOv2cuuE9rE5/Qt1oTarjDwc2oexlHwE8Asyo446uyzqubjTX4fnB8iZgHrAeJWRe2jPvvwHfrzVtRQm99/S0+1StfQLw55QAjQHWxSTgFuCjwJrA6+qG6SU9/TtziHV5F3DMEOO3r31+Y13WiXV5a9bxC4DL6zp6ISVMjq3jPgZ8s6ettwC/qc83pYTx/nXdvrG+3rCOnwv8ob5fEylB/HBPv2YAO/Wsr751/kLKRv4ddb6319fr97Q7v/Zrnfr6s4P0fRY1WGqNBwDPALvWYesDhwKT6/t4LvC9nvnnAn/W83oKZc/wmFrbrsB9wI51/J3A3vX5C6gB7GMY24+2C/Cxcj6AI4G7ljPNfGD/ntdvAhbU57MowTGhvp4GJLBHz/TzgIPq8znApT3j1uj9Ax9g2dcAB9bnRwN/6De+dyP3OkpgvIq6N1KHT6DsSezYM+x/AXN72rilZ9zk2oeNB6hnb0o49LZ/FjCnp39DBctTwJuHGP9/gG/3Wz93UPcuKMFyVM/4zwOn1efbUkJucn39TeBj9fmHgG/0W9ZPgXfV53OBT/SMm0LZszkUWGeIdf4O4PJ+4y8Bju5p96SecX8B/GSQvs+iBMmDwBOUPZAPDLGuZgIP9Lyey/OD5Qjgl/3m+Srwd/X5H+rvwfQ2/va68PAciwazGNhgOecrNqEcPupzWx32bBuZuaw+f6z+vLtn/GPA1J7XC/ueZOYzlENpmwBExDsj4pqIeDAiHgR2BjYYaN7+MvN84MvAqcA9EXF6REyv808aoA+b9ry+q6edR+vT3pr7bAIsrHUP1tZQFlP++x/M89Z1Xc7CwWoFHu2rMzNvoezBzI6IyZT/+L9Vp9sSOLxvvdZ1u1e/Wnrfl0coG+ZjgTsj4kcRscPy6q0GXbe99Q5iUWauRznHcgrlnwUAImJyRHw1Im6LiIeBXwDrDXHRw5bAHv36fCSwcR1/KGUP7raIuDAiXj1EXRqAwaLBXEL57/CgIaZZRPkj7bNFHTZSm/c9qSeJNwMWRcSWwD8D76ccSlkP+DXlsFafIW/TnZmnZOZuwI6Uwy9/Qzn88dQAfbhjBLUvAjbvd3J7OG39nLJBG6r9Z+uMiKCsrxVt/yzK4agDgRtr2EAJjW9k5no9jymZ+dmeeZ+3bjPzp5n5Rkr43ER5b4astxrpuu1d9hOUvayXRcRBdfAHgZdQ9oanA6+tw/t+P/r/biwELuzX56mZ+ed1GVdk5oGUQ7zfA749mppXRwaLBpSZD1GOzZ8aEQfV/wonRcR+EfH5OtlZwEkRsWFEbFCnP3MUi90tIg6pe0kfoATbpZTDL0k5R0O91HTnFW00Il4REXtExCTKeYrHgWfq3tS3gU9HxLQaYMePsA+XUf7rPrGup1nAbODsFZz/74A9I+ILEbFxrXvbiDgzItardb4lIl5f+/FByvq5eAXbPxvYl3Ke6Fs9w8+k7Mm8KSImRMTa9fLezQZqJCJeFBEHRsSUuvyllMNU/f0Y2L5erj4xIo6ghPoPV7DeQWXmk8A/Un7foBxmfQx4MCJeSFmXve4GXtzz+oe1tnfU92pS/R15aUSsWT+Ps25mPkU5nzRQ/zQEg0WDysx/pGxoT6Js1BdS9hq+Vyf5FHAl5Uqk6ylXcn1qFIv8PuUwS99J30My86nMvJGyIbmEspF4GeVqohU1nfJf9QOUwzGLgS/UccdRwub3lCvAvgV8bbiF143dbGA/yp7QV4B3ZuZNKzj/fODVlAsIboiIh4B/p6zfJZl5M3AU8KXa/mxgdl3uirR/J2X97Um5Yqxv+ELKXsxHee49/hsG3zasQfmdWES5oup/UMKq//IWA2+lBOBiysUGb83M+1ak3hXwNWCLiJhNuRpxHcp6uRT4Sb9p/x9wWEQ8EBGnZOYSSsi+rfbjLuBzlKscofzuLaiH1Y6lHCbTMEQ9WSW1KiLmANtm5lFt1yJpdNxjkSQ1ymCRJDXKQ2GSpEa5xyJJatTKeLO+cTdp0qTcZZdd2i5DklYp8+bNuy8zN+w/3ENhwLRp03LJkiVtlyFJq5SImJeZu/cf7qEwSVKjDBZgxoyhbtEkSRoOgwWYPn162yVIUmcYLMCjjz66/IkkSSvEYAEWLhz0juuSpGEyWCRJjTJYJEmNMlgkSY3yk/fA0qUQsfzpJK26/Cz4+HGPBVjxryWXJC2PwQLA1LYLkKTOMFiA8rXdkqQmGCwA3NF2AZLUGQaLJKlRBoskqVEGiySpUQaLJKlRBgsAm7ddgCR1hsECwOS2C5CkzjBYAHi47QIkqTMMFgDubLsASeoMg0WS1CiDRZLUKINFktQog0WS1CiDBYAt2y5AkjrDYAFg7bYLkKTOMFgAeKjtAiSpMwwWAO5quwBJ6gyDRZLUKINFktQog0WS1CiDRZLUKIMFgK3bLkCSOsNgAWDNtguQpM4wWAB4oO0CJKkzDBYA7mm7AEnqDINFktQog0WS1CiDRZLUKINFktQogwWAbdouQJI6w2ABYGLbBUhSZxgsACxuuwBJ6gyDBYD72i5AkjrDYJEkNcpgkSQ1ymCRJDXKYJEkNWpMgyUiNo6IsyNifkTMi4gfR8T2I2jnAxExeQTzLV2xKbcbbtOSpEGMWbBERADfBeZm5jaZuRvwEeBFI2juA8CAwRIRE0Zc5LPccZOkpozlJwP3AZ7KzNP6BmTmtVF8AdgPSOBTmXlORMwC5lCu/d0ZmAccBRwHbAJcEBH3ZeY+dU/kq8AbgL+MiFcC766L+ZfMPHl4pd47wi5Kat6ssWl1bJoFYO7cuWPX+CpoLIOlLxz6OwSYCewCbABcERG/qON2BXYCFgEXAa/JzFMi4nhgn8zs+8DJFOCyzPxgROwGHAPsAQRwWURcmJlXD1VcRLwPeF95tdZI+yhJ6qeNe5nsBZyVmcuAuyPiQuAVwMPA5Zl5O0BEXANsBfxqgDaWAf/e0953M/OROt9/AHsDQwZLZp4OnF7mmZaj65Kk5swdm1bHplkNYCxPLtwA7DbMeZ7oeb6MwYPv8RpMkqSVzFgGy/nAWvWQEwAR8SfAg8ARETEhIjYEXgtcvpy2lgDTBhn3S+CgiJgcEVOAg+swSVILxuxQWGZmRBwMnBwRHwIeBxZQrvCaClxLOXl/YmbeFRE7DNHc6cBPImJRZu7TbzlXRcTXeS6c/mV551ckSWMnMj29ELF7wpVtlyFpDLmpa15EzMvM3fsP9wMckqRGGSwA3N12AZLUGQYLUK4nkCQ1wWCRJDXKYJEkNcpgkSQ1ymABXA2S1By3qIDfxyJJzTFYJEmNMlgAuLPtAiSpMwwWoNyxX5LUBINFktQog0WS1CiDRZLUKIMFaOcbmiWpmwwWALZpuwBJ6gyDRZLUKIMFgDvaLkCSOsNgAWBp2wVIUmcYLJKkRhkskqRGGSySpEYZLACs2XYBktQZBgsAW7ddgCR1hsEiSWqUwQLAwrYLkKTOMFgAeLTtAiSpMwwWSVKjDBZJUqMMFklSowwWANZuuwBJ6gyDBYAt2y5AkjrDYJEkNcpgAeC2tguQpM4wWAB4vO0CJKkzDBZJUqMMFklSowwWSVKjJrZdwMpgo40mc/fdbVchSd3gHguw+eabt12CJHWGwSJJapTBAtx6661tlyBJnWGwAE8++WTbJUhSZxgskqRGGSySpEYZLJKkRhkswNSpU9suQZI6w2ABNt1007ZLkKTOMFgkSY0yWID58+e3XYIkdYbBAjz99NNtlyBJnWGwSJIaZbBIkhplsEiSGmWwANOnT2+7BEnqDIMFmDFjRtslSFJnGCySpEYZLMDvfve7tkuQpM4wWIBnnnmm7RIkqTMMFklSoya2XcDKYOlSiGi7Ckn6Y5ltVzB87rFIkhplsACwXtsFSFJnGCwAvKjtAiSpMwwWSVKjDBYAbm67AEnqDINFktQog0WS1CiDRZLUKINFktQogwWAF7ZdgCR1hsECwIZtFyBJnWGwAODdjSWpKQYLAH4fiyQ1xWCRJDXKYJEkNcpgkSQ1ymCRJDXKYAFgg7YLkKTOMFgAWL/tAiSpMwwWAJ5uuwBJ6gyDBYD5bRcgSZ1hsEiSGmWwSJIaZbBIkhplsEiSGmWwALBR2wVIUmcYLAC8oO0CJKkzDBYAnmy7AEnqDIMFgFvbLkCSOsNgkSQ1ymCRJDXKYJEkNcpgkSQ1ymABYOO2C5CkzjBYAFi37QIkqTMMFgAeb7sASeoMgwWA29ouQJI6w2CRJDXKYJEkNcpgkSQ1auJQIyNifeC/68uNgWXAvfX1KzNz2HdvjIhZwJOZefFw510ZlyNJer4hgyUzFwMzASJiDrA0M/+hb3xETMzMp4e5zFnAUmCsN/jDWM6MMS5FklYfQwbLQCLi65Trc3cFLoqIU4FTgQ2BR4H3ZuZNETEbOAlYE1gMHAmsAxwLLIuIo4DjgPcAj9X2NgLeDbwTeDVwWWYeXZe7L/BxYC1gPnBMZi6NiAXAGcBsYBJweK3vecvJzF8O3qvpw10NktRj1ti1PHZNM3fu3DFpd6TnWDYD9szM44HTKRvu3YATgK/UaX4FvCozdwXOBk7MzAXAacAXM3Nmz8b+BZQg+WvgPOCLwE7AyyJiZkRsQAmpN2Tmy4ErgeN76rmvDv8n4IQhlvOsiHhfRFwZEVfCohGuBklSf8PeY6nOzcxlETEV2BM4NyL6xq1Vf24GnBMRMyh7LUN96ckPMjMj4nrg7sy8HiAibgC2qm3tSNlDorZ3Sc/8/1F/zgMOWZEOZObplFAkYlquyDySNLC5Y9fy2DU9ZkYaLI/Un2sAD2bmzAGm+RLwfzPzvHoifc4Q7T1Rfz7T87zv9UTKRQM/y8y3L2f+ZYy8T5KkBozqcuPMfBi4NSIOB4hilzp6XeCO+vxdPbMtAaYNc1GXAq+JiG3rcqZExPbLmWcky5EkjVITn2M5EnhPRFwL3AAcWIfPoRwimwfc1zP9D4CDI+KaiNh7RRaQmfcCRwNnRcR1lMNgOyxntmEvR5I0epHp6YVyjmVJ22VI0h9ZmTfRETEvM3fvP9xP3gOwadsFSFJnGCwATG27AEnqDIMFKB/QlyQ1wWABnrt4TZI0WgaLJKlRBoskqVEGiySpUQaLJKlRBgsAm7ddgCR1hsECwOS2C5CkzjBYAHi47QIkqTMMFgDubLsASeoMg0WS1CiDRZLUKINFktQog0WS1CiDBYAt2y5AkjrDYAFg7bYLkKTOMFgAeKjtAiSpMwwWAO5quwBJ6gyDRZLUKINFktQog0WS1CiDRZLUKIMFgK3bLkCSOsNgAWDNtguQpM4wWAB4oO0CJKkzDBYA7mm7AEnqDINFktQog0WS1CiDRZLUKINFktQogwWAbdouQJI6w2ABYGLbBUhSZxgsACxuuwBJ6gyDBYD72i5AkjrDYJEkNcpgkSQ1yrPWwNSpsGRJ21VIUje4xyJJapTBAmy33XZtlyBJnWGwAGus4WqQpKa4RQXuvffetkuQpM4wWID777+/7RIkqTMMFklSowwWSVKjDBZJUqMMFklSoyIz266hdRGxBLi57TpatAGr9504V+f+r859B/s/2v5vmZkb9h/oLV2KmzNz97aLaEtEXGn/V8/+r859B/s/Vv33UJgkqVEGiySpUQZLcXrbBbTM/q++Vue+g/0fk/578l6S1Cj3WCRJjTJYJEmN6nSwRMSbI+LmiLglIj48wPi1IuKcOv6yiNiqZ9xH6vCbI+JN41p4Q0ba/4hYPyIuiIilEfHlcS+8IaPo/xsjYl5EXF9/vm7ci2/AKPr/yoi4pj6ujYiDx734Bozm77+O36L+DZwwbkU3aBTv/1YR8VjP78Bpw154ZnbyAUwA5gMvBtYErgV27DfNXwCn1edvA86pz3es068FbF3bmdB2n8ax/1OAvYBjgS+33ZcW+r8rsEl9vjNwR9v9Gef+TwYm1uczgHv6Xq8qj9H0v2f8d4BzgRPa7s84v/9bAb8ezfK7vMfySuCWzPx9Zj4JnA0c2G+aA4Ez6vPvAK+PiKjDz87MJzLzVuCW2t6qZMT9z8xHMvNXwOPjV27jRtP/qzNzUR1+A7BORKw1LlU3ZzT9fzQzn67D1wZWxSt8RvP3T0QcBNxKef9XRaPq/2h1OVg2BRb2vL69DhtwmvqH9BCw/grOu7IbTf+7oKn+HwpclZlPjFGdY2VU/Y+IPSLiBuB64NieoFlVjLj/ETEV+BDw8XGoc6yM9vd/64i4OiIujIi9h7twb+kiDSIidgI+B+zbdi3jLTMvA3aKiJcCZ0TEf2bmqrwHOxxzgC9m5tKG/oFf1dwJbJGZiyNiN+B7EbFTZj68og10eY/lDmDznteb1WEDThMRE4F1gcUrOO/KbjT974JR9T8iNgO+C7wzM+ePebXNa+T9z8zfAEsp55pWJaPp/x7A5yNiAfAB4KMR8f4xrrdpI+5/PQWwGCAz51HO1Ww/nIV3OViuALaLiK0jYk3Kyanz+k1zHvCu+vww4PwsZ6/OA95Wr5rYGtgOuHyc6m7KaPrfBSPuf0SsB/wI+HBmXjReBTdsNP3fum5oiIgtgR2ABeNTdmNG3P/M3Dszt8rMrYCTgc9k5qp2deRo3v8NI2ICQES8mLL9+/2wlt721Qtj+QD2B35LSdy/rcM+ARxQn69NuerjFkpwvLhn3r+t890M7Nd2X1ro/wLgfsp/q7fT74qSVeEx0v4DJwGPANf0PDZquz/j2P93UE5aXwNcBRzUdl/Gs//92pjDKnhV2Cjf/0P7vf+zh7tsb+kiSWpUlw+FSZJaYLBIkhplsEiSGmWwSJIaZbBIkhplsGi1FhEbR8TZETG/3sn4xxExrA+DjaWIOGCgO9OOsK1l9W61v46IH9TP6ww1/cyI2L+JZWv1YrBotVVvuPddYG5mbpOZuwEfAV7UQi0TBhqemedl5mcbWsxjmTkzM3emfEbpL5cz/UzKZyGkYTFYtDrbB3gqM5/9vonMvDYzfxnFF+p/99dHxBEAETErIuZGxHci4qaI+Gad9s0RcW5fO3W6H9bn+0bEJRFxVUScW29ySEQsiIjPRcRVwOER8VcRcWNEXBcRZ9dpjo76nTj1ezLOr+P/OyK2qMO/HhGnRMTFEfH7iDhsBfp+CfWmhFG+f+WSetPBiyPiJfXT2p8Ajqh7OUdExJSI+FpEXF6n7X+3XAnwJpRave0MzBtk3CGU/9h3ATYAroiIX9RxuwI7AYuAi4DXAD8HTo+IKZn5CHAEcHZEbED5JP8bMvORiPgQcDxlow3l3kwvB4iIRcDWmfnEIIepvgSckZlnRMS7gVOAg+q4GZTv0NmBcquO7wzW6bp39HrgX+ugm4C9M/PpiHgD5RYmh0bEx4DdM/P9db7PUG778e5a3+UR8fPaX+lZ7rFIA9sLOCszl2Xm3cCFwCvquMsz8/bMfIZy24utstx2/CfA7HqfrbcA3wdeRfniuIsi4hrKvZm27FnOOT3PrwO+GRFHAQPdpv7VwLfq82/UGvt8LzOfycwbGfxQ3jq1hrvqND+rw9cFzo2IXwNfpITmQPYFPlzbmEu5JcgWg0yr1ZjBotXZDcBuI5iv97tZlvHcnv/ZwJ8CrwOuzMwlQAA/q+c2Zmbmjpn5np75e//bfwtwKvByyh7ScI4o9NY02L3eH8vMmZRgC547x/JJ4IJ67mU2JTAGEsChPX3ZIsvdj6XnMVi0OjsfWCsi3tc3ICL+JMoXG/2Scn5hQkRsCLyW5d/h+kJKKLyXEjIAlwKviYhta/tTBrrqLCLWADbPzAsoXzK1LjC132QXU+5SC3BkrXHYMvNR4K+AD8Zzt0vvu6X60T2TLgGm9bz+KXBcveiBiNh1JMtX9xksWm1luQPrwcAb6uXGNwB/TzlU9F3KoalrKQF0YmbetZz2lgE/BParP8nMeykb67Mi4jrKSfMdBph9AnBmRFwPXA2ckpkP9pvmOOCY2s47gP893D731Ho1pX9vBz4P/H1EXM3zz7teAOzYd/KesmczCbiurqtPjnT56jbvbixJapR7LJKkRhkskqRGGSySpEYZLJKkRhkskqRGGSySpEYZLJKkRv1/ExOcSI7T/NEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot for the convertion rate and the standard error for each group.\n",
    "\n",
    "groups = ['Control', 'Treatment']\n",
    "conversion_rates = [p_hat_control, p_hat_treatment]\n",
    "errors = [ME, ME]\n",
    "fig, ax = plt.subplots()\n",
    "y_pos = np.arange(len(groups))\n",
    "ax.barh(y_pos, conversion_rates, xerr=errors, align='center', color='blue')\n",
    "ax.set_yticks(y_pos)\n",
    "ax.set_yticklabels(groups)\n",
    "ax.invert_yaxis()\n",
    "ax.set_xlabel('Conversion Rate')\n",
    "ax.set_title('Comparison of Conversion Rates')\n",
    "plt.axvline(x=0, color='black', linestyle='--')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "325e3cb4-343d-4ccc-8089-71e8b88936da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAAsTAAALEwEAmpwYAABVWElEQVR4nO2dd3hURdfAf4dQQhNpSoeI1ASSQAABURAQbCAKKljAV1S6FcGKirzqqyKioIIIiiBFBfkUXhUFhBdRWuhFmnQEBKVDyPn+mLvZTdgkm5DNlszveebZuffOnXvu7s49d2bOnCOqisVisVgsackXaAEsFovFEpxYBWGxWCwWr1gFYbFYLBavWAVhsVgsFq9YBWGxWCwWr1gFYbFYLBavWAVhCRlEpJqIqIjkD7QswYSIHBeRK0KlXn8hIlUcmSMCLUu4YBVEiOI0BFdKFpFTHtt35/C1LhWRj0Vkv4gcE5HNIjI4J6/hb9J8X650TkS2pVO+pYjszqFrzxeRnhkcdyk+l1w7svL9qmoxVfV6HxcjY07Um861dnj8X/eLyAQRKXax9arqTkfm8zkhp8UqiJDFaQjFVLUYsBO4xWPfpBy+3NtAMaAOUALoAGzJ4Wv4Fc/vy/nOagJ/AUMDLJonlzqydQaeF5G2gRbIj9zi3GscEA88HVhxLN6wCiKMEJFI582sjLP9rIgkicglzvZQERnh5EuIyKciclBE/hCR50Qkvf9DI2Cyqh5R1WRV3aiqX3hc9x0R2SUi/4jIchFp4XHsRRGZLiKfOb2PNSJSU0SeFpE/nfOu9yg/X0ReFZHfnPq+FpFS6dxvCREZJyL7RGSPiLziy/CCM0Q1Dfg/VR3v5XhRYA5QweOtvoKI5BORwSKyVUQOi8g0l2zOd/+Zs/+oiCwVkctFZBjQAnjPqee9zORT1WXAOszD0yXTv0Rkg4gcEZHvRKSqxzEVkSudfCEReVNEdorIARH5QEQKe5TtKCKJzne7VUTapydjmnrT/b+ISA8RWeRc94iIbBeRGzK7T+de9wPfpbnXq0RksfM9rhKRlh7HokTkZ+e/NFdERonIZ86xVEOQzm82S0T+EpEtIvKgRz0vOr/fp05d60QkwReZ8xSqalOIJ2AH0MbJ/wzc7uS/B7YCN3gc6+TkPwW+BooD1YDNwAPp1P8R5oF1P1DDy/F7gNJAfuAJYD8Q6Rx7ETgNtHOOfwpsB54FCgAPAts96poP7AFigKLAl8BnzrFqgAL5ne0ZwIdOucuA34CHffi+hgMrXTKmU6YlsDvNvkeAJUAloJBz7c+dYw8D/wcUASKAhsAlHvfUM4Nrpb2vq4CTHr9VR0yPrY7zHT4HLPY4X4ErnfzbwCyglPPb/h/wqnOsMfA30BbzclgRqJ2ejGnqTff/AvQAzjm/ZQTQG9gLiA//10rAGuAdZ7sicBi40ZGxrbNd1jn+C/AmUBC4Gvgng//Hz8BoIBKjgA4C16X5X97oyPwqsCTQbTnYUsAFsCkHfsTUDW4oMNJ5kOx3HmqvOY3kFOZBHgGcBep61PEwMD+d+gsDzwDLnQfBFhylk075I0Csk38R+MHj2C3AcSDC2S7uNOpLne35wGse5es6skZ4PgCAy4EzQGGPsl2BeZl8V7c78l2RSbmWXKggNgCtPbbLO99HfuBfwGKgvpe65uObgjjq/EbqPATFOT4HD+XtPDhPAlWdbQWuBAQ4AVT3KNsURwFjFNrb6chwgYwe9Wb4f8EoiC0ex4o455bL4P96HDjmlPvR4/cfBExMU/47oDtQBUgCingc+wwvCgKoDJwHinuUfRWY4PG/nJvmf3Yq0G052JIdYgo/FmAebg0wb2Y/ANdi3kq3qOphoAzm7f0Pj/P+wLy9XYCqnlLVf6tqQ4yCmQZM9xheedIZ/vhbRI5i5inKeFRxwCN/Cjik7onEU86n5yTlrjRyFUhTH0BVZ/8+ZyjiKOYBeJm3e3DkrAmMA7qrx+SruK1fjovI8fTOd645w+N6GzAPocuBiZgH2RQR2Ssi/xGRAhnU5Y0ymO/hCcxv6Dq/KvCOx3X/wiiDtL9XWczDeblH2f86+8E8NLdmUSaXXJn9X/a7Mqp60slmNPF8q6oWx9xnbdy/b1Wgi0t+5x6uxijjCsBfHvVD6v+KJ66yx3yRGaNwI8VayKXCKojwYzFQC+gELFDV9Zg3rxsxygPgEObNt6rHeVUwQzsZoqr/AP/GDOtEiZlveAq4AyipqpdihjHkIu6hchq5zjkye7IL04Moo6qXOukSVY32VqGIFMEMV32gqrPS3JPL+sU1gQ3mTTQtuzA9p0s9UqSq7lHVc6r6kqrWBZoBNwP3ZVCXV1T1vKoOxwx/9PG47sNprltYVRenOf0QRuFGe5Qr4XFPu4Dq6V06A7Gy/X/JDFVdAEzA9JjAyDgxzb0WVdXXgH1AKee3dFEZ7+x1yhbPaZnzElZBhBnO29VyoC9uhbAY6OXadt7epwHDRKS4M+H5OKa7fgEi8ryINBKRgiISiRm2OgpswgwRJWHGd/OLyAvAJRd5G/eISF3nQfAy8IWmMV1U1X2YOZa3ROQSZwK5uohcm06dH2DGsp/1UYYDQGkRKZGmjmGuCWIRKSsiHZ18KxGpJ2aS/B/MAzXZo66srid4DXjK+b4/AJ4WkWjnWiVEpEvaE1Q1GRgLvC0ilzllK4pIO6fIOOB+EWntfF8VRaR2ZjJm9f+SDUYAbUUk1qnzFhFpJyIRYib/W4pIJVX9A1gGvOj8F5tihiy9ybwL879/1amjPvBADsqcJ7AKIjxZgBkS+M1juzhm0s5Ff8x49TZgETAZ+Did+hQYj3mT3IuZOLxJVY9jhlX+i5m0/APz5ptet99XJmLeKvdj5k4GpFPuPsxk5XrMvMIXmKGIVIhIFeBezDDb35JmPYS3ilV1I/A5sM0Z6qgAvIOZAP5eRI5hJqybOKeUc67/D2boaYFzHzjndXYsfEb6+B1869zTg6o6A3gdM3z1D7AWSM9KaBBmjmiJU3YupkeJqv6GMTR4G9PLW4C7V5CZjFn5v2QJVT2ImQR/wXmwd8TMeR3E/JcG4n5W3Y2ZVzkMvAJMxfQkvdEVMy+xF2PQMERV5+aEzHkF1ySYxRIUiMh8zKTjR4GWJRRwTE3PYyasdwZantxGRKYCG1V1SKBlCUdsD8JiCW1iML22/ZkVDAecoc7qzhBZe0xvY2aAxQpb7Iy9xRKiiMjtwBhgkKqeDbQ8uUQ54CuMNd1uoLeqrgysSOGLHWKyWCwWi1fsEJPFYrFYvBI2Q0xlypTRatWqBVqM4GHTJvNZq1Zg5bBYgg3bNlKxfPnyQ6pa1tuxsFEQ1apVY9myZYEWI3ho2dJ8zp8fSCksluDDto1UiMgf6R2zQ0wWi8Vi8UrY9CAsaXjuuUBLYLEEJ7Zt+IxVEOFKmzaBlsBiCU5s2/AZO8QUriQmmmSxWFJj24bP2B5EuPLoo+bTTsRZLKmxbcNn/NqDEBPOcJMT7i/dIOwicrsTKjDBY9/TznmbPLxRWiwWiyWX8FsPwnF7PArj+XM3sFREZjnxCTzLFce4j/7VY19d4C4gGhP4Y66I1Ezr8tlisVgs/sOfPYjGmAhm2xw/MVMwjrXSMhTjyvi0x76OwBRVPaOq2zHuixv7UVaLJUdITjYjF4MGwTXXQPnyULAgRERAmTKQkAC9e8OMGXDqVKbVWSwBxZ9zEBVJHRdgN27f+QCISAOgsqp+KyID05y7JM25F4TDFJGHgIcAqlSpkkNiWyxZ58QJeO89+PBD2L7de5nDh01avhw++ACKF4eHH4bHHoMKFXJXXovFFwI2Se34sR+OCXieLVR1DMabJQkJCdbroCf//negJcgTJCfD2LEwZAgc8Ii8nS8fxMdDTIzpRUREwMGDsGEDLF0Kp0/DsWPw5pswciQ89RQ88wwULhy4e8kz2LbhM/5UEHtIHS+2EqnjwRbH+LKfLyJg3PjOEpEOPpxryYxmzQItQdizcyf06AHz5rn3RUebXkG3blC6tPfzTp6E2bPh3Xfh55/h7Fl45RWYPBk++wyaNs0V8fMutm34jD/nIJYCNUQkSkQKYiadU4LFq+rfqlpGVaupajXMkFIHVV3mlLtLRAqJSBRQA3f4TIsvLF5sksUv/Pe/UL++WzlUrWoe7qtXQ//+6SsHgCJFoHNnWLAAFi2CBg3M/m3bzLzFiBFgvfD7Eds2fMZvCkJVk4B+mJjFG4BpqrpORF52egkZnbsOEyR9PSbecV9rwZRFnnnGJEuOogrvvAM33QR//232PfggrFkDd99thpayQvPm8NtvMHw4FCgASUlmTqJnT5O3+AHbNnzGr3MQqjobmJ1m3wvplG2ZZnsYMMxvwlksWUTVuPFxDWEXLgyffAJdulxcvRERRik0b27q2rkTPv4YDh2CKVPsvIQlcFhXGxaLD6iaiWSXcihfHhYuvHjl4EnjxqY3ER9vtmfNgk6d4MyZnLuGxZIVrIKwWHzgpZeMxRFAlSpGOTRsmPPXufxys47CFbLgu+/grrvg3Lmcv5bFkhlWQVgsmTBmjFEQYCajFyyA6tX9d71LLoFvvjFDTgAzZ5rFdXbi2pLbWGd94cqIEYGWICz473/NwxnMSujvv4fciGxbtCh8+y20bm0W1o0bZyJkDhyY+bmWTLBtw2dEw+S1JCEhQW3IUUtOsm2bcY1x5IiZKJ43D5o0yfy8nGT/fjM3sWsXiJjeRIcMbQAtlqwhIstVNcHbMTvEFK7MnWuSJVucPGkmiI8cMdvjx+e+cgAoV84MNxUrZoaY7rsvfVceFh+xbcNnrIIIV155xSRLtnj4YbPoDeCJJ+DOOwMnS/36xpwWzNqLO+80q68t2cS2DZ+xCsJiScOkSWZVNECrVvDaa4GVB+C222DAAJNfutTORVhyB6sgLBYP/vgD+vQx+bJl4fPPIX+QmHK88QY0amTyI0fCnDmBlccS/lgFYbE4nD9vxvj/+cdsjxtn1iUECwULwtSpxk04GHccrjkSi8UfWAVhsTi8847xrgpmDuKWWwIrjzeioozfJoC9e93hlS0Wf2DNXMOVTZvMZ61agZUjRNi2zcRuOHUKrrwSEhPNWoRgRNU4C3QNMX39tTV9zRK2baTCmrnmRWrVsg3AR1ShVy93CNCxY4NXOYBZDzF2LFx6qdnu08cEH7L4iG0bPmMVRLjyf/9nkiVTPvsMfvjB5Hv2dPtBCmYqVoS33jL5PXtMRDuLj9i24TN2iClccT3l5s8PpBRBz5EjULOmca19+eUmJGjJkoGWyjeSk+Haa03QoXz5YNkytydYSwbYtpGKgA0xiUh7EdkkIltEZLCX471EZI2IJIrIIhGp6+yvJiKnnP2JIvKBP+W05F2GDDHKAYzpaKgoBzBK4f33jRlucrIZJktODrRUlnDCbwpCRCKAUcANQF2gq0sBeDBZVeupahzwH2C4x7GtqhrnpF7+ktOSd1m7FkaPNvmWLXM2tkNuERMDjz9u8r/95l5xbbHkBP7sQTQGtqjqNlU9C0wBOnoWUNV/PDaLAuEx3mUJelSNiej58+ZNfORIM/kbirzwgpmTABNJ8/jxwMpjCR/8qSAqArs8tnc7+1IhIn1FZCumBzHA41CUiKwUkQUi0sLbBUTkIRFZJiLLDh48mJOyW8KcmTPhxx9NvndvqFcvoOJcFEWLuiPd7d8Pr78eWHks4YPfJqlFpDPQXlV7Otv3Ak1UtV865bsB7VS1u4gUAoqp6mERaQjMBKLT9DhSYSep07DL0c2VKwdWjiDkzBmoXRt27IBSpeD3381nKJOcbLzNLlsGkZHG1L9KlUBLFaTYtpGKQE1S7wE8f4FKzr70mALcCqCqZ1T1sJNfDmwFavpHzDClcmXbANLh/feNcgB4+eXQVw5ghslcK6xPn4annw6sPEGNbRs+408FsRSoISJRIlIQuAuY5VlARGp4bN4E/O7sL+tMciMiVwA1gG1+lDX8mDrVJEsq/vkHhg0z+Ro14KGHAitPTtKiBXTubPKTJxuvrxYv2LbhM35TEKqaBPQDvgM2ANNUdZ2IvCwiLscA/URknYgkAo8D3Z391wCrnf1fAL1U9S9/yRqWvP++SZZUvPWW26x12DAoUCCw8uQ0//mP+56eeSawsgQttm34jF0oF67YxUAXcOAAVK8OJ05Aw4bmDTtULZcyYsAAePddk//xR7juusDKE3TYtpEK64vJYsEEETtxwuRfey08lQOYnkORIib/7LPGpNdiyQ5WQVjyBNu2wYcfmnybNiaFK+XKwSOPmPySJdbtkCX7WAVhyRO89BKcO2fywRBC1N8MHOj29vrss9YFhyV7WAURrnzxhUkWtmxxx5i+/XYz/xDulCwJTz1l8mvXwrRpgZUnqLBtw2esgghXypQxycKwYe436LzkFnvAABNXG8z8i+1FONi24TNWQYQrEyaYlMfZuhUmTjT5224LbZcaWaVoUXjySZNftw5mzAisPEGDbRs+YxVEuGIbAWB8FJ0/b/LPPx9YWQJB797uleIvv2x7EYBtG1nAKghL2LJ9O3z6qcl37AhxcQEVJyAUL+52B756tbVosmQNqyAsYcurr0JSksm/8EJgZQkk/fq5LZqGDrXrIiy+YxWEJSzZtcs9inDLLdCgQUDFCSglSrjXRSxfDnPmBFYeS+hgFYQlLBk+3L3uIS/OPaTlkUfMcBOYuQjbi7D4glUQ4crs2SblQf76C8aONfnWraFRo8DKEwyULAn9+5v8r7/CwoWBlSeg5OG2kVWsgghXihRxO+TJY4wa5fa5NGhQYGUJJh55BAoVMvk8HXUuD7eNrGIVRLgyerRJeYyTJ018aYD4+PD2uZRVLrsM7r/f5GfPhjVrAitPwMijbSM7WAURrkyblif9K4wf7473MGhQ+HpszS5PPGGizwG88UZgZQkYebRtZAerICxhQ1ISvPmmyUdFGb9LltRceaX7e/n8c9i5M7DyWIIbvyoIEWkvIptEZIuIDPZyvJeIrBGRRBFZJCJ1PY497Zy3SUTa+VNOS3gwfbo71vSTT0L+/AEVJ2hxOfFLSoK33w6sLJbgxm8KwokpPQq4AagLdPVUAA6TVbWeqsYB/wGGO+fWxcSwjgbaA6NdMaotFm+ouidey5Z1j7VbLiQhwR1lbuxYY/VlsXjDnz2IxsAWVd2mqmeBKUBHzwKq+o/HZlHAZZ3dEZiiqmdUdTuwxanPYvHK99/DqlUmP2AAFC4cWHmCHZd114kTNjyzJX38qSAqArs8tnc7+1IhIn1FZCumBzEgi+c+JCLLRGTZwYMHc0zwsGD+/DwVc9c14Vq0KPTpE1hZQoG2bd2+qd55B06fDqg4uUseaxsXQ8AnqVV1lKpWBwYBz2Xx3DGqmqCqCWVdju8teY516+DHH02+Z0+391JL+oiYqHMABw/C5MmBlccSnPhTQewBKntsV3L2pccU4NZsnmtJy5tvuk16wpx33zWfIsYxncU3unSBik6/fMSIPOR+Iw+1jYvFnwpiKVBDRKJEpCBm0nmWZwERqeGxeRPwu5OfBdwlIoVEJAqoAfzmR1nDj2++MSnMOXLE7dL7ppuMGafFNwoUcCvUNWvcvbCwJ4+0jZzAbwpCVZOAfsB3wAZgmqquE5GXRaSDU6yfiKwTkUTgcaC7c+46YBqwHvgv0FdVz/tLVkvo8tFHcOqUybs8llp856GH3BP61uTVkhbRMOlXJiQk6LJlywItRvDQsqX5DOPJuKQkqF7dLPaqU8fMRdiV01mnd2/44AOT37ABatcOrDx+Jw+0jawgIstVNcHbsYBPUlss2eX//s+9EnjAAKscssujj7rz77wTMDEsQYhVEOFK4cJhvxjA5ZSvRAm4997AyhLK1KoFN95o8p98AocPB1Yev5MH2kZOYRVEuDJnTliHDlu92j1C0LOnWf9gyT6PPWY+T51yx9IIW8K8beQkVkFYQhKXaWu+fNC3b2BlCQdat4Z69Uz+vffc0fgseRurIMKVoUNNCkMOHYLPPjP5Dh2M51bLxSHinovYs8c4Pgxbwrht5DRWQYQrP/4YtobtH33kdg0xYEDGZS2+062bCSoExuQ1TAwcLySM20ZOYxWEJaRISjIhRQFiYtwWi5aLJzLSmLwCLFsGv/wSWHksgccqCEtIMXMm7N5t8ta0Nefp1cussAb3PI8l72IVhCWkcJm2liwJd98dWFnCkXLljI8mgC++gL17AyuPJbBYBRGulC5tUhixciUsXGjyDz4IRYoEVp5wpX9/85mUBB9+GFhZ/EIYtg1/YV1tWEKG+++HCROMaev27VClSqAlCk9UoUkTWLoULr/crFYvWDDQUln8hXW1YQl5/vzTHbOgUyerHPyJiLsXceBAmJu8WjLEKohw5emnTQoTxo6Fs2dN3pq2+p877nCbvIbdZHWYtQ1/YhVEuPLLL2Fjp3juHIwebfKxsdCiRWDlyQsUKmRcgQP8+iv8Fk7RWMKobfgbqyAsQc9XX7mtaaxpa+7Rqxfkz2/yYdeLsPiEVRCWoMflgrp0aejaNbCy5CUqVoTbbjP5qVPNfIQlb+FXBSEi7UVkk4hsEZHBXo4/LiLrRWS1iPwoIlU9jp0XkUQnzUp7riVvsHSpezTg4Yetl+bcxjVZfe4cjBkTWFksuY/fFISIRACjgBuAukBXEambpthKIEFV6wNfAP/xOHZKVeOc1AFL1qhUyaQQxzW0ERHhdgNhyT2aN4e4OJP/4IMw8fIaJm0jN/BnD6IxsEVVt6nqWWAK0NGzgKrOU9WTzuYSwP5qOcVnn7ldnoYo+/fDlCkmf/vttk0HAhG31djevWY+KOQJg7aRW/hTQVQEdnls73b2pccDgGcUj0gRWSYiS0TkVm8niMhDTpllBw8evGiBLcHFmDHuN1Zr2ho47rrLvfDYTlbnLYJiklpE7gESgDc8dld1Vvd1A0aISPW056nqGFVNUNWEsmXL5pK0IcKjj6YONhxinD0L779v8g0aQLNmgZUnL1O4sHFtAvC//xmXJyFNiLeN3MSfCmIPUNlju5KzLxUi0gZ4Fuigqmdc+1V1j/O5DZgPxPtR1vAjMdGkEGX6dDPEBNa0NRjo3du4OIEw6EWEeNvITfypIJYCNUQkSkQKAncBqayRRCQe+BCjHP702F9SRAo5+TJAc2C9H2W1BBkur62XXWaGOCyBpUoVuPVWk5882UT1s4Q/flMQqpoE9AO+AzYA01R1nYi8LCIuq6Q3gGLA9DTmrHWAZSKyCpgHvKaqVkHkETxX7j78sFnVawk8LpPXM2eM6xNL+JPfn5Wr6mxgdpp9L3jk26Rz3mKgnj9lswQvrt5D/vxmNa8lOLj2WhPFb+1aMz80cKB7pbUlPAmKSWqLH6hZ06QQY+9emDbN5Lt0gQoVAiuPxY2nl9ddu+DrrwMrT7YJ0bYRCGw8CEtQ8cILMHSoyf/yC1x1VWDlsaTmxAmoXBmOHIFrroEFCwItkeVisfEgLCHBmTNmtS5Ao0YmaI0luChaFB54wOR//hlWrQqsPBb/4pOCEJGvROQmEbEKJVR46CG3v+YQYepUcK13fOQRa9oarPTtG+ImryHYNgKFrw/80ZgFa7+LyGsiUsuPMllygs2bTQoRVN2T0+XKmfkHS3BSrRp0dJzmTJoUgiavIdY2AolPCkJV56rq3UADYAcwV0QWi8j9IlLAnwJa8ga//ALLl5t8r142BnKw43J9cvo0fPRRYGWx+A+fh4xEpDTQA+iJ8cL6DkZh/OAXySx5ClfMhwIFzNoHS3Bz7bVQzzFEHzUKkpICK4/FP/g6BzEDWAgUAW5R1Q6qOlVV+2MWulks2Wb3bvjyS5O/804zxGQJbjy9vO7eDTNmBFYei3/wtQcxVlXrquqrqroPwOUKIz3zKEuAiYtzO/IPct5/H86fN3nrtTV06NYNSpUyedf8UUgQQm0j0Pi0DkJEVqhqg8z2BRK7DiI0OXXK+Pk5dAiaNoXFiwMtkSUrDB4Mr79u8suXG8+7ltAi2+sgRKSciDQECotIvIg0cFJLzHCTxXJReDp+c63StYQOffqEuMmrJUMy7EGISHfMxHQC4Pl6fgyYoKpBE1/K9iDScM895jOII2epQmwsrFkDFSvC9u1mktoSWnTubOaQChY0LjguuyzQEmVCCLSN3CTbPQhV/URVWwE9VLWVR+oQTMrB4oXdu00KYubPN8oBzJuoVQ6hiWve6OzZEPHyGgJtI1jIbIjJUbVUE5HH06ZckM8SxrhMWyMj7cLWUKZFC9MTBBg92h0m1hL6ZGbFVNT5LAYU95IslmyxdSvMcqJ/3HMPlCkTWHks2cfT5HXvXvjKji2EDRl6c1fVD53Pl3JHHEte4b33zBwEWNPWcKBbNxg0yBgcjBxp1rNYQh9fF8r9R0QuEZECIvKjiBz0GH7K6Lz2IrJJRLaIyGAvxx8XkfUistqpt6rHse4i8ruTumfttiw0bWpSEHLsGHz8sclfd517Ra4ldPEcJly8GILaXiSI20bQoaqZJiDR+ewEjANKAKsyOScC2ApcARQEVgF105RpBRRx8r2BqU6+FLDN+Szp5EtmdL2GDRuqJTQYOVLV9B9Uv/460NJYcopdu1QjIszveu+9gZbG4ivAMk3nuerrSmrXUNRNwHRV/duHcxoDW1R1m6qeBaYAHdMop3mqetLZXAJUcvLtgB9U9S9VPYLx99TeR1ktQUxysnvVbfXqcNNNgZXHknNUqgS3327yU6bA/v2Blcdy8fiqIL4RkY1AQ+BHESkLnM7knIrALo/t3c6+9HgAmJOVc0XkIRFZJiLLDroCCVgMt9/ubq1BxOzZsGWLyffvDxERgZXHkrO45pPOnYMxYwIrS7oEadsIRnx19z0YaAYkqOo54ARpegMXgzOfkQC8kZXzVHWMqiaoakLZsmVzSpzw4PBhk4IMl2lr8eJw//2BlcWS8zRr5na38f77Zm1E0BGkbSMYyUqEuNrAnSJyH9AZuD6T8nuAyh7blZx9qRCRNsCzQAdVPZOVcy2hxbp1MHeuyd9/P1xySWDlseQ8niav+/fDtGmBlcdycfhqxTQReBO4GmjkpMy8uC4FaohIlIgUBO4CZqWpNx74EKMc/vQ49B1wvYiUFJGSGGX0nS+yWoIXV+9BxPpdCmfuvNPtbuOtt9zmzJbQI8N1EB4kYCyQfP6pVTVJRPphHuwRwMequk5EXsbMms/CDCkVA6aLCUC8U40bj79EZChGyQC8rKp/+XptS/Bx6JDb9c3NN8OVVwZWHov/iIw0LwDPPw+JifDTT9C6daClsmQHXxXEWqAcsC8rlavqbGB2mn0veOTbZHDux8DHWbmexYMga5Hvv29cewM8+mhARbHkAr17w7//bX7zt94Ksr9jUAkT3PgaD2IeEAf8BrjmCVDVDn6TLItYb67By+nTJubDwYMQH2/iBpgOoyWc6dvX+GYCM/9Ut25g5bF4JyNvrr72IF7MOXEseY2JE41yAHjySasc8gqPPWZ6jqowfDh89FGgJbJkFV/NXBcAO4ACTn4psMKPclkulhtuMCnAJCebIQaAypWhS5fAymPJPa68Ejo6xvATJwbRwrkgaRuhgK9WTA8CX2AsjsAsWpvpJ5ksOcGpU+5B/wDy7bewaZPJP/qojfmQ13jySfN59iyMGhVYWVIIkrYRCvi6DqIv0Bz4B0BVfweCPW6UJQh4w1n6WKIEPPhgYGWx5D7NmkGTJiY/ejScPJlxeUtw4auCOOP4UwJARPID1rrZkiG//goLF5r8ww+b1dOWvIWIuxfx118wYUJAxbFkEV8VxAIReQYoLCJtgenA//lPLEs44Jp7yJ/fxnzIy3TqBFFRJv/223D+fGDlsfiOrwpiMHAQWAM8jFnb8Jy/hLLkADffbFKA2LbNBLIHE0ymYkZuGi1hTUSEsWgC46jx/wL9ahngthFK+LQOAsDx4IqqBqXbVLsOIrgYMADefdfkV6+2QYHyOsePGyu2o0eheXNYtCjQEllcZLQOIsMehBheFJFDwCZgkxNN7oWMzrPkbQ4fhnHjTL5dO6scLFCsmFldDfC//7nnpizBTWZDTI9hrJcaqWopVS0FNAGai8hjfpfOkn1atjQpALz7rttaxTVBabE88ojx0wTw6qsBFCSAbSPUyExB3At0VdXtrh2qug24B7jPn4JZQpNjx9wR4xISrNsbi5vLL4cHHjD5OXNg5crAymPJnMwURAFVPZR2pzMPYZc8WS7ggw/gyBGTf+YZ61bDkponn3RHEXzttcDKYsmczBRERvGggjFWlCWAnD7tNm2tW9ftZsFicVGtGtx9t8lPnw6bNwdUHEsmZKYgYkXkHy/pGGCnHi2p+PhjOHDA5J9+GvJlJV6hJc8waJD5VIXXXw+sLJaMybAJq2qEql7iJRVX1UyHmESkvYhsEpEtIjLYy/FrRGSFiCSJSOc0x86LSKKTZqU915IJd9xhUi5x7hz85z8mHxUFd92Va5e2hBh165rFcwCffgq7duWyALncNkIZX919ZxkRiQBGAW2B3cBSEZmlqus9iu0EegDebF1OqWqcv+QLe/r0ydXLTZ4Mf/xh8oMGmdXTFkt6PP00zJgBSUlmWHLEiFy8eC63jVDGn4MAjYEtqrrN8eM0BUg1Kq2qO1R1NZDsRznyJidP5ppntPPn3WaL5ctD9+65cllLCNOoEbRx4kmOGeOOF5Ir5GLbCHX8qSAqAp6dx93OPl+JFJFlIrJERG7NUcnyAjfeaFIuMGOG26X3E0+4bd0tlox45hnzeeqUCSiUa+Ri2wh1gnkasaqz/LsbMEJEqqctICIPOUpk2cFcfQWxuEhOhldeMflSpYzXVovFF1q2hKZNTf699+DQBQb1lkDjTwWxB6jssV3J2ecTqrrH+dwGzAfivZQZo6oJqppQtmzZi5PWki1mzoRVq0z+0UeNSwWLxRdE4KWXTP74cXjzzcDKY7kQfyqIpUANEYkSkYLAXYBP1kgiUlJECjn5Mhh3H+szPsuS2yQnw5AhJl+ypHGlYLFkhTZtTFAhML0IOxAQXPhNQahqEtAP+A7YAExT1XUi8rKIdAAQkUYishvoAnwoIuuc0+sAy0RkFTAPeC2N9ZMlCPjiC1i71uQHDoRLLgmsPJbQw7MXceKEOwKhJTjw2d13sGPdfafBFbqrRw+/VH/+vPHSumEDlClj4j/YiHGW7KAK11xjXIAXKQLbt8Nl/gxo7Oe2EWpk5O7bWquHK37+80+bZpQDwFNPWeVgyT6uXkTr1sb69D//8fN8hFUMPhPMVkyWi+HQIb+ZhSQlwYsvmvxll9l1R5aLp1Ur04sAGD0a9u/348X82DbCDasgwpXOnU3yA59/7nayNngwFC3ql8tY8hCecxGnTvnZ06sf20a4YRWEJUucPevuPZQrB716BVQcSxjRsiVcd53Jjx5t5iIsgcUqCEuWGDPGTEgDPPssFC4cWHks4YXLZcu5c/CCDWwccKyCsPjMsWPw8ssmf8UV8NBDgZXHEn40buwe/Zk0yb0I0xIYrIKw+Mzw4e6FTMOGQcGCgZXHEp4MG2aizqkar6+WwGHNXMOV3r1ztLoDB9ymhw0aWHf6Fv9Rsyb07AkffmhiV8+fb+YncowcbhvhjF0oZ/GJ/v2NKwSAH35wu2q2WPzBvn1w5ZVmXUTjxrBkiY1v7i8yWihnh5jClV27cixU15Yt8MEHJt+2rVUOFv9Tvrxx/gjw22/w5Zc5WHkOto1wxyqIcOXee03KAQYONIvjwM/26RaLB089BaVLm/ygQXDmTA5VnINtI9yxcxB5mHPnzrF7925Onz6dbpnTp02Mh4cfNgviChd2u9iwWPzNt9/CkSMmv3IllCiRcfnIyEgqVapEgQIF/C9cHsAqiDzM7t27KV68ONWqVUO8DPCqwvr1xhlfvnwQE2Mtlyy5S3Ky+Q+ePm3+g1deCek9+1WVw4cPs3v3bqKionJX0DDFDjHlYU6fPk3p0qW9KgcwJq2nTpl8+fJWOVhyn3z5oLITdiw5GfZkEHJMRChdunSGPWJL1rAKIo+TnnJISoK9e02+YEG4/PJcFMpi8aBECffQ0qFDJm5EeqT3f7ZkDzvEFK488cRFnb5vn3tiulIl8yZnsQSKypXhn3/MsOeuXVCr1kWYvV5k28hL+LXZi0h7EdkkIltEZLCX49eIyAoRSRKRzmmOdReR353U3Z9yhiW33GJSNjh50iyMAxNjumTJHJQrDcOGDSM6Opr69esTFxfHr7/+CsCIESM4efJkjl2nWrVqHLoIF88TJkygX79+Xvfny5eP1atXp+yLiYlhx44d2b5WVtmxYwcxMTFe9xcuXJj4+Hjq1KlD48aNmeAKlgPMmjWL1zIwS0tMTGT27NnpHl+2bBkDBgwA4MUXX+TNLAZxSPsb33jjjRw9etRr2chIdxCh48fhr7+ydKnUXETbyGv4TUGISAQwCrgBqAt0FZG6aYrtBHoAk9OcWwoYAjQBGgNDRMSPj6kwZNMmk7KIKuzc6d6uUsV/C5R++eUXvvnmG1asWMHq1auZO3culZ0B55xWEFnl/PnzPpetVKkSw4YNy5VrZZXq1auzcuVKNmzYwJQpUxgxYgTjx48HoEOHDgwefMF7WwoZKYikpCQSEhIYOXJktmVL+xvPnj2bSy+9NN3y5cu7J6h37XL3cLNMNttGXsSfPYjGwBZV3aaqZ4EpQEfPAqq6Q1VXA8lpzm0H/KCqf6nqEeAHoL0fZQ0/XLapPvLoo8adwdVXw913m1P794cbbzT7s5tci528sW/fPsqUKUOhQoUAKFOmDBUqVGDkyJHs3buXVq1a0apVKwB69+5NQkIC0dHRDBkyJKWOatWqMWTIEBo0aEC9evXYuHEjAIcPH+b6668nOjqanj174ukx4NZbb6Vhw4ZER0czZsyYlP3FihXjiSeeIDY2ll9++YXx48dTs2ZNGjduzP/+97907+Pmm29m3bp1bPLy0Pn888+pV68eMTExDBo0KN1rFStWjIEDBxIdHU2bNm347bffaNmyJVdccQWzZs0CTI+gRYsWNGjQgAYNGrB48eL0v1wvXHHFFQwfPjzloe7ZK5o+fToxMTHExsZyzTXXcPbsWV544QWmTp1KXFwcU6dO5cUXX+Tee++lefPm3HvvvcyfP5+bb745pf5Vq1bRtGlTatSowdixYwEuKNOvXz8mTJjg9Tf27OUNHz6cmJgYYmJiGDFiBAC7d++gS5c6vPLKg9x2WzTXXXc9p1xWFFkhi20jT6OqfklAZ+Ajj+17gffSKTsB6Oyx/STwnMf288CTXs57CFgGLKtSpYpaPLj2WpMyYP369amKm/5DzqaMRDh27JjGxsZqjRo1tHfv3jp//vyUY1WrVtWDBw+mbB8+fFhVVZOSkvTaa6/VVatWpZQbOXKkqqqOGjVKH3jgAVVV7d+/v7700kuqqvrNN98okFKfq66TJ09qdHS0Hjp0SFVVAZ06daqqqu7du1crV66sf/75p545c0abNWumffv2veAexo8fr3379tVPPvlE77vvPlVVjY6O1u3bt+uePXtS6jh37py2atVKZ8yYccG1XNuzZ89WVdVbb71V27Ztq2fPntXExESNjY1VVdUTJ07oqVOnVFV18+bN2rBhQ1VV3b59u0ZHR18gm7f9R44c0cjIyFSyq6rGxMTo7t27U8qkPa6qOmTIEG3QoIGePHlSVVXnzZunN910U8qx+vXr68mTJ/XgwYNaqVIl3bNnT6oyqqp9+/bV8ePHp/x2nr+xa3vZsmUaExOjx48f12PHjmndunV1xYoVun37do2IiNCZM1fq0qWqbdp00bFjJ15w357/a6/40DbyEsAyTec5HtKT1Ko6BhgDxhdTgMUJaeLijK35uXNmu3BhyJ8D/464uPSPFStWjOXLl7Nw4ULmzZvHnXfeyWuvvUYPLzGDp02bxpgxY0hKSmLfvn2sX7+e+vXrA3DbbbcB0LBhQ7766isAfv7555T8TTfdREmPiZSRI0cyY8YMAHbt2sXvv/9O6dKliYiI4Pbbbwfg119/pWXLlpQtWxaAO++8k82uMHpe6NatG8OGDWO7R5SbpUuXpqrj7rvv5ueff+bWW29NdS2AggUL0r696STXq1ePQoUKUaBAAerVq5cyn3Hu3Dn69etHYmIiERERGcqTHpqO77XmzZvTo0cP7rjjjpTv0xsdOnSgcDpBQDp27EjhwoUpXLgwrVq14rfffstwyCg9Fi1aRKdOnSjqhCq87bbbWLhwIR06dCAqKop27eJYtw5q127I6tU7ULV+mvyFPxXEHqCyx3YlZ5+v57ZMc+78HJHK4pWhQ93DsiVKmAVJudHoIiIiaNmyJS1btqRevXp88sknFyiI7du38+abb7J06VJKlixJjx49Utm6u4aoIiIiSMpkYHr+/PnMnTuXX375hSJFitCyZcuUuiIjI4mIiMjWfeTPn58nnniC119/3afyaa9VoECBFBPNfPnypdxTvnz5Uu7p7bff5vLLL2fVqlUkJycTGRmZZTlXrlxJnTp1Ltj/wQcf8Ouvv/Ltt9/SsGFDli9f7vX8ohnEl01rYioi5M+fn+Rk9wjyxa5RKFSoEJGRZj4iX74ITp06xYEDJrqhJefx5xzEUqCGiESJSEHgLmCWj+d+B1wvIiWdyenrnX0WP3D+PLiMbvLl8+/EtCebNm3i999/T9lOTEykatWqABQvXpxjx44B8M8//1C0aFFKlCjBgQMHmDNnTqZ1X3PNNUyebGwf5syZwxHHX8Pff/9NyZIlKVKkCBs3bmTJkiVez2/SpAkLFizg8OHDnDt3junTp2d6zR49ejB37lwOOkEzGjduzIIFCzh06BDnz5/n888/59prr820nvT4+++/KV++PPny5WPixIlZntzesWMHTz75JP3797/g2NatW2nSpAkvv/wyZcuWZdeuXal+A1/4+uuvOX36NIcPH2b+/Pk0atSIqlWrsn79es6cOcPRo0f58ccfU8qnV3+LFi2YOXMmJ0+e5MSJE8yYMYMWLVqkKlOunHvCes8e0/u15Dx+60GoapKI9MM82COAj1V1nYi8jBnzmiUijYAZQEngFhF5SVWjVfUvERmKUTIAL6vqxRi25T2ee87nonv2uB2hVawIzsur3zl+/Dj9+/fn6NGj5M+fnyuvvDJl0vihhx6iffv2VKhQgXnz5hEfH0/t2rWpXLkyzZs3z7TuIUOG0LVrV6Kjo2nWrBlVqlQBoH379nzwwQfUqVOHWrVqcdVVV3k9v3z58rz44os0bdqUSy+9lLiMxsocChYsyIABA3jkkUdS6njttddo1aoVqspNN91Ex44dM6klffr06cPtt9/Op59+Svv27TN8m3exdetW4uPjOX36NMWLF2fAgAFeh/AGDhzI77//jqrSunVrYmNjqVKlCq+99hpxcXE87UPknvr169OqVSsOHTrE888/T4UKFQC44447iImJISoqivj4+JTyaX9jFw0aNKBHjx40btwYgJ49exIfH5/KdDhfPmN+7VobsWNHFtZGZKFt5HVsPIg8zIYNG6hUqU7K0FKxYhe5AMliyWV27oQ//zT5SpVMz2LDhg1eh9Es3rHxIPIiiYkmZUBycuqhpWrVrHKwhBaePV6fh5p8aBsWQ0hbMVkywLUAYf78dIscPepuXBUrmtWqFksoERFhXmw2bTJDTdu3m88M8aFtWAy2B5FH+e9/wTU/WKyY242BxRJqFC/u/v+eOGHmJSw5g1UQeZD9+6G7493KDi1ZwgHPHvDRo/DLLwEVJ2ywCiKPkZxslINrYq9qVTu0ZAl9IiLgiivcLzpduxpFYbk4rILIYwwfDt9/b/JFi7pj/losoU6RIsaSCeCPP6BXLx/mIywZYhVEuPLvf5vkwbJl4DJnr14dSpVKe1I5QHIwZb68NSIigri4OGJiYrjllltS3D3v3buXzp07Z3wyxl2HN2bOnMn69eu9HnvxxRepWLEicXFx1K1bl88//zzT6zRr1izTMr7I8cILLzB37txs1eXJ/PnzKVGiBHFxcdSuXZsnn3wy23Vl5vY7lLjsMuMmBmDqVPj4Yy+FvLQNSzqk56Qp1JLLcZnFOwcPqlatahzo5c+v+ttv3pya+eOnyZiiRYum5O+77z595ZVXsnRfnud70r17d50+fbrXY0OGDNE33nhDVY3Tu+LFi+vZs2ezdF1fyUiOi8HTCd7Jkye1Vq1aumjRohy/TiiyZs16LVfO/NcjI1WXLw+0RMENGTjrsz2IcGXxYpMwrjS6djXdboDXX4dGjQIoWzo0bdqUPU7QYc8gOCdPnuSOO+6gbt26dOrUiSZNmuC5KPLZZ58lNjaWq666igMHDrB48WJmzZrFwIEDiYuLY+vWreles0aNGhQpUiTFFccbb7xBo0aNqF+/fiq34p49lfTKfPrpp9SvX5/Y2Fjuvfder3L06NGDL774AoAff/yR+Ph46tWrx7/+9S/OOMvZ03Nhnh6FCxcmLi4u5bv7/vvvadq0KQ0aNKBLly4cP34cMPEWateuTcOGDRkwYECKG25Pt987duzguuuuo379+rRu3ZqdTnCQHj16MGDAAJo1a8YVV1yRcg/BSEQETJ5sDDBOn4ZOnUyo0hQ82oYlY6yCCFeeecYkjGcB16jGnXfCY48FUK50OH/+PD/++CMdOnS44Njo0aMpWbIk69evZ+jQoakcyZ04cYKrrrqKVatWcc011zB27FiaNWtGhw4deOONN0hMTKR69erpXnfFihXUqFGDyy67jO+//57ff/+d3377jcTERJYvX87PP/+cqnx6ZdatW8crr7zCTz/9xKpVq3jnnXcylOP06dP06NGDqVOnsmbNGpKSknj//fdTjpcpU4YVK1bQu3fvTCO1HTlyhN9//51rrrmGQ4cO8corrzB37lxWrFhBQkICw4cP5/Tp0zz88MPMmTOH5cuXp/iLSkv//v3p3r07q1ev5u67706JGAcmfseiRYv45ptvMgw0FAy0agVvvGHyO3ea/32KH0ePtmHJGKsgwpwvvgDX8HJMDIwbF1wmradOnSIuLo5y5cpx4MAB2rZte0GZRYsWcddddwEmnKfLzTcY/0euN+GGDRv6HOrz7bffJjo6miZNmvDss88C5uH//fffEx8fT4MGDdi4cWMqZ4IZlfnpp5/o0qULZcqUAaDUhRM8qdi0aRNRUVHUrFkTgO7du6dSRp4uzNO7p4ULFxIbG0vFihVp164d5cqVY8mSJaxfv57mzZsTFxfHJ598wh9//MHGjRu54ooriIqKAqBr165e6/zll1/o1q0bAPfeey+LFi1KOXbrrbeSL18+6tatywFXTNog5rHHwPnb8NNP7vk3i+9YBRHG/HMM7rvP5EuUgK++MpZLwUThwoVJTEzkjz/+QFUZNWpUls73dJPti7tvF4899hjr1q3jyy+/5IEHHuD06dOoKk8//TSJiYkkJiayZcsWHnjggVTn+VImJ/DFhXmLFi1YtWoV69atY9y4cSQmJqKqtG3bNkW+9evXM27cuByVCdKPKxFMiMBHH4HrfeLNN+HTTwMrU6hhFUSYcvo0rF0Dp06ZsdjPP4caNQItVfoUKVKEkSNH8tZbb13wQGzevDnTpk0DYP369axZsybT+nx1Vd2hQwcSEhL45JNPaNeuHR9//HHKmP2ePXv407VgxCG9Mtdddx3Tp0/n8OHDAPz1118ZylGrVi127NjBli1bAJg4cWK2XYFHRUUxePBgXn/9da666ir+97//pdR74sQJNm/eTK1atdi2bVtKb2Tq1Kle62rWrBlTpkwBYNKkSRe42Q41ihaFGTPcFns9e8KRowEVKaSwCiIM+ftvWLMGzjrR4d57D264wZczL89hSbJWX3x8PPXr17/A7LRPnz4cPHiQunXr8txzzxEdHU2JEiUyrOuuu+7ijTfeID4+PsNJajCmp8OHD6dNmzZ069aNpk2bUq9ePTp37pzycHf1Uq6//nqvZaKjo3n22We59tpriY2N5fHHH89QjsjISMaPH0+XLl2oV68e+fLlo1evXln6vjzp1asXP//8MydOnGDChAl07dqV+vXr07RpUzZu3EjhwoUZPXo07du3p2HDhhQvXtzrd/juu+8yfvx46tevz8SJE3nnnXeyLVOwcMUVRkkULGgiJq5bCydOBlqqECE986ZQS9bM1XDqlGqrVqqxrNRYVupjj6VfNtPYvUFCUlJSSizmLVu2aLVq1fTMmTO5dv1Dhw5pOMQ8P3bsmKqqJicna+/evXX48OEBlsg/pPe/njTJmL7GslJvKL9C9+zJZcGCFMI1JrUlNefOGWsNE3sljo4t16VYcoQyJ0+epFWrVpw7dw5VZfTo0RQsWDBXrr13715atmx5UQvRgoWxY8fyySefcPbsWeLj43n44YcDLVKu0q2bcW//7LNxrNoHbdsmsWBBfhy7Aos30tMcOZGA9sAmYAsw2MvxQsBU5/ivQDVnfzXgFJDopA8yu1Ze70GcP696zz3mDQlUH6/ypp4ecVeG54RKD8JiyQoZ/a+Tk1VH3DxRW/ODgmqDBkl69GguCheEEIiFciISAYwCbgDqAl1FpG6aYg8AR1T1SuBtwDPi+1ZVjXNS9gdn8wDJydCnD3z2mdlOuGI5r0cOotBEuxjIYvFEBAYce513LzPrO1asiOCmm5I5cSLAggUp/pykbgxsUdVtqnoWmAKkDcjbEfjEyX8BtBYJJiv94Of8eXjwQfjwQ7Ndp+IG5jzVjvwRWQtob7HkFQSoXft3OnX6CoD//S8f7dsn2zgSXvCngqgI7PLY3u3s81pGVZOAvwGXf9EoEVkpIgtExKutnYg8JCLLRGRZeitDw5mkJLj/frdDstoVNjL36daUKX44sIJZLEGOiPL551254YbZACxalI82bZJxrJMtDsFq5roPqKKq8cDjwGQRuSRtIVUdo6oJqppQtmzZXBcykJw+bfwrTZxotmMqr2P+c9dSoeS+wApmsYQIhQqdZcaMTnTsOBOApUvz0apVMiGwSDzX8KcV0x6gssd2JWeftzK7RSQ/UAI47EycnAFQ1eUishWoCSzDwpEjcOut4PLMEFdtFT8MzoGew1fl4HQOto7Iy+G2/RkWKVasWMqis8w4ePAgN998M2fPnmXkyJGsWbOGPn36eC0bERFBvXr1SEpKIioqiokTJ3LppZemW/cHH3xAkSJFuM+19NxHjh49yuTJk1Pk2Lt3LwMGDMgRZ3YtW7Zk3759REZGUrBgQcaOHUtcXFy26rrxxhuZPHlyht9BXqRQobNMn96Fe++dyNSpd7F6dT6aNj3PnDkR1KoVaOkCjz97EEuBGiISJSIFgbuAWWnKzAKc4Jd0Bn5SVRWRss4kNyJyBVAD2OZHWUOGnTuheXO3cmhRexE/PdPyQuXwL+A5n1bHuclJ5eCH+n788Ufq1avHypUrqVy5MqNHj063rMuFx9q1aylVqlSmLjx69eqVZeUARkF4ylGhQoUc9XQ6adIkVq1aRZ8+fRg4cGC265k9e7ZVDi4+7A0fusMoFiiQxKRJd/OvfxmXJNu3R9C0aTILFwZKwODBbwrCmVPoB3wHbACmqeo6EXlZRFwuO8cBpUVkC2YoyeUi8hpgtYgkYiave6lqnh8dXLjQuOnesMFsd2nyJd8PakPJokcvLFwBqBaa4eK2bt2asuK3RYsWbNy4kcTERJ566im+/vpr4uLiGDRoEFu3biUuLi7TB6enG3FvdYMJIuTymppemQMHDtCpUydiY2OJjY1l8eLFDB48OJUcnm7KT58+zf3330+9evWIj49nnlmgwoQJE7jtttto3749NWrU4Kmnnsr0O/G8hxMnTvCvf/2Lxo0bEx8fz9dffw1k7Ba9WrVqHHJ8Xg8fPpyYmBhiYmIYMWIEYNx816lThwcffJDo6Giuv/56Tp065dsPFmrUqgS1IlLtiohI5qOPevLSSy8AcOSImZP45BNvFeQh0rN/DbUUzusgkpNV33vPBPpxrXN49IYRen6iqE7Ce3oC1RFdMqz3Anvx9Oq6mJQJ3gL+XHfddbp582ZVVV2yZIm2atVKVVXHjx+vffv2VVXV7du3a3R0dKb1JiUlaefOnXXOnDkZ1u0ZRCi9MnfccYe+/fbbKfUePXr0Ajk8t9988029//77VVV1w4YNWrlyZT116pSOHz9eo6Ki9OjRo3rq1CmtUqWK7ty584J7uPbaa3Xp0qWqqvr222/r008/raqqTz/9tE6cOFFVVY8cOaI1atTQ48eP6xtvvKEPPfSQqqquWbNGIyIiUs6vWrWqHjx4UJctW6YxMTF6/PhxPXbsmNatW1dXrFih27dv14iICF25cqWqqnbp0iXlGqFGput7Zj2rOquwpvc4mTDhPs2f/2xKW+vbVzUXF+7nOtiV1KHL8ePQrx8pbzIF859h9P19eKClt1iKHswGivwKj/hdxBzl+PHjLF68mC5duqTscwXSyQouN+J79uyhTp06tG3b1qe6Myrz008/8anjDjQiIoISJUqkBBryxqJFi+jfvz8AtWvXpmrVqmzevBmA1q1bp/hCqlu3Ln/88QeVK1e+oI67776bs2fPcvz4cRITEwHjcnzWrFkpPZ7Tp0+zc+dOFi1axCOPmB88rVt0T5k6depEUcet72233cbChQvp0KEDUVFRKXMcWXGdHnK89TVwFm7xfrh790+pUmUnd9wxjUOHyjJqFKxcqUybJlRMa4cZ5lgFEcSsWGH82btCElQouZcvH7mNq2r8GljB/EhycjKXXnppysMwu7jmIE6ePEm7du0YNWoUPXr0yLTunLp+Zni6zs7IpfekSZNo2LAhAwcOpH///nz11VeoKl9++SW1cngWNa1MYTvE5AOtWs1n+fKGdO78BUuXNmbxYqFevWTGjMmHD6HSw4ZgNXPN0yQlwX/+A1dd5VYO10XPY9nQhmGtHAAuueQSoqKimD59OmCGQFetWnVBOV/deXu6ES9SpEimdWd0/datW6dEfTt//jx///13hnK0aNGCSZMmAbB582Z27tyZrYe6iDB06FCWLFnCxo0badeuHe+++67LXQ0rV64EfHOL3qJFC2bOnMnJkyc5ceIEM2bMCHmX3v6iSpVd/PzzNTz44BjAzEt06QI9epBnFtVZBRFkrFwJTZrAoEHG+V7+iHO8eufTfD+4DeVLZmwyetFE5rC7bx/qO3nyJJUqVUpJw4cPZ9KkSYwbN47Y2Fiio6NTJmE9KV26NM2bNycmJibTSWpPN+IZ1e1axJ9emXfeeYd58+ZRr149GjZsyPr16zOUo0+fPiQnJ1OvXj3uvPNOJkyYkOotPSsULlyYJ554gjfeeIPnn3+ec+fOUb9+faKjo3n++edTrpeZW/QGDRrQo0cPGjduTJMmTejZsyfx8fHZkikvEBl5hjFjHmb69M6UKmUsBT/5BOrXV779NsDC5QLiegsJdRISEtQzkH2oceIEDB1qol6dd7xk1Cy/mYm976Fx9aVZr/AVoEgVWPZHukU2bNhAnTp1sidwmNG/f38aNGjA/fffH2hRss358+c5d+4ckZGRbN26lTZt2rBp06Zc83wbLGT6v25ZD9gA87Pmjmbv3vL06DGBH364PmXfbbfBO+9ApUrZFDYIEJHlqprg7ZjtQQSY5GSYMAFq1oTXXzfKIX/EOZ699d+s+nf97CkHgN7AKx0yLWaB559/nl9//ZUOHUL7+zp58iRXX301sbGxdOrUKVfdoocUEx+FiYWzfFqFCvv473/bM3p0b0qUOAqYML516iivvWaiN4YbtgcRIFRNIPWBA82wkotG1ZfxUc9/Ub9K5mE1M6XuMxA3LN3DtgdhCUcy/1/PAu4Fsj+RsH//5TzxxFtMnnx3yr6KFZWXXhK6d4f8IWT+Y3sQQYQqfPcdXH01tGnjVg4VSu5lwsPdWfJi45xRDr8A362/+HoslnBj6kKYeu6iqihX7gCTJt3D99+3JSbGtNc9e4SePaFePeXTT80cYqhjFUQuce4cTJtmLJPat4fFTqiGIoVO8OJtL7L5zRp0v+ZT8uXLoR7dj8D0FTlTl8USTrz/X3j/bI5U1bbtXBIT45gwoTuVK+8EYONG04u48kpl5EhCOtaEVRB+Zv9+ePllqFrVhAP97Tezv1jkMQbd8jrb345iyO0vUTTSRlG3WEKRiIhkunf/lM2ba/LWW49TvvxeAHbuFB55BCpUUPr1Ay9Wx0GPVRB+4MQJmDwZbrjBWDcMGQL7HC/cpYod5vlOL7NjRDVeu2swl5XIe3EsLJZwJDLyDI8//jbbt0cxZsyDXHmlWcT0zz/CqFFQv75xtDl6NPz5Z4CF9RGrIHKIv/82Q0j33AOXXw533w3//a/bZDW+WiIfP3Q/u9+txMudh1C6eJ73PQjA/v37ueuuu6hevToNGzbkxhtvTHFHkZZmzZoBxrHc5MmTU/YvW7aMAQMGZOv6LVu2JK1xw0svvcTTTz+dal9iYmKGE5+ezv4seZtChc7y4IMfsXFjbWbO7Ej79nMQSQbM0HLfvlC+vHL99TBunBllCFZCaK49uEhOhtWrYd48+PZbWLDArID2pFSxw9x11VTuvfpTmlz5KzaYampUlU6dOtG9e3emTJkCwKpVqzhw4AA1a9ZMKZeUlET+/PlZ7EzcuBREt27dAEhISCAhwasRRrbo2rUr7du359VXX03ZN2XKFLp27Zpj17CEPxERyXTsOIuOHWexbVsUY8c+yKRJd7NrVxWSk4UffoAffjBlGzY0Iw433AAJCRAs1sm2B+EjJ04Y7T98OHToAKVLQ3w8PP44/PijWzkUizxGlyZf8PXjHdg3qjyj7u/LVTUCoBweAd64LWvntGx5YXLFOjh50vvxCRPM8UOHLjyWCfPmzaNAgQL06tUrZV9sbCwtWrRg/vz5tGjRgg4dOlC3bl3ABBcCGDx4MAsXLiQuLo63336b+fPnc/PNNwPG2Z7LxXb9+vX58ssvAejduzcJCQlER0czZMiQDOWqWbMmJUuW5Ndf3W5Npk2bRteuXRk7diyNGjUiNjaW22+/nZMnL5w78uyVHDp0iGrVqgFmIdvAgQNp1KgR9evX50NXIHFL7vLFIPgi6+sgLoYrrtjOq68+w44d1Vi0qDn9+r3LZZe546UsXw6vvGKGoC69FFq1guefh++/h6NHc1XUVNgeRBqSk2HXLti0CdauNQ7zVqyAjRuNiao3qpTZyS3x/8ctDWbRss58ChXIGQuJi6I4ULJIoKXIkLVr19KwYcN0j69YsYK1a9cSFRWVav9rr73Gm2++yTfffAPA/PnzU44NHTqUEiVKpPghcnlbHTZsGKVKleL8+fO0bt2a1atXe/V26qJr165MmTKFJk2asGTJEkqVKkWNGjUoVaoUDz74IADPPfcc48aNS/HYmhnjxo2jRIkSLF26lDNnztC8eXOuv/76C+7P4mfKXEKg3o3z5VOaN19M8+aLGTHiUX79tQmzZ9/I7Nk3snJlA8AsuJs/3yQX1apBXJxJsbFQr54xfPH3eos8ryDmzIFFi4xC2LzZOMc7fTrjcyqX3sW1tRfQsu48rq29gOqXbw2+4aMFwJbVEJeFczz/kWkpUiTj42XKZHw8GzRu3DjLD8+5c+emDFcBlCxZEjA9gDFjxpCUlMS+fftYv359hgrizjvvpFmzZrz11luphpfWrl3Lc889x9GjRzl+/Djt2rXzWbbvv/+e1atXp0Sc+/vvv/n999+tgshtJvwInIUegRUjIiKZZs1+oVmzX3jllefZt68c8+a1YuHCFixadDVr19ZLKbtjh0kzZ7rPL1AAqlc3Xhhq1jTzntmMSJsuflUQItIeeAeIAD5S1dfSHC8EfAo0BA4Dd6rqDufY08ADwHlggKp+5w8ZP/8cJk5M/3jZSw7SMGoFDastpUHUChpUW0HVMn8En0JIy0KgyGp4IdCCpE90dHSG4TldMQsulu3bt/Pmm2+ydOlSSpYsSY8ePTidyVtA5cqViYqKYsGCBXz55Zf88ssvAPTo0YOZM2cSGxvLhAkTUvVeXOTPn5/kZDMp6XkdVeXdd9/NklKx+IEJPwHnAq4g0lK+/H66dfucbt0+B+Cvv0qyeHEzVq6MJzExjsTEOLZtq55S/tw5M7LhBDykWbMQUhBOTOlRQFtgN7BURGapqufy3geAI6p6pYjcBbwO3CkidTExrKMxwTPnikhNVc2ady0fqFXLaPKoMlupWf53apbbRK3ym6hZfjO1ym+iQsm9wa8MQpTrrruOZ555hjFjxvDQQw8BsHr1av7+++8Mz8vIxXbbtm0ZNWpUSijNI0eO8M8//1C0aFFKlCjBgQMHmDNnDi19mCPp2rUrjz32GFdccQWVHG9sx44do3z58pw7d45JkyZR0UsEmWrVqrF8+XIaN26cSgG2a9eO999/n+uuu44CBQqwefNmKlasmGOK0BJelCp1hJtv/pabb3a7jf3nn+KsXl2fDRvqsHlzTX7/vQabN9dky5YrqVlTgex5C04Pf/YgGgNbVHUbgIhMAToCngqiI/Cik/8CeE+Mz+WOwBRVPQNsd2JWN8Y4kMhRHnkEBt75FQWX94DzIbzkMQQREWbMmMGjjz7K66+/TmRkJNWqVWPEiBEp8Ze9Ub9+fSIiIoiNjaVHjx6p3FU/99xz9O3bl5iYGCIiIhgyZAi33XYb8fHx1K5dm8qVK9O8eXOf5OvSpQsDBgzg3XffTdk3dOhQmjRpQtmyZWnSpIlXRfXkk09yxx13MGbMGG666aaU/T179mTHjh00aNAAVaVs2bLM9BwzsOQioemD7pJLjnH11f/j6qv/l2p/UlIE+fIdz/Hr+c1Zn4h0Btqrak9n+16giar28yiz1imz29neCjTBKI0lqvqZs38cMEdVv0hzjYeAhwCqVKnS8I8/0ndtnSF/LoS510D+Ytk7Pxh58TgUrwG/el9TANZZnyU8ydzddwNgJcwPo/ZOMvA32Xnnz8hZX0hPUqvqGGAMGG+u2a7oshbQLTTfKNJldMtAS2CxBCmXANcC8wMsR/DjTwWxB/CMwl7J2eetzG4RyQ+UwExW+3KuJSNmzw60BBZLcGLbhs/40xh4KVBDRKJEpCBm0nlWmjKzgO5OvjPwk5oxr1nAXSJSSESigBrAb36UNfwoUsSkTAiXeCAWC/j4f/axbVj82INQ1SQR6Qd8hzFz/VhV14nIy8AyVZ0FjAMmOpPQf2GUCE65aZgJ7SSgrz8smMIa1wroPn3SLRIZGcnhw4cpXbp0SjxmiyVUUVUOHz5MZGRkxgV9aBsWg40oF664zDgzWLx27tw5du/enemaAIslVIiMjKRSpUoUKFAg/UI+tI28RNhOUlsujgIFCthVvBaLJV2ssz6LxWKxeMUqCIvFYrF4xSoIi8VisXglbCapReQgkM2l1ACUAQ7lkDiBJFzuA+y9BCv2XoKT7N5LVVUt6+1A2CiIi0VElqU3kx9KhMt9gL2XYMXeS3Dij3uxQ0wWi8Vi8YpVEBaLxWLxilUQbsYEWoAcIlzuA+y9BCv2XoKTHL8XOwdhsVgsFq/YHoTFYrFYvGIVhMVisVi8YhWEg4gMFZHVIpIoIt+LSIVAy5RdROQNEdno3M8MEbk00DJlFxHpIiLrRCRZRELSHFFE2ovIJhHZIiKDAy1PdhGRj0XkTycSZMgiIpVFZJ6IrHf+W48EWqbsIiKRIvKbiKxy7uWlHK3fzkEYROQSVf3HyQ8A6qpqrwCLlS1E5HpMbI0kEXkdQFUHBVisbCEidTDxFD8EnlTVkHLZKyIRwGagLbAbEyelq6quz/DEIERErgGOA5+qakyg5ckuIlIeKK+qK0SkOLAcuDVEfxMBiqrqcREpACwCHlHVJTlRv+1BOLiUg0NRQjWqOaCq36tqkrO5BBORLyRR1Q2quinQclwEjYEtqrpNVc8CU4COAZYpW6jqz5i4LSGNqu5T1RVO/hiwAagYWKmyhxqOO5sFnJRjzy6rIDwQkWEisgu4G3gh0PLkEP8C5gRaiDxMRWCXx/ZuQvRhFI6ISDUgHvg1wKJkGxGJEJFE4E/gB1XNsXvJUwpCROaKyFovqSOAqj6rqpWBSUC/wEqbMZndi1PmWUxEvkmBkzRzfLkXiyWnEZFiwJfAo2lGEEIKVT2vqnGYkYLGIpJjw395KmCQqrbxsegkYDYwxI/iXBSZ3YuI9ABuBlprkE80ZeF3CUX2AJU9tis5+ywBxBmv/xKYpKpfBVqenEBVj4rIPKA9kCOGBHmqB5ERIlLDY7MjsDFQslwsItIeeArooKonAy1PHmcpUENEokSkICbu+qwAy5SncSZ2xwEbVHV4oOW5GESkrMtKUUQKY4whcuzZZa2YHETkS6AWxmLmD6CXqobkm56IbAEKAYedXUtC2CKrE/AuUBY4CiSqaruACpVFRORGYAQQAXysqsMCK1H2EJHPgZYYt9IHgCGqOi6gQmUDEbkaWAiswbR3gGdUdXbgpMoeIlIf+ATz38oHTFPVl3OsfqsgLBaLxeINO8RksVgsFq9YBWGxWCwWr1gFYbFYLBavWAVhsVgsFq9YBWGxWCwWr1gFYbFkgIh0cjz8eqZkEbkh0LJZLP7GmrlaLFlARB7C+OpqparJmZW/iOtEqOp5f9VvsfiC7UFYLD4iIjUxThzvTascRGSAE19gtYhMcfYVE5HxIrLG2X+7s7+rs2+tyx27s/+4iLwlIquApiJyj+PrP1FEPnRch1ssuYZVEBaLDzi+eyYDT6jqTi9FBgPxqlofcK1afx74W1XrOft/cgJRvQ5cB8QBjUTkVqd8UeBXVY3FrIK/E2juOGI7j+m5WCy5hlUQFotvDAXWqerUdI6vBiaJyD0YD7oAbYBRrgKqegRoBMxX1YNOzI5JwDVOkfMYB3IArYGGwFLHlXNr4Iqcux2LJXPylDdXiyU7iEhL4Haggce+8Zg4AntV9UbgJsyD/hbgWRGpl41LnfaYdxDgE1V9+iJEt1guCjtJbbFkgIiUBFYA3VT1l3TK5AOqqOoOZyjqD6AuZtgpUlUf9agrEhPlryFwBPgOeFdVvxaR46pazClbF/gaM8T0p4iUAoqr6h9+vF2LJRV2iMliyZhewGXA+2lMXe/0KBMBfCYia4CVwEhVPQq8ApR0JqNXYSyf9mEUxzxgFbBcVb9Oe1EnPvJzwPcishr4ASjvv9u0WC7E9iAsFovF4hXbg7BYLBaLV6yCsFgsFotXrIKwWCwWi1esgrBYLBaLV6yCsFgsFotXrIKwWCwWi1esgrBYLBaLV/4fQ3C9XdrmLNUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualizing the rejection region\n",
    "\n",
    "z = np.arange(-3, 3, 0.01)\n",
    "plt.plot(z, norm.pdf(z), label='Standard Normal Distribution', color='blue', linewidth=2.5)\n",
    "plt.fill_between(z[z > z_score], norm.pdf(z[z > z_score]), label='Right Rejection Region', color='yellow')\n",
    "plt.fill_between(z[z < -z_score], norm.pdf(z[z < -z_score]), label='Left Rejection Region', color='orange')\n",
    "plt.axvline(x=z_score, color='red', linestyle='--', label='Critical Value')\n",
    "plt.axvline(x=-z_score, color='red', linestyle='--')\n",
    "plt.title(\"Two Sample Z-test Rejection Region\")\n",
    "plt.xlabel('Z-score')\n",
    "plt.ylabel('Density')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c01e866-c810-4c62-b942-c50b3d898651",
   "metadata": {},
   "source": [
    "The above plot shows rejection region with a standard normal distribution where the rejection region is shaded in yelow and orange,and is separated from the acceptance region  by the critical value. This plot also shows PDF of the standard normal distribution, which represents the distribution of the test statistic under the null hypothesis.\n",
    "\n",
    "After analyzing the conversion rate results and rejecting the null hypothesis, we will now proceed to examine the revenue or the average amount spent per user to determine the statistical significance of launching the new feature as part of our A/B testing objectives."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cb65f16-4d39-44cd-a701-53309f4251af",
   "metadata": {},
   "source": [
    "## Revenue "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "4b9eb740-9c92-48bd-ae96-d7157d3ac7fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "group\n",
      "A    3.374518\n",
      "B    3.390867\n",
      "Name: amount_spent, dtype: float64\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAGACAYAAAC9TFsZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAAsTAAALEwEAmpwYAAAccElEQVR4nO3debgddZ3n8feHJAq4a66CbFHElVaBNOBoP40yNqgIrY2KG+CWbpVRetoZl1FEbafbHpUexRGjoAguMLhFBxccUUQFDRhRQGci0AIihDUsbsC3/6i6erjc5J4gdS+/m/freepJVf1+p+p7zr25n1NVv1MnVYUkSWrPJnNdgCRJumMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLE1awk30xyTZK7z3UtG4v+NX/5XNcxKsmSJJVk4VzXIs02Q1xNSrIE+AuggH0H2L6BoGnN1e+Gv5OajiGuVh0InAF8DDgIIMndk1ybZMfJTkkmkvw6yQP75X2SrOr7fTfJY0f6XpTk9UnOAW5MsjDJG5L8PMn1Sc5L8qyR/guSvCfJlUkuTHLI6BFhkvskOTrJZUkuTfKPSRZM92SS7Jrke31dlyU5MsndRtoryauS/P++lnck2b5/DmuTnDil/yuSrE5ydZIVSR7cr7/dUevo0XWSg5OcnuTd/VmOC5M8rW97J90bpyOT3JDkyGmex6ZJjk9yVf9cfpDkQSP7+ack3+9r/kKS+488dvf++Vyb5EdJ9phS4zuSfKd//l9LsrhvPq3/99q+ridMU9fhSU5KckL/+LOTPG6k/cFJPpNkTf+cXzPNY49PshY4eJrt3+YMxeTr2M8nyRFJruif948nf0f739l3J/lFksuTHJVks75tjySX9L+TvwI+OnW/ElXl5NTcBKwGXgXsAvweeFC//hjgnSP9Xg18pZ/fCbgC2A1YQBf+FwF379svAlYB2wCb9eueAzyY7g3v84AbgS37tr8DzgO2Bu4HfJ3uzMDCvv1zwIeAewAPBL4P/O06ns8uwO7AQmAJcD5w6Eh7AV8A7g08Bvgt8H+BhwL36es4qO/7FOBKYGfg7sD7gdP6tiWjNfbrvgm8vJ8/uH89X9G/Rq8Efglkat91PI+/Bb4IbN4/fhfg3iOPvRTYsX9NPgMc37dtBVwFPL1/rZ/aL0+MPPbnwMOBzfrlf17Xc5qmrsP757U/sAh4HXBhP78JcBZwGHC3/jW9ANhrymP/uu+72TTbv83r0r+Op/fze/Xbvy8Q4FH88XfoCGAFcH/gXv1r90992x7AzcC7+p/j7fbr5DTnBTg5begEPKn/o7q4X/4p8Pf9/H8Efj7S9zvAgf38B4F3TNnWz4C/7OcvAl46w75XAfv1899gJJT7fRddED+ILmg3G2l/PnDqmM/xUOBzI8sFPHFk+Szg9SPL7wH+tZ8/GviXkbZ79q/XkukCj9uH+OqRts37/ltM7buOul8KfBd47DRt36QP3n750cDv6ML+9cBxU/p/lT++Mfkm8OaRtlfxxzdnt3tO0+z7cOCMkeVNgMvozizsBvxiSv83Ah8deexpM/y8bvO6cNsQfwrw/+jepG0y0id0bwq3H1n3BODCfn6P/vXZdK7+rznd9SevsahFBwFfq6or++VP9uuOAE4FNk+yG3A58Hi6I2KA7YCDkvynkW3dje5Ie9LFoztKciDwn+mCArpAnDyN++Ap/Ufnt6M7yrssyeS6TaZuf2Q/DwfeCyylC86FdEE96vKR+V9Ps7zFSF1nTzZU1Q1JrqI72r10uv1P8auRx97U13/PMR4HcBzdmYxPJ7kvcDzw36rq93376PP/N7rXaDHd6/WcJM8caV9E9/O8XV3ATRtQ06Q/7Luqbk1yCd1rVcCDk1w70ncB8O3pHruhquob/aWHDwDbJfks3ZmATel+1meN/I6k3/ekNVX1mzu6b81/hria0l8vfC6woL9OCN2pxvsmeVxV/SjJiXRHvZcDX6qq6/t+F9Odan/nenbxh6/1S7Id8GFgT+B7VXVLklV0f2ihO5LbeuSx24zMX0x3JL64qm4e46l9EPgh8Pyquj7JoXSnfu+IX9KF4uTzuAfwALoAv7FfvTmwtp/fgvGt92sP+7B+G/C2dIMPT6Y723F032X0NdqW7gzBlXSv13FV9YoNqGWsmkb8Yd9JNqH72f2S7pT1hVW1w5+wjxvpXtNJt3lNq+p9wPvSjc04EfgvwFvp3nw9pqrW9ebKr5nUejmwTa35a+AWulOxj++nR9EdNR3Y9/kk3fXrF/bzkz4M/F2S3frBRvdI8owk91rHvu5B90d0DUCSl9Bdz510IvDaJFv1R52vn2yoqsuArwHvSXLvJJukG4j2l+vY173oQvWGJI+kuxZ9R30KeEmSx6f7+N1/B86sqouqag1dmL8o3cC8lwLbb8C2L6e7ZjytJE9O8mfpBvCtpQvpW0e6vCjJo5NsDrwdOKmqbqE7Yn9mkr36ujbtB3Ztffu93M6afh/rrKu3S5JnpxvUdyjdm6wz6MYqXN8PINus3/+OSf58jH1PWgU8O8nmSR4GvGyyIcmf979zi+jC/jfArVV1K93v5BH548DLrZLstQH71UbOEFdrDqK7VvmLqvrV5AQcCbwwycKqOpPuj+WDgS9PPrCqVtIN2DoSuIZucNzB69pRVZ1Hd635e3Th9Wd019gnfZguqM+hO4o+me6o7pa+/UC60/Xn9fs7CdhyHbt7HfAC4Pp+uyeM8Vqsq+6vA2+hGzh2GV1IHzDS5RV0R4JX0Q2S++4GbP5/AvunG7n+vmnat6B7nmvpBud9i+4U+6Tj6D5R8Cu608mv6Wu+GNgPeBNdKF/c1zjj36iqugl4J/CdfmT77uvo+gW6N3fXAC8Gnl1Vv+/fROxD94bwQrozAx+hGzA4riPorl9fDhwLfGKk7d50P9Nr6C4hXAX8j77t9XS/h2f0I9+/DjxiA/arjdzkiFNJf6J0H8U6qqq2m7HzRijJN+lGo39kDvZ9OPCwqnrRbO9bGpJH4tId1J96fXq6z5NvRXeN83MzPU6S7iyGuHTHhW4Q1zV0p9PPp/ussSTNCk+nS5LUKI/EJUlqlCEuSVKjmrvZy+LFi2vJkiVzXYYkSbPmrLPOurKqJqauby7ElyxZwsqVK+e6DEmSZk2Sf5tu/WCn0/s7Ln0/3VcKnpvkbdP0Obj/6r9V/fTy6bYlSZJub8gj8d8CT+m/fGERcHqSL1fVGVP6nVBVhwxYhyRJ89JgIV7dZ9du6BcX9ZOfZ5Mk6U4y6Oj0/osEVgFXAKf097Se6m+SnJPkpCTbTNNOkmVJViZZuWbNmiFLliSpGYOGeFXdUlWPp/vKv12T7DilyxeBJVX1WOAUui8OmG47y6tqaVUtnZi43eA8SZI2SrPyOfGquhY4Fdh7yvqrquq3/eJHgF1mox5JkuaDIUenT/TfsUySzYCnAj+d0mf0axn3pbv3tCRJGsOQo9O3BI5NsoDuzcKJVfWlJG8HVlbVCuA1Sfal+w7mq1nPdztLkqTbau4LUJYuXVre7EWStDFJclZVLZ263nunS5LUKENckqRGGeKSJDXKEJckqVHNfYuZJA1hyRv+z1yXoD/BRf/8jLkuYU54JC5JUqM8Er+L8WigbRvr0YCkueGRuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktQoQ1ySpEYZ4pIkNcoQlySpUYa4JEmNMsQlSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRg0W4kk2TfL9JD9Kcm6St03T5+5JTkiyOsmZSZYMVY8kSfPNkEfivwWeUlWPAx4P7J1k9yl9XgZcU1UPA44A3jVgPZIkzSuDhXh1bugXF/VTTem2H3BsP38SsGeSDFWTJEnzyaDXxJMsSLIKuAI4parOnNJlK+BigKq6GbgOeMA021mWZGWSlWvWrBmyZEmSmjFoiFfVLVX1eGBrYNckO97B7SyvqqVVtXRiYuJOrVGSpFbNyuj0qroWOBXYe0rTpcA2AEkWAvcBrpqNmiRJat2Qo9Mnkty3n98MeCrw0yndVgAH9fP7A9+oqqnXzSVJ0jQWDrjtLYFjkyyge7NwYlV9KcnbgZVVtQI4GjguyWrgauCAAeuRJGleGSzEq+ocYKdp1h82Mv8b4DlD1SBJ0nzmHdskSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktQoQ1ySpEYZ4pIkNcoQlySpUYa4JEmNMsQlSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1KjBQjzJNklOTXJeknOTvHaaPnskuS7Jqn46bKh6JEmabxYOuO2bgX+oqrOT3As4K8kpVXXelH7frqp9BqxDkqR5abAj8aq6rKrO7uevB84Hthpqf5IkbWxm5Zp4kiXATsCZ0zQ/IcmPknw5yWNmox5JkuaDIU+nA5DknsBngEOrau2U5rOB7arqhiRPBz4P7DDNNpYBywC23XbbYQuWJKkRgx6JJ1lEF+CfqKrPTm2vqrVVdUM/fzKwKMniafotr6qlVbV0YmJiyJIlSWrGkKPTAxwNnF9V711Hny36fiTZta/nqqFqkiRpPhnydPoTgRcDP06yql/3JmBbgKo6CtgfeGWSm4FfAwdUVQ1YkyRJ88ZgIV5VpwOZoc+RwJFD1SBJ0nzmHdskSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktQoQ1ySpEYZ4pIkNcoQlySpUYa4JEmNMsQlSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRg0W4km2SXJqkvOSnJvktdP0SZL3JVmd5JwkOw9VjyRJ883CAbd9M/APVXV2knsBZyU5parOG+nzNGCHftoN+GD/ryRJmsFgR+JVdVlVnd3PXw+cD2w1pdt+wMercwZw3yRbDlWTJEnzyaxcE0+yBNgJOHNK01bAxSPLl3D7oJckSdMYPMST3BP4DHBoVa29g9tYlmRlkpVr1qy5cwuUJKlRg4Z4kkV0Af6JqvrsNF0uBbYZWd66X3cbVbW8qpZW1dKJiYlhipUkqTFjhXiSJyV5ST8/keQhYzwmwNHA+VX13nV0WwEc2I9S3x24rqouG7N2SZI2ajOOTk/yVmAp8Ajgo8Ai4HjgiTM89InAi4EfJ1nVr3sTsC1AVR0FnAw8HVgN3AS8ZIOfgSRJG6lxPmL2LLpBaZMjzX/Zf2RsvarqdCAz9Cng1WPUIEmSphjndPrv+rAtgCT3GLYkSZI0jnFC/MQkH6L7DPcrgK8DHx62LEmSNJMZT6dX1buTPBVYS3dd/LCqOmXwyiRJ0nqNddvVPrQNbkmS7kLGGZ1+Pf318BHXASvp7o1+wRCFSZKk9RvnSPxf6W6H+km60eYHANvTjVY/BthjoNokSdJ6jDOwbd+q+lBVXV9Va6tqObBXVZ0A3G/g+iRJ0jqME+I3JXlukk366bnAb/q2qafZJUnSLBknxF9Id+e1K4DL+/kXJdkMOGTA2iRJ0nqM8xGzC4BnrqP59Du3HEmSNK5xRqdvCrwMeAyw6eT6qnrpgHVJkqQZjHM6/ThgC2Av4Ft0Xxd6/ZBFSZKkmY0T4g+rqrcAN1bVscAzgN2GLUuSJM1knBD/ff/vtUl2BO4DPHC4kiRJ0jjGudnL8iT3A94MrADuCbxl0KokSdKM1hviSTYB1lbVNcBpwENnpSpJkjSj9Z5Or6pbgf86S7VIkqQNMM418a8neV2SbZLcf3IavDJJkrRe41wTf17/76tH1hWeWpckaU6Nc8e2h8xGIZIkacPMeDo9yeZJ3pxkeb+8Q5J9hi9NkiStzzjXxD8K/A74D/3ypcA/DlaRJEkayzghvn1V/Qv9TV+q6iYgg1YlSZJmNE6I/67/2tECSLI98NtBq5IkSTMaZ3T64cBXgG2SfAJ4InDwgDVJkqQxjDM6/WtJzgJ2pzuN/tqqunLwyiRJ0nqN833iXwQ+CayoqhuHL0mSJI1jnGvi7wb+AjgvyUlJ9k+y6cB1SZKkGYxzOv1bwLeSLACeArwCOAa498C1SZKk9RhnYBv96PRn0t2CdWfgYwPWJEmSxjDOHdtOBM6nOwo/EjgIWDBwXZIkaQbjXBM/GngucD3d3dveRhfqkiRpDq3zdHqShwPP76crgROAVNWTZ6k2SZK0Huu7Jv5T4NvAPlW1GiDJ389KVZIkaUbrO53+bOAy4NQkH06yJ94zXZKku4x1hnhVfb6qDgAeCZwKHAo8MMkHk/zVLNUnSZLWYcaBbVV1Y1V9sqqeCWwN/BB4/eCVSZKk9RpndPofVNU1VbW8qvacqW+SY5JckeQn62jfI8l1SVb102EbUoskSRu7sW72cgd9jO5z5R9fT59vV9U+A9YgSdK8tUFH4huiqk4Drh5q+5IkbewGC/ExPSHJj5J8Oclj5rgWSZKaMuTp9JmcDWxXVTckeTrweWCH6TomWQYsA9h2221nrUBJku7K5uxIvKrWVtUN/fzJwKIki9fRd3lVLa2qpRMTE7NapyRJd1VzFuJJtkiSfn7Xvpar5qoeSZJaM9jp9CSfAvYAFie5BHgrsAigqo4C9gdemeRm4NfAAVVVQ9UjSdJ8M1iIV9XzZ2g/ku4jaJIk6Q6Y69HpkiTpDjLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktQoQ1ySpEYZ4pIkNcoQlySpUYa4JEmNMsQlSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktSowUI8yTFJrkjyk3W0J8n7kqxOck6SnYeqRZKk+WjII/GPAXuvp/1pwA79tAz44IC1SJI07wwW4lV1GnD1errsB3y8OmcA902y5VD1SJI038zlNfGtgItHli/p191OkmVJViZZuWbNmlkpTpKku7omBrZV1fKqWlpVSycmJua6HEmS7hLmMsQvBbYZWd66XydJksYwlyG+AjiwH6W+O3BdVV02h/VIktSUhUNtOMmngD2AxUkuAd4KLAKoqqOAk4GnA6uBm4CXDFWLJEnz0WAhXlXPn6G9gFcPtX9Jkua7Jga2SZKk2zPEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktQoQ1ySpEYZ4pIkNcoQlySpUYa4JEmNMsQlSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktSoQUM8yd5JfpZkdZI3TNN+cJI1SVb108uHrEeSpPlk4VAbTrIA+ADwVOAS4AdJVlTVeVO6nlBVhwxVhyRJ89WQR+K7Aqur6oKq+h3waWC/AfcnSdJGZcgQ3wq4eGT5kn7dVH+T5JwkJyXZZroNJVmWZGWSlWvWrBmiVkmSmjPXA9u+CCypqscCpwDHTtepqpZX1dKqWjoxMTGrBUqSdFc1ZIhfCoweWW/dr/uDqrqqqn7bL34E2GXAeiRJmleGDPEfADskeUiSuwEHACtGOyTZcmRxX+D8AeuRJGleGWx0elXdnOQQ4KvAAuCYqjo3yduBlVW1AnhNkn2Bm4GrgYOHqkeSpPlmsBAHqKqTgZOnrDtsZP6NwBuHrEGSpPlqrge2SZKkO8gQlySpUYa4JEmNMsQlSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlqlCEuSVKjDHFJkhpliEuS1ChDXJKkRhnikiQ1yhCXJKlRhrgkSY0yxCVJapQhLklSowxxSZIaZYhLktQoQ1ySpEYZ4pIkNcoQlySpUYa4JEmNMsQlSWqUIS5JUqMMcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjTLEJUlq1KAhnmTvJD9LsjrJG6Zpv3uSE/r2M5MsGbIeSZLmk8FCPMkC4APA04BHA89P8ugp3V4GXFNVDwOOAN41VD2SJM03Qx6J7wqsrqoLqup3wKeB/ab02Q84tp8/CdgzSQasSZKkeWPIEN8KuHhk+ZJ+3bR9qupm4DrgAQPWJEnSvLFwrgsYR5JlwLJ+8YYkP5vLevQnWQxcOddFDCVeENJdl//32rbddCuHDPFLgW1Glrfu103X55IkC4H7AFdN3VBVLQeWD1SnZlGSlVW1dK7rkDY2/t+bn4Y8nf4DYIckD0lyN+AAYMWUPiuAg/r5/YFvVFUNWJMkSfPGYEfiVXVzkkOArwILgGOq6twkbwdWVtUK4GjguCSrgavpgl6SJI0hHvhqNiVZ1l8ekTSL/L83PxnikiQ1ytuuSpLUKENckqRGGeKaVUmelOQDc12HJM0HTdzsRW1LshPwAuA5wIXAZ+e2Imnjk2QxcJUf451fPBLXIJI8PMlbk/wUeD/wC7qBlE+uqvfPcXnSvJZk9yTfTPLZJDsl+QnwE+DyJHvPdX268zg6XYNIcivwbeBlVbW6X3dBVT10biuT5r8kK4E30d0FcznwtKo6I8kjgU9V1U5zWqDuNB6JayjPBi4DTk3y4SR7An5DnTQ7FlbV16rqfwO/qqozAKrqp3Ncl+5khrgGUVWfr6oDgEcCpwKHAg9M8sEkfzWnxUnz360j87+e0ubp13nE0+maNUnuRze47XlVtedc1yPNV0luAW6kO/u1GXDTZBOwaVUtmqvadOcyxCVJapSn0yVJapQhLklSowxxaSOU5EFJPpnkgiRnJflekmfNdV2SNowhLm1kkgT4PHBaVT20qnYBDgC2ntLPOzpKd3EObJM2Mv1n9g+rqr+cpu1gus/43xNYADwLOAZ4KN0I52VVdU6Sw4Ebqurd/eN+AuzTb+YrwFnAzsC5wIFVdROS7nQeiUsbn8cAZ6+nfWdg/z7k3wb8sKoeS3cHsI+Psf1HAP+rqh4FrAVe9SfWK2kdDHFpI5fkA0l+lOQH/apTqurqfv5JwHEAVfUN4AFJ7j3DJi+uqu/088f325A0AENc2vicS3e0DUBVvRrYE5joV904xjZu5rZ/PzYdmZ96jc5rdtJADHFp4/MNYNMkrxxZt/k6+n4beCFAkj2AK6tqLXAR/RuBJDsDDxl5zLZJntDPvwA4/c4qXNJtObBN2ggl2RI4AtgNWEN39H0U3S06l1bVIX2/+zP9wLbNgC8AWwFnAk8AntZv/ivASmAX4DzgxQ5sk4ZhiEu60yRZAnypqnac61qkjYGn0yVJapRH4pIkNcojcUmSGmWIS5LUKENckqRGGeKSJDXKEJckqVGGuCRJjfp3KQ5+JbJ4fyAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# calculating the average amount spent per user in eahc group. In this calculation I used the origal data since it is an average.\n",
    "# In the proportion since we want to analyze which users actually converte, it was necessary to delete the duplicates.\n",
    "amount_spent_user = ab.groupby(['group','uid'])['amount_spent'].sum().groupby('group').mean()\n",
    "print(amount_spent_user)\n",
    "\n",
    "plt.figure(figsize=(8,6))\n",
    "amount_spent_user.plot(kind='bar')\n",
    "plt.title('Average amount spent per user')\n",
    "plt.xlabel('Group')\n",
    "plt.ylabel('Average')\n",
    "plt.savefig(\"average amount spent per user.png\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2968b2c-a9df-46a4-8fb8-685d512aea78",
   "metadata": {},
   "source": [
    "We can observe that the average amount spent per user in group b is slightly higher than group A with 3.391 and 3.375 respectivately. However, we cannot make any conslusions about whether this difference is statistically significant or not yet. So the next step is conduct a hypothesis test and calculate the confidence interval.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "722b92c4-6738-45c8-ad75-2981bbd4baa3",
   "metadata": {},
   "source": [
    "### Hypothesis\n",
    "\n",
    "- Ho : &mu;con = &mu;exp\n",
    "- Ha : &mu;con &ne; &mu;exp\n",
    "\n",
    "- Ho: &mu;con - &mu;exp = 0\n",
    "- Ha : &mu;con - &mu;exp &ne; 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cfe84f9-8237-4360-bb81-b7cb7206b159",
   "metadata": {},
   "source": [
    "Let's calculate the confidence intervals for group A or control group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "46d0b980-7431-428c-a2fa-8ed4f74fa4a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample mean; 3.37\n",
      "Sample standard deviation : 25.37\n",
      "Standard of error: 0.16\n",
      "t-critical value: 1.96\n",
      "Margin of error: 0.32\n",
      "95% Confidence Interval: [3.048 3.685]\n"
     ]
    }
   ],
   "source": [
    "# Importing the t stat since we are going to use t distribution for mean\n",
    "from scipy.stats import t\n",
    "\n",
    "#Subsetting the data to incluide only the control group\n",
    "control = ab[ab['group']=='A']\n",
    "\n",
    "#Calculating some statistics over control group\n",
    "\n",
    "NC= control.shape[0]\n",
    "sample_mean = control['amount_spent'].mean()\n",
    "sample_std = control['amount_spent'].std()\n",
    "\n",
    "#Calculating the standard error of the mean\n",
    "SOE = sample_std / np.sqrt(NC)\n",
    "\n",
    "#Claculating the t-critical value, in this case 95% confidence level\n",
    "t_critical = t.ppf(0.975, NC-1)\n",
    "\n",
    "#Margin of error\n",
    "MOE = t_critical * SOE\n",
    "\n",
    "#Confidence interval\n",
    "\n",
    "CIC = (sample_mean - MOE, sample_mean + MOE)\n",
    "print(f\"Sample mean; {sample_mean:.2f}\")\n",
    "print(f\"Sample standard deviation : {sample_std:.2f}\")\n",
    "print(f\"Standard of error: {SOE:.2f}\")\n",
    "print(f\"t-critical value: {t_critical:.2f}\")\n",
    "print(f\"Margin of error: {MOE:.2f}\")\n",
    "print(f\"95% Confidence Interval: {np.round(CIC, 3)}\")\n",
    "      "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52213856-2036-4289-9652-422cefa1e5a7",
   "metadata": {},
   "source": [
    "We can observe that the sample mean for group A or control group is 3.37, which is slightly lower than the sample mean for group B. With a confidence interval of [3.048 3.685] we can be reasonably confidence that the true population mean falls within this range. However, we need to calculate the confidence interval for group b to compare the two groups and draw any conclusion. We are going to do that next.\n",
    "\n",
    "Note: Note that our sample mean does not fall within our confidence interval, which ww have to consider when drawing conclusions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "041e49ba-76c5-4d5f-903a-51e4a3bf8f85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample mean: 3.380\n",
      "Sample standard deviation : 24.79\n",
      "Standard of error: 0.16\n",
      "t-critical value: 1.96\n",
      "Margin of error: 0.31\n",
      "95% Confidence Interval: [3.071 3.689]\n"
     ]
    }
   ],
   "source": [
    "#Subsetting the data to incluide only the control group\n",
    "treatment = ab[ab['group']=='B']\n",
    "\n",
    "#Calculating some statistics over control group\n",
    "\n",
    "NT= treatment.shape[0]\n",
    "sample_mean_t = treatment['amount_spent'].mean()\n",
    "sample_std_t = treatment['amount_spent'].std()\n",
    "\n",
    "#Calculating the standard error of the mean\n",
    "SET = sample_std_t / np.sqrt(NT)\n",
    "\n",
    "#Claculating the t-critical value, in this case 95% confidence level\n",
    "t_critical_t = t.ppf(0.975, NT-1)\n",
    "\n",
    "#Margin of error\n",
    "MET = t_critical * SET\n",
    "\n",
    "#Confidence interval\n",
    "\n",
    "CIT = (sample_mean_t - MET, sample_mean_t + MET)\n",
    "\n",
    "print(f\"Sample mean: {sample_mean_t:.3f}\")\n",
    "print(f\"Sample standard deviation : {sample_std_t:.2f}\")\n",
    "print(f\"Standard of error: {SET:.2f}\")\n",
    "print(f\"t-critical value: {t_critical_t:.2f}\")\n",
    "print(f\"Margin of error: {MET:.2f}\")\n",
    "print(f\"95% Confidence Interval: {np.round(CIT, 3)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18857398-ada5-48cc-a505-f41a9d99bd6e",
   "metadata": {},
   "source": [
    "Here we can say with 95% confidence that the true population mean of the average amount spent per user falls within the interval of [3.071 3.689]. Notice that our sample mean falls within this range in the treatmen group. However, we are going to calculate the confidence interval and p-value of the difference of the mean to determinate whether there is a statistical significance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "60b0998d-801f-475e-a5c4-3abd8edc7f73",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Difference in mean: 0.014\n",
      "95% Confidence Interval: (-0.43, 0.457)\n",
      "p-value:   0.952\n"
     ]
    }
   ],
   "source": [
    "# Calculating the confidence interval for the difference of mean using unequal variance and t-distribution\n",
    "\n",
    "diff = sample_mean_t - sample_mean\n",
    "\n",
    "#standard error of the difference\n",
    "se_diff = np.sqrt((sample_std**2/len(control)) + (sample_std_t**2/len(treatment)))\n",
    "\n",
    "#t-statistics and p-value\n",
    "t_stat, p_value = ttest_ind(treatment['amount_spent'], control['amount_spent'], equal_var =False)\n",
    "\n",
    "#Confidence interal for the difference\n",
    "ci_low,ci_high = diff - 1.96 * se_diff,diff + 1.96 * se_diff\n",
    "\n",
    "print(f\"Difference in mean: {diff:.3f}\")\n",
    "print(F\"95% Confidence Interval: {np.round(ci_low,3), np.round(ci_high, 3)}\")\n",
    "print(f\"p-value:  {p_value: .3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91e71482-9eea-4f87-89de-d81df977a09e",
   "metadata": {},
   "source": [
    "We can observe that the difference in means is very small (0.014) and the confidence interval for this difference contains 0, which mean we fail to reject the null hypothesis. This imply hat the average amount spent per user is the same in both groups. Aditionally, the p-value 0.952 is greather than our significance level 0.05, this supports the conclusion that there is no statistically significant difference in the average amount spent per user. Therefore, it we consider only this metric, the new feature did not have a significant impact on the amount spend per user."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dc63b20-a57e-4693-974d-a66961a37b28",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "- Based on the results of the A/B test, we can conclude that the new feature had a significant impact on the conversion rate, with the treatment group 4.63% conversion rate outperforming the control group 3.92%. However, when considering the average amount spent per user, there was no significant difference between the two groups.\n",
    "\n",
    "- Given the targeted objective of a 2% change in conversion rate and revenue, it seems that the new feature has met this benchmark for conversion rate. However, the impact on revenue is unclear given the lack of difference in the average amount spent per user.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de254785-831e-4028-a62f-ba4450638ed2",
   "metadata": {},
   "source": [
    "## Recommendation\n",
    "\n",
    "We recommend further analysis to understand the impact of the new feature on revenue, such as examining customer behavior and engagement with the new feature. Based on this additional analysis, stakeholders can make an informed decision about whether to launch the experience to all users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47656f40-257d-4c62-911b-36c1d490cc4c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}