code/FoodHygieneProject.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Analysing food hygiene data\n",
        "\n",
        "This notebook contains the code for analysing food hygiene data to establish the scale and nature of uninspected establishments.\n",
        "\n",
        "We need to:\n",
        "\n",
        "* Compile: fetch data from the FSA\n",
        "* Clean: convert these from XML files to dataframes\n",
        "* Question: find out how many haven't been inspected in 2 years or more (or other timeframes)\n",
        "* Question: find out how many haven't yet been inspected\n",
        "* Context: work this out as a percentage\n",
        "* Context: establish the makeup of those establishments (e.g. how many are rated below 3? How many just 3?)\n",
        "* Combine: repeat this for all authorities\n",
        "* Combine: fetch Google Places API data\n",
        "* Context: what's the average rating of those places not inspected?"
      ],
      "metadata": {
        "id": "GHimnbsKmNdH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Import the libraries"
      ],
      "metadata": {
        "id": "ahahvwKim6Zb"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-smCNYEdmCiP"
      },
      "outputs": [],
      "source": [
        "#import the libraries we'll need\n",
        "import requests\n",
        "import xml.etree.ElementTree as ET\n",
        "from bs4 import BeautifulSoup  # Change from ET to BeautifulSoup\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#import a library for downloading files\n",
        "from google.colab import files"
      ],
      "metadata": {
        "id": "zQNiAukaoxqm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#for using the isnan() function to check for missing values\n",
        "import math"
      ],
      "metadata": {
        "id": "AY1PC9_QWsCG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Fetch the XML file\n",
        "\n",
        "Each XML file is at a different URL on https://ratings.food.gov.uk/open-data - we try one."
      ],
      "metadata": {
        "id": "shcibyNSAFaj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Brentwood's FSA data URL\n",
        "url = \"https://ratings.food.gov.uk/api/open-data-files/FHRS111en-GB.xml\"\n",
        "\n",
        "# Fetch the URL\n",
        "response = requests.get(url)\n",
        "# Store the content\n",
        "xml_data = response.content"
      ],
      "metadata": {
        "id": "ZZLp4dO8AH9F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Parse into a 'soup' and then into a dataframe\n",
        "\n",
        "Now that we have a requests object, we need to convert that to a BeautifulSoup object to be able to parse it as a structured set of info.\n",
        "\n",
        "As we parse it we store the info in a `pandas` dataframe."
      ],
      "metadata": {
        "id": "xrEn7uSHAIUY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Parse the XML data\n",
        "soup = BeautifulSoup(xml_data, 'xml')  # Use BeautifulSoup\n",
        "\n",
        "# Create empty lists to store data\n",
        "establishments = []\n",
        "business_names = []\n",
        "address_line_1s = []\n",
        "address_line_2s = []\n",
        "address_line_3s = []\n",
        "address_line_4s = []\n",
        "post_codes = []\n",
        "rating_values = []\n",
        "rating_dates = []\n",
        "business_types = []\n",
        "las = []\n",
        "nrps = []\n",
        "lats = []\n",
        "lngs = []\n",
        "\n",
        "# Find all establishment details\n",
        "establishments_data = soup.find_all('EstablishmentDetail')\n",
        "\n",
        "# Extract data for each establishment\n",
        "for establishment in establishments_data:\n",
        "  business_names.append(establishment.find('BusinessName').text.strip() or \"\")\n",
        "  las.append(establishment.find('LocalAuthorityName').text.strip() or \"\")\n",
        "  nrps.append(establishment.find('NewRatingPending').text.strip() or \"\")\n",
        "  #if it is there\n",
        "  if establishment.find('AddressLine1') != None:\n",
        "    address_line_1s.append(establishment.find('AddressLine1').text.strip() or \"\")\n",
        "  else:\n",
        "    address_line_1s.append('')\n",
        "  if establishment.find('AddressLine2') != None:\n",
        "    address_line_2s.append(establishment.find('AddressLine2').text.strip() or \"\")\n",
        "  else:\n",
        "    address_line_2s.append('')\n",
        "  if establishment.find('AddressLine3') != None:\n",
        "    address_line_3s.append(establishment.find('AddressLine3').text.strip() or \"\")\n",
        "  else:\n",
        "    address_line_3s.append('')\n",
        "  if establishment.find('AddressLine4') != None:\n",
        "    address_line_4s.append(establishment.find('AddressLine4').text.strip() or \"\")\n",
        "  else:\n",
        "    address_line_4s.append('')\n",
        "  if establishment.find('PostCode') != None:\n",
        "    post_codes.append(establishment.find('PostCode').text.strip() or \"\")\n",
        "  else:\n",
        "    post_codes.append('')\n",
        "  rating_values.append(establishment.find('RatingValue').text.strip() or \"\")\n",
        "  rating_dates.append(establishment.find('RatingDate').text.strip() or \"\")\n",
        "  business_types.append(establishment.find('BusinessType').text.strip() or \"\")\n",
        "\n",
        "  # Find Geocode data (might not exist)\n",
        "  #if establishment.find('Geocode') != None:\n",
        "  #print(establishment.find('Geocode'))\n",
        "  geocode = establishment.find('Geocode')\n",
        "  #print(geocode)\n",
        "  if geocode.find('Latitude') != None:\n",
        "    lats.append(geocode.find('Latitude').text.strip())\n",
        "    lngs.append(geocode.find('Longitude').text.strip())\n",
        "  else:\n",
        "    lats.append(\"\")\n",
        "    lngs.append(\"\")\n",
        "\n",
        "# Create a dictionary from lists\n",
        "data = {\n",
        "    \"BusinessName\": business_names,\n",
        "    \"Authority\": las,\n",
        "    \"AddressLine1\": address_line_1s,\n",
        "    \"AddressLine2\": address_line_2s,\n",
        "    \"AddressLine3\": address_line_3s,\n",
        "    \"AddressLine4\": address_line_4s,\n",
        "    \"PostCode\": post_codes,\n",
        "    \"RatingValue\": rating_values,\n",
        "    \"NewRatingPending\": nrps,\n",
        "    \"RatingDate\": rating_dates,\n",
        "    \"BusinessType\": business_types,\n",
        "    \"Lat\": lats,\n",
        "    \"Lng\": lngs\n",
        "}\n",
        "\n",
        "# Create pandas dataframe\n",
        "df = pd.DataFrame(data)\n",
        "\n",
        "# Print\n",
        "df\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 669
        },
        "id": "q9wZq1HQ_3NK",
        "outputId": "823821de-d26e-4639-e17c-15f19a311abd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                          BusinessName  Authority  \\\n",
              "0    :ROSEBANK NURSING HOMES LTD T/A Ardtully Retir...  Brentwood   \n",
              "1    124 (Essex) Transport Squadron Rlc Army Reserv...  Brentwood   \n",
              "2                                         55 Above Ltd  Brentwood   \n",
              "3                                                A & S  Brentwood   \n",
              "4                                            A B Roots  Brentwood   \n",
              "..                                                 ...        ...   \n",
              "660                                      Yiamas \\& NYX  Brentwood   \n",
              "661                                            Yorkies  Brentwood   \n",
              "662                                            ZEBRANO  Brentwood   \n",
              "663                                              Zizzi  Brentwood   \n",
              "664                                      Zoe Stanbakes  Brentwood   \n",
              "\n",
              "                 AddressLine1    AddressLine2 AddressLine3    AddressLine4  \\\n",
              "0    Ardtully Retirement Home    Station Lane  Ingatestone           Essex   \n",
              "1     Territorial Army Centre      Clive Road       Warley       Brentwood   \n",
              "2                                                                            \n",
              "3         13 Eastham Crescent       Brentwood        Essex                   \n",
              "4                 Spring Farm  Blackmore Road  Ingatestone           Essex   \n",
              "..                        ...             ...          ...             ...   \n",
              "660                Restaurant          Yiamas   Ongar Road  Pilgrims Hatch   \n",
              "661           186 Warley Hill          Warley        Essex                   \n",
              "662            161 Kings Road       Brentwood        Essex                   \n",
              "663         72-74 High Street       Brentwood        Essex                   \n",
              "664                                                                          \n",
              "\n",
              "     PostCode RatingValue NewRatingPending  RatingDate  \\\n",
              "0     CM4 0BL           5            False  2023-11-20   \n",
              "1    CM13 3UJ           5            False  2023-07-31   \n",
              "2                       5            False  2020-01-21   \n",
              "3    CM13 2BN           5            False  2023-02-14   \n",
              "4     CM4 0NP           5            False  2019-07-15   \n",
              "..        ...         ...              ...         ...   \n",
              "660  CM15 9SS           5            False  2023-11-17   \n",
              "661  CM14 5HF           5            False  2023-02-13   \n",
              "662  CM14 4EG           5            False  2023-12-06   \n",
              "663  CM14 4AN           5            False  2022-06-17   \n",
              "664                     5            False  2021-07-30   \n",
              "\n",
              "                            BusinessType               Lat                Lng  \n",
              "0    Hospitals/Childcare/Caring Premises         51.666644           0.380843  \n",
              "1                      Pub/bar/nightclub         51.599399           0.296689  \n",
              "2                      Retailers - other                                       \n",
              "3                      Retailers - other         51.612116           0.327679  \n",
              "4                      Retailers - other         51.677724           0.355835  \n",
              "..                                   ...               ...                ...  \n",
              "660              Other catering premises        51.6544384          0.2683294  \n",
              "661              Restaurant/Cafe/Canteen        51.6057156          0.2965792  \n",
              "662              Other catering premises  51.6150512695313  0.299198001623154  \n",
              "663              Other catering premises         51.619841           0.301203  \n",
              "664              Other catering premises                                       \n",
              "\n",
              "[665 rows x 13 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9991d913-5c96-4ec3-a2e1-8c3aaa79f5c7\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>:ROSEBANK NURSING HOMES LTD T/A Ardtully Retir...</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Ardtully Retirement Home</td>\n",
              "      <td>Station Lane</td>\n",
              "      <td>Ingatestone</td>\n",
              "      <td>Essex</td>\n",
              "      <td>CM4 0BL</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-11-20</td>\n",
              "      <td>Hospitals/Childcare/Caring Premises</td>\n",
              "      <td>51.666644</td>\n",
              "      <td>0.380843</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>124 (Essex) Transport Squadron Rlc Army Reserv...</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Territorial Army Centre</td>\n",
              "      <td>Clive Road</td>\n",
              "      <td>Warley</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>CM13 3UJ</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-07-31</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>51.599399</td>\n",
              "      <td>0.296689</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>55 Above Ltd</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2020-01-21</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>A &amp; S</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>13 Eastham Crescent</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Essex</td>\n",
              "      <td></td>\n",
              "      <td>CM13 2BN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-02-14</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.612116</td>\n",
              "      <td>0.327679</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>A B Roots</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Spring Farm</td>\n",
              "      <td>Blackmore Road</td>\n",
              "      <td>Ingatestone</td>\n",
              "      <td>Essex</td>\n",
              "      <td>CM4 0NP</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2019-07-15</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.677724</td>\n",
              "      <td>0.355835</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>660</th>\n",
              "      <td>Yiamas \\&amp; NYX</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Restaurant</td>\n",
              "      <td>Yiamas</td>\n",
              "      <td>Ongar Road</td>\n",
              "      <td>Pilgrims Hatch</td>\n",
              "      <td>CM15 9SS</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-11-17</td>\n",
              "      <td>Other catering premises</td>\n",
              "      <td>51.6544384</td>\n",
              "      <td>0.2683294</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>661</th>\n",
              "      <td>Yorkies</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>186 Warley Hill</td>\n",
              "      <td>Warley</td>\n",
              "      <td>Essex</td>\n",
              "      <td></td>\n",
              "      <td>CM14 5HF</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-02-13</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>51.6057156</td>\n",
              "      <td>0.2965792</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>662</th>\n",
              "      <td>ZEBRANO</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>161 Kings Road</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Essex</td>\n",
              "      <td></td>\n",
              "      <td>CM14 4EG</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-12-06</td>\n",
              "      <td>Other catering premises</td>\n",
              "      <td>51.6150512695313</td>\n",
              "      <td>0.299198001623154</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>663</th>\n",
              "      <td>Zizzi</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>72-74 High Street</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Essex</td>\n",
              "      <td></td>\n",
              "      <td>CM14 4AN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-06-17</td>\n",
              "      <td>Other catering premises</td>\n",
              "      <td>51.619841</td>\n",
              "      <td>0.301203</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>664</th>\n",
              "      <td>Zoe Stanbakes</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2021-07-30</td>\n",
              "      <td>Other catering premises</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>665 rows × 13 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9991d913-5c96-4ec3-a2e1-8c3aaa79f5c7')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9991d913-5c96-4ec3-a2e1-8c3aaa79f5c7 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9991d913-5c96-4ec3-a2e1-8c3aaa79f5c7');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6d0873c9-ddc6-46c0-91e5-22618725ab55\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6d0873c9-ddc6-46c0-91e5-22618725ab55')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6d0873c9-ddc6-46c0-91e5-22618725ab55 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 665,\n  \"fields\": [\n    {\n      \"column\": \"BusinessName\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 648,\n        \"samples\": [\n          \"Woodland Bakery\",\n          \"Hatch Food & Wine\",\n          \"The White Horse (Wet Side ONLY)\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Authority\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"Brentwood\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AddressLine1\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 502,\n        \"samples\": [\n          \"Cleves Avenue\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AddressLine2\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 166,\n        \"samples\": [\n          \"Little Warley\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AddressLine3\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 46,\n        \"samples\": [\n          \"82 High Street\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AddressLine4\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 15,\n        \"samples\": [\n          \"Hutton\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PostCode\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 284,\n        \"samples\": [\n          \"CM5 9RG\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RatingValue\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"5\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"NewRatingPending\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"True\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RatingDate\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 371,\n        \"samples\": [\n          \"2018-08-14\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"BusinessType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 14,\n        \"samples\": [\n          \"Hotel/bed & breakfast/guest house\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Lat\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 388,\n        \"samples\": [\n          \"51.6131703\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Lng\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 390,\n        \"samples\": [\n          \"0.259968\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Identify the business types\n",
        "\n",
        "We are not going to look at all businesses, so we need a list of types that we might choose from."
      ],
      "metadata": {
        "id": "pF_ADC9FRB-c"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#show the unique values, and the count of each\n",
        "df['BusinessType'].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qP3b7u6KQ2tX",
        "outputId": "424c163d-1187-40bb-a613-b7ece359a97a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Restaurant/Cafe/Canteen                  122\n",
              "Other catering premises                  121\n",
              "Retailers - other                        116\n",
              "Pub/bar/nightclub                         89\n",
              "Takeaway/sandwich shop                    52\n",
              "School/college/university                 49\n",
              "Hospitals/Childcare/Caring Premises       44\n",
              "Mobile caterer                            36\n",
              "Retailers - supermarkets/hypermarkets     18\n",
              "Manufacturers/packers                      8\n",
              "Hotel/bed & breakfast/guest house          5\n",
              "Importers/Exporters                        2\n",
              "Distributors/Transporters                  2\n",
              "Farmers/growers                            1\n",
              "Name: BusinessType, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Filtering to select categories\n",
        "\n",
        "The categories we are going to go with as fitting into our criteria of 'places a person might eat out' are:\n",
        "\n",
        "* Restaurant/Cafe/Canteen\n",
        "* Pub/bar/nightclub\n",
        "* Takeaway/sandwich shop\n",
        "* Mobile caterer\n",
        "\n",
        "\n",
        "We are going to exclude 'Other catering premises' as inspection suggests this is almost entirely sports clubs and home-based cake/baking operations."
      ],
      "metadata": {
        "id": "sDZd3UT8Thm3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "filtereddf = df[\n",
        "    (df['BusinessType'] == 'Restaurant/Cafe/Canteen') |\n",
        "    (df['BusinessType'] == 'Pub/bar/nightclub') |\n",
        "    (df['BusinessType'] == 'Takeaway/sandwich shop') |\n",
        "    (df['BusinessType'] == 'Mobile caterer')\n",
        " ]\n",
        "\n",
        "filtereddf['BusinessType'].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VRzjHMoQUXcY",
        "outputId": "19b6df77-fe1f-43df-e760-d4b5d67cc566"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Restaurant/Cafe/Canteen    122\n",
              "Pub/bar/nightclub           89\n",
              "Takeaway/sandwich shop      52\n",
              "Mobile caterer              36\n",
              "Name: BusinessType, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### FUNCTION: parsefsaxml\n",
        "\n",
        "We are going to need to do this repeatedly, so let's store in a function."
      ],
      "metadata": {
        "id": "egvHf490Lkew"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#define the function - it takes one parameter we call 'url'\n",
        "def parsefsaxml(url):\n",
        "  # Fetch the URL\n",
        "  response = requests.get(url)\n",
        "  # Store the content\n",
        "  xml_data = response.content\n",
        "  # Parse the XML data\n",
        "  soup = BeautifulSoup(xml_data, 'xml')\n",
        "  # Create empty lists to store data\n",
        "  establishments = []\n",
        "  business_names = []\n",
        "  address_line_1s = []\n",
        "  address_line_2s = []\n",
        "  address_line_3s = []\n",
        "  address_line_4s = []\n",
        "  post_codes = []\n",
        "  rating_values = []\n",
        "  rating_dates = []\n",
        "  business_types = []\n",
        "  las = []\n",
        "  nrps = []\n",
        "  lats = []\n",
        "  lngs = []\n",
        "\n",
        "  # Find all establishment details\n",
        "  establishments_data = soup.find_all('EstablishmentDetail')\n",
        "\n",
        "  # Extract data for each establishment\n",
        "  for establishment in establishments_data:\n",
        "    business_names.append(establishment.find('BusinessName').text.strip() or \"\")\n",
        "    las.append(establishment.find('LocalAuthorityName').text.strip() or \"\")\n",
        "    nrps.append(establishment.find('NewRatingPending').text.strip() or \"\")\n",
        "    #if it is there\n",
        "    if establishment.find('AddressLine1') != None:\n",
        "      address_line_1s.append(establishment.find('AddressLine1').text.strip() or \"\")\n",
        "    else:\n",
        "      address_line_1s.append('')\n",
        "    if establishment.find('AddressLine2') != None:\n",
        "      address_line_2s.append(establishment.find('AddressLine2').text.strip() or \"\")\n",
        "    else:\n",
        "      address_line_2s.append('')\n",
        "    if establishment.find('AddressLine3') != None:\n",
        "      address_line_3s.append(establishment.find('AddressLine3').text.strip() or \"\")\n",
        "    else:\n",
        "      address_line_3s.append('')\n",
        "    if establishment.find('AddressLine4') != None:\n",
        "      address_line_4s.append(establishment.find('AddressLine4').text.strip() or \"\")\n",
        "    else:\n",
        "      address_line_4s.append('')\n",
        "    if establishment.find('PostCode') != None:\n",
        "      post_codes.append(establishment.find('PostCode').text.strip() or \"\")\n",
        "    else:\n",
        "      post_codes.append('')\n",
        "    #this trips up on https://ratings.food.gov.uk/api/open-data-files/FHRS527en-GB.xml\n",
        "    if establishment.find('RatingValue') != None:\n",
        "      rating_values.append(establishment.find('RatingValue').text.strip() or \"\")\n",
        "    else:\n",
        "      rating_values.append('')\n",
        "    rating_dates.append(establishment.find('RatingDate').text.strip() or \"\")\n",
        "    business_types.append(establishment.find('BusinessType').text.strip() or \"\")\n",
        "\n",
        "    # Find Geocode data (might not exist)\n",
        "    #if establishment.find('Geocode') != None:\n",
        "    #print(establishment.find('Geocode'))\n",
        "    geocode = establishment.find('Geocode')\n",
        "    #print(geocode)\n",
        "    if geocode.find('Latitude') != None:\n",
        "      lats.append(geocode.find('Latitude').text.strip())\n",
        "      lngs.append(geocode.find('Longitude').text.strip())\n",
        "    else:\n",
        "      lats.append(\"\")\n",
        "      lngs.append(\"\")\n",
        "\n",
        "  # Create a dictionary from lists\n",
        "  data = {\n",
        "      \"BusinessName\": business_names,\n",
        "      \"Authority\": las,\n",
        "      \"AddressLine1\": address_line_1s,\n",
        "      \"AddressLine2\": address_line_2s,\n",
        "      \"AddressLine3\": address_line_3s,\n",
        "      \"AddressLine4\": address_line_4s,\n",
        "      \"PostCode\": post_codes,\n",
        "      \"RatingValue\": rating_values,\n",
        "      \"NewRatingPending\": nrps,\n",
        "      \"RatingDate\": rating_dates,\n",
        "      \"BusinessType\": business_types,\n",
        "      \"Lat\": lats,\n",
        "      \"Lng\": lngs\n",
        "  }\n",
        "  # Create pandas dataframe\n",
        "  df = pd.DataFrame(data)\n",
        "  #return to whatever called the function\n",
        "  return(df)\n"
      ],
      "metadata": {
        "id": "ON4vluNjLwyU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Extract the year separately\n",
        "\n",
        "The `RatingDate` column is currently a text string. As we want to filter on year, we can extract that into a dedicated column."
      ],
      "metadata": {
        "id": "s47J_1LSBEBJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#use .to_datetime() from pandas to convert the column to datetime\n",
        "#add the method .dt.year to extract the year from the resulting list of datetime objects\n",
        "df['ratingYear'] = pd.to_datetime(df['RatingDate']).dt.year"
      ],
      "metadata": {
        "id": "oLZFIHfVBGJj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Filter to those inspected before 2019"
      ],
      "metadata": {
        "id": "lJHFx987BoOz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#\n",
        "before2019 = df[df['ratingYear'] < 2019]\n",
        "before2019"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 600
        },
        "id": "8MlT5NseBv9I",
        "outputId": "3301fdad-36a6-4c63-ec1f-12c8977b7564"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                      BusinessName          AddressLine1    AddressLine2  \\\n",
              "9                     A.S.K. Wines        88 Church Lane    Doddinghurst   \n",
              "15             Adele Bywater Cakes                                         \n",
              "35                      Bar Bar.Co        159 Kings Road       Brentwood   \n",
              "44   Bentley District Village Club  Bentley Village Hall      Ongar Road   \n",
              "59                    Boots UK Ltd        51 High Street       Brentwood   \n",
              "..                             ...                   ...             ...   \n",
              "626                 Travis Perkins  41 Coxtie Green Road  Pilgrims Hatch   \n",
              "629                      Vaporetto                                         \n",
              "636                  W H Smith Ltd  1 - 2 Baytree Centre       Brentwood   \n",
              "643                       Well Ltd     201 Rayleigh Road          Hutton   \n",
              "650                          Wilko      8 Baytree Centre       Brentwood   \n",
              "\n",
              "     PostCode RatingValue  RatingDate             BusinessType  \\\n",
              "9    CM15 0NG      Exempt  2018-02-12        Retailers - other   \n",
              "15                      5  2016-01-25  Other catering premises   \n",
              "35   CM14 4EG           5  2017-11-13        Retailers - other   \n",
              "44   CM15 9RZ           5  2014-10-21        Pub/bar/nightclub   \n",
              "59   CM14 4RH           5  2017-07-03        Retailers - other   \n",
              "..        ...         ...         ...                      ...   \n",
              "626  CM14 5PN           5  2017-12-08        Retailers - other   \n",
              "629                     5  2015-10-07           Mobile caterer   \n",
              "636  CM14 4BX      Exempt  2015-07-06        Retailers - other   \n",
              "643  CM13 1LZ      Exempt  2017-12-27        Retailers - other   \n",
              "650  CM14 4BX           5  2015-07-06        Retailers - other   \n",
              "\n",
              "                  Lat                Lng  ratingYear  \n",
              "9           51.667835           0.298024      2018.0  \n",
              "15                                            2016.0  \n",
              "35   51.6150512695313  0.299198001623154      2017.0  \n",
              "44          51.640834           0.275657      2014.0  \n",
              "59          51.620859           0.302945      2017.0  \n",
              "..                ...                ...         ...  \n",
              "626         51.639362            0.27502      2017.0  \n",
              "629                                           2015.0  \n",
              "636         51.619507           0.302212      2015.0  \n",
              "643  51.6329536437988  0.351101011037827      2017.0  \n",
              "650         51.619507           0.302212      2015.0  \n",
              "\n",
              "[93 rows x 10 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1dfaae37-8788-4429-9ce3-cd19ad19e11d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>A.S.K. Wines</td>\n",
              "      <td>88 Church Lane</td>\n",
              "      <td>Doddinghurst</td>\n",
              "      <td>CM15 0NG</td>\n",
              "      <td>Exempt</td>\n",
              "      <td>2018-02-12</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.667835</td>\n",
              "      <td>0.298024</td>\n",
              "      <td>2018.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>Adele Bywater Cakes</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>5</td>\n",
              "      <td>2016-01-25</td>\n",
              "      <td>Other catering premises</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>2016.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>Bar Bar.Co</td>\n",
              "      <td>159 Kings Road</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>CM14 4EG</td>\n",
              "      <td>5</td>\n",
              "      <td>2017-11-13</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.6150512695313</td>\n",
              "      <td>0.299198001623154</td>\n",
              "      <td>2017.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>Bentley District Village Club</td>\n",
              "      <td>Bentley Village Hall</td>\n",
              "      <td>Ongar Road</td>\n",
              "      <td>CM15 9RZ</td>\n",
              "      <td>5</td>\n",
              "      <td>2014-10-21</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>51.640834</td>\n",
              "      <td>0.275657</td>\n",
              "      <td>2014.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>59</th>\n",
              "      <td>Boots UK Ltd</td>\n",
              "      <td>51 High Street</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>CM14 4RH</td>\n",
              "      <td>5</td>\n",
              "      <td>2017-07-03</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.620859</td>\n",
              "      <td>0.302945</td>\n",
              "      <td>2017.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>626</th>\n",
              "      <td>Travis Perkins</td>\n",
              "      <td>41 Coxtie Green Road</td>\n",
              "      <td>Pilgrims Hatch</td>\n",
              "      <td>CM14 5PN</td>\n",
              "      <td>5</td>\n",
              "      <td>2017-12-08</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.639362</td>\n",
              "      <td>0.27502</td>\n",
              "      <td>2017.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>629</th>\n",
              "      <td>Vaporetto</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>5</td>\n",
              "      <td>2015-10-07</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>2015.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>636</th>\n",
              "      <td>W H Smith Ltd</td>\n",
              "      <td>1 - 2 Baytree Centre</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>CM14 4BX</td>\n",
              "      <td>Exempt</td>\n",
              "      <td>2015-07-06</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.619507</td>\n",
              "      <td>0.302212</td>\n",
              "      <td>2015.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>643</th>\n",
              "      <td>Well Ltd</td>\n",
              "      <td>201 Rayleigh Road</td>\n",
              "      <td>Hutton</td>\n",
              "      <td>CM13 1LZ</td>\n",
              "      <td>Exempt</td>\n",
              "      <td>2017-12-27</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.6329536437988</td>\n",
              "      <td>0.351101011037827</td>\n",
              "      <td>2017.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>650</th>\n",
              "      <td>Wilko</td>\n",
              "      <td>8 Baytree Centre</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>CM14 4BX</td>\n",
              "      <td>5</td>\n",
              "      <td>2015-07-06</td>\n",
              "      <td>Retailers - other</td>\n",
              "      <td>51.619507</td>\n",
              "      <td>0.302212</td>\n",
              "      <td>2015.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>93 rows × 10 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1dfaae37-8788-4429-9ce3-cd19ad19e11d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1dfaae37-8788-4429-9ce3-cd19ad19e11d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1dfaae37-8788-4429-9ce3-cd19ad19e11d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-fd6cea18-ccd8-4640-b341-536a6689339f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-fd6cea18-ccd8-4640-b341-536a6689339f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-fd6cea18-ccd8-4640-b341-536a6689339f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "before2019",
              "repr_error": "0"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Show that as a percentage\n",
        "\n",
        "We can see what that is as a percentage by dividing the length (number of rows) of the filtered dataset by the length of the unfiltered dataset."
      ],
      "metadata": {
        "id": "dIGpNsT8B-Yn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "len(before2019)/len(df)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HEJ-PigpB794",
        "outputId": "9f12a846-8bd5-4580-914b-42a64046a73b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.13922155688622753"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## What are these establishments like?\n",
        "\n",
        "We use `.value_counts()` to generate a pivot table of how many rows there are in each category (rating value)."
      ],
      "metadata": {
        "id": "fRoxBsCSDgXq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "before2019['RatingValue'].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hFQq7Nn8DxRZ",
        "outputId": "8c180475-a9c6-4a6b-c6a6-e60ababe72fc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5         71\n",
              "Exempt    17\n",
              "4          5\n",
              "Name: RatingValue, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#divide all by the rows to get as %\n",
        "before2019['RatingValue'].value_counts()/len(before2019)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iLOUCTZ-Xxld",
        "outputId": "2ec9c7c6-f177-4624-baf2-9dde1567ac13"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5         0.763441\n",
              "Exempt    0.182796\n",
              "4         0.053763\n",
              "Name: RatingValue, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Repeat for 'Awaiting inspection'\n",
        "\n",
        "Some establishments don't have any date because they are 'Awaiting inspection'. Let's look at them:"
      ],
      "metadata": {
        "id": "YjquRj-wCOAh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "len(df[df['RatingValue'] == 'AwaitingInspection' ])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zf2CZ8HWCWs2",
        "outputId": "7a94db7d-a46b-4d49-da78-18373b871f16"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "17"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#divide the part by the whole\n",
        "len(df[df['RatingValue'] == 'AwaitingInspection' ])/len(df)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "45OpCQoyCuLg",
        "outputId": "64224ab9-3345-40ef-e152-d7b50b5016db"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.025449101796407185"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#add the two together\n",
        "ai_perc = len(df[df['RatingValue'] == 'AwaitingInspection' ])/len(df)\n",
        "pre19perc = len(before2019)/len(df)\n",
        "ai_perc+pre19perc"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CewFfWOPC7Ko",
        "outputId": "212df82b-964e-47c0-e36d-0b6b60ef969f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.16467065868263472"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Fetch the codes for each authority\n",
        "\n",
        "We have collected the codes for each authority covered by the FSA, and stored them in a Google Doc, which is imported below.\n",
        "\n",
        "Because the ID codes are numeric, they will be imported as numbers unless we specify otherwise, so we add the `dtype=str` parameter below to ensure all data is imported as strings."
      ],
      "metadata": {
        "id": "S-93iEgnJwYI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#store the URL we've published the Google Sheet at (as a CSV)\n",
        "fsacodesurl = \"https://docs.google.com/spreadsheets/d/e/2PACX-1vT56kwmL6BGdve2HLvPqazY9qIOC9R9OC6-yzcmwnaKgca3MrImKe2-tPF7ltlE29OkPn9ioiSBuDSi/pub?gid=1903756103&single=true&output=csv\"\n",
        "#import, all fields as strings\n",
        "fsacodedf = pd.read_csv(fsacodesurl, dtype=str)\n",
        "#show\n",
        "fsacodedf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "id": "kuAG8V9OJ34O",
        "outputId": "199fd6da-0cf8-48d6-ff5a-0459d19f9c50"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "      ID             LA only\n",
              "0    297            Babergh \n",
              "1    109           Basildon \n",
              "2    701            Bedford \n",
              "3    110          Braintree \n",
              "4    227          Breckland \n",
              "..   ...                 ...\n",
              "358  567  Rhondda Cynon Taf \n",
              "359  568            Swansea \n",
              "360  569            Torfaen \n",
              "361  570  Vale of Glamorgan \n",
              "362  571            Wrexham \n",
              "\n",
              "[363 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9f4b7034-bdd4-4b97-a763-cefa365fc8e8\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>LA only</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>297</td>\n",
              "      <td>Babergh</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>109</td>\n",
              "      <td>Basildon</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>701</td>\n",
              "      <td>Bedford</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>110</td>\n",
              "      <td>Braintree</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>227</td>\n",
              "      <td>Breckland</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>358</th>\n",
              "      <td>567</td>\n",
              "      <td>Rhondda Cynon Taf</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>359</th>\n",
              "      <td>568</td>\n",
              "      <td>Swansea</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>360</th>\n",
              "      <td>569</td>\n",
              "      <td>Torfaen</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>361</th>\n",
              "      <td>570</td>\n",
              "      <td>Vale of Glamorgan</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>362</th>\n",
              "      <td>571</td>\n",
              "      <td>Wrexham</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>363 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9f4b7034-bdd4-4b97-a763-cefa365fc8e8')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-9f4b7034-bdd4-4b97-a763-cefa365fc8e8 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-9f4b7034-bdd4-4b97-a763-cefa365fc8e8');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6be7e7c6-f43d-41d4-9db9-d50e0233ed08\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6be7e7c6-f43d-41d4-9db9-d50e0233ed08')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6be7e7c6-f43d-41d4-9db9-d50e0233ed08 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "fsacodedf",
              "summary": "{\n  \"name\": \"fsacodedf\",\n  \"rows\": 363,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 363,\n        \"samples\": [\n          \"179\",\n          \"032\",\n          \"157\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"LA only\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 363,\n        \"samples\": [\n          \"Ashford \",\n          \"South Cambridgeshire \",\n          \"East Hertfordshire \"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Generate a list of URLs\n",
        "\n",
        "These codes mean we can now generate URLs for the API endpoint for each authority.\n",
        "\n",
        "The URLs look like this:\n",
        "\n",
        "`https://ratings.food.gov.uk/api/open-data-files/FHRS561en-GB.xml`\n",
        "\n",
        "The only bit that changes is the three-digit code after `FHRS`."
      ],
      "metadata": {
        "id": "IsENag0DKbgi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#create a list to store the urls\n",
        "apiurl_list = []\n",
        "#loop through the codes\n",
        "for i in fsacodedf['ID']:\n",
        "  #form the URL with that in the middle\n",
        "  apiurl = \"https://ratings.food.gov.uk/api/open-data-files/FHRS\"+i+\"en-GB.xml\"\n",
        "  #append to the list\n",
        "  apiurl_list.append(apiurl)\n",
        "\n",
        "#show the first 5 results\n",
        "apiurl_list[:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4w3_gfJsKujy",
        "outputId": "cb022723-97e0-4217-d615-c246c0401105"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['https://ratings.food.gov.uk/api/open-data-files/FHRS297en-GB.xml',\n",
              " 'https://ratings.food.gov.uk/api/open-data-files/FHRS109en-GB.xml',\n",
              " 'https://ratings.food.gov.uk/api/open-data-files/FHRS701en-GB.xml',\n",
              " 'https://ratings.food.gov.uk/api/open-data-files/FHRS110en-GB.xml',\n",
              " 'https://ratings.food.gov.uk/api/open-data-files/FHRS227en-GB.xml']"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Loop through in groups of 100 - filtering by business type\n",
        "\n",
        "Now we test our function on multiple XML files from the API.\n",
        "\n",
        "We start by testing 5 by adding an index slice in:\n",
        "\n",
        "`for i in apiurl_list[:5]:`\n",
        "\n",
        "Then we change it to `[:100]`, then to `[100:200]` and finally `[200:]`, each time storing the results in a different dataframe so we can recombine it later.\n",
        "\n",
        "This helps us deal with problems - at one point an empty rating cell trips up the process so this limits the impact to just one third of the total."
      ],
      "metadata": {
        "id": "siRbb4BaLQd6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### The first 100"
      ],
      "metadata": {
        "id": "YkV5pMfpr1i3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#create an empty list to store the results\n",
        "df_list = []\n",
        "\n",
        "#loop through api URLs\n",
        "for i in apiurl_list[:100]:\n",
        "  print(i)\n",
        "  idf = parsefsaxml(i)\n",
        "  #filter out exempt inspections\n",
        "  idf = idf[idf['RatingValue'] != 'Exempt']\n",
        "  #filter to the eating out categories\n",
        "  idf = idf[\n",
        "    (idf['BusinessType'] == 'Restaurant/Cafe/Canteen') |\n",
        "    (idf['BusinessType'] == 'Pub/bar/nightclub') |\n",
        "    (idf['BusinessType'] == 'Takeaway/sandwich shop') |\n",
        "    (idf['BusinessType'] == 'Mobile caterer')\n",
        "  ]\n",
        "  # Append the new DataFrame to the list\n",
        "  df_list.append(idf)\n",
        "\n",
        "# Concatenate all DataFrames in the list into a single DataFrame\n",
        "alldf = pd.concat(df_list)\n",
        "\n",
        "alldf\n",
        "\n",
        "#store this 100 in one data frame\n",
        "alldf0_99 = alldf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tXmAJFYMLcVL",
        "outputId": "8fd6b3fa-e5fd-4438-d304-d9b6ac7f09d1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "https://ratings.food.gov.uk/api/open-data-files/FHRS297en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS109en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS701en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS110en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS227en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS111en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS228en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS155en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS027en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS112en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS702en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS113en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS114en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS156en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS028en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS157en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS302en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS115en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS029en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS229en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS116en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS158en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS030en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS299en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS230en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS869en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS117en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS300en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS159en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS231en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS232en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS888en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS118en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS032en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS233en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS893en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS160en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS161en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS120en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS162en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS894en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS122en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS163en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS164en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS298en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS062en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS259en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS260en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS209en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS063en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS219en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS261en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS210en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS064en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS871en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS070en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS220en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS066en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS262en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS211en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS067en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS212en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS878en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS221en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS263en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS214en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS264en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS068en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS222en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS235en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS215en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS899en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS216en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS266en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS879en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS069en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS223en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS224en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS225en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS236en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS501en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS502en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS503en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS504en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS505en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS506en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS508en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS507en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS509en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS510en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS511en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS512en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS513en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS514en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS515en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS516en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS517en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS518en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS519en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS520en-GB.xml\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#export the selection\n",
        "alldf0_99.to_csv('alldf0_99.csv')\n",
        "#download the file\n",
        "files.download('alldf0_99.csv')"
      ],
      "metadata": {
        "id": "xLF_kw9qsFcS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### The second 100"
      ],
      "metadata": {
        "id": "--l2ywJ4r523"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#create an empty list to store the results\n",
        "df_list = []\n",
        "\n",
        "#loop through api URLs\n",
        "for i in apiurl_list[100:200]:\n",
        "  print(i)\n",
        "  idf = parsefsaxml(i)\n",
        "  #filter out exempt inspections\n",
        "  idf = idf[idf['RatingValue'] != 'Exempt']\n",
        "  #filter to the eating out categories\n",
        "  idf = idf[\n",
        "    (idf['BusinessType'] == 'Restaurant/Cafe/Canteen') |\n",
        "    (idf['BusinessType'] == 'Pub/bar/nightclub') |\n",
        "    (idf['BusinessType'] == 'Takeaway/sandwich shop') |\n",
        "    (idf['BusinessType'] == 'Mobile caterer')\n",
        "  ]\n",
        "  # Append the new DataFrame to the list\n",
        "  df_list.append(idf)\n",
        "\n",
        "# Concatenate all DataFrames in the list into a single DataFrame\n",
        "alldf = pd.concat(df_list)\n",
        "\n",
        "alldf\n",
        "\n",
        "#store this 100 in one data frame\n",
        "alldf100_199 = alldf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OlgnUKDJr6xG",
        "outputId": "c03592c0-952c-4fb1-982d-481ff1849d44"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "https://ratings.food.gov.uk/api/open-data-files/FHRS521en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS522en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS523en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS524en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS525en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS526en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS527en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS528en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS529en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS530en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS531en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS532en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS533en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS874en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS706en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS410en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS859en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS861en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS416en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS417en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS707en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS860en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS648en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS427en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS862en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS429en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS897en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS898en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS403en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS196en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS405en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS703en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS704en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS197en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS055en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS198en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS889en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS199en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS412en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS200en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS414en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS415en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS418en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS201en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS202en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS203en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS419en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS204en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS422en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS424en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS205en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS421en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS428en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS430en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS431en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS890en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS206en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS056en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS434en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS435en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS207en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS323en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS324en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS179en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS131en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS882en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS875en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS021en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS180en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS268en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS325en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS326en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS181en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS182en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS132en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS102en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS133en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS305en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS306en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS134en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS188en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS135en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS184en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS307en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS136en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS103en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS137en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS327en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS867en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS105en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS185en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS896en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS328en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS870en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS308en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS138en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS269en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS876en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS884en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS309en-GB.xml\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#export the selection\n",
        "alldf100_199.to_csv('alldf100_199.csv')\n",
        "#download the file\n",
        "files.download('alldf100_199.csv')"
      ],
      "metadata": {
        "id": "GDZTwpiDsKW4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### The third batch"
      ],
      "metadata": {
        "id": "ntyJZxL2sNZ7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#create an empty list to store the results\n",
        "df_list = []\n",
        "\n",
        "#loop through api URLs\n",
        "for i in apiurl_list[200:]:\n",
        "  print(i)\n",
        "  idf = parsefsaxml(i)\n",
        "  #filter out exempt inspections\n",
        "  idf = idf[idf['RatingValue'] != 'Exempt']\n",
        "  #filter to the eating out categories\n",
        "  idf = idf[\n",
        "    (idf['BusinessType'] == 'Restaurant/Cafe/Canteen') |\n",
        "    (idf['BusinessType'] == 'Pub/bar/nightclub') |\n",
        "    (idf['BusinessType'] == 'Takeaway/sandwich shop') |\n",
        "    (idf['BusinessType'] == 'Mobile caterer')\n",
        "  ]\n",
        "  # Append the new DataFrame to the list\n",
        "  df_list.append(idf)\n",
        "\n",
        "# Concatenate all DataFrames in the list into a single DataFrame\n",
        "alldf = pd.concat(df_list)\n",
        "\n",
        "alldf\n",
        "\n",
        "#store this batch in one data frame\n",
        "alldf200_ = alldf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5emumMBssU_P",
        "outputId": "3a40c65b-7b11-40f9-8d7e-f9a2a1874fce"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "https://ratings.food.gov.uk/api/open-data-files/FHRS106en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS310en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS140en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS187en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS885en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS270en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS877en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS311en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS312en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS189en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS313en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS142en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS190en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS191en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS192en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS271en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS314en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS107en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS883en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS272en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS143en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS886en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS315en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS887en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS329en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS857en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS872en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS855en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS124en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS705en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS125en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS085en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS072en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS073en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS126en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS127en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS336en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS074en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS075en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS858en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS891en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS281en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS856en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS077en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS128en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS881en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS078en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS129en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS892en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS080en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS081en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS709en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS402en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS145en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS287en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS407en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS409en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS288en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS895en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS289en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS148en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS290en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS317en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS318en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS149en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS319en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS423en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS708en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS426en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS291en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS292en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS293en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS880en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS320en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS295en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS900en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS433en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS321en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS436en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS151en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS152en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS153en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS401en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS404en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS406en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS408en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS863en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS626en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS866en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS411en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS413en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS865en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS864en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS250en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS420en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS425en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS432en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS868en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS801en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS802en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS803en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS807en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS805en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS813en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS816en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS809en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS804en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS811en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS814en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS760en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS761en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS762en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS763en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS768en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS791en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS770en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS772en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS764en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS771en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS781en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS787en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS773en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS774en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS775en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS776en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS777en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS778en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS783en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS784en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS765en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS779en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS785en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS786en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS788en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS767en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS789en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS766en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS780en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS790en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS769en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS782en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS551en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS552en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS553en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS555en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS556en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS558en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS557en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS550en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS559en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS560en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS554en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS561en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS562en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS563en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS564en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS565en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS566en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS567en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS568en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS569en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS570en-GB.xml\n",
            "https://ratings.food.gov.uk/api/open-data-files/FHRS571en-GB.xml\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#export the selection\n",
        "alldf200_.to_csv('alldf200_.csv')\n",
        "#download the file\n",
        "files.download('alldf200_.csv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "Dq06cfVUsbKP",
        "outputId": "618db910-4745-4db3-bbed-96b364102826"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_ed66e32f-0e53-4c22-9e8c-dfe1971a9d06\", \"alldf200_.csv\", 18242406)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Combine all 3 dataframes and export\n",
        "\n",
        "Now we can combine the three dataframes we've created for the three slices of the list."
      ],
      "metadata": {
        "id": "cBbHTwA7DFTm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#combine the 3 data frames in the list\n",
        "alldf363 = pd.concat([alldf0_99,alldf100_199,alldf200_])\n"
      ],
      "metadata": {
        "id": "H8CNGKGU_7Lp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#export as a CSV\n",
        "alldf363.to_csv('alldf363.csv')\n",
        "#download the file\n",
        "files.download('alldf363.csv')"
      ],
      "metadata": {
        "id": "OiXaqM8VZcXk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "c9a70349-8614-4a02-81e6-4244d2ead4b7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_80471f84-93cc-47f2-8a95-ddeacbbfc278\", \"alldf363.csv\", 39002548)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Get an overview of business types\n",
        "\n",
        "Let's check how many there are in each business type."
      ],
      "metadata": {
        "id": "eCK0jeyCtp76"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#get a count of each type of business\n",
        "alldf363['BusinessType'].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LHn2hW8_BHSb",
        "outputId": "4e27896f-c3c2-4331-c5ca-6509c4843357"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BusinessType\n",
              "Restaurant/Cafe/Canteen    132397\n",
              "Takeaway/sandwich shop      63182\n",
              "Pub/bar/nightclub           50997\n",
              "Mobile caterer              27335\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Add the years\n",
        "\n",
        "Each time we also extract the year of inspection into a new column."
      ],
      "metadata": {
        "id": "tV-LZURtZytJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#use .to_datetime() from pandas to convert the column to datetime\n",
        "#add the method .dt.year to extract the year from the resulting list of datetime objects\n",
        "alldf363['ratingYear'] = pd.to_datetime(alldf363['RatingDate']).dt.year\n",
        "alldf363['ratingYear'].value_counts()"
      ],
      "metadata": {
        "id": "pjmomGzRYaC3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "72175448-bcea-4671-af3d-c01a2491bf17"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ratingYear\n",
              "2023.0    108549\n",
              "2022.0     59659\n",
              "2024.0     32732\n",
              "2019.0     13963\n",
              "2021.0     13637\n",
              "2018.0      8102\n",
              "2020.0      7153\n",
              "2017.0      3710\n",
              "2016.0      1944\n",
              "2015.0      1238\n",
              "2014.0       910\n",
              "2013.0       566\n",
              "2012.0       517\n",
              "2011.0       481\n",
              "2010.0       434\n",
              "2009.0       297\n",
              "2008.0       156\n",
              "2007.0       102\n",
              "1999.0       100\n",
              "2006.0        90\n",
              "2005.0        77\n",
              "2004.0        61\n",
              "2003.0        55\n",
              "2002.0        35\n",
              "2001.0        25\n",
              "2000.0        13\n",
              "1998.0         5\n",
              "1997.0         1\n",
              "1995.0         1\n",
              "1996.0         1\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Add T/F columns for pre-2022\n",
        "\n",
        "We are interested in how many haven't been inspected in at least two years. A rough approximation of that can be given by how many have a year of inspection before 2022 (there will also be some inspected in the first few months of 2022 who haven't been inspected now for over two years, but we just want a rough idea for now).\n",
        "\n"
      ],
      "metadata": {
        "id": "JBu43sqwCY8N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alldf363['pre2022'] = alldf363['ratingYear'] < 2022\n",
        "alldf363['pre2022'].value_counts()"
      ],
      "metadata": {
        "id": "11k_iLuZ3-Md",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3ac9b9cd-03f1-4490-ba9b-22c96ee58331"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pre2022\n",
              "False    220237\n",
              "True      53674\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Export with years"
      ],
      "metadata": {
        "id": "bVB4BBdkBlkm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#export as a CSV\n",
        "alldf363.to_csv('alldf363.csv')\n",
        "#download the file\n",
        "files.download('alldf363.csv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "j1RlEaRMBo0Z",
        "outputId": "8ddcb99b-0693-4e65-c066-ed923336662e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_fcb3c1c8-80e0-4aff-8e83-f7a51d41b612\", \"alldf363.csv\", 42393954)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Check awaiting inspection\n",
        "\n",
        "We have a number of records where no date is given - these also return `False` for the year being before 2022."
      ],
      "metadata": {
        "id": "HODxLN7wdgOU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "len(alldf363[alldf363['RatingValue'] == 'AwaitingInspection' ])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HkIqbHmmdHyI",
        "outputId": "5bf497f5-73ed-4b93-b76f-a2535e45baa2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "18650"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Generate a pivot table showing numbers for each authority.\n",
        "\n",
        "We can get an idea of those awaiting inspection by using the `pivot_table()` function from pandas.\n",
        "\n",
        "Note that there are two ways this is stored: as 'Awaiting Inspection' and 'AwaitingInspection' (no space).\n",
        "\n",
        "Note also that Scottish authorities use a different rating system which has three levels: pass, pass and eat safe, and improvement required."
      ],
      "metadata": {
        "id": "D9cSDQVNd029"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alldf363.pivot_table(index=\"Authority\",\n",
        "                        values=\"BusinessName\",\n",
        "                        columns=\"RatingValue\",\n",
        "                        margins=True,\n",
        "                        aggfunc=\"count\").fillna(0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 527
        },
        "id": "PndREgnwaR9s",
        "outputId": "0c01994b-ea30-4547-e588-33dbbc72da2f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RatingValue        0       1       2        3        4         5  \\\n",
              "Authority                                                          \n",
              "Aberdeen City    0.0     0.0     0.0      0.0      0.0       0.0   \n",
              "Aberdeenshire    0.0     0.0     0.0      0.0      0.0       0.0   \n",
              "Adur             0.0     4.0     0.0     10.0     21.0     162.0   \n",
              "Amber Valley     0.0     4.0     2.0     41.0     74.0     386.0   \n",
              "Anglesey         0.0     3.0     4.0     13.0     37.0     292.0   \n",
              "...              ...     ...     ...      ...      ...       ...   \n",
              "Wychavon         0.0     4.0     6.0     32.0     81.0     303.0   \n",
              "Wyre             0.0    11.0    13.0     21.0     66.0     331.0   \n",
              "Wyre Forest      0.0     1.0     5.0     35.0     64.0     209.0   \n",
              "York             0.0    14.0    24.0     41.0    176.0     711.0   \n",
              "All            503.0  3580.0  4349.0  18670.0  38358.0  165683.0   \n",
              "\n",
              "RatingValue    Awaiting Inspection  AwaitingInspection  AwaitingPublication  \\\n",
              "Authority                                                                     \n",
              "Aberdeen City                 98.0                 0.0                  0.0   \n",
              "Aberdeenshire                101.0                 0.0                  0.0   \n",
              "Adur                           0.0                 3.0                  0.0   \n",
              "Amber Valley                   0.0                64.0                  0.0   \n",
              "Anglesey                       0.0                 1.0                  0.0   \n",
              "...                            ...                 ...                  ...   \n",
              "Wychavon                       0.0                49.0                  0.0   \n",
              "Wyre                           0.0                26.0                  0.0   \n",
              "Wyre Forest                    0.0                45.0                  0.0   \n",
              "York                           0.0                51.0                  0.0   \n",
              "All                         3455.0             18650.0                 24.0   \n",
              "\n",
              "RatingValue    Improvement Required     Pass  Pass and Eat Safe     All  \n",
              "Authority                                                                \n",
              "Aberdeen City                 217.0    761.0                1.0    1077  \n",
              "Aberdeenshire                  58.0    503.0                2.0     664  \n",
              "Adur                            0.0      0.0                0.0     200  \n",
              "Amber Valley                    0.0      0.0                0.0     571  \n",
              "Anglesey                        0.0      0.0                0.0     350  \n",
              "...                             ...      ...                ...     ...  \n",
              "Wychavon                        0.0      0.0                0.0     475  \n",
              "Wyre                            0.0      0.0                0.0     468  \n",
              "Wyre Forest                     0.0      0.0                0.0     359  \n",
              "York                            0.0      0.0                0.0    1017  \n",
              "All                          1724.0  18747.0              168.0  273911  \n",
              "\n",
              "[364 rows x 13 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-29bd7890-e6b1-45eb-bb26-9f4048ba244d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>RatingValue</th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>Awaiting Inspection</th>\n",
              "      <th>AwaitingInspection</th>\n",
              "      <th>AwaitingPublication</th>\n",
              "      <th>Improvement Required</th>\n",
              "      <th>Pass</th>\n",
              "      <th>Pass and Eat Safe</th>\n",
              "      <th>All</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Authority</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Aberdeen City</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>98.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>217.0</td>\n",
              "      <td>761.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1077</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aberdeenshire</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>101.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>503.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>664</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Adur</th>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>162.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Amber Valley</th>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>41.0</td>\n",
              "      <td>74.0</td>\n",
              "      <td>386.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>64.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Anglesey</th>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>37.0</td>\n",
              "      <td>292.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wychavon</th>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>32.0</td>\n",
              "      <td>81.0</td>\n",
              "      <td>303.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>49.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>475</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre</th>\n",
              "      <td>0.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>66.0</td>\n",
              "      <td>331.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>26.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>468</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre Forest</th>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>64.0</td>\n",
              "      <td>209.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>45.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>359</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>York</th>\n",
              "      <td>0.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>24.0</td>\n",
              "      <td>41.0</td>\n",
              "      <td>176.0</td>\n",
              "      <td>711.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>51.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>All</th>\n",
              "      <td>503.0</td>\n",
              "      <td>3580.0</td>\n",
              "      <td>4349.0</td>\n",
              "      <td>18670.0</td>\n",
              "      <td>38358.0</td>\n",
              "      <td>165683.0</td>\n",
              "      <td>3455.0</td>\n",
              "      <td>18650.0</td>\n",
              "      <td>24.0</td>\n",
              "      <td>1724.0</td>\n",
              "      <td>18747.0</td>\n",
              "      <td>168.0</td>\n",
              "      <td>273911</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>364 rows × 13 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-29bd7890-e6b1-45eb-bb26-9f4048ba244d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-29bd7890-e6b1-45eb-bb26-9f4048ba244d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-29bd7890-e6b1-45eb-bb26-9f4048ba244d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-fd8b598e-77e6-4f8a-8bd4-1ece496b764e\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-fd8b598e-77e6-4f8a-8bd4-1ece496b764e')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-fd8b598e-77e6-4f8a-8bd4-1ece496b764e button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"                        aggfunc=\\\"count\\\")\",\n  \"rows\": 364,\n  \"fields\": [\n    {\n      \"column\": \"Authority\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 364,\n        \"samples\": [\n          \"Mid Sussex\",\n          \"Bracknell Forest\",\n          \"Barnet\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"0\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 26.706530877512314,\n        \"min\": 0.0,\n        \"max\": 503.0,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.0,\n          3.0,\n          10.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"1\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 187.74882435216753,\n        \"min\": 0.0,\n        \"max\": 3580.0,\n        \"num_unique_values\": 49,\n        \"samples\": [\n          185.0,\n          72.0,\n          64.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"2\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 228.0683650885829,\n        \"min\": 0.0,\n        \"max\": 4349.0,\n        \"num_unique_values\": 55,\n        \"samples\": [\n          38.0,\n          11.0,\n          32.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"3\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 977.6704596252724,\n        \"min\": 0.0,\n        \"max\": 18670.0,\n        \"num_unique_values\": 128,\n        \"samples\": [\n          83.0,\n          130.0,\n          99.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"4\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2007.6069967096823,\n        \"min\": 0.0,\n        \"max\": 38358.0,\n        \"num_unique_values\": 184,\n        \"samples\": [\n          311.0,\n          291.0,\n          94.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"5\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 8668.455973860426,\n        \"min\": 0.0,\n        \"max\": 165683.0,\n        \"num_unique_values\": 269,\n        \"samples\": [\n          195.0,\n          627.0,\n          328.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Awaiting Inspection\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 195.41218639726375,\n        \"min\": 0.0,\n        \"max\": 3455.0,\n        \"num_unique_values\": 28,\n        \"samples\": [\n          5.0,\n          37.0,\n          51.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AwaitingInspection\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 979.1363326528472,\n        \"min\": 0.0,\n        \"max\": 18650.0,\n        \"num_unique_values\": 125,\n        \"samples\": [\n          2.0,\n          107.0,\n          67.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AwaitingPublication\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.2895749931677736,\n        \"min\": 0.0,\n        \"max\": 24.0,\n        \"num_unique_values\": 4,\n        \"samples\": [\n          1.0,\n          24.0,\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Improvement Required\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 94.12667634153416,\n        \"min\": 0.0,\n        \"max\": 1724.0,\n        \"num_unique_values\": 28,\n        \"samples\": [\n          12.0,\n          166.0,\n          65.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Pass\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1006.9932715790806,\n        \"min\": 0.0,\n        \"max\": 18747.0,\n        \"num_unique_values\": 34,\n        \"samples\": [\n          1584.0,\n          278.0,\n          44.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Pass and Eat Safe\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 9.27567483660932,\n        \"min\": 0.0,\n        \"max\": 168.0,\n        \"num_unique_values\": 14,\n        \"samples\": [\n          26.0,\n          23.0,\n          1.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"All\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 14330,\n        \"min\": 4,\n        \"max\": 273911,\n        \"num_unique_values\": 320,\n        \"samples\": [\n          319,\n          658,\n          278\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#store in a dataframe\n",
        "pivot_rating_la = alldf363.pivot_table(index=\"Authority\",\n",
        "                        values=\"BusinessName\",\n",
        "                        columns=\"RatingValue\",\n",
        "                        margins=True,\n",
        "                        aggfunc=\"count\").fillna(0)\n",
        "#export as a CSV\n",
        "pivot_rating_la.to_csv('pivot_rating_la.csv')"
      ],
      "metadata": {
        "id": "1GblzM2HeUIG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Generate a pivot showing number of inspections pre-2022 for each authority\n",
        "\n",
        "We can repeat this for the pre2022 column, to show the numbers in each authority which are `True` (pre-2022 inspections) and `False` (not pre-2022).\n",
        "\n",
        "Note that those Awaiting Inspection will be counted as False here, so we need to combine the previous pivot table with this to get a more accurate figure."
      ],
      "metadata": {
        "id": "CkDqpOHont_-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pivot_pre22_la = alldf363.pivot_table(index=\"Authority\",\n",
        "                        values=\"BusinessName\",\n",
        "                        columns=\"pre2022\",\n",
        "                        margins=True,\n",
        "                        aggfunc=\"count\").fillna(0).astype(int)\n",
        "\n",
        "pivot_pre22_la\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 455
        },
        "id": "XRdhfNNucNrP",
        "outputId": "2e8913c6-4386-40ed-a779-f6c0ca205af4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pre2022         False   True     All\n",
              "Authority                           \n",
              "Aberdeen City     390    687    1077\n",
              "Aberdeenshire     367    297     664\n",
              "Adur              195      5     200\n",
              "Amber Valley      390    181     571\n",
              "Anglesey          293     57     350\n",
              "...               ...    ...     ...\n",
              "Wychavon          438     37     475\n",
              "Wyre              406     62     468\n",
              "Wyre Forest       321     38     359\n",
              "York              789    228    1017\n",
              "All            220237  53674  273911\n",
              "\n",
              "[364 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-aa60c7c9-9f33-413d-90dc-3cd03aeab12e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>pre2022</th>\n",
              "      <th>False</th>\n",
              "      <th>True</th>\n",
              "      <th>All</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Authority</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Aberdeen City</th>\n",
              "      <td>390</td>\n",
              "      <td>687</td>\n",
              "      <td>1077</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aberdeenshire</th>\n",
              "      <td>367</td>\n",
              "      <td>297</td>\n",
              "      <td>664</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Adur</th>\n",
              "      <td>195</td>\n",
              "      <td>5</td>\n",
              "      <td>200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Amber Valley</th>\n",
              "      <td>390</td>\n",
              "      <td>181</td>\n",
              "      <td>571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Anglesey</th>\n",
              "      <td>293</td>\n",
              "      <td>57</td>\n",
              "      <td>350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wychavon</th>\n",
              "      <td>438</td>\n",
              "      <td>37</td>\n",
              "      <td>475</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre</th>\n",
              "      <td>406</td>\n",
              "      <td>62</td>\n",
              "      <td>468</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre Forest</th>\n",
              "      <td>321</td>\n",
              "      <td>38</td>\n",
              "      <td>359</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>York</th>\n",
              "      <td>789</td>\n",
              "      <td>228</td>\n",
              "      <td>1017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>All</th>\n",
              "      <td>220237</td>\n",
              "      <td>53674</td>\n",
              "      <td>273911</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>364 rows × 3 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-aa60c7c9-9f33-413d-90dc-3cd03aeab12e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-aa60c7c9-9f33-413d-90dc-3cd03aeab12e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-aa60c7c9-9f33-413d-90dc-3cd03aeab12e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-99080869-6437-4b8b-a13e-8dba73e3fcae\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-99080869-6437-4b8b-a13e-8dba73e3fcae')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-99080869-6437-4b8b-a13e-8dba73e3fcae button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "pivot_pre22_la",
              "summary": "{\n  \"name\": \"pivot_pre22_la\",\n  \"rows\": 364,\n  \"fields\": [\n    {\n      \"column\": \"Authority\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 364,\n        \"samples\": [\n          \"Mid Sussex\",\n          \"Bracknell Forest\",\n          \"Barnet\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": false,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11520,\n        \"min\": 4,\n        \"max\": 220237,\n        \"num_unique_values\": 305,\n        \"samples\": [\n          445,\n          2551,\n          288\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": true,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2813,\n        \"min\": 0,\n        \"max\": 53674,\n        \"num_unique_values\": 204,\n        \"samples\": [\n          210,\n          61,\n          144\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"All\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 14330,\n        \"min\": 4,\n        \"max\": 273911,\n        \"num_unique_values\": 320,\n        \"samples\": [\n          319,\n          658,\n          278\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#export as a CSV\n",
        "pivot_pre22_la.to_csv('pivot_pre22_la.csv')"
      ],
      "metadata": {
        "id": "kd2jbOQhypPS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Calculate the pre22 numbers as %\n",
        "pivot_pre22_la[True]/pivot_pre22_la['All']"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uA3AVrqG0nhY",
        "outputId": "472164d8-75d6-4101-f98e-66b747c30630"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Authority\n",
              "Aberdeen City    0.637883\n",
              "Aberdeenshire    0.447289\n",
              "Adur             0.025000\n",
              "Amber Valley     0.316988\n",
              "Anglesey         0.162857\n",
              "                   ...   \n",
              "Wychavon         0.077895\n",
              "Wyre             0.132479\n",
              "Wyre Forest      0.105850\n",
              "York             0.224189\n",
              "All              0.195954\n",
              "Length: 364, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pivot_pre22_la['percPre22'] = pivot_pre22_la[True]/pivot_pre22_la['All']\n",
        "pivot_pre22_la"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 455
        },
        "id": "xw9_njYb08VP",
        "outputId": "22cc74c6-a746-427d-9d74-e11761215a84"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pre2022         False   True     All  percPre22\n",
              "Authority                                      \n",
              "Aberdeen City     390    687    1077   0.637883\n",
              "Aberdeenshire     367    297     664   0.447289\n",
              "Adur              195      5     200   0.025000\n",
              "Amber Valley      390    181     571   0.316988\n",
              "Anglesey          293     57     350   0.162857\n",
              "...               ...    ...     ...        ...\n",
              "Wychavon          438     37     475   0.077895\n",
              "Wyre              406     62     468   0.132479\n",
              "Wyre Forest       321     38     359   0.105850\n",
              "York              789    228    1017   0.224189\n",
              "All            220237  53674  273911   0.195954\n",
              "\n",
              "[364 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fd46d385-259c-4bf3-840f-8533755920c6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>pre2022</th>\n",
              "      <th>False</th>\n",
              "      <th>True</th>\n",
              "      <th>All</th>\n",
              "      <th>percPre22</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Authority</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Aberdeen City</th>\n",
              "      <td>390</td>\n",
              "      <td>687</td>\n",
              "      <td>1077</td>\n",
              "      <td>0.637883</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aberdeenshire</th>\n",
              "      <td>367</td>\n",
              "      <td>297</td>\n",
              "      <td>664</td>\n",
              "      <td>0.447289</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Adur</th>\n",
              "      <td>195</td>\n",
              "      <td>5</td>\n",
              "      <td>200</td>\n",
              "      <td>0.025000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Amber Valley</th>\n",
              "      <td>390</td>\n",
              "      <td>181</td>\n",
              "      <td>571</td>\n",
              "      <td>0.316988</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Anglesey</th>\n",
              "      <td>293</td>\n",
              "      <td>57</td>\n",
              "      <td>350</td>\n",
              "      <td>0.162857</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wychavon</th>\n",
              "      <td>438</td>\n",
              "      <td>37</td>\n",
              "      <td>475</td>\n",
              "      <td>0.077895</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre</th>\n",
              "      <td>406</td>\n",
              "      <td>62</td>\n",
              "      <td>468</td>\n",
              "      <td>0.132479</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre Forest</th>\n",
              "      <td>321</td>\n",
              "      <td>38</td>\n",
              "      <td>359</td>\n",
              "      <td>0.105850</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>York</th>\n",
              "      <td>789</td>\n",
              "      <td>228</td>\n",
              "      <td>1017</td>\n",
              "      <td>0.224189</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>All</th>\n",
              "      <td>220237</td>\n",
              "      <td>53674</td>\n",
              "      <td>273911</td>\n",
              "      <td>0.195954</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>364 rows × 4 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fd46d385-259c-4bf3-840f-8533755920c6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-fd46d385-259c-4bf3-840f-8533755920c6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-fd46d385-259c-4bf3-840f-8533755920c6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-84532d70-a846-4c97-8ba4-fefcdd21d976\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-84532d70-a846-4c97-8ba4-fefcdd21d976')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-84532d70-a846-4c97-8ba4-fefcdd21d976 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "pivot_pre22_la",
              "summary": "{\n  \"name\": \"pivot_pre22_la\",\n  \"rows\": 364,\n  \"fields\": [\n    {\n      \"column\": \"Authority\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 364,\n        \"samples\": [\n          \"Mid Sussex\",\n          \"Bracknell Forest\",\n          \"Barnet\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": false,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11520,\n        \"min\": 4,\n        \"max\": 220237,\n        \"num_unique_values\": 305,\n        \"samples\": [\n          445,\n          2551,\n          288\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": true,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2813,\n        \"min\": 0,\n        \"max\": 53674,\n        \"num_unique_values\": 204,\n        \"samples\": [\n          210,\n          61,\n          144\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"All\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 14330,\n        \"min\": 4,\n        \"max\": 273911,\n        \"num_unique_values\": 320,\n        \"samples\": [\n          319,\n          658,\n          278\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"percPre22\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.12510059381812863,\n        \"min\": 0.0,\n        \"max\": 0.6558441558441559,\n        \"num_unique_values\": 362,\n        \"samples\": [\n          0.02857142857142857,\n          0.26573426573426573,\n          0.11439842209072978\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data checking: duplicates\n",
        "\n",
        "Let's see if there are any duplicates."
      ],
      "metadata": {
        "id": "ATsxHN7KDnKv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#create a list of True/False values indicating whether a row is a duplicate\n",
        "dupes = alldf363.duplicated()\n",
        "#Get a count of T/F\n",
        "dupes.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DMGefphcD5Ca",
        "outputId": "bc6460ba-56f0-46ca-cefb-64ac305ced73"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False    273820\n",
              "True         91\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Is there any pattern to the authorities involved?"
      ],
      "metadata": {
        "id": "zMah9IUEEGq3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Use that list to filter to duplicate entries, and get a count of the authorities\n",
        "alldf363['Authority'][dupes].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1uni0jCSEDAc",
        "outputId": "476c719f-8694-4641-f4c4-744985cff8dc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Authority\n",
              "Hackney                                8\n",
              "Stirling                               7\n",
              "Lewisham                               7\n",
              "Bournemouth, Christchurch and Poole    5\n",
              "Wirral                                 5\n",
              "South Hams                             3\n",
              "West Lothian                           3\n",
              "Leeds                                  3\n",
              "City of London Corporation             3\n",
              "South Ribble                           3\n",
              "Gateshead                              2\n",
              "Bristol                                2\n",
              "East Riding of Yorkshire               2\n",
              "Hull City                              2\n",
              "Preston                                2\n",
              "Milton Keynes                          2\n",
              "Kensington and Chelsea                 2\n",
              "Glasgow City                           2\n",
              "West Northamptonshire                  2\n",
              "Renfrewshire                           2\n",
              "Bromsgrove                             1\n",
              "Dumfries and Galloway                  1\n",
              "Highland                               1\n",
              "North Lanarkshire                      1\n",
              "Scottish Borders                       1\n",
              "Warwick                                1\n",
              "Stratford-on-Avon                      1\n",
              "Edinburgh (City of)                    1\n",
              "Babergh                                1\n",
              "Plymouth City                          1\n",
              "Gloucester City                        1\n",
              "Brentwood                              1\n",
              "Elmbridge                              1\n",
              "Warrington                             1\n",
              "Trafford                               1\n",
              "Wandsworth                             1\n",
              "Camden                                 1\n",
              "West Lindsey                           1\n",
              "Leicester City                         1\n",
              "Charnwood                              1\n",
              "Rochford                               1\n",
              "Colchester                             1\n",
              "Chelmsford                             1\n",
              "Powys                                  1\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Use that list to filter to duplicate entries, and get a count of the authorities\n",
        "alldf363['BusinessName'][dupes].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZgsL-oA0EKIb",
        "outputId": "5028460f-ba06-4851-d557-cc21da7c3924"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BusinessName\n",
              "W Asher And Sons                  7\n",
              "Big Chief Ices                    5\n",
              "The Bear and Blacksmith           2\n",
              "Fotheringhams Ice Cream           2\n",
              "Food 4 Festival (Greek)           2\n",
              "                                 ..\n",
              "Mye Green Mango                   1\n",
              "Imma Kandey African Restaurant    1\n",
              "Fortune House                     1\n",
              "Daku Coffee One Limited           1\n",
              "ESS Compass Group                 1\n",
              "Name: count, Length: 76, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's just look at them."
      ],
      "metadata": {
        "id": "ScHF5kubETBg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Use that list to filter to duplicate entries, and get a count of the authorities\n",
        "alldf363[dupes]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 652
        },
        "id": "UCCUZh57EQX3",
        "outputId": "59c70359-cf1f-48cc-ab28-b9195acbadf9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                           BusinessName     Authority  \\\n",
              "70                  Bildeston Baptist Church Lunch Club       Babergh   \n",
              "401                                 Ongar Road Fish Bar     Brentwood   \n",
              "1148  Streetfood Feeding The Homeless Within The Com...    Chelmsford   \n",
              "367                                         CornDog Row    Colchester   \n",
              "269                                        Hockley Cafe      Rochford   \n",
              "...                                                 ...           ...   \n",
              "1287                                   W Asher And Sons      Stirling   \n",
              "362                             Fotheringhams Ice Cream  West Lothian   \n",
              "364                             Fotheringhams Ice Cream  West Lothian   \n",
              "645                                            Mr Softy  West Lothian   \n",
              "756                                   ESS Compass Group         Powys   \n",
              "\n",
              "                  AddressLine1       AddressLine2             AddressLine3  \\\n",
              "70    Bildeston Baptist Church        Duke Street                Bildeston   \n",
              "401              71 Ongar Road          Brentwood                    Essex   \n",
              "1148                                                                         \n",
              "367                                                                          \n",
              "269                26 Spa Road            Hockley                    Essex   \n",
              "...                        ...                ...                      ...   \n",
              "1287             156 Drip Road            Raploch                 Stirling   \n",
              "362                     Mobile  Various Locations  Trading in West Lothian   \n",
              "364                     Mobile  Various Locations  Trading in West Lothian   \n",
              "645                     Mobile  Various Locations  Trading in West Lothian   \n",
              "756                                                                          \n",
              "\n",
              "     AddressLine4  PostCode          RatingValue NewRatingPending  RatingDate  \\\n",
              "70        IPSWICH   IP7 7EW                    5            False  2023-01-11   \n",
              "401                CM15 9AZ                    5            False  2023-11-20   \n",
              "1148                                           5            False  2024-03-11   \n",
              "367                           AwaitingInspection            False               \n",
              "269                 SS5 4PH                    5            False  2019-03-18   \n",
              "...           ...       ...                  ...              ...         ...   \n",
              "1287                FK8 1RR                 Pass            False  2019-05-15   \n",
              "362                EH54 6FF  Awaiting Inspection            False  2022-06-22   \n",
              "364                EH54 6FF  Awaiting Inspection            False  2022-06-22   \n",
              "645                EH54 6FF                 Pass            False  2017-10-17   \n",
              "756                                            5            False  2023-01-11   \n",
              "\n",
              "                 BusinessType         Lat         Lng  ratingYear  pre2022  \n",
              "70    Restaurant/Cafe/Canteen                              2023.0    False  \n",
              "401    Takeaway/sandwich shop  51.6232121   0.3036676      2023.0    False  \n",
              "1148           Mobile caterer                              2024.0    False  \n",
              "367    Takeaway/sandwich shop                                 NaN    False  \n",
              "269   Restaurant/Cafe/Canteen   51.601448    0.655933      2019.0     True  \n",
              "...                       ...         ...         ...         ...      ...  \n",
              "1287           Mobile caterer   56.131784  -3.9563926      2019.0     True  \n",
              "362            Mobile caterer    55.88695   -3.516103      2022.0    False  \n",
              "364            Mobile caterer    55.88695   -3.516103      2022.0    False  \n",
              "645            Mobile caterer    55.88695   -3.516103      2017.0     True  \n",
              "756   Restaurant/Cafe/Canteen                              2023.0    False  \n",
              "\n",
              "[91 rows x 15 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7897ad2e-fd76-42c6-9345-5eb4d54005c6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>70</th>\n",
              "      <td>Bildeston Baptist Church Lunch Club</td>\n",
              "      <td>Babergh</td>\n",
              "      <td>Bildeston Baptist Church</td>\n",
              "      <td>Duke Street</td>\n",
              "      <td>Bildeston</td>\n",
              "      <td>IPSWICH</td>\n",
              "      <td>IP7 7EW</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-01-11</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>401</th>\n",
              "      <td>Ongar Road Fish Bar</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>71 Ongar Road</td>\n",
              "      <td>Brentwood</td>\n",
              "      <td>Essex</td>\n",
              "      <td></td>\n",
              "      <td>CM15 9AZ</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-11-20</td>\n",
              "      <td>Takeaway/sandwich shop</td>\n",
              "      <td>51.6232121</td>\n",
              "      <td>0.3036676</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1148</th>\n",
              "      <td>Streetfood Feeding The Homeless Within The Com...</td>\n",
              "      <td>Chelmsford</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2024-03-11</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>2024.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>367</th>\n",
              "      <td>CornDog Row</td>\n",
              "      <td>Colchester</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td></td>\n",
              "      <td>Takeaway/sandwich shop</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>269</th>\n",
              "      <td>Hockley Cafe</td>\n",
              "      <td>Rochford</td>\n",
              "      <td>26 Spa Road</td>\n",
              "      <td>Hockley</td>\n",
              "      <td>Essex</td>\n",
              "      <td></td>\n",
              "      <td>SS5 4PH</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2019-03-18</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>51.601448</td>\n",
              "      <td>0.655933</td>\n",
              "      <td>2019.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1287</th>\n",
              "      <td>W Asher And Sons</td>\n",
              "      <td>Stirling</td>\n",
              "      <td>156 Drip Road</td>\n",
              "      <td>Raploch</td>\n",
              "      <td>Stirling</td>\n",
              "      <td></td>\n",
              "      <td>FK8 1RR</td>\n",
              "      <td>Pass</td>\n",
              "      <td>False</td>\n",
              "      <td>2019-05-15</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>56.131784</td>\n",
              "      <td>-3.9563926</td>\n",
              "      <td>2019.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>362</th>\n",
              "      <td>Fotheringhams Ice Cream</td>\n",
              "      <td>West Lothian</td>\n",
              "      <td>Mobile</td>\n",
              "      <td>Various Locations</td>\n",
              "      <td>Trading in West Lothian</td>\n",
              "      <td></td>\n",
              "      <td>EH54 6FF</td>\n",
              "      <td>Awaiting Inspection</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-06-22</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>55.88695</td>\n",
              "      <td>-3.516103</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>364</th>\n",
              "      <td>Fotheringhams Ice Cream</td>\n",
              "      <td>West Lothian</td>\n",
              "      <td>Mobile</td>\n",
              "      <td>Various Locations</td>\n",
              "      <td>Trading in West Lothian</td>\n",
              "      <td></td>\n",
              "      <td>EH54 6FF</td>\n",
              "      <td>Awaiting Inspection</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-06-22</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>55.88695</td>\n",
              "      <td>-3.516103</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>645</th>\n",
              "      <td>Mr Softy</td>\n",
              "      <td>West Lothian</td>\n",
              "      <td>Mobile</td>\n",
              "      <td>Various Locations</td>\n",
              "      <td>Trading in West Lothian</td>\n",
              "      <td></td>\n",
              "      <td>EH54 6FF</td>\n",
              "      <td>Pass</td>\n",
              "      <td>False</td>\n",
              "      <td>2017-10-17</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>55.88695</td>\n",
              "      <td>-3.516103</td>\n",
              "      <td>2017.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>756</th>\n",
              "      <td>ESS Compass Group</td>\n",
              "      <td>Powys</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-01-11</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td></td>\n",
              "      <td></td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>91 rows × 15 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7897ad2e-fd76-42c6-9345-5eb4d54005c6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7897ad2e-fd76-42c6-9345-5eb4d54005c6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7897ad2e-fd76-42c6-9345-5eb4d54005c6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-e86567dd-96c3-4ee8-acd6-ca539e62b143\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-e86567dd-96c3-4ee8-acd6-ca539e62b143')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-e86567dd-96c3-4ee8-acd6-ca539e62b143 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "0"
            }
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Export deduplicated\n",
        "\n",
        "The numbers involved here are around 0.3% but we will remove them anyway."
      ],
      "metadata": {
        "id": "80EpnDO1EpTb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "alldf363deduplicated = alldf363.drop_duplicates()\n",
        "#export as a CSV\n",
        "alldf363deduplicated.to_csv('alldf363deduplicated.csv')\n",
        "#download the file\n",
        "files.download('alldf363deduplicated.csv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "id": "ITQVhwn1Evj7",
        "outputId": "37b049ea-8763-4e2e-aac6-16fb86483143"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_1625267e-416e-44f8-897d-f9cd6bcaa35e\", \"alldf363deduplicated.csv\", 42381376)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Import and deduplicate again by BusinessName/latlong\n",
        "\n",
        "Although we have deduplicated where rows are the same, there might also be rows which are duplicates in the sense of them being the same business, but a different rating/inspection date.\n",
        "\n",
        "Pilar Thomas conducted some spot-checking on the exported data: first, a preliminary conditional formatting in Excel to have a general view, and then using OpenRefine.\n",
        "\n",
        "The process followed was:\n",
        "\n",
        "1. Reorder spreadsheet so Lat column is sorted a – z, and select Reorder Rows Permanently.\n",
        "2. Apply Duplicates facet to Lat and select True, so exact latitude figures will be ordered in consecutive rows.\n",
        "3. Blank down cells in column BusinessName – this will identify two consecutive columns with the same BusinessName and then delete the name in the second row so it's easier to filter later. The reasoning behind this is that we'll probably then be able to detect rows with the same latitude and same business name, which could easily be duplicates.\n",
        "4. Apply Facet by blank to column BusinessName.\n",
        "Star these rows (732 in total) so they are easier to check later on, and then delete Facet by blank.\n",
        "5. Sort A-Z by BusinessName and drag Blanks (starred rows) so they are in top of the column.\n",
        "Apply Text filter to column AddressLine1.\n",
        "\n",
        "Then random spot-checking:\n",
        "6. Starting with the first row (number 1222.), search its AddressLine1 (Bickels Yard Cafe ( Fusion )) in the Text filter box.\n",
        "7. Two exact rows appear with the same latitude, so we flag the starred one (with the blank BusinessName), as Bickels Yard Cafe is clearly duplicated.\n",
        "8. Reset AddressLine1 Text filter and repeat this process with other random rows.\n",
        "\n",
        "We now try to codify this process in Python.\n"
      ],
      "metadata": {
        "id": "aMTV29HLR2Lc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#import the exported data again if needed\n",
        "exporturl = \"https://docs.google.com/spreadsheets/d/e/2PACX-1vSNbsW3eA3efpYVJd_6M2Y2LttxcKPvluqmMFXnOU9_3uGnLrbwglgftFYAjgWx2PUZgMEo5q2faUCw/pub?gid=681915056&single=true&output=csv\"\n",
        "exportdf = pd.read_csv(exporturl)\n",
        "#check how many rows\n",
        "len(exportdf)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I7gkzKzTSkP4",
        "outputId": "4da16fdf-f4a0-41a7-def3-1ba0bba6f4db"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "273820"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#remove duplicates - specifying which columns we want to deduplicate on\n",
        "deduped_BNLL = exportdf.drop_duplicates(subset=['BusinessName','Lat','Lng'])\n",
        "#how many does that leave\n",
        "len(deduped_BNLL)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zXidrCiwTntC",
        "outputId": "4e4d9f88-7c3e-4d63-8fca-9ce0f1e5ec2b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "270881"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Count how many duplicates - and check\n",
        "\n",
        "We now look at what rows have been identified as duplicates."
      ],
      "metadata": {
        "id": "iE5XZa7kVAnk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#create a list of True/False values indicating whether a row is a duplicate\n",
        "dupes = exportdf.duplicated(subset=['BusinessName','Lat','Lng'],\n",
        "                            keep = False) #keep all duplicates\n",
        "#Get a count of T/F\n",
        "dupes.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wTvfSB1jUiE5",
        "outputId": "ff88bbe7-ca36-42f2-894c-874f03d1d983"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False    269435\n",
              "True       4385\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "An inspection of the results flags a potential cause of false positives in the deduplication: rows where Lat and Lng are `NaN`. In these cases, only the BusinessName field is left to deduplicate on, so 14Forty in two different places is treated as a duplicate."
      ],
      "metadata": {
        "id": "cJD1RHj0VL6s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Use that list to filter to duplicate entries, and sort it by BusinessName so we can see them together\n",
        "exportdf[dupes].sort_values(by = ['BusinessName'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 721
        },
        "id": "9BN9V_aqUqvn",
        "outputId": "e57e040b-1552-4d17-8844-7673ddac85af"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0                    BusinessName        Authority  \\\n",
              "190913           1                         14Forty    Herefordshire   \n",
              "193859           1                         14Forty            Rugby   \n",
              "142498           2                         14Forty    Milton Keynes   \n",
              "167813           7  23.5 Degrees Ltd T/A Starbucks           Dorset   \n",
              "167812           6  23.5 Degrees Ltd T/A Starbucks           Dorset   \n",
              "...            ...                             ...              ...   \n",
              "143539        2334                    Zorro's Cafe    Milton Keynes   \n",
              "250600        6301                    Zumuku Sushi     Glasgow City   \n",
              "109130        5785                    Zumuku Sushi       Manchester   \n",
              "170377         137                          cafe2u      Exeter City   \n",
              "22102          100                          cafe2u  Welwyn Hatfield   \n",
              "\n",
              "                                             AddressLine1       AddressLine2  \\\n",
              "190913                                        Cider Mills        Plough Lane   \n",
              "193859                   , DHL for NHS Supply Chain Valle       Cosford Lane   \n",
              "142498                                       DHL Nautilus   Pendeen Crescent   \n",
              "167813                                          Starbucks                NaN   \n",
              "167812                                                209                NaN   \n",
              "...                                                   ...                ...   \n",
              "143539  Trading Location, Central Milton Keynes Market...                NaN   \n",
              "250600                                                NaN  1 Scotland Street   \n",
              "109130              New Century Kitchen, New Century Hall       Mayes Street   \n",
              "170377                                                NaN                NaN   \n",
              "22102                                                 NaN                NaN   \n",
              "\n",
              "                   AddressLine3              AddressLine4  PostCode  \\\n",
              "190913                      NaN                  Hereford   HR4 0EG   \n",
              "193859                      NaN  Swift Valley Ind. Estate  CV21 1QN   \n",
              "142498            Milton Keynes             Milton Keynes   MK4 4BX   \n",
              "167813             Corfe Mullen                    Dorset  BH21 3HA   \n",
              "167812  St Leonards And St Ives                    Dorset  BH24 2QA   \n",
              "...                         ...                       ...       ...   \n",
              "143539                      NaN                       NaN       NaN   \n",
              "250600                  Glasgow                       NaN    G5 8LS   \n",
              "109130               Manchester                       NaN   M60 4ES   \n",
              "170377                      NaN                       NaN       NaN   \n",
              "22102                       NaN                       NaN       NaN   \n",
              "\n",
              "                RatingValue  NewRatingPending  RatingDate  \\\n",
              "190913                    5             False  2022-03-18   \n",
              "193859                    5             False  2019-05-07   \n",
              "142498                    4             False  2023-05-22   \n",
              "167813                    5             False  2023-07-03   \n",
              "167812                    5             False  2022-07-15   \n",
              "...                     ...               ...         ...   \n",
              "143539   AwaitingInspection             False         NaN   \n",
              "250600  Awaiting Inspection             False  2023-07-16   \n",
              "109130                    5             False  2023-03-21   \n",
              "170377                    4             False  2023-06-23   \n",
              "22102                     5             False  2022-09-05   \n",
              "\n",
              "                   BusinessType  Lat  Lng  ratingYear  pre2022  \n",
              "190913  Restaurant/Cafe/Canteen  NaN  NaN      2022.0    False  \n",
              "193859  Restaurant/Cafe/Canteen  NaN  NaN      2019.0     True  \n",
              "142498  Restaurant/Cafe/Canteen  NaN  NaN      2023.0    False  \n",
              "167813  Restaurant/Cafe/Canteen  NaN  NaN      2023.0    False  \n",
              "167812  Restaurant/Cafe/Canteen  NaN  NaN      2022.0    False  \n",
              "...                         ...  ...  ...         ...      ...  \n",
              "143539  Restaurant/Cafe/Canteen  NaN  NaN         NaN    False  \n",
              "250600   Takeaway/sandwich shop  NaN  NaN      2023.0    False  \n",
              "109130  Restaurant/Cafe/Canteen  NaN  NaN      2023.0    False  \n",
              "170377           Mobile caterer  NaN  NaN      2023.0    False  \n",
              "22102            Mobile caterer  NaN  NaN      2022.0    False  \n",
              "\n",
              "[4385 rows x 16 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-de7cefc0-267b-431d-86b1-e5c014712767\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>190913</th>\n",
              "      <td>1</td>\n",
              "      <td>14Forty</td>\n",
              "      <td>Herefordshire</td>\n",
              "      <td>Cider Mills</td>\n",
              "      <td>Plough Lane</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Hereford</td>\n",
              "      <td>HR4 0EG</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-03-18</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>193859</th>\n",
              "      <td>1</td>\n",
              "      <td>14Forty</td>\n",
              "      <td>Rugby</td>\n",
              "      <td>, DHL for NHS Supply Chain Valle</td>\n",
              "      <td>Cosford Lane</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Swift Valley Ind. Estate</td>\n",
              "      <td>CV21 1QN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2019-05-07</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2019.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>142498</th>\n",
              "      <td>2</td>\n",
              "      <td>14Forty</td>\n",
              "      <td>Milton Keynes</td>\n",
              "      <td>DHL Nautilus</td>\n",
              "      <td>Pendeen Crescent</td>\n",
              "      <td>Milton Keynes</td>\n",
              "      <td>Milton Keynes</td>\n",
              "      <td>MK4 4BX</td>\n",
              "      <td>4</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-05-22</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>167813</th>\n",
              "      <td>7</td>\n",
              "      <td>23.5 Degrees Ltd T/A Starbucks</td>\n",
              "      <td>Dorset</td>\n",
              "      <td>Starbucks</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Corfe Mullen</td>\n",
              "      <td>Dorset</td>\n",
              "      <td>BH21 3HA</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-07-03</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>167812</th>\n",
              "      <td>6</td>\n",
              "      <td>23.5 Degrees Ltd T/A Starbucks</td>\n",
              "      <td>Dorset</td>\n",
              "      <td>209</td>\n",
              "      <td>NaN</td>\n",
              "      <td>St Leonards And St Ives</td>\n",
              "      <td>Dorset</td>\n",
              "      <td>BH24 2QA</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-07-15</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>143539</th>\n",
              "      <td>2334</td>\n",
              "      <td>Zorro's Cafe</td>\n",
              "      <td>Milton Keynes</td>\n",
              "      <td>Trading Location, Central Milton Keynes Market...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>250600</th>\n",
              "      <td>6301</td>\n",
              "      <td>Zumuku Sushi</td>\n",
              "      <td>Glasgow City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1 Scotland Street</td>\n",
              "      <td>Glasgow</td>\n",
              "      <td>NaN</td>\n",
              "      <td>G5 8LS</td>\n",
              "      <td>Awaiting Inspection</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-07-16</td>\n",
              "      <td>Takeaway/sandwich shop</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109130</th>\n",
              "      <td>5785</td>\n",
              "      <td>Zumuku Sushi</td>\n",
              "      <td>Manchester</td>\n",
              "      <td>New Century Kitchen, New Century Hall</td>\n",
              "      <td>Mayes Street</td>\n",
              "      <td>Manchester</td>\n",
              "      <td>NaN</td>\n",
              "      <td>M60 4ES</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-03-21</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>170377</th>\n",
              "      <td>137</td>\n",
              "      <td>cafe2u</td>\n",
              "      <td>Exeter City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-06-23</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22102</th>\n",
              "      <td>100</td>\n",
              "      <td>cafe2u</td>\n",
              "      <td>Welwyn Hatfield</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-09-05</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>4385 rows × 16 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-de7cefc0-267b-431d-86b1-e5c014712767')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-de7cefc0-267b-431d-86b1-e5c014712767 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-de7cefc0-267b-431d-86b1-e5c014712767');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-d6b5cbc2-104e-43aa-99cb-16e90fae3eb5\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d6b5cbc2-104e-43aa-99cb-16e90fae3eb5')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-d6b5cbc2-104e-43aa-99cb-16e90fae3eb5 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "0"
            }
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can do a count to see how many there are of each Lat."
      ],
      "metadata": {
        "id": "FpqzGSHSWDpE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Use that list to filter to duplicate entries\n",
        "#count how many of each value\n",
        "exportdf['Lat'][dupes].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L8vJkcskV53e",
        "outputId": "0350968d-7b11-4c72-85d5-ddd1132576dd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Lat\n",
              "53.773205    12\n",
              "55.948386     8\n",
              "51.304118     8\n",
              "52.083159     6\n",
              "53.466187     6\n",
              "             ..\n",
              "51.492100     2\n",
              "51.361053     2\n",
              "51.512870     2\n",
              "51.520237     2\n",
              "51.738945     2\n",
              "Name: count, Length: 243, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Check for `NaN` values in Lat\n",
        "\n",
        "...but that doesn't count NaN values (it does indicate these are the minority, and that some only appear once, which shouldn't be the case if they are duplicates)\n",
        "\n",
        "So [we need the `isnan()` function](https://www.turing.com/kb/nan-values-in-python) from the library `math`"
      ],
      "metadata": {
        "id": "sloOP3qvXSBi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#loop through each value in Lat and use in isnan() function to return a list of True/False\n",
        "#create a data frame from that list\n",
        "#apply value_counts() to get a total of True and False\n",
        "pd.DataFrame([math.isnan(i) for i in exportdf['Lat'][dupes]]).value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tMjUj58tWcDd",
        "outputId": "14f8ea33-42e6-4c79-f028-ea82842007d0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True     3843\n",
              "False     542\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Deduplicating by name and postcode\n",
        "\n",
        "Let's try doing it by postcode instead."
      ],
      "metadata": {
        "id": "aJS-bh5zV1Q5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#remove duplicates - specifying which columns we want to deduplicate on\n",
        "deduped_BNP = exportdf.drop_duplicates(subset=['BusinessName','PostCode'])\n",
        "#how many does that leave\n",
        "len(deduped_BNP)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PbKCifzhVKYz",
        "outputId": "25e61852-8608-48be-8a0e-ad6269398f61"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "272192"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#create a list of True/False values indicating whether a row is a duplicate\n",
        "dupes = exportdf.duplicated(subset=['BusinessName','PostCode'],\n",
        "                            keep = False) #keep both duplicates\n",
        "#Get a count of T/F\n",
        "dupes.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eQ3vzWY1XuPu",
        "outputId": "6904e6ee-9401-4244-e62e-8b22ed650013"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False    271094\n",
              "True       2726\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Use that list to filter to duplicate entries, and sort it by BusinessName so we can see them together\n",
        "exportdf[dupes].sort_values(by = ['BusinessName'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 738
        },
        "id": "ufacjsdeYW68",
        "outputId": "5c52ebe2-7fb7-49a1-f383-49fa1dc10263"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0      BusinessName        Authority  \\\n",
              "179252           1     A Bite To Eat      Teignbridge   \n",
              "31329           35     A Bite To Eat   Leicester City   \n",
              "177650           7        A Pizza Me       South Hams   \n",
              "43573            4        A Pizza Me           Bexley   \n",
              "141248          14  A Touch Of Class           Medway   \n",
              "...            ...               ...              ...   \n",
              "222706        6559             Zizzi  North Yorkshire   \n",
              "143539        2334      Zorro's Cafe    Milton Keynes   \n",
              "143540        2335      Zorro's Cafe    Milton Keynes   \n",
              "22102          100            cafe2u  Welwyn Hatfield   \n",
              "170377         137            cafe2u      Exeter City   \n",
              "\n",
              "                                             AddressLine1 AddressLine2  \\\n",
              "179252            Town Quay Forde Road Newton Abbot Devon          NaN   \n",
              "31329                                                 NaN          NaN   \n",
              "177650                                                NaN          NaN   \n",
              "43573                                                 NaN          NaN   \n",
              "141248                                                NaN          NaN   \n",
              "...                                                   ...          ...   \n",
              "222706                                   1 Station Square    Harrogate   \n",
              "143539  Trading Location, Central Milton Keynes Market...          NaN   \n",
              "143540  Trading Location, Central Milton Keynes Market...          NaN   \n",
              "22102                                                 NaN          NaN   \n",
              "170377                                                NaN          NaN   \n",
              "\n",
              "           AddressLine3 AddressLine4 PostCode         RatingValue  \\\n",
              "179252              NaN          NaN      NaN                   5   \n",
              "31329               NaN          NaN      NaN                   5   \n",
              "177650              NaN          NaN      NaN                   5   \n",
              "43573               NaN          NaN      NaN                   5   \n",
              "141248              NaN          NaN      NaN                   5   \n",
              "...                 ...          ...      ...                 ...   \n",
              "222706  North Yorkshire          NaN      NaN                   5   \n",
              "143539              NaN          NaN      NaN  AwaitingInspection   \n",
              "143540              NaN          NaN      NaN  AwaitingInspection   \n",
              "22102               NaN          NaN      NaN                   5   \n",
              "170377              NaN          NaN      NaN                   4   \n",
              "\n",
              "        NewRatingPending  RatingDate             BusinessType        Lat  \\\n",
              "179252             False  2022-11-25           Mobile caterer  50.532931   \n",
              "31329              False  2023-10-26           Mobile caterer        NaN   \n",
              "177650             False  2020-02-27   Takeaway/sandwich shop        NaN   \n",
              "43573              False  2023-08-21           Mobile caterer        NaN   \n",
              "141248             False  2022-03-18           Mobile caterer        NaN   \n",
              "...                  ...         ...                      ...        ...   \n",
              "222706             False  2023-05-19  Restaurant/Cafe/Canteen  53.992024   \n",
              "143539             False         NaN  Restaurant/Cafe/Canteen        NaN   \n",
              "143540             False         NaN           Mobile caterer        NaN   \n",
              "22102              False  2022-09-05           Mobile caterer        NaN   \n",
              "170377             False  2023-06-23           Mobile caterer        NaN   \n",
              "\n",
              "             Lng  ratingYear  pre2022  \n",
              "179252 -3.596134      2022.0    False  \n",
              "31329        NaN      2023.0    False  \n",
              "177650       NaN      2020.0     True  \n",
              "43573        NaN      2023.0    False  \n",
              "141248       NaN      2022.0    False  \n",
              "...          ...         ...      ...  \n",
              "222706 -1.538099      2023.0    False  \n",
              "143539       NaN         NaN    False  \n",
              "143540       NaN         NaN    False  \n",
              "22102        NaN      2022.0    False  \n",
              "170377       NaN      2023.0    False  \n",
              "\n",
              "[2726 rows x 16 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-48b43d6a-d30a-4383-b8c2-305ffe454fa9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>179252</th>\n",
              "      <td>1</td>\n",
              "      <td>A Bite To Eat</td>\n",
              "      <td>Teignbridge</td>\n",
              "      <td>Town Quay Forde Road Newton Abbot Devon</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-11-25</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>50.532931</td>\n",
              "      <td>-3.596134</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31329</th>\n",
              "      <td>35</td>\n",
              "      <td>A Bite To Eat</td>\n",
              "      <td>Leicester City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-10-26</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>177650</th>\n",
              "      <td>7</td>\n",
              "      <td>A Pizza Me</td>\n",
              "      <td>South Hams</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2020-02-27</td>\n",
              "      <td>Takeaway/sandwich shop</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2020.0</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43573</th>\n",
              "      <td>4</td>\n",
              "      <td>A Pizza Me</td>\n",
              "      <td>Bexley</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-08-21</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>141248</th>\n",
              "      <td>14</td>\n",
              "      <td>A Touch Of Class</td>\n",
              "      <td>Medway</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-03-18</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>222706</th>\n",
              "      <td>6559</td>\n",
              "      <td>Zizzi</td>\n",
              "      <td>North Yorkshire</td>\n",
              "      <td>1 Station Square</td>\n",
              "      <td>Harrogate</td>\n",
              "      <td>North Yorkshire</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-05-19</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>53.992024</td>\n",
              "      <td>-1.538099</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>143539</th>\n",
              "      <td>2334</td>\n",
              "      <td>Zorro's Cafe</td>\n",
              "      <td>Milton Keynes</td>\n",
              "      <td>Trading Location, Central Milton Keynes Market...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>143540</th>\n",
              "      <td>2335</td>\n",
              "      <td>Zorro's Cafe</td>\n",
              "      <td>Milton Keynes</td>\n",
              "      <td>Trading Location, Central Milton Keynes Market...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22102</th>\n",
              "      <td>100</td>\n",
              "      <td>cafe2u</td>\n",
              "      <td>Welwyn Hatfield</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2022-09-05</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2022.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>170377</th>\n",
              "      <td>137</td>\n",
              "      <td>cafe2u</td>\n",
              "      <td>Exeter City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>4</td>\n",
              "      <td>False</td>\n",
              "      <td>2023-06-23</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2023.0</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2726 rows × 16 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-48b43d6a-d30a-4383-b8c2-305ffe454fa9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-48b43d6a-d30a-4383-b8c2-305ffe454fa9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-48b43d6a-d30a-4383-b8c2-305ffe454fa9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-4cfceabb-2ab6-4e08-8178-6bfd9945054b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4cfceabb-2ab6-4e08-8178-6bfd9945054b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-4cfceabb-2ab6-4e08-8178-6bfd9945054b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "0"
            }
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Check for missing PostCode field\n",
        "\n",
        "Again, could this be caused by them having no postcode at all?\n",
        "\n",
        "To check this we can't use `isnan()` because the column isn't numerical. So we use `isinstance()` which indicates if a variable is a specified type (in this case, `str`). An empty cell would not be a string."
      ],
      "metadata": {
        "id": "1zGkxid5YKdl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#check a postcode string\n",
        "print(isinstance('B42 2SU', str))\n",
        "#find a row with a nan value\n",
        "print(exportdf['PostCode'][dupes][15])\n",
        "#check that\n",
        "print(isinstance(exportdf['PostCode'][dupes][15], str))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9jqziBY0dsXN",
        "outputId": "6ecc7037-5f5f-4fdc-fda6-fb6455e5373f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "True\n",
            "nan\n",
            "False\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#loop through each value in PostCode and use in isinstance() function to return a list of True/False\n",
        "#create a data frame from that list\n",
        "#apply value_counts() to get a total of True and False\n",
        "pd.DataFrame([isinstance(i, str) for i in exportdf['PostCode'][dupes]]).value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nn8pa2jZX25B",
        "outputId": "23b0c97c-b4a7-42c2-b846-ab0e5416fefd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False    1842\n",
              "True      884\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "So most of the duplicates here are NaN for PostCode too."
      ],
      "metadata": {
        "id": "_4A2wkAZeMA2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Deduplicating only where there is not a NaN\n",
        "\n",
        "We need to do the following:\n",
        "\n",
        "* Sort by inspection date and BusinessName so that when we remove duplicates we remove the older inspection record\n",
        "* Create a T/F column identifying duplicates based on BusinessName/Lat/Lng\n",
        "* Create a T/F column identifying `NaN` entries in the Lat column\n",
        "* Filter out duplicates based on BusinessName/Lat/Lng where the Lat NaN column is False\n",
        "* Create a T/F column identifying duplicates based on BusinessName/PostCode\n",
        "* Create a T/F column identifying `NaN` entries in the PostCode column\n",
        "* Filter out duplicates based on BusinessName/PostCode where the PostCode NaN column is False\n",
        "* Inspect the duplicate-but-NaN results to identify any other obvious duplicates\n"
      ],
      "metadata": {
        "id": "cSifqxPWeQfJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Sorting by inspection date: `sort_values()`"
      ],
      "metadata": {
        "id": "z5POXbcSqPuu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#sort by RatingDate\n",
        "exportdf = exportdf.sort_values(by = ['RatingDate'])\n",
        "exportdf.head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        },
        "id": "KLPRprDWqQ7f",
        "outputId": "ac5192ef-041e-4493-a902-127da6554e1f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0                             BusinessName  \\\n",
              "150414        1338  Shirley And Millbrook Conservative Club   \n",
              "150673        1834   Woolston And Sholing Conservative Club   \n",
              "120371         596               Corinthians FC Social Club   \n",
              "\n",
              "                      Authority       AddressLine1  AddressLine2 AddressLine3  \\\n",
              "150414              Southampton   22 Anglesea Road   Southampton          NaN   \n",
              "150673              Southampton  74 St. Annes Road   Southampton          NaN   \n",
              "120371  Westmorland and Furness                NaN  Strands Lane          NaN   \n",
              "\n",
              "       AddressLine4  PostCode RatingValue  NewRatingPending  RatingDate  \\\n",
              "150414          NaN  SO15 5QJ           5             False  1995-04-06   \n",
              "150673          NaN  SO19 9FF           5             False  1996-06-12   \n",
              "120371   Milnthorpe   LA7 7AE           3             False  1997-11-28   \n",
              "\n",
              "             BusinessType        Lat       Lng  ratingYear  pre2022  \\\n",
              "150414  Pub/bar/nightclub  50.924719 -1.434945      1995.0     True   \n",
              "150673  Pub/bar/nightclub  50.893923 -1.368832      1996.0     True   \n",
              "120371  Pub/bar/nightclub  54.225384 -2.776882      1997.0     True   \n",
              "\n",
              "        duplicateBNLL  latNAN  duplicateBNP  postCodeSTR  \n",
              "150414          False   False         False         True  \n",
              "150673          False   False         False         True  \n",
              "120371          False   False         False         True  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7766fa30-7d5a-4411-ac8b-bb3868e22f07\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "      <th>duplicateBNLL</th>\n",
              "      <th>latNAN</th>\n",
              "      <th>duplicateBNP</th>\n",
              "      <th>postCodeSTR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>150414</th>\n",
              "      <td>1338</td>\n",
              "      <td>Shirley And Millbrook Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>22 Anglesea Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO15 5QJ</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1995-04-06</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.924719</td>\n",
              "      <td>-1.434945</td>\n",
              "      <td>1995.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150673</th>\n",
              "      <td>1834</td>\n",
              "      <td>Woolston And Sholing Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>74 St. Annes Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO19 9FF</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1996-06-12</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.893923</td>\n",
              "      <td>-1.368832</td>\n",
              "      <td>1996.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>120371</th>\n",
              "      <td>596</td>\n",
              "      <td>Corinthians FC Social Club</td>\n",
              "      <td>Westmorland and Furness</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Strands Lane</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Milnthorpe</td>\n",
              "      <td>LA7 7AE</td>\n",
              "      <td>3</td>\n",
              "      <td>False</td>\n",
              "      <td>1997-11-28</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>54.225384</td>\n",
              "      <td>-2.776882</td>\n",
              "      <td>1997.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7766fa30-7d5a-4411-ac8b-bb3868e22f07')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7766fa30-7d5a-4411-ac8b-bb3868e22f07 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7766fa30-7d5a-4411-ac8b-bb3868e22f07');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-fe7c8580-4620-4341-9634-b9bb83aaa6ca\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-fe7c8580-4620-4341-9634-b9bb83aaa6ca')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-fe7c8580-4620-4341-9634-b9bb83aaa6ca button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "exportdf"
            }
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#show the last rows\n",
        "exportdf.tail(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        },
        "id": "VTO8vvwnqpPZ",
        "outputId": "ce13ad35-c42f-4284-a958-c80010897f1a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0      BusinessName          Authority  \\\n",
              "273310        1199  Wild Dough Pizza  Vale of Glamorgan   \n",
              "273321        1217          Y Castel  Vale of Glamorgan   \n",
              "273325        1240       YUM N' BASS  Vale of Glamorgan   \n",
              "\n",
              "                                             AddressLine1 AddressLine2  \\\n",
              "273310                                                NaN          NaN   \n",
              "273321  Aeros Flight Training, Cardiff International A...          NaN   \n",
              "273325                                                NaN          NaN   \n",
              "\n",
              "       AddressLine3       AddressLine4  PostCode         RatingValue  \\\n",
              "273310          NaN                NaN       NaN  AwaitingInspection   \n",
              "273321       Rhoose  Vale Of Glamorgan  CF62 3BD  AwaitingInspection   \n",
              "273325          NaN                NaN       NaN  AwaitingInspection   \n",
              "\n",
              "        NewRatingPending RatingDate             BusinessType        Lat  \\\n",
              "273310             False        NaN           Mobile caterer        NaN   \n",
              "273321             False        NaN  Restaurant/Cafe/Canteen  51.398575   \n",
              "273325             False        NaN           Mobile caterer        NaN   \n",
              "\n",
              "             Lng  ratingYear  pre2022  duplicateBNLL  latNAN  duplicateBNP  \\\n",
              "273310       NaN         NaN    False          False    True         False   \n",
              "273321 -3.339125         NaN    False          False   False         False   \n",
              "273325       NaN         NaN    False          False    True         False   \n",
              "\n",
              "        postCodeSTR  \n",
              "273310        False  \n",
              "273321         True  \n",
              "273325        False  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7e171831-997c-4431-9363-32970ff4af95\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "      <th>duplicateBNLL</th>\n",
              "      <th>latNAN</th>\n",
              "      <th>duplicateBNP</th>\n",
              "      <th>postCodeSTR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>273310</th>\n",
              "      <td>1199</td>\n",
              "      <td>Wild Dough Pizza</td>\n",
              "      <td>Vale of Glamorgan</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>273321</th>\n",
              "      <td>1217</td>\n",
              "      <td>Y Castel</td>\n",
              "      <td>Vale of Glamorgan</td>\n",
              "      <td>Aeros Flight Training, Cardiff International A...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Rhoose</td>\n",
              "      <td>Vale Of Glamorgan</td>\n",
              "      <td>CF62 3BD</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>51.398575</td>\n",
              "      <td>-3.339125</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>273325</th>\n",
              "      <td>1240</td>\n",
              "      <td>YUM N' BASS</td>\n",
              "      <td>Vale of Glamorgan</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7e171831-997c-4431-9363-32970ff4af95')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7e171831-997c-4431-9363-32970ff4af95 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7e171831-997c-4431-9363-32970ff4af95');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-d7f26a16-cf85-4181-899e-e1f9de913761\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-d7f26a16-cf85-4181-899e-e1f9de913761')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-d7f26a16-cf85-4181-899e-e1f9de913761 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "0"
            }
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create True/False columns for duplicates and NaN\n",
        "\n",
        "This time we set the `keep =` parameter to `'last'` rather than `False`. This ensures that we will only mark rows as a duplicate if they are 'older' entries (as the data is sorted by RatingDate so the last entry will be the latest)."
      ],
      "metadata": {
        "id": "2IgtX33bpX_0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#add a T/F column identifying duplicates based on BusinessName/lat/long\n",
        "exportdf['duplicateBNLL'] = exportdf.duplicated(subset=['BusinessName','Lat','Lng'],\n",
        "                            keep = 'last') #Mark duplicates as True except for the last occurrence.\n",
        "\n",
        "#show the first few rows\n",
        "exportdf.head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        },
        "id": "qLi6pDhxoP-h",
        "outputId": "0dc5f697-9c62-4273-9088-c6395d2b1426"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0                             BusinessName  \\\n",
              "150414        1338  Shirley And Millbrook Conservative Club   \n",
              "150673        1834   Woolston And Sholing Conservative Club   \n",
              "120371         596               Corinthians FC Social Club   \n",
              "\n",
              "                      Authority       AddressLine1  AddressLine2 AddressLine3  \\\n",
              "150414              Southampton   22 Anglesea Road   Southampton          NaN   \n",
              "150673              Southampton  74 St. Annes Road   Southampton          NaN   \n",
              "120371  Westmorland and Furness                NaN  Strands Lane          NaN   \n",
              "\n",
              "       AddressLine4  PostCode RatingValue  NewRatingPending  RatingDate  \\\n",
              "150414          NaN  SO15 5QJ           5             False  1995-04-06   \n",
              "150673          NaN  SO19 9FF           5             False  1996-06-12   \n",
              "120371   Milnthorpe   LA7 7AE           3             False  1997-11-28   \n",
              "\n",
              "             BusinessType        Lat       Lng  ratingYear  pre2022  \\\n",
              "150414  Pub/bar/nightclub  50.924719 -1.434945      1995.0     True   \n",
              "150673  Pub/bar/nightclub  50.893923 -1.368832      1996.0     True   \n",
              "120371  Pub/bar/nightclub  54.225384 -2.776882      1997.0     True   \n",
              "\n",
              "        duplicateBNLL  latNAN  duplicateBNP  postCodeSTR  \n",
              "150414          False   False         False         True  \n",
              "150673          False   False         False         True  \n",
              "120371          False   False         False         True  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4ab2ec7c-d76d-4b14-b43f-fa5f612f2a36\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "      <th>duplicateBNLL</th>\n",
              "      <th>latNAN</th>\n",
              "      <th>duplicateBNP</th>\n",
              "      <th>postCodeSTR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>150414</th>\n",
              "      <td>1338</td>\n",
              "      <td>Shirley And Millbrook Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>22 Anglesea Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO15 5QJ</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1995-04-06</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.924719</td>\n",
              "      <td>-1.434945</td>\n",
              "      <td>1995.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150673</th>\n",
              "      <td>1834</td>\n",
              "      <td>Woolston And Sholing Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>74 St. Annes Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO19 9FF</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1996-06-12</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.893923</td>\n",
              "      <td>-1.368832</td>\n",
              "      <td>1996.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>120371</th>\n",
              "      <td>596</td>\n",
              "      <td>Corinthians FC Social Club</td>\n",
              "      <td>Westmorland and Furness</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Strands Lane</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Milnthorpe</td>\n",
              "      <td>LA7 7AE</td>\n",
              "      <td>3</td>\n",
              "      <td>False</td>\n",
              "      <td>1997-11-28</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>54.225384</td>\n",
              "      <td>-2.776882</td>\n",
              "      <td>1997.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4ab2ec7c-d76d-4b14-b43f-fa5f612f2a36')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-4ab2ec7c-d76d-4b14-b43f-fa5f612f2a36 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-4ab2ec7c-d76d-4b14-b43f-fa5f612f2a36');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-9e58d9b8-df54-4503-a697-2921ffcd74be\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-9e58d9b8-df54-4503-a697-2921ffcd74be')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-9e58d9b8-df54-4503-a697-2921ffcd74be button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "exportdf"
            }
          },
          "metadata": {},
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#add a T/F column identifying lat = NaN\n",
        "exportdf['latNAN'] = [math.isnan(i) for i in exportdf['Lat']]\n",
        "\n",
        "#show the first few rows\n",
        "exportdf.head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        },
        "id": "Zxz1KGPDovO_",
        "outputId": "24247a4d-50b5-42fc-e521-1b8ebb027418"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0                             BusinessName  \\\n",
              "150414        1338  Shirley And Millbrook Conservative Club   \n",
              "150673        1834   Woolston And Sholing Conservative Club   \n",
              "120371         596               Corinthians FC Social Club   \n",
              "\n",
              "                      Authority       AddressLine1  AddressLine2 AddressLine3  \\\n",
              "150414              Southampton   22 Anglesea Road   Southampton          NaN   \n",
              "150673              Southampton  74 St. Annes Road   Southampton          NaN   \n",
              "120371  Westmorland and Furness                NaN  Strands Lane          NaN   \n",
              "\n",
              "       AddressLine4  PostCode RatingValue  NewRatingPending  RatingDate  \\\n",
              "150414          NaN  SO15 5QJ           5             False  1995-04-06   \n",
              "150673          NaN  SO19 9FF           5             False  1996-06-12   \n",
              "120371   Milnthorpe   LA7 7AE           3             False  1997-11-28   \n",
              "\n",
              "             BusinessType        Lat       Lng  ratingYear  pre2022  \\\n",
              "150414  Pub/bar/nightclub  50.924719 -1.434945      1995.0     True   \n",
              "150673  Pub/bar/nightclub  50.893923 -1.368832      1996.0     True   \n",
              "120371  Pub/bar/nightclub  54.225384 -2.776882      1997.0     True   \n",
              "\n",
              "        duplicateBNLL  latNAN  duplicateBNP  postCodeSTR  \n",
              "150414          False   False         False         True  \n",
              "150673          False   False         False         True  \n",
              "120371          False   False         False         True  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-26c4a005-c0a9-44f2-82dc-e0637269fcd9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "      <th>duplicateBNLL</th>\n",
              "      <th>latNAN</th>\n",
              "      <th>duplicateBNP</th>\n",
              "      <th>postCodeSTR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>150414</th>\n",
              "      <td>1338</td>\n",
              "      <td>Shirley And Millbrook Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>22 Anglesea Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO15 5QJ</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1995-04-06</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.924719</td>\n",
              "      <td>-1.434945</td>\n",
              "      <td>1995.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150673</th>\n",
              "      <td>1834</td>\n",
              "      <td>Woolston And Sholing Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>74 St. Annes Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO19 9FF</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1996-06-12</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.893923</td>\n",
              "      <td>-1.368832</td>\n",
              "      <td>1996.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>120371</th>\n",
              "      <td>596</td>\n",
              "      <td>Corinthians FC Social Club</td>\n",
              "      <td>Westmorland and Furness</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Strands Lane</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Milnthorpe</td>\n",
              "      <td>LA7 7AE</td>\n",
              "      <td>3</td>\n",
              "      <td>False</td>\n",
              "      <td>1997-11-28</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>54.225384</td>\n",
              "      <td>-2.776882</td>\n",
              "      <td>1997.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-26c4a005-c0a9-44f2-82dc-e0637269fcd9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-26c4a005-c0a9-44f2-82dc-e0637269fcd9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-26c4a005-c0a9-44f2-82dc-e0637269fcd9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c942db91-2ae1-4e08-bfa4-996191047c2b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c942db91-2ae1-4e08-bfa4-996191047c2b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c942db91-2ae1-4e08-bfa4-996191047c2b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "exportdf"
            }
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#add a T/F column identifying duplicates based on BusinessName/PostCode\n",
        "exportdf['duplicateBNP'] = exportdf.duplicated(subset=['BusinessName','PostCode'],\n",
        "                            keep = 'last') #only mark older entries as duplicates\n",
        "#add a T/F column identifying PostCode = NaN\n",
        "exportdf['postCodeSTR'] = [isinstance(i, str) for i in exportdf['PostCode']]\n",
        "\n",
        "#show the first few rows\n",
        "exportdf.head(3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        },
        "id": "gqlkjIwWpBOa",
        "outputId": "d58344a7-b846-455a-f858-95ed359b0f45"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0                             BusinessName  \\\n",
              "150414        1338  Shirley And Millbrook Conservative Club   \n",
              "150673        1834   Woolston And Sholing Conservative Club   \n",
              "120371         596               Corinthians FC Social Club   \n",
              "\n",
              "                      Authority       AddressLine1  AddressLine2 AddressLine3  \\\n",
              "150414              Southampton   22 Anglesea Road   Southampton          NaN   \n",
              "150673              Southampton  74 St. Annes Road   Southampton          NaN   \n",
              "120371  Westmorland and Furness                NaN  Strands Lane          NaN   \n",
              "\n",
              "       AddressLine4  PostCode RatingValue  NewRatingPending  RatingDate  \\\n",
              "150414          NaN  SO15 5QJ           5             False  1995-04-06   \n",
              "150673          NaN  SO19 9FF           5             False  1996-06-12   \n",
              "120371   Milnthorpe   LA7 7AE           3             False  1997-11-28   \n",
              "\n",
              "             BusinessType        Lat       Lng  ratingYear  pre2022  \\\n",
              "150414  Pub/bar/nightclub  50.924719 -1.434945      1995.0     True   \n",
              "150673  Pub/bar/nightclub  50.893923 -1.368832      1996.0     True   \n",
              "120371  Pub/bar/nightclub  54.225384 -2.776882      1997.0     True   \n",
              "\n",
              "        duplicateBNLL  latNAN  duplicateBNP  postCodeSTR  \n",
              "150414          False   False         False         True  \n",
              "150673          False   False         False         True  \n",
              "120371          False   False         False         True  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fd498ef8-ca48-49b7-9978-e1829988b30b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "      <th>duplicateBNLL</th>\n",
              "      <th>latNAN</th>\n",
              "      <th>duplicateBNP</th>\n",
              "      <th>postCodeSTR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>150414</th>\n",
              "      <td>1338</td>\n",
              "      <td>Shirley And Millbrook Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>22 Anglesea Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO15 5QJ</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1995-04-06</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.924719</td>\n",
              "      <td>-1.434945</td>\n",
              "      <td>1995.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>150673</th>\n",
              "      <td>1834</td>\n",
              "      <td>Woolston And Sholing Conservative Club</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>74 St. Annes Road</td>\n",
              "      <td>Southampton</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SO19 9FF</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>1996-06-12</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>50.893923</td>\n",
              "      <td>-1.368832</td>\n",
              "      <td>1996.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>120371</th>\n",
              "      <td>596</td>\n",
              "      <td>Corinthians FC Social Club</td>\n",
              "      <td>Westmorland and Furness</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Strands Lane</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Milnthorpe</td>\n",
              "      <td>LA7 7AE</td>\n",
              "      <td>3</td>\n",
              "      <td>False</td>\n",
              "      <td>1997-11-28</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>54.225384</td>\n",
              "      <td>-2.776882</td>\n",
              "      <td>1997.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fd498ef8-ca48-49b7-9978-e1829988b30b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-fd498ef8-ca48-49b7-9978-e1829988b30b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-fd498ef8-ca48-49b7-9978-e1829988b30b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-902f6e41-d436-424c-af04-9b0273b694f4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-902f6e41-d436-424c-af04-9b0273b694f4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-902f6e41-d436-424c-af04-9b0273b694f4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "exportdf"
            }
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Filtering out the duplicates (but not NaN)\n",
        "\n",
        "We can now use those T/F columns as indices to filter to the subset that we want.\n",
        "\n",
        "For example this will use the T/F list of `postCodeSTR` to only show rows at the positions of True values (i.e. it will only show those with postcode strings):\n",
        "\n",
        "`exportdf[exportdf['postCodeSTR']]`\n",
        "\n",
        "We can then add another index to the end of that to filter it again: `[exportdf['duplicateBNP']` is another T/F column, where True means that row is a duplicate (but not the most recent duplicate)."
      ],
      "metadata": {
        "id": "QZrE5pptpdii"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "exportdf[exportdf['postCodeSTR']][exportdf['duplicateBNP']]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 669
        },
        "id": "xYXlKJpoq1HO",
        "outputId": "3e42faa2-6886-4401-aa47-c1eb4fafcdad"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-79-53d9b496acb3>:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  exportdf[exportdf['postCodeSTR']][exportdf['duplicateBNP']]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Unnamed: 0         BusinessName      Authority         AddressLine1  \\\n",
              "236535        2157      ZULUS PERI PERI  Aberdeen City                  NaN   \n",
              "236168        1430            PARAMOUNT  Aberdeen City                  NaN   \n",
              "235876         838         GOLDEN GRILL  Aberdeen City                  NaN   \n",
              "215168        1047  Carlton Social Club          Leeds          Main Street   \n",
              "235864         814        GEORGE'S ICES  Aberdeen City                  NaN   \n",
              "...            ...                  ...            ...                  ...   \n",
              "184177        2988     Fast Food Centre     Birmingham  140 Bordesley Green   \n",
              "185000        4873           Lee Garden     Birmingham   33-47 Station Road   \n",
              "185102        5278              MAJAJAN     Birmingham              Unit 2A   \n",
              "186480        8484      Taikula chicken     Birmingham      65 Hurst Street   \n",
              "264780        1583         Tyniholidays          Conwy                Ty Ni   \n",
              "\n",
              "                     AddressLine2              AddressLine3   AddressLine4  \\\n",
              "236535  32 Bridge Street Aberdeen                       NaN            NaN   \n",
              "236168    21-25 Bon Accord Street                  Aberdeen            NaN   \n",
              "235876        Silverburn Crescent                  Aberdeen            NaN   \n",
              "215168                    Carlton                  Rothwell          Leeds   \n",
              "235864                    Stt/4/3  Registration No J930 Gck            NaN   \n",
              "...                           ...                       ...            ...   \n",
              "184177                   Nechells                Birmingham            NaN   \n",
              "185000                  Erdington                Birmingham            NaN   \n",
              "185102     Cobham Business Centre               Cobham Road     Birmingham   \n",
              "186480                 Birmingham                       NaN            NaN   \n",
              "264780                        NaN                  Trofarth  Betws Yn Rhos   \n",
              "\n",
              "        PostCode          RatingValue  NewRatingPending  RatingDate  \\\n",
              "236535  AB11 6JN  Awaiting Inspection             False  1999-04-01   \n",
              "236168  AB11 6EA  Awaiting Inspection             False  1999-04-01   \n",
              "235876  AB23 8EW  Awaiting Inspection             False  1999-04-01   \n",
              "215168   WF3 3RW                    5             False  2001-03-30   \n",
              "235864  AB23 8NT                 Pass             False  2008-06-02   \n",
              "...          ...                  ...               ...         ...   \n",
              "184177    B9 4SU   AwaitingInspection             False         NaN   \n",
              "185000   B23 6UE   AwaitingInspection             False         NaN   \n",
              "185102    B9 4UP   AwaitingInspection             False         NaN   \n",
              "186480    B5 4TE   AwaitingInspection             False         NaN   \n",
              "264780  LL22 8BE   AwaitingInspection             False         NaN   \n",
              "\n",
              "                   BusinessType        Lat       Lng  ratingYear  pre2022  \\\n",
              "236535  Restaurant/Cafe/Canteen  57.144457 -2.101305      1999.0     True   \n",
              "236168        Pub/bar/nightclub  57.143660 -2.105679      1999.0     True   \n",
              "235876   Takeaway/sandwich shop  57.187503 -2.095280      1999.0     True   \n",
              "215168        Pub/bar/nightclub  53.740684 -1.488365      2001.0     True   \n",
              "235864           Mobile caterer        NaN       NaN      2008.0     True   \n",
              "...                         ...        ...       ...         ...      ...   \n",
              "184177  Restaurant/Cafe/Canteen  52.477075 -1.855753         NaN    False   \n",
              "185000   Takeaway/sandwich shop  52.528019 -1.840093         NaN    False   \n",
              "185102  Restaurant/Cafe/Canteen  52.480626 -1.855565         NaN    False   \n",
              "186480  Restaurant/Cafe/Canteen  52.474197 -1.897146         NaN    False   \n",
              "264780   Takeaway/sandwich shop        NaN       NaN         NaN    False   \n",
              "\n",
              "        duplicateBNLL  latNAN  duplicateBNP  postCodeSTR  \n",
              "236535          False   False          True         True  \n",
              "236168          False   False          True         True  \n",
              "235876          False   False          True         True  \n",
              "215168          False   False          True         True  \n",
              "235864           True    True          True         True  \n",
              "...               ...     ...           ...          ...  \n",
              "184177          False   False          True         True  \n",
              "185000          False   False          True         True  \n",
              "185102          False   False          True         True  \n",
              "186480          False   False          True         True  \n",
              "264780           True    True          True         True  \n",
              "\n",
              "[461 rows x 20 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-fbed45af-37f4-41b6-8937-d45e6b62ca45\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>BusinessName</th>\n",
              "      <th>Authority</th>\n",
              "      <th>AddressLine1</th>\n",
              "      <th>AddressLine2</th>\n",
              "      <th>AddressLine3</th>\n",
              "      <th>AddressLine4</th>\n",
              "      <th>PostCode</th>\n",
              "      <th>RatingValue</th>\n",
              "      <th>NewRatingPending</th>\n",
              "      <th>RatingDate</th>\n",
              "      <th>BusinessType</th>\n",
              "      <th>Lat</th>\n",
              "      <th>Lng</th>\n",
              "      <th>ratingYear</th>\n",
              "      <th>pre2022</th>\n",
              "      <th>duplicateBNLL</th>\n",
              "      <th>latNAN</th>\n",
              "      <th>duplicateBNP</th>\n",
              "      <th>postCodeSTR</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>236535</th>\n",
              "      <td>2157</td>\n",
              "      <td>ZULUS PERI PERI</td>\n",
              "      <td>Aberdeen City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>32 Bridge Street Aberdeen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AB11 6JN</td>\n",
              "      <td>Awaiting Inspection</td>\n",
              "      <td>False</td>\n",
              "      <td>1999-04-01</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>57.144457</td>\n",
              "      <td>-2.101305</td>\n",
              "      <td>1999.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>236168</th>\n",
              "      <td>1430</td>\n",
              "      <td>PARAMOUNT</td>\n",
              "      <td>Aberdeen City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>21-25 Bon Accord Street</td>\n",
              "      <td>Aberdeen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AB11 6EA</td>\n",
              "      <td>Awaiting Inspection</td>\n",
              "      <td>False</td>\n",
              "      <td>1999-04-01</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>57.143660</td>\n",
              "      <td>-2.105679</td>\n",
              "      <td>1999.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>235876</th>\n",
              "      <td>838</td>\n",
              "      <td>GOLDEN GRILL</td>\n",
              "      <td>Aberdeen City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Silverburn Crescent</td>\n",
              "      <td>Aberdeen</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AB23 8EW</td>\n",
              "      <td>Awaiting Inspection</td>\n",
              "      <td>False</td>\n",
              "      <td>1999-04-01</td>\n",
              "      <td>Takeaway/sandwich shop</td>\n",
              "      <td>57.187503</td>\n",
              "      <td>-2.095280</td>\n",
              "      <td>1999.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>215168</th>\n",
              "      <td>1047</td>\n",
              "      <td>Carlton Social Club</td>\n",
              "      <td>Leeds</td>\n",
              "      <td>Main Street</td>\n",
              "      <td>Carlton</td>\n",
              "      <td>Rothwell</td>\n",
              "      <td>Leeds</td>\n",
              "      <td>WF3 3RW</td>\n",
              "      <td>5</td>\n",
              "      <td>False</td>\n",
              "      <td>2001-03-30</td>\n",
              "      <td>Pub/bar/nightclub</td>\n",
              "      <td>53.740684</td>\n",
              "      <td>-1.488365</td>\n",
              "      <td>2001.0</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>235864</th>\n",
              "      <td>814</td>\n",
              "      <td>GEORGE'S ICES</td>\n",
              "      <td>Aberdeen City</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Stt/4/3</td>\n",
              "      <td>Registration No J930 Gck</td>\n",
              "      <td>NaN</td>\n",
              "      <td>AB23 8NT</td>\n",
              "      <td>Pass</td>\n",
              "      <td>False</td>\n",
              "      <td>2008-06-02</td>\n",
              "      <td>Mobile caterer</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2008.0</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>184177</th>\n",
              "      <td>2988</td>\n",
              "      <td>Fast Food Centre</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>140 Bordesley Green</td>\n",
              "      <td>Nechells</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>NaN</td>\n",
              "      <td>B9 4SU</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>52.477075</td>\n",
              "      <td>-1.855753</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>185000</th>\n",
              "      <td>4873</td>\n",
              "      <td>Lee Garden</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>33-47 Station Road</td>\n",
              "      <td>Erdington</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>NaN</td>\n",
              "      <td>B23 6UE</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Takeaway/sandwich shop</td>\n",
              "      <td>52.528019</td>\n",
              "      <td>-1.840093</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>185102</th>\n",
              "      <td>5278</td>\n",
              "      <td>MAJAJAN</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>Unit 2A</td>\n",
              "      <td>Cobham Business Centre</td>\n",
              "      <td>Cobham Road</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>B9 4UP</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>52.480626</td>\n",
              "      <td>-1.855565</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>186480</th>\n",
              "      <td>8484</td>\n",
              "      <td>Taikula chicken</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>65 Hurst Street</td>\n",
              "      <td>Birmingham</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>B5 4TE</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Restaurant/Cafe/Canteen</td>\n",
              "      <td>52.474197</td>\n",
              "      <td>-1.897146</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>264780</th>\n",
              "      <td>1583</td>\n",
              "      <td>Tyniholidays</td>\n",
              "      <td>Conwy</td>\n",
              "      <td>Ty Ni</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Trofarth</td>\n",
              "      <td>Betws Yn Rhos</td>\n",
              "      <td>LL22 8BE</td>\n",
              "      <td>AwaitingInspection</td>\n",
              "      <td>False</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Takeaway/sandwich shop</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>461 rows × 20 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-fbed45af-37f4-41b6-8937-d45e6b62ca45')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-fbed45af-37f4-41b6-8937-d45e6b62ca45 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-fbed45af-37f4-41b6-8937-d45e6b62ca45');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-730e5021-c109-4ab0-9c1b-355e05324166\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-730e5021-c109-4ab0-9c1b-355e05324166')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-730e5021-c109-4ab0-9c1b-355e05324166 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "repr_error": "0"
            }
          },
          "metadata": {},
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "There are around 461 establishments that fit this pattern. If we removed these from the data frame, how many should there be left?"
      ],
      "metadata": {
        "id": "Qd73g6IitC1t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "len(exportdf) - 461"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3ymt0maIucBk",
        "outputId": "855a1821-ab55-4d5e-b0f7-c8439a1cac4f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "273359"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can add a minus sign before a T/F list to reverse them (Trues become Falses and vice versa) and filter the other way, only showing records where duplicate is `False`:\n",
        "\n",
        "`[-exportdf['duplicateBNP']`\n",
        "\n",
        "But if we make both filters negative we don't get the 273,359 rows - we just get those which are not postcode strings and not duplicates"
      ],
      "metadata": {
        "id": "lobl_f21ukgm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#filter where postCodeSTR is True (the postcode is a string) and duplicate is false\n",
        "#try one order of the indexing\n",
        "print(len(exportdf[-exportdf['postCodeSTR']][-exportdf['duplicateBNP']]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5QOPuGV3tdLi",
        "outputId": "3e6743a5-261f-4d3b-820d-cbc4c7e9ec51"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "25198\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-77-7e9d5945a0c4>:3: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  print(len(exportdf[-exportdf['postCodeSTR']][-exportdf['duplicateBNP']]))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "So we need to map out what we want:\n",
        "\n",
        "* duplicateBNP = `False`\n",
        "* duplicateBNP = `True` but postCodeSTR = `False`\n",
        "\n",
        "Let's check that if we did both and added the results together they would add up to the expected number of rows."
      ],
      "metadata": {
        "id": "cDfxqhJ319Fo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#add the results of both filters\n",
        "len(exportdf[-exportdf['duplicateBNP']]) + len(exportdf[exportdf['duplicateBNP']][-exportdf['postCodeSTR']])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "llauSpGjvc8E",
        "outputId": "47fb770d-98d4-48de-b89b-fb18783be1eb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-85-414523b58638>:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  len(exportdf[-exportdf['duplicateBNP']]) + len(exportdf[exportdf['duplicateBNP']][-exportdf['postCodeSTR']])\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "273359"
            ]
          },
          "metadata": {},
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "That adds up, so we now concatenate."
      ],
      "metadata": {
        "id": "0SFQaHld3dLM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#concatenate the results of both filters\n",
        "deduped_BNP = pd.concat([exportdf[-exportdf['duplicateBNP']],\n",
        "              exportdf[exportdf['duplicateBNP']][-exportdf['postCodeSTR']]])\n",
        "\n",
        "len(deduped_BNP)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FMriijbP20vp",
        "outputId": "51f89904-1457-40e8-a1a8-f02d1a8850f1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-89-22de062d078d>:3: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  exportdf[exportdf['duplicateBNP']][-exportdf['postCodeSTR']]])\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "273359"
            ]
          },
          "metadata": {},
          "execution_count": 89
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Filter again to deduplicate on lat/long\n",
        "\n",
        "This time we want:\n",
        "\n",
        "* duplicateBNLL = `False` OR\n",
        "* duplicateBNLL = `True` but latNAN = `True` (so only duplicates where there is a Lat)\n",
        "\n",
        "We count how many fit the latter category in the dataset before we removed the duplicate postcodes:"
      ],
      "metadata": {
        "id": "iawkP6TS3lwq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#duplicateBNLL = True AND latNAN = False\n",
        "len(exportdf[exportdf['duplicateBNLL']][-exportdf['latNAN']])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5ypNoPPz4tCJ",
        "outputId": "7b138994-5fbd-4e8a-b096-5f4b561bb900"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-96-68c09d99b2f1>:1: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  len(exportdf[exportdf['duplicateBNLL']][-exportdf['latNAN']])\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "279"
            ]
          },
          "metadata": {},
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And how many now we've removed those postcode duplicates"
      ],
      "metadata": {
        "id": "FKCz5HXo44Id"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#duplicateBNLL = True AND latNAN = False\n",
        "len(deduped_BNP[deduped_BNP['duplicateBNLL']][-deduped_BNP['latNAN']])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XcC2oCoR3uwp",
        "outputId": "51c4fffb-d350-43aa-fc5d-8a358858bcef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-99-95a7f6e5dd7b>:2: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  len(deduped_BNP[deduped_BNP['duplicateBNLL']][-deduped_BNP['latNAN']])\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "18"
            ]
          },
          "metadata": {},
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#concatenate the results of two filters\n",
        "#duplicateBNLL = False\n",
        "deduped_BNPLL = pd.concat([deduped_BNP[-deduped_BNP['duplicateBNLL']],\n",
        "              deduped_BNP[deduped_BNP['duplicateBNLL']][deduped_BNP['latNAN']]])\n",
        "              #duplicateBNLL = True AND latNAN = True, i.e. it's only a duplicate name, not location\n",
        "\n",
        "len(deduped_BNPLL)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_p4tg62V5APk",
        "outputId": "3a210b43-96e6-4139-8ee2-bb479fcd6233"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-100-1db575291be7>:4: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  deduped_BNP[deduped_BNP['duplicateBNLL']][deduped_BNP['latNAN']]])\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "273341"
            ]
          },
          "metadata": {},
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Export df with duplicate businesses removed"
      ],
      "metadata": {
        "id": "2E4_4cCF6oQQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "deduped_BNPLL.to_csv('deduped_BNPLL.csv')\n",
        "files.download('deduped_BNPLL.csv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "32LGOUnx6pkP",
        "outputId": "478fe849-ff8b-4da4-d91e-362e58c8de37"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_135c3dee-22f5-4a80-99c8-af5d8d80fabc\", \"deduped_BNPLL.csv\", 50092181)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Pivot deduped: inspections pre-2022 by authority\n",
        "\n",
        "We can now repeat the analysis from earlier."
      ],
      "metadata": {
        "id": "J_1pK6MrKTYp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pivot_pre22_la = deduped_BNPLL.pivot_table(index=\"Authority\",\n",
        "                        values=\"BusinessName\",\n",
        "                        columns=\"pre2022\",\n",
        "                        margins=True,\n",
        "                        aggfunc=\"count\").fillna(0).astype(int)\n",
        "\n",
        "pivot_pre22_la\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 455
        },
        "id": "l6JDt91pKa9i",
        "outputId": "81cff47d-36bf-4a6e-cf2b-5cb08b9cc123"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pre2022         False   True     All\n",
              "Authority                           \n",
              "Aberdeen City     390    679    1069\n",
              "Aberdeenshire     366    297     663\n",
              "Adur              195      5     200\n",
              "Amber Valley      390    181     571\n",
              "Anglesey          293     57     350\n",
              "...               ...    ...     ...\n",
              "Wychavon          437     37     474\n",
              "Wyre              405     62     467\n",
              "Wyre Forest       321     38     359\n",
              "York              789    228    1017\n",
              "All            219856  53482  273338\n",
              "\n",
              "[364 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b78bc055-3384-4bfb-a0f9-273f706f4fe9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>pre2022</th>\n",
              "      <th>False</th>\n",
              "      <th>True</th>\n",
              "      <th>All</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Authority</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Aberdeen City</th>\n",
              "      <td>390</td>\n",
              "      <td>679</td>\n",
              "      <td>1069</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aberdeenshire</th>\n",
              "      <td>366</td>\n",
              "      <td>297</td>\n",
              "      <td>663</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Adur</th>\n",
              "      <td>195</td>\n",
              "      <td>5</td>\n",
              "      <td>200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Amber Valley</th>\n",
              "      <td>390</td>\n",
              "      <td>181</td>\n",
              "      <td>571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Anglesey</th>\n",
              "      <td>293</td>\n",
              "      <td>57</td>\n",
              "      <td>350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wychavon</th>\n",
              "      <td>437</td>\n",
              "      <td>37</td>\n",
              "      <td>474</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre</th>\n",
              "      <td>405</td>\n",
              "      <td>62</td>\n",
              "      <td>467</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre Forest</th>\n",
              "      <td>321</td>\n",
              "      <td>38</td>\n",
              "      <td>359</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>York</th>\n",
              "      <td>789</td>\n",
              "      <td>228</td>\n",
              "      <td>1017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>All</th>\n",
              "      <td>219856</td>\n",
              "      <td>53482</td>\n",
              "      <td>273338</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>364 rows × 3 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b78bc055-3384-4bfb-a0f9-273f706f4fe9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b78bc055-3384-4bfb-a0f9-273f706f4fe9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b78bc055-3384-4bfb-a0f9-273f706f4fe9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-ffcabe6f-c714-46c9-9103-1b5583801a86\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-ffcabe6f-c714-46c9-9103-1b5583801a86')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-ffcabe6f-c714-46c9-9103-1b5583801a86 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "pivot_pre22_la",
              "summary": "{\n  \"name\": \"pivot_pre22_la\",\n  \"rows\": 364,\n  \"fields\": [\n    {\n      \"column\": \"Authority\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 364,\n        \"samples\": [\n          \"Mid Sussex\",\n          \"Bracknell Forest\",\n          \"Barnet\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": false,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11500,\n        \"min\": 4,\n        \"max\": 219856,\n        \"num_unique_values\": 311,\n        \"samples\": [\n          1520,\n          497,\n          522\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": true,\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2803,\n        \"min\": 0,\n        \"max\": 53482,\n        \"num_unique_values\": 209,\n        \"samples\": [\n          73,\n          408,\n          85\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"All\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 14300,\n        \"min\": 4,\n        \"max\": 273338,\n        \"num_unique_values\": 315,\n        \"samples\": [\n          1075,\n          286,\n          1385\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Rename the columns\n",
        "\n",
        "Let's make the column names clearer"
      ],
      "metadata": {
        "id": "ntl3nRFRNBDc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pivot_pre22_la = pivot_pre22_la.rename(columns={0:'post22',1:'pre22'})\n",
        "pivot_pre22_la\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 455
        },
        "id": "t9IOqCGwMc41",
        "outputId": "ccb2cdb7-cd96-4b36-de39-51b16c6a3662"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pre2022        post22  pre22     All\n",
              "Authority                           \n",
              "Aberdeen City     390    679    1069\n",
              "Aberdeenshire     366    297     663\n",
              "Adur              195      5     200\n",
              "Amber Valley      390    181     571\n",
              "Anglesey          293     57     350\n",
              "...               ...    ...     ...\n",
              "Wychavon          437     37     474\n",
              "Wyre              405     62     467\n",
              "Wyre Forest       321     38     359\n",
              "York              789    228    1017\n",
              "All            219856  53482  273338\n",
              "\n",
              "[364 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6ce50e4e-2d6a-4386-be82-35baffeb4906\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>pre2022</th>\n",
              "      <th>post22</th>\n",
              "      <th>pre22</th>\n",
              "      <th>All</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Authority</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Aberdeen City</th>\n",
              "      <td>390</td>\n",
              "      <td>679</td>\n",
              "      <td>1069</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aberdeenshire</th>\n",
              "      <td>366</td>\n",
              "      <td>297</td>\n",
              "      <td>663</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Adur</th>\n",
              "      <td>195</td>\n",
              "      <td>5</td>\n",
              "      <td>200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Amber Valley</th>\n",
              "      <td>390</td>\n",
              "      <td>181</td>\n",
              "      <td>571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Anglesey</th>\n",
              "      <td>293</td>\n",
              "      <td>57</td>\n",
              "      <td>350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wychavon</th>\n",
              "      <td>437</td>\n",
              "      <td>37</td>\n",
              "      <td>474</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre</th>\n",
              "      <td>405</td>\n",
              "      <td>62</td>\n",
              "      <td>467</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre Forest</th>\n",
              "      <td>321</td>\n",
              "      <td>38</td>\n",
              "      <td>359</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>York</th>\n",
              "      <td>789</td>\n",
              "      <td>228</td>\n",
              "      <td>1017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>All</th>\n",
              "      <td>219856</td>\n",
              "      <td>53482</td>\n",
              "      <td>273338</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>364 rows × 3 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6ce50e4e-2d6a-4386-be82-35baffeb4906')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6ce50e4e-2d6a-4386-be82-35baffeb4906 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6ce50e4e-2d6a-4386-be82-35baffeb4906');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-0922c728-608b-4e5e-bde4-aa5785fd322f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0922c728-608b-4e5e-bde4-aa5785fd322f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-0922c728-608b-4e5e-bde4-aa5785fd322f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pivot_pre22_la\",\n  \"rows\": 364,\n  \"fields\": [\n    {\n      \"column\": \"Authority\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 364,\n        \"samples\": [\n          \"Mid Sussex\",\n          \"Bracknell Forest\",\n          \"Barnet\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"post22\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 11500,\n        \"min\": 4,\n        \"max\": 219856,\n        \"num_unique_values\": 311,\n        \"samples\": [\n          1520,\n          497,\n          522\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"pre22\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2803,\n        \"min\": 0,\n        \"max\": 53482,\n        \"num_unique_values\": 209,\n        \"samples\": [\n          73,\n          408,\n          85\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"All\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 14300,\n        \"min\": 4,\n        \"max\": 273338,\n        \"num_unique_values\": 315,\n        \"samples\": [\n          1075,\n          286,\n          1385\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Pivot deduped: rating by authority"
      ],
      "metadata": {
        "id": "jGEuI7OzNSUZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#store in a dataframe\n",
        "pivot_rating_la = deduped_BNPLL.pivot_table(index=\"Authority\",\n",
        "                        values=\"BusinessName\",\n",
        "                        columns=\"RatingValue\",\n",
        "                        margins=True,\n",
        "                        aggfunc=\"count\").fillna(0)\n",
        "pivot_rating_la"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 527
        },
        "id": "ygCHbbh5NTLC",
        "outputId": "2f2d2f0f-8a8a-4bd8-9002-37aaac7c4a75"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RatingValue        0       1       2        3        4         5  \\\n",
              "Authority                                                          \n",
              "Aberdeen City    0.0     0.0     0.0      0.0      0.0       0.0   \n",
              "Aberdeenshire    0.0     0.0     0.0      0.0      0.0       0.0   \n",
              "Adur             0.0     4.0     0.0     10.0     21.0     162.0   \n",
              "Amber Valley     0.0     4.0     2.0     41.0     74.0     386.0   \n",
              "Anglesey         0.0     3.0     4.0     13.0     37.0     292.0   \n",
              "...              ...     ...     ...      ...      ...       ...   \n",
              "Wychavon         0.0     4.0     6.0     31.0     81.0     303.0   \n",
              "Wyre             0.0    11.0    13.0     21.0     66.0     331.0   \n",
              "Wyre Forest      0.0     1.0     5.0     35.0     64.0     209.0   \n",
              "York             0.0    14.0    24.0     41.0    176.0     711.0   \n",
              "All            502.0  3574.0  4341.0  18640.0  38303.0  165364.0   \n",
              "\n",
              "RatingValue    Awaiting Inspection  AwaitingInspection  AwaitingPublication  \\\n",
              "Authority                                                                     \n",
              "Aberdeen City                 95.0                 0.0                  0.0   \n",
              "Aberdeenshire                101.0                 0.0                  0.0   \n",
              "Adur                           0.0                 3.0                  0.0   \n",
              "Amber Valley                   0.0                64.0                  0.0   \n",
              "Anglesey                       0.0                 1.0                  0.0   \n",
              "...                            ...                 ...                  ...   \n",
              "Wychavon                       0.0                49.0                  0.0   \n",
              "Wyre                           0.0                25.0                  0.0   \n",
              "Wyre Forest                    0.0                45.0                  0.0   \n",
              "York                           0.0                51.0                  0.0   \n",
              "All                         3424.0             18607.0                 24.0   \n",
              "\n",
              "RatingValue    Improvement Required     Pass  Pass and Eat Safe     All  \n",
              "Authority                                                                \n",
              "Aberdeen City                 217.0    756.0                1.0    1069  \n",
              "Aberdeenshire                  58.0    502.0                2.0     663  \n",
              "Adur                            0.0      0.0                0.0     200  \n",
              "Amber Valley                    0.0      0.0                0.0     571  \n",
              "Anglesey                        0.0      0.0                0.0     350  \n",
              "...                             ...      ...                ...     ...  \n",
              "Wychavon                        0.0      0.0                0.0     474  \n",
              "Wyre                            0.0      0.0                0.0     467  \n",
              "Wyre Forest                     0.0      0.0                0.0     359  \n",
              "York                            0.0      0.0                0.0    1017  \n",
              "All                          1719.0  18673.0              167.0  273338  \n",
              "\n",
              "[364 rows x 13 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-577b83cf-70e1-418f-adfd-c5afd41b587d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>RatingValue</th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "      <th>5</th>\n",
              "      <th>Awaiting Inspection</th>\n",
              "      <th>AwaitingInspection</th>\n",
              "      <th>AwaitingPublication</th>\n",
              "      <th>Improvement Required</th>\n",
              "      <th>Pass</th>\n",
              "      <th>Pass and Eat Safe</th>\n",
              "      <th>All</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Authority</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Aberdeen City</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>95.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>217.0</td>\n",
              "      <td>756.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1069</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aberdeenshire</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>101.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>502.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>663</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Adur</th>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>162.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Amber Valley</th>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>41.0</td>\n",
              "      <td>74.0</td>\n",
              "      <td>386.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>64.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Anglesey</th>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>37.0</td>\n",
              "      <td>292.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>350</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wychavon</th>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>31.0</td>\n",
              "      <td>81.0</td>\n",
              "      <td>303.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>49.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>474</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre</th>\n",
              "      <td>0.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>13.0</td>\n",
              "      <td>21.0</td>\n",
              "      <td>66.0</td>\n",
              "      <td>331.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>25.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>467</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Wyre Forest</th>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>35.0</td>\n",
              "      <td>64.0</td>\n",
              "      <td>209.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>45.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>359</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>York</th>\n",
              "      <td>0.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>24.0</td>\n",
              "      <td>41.0</td>\n",
              "      <td>176.0</td>\n",
              "      <td>711.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>51.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>All</th>\n",
              "      <td>502.0</td>\n",
              "      <td>3574.0</td>\n",
              "      <td>4341.0</td>\n",
              "      <td>18640.0</td>\n",
              "      <td>38303.0</td>\n",
              "      <td>165364.0</td>\n",
              "      <td>3424.0</td>\n",
              "      <td>18607.0</td>\n",
              "      <td>24.0</td>\n",
              "      <td>1719.0</td>\n",
              "      <td>18673.0</td>\n",
              "      <td>167.0</td>\n",
              "      <td>273338</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>364 rows × 13 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-577b83cf-70e1-418f-adfd-c5afd41b587d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-577b83cf-70e1-418f-adfd-c5afd41b587d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-577b83cf-70e1-418f-adfd-c5afd41b587d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a25e83af-c777-4023-b5b9-56be2bedfd3f\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a25e83af-c777-4023-b5b9-56be2bedfd3f')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a25e83af-c777-4023-b5b9-56be2bedfd3f button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "pivot_rating_la",
              "summary": "{\n  \"name\": \"pivot_rating_la\",\n  \"rows\": 364,\n  \"fields\": [\n    {\n      \"column\": \"Authority\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 364,\n        \"samples\": [\n          \"Mid Sussex\",\n          \"Bracknell Forest\",\n          \"Barnet\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"0\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 26.655165836486425,\n        \"min\": 0.0,\n        \"max\": 502.0,\n        \"num_unique_values\": 19,\n        \"samples\": [\n          0.0,\n          3.0,\n          10.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"1\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 187.43517617260204,\n        \"min\": 0.0,\n        \"max\": 3574.0,\n        \"num_unique_values\": 47,\n        \"samples\": [\n          34.0,\n          17.0,\n          16.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"2\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 227.65005034981365,\n        \"min\": 0.0,\n        \"max\": 4341.0,\n        \"num_unique_values\": 54,\n        \"samples\": [\n          14.0,\n          26.0,\n          21.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"3\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 976.0949273703948,\n        \"min\": 0.0,\n        \"max\": 18640.0,\n        \"num_unique_values\": 127,\n        \"samples\": [\n          8.0,\n          127.0,\n          101.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"4\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2004.7228082723673,\n        \"min\": 0.0,\n        \"max\": 38303.0,\n        \"num_unique_values\": 178,\n        \"samples\": [\n          311.0,\n          449.0,\n          160.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"5\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 8651.765949422856,\n        \"min\": 0.0,\n        \"max\": 165364.0,\n        \"num_unique_values\": 269,\n        \"samples\": [\n          195.0,\n          627.0,\n          858.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Awaiting Inspection\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 193.51720118235642,\n        \"min\": 0.0,\n        \"max\": 3424.0,\n        \"num_unique_values\": 28,\n        \"samples\": [\n          5.0,\n          37.0,\n          51.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AwaitingInspection\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 976.8595739517854,\n        \"min\": 0.0,\n        \"max\": 18607.0,\n        \"num_unique_values\": 128,\n        \"samples\": [\n          44.0,\n          22.0,\n          59.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AwaitingPublication\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.2895749931677736,\n        \"min\": 0.0,\n        \"max\": 24.0,\n        \"num_unique_values\": 4,\n        \"samples\": [\n          1.0,\n          24.0,\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Improvement Required\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 93.8544958657381,\n        \"min\": 0.0,\n        \"max\": 1719.0,\n        \"num_unique_values\": 27,\n        \"samples\": [\n          65.0,\n          9.0,\n          12.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Pass\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1003.0290288001983,\n        \"min\": 0.0,\n        \"max\": 18673.0,\n        \"num_unique_values\": 34,\n        \"samples\": [\n          1580.0,\n          278.0,\n          44.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Pass and Eat Safe\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 9.218129142238993,\n        \"min\": 0.0,\n        \"max\": 167.0,\n        \"num_unique_values\": 14,\n        \"samples\": [\n          26.0,\n          23.0,\n          1.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"All\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 14300,\n        \"min\": 4,\n        \"max\": 273338,\n        \"num_unique_values\": 315,\n        \"samples\": [\n          1075,\n          286,\n          1385\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 126
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Export"
      ],
      "metadata": {
        "id": "_8uN_ZKpNRlF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#export as a CSV\n",
        "pivot_rating_la.to_csv('pivot_rating_la.csv')\n",
        "pivot_pre22_la.to_csv('pivot_pre22_la.csv')\n",
        "#download\n",
        "files.download('pivot_pre22_la.csv')\n",
        "files.download('pivot_rating_la.csv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "id": "Foku7Me9K2SN",
        "outputId": "d0212bbb-3b08-479b-d07c-12bc721ce327"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_cf66ccd2-3979-40e9-bfba-419bd11a2504\", \"pivot_pre22_la.csv\", 8913)"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_0fa07624-7010-419e-a694-bc552df212b1\", \"pivot_rating_la.csv\", 26065)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Some previous data checking\n",
        "\n",
        "There were some duplicates in a previous run of this notebook. This is how we saw if there's a pattern before we removed them."
      ],
      "metadata": {
        "id": "aONhNVldlNc2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#import the exported data\n",
        "exporturl = \"https://docs.google.com/spreadsheets/d/e/2PACX-1vQCqMuFBH7EA88vdfYXhoQ42wUT-BIlVYyMhJmIkwSYaXvEE6A4IKN_WM3cztF2tFVUIx0ImZDo63RE/pub?gid=2131543290&single=true&output=csv\"\n",
        "exportdf = pd.read_csv(exporturl)\n",
        "#check how many rows\n",
        "len(exportdf)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5DL-IYfQlRSN",
        "outputId": "333b5c30-6394-4152-bbb3-248a9e10d8e3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "272106"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#remove duplicates\n",
        "deduped = exportdf.drop_duplicates()\n",
        "#how many does that leave\n",
        "len(deduped)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pws7xCcllhSW",
        "outputId": "f90bbfd9-297d-40ab-c279-c5028485bd02"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "257783"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#create a list of True/False values indicating whether a row is a duplicate\n",
        "dupes = exportdf.duplicated()\n",
        "#Get a count of T/F\n",
        "dupes.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kfso3nkLlv31",
        "outputId": "3b737938-c41e-49cf-8255-51dde44a9066"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False    257783\n",
              "True      14323\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Use that list to filter to duplicate entries, and get a count of the authorities\n",
        "exportdf['Authority'][dupes].value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a6okf5RnmAAb",
        "outputId": "82714871-504b-4a52-cf6c-d94b464c1090"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Authority\n",
              "Cardiff              1463\n",
              "Swansea              1083\n",
              "Powys                 978\n",
              "Rhondda Cynon Taf     943\n",
              "Pembrokeshire         911\n",
              "Carmarthenshire       902\n",
              "Gwynedd               900\n",
              "Conwy                 729\n",
              "Caerphilly            619\n",
              "Neath Port Talbot     604\n",
              "Newport               578\n",
              "Bridgend              562\n",
              "Vale of Glamorgan     523\n",
              "Flintshire            514\n",
              "Wrexham               491\n",
              "Denbighshire          464\n",
              "Monmouthshire         447\n",
              "Ceredigion            399\n",
              "Anglesey              350\n",
              "Torfaen               323\n",
              "Blaenau Gwent         272\n",
              "Merthyr Tydfil        268\n",
              "Name: count, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(alldf.drop_duplicates())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e2mOU7voonFj",
        "outputId": "be537451-f12b-47e3-c68e-cbfd9aab2582"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2699"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    }
  ]
}