week1_session3_matplotlib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# AI in Medicine: Data Science - Basics II"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Programming: Data Visualisation using *matplotlib* and *Seaborn*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- **Instructor**: Moritz Seiler, AG Ritter, Charité - Universitätsmedizin Berlin (moritz.seiler@charite.de) <br>\n",
    "- **Target audience**: Medical students from Charité<br>\n",
    "- **Course date**: January 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**First, please make sure, that you're using the Python 3.6 or later kernel!**<br> You can check that in the top right corner of your browser window. If you're using a different Version go to the tab *Kernel* --> *Change Kernel* and select *Python 3.6*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Aim of this session."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial introduces **data visualisation** in Python using ***matplotlib***. You'll learn how to create basic statistical plots using RKI's Covid-19 dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Learning goals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data visualisation\n",
    "- Definition\n",
    "- Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "1. Data set<br>\n",
    "2. matplotlib\n",
    "    - Installation & Import\n",
    "    - Basics in *matplotlib*\n",
    "    - Workflow in *matplotlib*\n",
    "    - Charting types in *matplotlib*\n",
    "    - Examples \n",
    "    - Save plot\n",
    "    - Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Data visualisation<br>\n",
    "https://www.cs.unm.edu/~pgk/IVCDs14/minitufte.pdf<br>\n",
    "https://www.oreilly.com/library/view/practical-statistics-for/9781491952955/ch01.html\n",
    "- matplotlib<br>\n",
    "https://matplotlib.org/tutorials/introductory/usage.html#sphx-glr-tutorials-introductory-usage-py<br>\n",
    "https://matplotlib.org/tutorials/introductory/sample_plots.html#sphx-glr-tutorials-introductory-sample-plots-py\n",
    "- RKI COVID-19 dataset for Berlin<br>\n",
    "https://opendata.arcgis.com/datasets/dd4580c810204019a7b8eb3e0b329dd6_0.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Theory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See accompanying slide deck."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Pratical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll again work with the Robert-Koch-Institut (RKI) Covid-19 dataset that you already know from the previous tutorials.\n",
    "\n",
    "*Source: Robert Koch-Institut (RKI), dl-de/by-2-0*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's first read in the dataset as you've learned it in the previous tutorial using the `read_csv()` function in *pandas*. Furthermore, we transform the entries in the column *Meldedatum* from string into a datetime format with the *pandas* function `to_datetime()` which allows you e.g. to do calculations with those dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the Pandas and NumPy library\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Read in the RKI Covid-19 data set\n",
    "covid19 = pd.read_csv('https://opendata.arcgis.com/datasets/dd4580c810204019a7b8eb3e0b329dd6_0.csv')\n",
    "\n",
    "# Transform string into datetime format\n",
    "covid19['Meldedatum'] = pd.to_datetime(covid19['Meldedatum'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "covid19.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "covid19.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### 5.2 matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matplotlib is a Python 2D plotting library which produces publication-quality figures. It was designed to serve two purposes:\n",
    "\n",
    "1) allow for interactive, cross-platform control of figures and plots,<br>\n",
    "2) make it easy to produce static raster or vector graphics files without the need for any GUIs.   \n",
    "\n",
    "<br><br>\n",
    "\n",
    "<center><i>\"Matplotlib tries to make easy things easy and hard things possible.\"</center>\n",
    "    \n",
    "<img src=\"https://matplotlib.org/_static/logo2_compressed.svg\" width=\"40%\">\n",
    "\n",
    "*Source: https://matplotlib.org/_static/logo2_compressed.svg*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an installation of the *matplotlib* library, you have to enter the following command in the terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "!pip install matplotlib\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing the library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A library is imported using the command *import*. In the following, we will use the *pyplot* module, which we will call *plt*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the pyplot module\n",
    "import matplotlib.pyplot as plt\n",
    "#plt.style.use('fivethirtyeight')\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting interactively within an IPython notebook can be done with the magic ```%matplotlib``` command. In the IPython notebook, you also have the option of embedding graphics directly in the notebook, with two possible options:<br>\n",
    "    ```%matplotlib notebook``` will lead to interactive plots embedded within the notebook <br>\n",
    "    ```%matplotlib inline``` will lead to static images of your plot embedded in the notebook\n",
    "\n",
    "For this book, we will generally opt for ```%matplotlib inline```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basics in *matplotlib*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In *matplotlib*, there are several common approaches to plotting with matplotlib:\n",
    "\n",
    "- object-oriented API,\n",
    "- *pyplot* API.\n",
    "- *pylab* API *(not recommended)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll focus on the object-oriented approach which is recommended to use, especially when you want to create complex graphics, because it gives you more control over the figure. *Pyplot* on the other hand is just a *MATLAB*-like module of the *matplotlib* library which is mainly intended for interactive\n",
    "plots and simple cases of programmatic plot generation. The syntax of both approaches is quite similiar, so if you know the object-oriented approach, you won't have a problem to work with the *pyplot* module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we'll create our first plot, we need to look at the parts of a *matplotlib* figure:\n",
    "\n",
    "-**Figure**<br>\n",
    "The whole figure.<br>\n",
    "-**Axes**<br>\n",
    "This is what you think of as ’a plot’, it is the region of the image with the\n",
    "data space. A given figure can contain many Axes, but a given Axes\n",
    "object can only be in one Figure.<br>\n",
    "-**Axis**<br>\n",
    "These are the number-line-like objects. They take care of setting the\n",
    "graph limits and generating the ticks (the marks on the axis) and\n",
    "ticklabels (strings labeling the ticks)<br>\n",
    "-**Artist**<br>\n",
    "Basically everything you can see on the figure is an artist (even the\n",
    "Figure, Axes, and Axis objects)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://matplotlib.org/3.1.1/_images/sphx_glr_anatomy_001.png\" width=\"50%\">\n",
    "\n",
    "*Source: https://matplotlib.org/3.1.1/_images/sphx_glr_anatomy_001.png*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's create our first plot using *matplotlib*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow in *matplotlib*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting in *matplotlib* follows a certain rountine which includes the following steps:\n",
    "\n",
    "1. **Prepare data**\n",
    "2. **Create plot**\n",
    "3. **Plot**\n",
    "4. **Customise plot**\n",
    "5. **Save plot**\n",
    "6. (**Show plot**)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Prepare data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's first create a small data set for a simple visualization using *matplotlib*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate data set (STEP 1)\n",
    "x = [1,2,3,4]\n",
    "y = [1,2,3,4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Create a plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using the object-oriented approach, we have to initialize the objects *Figure* and *Axes* first. This is commonly done by calling the `pyplot.figure()` function and adding the *Axes* object via `pyplot.add_axes()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Step 1: Prepare data\n",
    "x = [1,2,3,4]\n",
    "y = [1,2,3,4]\n",
    "\n",
    "### Step 2: Create plot\n",
    "# Initialize `Figure` and `Axes` object\n",
    "fig = plt.figure()\n",
    "ax = fig.add_axes( [0, 0, 1, 1]) # [bottom, top, width, height]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can use `pyplot.subplots()` to initialize both the *Figure* and the *Axes* object. You can also generate several *Axes* objects in one *Figure*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Step 1: Prepare data\n",
    "x = [1,2,3,4]\n",
    "y = [1,2,3,4]\n",
    "\n",
    "### Step 2: Create plot\n",
    "# Initialize `Figure` and `Axes` object\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1) # 1 row, 1 column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Plot + 4. Customise Plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we can simply use a plotting method on the *Axes* object. In this small example we're going to create a simple line graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Step 1: Prepare data\n",
    "x = [1,2,3,4]\n",
    "y = [1,2,3,4]\n",
    "\n",
    "### Step 2: Create plot\n",
    "# Initialize `Figure` and `Axes` object\n",
    "fig, ax = plt.subplots(nrows=1, ncols=1) # 1 row, 1 column\n",
    "\n",
    "### Step 3: Plot\n",
    "ax.plot(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we move on, let's have a look at the *matplotlib* documentation of the line plot method:\n",
    "\n",
    "https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.axes.Axes.plot.html#matplotlib.axes.Axes.plot\n",
    "\n",
    "Alternatively, we can use the following command to open the documentation in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax.plot?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Charting types in *matplotlib*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Matplotlib* offers a huge amount of plotting methods/functions. A good starting point, so you don't have to start from scratch, is the *matplotlib* gallery, where you can find example code for some of these methods/functions (https://matplotlib.org/gallery/index.html). \n",
    "\n",
    "*Note: Interestingly, just about all methods of an Axes object exist as a function in the pyplot module (and vice-versa).*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A small list of *matplotlib* *Axes* methods:\n",
    "\n",
    "- **Histogram:** ax.hist(x)\n",
    "    * x: numerical (converted into categorical)\n",
    "    * distribution of x\n",
    "    \n",
    "- **Box Plot:** ax.boxplot(x)\n",
    "    * x: numerical\n",
    "    * distribution of x / identification of outliers\n",
    "- **Violin Plot:** ax.violinplot(x)\n",
    "    * x: numerical\n",
    "    * distribution of x \n",
    "- **Pie Chart:** ax.pie(x) *(not recommended)*\n",
    "    * x: categorical\n",
    "    * proportions of categories in x\n",
    "- **Scatter Plot**:  ax.scatter(x,y)\n",
    "    * x: numerical\n",
    "    * y: numerical\n",
    "    * joint distribution of x and y / identification of outliers\n",
    "- **Line Plot**:  ax.plot(x, y)\n",
    "    * x: numerical/categorical\n",
    "    * y: numerical\n",
    "    * function *(math.)* / time-series\n",
    "- **Bar Plot**:  ax.bar(x, y) \n",
    "    * x: categorical\n",
    "    * y: numerical (often count of x)\n",
    "    * joint distribution of x and y / identification of outliers\n",
    "- **Contour Plot**: ax.contour(X, Y, Z)\n",
    "    * X: numerical\n",
    "    * Y: numerical\n",
    "    * Z: numerical\n",
    "    * 3D surface on a 2D format\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at some examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Bar plot***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our first example, we want to visualize the overall number of reported Covid-19 cases per state. For this, we use the variables *Bundesland* and *AnzahlFall*. Let's first have a look at those two variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "covid19[['Bundesland', 'AnzahlFall']] # Step 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the each state occurs several times in our dataset. Let's add the variable *Meldedatum* and print the first 20 rows of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "covid19[['Bundesland', 'AnzahlFall', 'Meldedatum']].head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see, that we have time-series of reported cases. Since we are interested in the cumulative number of cases per state, we have to take the sum of the variable *AnzahlFall* for each state. This can simply be done by using the `groupby()` method from the *Pandas* library. We group our dataset by the variable *Bundesland* and then define the operation that is then applied to the grouped data which is the sum of the remaining variable *AnzahlFall*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases_state = covid19[['Bundesland','AnzahlFall']].groupby('Bundesland').sum() # Step 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we see that the structure of our dataframe has changed; to restore the old structure we can use the `reset_index()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases_state = cases_state.reset_index() # Step 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print the result again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have a dataframe with the old structure which can be used for our visualisation. So, let's use a barplot to visualise this table with states along the x-axis and the sums along the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(16,6)) # Step 2\n",
    "ax.bar(cases_state['Bundesland'],cases_state['AnzahlFall']) #Step 3\n",
    "ax.set_xlabel('Bundesland') # Step 4\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Übermittelte Covid19-Infetionen nach Bundesland in Deutschland')\n",
    "plt.show() # Step 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The x-axis is a bit messy since some of the state names are too long and overlap. So, let's flip the axes to solve this issue by using the `barh()` method for horizontal barplots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,10)) \n",
    "ax.barh(cases_state['Bundesland'],cases_state['AnzahlFall']) # barh() instead of bar()\n",
    "ax.set_ylabel('Bundesland')\n",
    "ax.set_xlabel('Anzahl')\n",
    "ax.set_title('Übermittelte Covid19-Infetionen nach Bundesland in Deutschland')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This look a lot better than our vertical bar plot. To further improve the readability of this chart, we sort the states by the number of cases instead of their alphabetical order. For this, we use the `sort_values()` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases_state_sorted = cases_state.sort_values('AnzahlFall')\n",
    "cases_state_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the result again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,10))\n",
    "ax.barh(cases_state_sorted['Bundesland'],cases_state_sorted['AnzahlFall'])\n",
    "ax.set_ylabel('Bundesland') # Step 4\n",
    "ax.set_xlabel('Anzahl')\n",
    "ax.set_title('Übermittelte Covid19-Infetionen nach Bundesland in Deutschland')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In *matplotlib* you can easily overlay some plots by using multiple methods on the same *Axes* object. Let's split the confirmed number of cases into *infected*, *recovered* and *deceased* plot them in the same figure. For this task, we need the variables *Bundesland*, *AnzahlFall*, *AnzahlGenesen* und *AnzahlTodesfall*. Following the same procedure as before, we group the data by the variable *Bundesland* and add up all the cases. Then, we sort them by the variable *AnzahlFall*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_state = covid19[['Bundesland','AnzahlFall','AnzahlGenesen','AnzahlTodesfall']]\n",
    "data_state = data_state.groupby('Bundesland').sum()\n",
    "data_state = data_state.reset_index()\n",
    "data_state_sorted = data_state.sort_values('AnzahlFall')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_state_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we again create our *Figure* and one *Axes* object which will then be used for all plots. This way, we can simply overlay our 3 barplots by first plotting the largest bar, then the second largest and finally the smallest bars. We can do it this way because the variable *AnzahlFall* **includes** the other two variables. When overlaying the other two bar plots, we have to consider that those variables are **exclusive** which means we have to add the number of deaths to the recovered cases to report them correctly in an overlayed plot. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(10,10))\n",
    "\n",
    "# Use same Axes object (ax) for all the 3 plots\n",
    "ax.barh(data_state_sorted['Bundesland'],data_state_sorted['AnzahlFall'],label='infected')\n",
    "ax.barh(data_state_sorted['Bundesland'],data_state_sorted['AnzahlGenesen']+data_state_sorted['AnzahlTodesfall'],label='recovered')\n",
    "ax.barh(data_state_sorted['Bundesland'],data_state_sorted['AnzahlTodesfall'], label='deceased')\n",
    "ax.set_xlabel('Anzahl')\n",
    "ax.set_ylabel('Bundesland')\n",
    "ax.set_title('Übermittelte Covid19-Infektionen pro Bundesland');\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Line plot***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous example, we have looked at the cumulative cases which means we haven't captured the longitudinal aspect of our data. Therefore, we will here plot the cumulative cases over time and not just at the last time point. This means, we again need the variables *Bundesland*, *Meldedatum* and *AnzahlFall* for this task. Different from the task above, we now group the data by two variables *Bundesland* and *Meldedatum* to add the cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases = covid19[['Bundesland', 'AnzahlFall', 'Meldedatum']].groupby(['Bundesland','Meldedatum'])['AnzahlFall'].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this gives us the sum of cases per state and date, we have to group the data again to calculate the cumulative sums per state and date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases_cum = cases.groupby(level=0).cumsum()\n",
    "cases_cum = cases_cum.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cases_cum.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can plot the temporal evolution of Covid-19 cases for individual states (e.g. Bavaria, Berlin) by first creating subsets of the data for the corresponding state and then using the method `plot()` to create the line charts. We overlay the line charts by using the same *Axes* object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(14,10))\n",
    "\n",
    "bavaria = cases_cum[cases_cum['Bundesland']=='Bayern']\n",
    "berlin = cases_cum[cases_cum['Bundesland']=='Berlin']\n",
    "ax.plot(bavaria['Meldedatum'], bavaria['AnzahlFall'], label='Bayern')\n",
    "ax.plot(berlin['Meldedatum'], berlin['AnzahlFall'], label='Berlin')\n",
    "\n",
    "ax.legend()\n",
    "ax.set_xlabel('Meldedatum')\n",
    "ax.set_ylabel('Anzahl')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to plot a line for every state, you can use a *for-loop* to create the subsets and the line charts. You have to first create an array with unique state names which can be done by using *pandas* `unique()` method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states = cases_cum['Bundesland'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, you can simply loop through this array to create the subsets and the plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(14,10))\n",
    "\n",
    "for state in states:\n",
    "    x = cases_cum[cases_cum['Bundesland']==state]['Meldedatum']\n",
    "    y = cases_cum[cases_cum['Bundesland']==state]['AnzahlFall']\n",
    "    ax.plot(x,y,label=state, alpha=0.5)\n",
    "    \n",
    "ax.legend()\n",
    "ax.set_xlabel('Meldedatum')\n",
    "ax.set_ylabel('Anzahl')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Save plots + 6. Show Plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily save your created figures by adding the function ```fig.savefig()``` to your code. The only argument you need to pass to the function is a complete path/just the file name (you then save it in your directory).\n",
    "\n",
    "*Recommendation: Use vector-based graphics; so, you don't get blurry figures in your document later on (e.g. SVG, EPS, PDF)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Step 1: Prepare data\n",
    "x = [1,2,3,4]\n",
    "y = [1,2,3,4]\n",
    "\n",
    "### Step 2: Create plot\n",
    "# Initialize `Figure` and `Axes` object\n",
    "fig, ax = plt.subplots(1, 1) # Initializing the 'Figure' and 'Axes' objects \n",
    "\n",
    "### Step 3: Plot\n",
    "ax.plot(x,y) # Line plot \n",
    "\n",
    "### Step 4: Customise plot\n",
    "ax.plot(x, y); \n",
    "ax.set_xlabel('x'); #  Set label for x-axis \n",
    "ax.set_ylabel('y'); # Set label for y-axis\n",
    "ax.set_title('Mein erster Plot mit matplotlib'); # Set title\n",
    "\n",
    "### Step 5: Save plot\n",
    "from pathlib import Path\n",
    "Path(\"results/week1_session3_matplotlib\").mkdir(parents=True, exist_ok=True)\n",
    "fig.savefig('results/week1_session3_matplotlib/line.pdf', \n",
    "            format='pdf', \n",
    "            bbox_inches = 'tight'); # bbox_inches = 'tight' created a padding around the figure\n",
    "\n",
    "### Step 6: Show plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Discusison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, data visualisation using the *matplotlib* library was introduced. The workflow to create a plot was presented using a step-by-step introduction.\n",
    "\n",
    "- **Data Preparation:** We selected different data features from the RKI Covid-19 dataset.\n",
    "- **Create Plot:** We instantiated *Figure* and *Axes* objects.\n",
    "- **Plot:** We introduced different chart types.\n",
    "- **Customisation:** We added axis labels, a title and a legend to our plots.\n",
    "- **Save:** We saved our plots in an arbitrary image format.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, it's time for you to get your hands dirty. Please answer the following questions by choosing a **suitable graphical representation** of the data *(think about scales)* including axis labels, title and legend (if necessary).\n",
    "\n",
    "*Tip:* The following resources might be helpful:\n",
    "- www.stackoverflow.com\n",
    "- matplotlib documentation: https://matplotlib.org/api/axes_api.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) **Create a Covid-19 subset of the states Berlin and Mecklenburg-Vorpommern.** <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2) **Plot the Covid-19 cases by sex in Berlin.** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.1 Select the required variables and store them in a separate dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.2 Group the data by sex and take the sum of the required variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.3 Plot the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3) Plot the Covid-19 case fatalaty rate (CFR) by age group in Berlin and Mecklenburg-Vorpommern.**\n",
    "\n",
    "*Note:* CFT is defined as\n",
    "$$CFT=\\frac{\\text{number of deaths from disease}}{\\text{number of confirmed cases of disease}}$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.1 Select the required variables and store them in a separate dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.2 Group the data by age group and compute the CFR per age group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.3 Plot the data (overlay the plots for both states using 'alpha=0.5')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4) Plot the total CFR in Berlin and Mecklenburg-Vorpommern.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.1 Repeat the steps from 3) without grouping the values by age groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.2 Compare the results from 4.1 with the results from 3.3. What do you notice? How is that possible?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5) **Plot the number of reported daily Covid-19 infections in Berlin over time.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.1 Select the required variables and store them in a separate dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.2 Group the dataframe by the required variable and use correct mathematical operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.3 Reset the indices of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.4 Plot the number of reported daily infections over time using a suitable plotting method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6) **Plot the cumulative cases by age group over time.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.1 Select the required variables and store them in a separate dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.2 Group the data by age group as well as reporting date and take the sum of the required variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.3 Group the data again by *level=0* and take the cumulative sum.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.4 Reset the index of the resulting dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.5 Create an array with unique ages groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6.6 Loop through the age group array and plot the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7) **Plot the distributions of daily reported Covid-19 cases in Berlin and Brandenburg. Interpret the results.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.1 Calculate the daily reported number of Covid-19 infections in Berlin and Brandenburg similar to the excercises above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.2 Plot the distribution of daily cases for both states."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.3 Interpret the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1) Create a Covid-19 subset of the states Berlin and Mecklenburg-Vorpommern.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "berlin = covid19[covid19['Bundesland']=='Berlin']\n",
    "mv = covid19[covid19['Bundesland']=='Mecklenburg-Vorpommern']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2) Plot the Covid-19 cases by sex in Berlin.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 2.1\n",
    "berlin_cases = berlin[['Geschlecht', 'AnzahlFall']]\n",
    "\n",
    "### 2.2\n",
    "berlin_cases_sex = berlin_cases.groupby('Geschlecht').sum().reset_index()\n",
    "\n",
    "### 2.3\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "\n",
    "ax.bar(berlin_cases_sex['Geschlecht'], berlin_cases_sex['AnzahlFall'])\n",
    "ax.set_xlabel('Geschlecht')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Gechlechterverteilung der übermittelten Covid-19 positiv Fälle in Berlin')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3) Plot the Covid-19 case fatalaty rate (CFR) by age group in Berlin and Mecklenburg-Vorpommern.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 3.1\n",
    "berlin_cases = berlin[['Altersgruppe','AnzahlFall']]\n",
    "berlin_ceased = berlin[['Altersgruppe','AnzahlTodesfall']]\n",
    "\n",
    "mv_cases = mv[['Altersgruppe','AnzahlFall']]\n",
    "mv_ceased = mv[['Altersgruppe','AnzahlTodesfall']]\n",
    "\n",
    "### 3.2\n",
    "berlin_cases_age = berlin_cases.groupby('Altersgruppe')['AnzahlFall'].sum().reset_index()\n",
    "berlin_ceased_age = berlin_ceased.groupby('Altersgruppe')['AnzahlTodesfall'].sum().reset_index()\n",
    "berlin_cfr_age = berlin_ceased_age['AnzahlTodesfall'] / berlin_cases_age['AnzahlFall']\n",
    "\n",
    "mv_cases_age = mv_cases.groupby('Altersgruppe')['AnzahlFall'].sum().reset_index()\n",
    "mv_ceased_age = mv_ceased.groupby('Altersgruppe')['AnzahlTodesfall'].sum().reset_index()\n",
    "mv_cfr_age = mv_ceased_age['AnzahlTodesfall'] / mv_cases_age['AnzahlFall']\n",
    "\n",
    "### 3.3\n",
    "fig, ax = plt.subplots(1,1,figsize=(12,6))\n",
    "ax.bar(berlin_ceased_age['Altersgruppe'], berlin_cfr_age, alpha=0.5, label='Berlin')\n",
    "ax.bar(mv_ceased_age['Altersgruppe'], mv_cfr_age, alpha=0.5, label='Mecklenburg-Vorpommern')\n",
    "ax.set_xlabel('Altersgruppe')\n",
    "ax.set_ylabel('CFR')\n",
    "ax.set_title('CFR nach Altersgruppe in Berlin und Mecklenburg-Vorpommern')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4) Plot the total CFR in Berlin and Mecklenburg-Vorpommern.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 4.1\n",
    "berlin_cfr = berlin['AnzahlTodesfall'].sum() / berlin['AnzahlFall'].sum()\n",
    "mv_cfr = mv['AnzahlTodesfall'].sum() / mv['AnzahlFall'].sum()\n",
    "\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "ax.bar(0, berlin_cfr, alpha=0.5, label='Berlin')\n",
    "ax.bar(1, mv_cfr, alpha=0.5, label='Mecklenburg-Vorpommern')\n",
    "ax.set_xticks([0,1])\n",
    "ax.set_xticklabels(['Berlin', 'Mecklenburg-Vorpommern'])\n",
    "ax.set_ylabel('CFR')\n",
    "ax.set_title('CFR in Berlin und Mecklenburg-Vorpommern')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.2:<br>\n",
    "Although, the CFR per age group is lower in Mecklenburg-Vorpommern compared to Berlin, the total CFR is higher. This phenomenon is known in statistics as *Simpson's paradox*. Where trends appear when comparing different groups which then disappear or reverse when the groups are combined. To resolve this problem, the causal relations have to be addressed. The reason here is the difference in the age distributions of Covid19 patients in the two states. As you can see from the plot below, the number of 'older' patients in Mecklenburg-Vorpommern is relatively higher compared to Berlin, leading to a higher overall CFR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "ax.bar(berlin_cases_age['Altersgruppe'], berlin_cases_age['AnzahlFall'], alpha=0.5, label='Berlin')\n",
    "ax.bar(mv_cases_age['Altersgruppe'], mv_cases_age['AnzahlFall'], alpha=0.5, label='Mecklenburg-Vorpommern')\n",
    "ax.set_xlabel('Altersgruppe')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Übermittelte Covid19-Infektionen nach Altersgruppe in Berlin and Mecklenburg-Vorpommern')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "ax.bar(berlin_ceased_age['Altersgruppe'], berlin_ceased_age['AnzahlTodesfall'], alpha=0.5, label='Berlin')\n",
    "ax.bar(mv_ceased_age['Altersgruppe'], mv_ceased_age['AnzahlTodesfall'], alpha=0.5, label='Mecklenburg-Vorpommern')\n",
    "ax.set_xlabel('Altersgruppe')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Übermittelte Covid19-Todeszahlen nach Altersgruppe in Berlin und Mecklenburg-Vorpommern')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5) Plot the number of daily Covid-19 infections in Berlin over time.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 5.1\n",
    "berlin_cases = berlin[['AnzahlFall','Meldedatum']]\n",
    "### 5.2\n",
    "berlin_daily_cases = berlin_cases.groupby('Meldedatum').sum()\n",
    "### 5.3\n",
    "berlin_daily_cases = berlin_daily_cases.reset_index()\n",
    "\n",
    "### 5.4\n",
    "fig, ax = plt.subplots(1,1,figsize=(12,6))\n",
    "ax.bar(berlin_daily_cases['Meldedatum'],berlin_daily_cases['AnzahlFall'])\n",
    "ax.set_xlabel('Meldedatum')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Verlauf der übermittelten täglichen Covid-19 Infektionen in Berlin')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6) Plot the cumulative cases by age group over time.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 6.1\n",
    "berlin_cases = berlin[['Altersgruppe','AnzahlFall','Meldedatum']]\n",
    "### 6.2\n",
    "berlin_cases_grouped = berlin_cases.groupby(['Altersgruppe','Meldedatum'])['AnzahlFall'].sum()\n",
    "### 6.3\n",
    "berlin_cases_cum = berlin_cases_grouped.groupby(level=0).cumsum()\n",
    "### 6.4\n",
    "berlin_cases_cum = berlin_cases_cum.reset_index()\n",
    "### 6.5\n",
    "age_groups = berlin_cases_cum['Altersgruppe'].unique()\n",
    "### 6.6\n",
    "fig, ax= plt.subplots(1, 1, figsize=(14,6))\n",
    "\n",
    "for age_group in age_groups:\n",
    "    x = berlin_cases_cum[berlin_cases_cum['Altersgruppe']==age_group]['Meldedatum']\n",
    "    y = berlin_cases_cum[berlin_cases_cum['Altersgruppe']==age_group]['AnzahlFall']\n",
    "    ax.plot(x,y,label=age_group, alpha=0.5)\n",
    "\n",
    "ax.set_xlabel('Meldedatum')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Übermittelte Covid19-Infetionen im Verlauf nach Altergruppe in Berlin')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7) Plot the distributions of daily reported Covid-19 cases in Berlin and Brandenburg. Interpret the results.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 7.1\n",
    "berlin = covid19[covid19['Bundesland']=='Berlin']\n",
    "berlin_cases = berlin[['Meldedatum', 'AnzahlFall']]\n",
    "berlin_cases_daily = berlin_cases.groupby('Meldedatum').sum().reset_index()\n",
    "\n",
    "brandenburg = covid19[covid19['Bundesland']=='Brandenburg']\n",
    "brandenburg_cases = brandenburg[['Meldedatum', 'AnzahlFall']]\n",
    "brandenburg_cases_daily = brandenburg_cases.groupby('Meldedatum').sum().reset_index()\n",
    "\n",
    "### 7.2 (Boxplot)\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "\n",
    "ax.boxplot(berlin_cases_daily['AnzahlFall'], positions=[0])\n",
    "ax.boxplot(brandenburg_cases_daily['AnzahlFall'], positions=[1])\n",
    "ax.set_xticklabels(['Berlin', 'Brandenburg'])\n",
    "\n",
    "ax.set_xlabel('Bundesland')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Verteilung übermittelter täglicher Covid-19 Infektionszahlen in Berlin und Brandenburg')\n",
    "plt.show()\n",
    "\n",
    "### alternatively: Histogram\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "\n",
    "ax.hist(berlin_cases_daily['AnzahlFall'], alpha=0.5, label='Berlin')\n",
    "ax.hist(brandenburg_cases_daily['AnzahlFall'], alpha=0.5, label='Brandenburg')\n",
    "\n",
    "ax.set_xlabel('Anzahl übermittelter Covid-19 Fälle')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Verteilung übermittelter täglicher Covid-19 Infektionszahlen in Berlin und Brandenburg')\n",
    "ax.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7.3: \n",
    "\n",
    "The distributions of reported daily Covid-19 cases show an overall lower absolute number of cases in Brandenburg, as indicated the Median and the box. Both distributions are right-skewed due to low rates of infection. Large numbers of infection result from the exponential growth in the 'second-wave'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seaborn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matplotlib has proven to be an incredibly useful and popular visualization tool, but it comes with drawbacks:\n",
    "\n",
    "- defaults are not exactly the best choices,\n",
    "- API is relatively low level,\n",
    "- lacking compatability with *pandas DataFrame*s\n",
    "\n",
    "Adressing these drawbacks, *Seaborn* is statistical plotting library build on top of *matplotlib* and provides an API that offers sane choices for plot style and color defaults, defines simple high-level functions for common statistical plot types, and provides a close integration with *pandas* data structures.\n",
    "<br><br>\n",
    "\n",
    "<center><i>“If matplotlib ’tries to make easy things easy and hard things possible’, seaborn tries to make a well-defined set of hard things easy too.”</center>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; - Michael Waskom, Introduction to Seaborn\n",
    "\n",
    "Behind the scenes, *Seaborn* uses matplotlib to draw plots; so, if you know *matplotlib*, you’re already halfway through *Seaborn*. Many tasks can be accomplished with only seaborn functions, but further customization might require using *matplotlib* directly. \n",
    "\n",
    "<img src=\"https://www.datacouncil.ai/hs-fs/hubfs/DataEngConf/NYC16/OSS%20Projects/Seaborn-Logo_256px.png?width=256&name=Seaborn-Logo_256px.png\" width=\"20%\">\n",
    "\n",
    "*Source: https://www.datacouncil.ai/hs-fs/hubfs/DataEngConf/NYC16/OSS%20Projects/Seaborn-Logo_256px.png?width=256&name=Seaborn-Logo_256px.png*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an installation of the *Seaborn* library, you have to enter the following command in the terminal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "!pip install seaborn\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing the library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A library is imported using the command import. In the following, we will call the library ```sns```.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Laden der seaborn library\n",
    "import seaborn as sns\n",
    "\n",
    "# Default seaborn theme\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at some of the drawbacks of *matplotlib* and how *Seaborn* adresses these."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Close integration with *pandas*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned before, *Seaborn* is closely integrated with *pandas DataFrame*s; so, for most of the functions you simply specify your data set in your plotting function and call a variable by using its header/column name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert our toy data set into a dataframe\n",
    "df = pd.DataFrame({'x': x, 'y': y})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Line plot\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "sns.lineplot(data=df,\n",
    "             x='x',\n",
    "             y='y', \n",
    "             ax=ax)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, another example of the close integration with *pandas* can be seen in the plot; *Seaborn* loads the full data set and uses the selected header/column names as axis labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "berlin_cases = berlin.groupby('Meldedatum')['AnzahlFall'].sum().reset_index()\n",
    "\n",
    "# Histogram\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "sns.histplot(x='AnzahlFall', data=berlin_cases, bins=50, ax=ax)\n",
    "ax.set_xlabel('Anzahl übermittelter Fälle pro Tag')\n",
    "ax.set_ylabel('Anzahl')\n",
    "ax.set_title('Verteilung der übermittelten täglichen Covid19-Neuinfektionen in Berlin')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In ``Seaborn``, the ``hue`` parameters allows to easily compare different groups. This parameter determines which column in the data frame should be used for colour encoding. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate cumulative cases\n",
    "cases = covid19[['Bundesland', 'AnzahlFall', 'Meldedatum']].groupby(\n",
    "    ['Bundesland','Meldedatum'])['AnzahlFall'].sum()\n",
    "cases_cum = cases.groupby(level=0).cumsum()\n",
    "cases_cum = cases_cum.reset_index()\n",
    "\n",
    "# Plot cumulative cases for all states over time\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "sns.lineplot(x='Meldedatum', y='AnzahlFall', data=cases_cum, hue='Bundesland')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### High-level functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In *matplotlib*, tasks like plotting counts for categorical data, like you did in ex.1 and ex.2, is tedious. First, you have to count your categories and then pass these to the bar plot function. In contrast, *Seaborn* provides high-level functions for these tasks which will do all the work for you. Let's create the plot from ex.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bar plot with counts\n",
    "fig, ax = plt.subplots(1, 1, figsize=(12,6))\n",
    "sns.barplot(data=berlin,\n",
    "            y='AnzahlFall',\n",
    "            x='Geschlecht',\n",
    "            #hue='Altersgruppe',\n",
    "            estimator=sum, \n",
    "            ci=False, \n",
    "            ax=ax)\n",
    "\n",
    "ax.set_title('Gechlechterverteilung der übermittelten Covid-19-Infektionen in Berlin')\n",
    "ax.set_ylabel('Anzahl')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that we don't have to build a subset for every category like in *matplotlib*, we can simply split the data by passing a conditioning variable to the argument *hue*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A small list of *Seaborn* functions:\n",
    "\n",
    "- **Boxplot:** sns.boxplot(x)\n",
    "- **Violin Plot:** sns.violinplot(x)\n",
    "- **Bar Plot:** sns.barplot(x,y)\n",
    "- **Line Plot:** sns.lineplot(x,y)\n",
    "- **Scatter Plot:** sns.scatterplot(x,y)\n",
    "- **Faceted Scatter Plot:** sns.relplot(x,y)\n",
    "- **Scatter Plot with regression line:** sns.lmplot(x,y)\n",
    "- **KDE Plot:** sns.kde(x,y)\n",
    "\n",
    "Visualizations with multiple plots\n",
    "\n",
    "- **Scatter Plot with histograms:** sns.jointplot(x,y)\n",
    "- **Scatter Plot Matrix:** sns.pairplot(df[['x','y',...]])\n",
    "- ...\n",
    "\n",
    "Optimized function for plotting categorical data:\n",
    "- **Plotting categorical data:** \n",
    "    - sns.catplot(x,y,kind) *(kind=['swarm', 'violin', 'boxplot'])*\n",
    "    - sns.sns.stripplot(x,y)\n",
    "    - sns.swarmplot(x,y)\n",
    "\n",
    "Check out the tutorial or the gallery for more examples.\n",
    "\n",
    "https://seaborn.pydata.org/tutorial.html\n",
    "\n",
    "https://seaborn.pydata.org/examples/index.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}