02_Working_with_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Lesson 02: Working With Data\n",
    "===============\n",
    "\n",
    "As scientists, we often use Python to organize, analyze, and display data in a variety of ways. We'll learn some of the basic techniques you'll need in order to read a data file, ask questions of the data in it, display data in a way that makes sense, and store that information. We're going to take a problem-based approach, and will use data from a real study: [_Pliocene Warmth, Polar Amplification, and Stepped Pleistocene Cooling Recorded in NE Arctic Russia_ by J. Brigham-Grette _et al._](https://www.dropbox.com/s/moscw2rr6ksqgw1/Brigham-Grette2014_Pliocene_Pleistocene_Russia.pdf?dl=0). This is what climatologists call a _multiproxy_ study - a study in which multiple indicators of past climates (proxies) are put together to reconstruct characteristics of that past climate. In the next tutorial, we'll recreate some of Brigham-Grette's graphs and make a few of our own. This time, we'll just explore the data.\n",
    "\n",
    "Goals of this Notebook\n",
    "-----------------------\n",
    "1. Read, store, access, and save data using Python, Numpy, and Pandas\n",
    "2. Choose data structures appropriate for common tasks, including scalar data types, strings, lists, arrays, and data frames\n",
    "3. Understand basic Python error messages\n",
    "4. Load and use Python packages\n",
    "5. Perform simple mathematical and data organization operations\n",
    "6. View the contents of variables\n",
    "\n",
    "Variables\n",
    "----------\n",
    "A key piece of the puzzle is understanding how information is stored in a computer's _memory_. We'll sidestep the somewhat more complex issue of how information is stored on you computer's hard disk for now. \n",
    "\n",
    "Python stores information in what are called _variables_. I like to think of variables as bins in which Python puts information. every variable can hold a certain type of data. Some variables contain single numbers - integers (`0`, `1`, `2`, ... etc.; could be negative), decimal numbers (e.g. `1.2345`), or characters (e.g. `a`, `3`, `$`, `N`, etc.). _Sequences_ contain multiples of a single type of data. Multiple characters make up a _string_. Both strings and characters usually are identified by single or double quotation marks (e.g. `\"this is a string\"`; `'Another f#&*ing string.'`; `\"X\"` is a single character). _Lists_, a special type of sequence, are what they sound like: lists of data of a variety of types, identified by brackets (e.g. `['a', 1, 2.5, -637, 'Hubert']`). Before going much further, let's play with a few data types.\n",
    "\n",
    "#### Now Try This\n",
    "1. Run the code block below. Notice that you can use one variable to define the contents of another variable. Also notice that the `print` command outputs the contents of a variable.\n",
    "2. Modify the code block so that instead of adding `a + b`, you add `a + c`. What happens when you try to add variables with two different types of data? What potentially helpful information does Python give you? Fix the mistake by returning the line to its original state.\n",
    "3. Modify the code block so that `d` is equal to `a` minus 0.5. You are now subtracting a decimal from an integer. What happens? Do you get the same error for adding integers and decimals? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "a = 1\n",
    "b = 0\n",
    "print (a)\n",
    "\n",
    "c = 'cat'\n",
    "print (c)\n",
    "\n",
    "a + b\n",
    "\n",
    "d = a - 5\n",
    "print (d)\n",
    "\n",
    "mylist = [1, 2, -3, 'dog']\n",
    "print ('My List:',mylist)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Packages\n",
    "---------\n",
    "We often want to do things with data that are difficult or complicated to do with standard Python. Other clever programmers have often accomplished these tasks, and made the tools to do them available as _packages_. Packages are somewhat like plug-ins in, say, MS Word: they are extra features that you install after you've set up the main application. \n",
    "\n",
    "There are two steps to using a Python package. First, you need to download the package and put it in the right place on your computer. There are two ways to install packages, both from the command line. Suppose you wanted to install the [_Pandas_](https://pandas.pydata.org/) data analysis and management package. (Note that this was likely installed already when you first set up Anaconda.) It's better to use Anaconda's installer, `conda`. To do that, type:\n",
    "```bash\n",
    "% conda install pandas\n",
    "```\n",
    "Alternatively, you could use the `pip` installer:\n",
    "```bash\n",
    "% pip install pandas\n",
    "```\n",
    "This might put packages in a different location on your comupter than `conda` does, so it's better to use `conda` when you can.\n",
    "\n",
    "The second step is to tell the Python interpreter to use the package. We do this in the next block of code using the `import` command.\n",
    "\n",
    "#### Now Try This\n",
    "1. Run the next block of code. Make sure there are no error messages.\n",
    "2. Every command from the Pandas package now starts with the prefix \"pd.\" So `pd.DataFrame` refers to the `DataFrame` command in Pandas.\n",
    "3. You'll also want to use the [_NumPy_](http://www.numpy.org/) scientific computing package. Add a line to the code block below to import `numpy` as `np`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "INSTALLED VERSIONS\n",
      "------------------\n",
      "commit: None\n",
      "python: 3.6.3.final.0\n",
      "python-bits: 64\n",
      "OS: Windows\n",
      "OS-release: 10\n",
      "machine: AMD64\n",
      "processor: Intel64 Family 6 Model 61 Stepping 4, GenuineIntel\n",
      "byteorder: little\n",
      "LC_ALL: None\n",
      "LANG: None\n",
      "LOCALE: None.None\n",
      "\n",
      "pandas: 0.22.0\n",
      "pytest: 3.2.1\n",
      "pip: 9.0.1\n",
      "setuptools: 38.4.0\n",
      "Cython: 0.26.1\n",
      "numpy: 1.13.3\n",
      "scipy: 1.0.0\n",
      "pyarrow: None\n",
      "xarray: None\n",
      "IPython: 6.1.0\n",
      "sphinx: 1.6.3\n",
      "patsy: 0.4.1\n",
      "dateutil: 2.6.1\n",
      "pytz: 2018.3\n",
      "blosc: None\n",
      "bottleneck: 1.2.1\n",
      "tables: 3.4.2\n",
      "numexpr: 2.6.2\n",
      "feather: None\n",
      "matplotlib: 2.1.2\n",
      "openpyxl: 2.4.8\n",
      "xlrd: 1.1.0\n",
      "xlwt: 1.3.0\n",
      "xlsxwriter: 1.0.2\n",
      "lxml: 4.1.0\n",
      "bs4: 4.6.0\n",
      "html5lib: 0.999999999\n",
      "sqlalchemy: 1.1.13\n",
      "pymysql: None\n",
      "psycopg2: None\n",
      "jinja2: 2.9.6\n",
      "s3fs: None\n",
      "fastparquet: None\n",
      "pandas_gbq: None\n",
      "pandas_datareader: None\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.show_versions()\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy Arrays\n",
    "-------------\n",
    "NumPy is an improvement over many of the mathematical and data-processing tools available in standard Python. One of the main improvements is in NumPy's ability to handle _arrays_ quickly and easily. Arrays, like lists, are sequences. Unlike lists, each array only holds one data type, usually numbers. Arrays can be multi-dimensional, meaning that they can hold 2-D grids of numbers, 3-D blocks, or even more complex structures with multiple dimensions if you need them. The command `np.array()` converts a list to an array.\n",
    "\n",
    "#### Now Try This\n",
    "1. Run the code block below. How do you know that B is an array of decimal numbers, whereas A contains integers? What happens when one element of an array is a decimal number?\n",
    "2. What happens when you add A and B? What happens when you multiply A and B?\n",
    "\n",
    "\n",
    "You can access a particular element of a list or an array using _indexing_. In indexing, you tell Python which item in a list or an array you want to print, change, or otherwise use. To access the first element of array A, you would type `A[0]`, because the numbering of elements starts at 0. So `A` has items numbered from 0 to 2. In multidimensional arrays, you specify elements using two or more \"coordinates\": for example `B[1,1]`.\n",
    "\n",
    "#### Now Try This\n",
    "1. What do you guess element [2,1] of array B will be? Print out element [2,1] of array B.\n",
    "2. What error message do you see when you ask Python for A[3]?\n",
    "3. You can access part of an array by *slicing*, or designating a range of elements. For example, A[0:2] will give an array with the first two elements of A (A[0] and A[1], where the elements are *below 2*). How would you access the bottom right 2x2 grid from array B? \n",
    "\n",
    "Array Attributes\n",
    "-----------------\n",
    "\n",
    "In addition to their contents, arrays have a set of useful _attributes_, pieces of information about them that you might want to know. Attributes are accessed by a dot and the name of the attribute. Here's a list of some of the useful attributes [here](https://docs.scipy.org/doc/numpy-dev/user/quickstart.html). \n",
    "\n",
    "#### Now Try This\n",
    "1. Find the `shape` (number of rows and columns), `size` (number of elements), and `dtype` (data type) of both arrays B and A. For example, for the shape, type 'A.shape()'.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n",
      "[[ 1.  2.  3.]\n",
      " [ 4.  5.  6.]\n",
      " [ 7.  8.  9.]]\n"
     ]
    }
   ],
   "source": [
    "A = np.array([1, 2, 3])\n",
    "print (A)\n",
    "\n",
    "B = np.array([[1, 2, 3],[4, 5, 6],[7,8,9.0]])\n",
    "print (B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Pandas Data Frames\n",
    "-------------------\n",
    "Pandas is a step above Numpy when it comes to analyzing large datasets with multiple columns of different kinds of information. This is great for Excel files with sample IDs, measurements, times, dates, etc. Data frames are kind of like data tables in publications. Each column has one kind of data in it.\n",
    "\n",
    "To create a data frame from an array, you just use the `pd.DataFrame()` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     0    1    2\n",
      "0  1.0  2.0  3.0\n",
      "1  4.0  5.0  6.0\n",
      "2  7.0  8.0  9.0\n"
     ]
    }
   ],
   "source": [
    "df_B=pd.DataFrame(B)\n",
    "print(df_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's more helpful if the columns in your data frame have meaningful names. Let's name them using the `columns` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   first  second  third\n",
      "0    1.0     2.0    3.0\n",
      "1    4.0     5.0    6.0\n",
      "2    7.0     8.0    9.0\n"
     ]
    }
   ],
   "source": [
    "df_B.columns=['first','second','third']\n",
    "print(df_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access a column, you specify its name in the same way you specify an element in an array: using the `[]` brackets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    3.0\n",
      "1    6.0\n",
      "2    9.0\n",
      "Name: third, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(df_B['third'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Technically, each column is called a pandas `Series`. We'll just call them columns.\n",
    "\n",
    "If you want to access a particular element in a data frame, use `.loc[]` or `.iloc[]` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.0\n"
     ]
    }
   ],
   "source": [
    "print(df_B.loc[1,'second'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n"
     ]
    }
   ],
   "source": [
    "print(df_B.iloc[0,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's the difference? The `.loc[]` attribute lets you use the names of rows and columns, whereas `.iloc[]` uses indexes (like Numpy does with arrays).\n",
    "\n",
    "You can use slices with `.loc[]` and `.iloc[]` as well. Note that they work differently with the two attributes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1    6.0\n",
      "2    9.0\n",
      "Name: third, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "print(df_B.iloc[1:3,2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   first  second\n",
      "0    1.0     2.0\n",
      "1    4.0     5.0\n"
     ]
    }
   ],
   "source": [
    "print(df_B.loc[0:1,'first':'second'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading Data Files in Pandas\n",
    "------------------------------\n",
    "One of the great things about pandas is its ability to read files and manipulate them as data frames. This allows you to do the kinds of tasks you do in Excel, but faster. Let's read the Lake Elgygytgyn data file into memory as a data frame. We do this using the `read_csv()` command. This command takes several *parameters* - aspects of the command you need to specify (designated in parentheses):\n",
    "1. The file name (`'elgygytgyn2013_reconstruction.csv'`). This is the first parameter you specify.\n",
    "2. The number of rows to skip. This is a *named* parameter: you need to specify \"`skiprows=`\" to tell pandas to skip this many lines at the beginning of the file. Other useful named parameters include `sep` (column separator; useful if you have something other than commas between the columns in a file), `comment` (set this if you want to skip lines starting with a certain character) and `skipfooter` (skip this many lines at the end)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     Age [ka BP]  Mean Temperature, warmest month [deg C]  \\\n",
      "0       2150.300                                 5.640586   \n",
      "1       2152.460                                 9.207488   \n",
      "2       2155.000                                10.511760   \n",
      "3       2156.820                                 7.843209   \n",
      "4       2157.310                                10.720896   \n",
      "5       2159.330                                11.119363   \n",
      "6       2160.320                                10.217688   \n",
      "7       2162.620                                11.517836   \n",
      "8       2165.180                                11.804156   \n",
      "9       2167.740                                12.404994   \n",
      "10      2169.170                                12.837006   \n",
      "11      2175.220                                12.327336   \n",
      "12      2177.140                                12.168423   \n",
      "13      2179.050                                12.470264   \n",
      "14      2182.240                                12.741446   \n",
      "15      2183.930                                12.650268   \n",
      "16      2185.620                                12.806875   \n",
      "17      2187.310                                12.816709   \n",
      "18      2189.080                                12.270645   \n",
      "19      2192.000                                12.218701   \n",
      "20      2193.020                                11.717243   \n",
      "21      2196.910                                 9.050066   \n",
      "22      2199.190                                11.477641   \n",
      "23      2201.020                                10.536647   \n",
      "24      2202.680                                11.665339   \n",
      "25      2205.345                                12.038355   \n",
      "26      2207.120                                12.039148   \n",
      "27      2209.780                                12.676553   \n",
      "28      2211.550                                12.504494   \n",
      "29      2213.350                                11.888255   \n",
      "..           ...                                      ...   \n",
      "347     3404.410                                16.737540   \n",
      "348     3405.480                                16.454261   \n",
      "349     3411.530                                16.697976   \n",
      "350     3416.000                                16.614803   \n",
      "351     3417.840                                16.581522   \n",
      "352     3420.630                                12.914661   \n",
      "353     3425.150                                13.851219   \n",
      "354     3428.350                                13.195106   \n",
      "355     3430.800                                15.133177   \n",
      "356     3437.400                                15.891887   \n",
      "357     3445.210                                12.531218   \n",
      "358     3454.770                                15.473760   \n",
      "359     3463.210                                13.016791   \n",
      "360     3467.770                                 9.920290   \n",
      "361     3472.440                                13.216081   \n",
      "362     3478.340                                15.678593   \n",
      "363     3483.330                                13.282589   \n",
      "364     3484.570                                15.684615   \n",
      "365     3492.430                                16.122378   \n",
      "366     3496.790                                15.430420   \n",
      "367     3508.040                                15.753448   \n",
      "368     3510.780                                15.428219   \n",
      "369     3515.710                                16.439388   \n",
      "370     3518.430                                15.251257   \n",
      "371     3523.360                                13.011299   \n",
      "372     3526.400                                15.322983   \n",
      "373     3535.550                                16.300991   \n",
      "374     3541.450                                15.169896   \n",
      "375     3552.770                                12.049301   \n",
      "376     3561.480                                15.689877   \n",
      "\n",
      "     Minimum Mean Temperature, warmest month [deg C]  \\\n",
      "0                                           2.678636   \n",
      "1                                           1.885963   \n",
      "2                                           7.962232   \n",
      "3                                           1.301907   \n",
      "4                                           8.203799   \n",
      "5                                           8.736924   \n",
      "6                                           6.723368   \n",
      "7                                           8.404997   \n",
      "8                                           9.372140   \n",
      "9                                           9.817912   \n",
      "10                                         10.706406   \n",
      "11                                          9.330589   \n",
      "12                                          9.975091   \n",
      "13                                          9.930171   \n",
      "14                                         10.878933   \n",
      "15                                         10.265249   \n",
      "16                                         11.202442   \n",
      "17                                         10.974084   \n",
      "18                                         10.828023   \n",
      "19                                         11.582788   \n",
      "20                                         11.421192   \n",
      "21                                          2.329098   \n",
      "22                                          7.385733   \n",
      "23                                          7.539287   \n",
      "24                                         10.799299   \n",
      "25                                         11.418984   \n",
      "26                                         10.618542   \n",
      "27                                         11.685299   \n",
      "28                                         10.517353   \n",
      "29                                         10.092746   \n",
      "..                                               ...   \n",
      "347                                        15.358932   \n",
      "348                                        15.140938   \n",
      "349                                        15.573217   \n",
      "350                                        15.260324   \n",
      "351                                        15.261901   \n",
      "352                                        10.818804   \n",
      "353                                        10.164776   \n",
      "354                                         9.814812   \n",
      "355                                        12.069606   \n",
      "356                                        14.034197   \n",
      "357                                         5.500000   \n",
      "358                                        12.778169   \n",
      "359                                        10.213754   \n",
      "360                                         7.740428   \n",
      "361                                        10.640471   \n",
      "362                                         8.698485   \n",
      "363                                        10.630669   \n",
      "364                                        15.505810   \n",
      "365                                        15.734821   \n",
      "366                                        12.754212   \n",
      "367                                        15.578930   \n",
      "368                                        12.607821   \n",
      "369                                        15.262727   \n",
      "370                                        11.500000   \n",
      "371                                        10.300000   \n",
      "372                                        12.162873   \n",
      "373                                        13.919873   \n",
      "374                                        12.708565   \n",
      "375                                        10.300000   \n",
      "376                                        10.969776   \n",
      "\n",
      "     Maximum Mean Temperature, warmest month [deg C]  \\\n",
      "0                                          12.178636   \n",
      "1                                          14.785963   \n",
      "2                                          15.062232   \n",
      "3                                          14.201907   \n",
      "4                                          14.503799   \n",
      "5                                          12.136924   \n",
      "6                                          13.823368   \n",
      "7                                          14.704997   \n",
      "8                                          13.472140   \n",
      "9                                          16.017912   \n",
      "10                                         14.806406   \n",
      "11                                         16.130589   \n",
      "12                                         13.375091   \n",
      "13                                         13.330171   \n",
      "14                                         16.778933   \n",
      "15                                         17.065249   \n",
      "16                                         16.802442   \n",
      "17                                         17.074084   \n",
      "18                                         13.328023   \n",
      "19                                         14.082788   \n",
      "20                                         12.221192   \n",
      "21                                         15.229098   \n",
      "22                                         13.685733   \n",
      "23                                         14.639287   \n",
      "24                                         13.999299   \n",
      "25                                         14.618984   \n",
      "26                                         14.018542   \n",
      "27                                         13.385299   \n",
      "28                                         13.917353   \n",
      "29                                         13.892746   \n",
      "..                                               ...   \n",
      "347                                        17.858932   \n",
      "348                                        17.640938   \n",
      "349                                        17.373217   \n",
      "350                                        17.560324   \n",
      "351                                        17.561901   \n",
      "352                                        15.618804   \n",
      "353                                        16.664776   \n",
      "354                                        15.614812   \n",
      "355                                        16.569606   \n",
      "356                                        16.134197   \n",
      "357                                        16.800000   \n",
      "358                                        16.578169   \n",
      "359                                        15.513754   \n",
      "360                                        14.640428   \n",
      "361                                        16.440471   \n",
      "362                                        18.198485   \n",
      "363                                        16.530669   \n",
      "364                                        15.705810   \n",
      "365                                        17.734821   \n",
      "366                                        17.954212   \n",
      "367                                        15.778930   \n",
      "368                                        17.807821   \n",
      "369                                        17.562727   \n",
      "370                                        15.800000   \n",
      "371                                        15.400000   \n",
      "372                                        17.362873   \n",
      "373                                        17.519873   \n",
      "374                                        17.508565   \n",
      "375                                        15.100000   \n",
      "376                                        17.669776   \n",
      "\n",
      "     Precipitation, annual mean [mm]  Minimum Precipitation, annual mean [mm]  \\\n",
      "0                         238.477111                               150.609472   \n",
      "1                         262.107504                               156.246880   \n",
      "2                         260.865459                               208.977058   \n",
      "3                         260.351720                               180.263587   \n",
      "4                         259.293757                               210.501706   \n",
      "5                         269.803083                               226.075946   \n",
      "6                         260.734227                               219.461385   \n",
      "7                         277.003559                               224.170555   \n",
      "8                         270.993506                               230.852064   \n",
      "9                         259.030959                               190.876054   \n",
      "10                        251.668945                               229.521375   \n",
      "11                        275.362289                               208.297980   \n",
      "12                        269.767584                               200.343232   \n",
      "13                        283.300683                               201.359029   \n",
      "14                        292.786410                               199.800605   \n",
      "15                        288.392084                               226.115184   \n",
      "16                        288.284782                               205.534841   \n",
      "17                        291.521727                               192.049625   \n",
      "18                        284.374415                               235.441259   \n",
      "19                        286.736846                               246.381958   \n",
      "20                        285.581447                               256.880359   \n",
      "21                        243.930166                               156.628958   \n",
      "22                        252.852330                               220.528538   \n",
      "23                        281.811852                               219.834566   \n",
      "24                        266.795139                               221.080770   \n",
      "25                        278.660801                               217.206103   \n",
      "26                        284.655727                               227.418374   \n",
      "27                        283.345848                               193.436237   \n",
      "28                        271.756416                               198.591867   \n",
      "29                        278.496421                               240.056330   \n",
      "..                               ...                                      ...   \n",
      "347                       674.827273                               581.037697   \n",
      "348                       671.563817                               585.396889   \n",
      "349                       659.737168                               609.408848   \n",
      "350                       666.845675                               602.858346   \n",
      "351                       700.696439                               616.047690   \n",
      "352                       344.469443                               238.239392   \n",
      "353                       421.733203                               165.352977   \n",
      "354                       427.558867                               228.287773   \n",
      "355                       600.254271                               432.297915   \n",
      "356                       649.420450                               421.842493   \n",
      "357                       556.350876                               327.927000   \n",
      "358                       617.493121                               449.358219   \n",
      "359                       467.579815                               248.419357   \n",
      "360                       338.312292                               171.152123   \n",
      "361                       331.165239                               183.945322   \n",
      "362                       601.282518                               331.048270   \n",
      "363                       391.305892                               254.296385   \n",
      "364                       677.093946                               667.058109   \n",
      "365                       662.049581                               592.320535   \n",
      "366                       592.199596                               312.782961   \n",
      "367                       666.472788                               656.798532   \n",
      "368                       597.415749                               462.727236   \n",
      "369                       638.395850                               582.907484   \n",
      "370                       620.830728                               290.686000   \n",
      "371                       317.766176                               212.119000   \n",
      "372                       621.771942                               497.668850   \n",
      "373                       630.333045                               362.692443   \n",
      "374                       516.306328                               197.615571   \n",
      "375                       372.617990                               212.119000   \n",
      "376                       649.845366                               327.349627   \n",
      "\n",
      "     Maximum Precipitation, annual mean [mm]  Trees & Shrubs [%]  Picea [%]   \n",
      "0                                 347.795471            8.058608    0.000000  \n",
      "1                                 353.432880           20.425532    0.000000  \n",
      "2                                 320.479059           23.484848    0.000000  \n",
      "3                                 329.738587           24.045802    0.000000  \n",
      "4                                 309.687706           43.231441    0.000000  \n",
      "5                                 313.259946           54.183267    0.000000  \n",
      "6                                 293.722385           31.418919    0.000000  \n",
      "7                                 323.356555           54.618474    0.000000  \n",
      "8                                 323.018064           64.960630    0.787402  \n",
      "9                                 460.749054           82.175227    4.229607  \n",
      "10                                269.965375           80.851064    0.000000  \n",
      "11                                342.887980           87.632509    0.000000  \n",
      "12                                317.201232           80.633803    0.000000  \n",
      "13                                332.812029           85.551331    0.380228  \n",
      "14                                319.795605           87.500000    0.000000  \n",
      "15                                320.261184           85.833333    0.416667  \n",
      "16                                322.392841           85.815603    0.000000  \n",
      "17                                320.407625           91.266376    0.000000  \n",
      "18                                305.990259           81.218274    0.000000  \n",
      "19                                316.930958           82.061069    0.000000  \n",
      "20                                327.429360           67.399267    0.000000  \n",
      "21                                316.573958           26.943005    0.000000  \n",
      "22                                312.694538           41.365462    0.000000  \n",
      "23                                331.336566           40.000000    0.000000  \n",
      "24                                295.341770           57.142857    0.000000  \n",
      "25                                328.452103           71.008403    0.000000  \n",
      "26                                318.427374           81.632653    0.000000  \n",
      "27                                324.889237           86.554622    0.000000  \n",
      "28                                330.044867           86.666667    0.350877  \n",
      "29                                331.065330           77.573529    0.000000  \n",
      "..                                       ...                 ...         ...  \n",
      "347                               793.484697           90.612245   19.591837  \n",
      "348                               780.339889           94.964029   25.539568  \n",
      "349                               804.351848           95.275591   14.566929  \n",
      "350                               797.801346           95.261845   22.693267  \n",
      "351                               810.486690           83.653846   16.346154  \n",
      "352                               718.180392           67.222222    2.777778  \n",
      "353                               734.509977           72.180451    6.766917  \n",
      "354                               708.228773           71.428571    6.122449  \n",
      "355                               710.895915           75.418994   11.173184  \n",
      "356                               684.296493           83.766234   13.636364  \n",
      "357                               781.276000           69.642857   16.071429  \n",
      "358                               673.924219           95.454545   47.933884  \n",
      "359                               740.190357           75.000000   20.312500  \n",
      "360                               639.532123           64.673913    5.434783  \n",
      "361                               663.886322           76.884422    5.527638  \n",
      "362                               784.397270           90.322581   35.483871  \n",
      "363                               734.237385           84.883721    9.302326  \n",
      "364                               682.441109           93.562232   20.600858  \n",
      "365                               677.437535           88.127854   25.570776  \n",
      "366                               737.903961           93.527508   38.511327  \n",
      "367                               672.181532           93.721973   20.179372  \n",
      "368                               698.817236           92.086331   34.532374  \n",
      "369                               700.553483           93.277311   24.369748  \n",
      "370                               671.954000           87.203791   15.639810  \n",
      "371                               638.561000           89.552239   11.044776  \n",
      "372                               743.901850           83.471074   41.735537  \n",
      "373                               771.021443           93.506494   10.389610  \n",
      "374                               649.443571           84.905660   28.301887  \n",
      "375                               692.060000           77.272727    4.545455  \n",
      "376                               788.435627           75.862069   17.241379  \n",
      "\n",
      "[377 rows x 9 columns]\n"
     ]
    }
   ],
   "source": [
    "elgy=pd.read_csv('elgygytgyn2013_reconstruction.csv',skiprows=12)\n",
    "print(elgy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...that's a big file. We can just see the first few lines of the data frame using `.head()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Age [ka BP]  Mean Temperature, warmest month [deg C]  \\\n",
      "0      2150.30                                 5.640586   \n",
      "1      2152.46                                 9.207488   \n",
      "2      2155.00                                10.511760   \n",
      "3      2156.82                                 7.843209   \n",
      "4      2157.31                                10.720896   \n",
      "\n",
      "   Minimum Mean Temperature, warmest month [deg C]  \\\n",
      "0                                         2.678636   \n",
      "1                                         1.885963   \n",
      "2                                         7.962232   \n",
      "3                                         1.301907   \n",
      "4                                         8.203799   \n",
      "\n",
      "   Maximum Mean Temperature, warmest month [deg C]  \\\n",
      "0                                        12.178636   \n",
      "1                                        14.785963   \n",
      "2                                        15.062232   \n",
      "3                                        14.201907   \n",
      "4                                        14.503799   \n",
      "\n",
      "   Precipitation, annual mean [mm]  Minimum Precipitation, annual mean [mm]  \\\n",
      "0                       238.477111                               150.609472   \n",
      "1                       262.107504                               156.246880   \n",
      "2                       260.865459                               208.977058   \n",
      "3                       260.351720                               180.263587   \n",
      "4                       259.293757                               210.501706   \n",
      "\n",
      "   Maximum Precipitation, annual mean [mm]  Trees & Shrubs [%]  Picea [%]   \n",
      "0                               347.795471            8.058608         0.0  \n",
      "1                               353.432880           20.425532         0.0  \n",
      "2                               320.479059           23.484848         0.0  \n",
      "3                               329.738587           24.045802         0.0  \n",
      "4                               309.687706           43.231441         0.0  \n"
     ]
    }
   ],
   "source": [
    "print(elgy.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say we wanted to know the range in precipitation between the minimum and maximum, year by year. All you have to do is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      197.186\n",
       "1      197.186\n",
       "2      111.502\n",
       "3      149.475\n",
       "4       99.186\n",
       "5       87.184\n",
       "6       74.261\n",
       "7       99.186\n",
       "8       92.166\n",
       "9      269.873\n",
       "10      40.444\n",
       "11     134.590\n",
       "12     116.858\n",
       "13     131.453\n",
       "14     119.995\n",
       "15      94.146\n",
       "16     116.858\n",
       "17     128.358\n",
       "18      70.549\n",
       "19      70.549\n",
       "20      70.549\n",
       "21     159.945\n",
       "22      92.166\n",
       "23     111.502\n",
       "24      74.261\n",
       "25     111.246\n",
       "26      91.009\n",
       "27     131.453\n",
       "28     131.453\n",
       "29      91.009\n",
       "        ...   \n",
       "347    212.447\n",
       "348    194.943\n",
       "349    194.943\n",
       "350    194.943\n",
       "351    194.439\n",
       "352    479.941\n",
       "353    569.157\n",
       "354    479.941\n",
       "355    278.598\n",
       "356    262.454\n",
       "357    453.349\n",
       "358    224.566\n",
       "359    491.771\n",
       "360    468.380\n",
       "361    479.941\n",
       "362    453.349\n",
       "363    479.941\n",
       "364     15.383\n",
       "365     85.117\n",
       "366    425.121\n",
       "367     15.383\n",
       "368    236.090\n",
       "369    117.646\n",
       "370    381.268\n",
       "371    426.442\n",
       "372    246.233\n",
       "373    408.329\n",
       "374    451.828\n",
       "375    479.941\n",
       "376    461.086\n",
       "Length: 377, dtype: float64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elgy['Maximum Precipitation, annual mean [mm]']-elgy['Minimum Precipitation, annual mean [mm]']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    197.186\n",
      "1    197.186\n",
      "2    111.502\n",
      "3    149.475\n",
      "4     99.186\n",
      "Name: Precipitation Range (mm), dtype: float64\n"
     ]
    }
   ],
   "source": [
    "elgy['Precipitation Range (mm)']=elgy['Maximum Precipitation, annual mean [mm]']-elgy['Minimum Precipitation, annual mean [mm]']\n",
    "print(elgy['Precipitation Range (mm)'].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and if we wanted the average, standard deviation, min, and max of all the precipitation ranges?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average:  205.96519628647218\n",
      "Std Dev:  166.0245460289239\n",
      "Min:  15.383\n",
      "Max:  582.756\n"
     ]
    }
   ],
   "source": [
    "print('Average: ',elgy['Precipitation Range (mm)'].mean())\n",
    "print('Std Dev: ',elgy['Precipitation Range (mm)'].std())\n",
    "print('Min: ',elgy['Precipitation Range (mm)'].min())\n",
    "print('Max: ',elgy['Precipitation Range (mm)'].max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we wanted the average amount of tree and shrub pollen in this reconstruction, starting 2.5 million years ago? We'd have to find the slice of the data frame where `Age [ka BP]` is less than 2500:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "56.8791720872578\n"
     ]
    }
   ],
   "source": [
    "print(elgy.loc[elgy['Age [ka BP]']<2500.,'Trees & Shrubs [%]'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make our request even more ridiculous. What if we wanted to average only the years when the warmest month was over 10$^\\circ$ C?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "62.35154920481818\n"
     ]
    }
   ],
   "source": [
    "print(elgy.loc[(elgy['Age [ka BP]']<2500.)&(elgy['Mean Temperature, warmest month [deg C]']>10.),'Trees & Shrubs [%]'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now Try This\n",
    "1. Find the average tree/shrub pollen content between 2.5 and 3.0 million years ago.\n",
    "2. What's *Picea* (look it up)? What are the maximum and minimum values in that column?\n",
    "3. What years have had over 1% *Picea* content?\n",
    "4. When did *Picea* reach its maximum?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}