Assignment 0v2 - Numpy intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Short Numpy Tutorial\n",
    "\n",
    "This is a very short introduction to the library numpy (http://www.numpy.org), focused on one of its basic data structures, `ndarray`. Numpy is the most important scientific package in the Python ecosystem, because it provides a common datastructure on which many other packages are build on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Python scientific ecosystem](http://luispedro.org/files/talks/2013/EuBIAS/figures/sciwheel.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make this tutorial work on Python 3, let's import some future features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function, division"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# np is the standard abbreviation for numpy in the code\n",
    "# Even the numpy docs use it\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is an ndarray?\n",
    "\n",
    "The `ndarray` is the biggest contribution of numpy. An ndarray is\n",
    "\n",
    "- a regular grid of N-dimensions,\n",
    "- homogeneous by default (all the elements have the same type),\n",
    "- contiguous block of memory with types corresponding to machine types (8-bit ints, 32 bit floats, 64-bit longs, ...).\n",
    "- by default float64 is considered.\n",
    "\n",
    "### Building an array (inline)\n",
    "\n",
    "We can build an array explicitely from a Python list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.2 2.3 4. ]\n",
      " [1.2 3.4 5.2]\n",
      " [0.  1.  1.3]\n",
      " [0.  1.  0.2]]\n"
     ]
    }
   ],
   "source": [
    "arr = np.array([\n",
    "    [1.2, 2.3, 4.0],\n",
    "    [1.2, 3.4, 5.2],\n",
    "    [0.0, 1.0, 1.3],\n",
    "    [0.0, 1.0, 2e-1]])\n",
    "\n",
    "print(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** \n",
    "\n",
    "Check what parameters the array() command has e.g. how to specify the type of the array. \n",
    "\n",
    "Create a 2D matrix of 3 rows and 4 columns initializing with values chosen by you and specify the type of the elements as float.\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64\n"
     ]
    }
   ],
   "source": [
    "print (arr.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Check in [numpy](docs.scipy.org) what other ways there are to create an array?()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a three-dimensional array of 100x100x3 elements of type integer stored by 32 bits. How many ways to do it, do you find?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Help**: [Array creation.](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.creation.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspecting array properties"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64\n",
      "2\n",
      "(4, 3)\n"
     ]
    }
   ],
   "source": [
    "print(arr.dtype)\n",
    "print(arr.ndim)\n",
    "print(arr.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This array is of `float64`, it has 2 dimensions and its shape is 4 rows and 3 columns.\n",
    "\n",
    "When constructing an array, we can explicitly specify the type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n"
     ]
    }
   ],
   "source": [
    "iarr = np.array([1,2,3], dtype='uint8')\n",
    "print(iarr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arithmetic operations on the array : we should take into account that the type has to be respected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 3.    5.75 10.  ]\n",
      " [ 3.    8.5  13.  ]\n",
      " [ 0.    2.5   3.25]\n",
      " [ 0.    2.5   0.5 ]]\n",
      "[2 4 6]\n"
     ]
    }
   ],
   "source": [
    "arr *= 2.5\n",
    "iarr *= 2\n",
    "print(arr)\n",
    "print(iarr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ex: What is the problem of:\n",
    "\n",
    "`iarr *= 2.5 ?`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "UFuncTypeError",
     "evalue": "Cannot cast ufunc 'multiply' output from dtype('float64') to dtype('uint8') with casting rule 'same_kind'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mUFuncTypeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/z7/nxq36knx5rl5215wg26wwgkm0000gn/T/ipykernel_31074/904203291.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0miarr\u001b[0m \u001b[0;34m*=\u001b[0m \u001b[0;36m2.5\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mUFuncTypeError\u001b[0m: Cannot cast ufunc 'multiply' output from dtype('float64') to dtype('uint8') with casting rule 'same_kind'"
     ]
    }
   ],
   "source": [
    "iarr *= 2.5 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The solution is to convert the array elements to float.\n",
    "\n",
    "Check in [numpy](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.types.html) the different types and how to convert an array to a different type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5., 10., 15.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iarr.astype(float)*2.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Has the type of the `iarr` array chaged?\n",
    "Notice that numpy array creates variable with a certain type. If we do not consider it, our code will not work!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "uint8\n",
      "[2 4 6]\n"
     ]
    }
   ],
   "source": [
    "print(iarr.dtype)\n",
    "print(iarr)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Slicing & Dicing\n",
    "\n",
    "We can use Python's `[]` operator to slice and address the array:\n",
    "\n",
    "Below, you can see some examples of how we can read a matrix: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 3.    5.75 10.  ]\n",
      " [ 3.    8.5  13.  ]\n",
      " [ 0.    2.5   3.25]\n",
      " [ 0.    2.5   0.5 ]]\n",
      "3.0\n",
      "[ 3.   8.5 13. ]\n",
      "[10.   13.    3.25  0.5 ]\n"
     ]
    }
   ],
   "source": [
    "print(arr) # The whole matrix\n",
    "print(arr[0,0]) # 0 row, 0 column\n",
    "print(arr[1]) # The whole 1 row\n",
    "print(arr[:,2]) # The whole 2 column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with slices of an array.\n",
    "\n",
    "Slices share memory with the original array! In the following code, the variable `view` corresponds to a slice of the array `arr`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: 3.0\n",
      "After: 103.0\n"
     ]
    }
   ],
   "source": [
    "# The position arr[1,0] = x. If we move its value to view, and modify view, we can see how arr[1,0] also is modifed.\n",
    "\n",
    "print(\"Before: {}\".format(arr[1,0]))\n",
    "\n",
    "# adding 100\n",
    "view = arr[1]\n",
    "view[0] += 100\n",
    "\n",
    "print(\"After: {}\".format(arr[1,0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that by default Python assignation assumes sharing internal memory.\n",
    "\n",
    "**Exercise:** How can we avoid memory sharing between variables? We should use the `.copy()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: 103.0\n",
      "After: 103.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Before: {}\".format(arr[1,0]))\n",
    "view = arr.copy()[1]\n",
    "view[0] += 100\n",
    "print(\"After: {}\".format(arr[1,0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visual illustration of slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.array([\n",
    "       [ 0,  1,  2,  3,  4,  5],\n",
    "       [10, 11, 12, 13, 14, 15],\n",
    "       [20, 21, 22, 23, 24, 25],\n",
    "       [30, 31, 32, 33, 34, 35],\n",
    "       [40, 41, 42, 43, 44, 45],\n",
    "       [50, 51, 52, 53, 54, 55]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![slicing](https://scipy-lectures.github.io/_images/numpy_indexing.png)\n",
    "\n",
    "This image is taken from [scipy-lectures](https://scipy-lectures.github.io/intro/numpy/array_object.html), a more complete tutorial on numpy than what we have here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Boolean operations](https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.logic.html)\n",
    "\n",
    "An important subset of operations with numpy arrays concerns using logical operators to build boolean arrays. For example:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  3.     5.75  10.  ]\n",
      " [103.     8.5   13.  ]\n",
      " [  0.     2.5    3.25]\n",
      " [  0.     2.5    0.5 ]]\n",
      "[[ True  True  True]\n",
      " [ True  True  True]\n",
      " [False  True  True]\n",
      " [False  True False]]\n"
     ]
    }
   ],
   "source": [
    "print (arr)\n",
    "is_greater_one = (arr >= 1.)\n",
    "print(is_greater_one)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Put -100 in all elements of `arr` that are bigger than 10:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   3.      5.75   10.  ]\n",
      " [-100.      8.5  -100.  ]\n",
      " [   0.      2.5     3.25]\n",
      " [   0.      2.5     0.5 ]]\n"
     ]
    }
   ],
   "source": [
    "arr[(arr>10)] = -100\n",
    "print(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct a second array `arr2` that contains only the values of `arr` that are between 5 and 10:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5.75 8.5 ]\n"
     ]
    }
   ],
   "source": [
    "arr2 = arr[(arr>5)&(arr<10)]\n",
    "print(arr2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check what other [logical operations](https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.logic.html) there are in numpy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Basic functions on arrays](http://www.scipy-lectures.org/intro/numpy/operations.html)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-13.666666666666666"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also available: `max`, `min`, `sum`, `ptp` (point-to-point, i.e., difference between maximum and minimum values).\n",
    "\n",
    "These functions can also work *axis-wise*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-24.25  ,   4.8125, -21.5625])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arr.mean(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to *save* code lines, an important trick is to combine logical operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ True  True  True]\n",
      " [False  True False]\n",
      " [False  True  True]\n",
      " [False  True False]]\n",
      "0.5833333333333334\n"
     ]
    }
   ],
   "source": [
    "is_greater_one = (arr > 1)\n",
    "print(is_greater_one)\n",
    "print(is_greater_one.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Broadcasting\n",
    "\n",
    "You can often perform operations along the array rows or columns: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   3.      5.75   10.  ]\n",
      " [-100.      8.5  -100.  ]\n",
      " [   0.      2.5     3.25]\n",
      " [   0.      2.5     0.5 ]]\n",
      "Now adding [1,1,0] to *every row*\n",
      "\n",
      "[[   4.      6.75   10.  ]\n",
      " [ -99.      9.5  -100.  ]\n",
      " [   1.      3.5     3.25]\n",
      " [   1.      3.5     0.5 ]]\n"
     ]
    }
   ],
   "source": [
    "print(arr)\n",
    "print(\"Now adding [1,1,0] to *every row*\")\n",
    "print()\n",
    "arr += np.array([1,1,0])\n",
    "print(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the vector [1,2,3,4] to each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   4.      6.75   10.  ]\n",
      " [ -99.      9.5  -100.  ]\n",
      " [   1.      3.5     3.25]\n",
      " [   1.      3.5     0.5 ]]\n",
      "[[  5.     7.75  11.  ]\n",
      " [-97.    11.5  -98.  ]\n",
      " [  4.     6.5    6.25]\n",
      " [  5.     7.5    4.5 ]]\n"
     ]
    }
   ],
   "source": [
    "print(arr)\n",
    "arr3 = arr.transpose()\n",
    "arr3 += np.array([1,2,3,4])\n",
    "arr = arr3.transpose()\n",
    "print(arr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The exact [rules of how broadcasting works](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) are a bit complex to explain, but it generally works as expected. For example, if your data is a set of measurements for a sample, and your columns are the different types of measurements, then, you can easily remove the mean like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Footnotes\n",
    "\n",
    "[homogeneous]: There is a loophole to get heterogeneous arrays, namely an array of `object`. Then, you can store any Python object. This comes at the cost of decreased computational efficiency (both in terms of processing time and memory usage)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original matrix is:\n",
      "  [[1.2 2.3 4. ]\n",
      " [1.2 3.4 5.2]\n",
      " [0.  1.  1.3]\n",
      " [0.  1.  0.2]]\n",
      "\n",
      "The average value per column is:  [0.6   1.925 2.675]\n",
      "The average value per row is:  [2.5        3.26666667 0.76666667 0.4       ]\n",
      "\n",
      "The average value after subtracting the average values per row is: \n",
      "[[ 0.6    0.375  1.325]\n",
      " [ 0.6    1.475  2.525]\n",
      " [-0.6   -0.925 -1.375]\n",
      " [-0.6   -0.925 -2.475]]\n",
      "The average value after subtracting the average values per coloumn is: \n",
      "[[-1.3        -2.06666667 -0.76666667 -0.4       ]\n",
      " [-0.2         0.13333333  0.23333333  0.6       ]\n",
      " [ 1.5         1.93333333  0.53333333 -0.2       ]]\n",
      "\n",
      "1.7333333333333332\n",
      "The original matrix after normalizing is:\n",
      "  [[0.69230769 1.32692308 2.30769231]\n",
      " [0.69230769 1.96153846 3.        ]\n",
      " [0.         0.57692308 0.75      ]\n",
      " [0.         0.57692308 0.11538462]]\n"
     ]
    }
   ],
   "source": [
    "arr = np.array([\n",
    "    [1.2, 2.3, 4.0],\n",
    "    [1.2, 3.4, 5.2],\n",
    "    [0.0, 1.0, 1.3],\n",
    "    [0.0, 1.0, 2e-1]])\n",
    "print ('The original matrix is:\\n ', arr)\n",
    "print()\n",
    "print('The average value per column is: ',arr.mean(0))\n",
    "print('The average value per row is: ',arr.mean(1))\n",
    "print()\n",
    "# here we make a copy of the variable arr since we will modify it several times\n",
    "arr_aux1 = arr.copy()\n",
    "arr_aux2 = arr.copy()\n",
    "\n",
    "# we substract the average values to the whole matrix, first row based and later column based\n",
    "\n",
    "arr_aux1 -= arr.mean(0)\n",
    "print('The average value after subtracting the average values per row is: ') \n",
    "print(arr_aux1)\n",
    "\n",
    "arr_aux2 = arr_aux2.transpose()\n",
    "arr_aux2 -= arr.mean(1)\n",
    "print('The average value after subtracting the average values per coloumn is: ')\n",
    "print(arr_aux2)\n",
    "print()\n",
    "\n",
    "# The normalization is performed by dividing the matrix by its mean. \n",
    "print(arr.mean())\n",
    "print ('The original matrix after normalizing is:\\n ', arr/arr.mean())"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}