pytorch_basics/pytorch_intro.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyN9y+zYt0XPYRKtiGRGNpAq",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jngadiub/ML_course_Pavia_23/blob/main/pytorch_basics/pytorch_intro.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Introduction to Pytorch\n",
        "\n",
        "Adapted from official [Pytorch tutorial](https://pytorch.org/tutorials/beginner/basics/intro.html) for dealing with PyTorch tensors, datasets, building neural networks etc., also has an accompanying video series."
      ],
      "metadata": {
        "id": "xOl7MvFyyOOV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tensors\n",
        "\n",
        "Tensors are a specialized data structure that are very similar to NumPy `ndarrays`. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters. Tensors are also optimized for automatic differentiation."
      ],
      "metadata": {
        "id": "D6ltEl0nzUXI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import numpy as np"
      ],
      "metadata": {
        "id": "lh3knnd-ahfZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Initializing a Tensor\n",
        "\n",
        "Tensors can be initialized in various ways. Take a look at the following examples:"
      ],
      "metadata": {
        "id": "OHiKPM0DapMU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Directly from data**\n",
        "\n",
        "Tensors can be created directly from data. The data type is automatically inferred."
      ],
      "metadata": {
        "id": "7VMS73xDatXQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data = [[1, 2],[3, 4]]\n",
        "x_data = torch.tensor(data)"
      ],
      "metadata": {
        "id": "akgMhYw8ayXS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**From a NumPy array**\n",
        "\n",
        "Tensors can be created from NumPy arrays (and vice versa)."
      ],
      "metadata": {
        "id": "jqBZTAHda8BY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "np_array = np.array(data)\n",
        "x_np = torch.from_numpy(np_array)"
      ],
      "metadata": {
        "id": "EE2gC9rIao5v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**From another tensor**\n",
        "\n",
        "The new tensor retains the properties (shape, datatype) of the argument tensor, unless explicitly overridden."
      ],
      "metadata": {
        "id": "fnqjE__vbKEq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x_ones = torch.ones_like(x_data) # retains the properties of x_data\n",
        "print(f\"Ones Tensor: \\n {x_ones} \\n\")\n",
        "\n",
        "x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data\n",
        "print(f\"Random Tensor: \\n {x_rand} \\n\")"
      ],
      "metadata": {
        "id": "nVhIJMk7bM-q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**With random or constant values**\n",
        "\n",
        "`shape` is a tuple of tensor dimensions. In the functions below, it determines the dimensionality of the output tensor."
      ],
      "metadata": {
        "id": "pc86VvLZbePF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "shape = (2,3,)\n",
        "rand_tensor = torch.rand(shape)\n",
        "ones_tensor = torch.ones(shape)\n",
        "zeros_tensor = torch.zeros(shape)\n",
        "\n",
        "print(f\"Random Tensor: \\n {rand_tensor} \\n\")\n",
        "print(f\"Ones Tensor: \\n {ones_tensor} \\n\")\n",
        "print(f\"Zeros Tensor: \\n {zeros_tensor}\")"
      ],
      "metadata": {
        "id": "Xoq55JAebjuG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Attributes of a Tensor\n",
        "\n",
        "Tensor attributes describe their shape, datatype, and the device on which they are stored."
      ],
      "metadata": {
        "id": "m2jfWJYvb4tU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tensor = torch.rand(3,4)\n",
        "\n",
        "print(f\"Shape of tensor: {tensor.shape}\")\n",
        "print(f\"Datatype of tensor: {tensor.dtype}\")\n",
        "print(f\"Device tensor is stored on: {tensor.device}\")"
      ],
      "metadata": {
        "id": "j-lqGiF0cAhS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Operations on Tensors\n",
        "\n",
        "Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing, indexing, slicing), sampling and more are comprehensively described [here](https://pytorch.org/docs/stable/torch.html).\n",
        "\n",
        "Each of these operations can be run on the GPU. By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using `.to` method (after checking for GPU availability)."
      ],
      "metadata": {
        "id": "aUsjyaH7cKB4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# We move our tensor to the GPU if available\n",
        "if torch.cuda.is_available():\n",
        "    print(\"Found GPU\")\n",
        "    tensor = tensor.to(\"cuda\")"
      ],
      "metadata": {
        "id": "QMkxXospchGt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Standard numpy-like indexing and slicing**"
      ],
      "metadata": {
        "id": "GTKR8atYcr_F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tensor = torch.rand(4, 4)\n",
        "print(tensor)\n",
        "print(f\"First row: {tensor[0,:]}\")\n",
        "print(f\"First column: {tensor[:,0]}\")\n",
        "print(f\"Last column: {tensor[:,-1]}\")\n",
        "tensor[:,1] = 0\n",
        "print(tensor)"
      ],
      "metadata": {
        "id": "rpdU44nMcu9Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Joining tensors**\n",
        "\n",
        "You can use `torch.cat` to concatenate a sequence of tensors along a given dimension."
      ],
      "metadata": {
        "id": "g5g0YabNdjXR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "t1 = torch.cat([tensor, tensor, tensor], dim=1)\n",
        "print(t1.shape)\n",
        "print(t1)"
      ],
      "metadata": {
        "id": "ll7u18V0dp7V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Arithmetic operations**"
      ],
      "metadata": {
        "id": "i86MON9seCYq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value\n",
        "# ``tensor.T`` returns the transpose of a tensor\n",
        "y1 = tensor @ tensor.T\n",
        "y2 = tensor.matmul(tensor.T)\n",
        "\n",
        "y3 = torch.rand_like(y1)\n",
        "torch.matmul(tensor, tensor.T, out=y3)\n",
        "\n",
        "\n",
        "# This computes the element-wise product. z1, z2, z3 will have the same value\n",
        "z1 = tensor * tensor\n",
        "z2 = tensor.mul(tensor)\n",
        "\n",
        "z3 = torch.rand_like(tensor)\n",
        "torch.mul(tensor, tensor, out=z3)"
      ],
      "metadata": {
        "id": "e-GD0INYeGe6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Single-element tensors**\n",
        "\n",
        "If you have a one-element tensor, for example by aggregating all values of a tensor into one value, you can convert it to a Python numerical value using `item()`:"
      ],
      "metadata": {
        "id": "Os1ZcNNrx3x_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "agg = tensor.sum()\n",
        "agg_item = agg.item()\n",
        "print(agg_item, type(agg_item))"
      ],
      "metadata": {
        "id": "K0YYD_QFx9hi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**In-place operations**\n",
        "\n",
        "Operations that store the result into the operand are called in-place. They are denoted by a `_` suffix. For example: `x.copy_(y)`, `x.t_()`, will change `x`."
      ],
      "metadata": {
        "id": "3eF6poF7yZV9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(f\"{tensor} \\n\")\n",
        "tensor.add_(5)\n",
        "print(tensor)"
      ],
      "metadata": {
        "id": "gGgGfsqVylCy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Bridge with NumPy\n",
        "\n",
        "Tensors on the CPU and NumPy arrays can share their underlying memory locations, and changing one will change the other."
      ],
      "metadata": {
        "id": "v3UR8cWty1Bn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Tensor to NumPy array**"
      ],
      "metadata": {
        "id": "VV5b4q_dy6lC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "t = torch.ones(5)\n",
        "print(f\"t: {t}\")\n",
        "n = t.numpy()\n",
        "print(f\"n: {n}\")"
      ],
      "metadata": {
        "id": "QnARXAHIy9Mf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "A change in the tensor reflects in the NumPy array."
      ],
      "metadata": {
        "id": "Oml36m6kzDuU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "t.add_(1)\n",
        "print(f\"t: {t}\")\n",
        "print(f\"n: {n}\")"
      ],
      "metadata": {
        "id": "-ck0MgA_zEIu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**NumPy array to Tensor**"
      ],
      "metadata": {
        "id": "6QMiDGoszJ88"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "n = np.ones(5)\n",
        "t = torch.from_numpy(n)"
      ],
      "metadata": {
        "id": "D4gjyUhDzLhk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Changes in the NumPy array reflects in the tensor."
      ],
      "metadata": {
        "id": "t7AIcf7LzNzl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "np.add(n, 1, out=n)\n",
        "print(f\"t: {t}\")\n",
        "print(f\"n: {n}\")"
      ],
      "metadata": {
        "id": "0HSxoc7pzPYU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Working with data\n",
        "\n",
        "PyTorch has [two primitives to work with data](https://pytorch.org/docs/stable/data.html): `torch.utils.data.DataLoader` and `torch.utils.data.Dataset`:\n",
        "* **Dataset** stores the samples and their corresponding labels;\n",
        "* **DataLoader** wraps an iterable around the `Dataset` to enable easy access to the samples.\n",
        "\n",
        "PyTorch domain libraries provide a number of pre-loaded datasets (such as FashionMNIST) that subclass `torch.utils.data.Dataset` and implement functions specific to the particular data. They can be used to prototype and benchmark your model. You can find them here: [Image Datasets](https://pytorch.org/vision/stable/datasets.html), [Text Datasets](https://pytorch.org/text/stable/datasets.html), and [Audio Datasets](https://pytorch.org/audio/stable/datasets.html)."
      ],
      "metadata": {
        "id": "xsmgaDDazbiF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Loading a Dataset\n",
        "\n",
        "Here is an example of how to load the [Fashion-MNIST dataset](https://github.com/zalandoresearch/fashion-mnist) from [TorchVision](https://pytorch.org/vision/stable/datasets.html). Fashion-MNIST is a dataset of Zalando’s article images consisting of 60,000 training examples and 10,000 test examples. Each example comprises a 28×28 grayscale image and an associated label from one of 10 classes.\n",
        "\n",
        "We load the [FashionMNIST Dataset](https://pytorch.org/vision/stable/datasets.html#fashion-mnist) with the following parameters:\n",
        "\n",
        "* `root` is the path where the train/test data is stored;\n",
        "* `train` specifies training or test dataset;\n",
        "* `download=True` downloads the data from the internet if it’s not available at `root`;\n",
        "* `transform` and `target_transform` specify the feature and label transformations.\n"
      ],
      "metadata": {
        "id": "Chww-ZVMzqPv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from torch.utils.data import Dataset\n",
        "from torchvision import datasets\n",
        "from torchvision.transforms import ToTensor, Lambda\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "training_data = datasets.FashionMNIST(\n",
        "    root=\"data\",\n",
        "    train=True,\n",
        "    download=True,\n",
        "    transform=ToTensor()\n",
        ")\n",
        "\n",
        "test_data = datasets.FashionMNIST(\n",
        "    root=\"data\",\n",
        "    train=False,\n",
        "    download=True,\n",
        "    transform=ToTensor()\n",
        ")"
      ],
      "metadata": {
        "id": "0P0z5TOdz2jf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Iterating and Visualizing the Dataset\n",
        "\n",
        "We can index `Datasets` manually like a list: `training_data[index]`. We use `matplotlib` to visualize some samples in our training data."
      ],
      "metadata": {
        "id": "J__5cWXL1IOi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "labels_map = {\n",
        "    0: \"T-Shirt\",\n",
        "    1: \"Trouser\",\n",
        "    2: \"Pullover\",\n",
        "    3: \"Dress\",\n",
        "    4: \"Coat\",\n",
        "    5: \"Sandal\",\n",
        "    6: \"Shirt\",\n",
        "    7: \"Sneaker\",\n",
        "    8: \"Bag\",\n",
        "    9: \"Ankle Boot\",\n",
        "}\n",
        "figure = plt.figure(figsize=(8, 8))\n",
        "cols, rows = 3, 3\n",
        "for i in range(1, cols * rows + 1):\n",
        "    sample_idx = torch.randint(len(training_data), size=(1,)).item()\n",
        "    img, label = training_data[sample_idx]\n",
        "    figure.add_subplot(rows, cols, i)\n",
        "    plt.title(labels_map[label])\n",
        "    plt.axis(\"off\")\n",
        "    plt.imshow(img.squeeze(), cmap=\"gray\")\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "zRq8kFBu1ox6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating a Custom Dataset for your files\n",
        "\n",
        "A custom `Dataset` class must implement three functions: `__init__`, `__len__`, and `__getitem__`. Take a look at this implementation; the FashionMNIST images are stored in a directory `img_dir`, and their labels are stored separately in a CSV file `annotations_file`.\n",
        "\n",
        "In the next sections, we’ll break down what’s happening in each of these functions."
      ],
      "metadata": {
        "id": "0ogeFMzZ15Qw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import pandas as pd\n",
        "from torchvision.io import read_image\n",
        "\n",
        "class CustomImageDataset(Dataset):\n",
        "    def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):\n",
        "        self.img_labels = pd.read_csv(annotations_file)\n",
        "        self.img_dir = img_dir\n",
        "        self.transform = transform\n",
        "        self.target_transform = target_transform\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.img_labels)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx, 0])\n",
        "        image = read_image(img_path)\n",
        "        label = self.img_labels.iloc[idx, 1]\n",
        "        if self.transform:\n",
        "            image = self.transform(image)\n",
        "        if self.target_transform:\n",
        "            label = self.target_transform(label)\n",
        "        return image, label"
      ],
      "metadata": {
        "id": "E5LyOQ-22Dum"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**`__init_`**\n",
        "\n",
        "The `__init__` function is run once when instantiating the `Dataset` object. We initialize the directory containing the images, the annotations file, and both transforms (covered in more detail in the next section).\n",
        "\n",
        "The `labels.csv` file looks like:\n",
        "\n",
        "```\n",
        "tshirt1.jpg, 0\n",
        "tshirt2.jpg, 0\n",
        "......\n",
        "ankleboot999.jpg, 9\n",
        "```\n",
        "\n",
        "**`__len__`**\n",
        "\n",
        "The `__len__` function returns the number of samples in our dataset.\n",
        "\n",
        "**`__getitem__`**\n",
        "\n",
        "The `__getitem__` function loads and returns a sample from the dataset at the given index `idx`. Based on the index, it identifies the image’s location on disk, converts that to a tensor using `read_image`, retrieves the corresponding label from the csv data in `self.img_labels`, calls the `transform` functions on them (if applicable), and returns the tensor image and corresponding label in a tuple."
      ],
      "metadata": {
        "id": "eF3dbHy72PGU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Preparing your data for training with DataLoaders\n",
        "\n",
        "The `Dataset` retrieves our dataset’s features and labels one sample at a time. While training a model, we typically want to pass samples in *minibatches*, reshuffle the data at every epoch to reduce model overfitting, and use Python’s multiprocessing to speed up data retrieval.\n",
        "\n",
        "`DataLoader` is an iterable that abstracts this complexity for us in an easy API."
      ],
      "metadata": {
        "id": "MwYgbBlS3g_8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "train_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)\n",
        "test_dataloader = DataLoader(test_data, batch_size=64, shuffle=True)"
      ],
      "metadata": {
        "id": "MqqX62f73tu7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Iterate through the DataLoader\n",
        "\n",
        "We have loaded that dataset into the `DataLoader` and can iterate through the dataset as needed. Each iteration below returns a batch of `train_features` and `train_labels` (containing `batch_size=64` features and labels respectively). Because we specified `shuffle=True`, after we iterate over all batches the data is shuffled (for finer-grained control over the data loading order, take a look at [Samplers](https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler))."
      ],
      "metadata": {
        "id": "nIp7wKup3xw3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display image and label.\n",
        "train_features, train_labels = next(iter(train_dataloader))\n",
        "print(f\"Feature batch shape: {train_features.size()}\")\n",
        "print(f\"Labels batch shape: {train_labels.size()}\")\n",
        "img = train_features[0].squeeze() #Returns a tensor with all the dimensions of input of size 1 removed\n",
        "label = train_labels[0]\n",
        "plt.imshow(img, cmap=\"gray\")\n",
        "plt.show()\n",
        "print(f\"Label: {label}\")"
      ],
      "metadata": {
        "id": "mfMgStmm4iks"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Transforms\n",
        "\n",
        "Data does not always come in its final processed form that is required for training machine learning algorithms. We use **transforms** to perform some manipulation of the data and make it suitable for training.\n",
        "\n",
        "All TorchVision datasets have two parameters: `transform` to modify the features and `target_transform` to modify the labels - that accept callables containing the transformation logic. The [`torchvision.transforms`](https://pytorch.org/vision/stable/transforms.html) module offers several commonly-used transforms out of the box.\n",
        "\n",
        "The FashionMNIST features are in Python Imaging Library (PIL) format, and the labels are integers. For training, we need the features as normalized tensors, and the labels as one-hot encoded tensors. To make these transformations, we use `ToTensor` and `Lambda`."
      ],
      "metadata": {
        "id": "jB4pE5hL57Ae"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ds = datasets.FashionMNIST(\n",
        "    root=\"data\",\n",
        "    train=True,\n",
        "    download=True,\n",
        "    transform=ToTensor(),\n",
        "    target_transform=Lambda(lambda y: torch.zeros(10, dtype=torch.float).scatter_(0, torch.tensor(y), value=1))\n",
        ")"
      ],
      "metadata": {
        "id": "kY7iscx96lR6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**ToTensor()**\n",
        "\n",
        "[`ToTensor`](https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.ToTensor) converts a PIL image or NumPy `ndarray` into a `FloatTensor`. and scales the image’s pixel intensity values in the range `[0., 1.]`\n",
        "\n",
        "**Lambda Transforms**\n",
        "\n",
        "`Lambda` transforms apply any user-defined lambda function. Here, we define a function to turn the integer into a one-hot encoded tensor. It first creates a zero tensor of size 10 (the number of labels in our dataset) and calls [`scatter_`](https://pytorch.org/docs/stable/generated/torch.Tensor.scatter_.html) which assigns a `value=1` on the index as given by the label `y`."
      ],
      "metadata": {
        "id": "FcWupzLI7CNL"
      }
    }
  ]
}