CMIYC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6b601c1d-7699-416f-95d4-2d13f9504250",
   "metadata": {},
   "source": [
    "![title](pics/cmiyc.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60189699-f9ab-47df-86a5-1e4d350ab622",
   "metadata": {},
   "source": [
    "# Re-Developing of *Crash Me If You Can*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ee02cc0-5c7c-47fc-96a5-511f6f6d3d77",
   "metadata": {},
   "source": [
    "**Assuming you'd have to re-develop the software for CMIYC, how would you proceed?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44827e68-08c4-4e23-a382-fd85c56a3445",
   "metadata": {},
   "source": [
    "**Given**:\n",
    "* All hardware, in particular,\n",
    "  * a car-mounted camera that constantly streams images from the race track (which you can easily access), and\n",
    "  * a way to control the speed for the car through simple software instructions.\n",
    "    \n",
    "**Objective**:\n",
    "* AI system that automatically adjusts the car speed based on traffic signs (speed limits, stop signs) next to the race track.\n",
    "  * The overall system thus consists of a DL model to recognize traffic signs and a (simple) rule-based AI to set the car speed based on the signs.\n",
    "  * The focus should be on the DL model for traffic sign recognition."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1093e3c7-39d5-48bc-b37f-a1421b4b9c33",
   "metadata": {},
   "source": [
    "**Examples**:\n",
    "\n",
    "<img src=\"pics/frame1.jpg\" alt=\"drawing\" width=\"350\"/><img src=\"pics/frame2.jpg\" alt=\"drawing\" width=\"350\"/><img src=\"pics/frame3.jpg\" alt=\"drawing\" width=\"350\"/>\n",
    "\n",
    "Left: car speed should be 50 km/h (default speed)<br>\n",
    "Middle: car speed should be temporarily set to 70 km/h<br>\n",
    "Right: car speed should be temporarily set to 30 km/h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "192469bb-3739-48d7-ae1a-7a56a5b3cdfd",
   "metadata": {},
   "source": [
    "Try to solve the task on a high level of abstraction. The following list of questions may serve as an inspiration, but is not exhaustive:\n",
    "* What data is needed? How can it be *efficiently* obtained/collected/labeled?\n",
    "* What must be changed in the MNIST handwritten digits example to accommodate the new problem setting, with RGB images and a different number of classes?\n",
    "* Is DL only needed for image classification, or are there other CV-related tasks that can/must be solved with DL in this case?\n",
    "* What could be the overall workflow / individual stages of the AI system?\n",
    "* Which data augmentation strategies might be useful?\n",
    "* How can traffic sign recognition be made more robust? In particular, we're not dealing with single, isolated images, but with video frames ...\n",
    "* How can the quality of the traffic sign recognition system be evaluated?\n",
    "* What are potential corner cases one must consider?\n",
    "* In an actual traffic sign recognition system for real cars, what might be additional challenges?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b8e45e0-ce1f-43ab-be43-6ad7adf44364",
   "metadata": {},
   "source": [
    "What do you think, which step takes most time?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d5afdec-b5d9-418b-b9ca-f4928ef2138d",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Our Solution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fef22bb-5c0b-4aac-9e0f-22b44fc3856b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from PIL import Image, ImageDraw\n",
    "from matplotlib import pyplot as plt\n",
    "from pathlib import Path\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "527ced36-6c90-4117-bc7d-1984ef3fe193",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cmiyc.classifier import Classifier\n",
    "from cmiyc.app import load_detector, CMIYCApp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba35a927-4d08-4215-b231-7bff75a3830b",
   "metadata": {},
   "source": [
    "## Overall Traffic Sign Recognition Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5f045fb-dfb4-4971-99f0-8349d86d6740",
   "metadata": {},
   "source": [
    "![pipeline](pics/cmiyc_pipeline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcb20771-213a-478c-8200-b51ea169ecbf",
   "metadata": {},
   "source": [
    "1. **Input**: RGB images of traffic scenes\n",
    "2. *Detector* locates traffic signs in images (bounding boxes) and classifies rough categories (warning, prohibitory, etc.)\n",
    "   * Neural network\n",
    "3. *Classifier* classifies each found traffic sign\n",
    "   * Neural network\n",
    "4. **Output**: Detailed information about each detected sign: location within image, class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab9b2672-e9f3-4d65-99f4-83b054117425",
   "metadata": {},
   "source": [
    "## Traffic Sign Detection"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "337827a6-9178-4211-9976-099a90946e2d",
   "metadata": {},
   "source": [
    "* Popular neural network architecture for general object detection in images: [YOLO-family](https://10.1109/CVPR.2016.91)\n",
    "  * **Y**ou **O**nly **L**ook **O**nce\n",
    "  * We used YOLOv4 (released 2020), current version is YOLOv9\n",
    "  * Convolutional neural network\n",
    "  * **Input**: $1600\\times 1600$ pixel RGB images\n",
    "  * **Output**: coordinates of bounding boxes of detected objects\n",
    "  * 5,883,356 trainable parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8663b759-59a2-4f81-bf63-4d9e1becb8e4",
   "metadata": {},
   "source": [
    "* Training data: [ATSD-Scenes](https://github.com/risc-mi/atsd)\n",
    "  * 7,454 high-res images from Austrian highways in 2014\n",
    "  * 28,000 detailed traffic sign annotations\n",
    "  * Created by RISC-SW and ASFINAG in research project *SafeSign*\n",
    "    ![atsd](pics/atsd.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91de5125-43e4-462b-913f-e1a31cc75a76",
   "metadata": {},
   "source": [
    "* Training and evaluation:\n",
    "  * Training takes about 10h on GPU\n",
    "  * Final performance on test set of ATSD-Scenes: $94.82\\%$ [mAP](https://www.v7labs.com/blog/mean-average-precision)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "080c506e-b00a-4209-a4ac-31c0a3c934dc",
   "metadata": {},
   "source": [
    "* Application in CMIYC:\n",
    "  * **Detector works exceptionally well also on CMIYC video frames** => model generalizes from highways to Carrera race tracks\n",
    "  * Restrict to at most one traffic sign per image, with largest bounding box"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc53941-3921-4dba-914a-d2d48f841387",
   "metadata": {},
   "source": [
    "Let's load the detector and apply it to some test images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69410600-d560-4e3c-8d87-f90f98e889b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "detector = load_detector()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d17be70-47d7-41ea-99c5-b462a0d31301",
   "metadata": {},
   "outputs": [],
   "source": [
    "img = Image.open('pics/frame1.jpg')\n",
    "img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3e9a6b8-fc52-4b10-aa4d-044d5b3206e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_result = detector.detect(np.asarray(img))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9880d7e6-f811-4248-bbbb-66976a333957",
   "metadata": {},
   "outputs": [],
   "source": [
    "detection_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "055b4a42-50ef-4246-b20c-833c2bea22ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize bounding box of all detected signs\n",
    "draw = ImageDraw.Draw(img)\n",
    "for bb in detection_result[2]:\n",
    "    draw.rectangle(\n",
    "        # convert bounding box coordinates from (left, top, width, height) to (left, top, right, bottom)\n",
    "        tuple(bb[:2]) + tuple(bb[:2] + bb[2:]),\n",
    "        width=6,\n",
    "        outline=(0, 255, 0)\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9440ddc9-bf9d-41d6-92d9-157cc8602ad5",
   "metadata": {},
   "outputs": [],
   "source": [
    "img"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8439080e-9efa-4ec9-acac-23395ea748bf",
   "metadata": {},
   "source": [
    "## Traffic Sign Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e684fad-a77a-4d7f-8b28-8d593c8e80ec",
   "metadata": {},
   "source": [
    "* Neural network architecture: [Li & Wang, 2019](https://doi.org/10.1109/TITS.2018.2843815)\n",
    "  * Convolutional neural network\n",
    "  * Similar architecture as MNIST classifier, but more layers (and more parameters)\n",
    "  * **Input**: $48\\times 48$ pixel RGB images\n",
    "  * **Output**: index of one of 19 traffic sign classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73cb1485-ed35-49bf-b8b7-67bb19e47ae6",
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = Classifier(19).eval()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e16be8fb-3a6b-4bfb-9a32-78efe7ac3b45",
   "metadata": {},
   "outputs": [],
   "source": [
    "sum(p.numel() for p in classifier.parameters())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78da7662-344e-4d69-9de8-03da38c6d8b4",
   "metadata": {},
   "source": [
    "* The network architecture can be summarized as follows:\n",
    "\n",
    "![network architecture](pics/cmiyc_classifier.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c135f818-0375-4968-9ec5-09efff2ecde1",
   "metadata": {},
   "source": [
    "* Training:\n",
    "  * Data: 18,402 images acquired with car on race track\n",
    "    * Using classifier trained on ATSD did not work very well, in contrast to detector\n",
    "  * Augmentation: rotation, zoom, noise, ...\n",
    "  * Training takes about 1 hour on GPU\n",
    "  * Final test-set performance: $99.55\\%$ accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80a3c1cf-0f56-4f3c-8e24-fc5619102528",
   "metadata": {},
   "source": [
    "Let's load the trained parameters and classify some test images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c8b3e92-f5e1-491a-92c5-2a9ba6836e2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier.load_state_dict(torch.load('cmiyc/classifier.pt'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f82d62e7-679a-4c99-9907-9f47eb9fdeb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "img = Image.open('pics/speed_limit.png')\n",
    "img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4b2337d-6841-4515-b6ca-17af47462b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "classification_result = classifier.classify(np.asarray(img))\n",
    "classification_result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83c29125-1356-447e-a220-e8c93001af1c",
   "metadata": {},
   "source": [
    "To see which traffic sign class this index corresponds to, we can visualize the template class image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0887bf0-33de-4e1b-8785-e287edafe07b",
   "metadata": {},
   "outputs": [],
   "source": [
    "Image.open('cmiyc/class_imgs/{}.png'.format(classification_result))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "536b14ce-0739-4efc-86c4-19e12279009b",
   "metadata": {},
   "source": [
    "## Webcam App"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85b32c89-5454-498a-a5de-01d673fa29a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "app = CMIYCApp(classifier=classifier, detector=detector, detect=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a00d714-bae3-47f1-8405-919362c9dcee",
   "metadata": {},
   "outputs": [],
   "source": [
    "app.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd72cc35-f2c7-4a81-9433-f11e505b56e5",
   "metadata": {},
   "source": [
    "**Note**: In the live-view window, press \"q\" to shut down the app."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22dd4c62-1d8b-46f2-ac9d-d4c7cfc3435b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Concluding Remarks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7c32dc2-ec61-4f5d-9adb-7c8b7cbe0932",
   "metadata": {},
   "source": [
    "* We've encountered two important CV tasks: *image classification* and *object detection*\n",
    "  * Many other tasks of varying complexity exist: semantic segmentation, instance segmentation, captioning, visual Q&A, image generation, ...\n",
    "* **Trustworthiness** is an important aspect of DL (and AI in general) and actively researched:\n",
    "  * Explainable AI (XAI): How does a DL model arrive at its predictions? What are the most important input features? How do these features contribute to the output?\n",
    "  * Uncertainty quantification: How (un)certain is the model about its predictions? Can we trust it?\n",
    "  * Out-of-distribution detection: Is the input drawn from the same distribution as the training data? Are there systematic differences? Maybe even adversarial attacks?\n",
    "* **Neurosymbolic AI** can bridge the gap between DL and symbolic AI:\n",
    "  * For instance, use DL to extract high-level semantic content from unstructured, fuzzy input data (e.g., images, natural language text), and process this content using symbolic AI and automated reasoning\n",
    "    * Example: In autonomous driving, DL can be used to process images and other sensor data, whereas symbolic AI controls car based on obtained information\n",
    "  * Different approach: tackle difficult problems from symbolic AI, e.g., symbolic integration, with DL\n",
    "  * Great potential for more robust, trustworthy, understandable AI!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f56bda0d-cd22-451a-95ea-668122b27c6c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "# Additional Material"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a21599b-a29a-42e4-a191-735eae56da08",
   "metadata": {},
   "source": [
    "## Public CV Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b190a7c-fe66-4725-80c5-668e69efa900",
   "metadata": {},
   "source": [
    "* MNIST: small gray-scale images of handwritten digits\n",
    "* [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist): 60,000 small gray-scale images of fashion items\n",
    "  * drop-in replacement for MNIST\n",
    "  * more challenging\n",
    "* [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html): 60,000 small RGB images (photographs), either 10 or 100 different classes\n",
    "* [ImageNet](https://image-net.org/index.php): 1.4M RGB images (photographs), 1000 different classes\n",
    "* [COCO](https://cocodataset.org/#home): 123,000 RGB images (photographs), detailed object-level annotations for detection and segmentation\n",
    "* ... and many, many more, from all kinds of domains and for all sorts of tasks\n",
    "\n",
    "**Note: Many of these datasets can easily be accessed (downloaded) with [torchvision](https://pytorch.org/vision/stable/datasets.html)!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae39d3d4-dcb9-4ecf-99ee-065f9054bb71",
   "metadata": {},
   "source": [
    "## Useful Technologies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "992553f0-68aa-4c3c-a5fb-700934345e98",
   "metadata": {},
   "source": [
    "* [torchvision](https://pytorch.org/vision/stable/index.html): Add-on library for PyTorch, very useful for computer vision:\n",
    "  * [public datasets](https://pytorch.org/vision/stable/datasets.html)\n",
    "  * [neural network architectures and pre-trained models](https://pytorch.org/vision/stable/models.html)\n",
    "  * [image augmentations](https://pytorch.org/vision/stable/transforms.html)\n",
    "* [PyTorch-Lightning](https://lightning.ai/docs/pytorch/stable/), [fastai](https://docs.fast.ai/): High-level interface to PyTorch, facilitates especially model training\n",
    "* [TensorBoard](https://www.tensorflow.org/tensorboard), [Wandb](https://wandb.ai/site): Browser-based real-time monitoring of training progress (loss, accuracy, etc.), and organizing/summarizing results\n",
    "  * seamlessly integrated into PyTorch-Lightning\n",
    "  * TensorBoard does not require TensorFlow installation\n",
    "* [Huggingface](https://huggingface.co/): Huge collection of trained ML models and curated datasets for all kinds of tasks in computer vision, natural language processing, etc.\n",
    "* [Kaggle](https://www.kaggle.com/): Huge collection of datasets, models, notebooks, competitions\n",
    "  * particularly nice for learning ML, DL, data science, etc.\n",
    "* [Docker](https://www.docker.com/): Lots of existing images for ML and DL, with Python, PyTorch and many other useful packages pre-installed\n",
    "* [Visual Studio Code](https://code.visualstudio.com/): Excellent Python IDE, with integrated Jupyter notebook support"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1b71ba4-2f05-4db9-8104-7fb90b57d139",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}