ML-Accountability/CO2-Emission-tracking/carbon_emission_cv.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hSBvPA0qiPxB"
      },
      "source": [
        "# Architectural choices in computer vision and their impact on energy consumption\n",
        "\n",
        "<a href=\"https://colab.research.google.com/drive/1G3tP5kLD1MUjdVVOpg3vDWeqpI0Q03gh\" target=\"_blank\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\">\n",
        "</a>\n",
        "\n",
        "Return to the [Castle](https://github.com/Nkluge-correa/TeenyTinyCastle)\n",
        "\n",
        "Convolutional neural networks (CNNs) have led to massive achievements and progress in Computer Vision. However, these networks come at a cost, given that training them can be relatively costly regarding resources like energy consumption. When you have a massive dataset -which is necessary when you aim to achieve extremely high performance - training such models results in substantial training times, which in the end may equate to non-trivial carbon emissions tied to the development of a CV model.\n",
        "\n",
        "The carbon emissions associated with training CNNs can vary depending on factors such as the network's size, the training process's length, and the energy efficiency of the hardware used. Currently, efforts are being made to develop more energy-efficient hardware and optimize training algorithms to reduce carbon emissions associated with training CNNs for computer vision. Something that is usually debated under the umbrella term of \"_Sustaineble AI_\".\n",
        "\n",
        "<img src=\"https://lh5.googleusercontent.com/prAfirs8L4UBqWWkX9dPoAEQwYHIJ0CLR9sUDNSrRMC44R3vXaQfGFycHjq68rT6Z5_B6sJJ9jlQmhun0adWX2BBfVFr6BZ8OFTXQskjPNqTBkPfl5ysmdMinxn7CPEgkGoXL1hT=s0\" alt=\"image\" width=\"600\">\n",
        "\n",
        "Source: [The Imperative for Sustainable AI Systems](https://thegradient.pub/sustainable-ai/).\n",
        "\n",
        "> For more information on the matter (Sustainable Computer Vision), we recommend \"_[Highlighting the Importance of Reducing Research Bias and Carbon Emissions in CNNs](https://arxiv.org/abs/2106.03242)_\".\n",
        "\n",
        "For this tutorial, we want to compare different CNN architectural choices. More specifically, we want to compare the energy consumption and carbon emissions generated by networks that use convolutional layers: `Conv2D` and `SeparableConv2D`.\n",
        "\n",
        "In other words, we want to compare how architectural choices can impact the carbon footprint related to training such models. We will be using [`CodeCarbon`](https://github.com/mlco2/codecarbon) for measuring our consumption and emissions, as already demonstrated in [this notebook](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Accountability/CO2-Emission-tracking/emission_tracker.ipynb).\n",
        "\n",
        "But first, let us understand wht is the difference between traditional convolutions (`Conv2D`) and depthwise separable convolutions (`SeparableConv2D`).\n",
        "\n",
        "## `Conv2D` versus `SeparableConv2D`\n",
        "\n",
        "Imagine you are a traditional convolutional layer trained on 15x15x3 pixel images. A forward pass on this layer will require more than 45,000 multiplications per image. Spatially separable convolutions help solve this problem. They are convolutions that can be separated across their spatial axis, meaning that one large convolution (e.g., the original `Conv2D` layer) can be split into smaller ones that, when convolved sequentially, produce the same result. For example, on our 15x15x3 pixel image, we would only require around 9,000 multiplications for the same result: an 80% decrease in multiplication operations!\n",
        "\n",
        "One gain of performing this sequential way to perform convolutions is a decrease in the number of multiplications. Less multiplication -> less computation -> less energy consumption.\n",
        "\n",
        "> **Note: To learn more about traditional convolutions and separable convolutions, we recommend the [following explanation](https://machinecurve.com/index.php/2019/09/23/understanding-separable-convolutions#how-many-multiplications-do-we-save). We also recommend the original article, \"_[Xception: Deep Learning with Depthwise Separable Convolutions](https://arxiv.org/abs/1610.02357)_\", where depthwise separable convolutions where introduced.**\n",
        "\n",
        "In principle, depthwise separable convolutional layers may significantly optimize your model's energy efficiency while preserving its performance. Let us test this. First, we create a standard convolutional network using the original `Conv2D` layer. Our model uses the  `Sequential`  API provided by Keras and stacks all layers on top of each other. We employ `Conv2D`  twice, followed by Max Pooling and Dropout, before we flatten the abstract feature map and classify the data using densely connected layers. Our loss function is categorical cross-entropy, and the optimizer is [Adam](https://paperswithcode.com/method/adam).\n",
        "\n",
        "For this tutorial, we will use the MNIST dataset to train our model, which we can load straight from TensorFlow."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lBbZWYnHiN8N",
        "outputId": "92abfedb-13d1-4978-9518-fb52d5774794"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11490434/11490434 [==============================] - 2s 0us/step\n",
            "Version:  2.15.0\n",
            "Eager mode:  True\n",
            "GPU is available\n",
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d (Conv2D)             (None, 26, 26, 32)        320       \n",
            "                                                                 \n",
            " max_pooling2d (MaxPooling2  (None, 13, 13, 32)        0         \n",
            " D)                                                              \n",
            "                                                                 \n",
            " dropout (Dropout)           (None, 13, 13, 32)        0         \n",
            "                                                                 \n",
            " conv2d_1 (Conv2D)           (None, 11, 11, 64)        18496     \n",
            "                                                                 \n",
            " max_pooling2d_1 (MaxPoolin  (None, 5, 5, 64)          0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " dropout_1 (Dropout)         (None, 5, 5, 64)          0         \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 1600)              0         \n",
            "                                                                 \n",
            " dense (Dense)               (None, 256)               409856    \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 10)                2570      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 431242 (1.65 MB)\n",
            "Trainable params: 431242 (1.65 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# Model configuration\n",
        "img_width, img_height = 28, 28\n",
        "batch_size = 250\n",
        "no_epochs = 25\n",
        "no_classes = 10\n",
        "validation_split = 0.2\n",
        "verbosity = 1\n",
        "\n",
        "# Load MNIST dataset\n",
        "(input_train, target_train), (input_test, target_test) = tf.keras.datasets.mnist.load_data()\n",
        "\n",
        "# Reshape the data\n",
        "input_train = input_train.reshape(input_train.shape[0], img_width, img_height, 1)\n",
        "input_test = input_test.reshape(input_test.shape[0], img_width, img_height, 1)\n",
        "input_shape = (img_width, img_height, 1)\n",
        "\n",
        "# Parse numbers as floats\n",
        "input_train = input_train.astype('float32')\n",
        "input_test = input_test.astype('float32')\n",
        "\n",
        "# Scale data\n",
        "input_train = input_train / 255\n",
        "input_test = input_test / 255\n",
        "\n",
        "# Convert target vectors to categorical targets\n",
        "target_train = tf.keras.utils.to_categorical(target_train, no_classes)\n",
        "target_test = tf.keras.utils.to_categorical(target_test, no_classes)\n",
        "\n",
        "# Create a CNN using `Conv2D` layers\n",
        "model_Conv2D = tf.keras.models.Sequential()\n",
        "model_Conv2D.add(tf.keras.layers.Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))\n",
        "model_Conv2D.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "model_Conv2D.add(tf.keras.layers.Dropout(0.25))\n",
        "model_Conv2D.add(tf.keras.layers.Conv2D(64, kernel_size=(3, 3), activation='relu'))\n",
        "model_Conv2D.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "model_Conv2D.add(tf.keras.layers.Dropout(0.25))\n",
        "model_Conv2D.add(tf.keras.layers.Flatten())\n",
        "model_Conv2D.add(tf.keras.layers.Dense(256, activation='relu'))\n",
        "model_Conv2D.add(tf.keras.layers.Dense(no_classes, activation='softmax'))\n",
        "\n",
        "# Compile the model\n",
        "model_Conv2D.compile(loss=tf.keras.losses.categorical_crossentropy,\n",
        "              optimizer=tf.keras.optimizers.Adam(),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Display a model summary\n",
        "print(\"Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n",
        "model_Conv2D.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2J6n8yNpijyr"
      },
      "source": [
        "In this tutorial, we will use the `EmissionsTracker` from CodeCarbon to track our experiments. To learm more on how to use CodeCarbon, visit this [tutorial](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Accountability/CO2-Emission-tracking/emission_tracker.ipynb).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p15VkXnrijEX",
        "outputId": "565ac929-95cc-4547-974d-e16a9652df20"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m181.6/181.6 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m66.4/66.4 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.1/53.1 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m43.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hEpoch 1/25\n",
            "192/192 [==============================] - 7s 10ms/step - loss: 0.3560 - accuracy: 0.8934 - val_loss: 0.0963 - val_accuracy: 0.9718\n",
            "Epoch 2/25\n",
            "192/192 [==============================] - 2s 9ms/step - loss: 0.0991 - accuracy: 0.9695 - val_loss: 0.0617 - val_accuracy: 0.9825\n",
            "Epoch 3/25\n",
            "192/192 [==============================] - 2s 9ms/step - loss: 0.0738 - accuracy: 0.9773 - val_loss: 0.0557 - val_accuracy: 0.9844\n",
            "Epoch 4/25\n",
            "192/192 [==============================] - 1s 8ms/step - loss: 0.0571 - accuracy: 0.9822 - val_loss: 0.0417 - val_accuracy: 0.9878\n",
            "Epoch 5/25\n",
            "192/192 [==============================] - 2s 8ms/step - loss: 0.0471 - accuracy: 0.9853 - val_loss: 0.0398 - val_accuracy: 0.9880\n",
            "Epoch 6/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0422 - accuracy: 0.9867 - val_loss: 0.0392 - val_accuracy: 0.9891\n",
            "Epoch 7/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0363 - accuracy: 0.9881 - val_loss: 0.0350 - val_accuracy: 0.9898\n",
            "Epoch 8/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0321 - accuracy: 0.9888 - val_loss: 0.0333 - val_accuracy: 0.9897\n",
            "Epoch 9/25\n",
            "192/192 [==============================] - 2s 9ms/step - loss: 0.0296 - accuracy: 0.9904 - val_loss: 0.0304 - val_accuracy: 0.9910\n",
            "Epoch 10/25\n",
            "192/192 [==============================] - 2s 13ms/step - loss: 0.0258 - accuracy: 0.9916 - val_loss: 0.0315 - val_accuracy: 0.9912\n",
            "Epoch 11/25\n",
            "192/192 [==============================] - 2s 10ms/step - loss: 0.0220 - accuracy: 0.9926 - val_loss: 0.0305 - val_accuracy: 0.9916\n",
            "Epoch 12/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0207 - accuracy: 0.9929 - val_loss: 0.0301 - val_accuracy: 0.9923\n",
            "Epoch 13/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0179 - accuracy: 0.9942 - val_loss: 0.0301 - val_accuracy: 0.9923\n",
            "Epoch 14/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0175 - accuracy: 0.9941 - val_loss: 0.0381 - val_accuracy: 0.9902\n",
            "Epoch 15/25\n",
            "192/192 [==============================] - 1s 8ms/step - loss: 0.0156 - accuracy: 0.9949 - val_loss: 0.0320 - val_accuracy: 0.9917\n",
            "Epoch 16/25\n",
            "192/192 [==============================] - 1s 8ms/step - loss: 0.0149 - accuracy: 0.9947 - val_loss: 0.0338 - val_accuracy: 0.9906\n",
            "Epoch 17/25\n",
            "192/192 [==============================] - 2s 10ms/step - loss: 0.0125 - accuracy: 0.9959 - val_loss: 0.0334 - val_accuracy: 0.9912\n",
            "Epoch 18/25\n",
            "192/192 [==============================] - 1s 8ms/step - loss: 0.0119 - accuracy: 0.9959 - val_loss: 0.0332 - val_accuracy: 0.9923\n",
            "Epoch 19/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0110 - accuracy: 0.9963 - val_loss: 0.0362 - val_accuracy: 0.9912\n",
            "Epoch 20/25\n",
            "192/192 [==============================] - 1s 8ms/step - loss: 0.0117 - accuracy: 0.9960 - val_loss: 0.0337 - val_accuracy: 0.9918\n",
            "Epoch 21/25\n",
            "192/192 [==============================] - 1s 8ms/step - loss: 0.0098 - accuracy: 0.9968 - val_loss: 0.0357 - val_accuracy: 0.9917\n",
            "Epoch 22/25\n",
            "192/192 [==============================] - 1s 8ms/step - loss: 0.0104 - accuracy: 0.9965 - val_loss: 0.0331 - val_accuracy: 0.9918\n",
            "Epoch 23/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0091 - accuracy: 0.9967 - val_loss: 0.0356 - val_accuracy: 0.9909\n",
            "Epoch 24/25\n",
            "192/192 [==============================] - 2s 8ms/step - loss: 0.0076 - accuracy: 0.9974 - val_loss: 0.0342 - val_accuracy: 0.9919\n",
            "Epoch 25/25\n",
            "192/192 [==============================] - 2s 10ms/step - loss: 0.0085 - accuracy: 0.9968 - val_loss: 0.0346 - val_accuracy: 0.9909\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0.001056536112822383"
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "!pip install codecarbon -q\n",
        "\n",
        "from codecarbon import EmissionsTracker\n",
        "\n",
        "tracker = EmissionsTracker(\n",
        "    project_name=\"Conv2D\",\n",
        "    log_level=\"critical\",\n",
        "    measure_power_secs=15,\n",
        "    output_dir=\"./\",\n",
        "    output_file=\"emissions-conv2d.csv\",\n",
        "    tracking_mode='machine',\n",
        ")\n",
        "\n",
        "tracker.start()\n",
        "\n",
        "# Train the model\n",
        "model_Conv2D.fit(input_train, target_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=no_epochs,\n",
        "          verbose=verbosity,\n",
        "          validation_split=validation_split)\n",
        "\n",
        "tracker.stop()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OaJOQQXfipV2"
      },
      "source": [
        "Now, let us introduce the  `SeparableConv2D`  convolutional layer in our model. The layer is very similar to the traditional  `Conv2D`  layer and can be added to your model easily, given that it is already implemented in the library. However, it comes with some separation-specific configuration options that must be set before training is commenced. The [Keras documentation](https://keras.io/api/layers/convolution_layers/separable_convolution2d/)  defines the  `SeparableConv2D`  layer as follows:\n",
        "\n",
        "```python\n",
        "keras.layers.SeparableConv2D(\n",
        "    filters,\n",
        "    kernel_size,\n",
        "    strides=(1, 1),\n",
        "    padding=\"valid\",\n",
        "    data_format=None,\n",
        "    dilation_rate=(1, 1),\n",
        "    depth_multiplier=1,\n",
        "    activation=None,\n",
        "    use_bias=True,\n",
        "    depthwise_initializer=\"glorot_uniform\",\n",
        "    pointwise_initializer=\"glorot_uniform\",\n",
        "    bias_initializer=\"zeros\",\n",
        "    depthwise_regularizer=None,\n",
        "    pointwise_regularizer=None,\n",
        "    bias_regularizer=None,\n",
        "    activity_regularizer=None,\n",
        "    depthwise_constraint=None,\n",
        "    pointwise_constraint=None,\n",
        "    bias_constraint=None,\n",
        "    **kwargs\n",
        ")\n",
        "```\n",
        "\n",
        "Where:\n",
        "\n",
        "-   **filters**: int, the dimensionality of the output space (i.e. the number of filters in the pointwise convolution).\n",
        "-   **kernel_size**: int or tuple/list of 2 integers, specifying the size of the depthwise convolution window.\n",
        "-   **strides**: int or tuple/list of 2 integers, specifying the stride length of the depthwise convolution. If only one int is specified, the same stride size will be used for all dimensions.  `strides > 1`  is incompatible with  `dilation_rate > 1`.\n",
        "-   **padding**: string, either  `\"valid\"`  or  `\"same\"`  (case-insensitive).  `\"valid\"`  means no padding.  `\"same\"`  results in padding evenly to the left/right or up/down of the input. When  `padding=\"same\"`  and  `strides=1`, the output has the same size as the input.\n",
        "-   **data_format**: string, either  `\"channels_last\"`  or  `\"channels_first\"`. The ordering of the dimensions in the inputs.  `\"channels_last\"`  corresponds to inputs with shape  `(batch, height, width, channels)`  while  `\"channels_first\"`  corresponds to inputs with shape  `(batch, channels, height, width)`. It defaults to the  `image_data_format`  value found in your Keras config file at  `~/.keras/keras.json`. If you never set it, then it will be  `\"channels_last\"`.\n",
        "-   **dilation_rate**: int or tuple/list of 2 integers, specifying the dilation rate to use for dilated convolution. If only one int is specified, the same dilation rate will be used for all dimensions.\n",
        "-   **depth_multiplier**: The number of depthwise convolution output channels for each input channel. The total number of depthwise convolution output channels will be equal to  `input_channel * depth_multiplier`.\n",
        "-   **activation**: Activation function. If  `None`, no activation is applied.\n",
        "-   **use_bias**: bool, if  `True`, bias will be added to the output.\n",
        "-   **depthwise_initializer**: An initializer for the depthwise convolution kernel. If None, then the default initializer (`\"glorot_uniform\"`) will be used.\n",
        "-   **pointwise_initializer**: An initializer for the pointwise convolution kernel. If None, then the default initializer (`\"glorot_uniform\"`) will be used.\n",
        "-   **bias_initializer**: An initializer for the bias vector. If None, the default initializer ('\"zeros\"') will be used.\n",
        "-   **depthwise_regularizer**: Optional regularizer for the depthwise convolution kernel.\n",
        "-   **pointwise_regularizer**: Optional regularizer for the pointwise convolution kernel.\n",
        "-   **bias_regularizer**: Optional regularizer for the bias vector.\n",
        "-   **activity_regularizer**: Optional regularizer function for the output.\n",
        "-   **depthwise_constraint**: Optional projection function to be applied to the depthwise kernel after being updated by an  `Optimizer`  (e.g. used for norm constraints or value constraints for layer weights). The function must take as input the unprojected variable and must return the projected variable (which must have the same shape).\n",
        "-   **pointwise_constraint**: Optional projection function to be applied to the pointwise kernel after being updated by an  `Optimizer`.\n",
        "-   **bias_constraint**: Optional projection function to be applied to the bias after being updated by an  `Optimizer`.\n",
        "\n",
        "Now that we understand how to set a depthwise separable convolutional layer in Keras, we can adapt our CNN from above to use depthwise separable convolutions, a.k.a., replace `Conv2D`  with  `SeparableConv2D`  and add the extra configuration that we need."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JG3jDamVipHY",
        "outputId": "1f83c150-d33c-4468-9068-170ebbb0b56d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Version:  2.15.0\n",
            "Eager mode:  True\n",
            "GPU is available\n",
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " separable_conv2d (Separabl  (None, 26, 26, 32)        73        \n",
            " eConv2D)                                                        \n",
            "                                                                 \n",
            " max_pooling2d_2 (MaxPoolin  (None, 13, 13, 32)        0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " dropout_2 (Dropout)         (None, 13, 13, 32)        0         \n",
            "                                                                 \n",
            " separable_conv2d_1 (Separa  (None, 11, 11, 64)        2400      \n",
            " bleConv2D)                                                      \n",
            "                                                                 \n",
            " max_pooling2d_3 (MaxPoolin  (None, 5, 5, 64)          0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " dropout_3 (Dropout)         (None, 5, 5, 64)          0         \n",
            "                                                                 \n",
            " flatten_1 (Flatten)         (None, 1600)              0         \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 256)               409856    \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 10)                2570      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 414899 (1.58 MB)\n",
            "Trainable params: 414899 (1.58 MB)\n",
            "Non-trainable params: 0 (0.00 Byte)\n",
            "_________________________________________________________________\n",
            "This model has 16343 parameters less!\n"
          ]
        }
      ],
      "source": [
        "# Create a CNN using `SeparableConv2D` layers\n",
        "model_SeparableConv2D = tf.keras.models.Sequential()\n",
        "model_SeparableConv2D.add(tf.keras.layers.SeparableConv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))\n",
        "model_SeparableConv2D.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "model_SeparableConv2D.add(tf.keras.layers.Dropout(0.25))\n",
        "model_SeparableConv2D.add(tf.keras.layers.SeparableConv2D(64, kernel_size=(3, 3), activation='relu'))\n",
        "model_SeparableConv2D.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "model_SeparableConv2D.add(tf.keras.layers.Dropout(0.25))\n",
        "model_SeparableConv2D.add(tf.keras.layers.Flatten())\n",
        "model_SeparableConv2D.add(tf.keras.layers.Dense(256, activation='relu'))\n",
        "model_SeparableConv2D.add(tf.keras.layers.Dense(no_classes, activation='softmax'))\n",
        "\n",
        "# Compile the model\n",
        "model_SeparableConv2D.compile(loss=tf.keras.losses.categorical_crossentropy,\n",
        "              optimizer=tf.keras.optimizers.Adam(),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "# Display a model summary\n",
        "print(\"Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n",
        "model_SeparableConv2D.summary()\n",
        "\n",
        "print(f\"This model has {model_Conv2D.count_params() - model_SeparableConv2D.count_params()} parameters less!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZOgRxiEfjXhy"
      },
      "source": [
        "Again, we use the `EmissionsTracker` from CodeCarbon to track our experiments."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bWini_WdkHoJ",
        "outputId": "448c23cd-7edb-4875-e0e8-a9991feef094"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/25\n",
            "192/192 [==============================] - 4s 10ms/step - loss: 0.5971 - accuracy: 0.8319 - val_loss: 0.1836 - val_accuracy: 0.9479\n",
            "Epoch 2/25\n",
            "192/192 [==============================] - 2s 11ms/step - loss: 0.1902 - accuracy: 0.9424 - val_loss: 0.1172 - val_accuracy: 0.9666\n",
            "Epoch 3/25\n",
            "192/192 [==============================] - 2s 11ms/step - loss: 0.1398 - accuracy: 0.9569 - val_loss: 0.0906 - val_accuracy: 0.9750\n",
            "Epoch 4/25\n",
            "192/192 [==============================] - 2s 10ms/step - loss: 0.1123 - accuracy: 0.9651 - val_loss: 0.0781 - val_accuracy: 0.9777\n",
            "Epoch 5/25\n",
            "192/192 [==============================] - 2s 11ms/step - loss: 0.0965 - accuracy: 0.9704 - val_loss: 0.0680 - val_accuracy: 0.9797\n",
            "Epoch 6/25\n",
            "192/192 [==============================] - 2s 8ms/step - loss: 0.0843 - accuracy: 0.9731 - val_loss: 0.0618 - val_accuracy: 0.9812\n",
            "Epoch 7/25\n",
            "192/192 [==============================] - 2s 8ms/step - loss: 0.0737 - accuracy: 0.9770 - val_loss: 0.0584 - val_accuracy: 0.9834\n",
            "Epoch 8/25\n",
            "192/192 [==============================] - 2s 9ms/step - loss: 0.0670 - accuracy: 0.9784 - val_loss: 0.0566 - val_accuracy: 0.9830\n",
            "Epoch 9/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0612 - accuracy: 0.9805 - val_loss: 0.0550 - val_accuracy: 0.9830\n",
            "Epoch 10/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0557 - accuracy: 0.9821 - val_loss: 0.0491 - val_accuracy: 0.9849\n",
            "Epoch 11/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0509 - accuracy: 0.9836 - val_loss: 0.0503 - val_accuracy: 0.9840\n",
            "Epoch 12/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0476 - accuracy: 0.9844 - val_loss: 0.0474 - val_accuracy: 0.9857\n",
            "Epoch 13/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0429 - accuracy: 0.9863 - val_loss: 0.0449 - val_accuracy: 0.9862\n",
            "Epoch 14/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0420 - accuracy: 0.9863 - val_loss: 0.0477 - val_accuracy: 0.9857\n",
            "Epoch 15/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0381 - accuracy: 0.9878 - val_loss: 0.0419 - val_accuracy: 0.9879\n",
            "Epoch 16/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0362 - accuracy: 0.9876 - val_loss: 0.0423 - val_accuracy: 0.9868\n",
            "Epoch 17/25\n",
            "192/192 [==============================] - 2s 9ms/step - loss: 0.0341 - accuracy: 0.9885 - val_loss: 0.0418 - val_accuracy: 0.9879\n",
            "Epoch 18/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0314 - accuracy: 0.9895 - val_loss: 0.0420 - val_accuracy: 0.9878\n",
            "Epoch 19/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0285 - accuracy: 0.9906 - val_loss: 0.0408 - val_accuracy: 0.9877\n",
            "Epoch 20/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0269 - accuracy: 0.9910 - val_loss: 0.0443 - val_accuracy: 0.9872\n",
            "Epoch 21/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0266 - accuracy: 0.9911 - val_loss: 0.0437 - val_accuracy: 0.9876\n",
            "Epoch 22/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0246 - accuracy: 0.9913 - val_loss: 0.0446 - val_accuracy: 0.9868\n",
            "Epoch 23/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0234 - accuracy: 0.9918 - val_loss: 0.0453 - val_accuracy: 0.9864\n",
            "Epoch 24/25\n",
            "192/192 [==============================] - 1s 7ms/step - loss: 0.0232 - accuracy: 0.9919 - val_loss: 0.0445 - val_accuracy: 0.9877\n",
            "Epoch 25/25\n",
            "192/192 [==============================] - 2s 8ms/step - loss: 0.0211 - accuracy: 0.9927 - val_loss: 0.0388 - val_accuracy: 0.9891\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "0.0005982410172611527"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tracker = EmissionsTracker(\n",
        "    project_name=\"SeparableConv2D\",\n",
        "    log_level=\"critical\",\n",
        "    measure_power_secs=15,\n",
        "    output_dir=\"./\",\n",
        "    output_file=\"emissions-separableconv2d.csv\",\n",
        "    tracking_mode='machine',\n",
        ")\n",
        "\n",
        "tracker.start()\n",
        "\n",
        "# Train the model\n",
        "model_SeparableConv2D.fit(input_train, target_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=no_epochs,\n",
        "          verbose=verbosity,\n",
        "          validation_split=validation_split)\n",
        "\n",
        "tracker.stop()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I45yN6hdkX2Z"
      },
      "source": [
        "### Traditional vs. Depthwise separable CNN: energy consumption comparison\n",
        "\n",
        "Now is the time for us to compare the performance of our two networks and their energy consumption and estimated emissions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4y6Eh6COlHrL",
        "outputId": "a10ef14e-1efc-4bff-b414-59fb0ae14841"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[0.023584142327308655, 0.9930999875068665]"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "perf_model_Conv2D"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 237
        },
        "id": "_3UGbA4fkYwC",
        "outputId": "c65d7de6-efc0-444d-8f48-686a29547fb9"
      },
      "outputs": [
        {
          "data": {
            "text/markdown": [
              "|                 |       Conv2D |   SeparableConv2D |\n",
              "|:----------------|-------------:|------------------:|\n",
              "| accuracy        |  0.9931      |       0.9895      |\n",
              "| loss            |  0.0235841   |       0.0362006   |\n",
              "| duration        | 83.0816      |      40.3688      |\n",
              "| energy_consumed |  0.00216531  |       0.00122606  |\n",
              "| emissions       |  0.00105654  |       0.000598241 |\n",
              "| emissions_rate  |  1.27168e-05 |       1.48194e-05 |"
            ],
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "from IPython.display import Markdown\n",
        "\n",
        "# Evaluate the models\n",
        "loss_Conv2D, acc_Conv2D = model_Conv2D.evaluate(input_test, target_test, verbose=0)\n",
        "loss_SeparableConv2D, acc_SeparableConv2D = model_SeparableConv2D.evaluate(input_test, target_test, verbose=0)\n",
        "\n",
        "# Read the generated emissions data\n",
        "emissions_conv2d = pd.read_csv(\"emissions-conv2d.csv\")\n",
        "emissions_separableconv2d = pd.read_csv(\"emissions-separableconv2d.csv\")\n",
        "\n",
        "# Create a dataframe with the combined results\n",
        "emissions_conv2d['accuracy'] = acc_Conv2D\n",
        "emissions_conv2d['loss'] = loss_Conv2D\n",
        "emissions_separableconv2d['accuracy'] = acc_SeparableConv2D\n",
        "emissions_separableconv2d['loss'] = loss_SeparableConv2D\n",
        "\n",
        "emissions_conv2d.index = ['Conv2D']\n",
        "emissions_conv2d = emissions_conv2d[['accuracy','loss', 'duration','energy_consumed',\n",
        "                    'emissions','emissions_rate']]\n",
        "\n",
        "emissions_separableconv2d.index = ['SeparableConv2D']\n",
        "emissions_separableconv2d = emissions_separableconv2d[['accuracy', 'loss', 'duration','energy_consumed',\n",
        "                    'emissions','emissions_rate']]\n",
        "\n",
        "# Concatenate the dataframes and display the results\n",
        "emissions_report = pd.concat([emissions_conv2d, emissions_separableconv2d])\n",
        "display(Markdown(emissions_report.transpose().to_markdown()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C0Ujf7k6nlJ6"
      },
      "source": [
        "In terms of accuracy, Conv2D marginally outperforms SeparableConv2D. However, when considering energy efficiency metrics, SeparableConv2D exhibits better results. It consumes significantly less energy and emits fewer carbon emissions (an approximate 40% decrease). Also, the experiments' duration is influenced by the type of letter we are using, with the network implementing SeparableConv2D taking half the time to train. These findings highlight a tradeoff between model accuracy and energy efficiency, where Conv2D excels in accuracy but consumes more energy and emits more carbon compared to SeparableConv2D. The choice between the two models depends on the specific priorities and constraints of the application, emphasizing the importance of considering both performance and energy efficiency in model selection.\n",
        "\n",
        "For example:\n",
        "\n",
        "> **_How do we deal with the tradeoff between accuracy and sustainability if low accuracy means that children with pneumonia will likely receive false negative results?_**\n",
        "\n",
        "We do not have answers to these questions, but the CV field is working to optimize our current techniques to reduce their environmental footprint. But the point here is that, when discussing sustainability in certain areas of application, values sometimes collide: intergenerational justice (to be fair to those who haven't yet arrived) and beneficence/non-maleficence. What would you choose? 🤔\n",
        "\n",
        "---\n",
        "\n",
        "Return to the [castle](https://github.com/Nkluge-correa/TeenyTinyCastle)."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}