decoder_notebook.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h_Iw1qCBlT-z"
      },
      "source": [
        "<a name='0'></a>\n",
        "## Import"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cqXowf9MlT-1"
      },
      "outputs": [],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eh1JdQmwlT-3"
      },
      "outputs": [],
      "source": [
        "!cp /content/drive/MyDrive/transformer_soc/rolling_and_plot_dc.py .\n",
        "!cp /content/drive/MyDrive/transformer_soc/sim_data.csv .\n",
        "!cp /content/drive/MyDrive/transformer_soc/transformer_helper_dc.py ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_OpwqWL2QH5G"
      },
      "outputs": [],
      "source": [
        "# from os import environ\n",
        "# environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"1\"\n",
        "# # removes tensorflow warnings triggered because of Tensorflow incompatibility with my Apple M1 chip.\n",
        "# # ignore this when using a non Apple Silicon device, ie. Google Colab or the likes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_DOA-JbhlT-4"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "# !pip install jupyterplot\n",
        "from jupyterplot import ProgressPlot as PP\n",
        "\n",
        "from global_dataclass import G\n",
        "from battery_transformer import *\n",
        "from transformer_helper_dc import *\n",
        "from rolling_and_plot_dc import data_plot, rolling_split, normalize, validate\n",
        "\n",
        "%reload_ext autoreload\n",
        "%autoreload 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AKXJW2TznWtG"
      },
      "outputs": [],
      "source": [
        "print(\"Num GPUs Available: \", len(tf.config.list_physical_devices('GPU')))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RUteRx9dlT-5"
      },
      "source": [
        "## Table of Contents\n",
        "\n",
        "- [Import](#0)\n",
        "- [Preprocessing](#win)\n",
        "- [Model](#model)\n",
        "- [Learning Rate Scheduler](#loss)\n",
        "- [Training](#train)\n",
        "- [Validate](#val)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "0EL21GdslT-5"
      },
      "source": [
        "**Literature:**\n",
        "\n",
        "\n",
        "According to [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://dl.acm.org/doi/abs/10.1145/3447548.3467401):\n",
        "Using **Batch Normalization is significantly more effective** for multivariate time-series than using the traditional Layer Normalization method found in NLP.\n",
        "\n",
        "In addition, according to [Deep learning approach towards accurate state of charge estimation for lithium-ion batteries using self-supervised transformer model](https://www.nature.com/articles/s41598-021-98915-8#Sec9):\n",
        "Using a transformer network while **forgoing the Decoder Layer** is more effective for the application of State-of-Charge estimation."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "VG0gPyv0oDBi"
      },
      "source": [
        "**Self-Attention**\n",
        "$$\n",
        "\\text { Attention }(Q, K, V)=\\operatorname{softmax}\\left(\\frac{Q K^{T}}{\\sqrt{d_{k}}}+{M}\\right) V\n",
        "$$"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "k2DSwSOZlT-7"
      },
      "source": [
        "\n",
        "**INPUT:** Voltage, Current, SOC at times:\n",
        "$$t - window\\_size - 1 \\rightarrow t - 1 $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bw-WpE1ulT-9"
      },
      "source": [
        "**Note**\n",
        "\n",
        "Cannot use embedding layers with battery data because of floating point values and negative values"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "prIueTe-lT-9"
      },
      "source": [
        "<a id=\"win\"></a>\n",
        "# Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "il6DI4Z7lT--"
      },
      "outputs": [],
      "source": [
        "# from google.colab import files\n",
        "# file = pd.read_csv(\"/content/sim_data.csv\")\n",
        "#if using sim_data.csv:\n",
        "file[\"soc\"] *= 100.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SLQrFOvrlT--"
      },
      "outputs": [],
      "source": [
        "data_plot(data = [file],\n",
        "          title=\"OCV v SOC\",\n",
        "          x = [\"test time (sec)\"],\n",
        "          y = [\"soc\"],\n",
        "          markers = \"lines\",\n",
        "          color = \"darkorchid\",\n",
        "          x_title = \"Test Time (sec)\",\n",
        "          y_title = \"SOC\"\n",
        "         )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_f7QighFlT--"
      },
      "outputs": [],
      "source": [
        "file = normalize(file.loc[:,[\"current\",\"voltage\",\"soc\"]].iloc[::G.slicing], G.capacity)\n",
        "#uses sklearn.preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x79KvZ3ilT--"
      },
      "outputs": [],
      "source": [
        "x_train, x_test, y_train, y_test = rolling_split(file, G.window_size, G.tgt_len, train=True)\n",
        "print(x_train.shape, x_test.shape, y_train.shape, y_test.shape)\n",
        "#uses sklearn.model_selection\n",
        "\n",
        "train_dataloader = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(G.batch_size, drop_remainder=True)\n",
        "test_dataloader = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(G.batch_size, drop_remainder=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yRmivoyVlT-_"
      },
      "outputs": [],
      "source": [
        "for x, y in train_dataloader:\n",
        "    print(f\"Shape of X [window, features]: {x.shape}\")\n",
        "    print(f\"Shape of y: {y.shape} {y.dtype}\")\n",
        "    break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HNB-qBpp7XFN"
      },
      "source": [
        "<a id =\"model\"></a>\n",
        "# Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BB7qoG617XFR"
      },
      "source": [
        "Build Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KptM3t5ol8jS"
      },
      "outputs": [],
      "source": [
        "tf.keras.backend.clear_session()\n",
        "model = Transformer()\n",
        "model.build((G.batch_size, G.window_size, G.num_features))\n",
        "model.summary(expand_nested=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xsipoJnw7XFR"
      },
      "source": [
        "**Loading Already Saved Progress**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JUcLoUmWlT_D"
      },
      "outputs": [],
      "source": [
        "model.load_weights(\"/content/drive/MyDrive/transformer_soc/decoder/model_weights.tf\")\n",
        "\n",
        "# scheduler_state = np.load(\"/content/drive/MyDrive/transformer_soc/decoder/scheduler_state.npy\")\n",
        "# print(f\"Saved learning_rate, T_cur, and T_i: {scheduler_state}\")\n",
        "\n",
        "# try:\n",
        "#     G.learning_rate, G.T_cur, G.T_i = scheduler_state\n",
        "# except NameError:\n",
        "#     \"global_dataclass.py has not been imported\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yYtQv1TtlT_D"
      },
      "source": [
        "<a id = \"loss\"></a>\n",
        "# LR Scheduler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aTN3TiSblT_D"
      },
      "source": [
        "**Learning Rate Scheduler**\n",
        "\n",
        "Cosine Annealing with Warm Restarts proposed by Loshchilov et al. in [SGDR: Stochastic Gradient Descent with Warm Restarts](https://doi.org/10.48550/arXiv.1608.03983)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xWt1eUd9o6WA"
      },
      "source": [
        "$$\\mu_t = \\mu_{min} + \\frac{1}{2}(\\mu_{max} - \\mu_{min})\\cdot (1 + \\cos (\\frac{T_{cur}}{T_i}\\pi))$$\n",
        "\n",
        "Where:\n",
        " - $\\mu$ is the learning_rate, subscript $t$ is for time = $t$\n",
        " - $T_{cur}$ is the number of epochs since the last restart\n",
        " - $T_i$ is the number of epochs between two restarts\n",
        "\n",
        "Note:\n",
        " - When $T_{cur} = T_i \\rightarrow \\mu_t = \\mu_{min}$\n",
        " - When $T_{cur} = 0 \\rightarrow \\mu_t = \\mu_{max}$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ciqL4LQCl8jT"
      },
      "outputs": [],
      "source": [
        "def schedule(batch, logs):\n",
        "        '''\n",
        "        This is a dummy function for the LearningRateScheduler Class\n",
        "        Returns a new learning rate based on the schedule described below\n",
        "        Call after every batch\n",
        "        '''\n",
        "        mu_i = G.min_learning_rate + 0.5 * (\n",
        "                G.learning_rate - G.min_learning_rate) * (\n",
        "                    1 + tf.math.cos(np.pi * G.T_cur / G.T_i))\n",
        "        \n",
        "        G.T_cur += G.batch_size / len(x_train)\n",
        "        if np.isclose(G.T_cur, G.T_i):\n",
        "            G.T_i *= G.T_mult\n",
        "            G.T_cur = 0.0\n",
        "        K.set_value(model.optimizer.learning_rate, mu_i)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EMewwSwzl8jT"
      },
      "source": [
        "### Save Model Progress Callbacks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iQ3M31YBl8jT"
      },
      "outputs": [],
      "source": [
        "# class SaveModel(tf.keras.callbacks.Callback):\n",
        "#     def on_epoch_end(self, epoch, logs = None):\n",
        "#         if epoch != 0 and epoch % 15 == 0:\n",
        "#             self.model.save_weights(\"/content/drive/MyDrive/transformer_soc/model_weights.h5\")\n",
        "\n",
        "# model_save = SaveModel() #This is optional"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-wuAcjicl8jT"
      },
      "source": [
        "### Early Stopping and Saving Best Model checkpoint Callbacks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wiDNXqaKl8jT"
      },
      "outputs": [],
      "source": [
        "# model_options = tf.saved_model.SaveOptions(experimental_io_device=\"/job:localhost\")\n",
        "# earlystopping = EarlyStopping(monitor='val_mean_absolute_percentage_error',\n",
        "#                               patience=150,\n",
        "#                               verbose=0,\n",
        "#                               mode='min')\n",
        "# mcp_save = ModelCheckpoint('/content/drive/MyDrive/transformer_soc/decoder/model_weights',\n",
        "#                            save_format = \"tf\",\n",
        "#                            save_best_only=True,\n",
        "#                            monitor='val_mean_absolute_percentage_error',\n",
        "#                            mode='min')\n",
        "# #                            options = model_options)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MODzuuNjl8jT"
      },
      "source": [
        "**ProgressPlot Callback**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7PrLnMq4l8jT"
      },
      "outputs": [],
      "source": [
        "class ProgressCallback(tf.keras.callbacks.Callback):\n",
        "    def on_epoch_end(self, epoch, logs = None):\n",
        "        train_loss = logs[\"loss\"]\n",
        "        train_acc = 100.0 - logs[\"mean_absolute_percentage_error\"]\n",
        "        pp.update([[train_loss],\n",
        "                   [train_acc]])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "d69cDpRel8jT"
      },
      "source": [
        "<a id =\"train\"></a>\n",
        "\n",
        "# Training \n",
        "reset the cos_anneal scheduler $\\downarrow$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Lj0tZcs7vD4y"
      },
      "outputs": [],
      "source": [
        "G.T_i = 1\n",
        "G.T_mult = 2\n",
        "G.T_cur = 0.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxiW8nx6l8jT"
      },
      "source": [
        "`model.fit()` and `model.compile()` asset declaration $\\downarrow$"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mArN7M42l8jT"
      },
      "outputs": [],
      "source": [
        "loss_object = tf.keras.losses.LogCosh()\n",
        "\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate = G.learning_rate,\n",
        "                                     beta_1 = 0.9,\n",
        "                                     beta_2 = 0.999\n",
        "                                    )\n",
        "\n",
        "#cos_anneal is for the model.fit() call\n",
        "cos_anneal = tf.keras.callbacks.LambdaCallback(on_batch_end = schedule)\n",
        "\n",
        "#progress plot callback\n",
        "pp_update = ProgressCallback()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Note:** can add `model_save` or `mcp_save` to the `callbacks` argument in `model.fit()`\n",
        "it saves the model params, or saves model checkpoints to the google drive, respectively.\n",
        "there is also the earlystopping callback but don't worry about it"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#only run this cell once per model, notebook will crash if you compile an already compiled model\n",
        "model.compile(optimizer, loss_object, metrics=[\"mean_absolute_percentage_error\"])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w_x0rV-el8jT"
      },
      "outputs": [],
      "source": [
        "pp = PP(plot_names = [\"Mean Log Loss\", \"% Accuracy\"],\n",
        "        line_names = [\"Train Loop\"],\n",
        "        x_label = \"epochs\"\n",
        "       )\n",
        "# Dont compile after training, it causes issues.\n",
        "history = model.fit(train_dataloader,\n",
        "                    epochs = 4,\n",
        "                    verbose = 1,\n",
        "                    callbacks = [cos_anneal, pp_update]\n",
        "                    )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zl1wsaxlvksT"
      },
      "outputs": [],
      "source": [
        "model.evaluate(test_dataloader,verbose = 1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5pSwH7QlT_I"
      },
      "source": [
        "<a id = \"val\"></a>\n",
        "# Validate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JYr2y9eulT_I"
      },
      "source": [
        "**Dev Set**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CuY9saCblT_I"
      },
      "outputs": [],
      "source": [
        "visualize_dev = validate(model, test_dataloader, dev = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5uLkWkLlT_I"
      },
      "source": [
        "**Entire Dataset**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gjvsvbIllT_I"
      },
      "outputs": [],
      "source": [
        "x_set, y_set = rolling_split(file, G.window_size, G.tgt_len, train = False)\n",
        "\n",
        "set_dataloader = tf.data.Dataset.from_tensor_slices((x_set, y_set)).batch(G.batch_size, drop_remainder=True)\n",
        "for x,y in set_dataloader:\n",
        "    print(x.shape, y.shape)\n",
        "    break\n",
        "\n",
        "visualize = validate(model, set_dataloader, dev = False)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "transform_notebook.ipynb",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "tflow",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.5"
    },
    "vscode": {
      "interpreter": {
        "hash": "4a0aa943aa44ec2cea40b2a3e959790e4ce20a8627960ed37a1774af4e3759e0"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}