docs/examples/individual_pipeline/1.individual_data_preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2cff9a48",
   "metadata": {},
   "source": [
    "# Preparing Your Data for wearablehrv \n",
    "This example shows you how to prepare and name your files, such that it is readble for `wearablehrv`. It also involves importing all your recorded datasets in one go, as well as defining and importing experimental events. We will provide some sample data and a detailed explanation of this data to help you get started with the package and give you an idea of how you could conduct a similar validation study.\n",
    "\n",
    "**Note**: The notebook examples located in the \"individual\" folder utilize the `individual` module from `wearablehrv`. These examples are designed to handle data from <u>a single participant</u>, who may have worn multiple wearables simultaneously. As long as the detected interbeat intervals and timestamps are stored, `wearablehrv` facilitates the journey from raw data to group statistical analysis. It offers a range of plotting functionalities to assist in decision-making or data pre-processing, along with user-friendly graphical user interfaces.\n",
    "\n",
    "*Tip: If you use Empatica Embrace Plus, Labfront, or VU-AMS, the pipeline is compatible with their data output. To learn more, consult this notebook: `individual_compatibility.ipynb`.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c77982f",
   "metadata": {},
   "source": [
    "## Previous Steps\n",
    "\n",
    "If you have not done so, first take a look at the following notebooks:\n",
    "\n",
    "- [Installation guide for wearablehrv](https://github.com/Aminsinichi/wearable-hrv/blob/master/docs/examples/individual_pipeline/installation.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1bdea58",
   "metadata": {},
   "source": [
    "### Shape of datasets\n",
    "\n",
    "Datasets should be saved as .csv file with this format:\n",
    "\n",
    "* [Participant ID]_[name of the device].csv\n",
    "* Example: P01_polar.csv\n",
    "\n",
    "This dataset for each device is a continuous recording that contains all conditions. It should follow this structure: the first column is time in UNIX format, in milliseconds; the second column contains inter-beat intervals. \n",
    "\n",
    "**Example:**\n",
    "\n",
    "| timestamp    | rr   |\n",
    "|--------------|------|\n",
    "| 1688126297704| 919  |\n",
    "| 1688126298681| 1207 |\n",
    "| 1688126298693| 695  |\n",
    "| 1688126304713| 1167 |\n",
    "| 1688126305707| 1047 |\n",
    "| 1688126307686| 951  |\n",
    "| 1688126309693| 1111 |\n",
    "\n",
    "- **timestamp**: Millisecond precision in Unix format.\n",
    "- **rr**: Contains inter-beat intervals (sourced from either PPG or ECG devices).\n",
    "\n",
    "Let's import the package. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "558ffdc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing Module\n",
    "import wearablehrv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0afd784-b28c-4a99-b4dd-c59ecbb44bd6",
   "metadata": {},
   "source": [
    "Once you have restructured (if necessary) your data and renamed it accordingly, you can define a few variables before calling the functions.\n",
    "\n",
    "One variable is the `path`; this is where all your files from different wearables for a particular participant are located. \n",
    "\n",
    "For now, we will use a function from `wearablehrv` that will download the sample data from GitHub to your Cache and save the location of it in the path variable. You can change this path accordingly to point to where your files are located. **It can be a string or a Path object.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe8b4f9a-e038-4549-b5c5-2e235bfdce61",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define where the files are located \n",
    "# For now, we are downloading some example data\n",
    "path = wearablehrv.data.download_data_and_get_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e28e3a5-1788-4562-9603-f9db45e341e6",
   "metadata": {},
   "source": [
    "The following section needs to be modified based on your datasets:\n",
    "\n",
    "* `pp`: the participant ID. Please ensure that this is exactly the same as the one used to save your files (in our example, this can be set to \"test\".)\n",
    "* `conditions`: the experimental conditions you used should be listed, for example: `['sitting', 'breathing']`.\n",
    "* `devices`: the devices you used should be listed, for example: `['empatica', 'heartmath']`. Please ensure that this is exactly the same as the names used to save your files. Also, ensure that you always specify the criterion device as the last element of the list.\n",
    "* `criterion`: specify the name of your criterion device, for example: `vu`. Please ensure that this is exactly the same as the name used to save your files.\n",
    "\n",
    "**Note**: Throughout the example notebooks and also in the code, we used the term \"<u>criterion</u>,\" which refers to the device that the rest of the devices are compared against. This is also referred to as \"reference system,\" \"ground truth,\" and \"gold standard\" in the literature. This is usually an electrocardiography (ECG) device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98fb4677",
   "metadata": {},
   "outputs": [],
   "source": [
    "pp = \"test\" # Define the participant ID here\n",
    "\n",
    "# Define your experimental conditions\n",
    "conditions = ['sitting', 'arithmetic', 'recovery', 'standing', 'breathing', 'neurotask', 'walking', 'biking'] \n",
    "\n",
    "# Define the devices you want to validate against the criterion. \n",
    "# Note: MAKE SURE TO PUT THE CRITERION DEVICE THE LAST ONE IN THE LIST \n",
    "devices = [\"kyto\", \"heartmath\", \"rhythm\", \"empatica\", \"vu\"] \n",
    "\n",
    "# Redefine the name of the criterion device\n",
    "criterion = \"vu\" "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8742fac",
   "metadata": {},
   "source": [
    "Once you have set all these up, **it is very easy to read all your files from all the devices in one go**. You just need to run the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d2952b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = wearablehrv.individual.import_data (path, pp, devices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d373deb",
   "metadata": {},
   "source": [
    "Given that we are working with some sample data, it is good for us to become familiar with how this dataset is recorded for validation.\n",
    "\n",
    "\n",
    "We wanted to validate a few wearables against a criterion ECG wearable to see how accurate they are in terms of HR and HRV.\n",
    "\n",
    "The names of the devices are as follows:\n",
    "\n",
    "- [Empatica EmbracePlus](https://www.empatica.com/en-int/embraceplus/?utm_campaign=ADW_AO_BRND_CNV&utm_campaign=ADW_AO_BRND_CNV&utm_medium=sitelink&utm_medium=Paid&utm_source=google&utm_source=Google) (empatica)\n",
    "- HeartMath Inner Balance Bluetooth (heartmath)\n",
    "- [KYTO2935 Earlobe Device](https://kytofitness.com/products/bluetooth-mobile-heart-rate-monitor-with-ear-clip-kyto2935) (kyto)\n",
    "- [Scosche Rhythm 24](https://www.scosche.com/rhythm24-waterproof-armband-heart-rate-monitor) (rhythm)\n",
    "- [VU-AMS](https://vu-ams.nl/) (vu - the criterion device)\n",
    "\n",
    "\n",
    "<img src=\"https://github.com/Aminsinichi/wearable-hrv/blob/master/docs/img/Sensor%20Placement.png?raw=true\" alt=\"Sensor Placement\" width=\"width_value\" height=\"height_value\">\n",
    "\n",
    "\n",
    "All devices were worn simultaneously. The VU-AMS ECewass the criterion device. Ultimately, from every device, the detected inter-beat intervals and timestamps were extracted and saved as .csv file, similar to the format explained above (For PPG devices, we used an application called [HRV Logger](https://www.hrv.tools/hrv-logger-faq.html)). Participants did a series of conditions consecutively. \n",
    "\n",
    "These **conditions** are as follows:\n",
    "\n",
    "- Sitting at rest for 5 minutes (\"sitting\")\n",
    "- A backward counting task from 1022, decreasing by 7 each time (\"arithmetic\") for 3 minutes\n",
    "- Recovery, sitting rested for an additional 3 minutes post-task (\"recovery\")\n",
    "- Posture manipulation, standing for 3 minutes (\"standing\")\n",
    "- Slow-paced breathing: 4 seconds inhale, 6 seconds exhale for 3 minutes (\"breathing\")\n",
    "- A cognitive task, emotional go-no-go, lasting around 10 minutes (\"neurotask\")\n",
    "- Slow-paced walking for 3 minutes (\"walking\")\n",
    "- Stationary biking for 3 minutes (\"biking\")\n",
    "\n",
    "**You have already downloaded these data**, and they should exist on your local drive at the following location:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47eb26a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdb56281",
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can check them out by running this code:\n",
    "\n",
    "import os\n",
    "os.listdir (path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3a5c30",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Below are links to GitHub for the files explained above, along with a further explanation of each of them.\n",
    "\n",
    "- [test_empatica.csv](https://github.com/Aminsinichi/wearable-hrv/blob/develop/tests/test_individual/test_empatica.csv) - Continuous data comprising timestamps and inter-beat intervals for the Empatica device\n",
    "- [test_heartmath.csv](https://github.com/Aminsinichi/wearable-hrv/blob/develop/tests/test_individual/test_heartmath.csv) - Continuous data featuring timestamps and inter-beat intervals for the HeartMath device\n",
    "- [test_kyto.csv](https://github.com/Aminsinichi/wearable-hrv/blob/develop/tests/test_individual/test_kyto.csv) - Continuous data with timestamps and inter-beat intervals for the Kyto device\n",
    "- [test_rhythm.csv](https://github.com/Aminsinichi/wearable-hrv/blob/develop/tests/test_individual/test_rhythm.csv) - Continuous data that captures timestamps and inter-beat intervals for the Rhythm device\n",
    "- [test_vu.txt](https://github.com/Aminsinichi/wearable-hrv/blob/develop/tests/test_individual/test_vu.txt) - Continuous data detailing timestamps and inter-beat intervals for the VU-AMS device. Note that this is a text file, solely because this specific device saves data as .txt. However, a .csv format can also be employed, for instance, with an ECG device used as a criterion, like the Polar."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1568663f",
   "metadata": {},
   "source": [
    "## Defining Events\n",
    "\n",
    "Participants were wearing the devices simultaneously, engaging in a variety of conditions, and the data was being recorded from all the devices continuously. In order to know the start and end of each condition, an experimenter keeps track of this. Then, you should also import this into the pipeline so that your continuous data can be cropped into smaller pieces, for each condition.\n",
    "\n",
    "In order to define the events, you can use the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f1217bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "events = wearablehrv.individual.define_events (path, pp, conditions, already_saved= True, save_as_csv= False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0566066",
   "metadata": {},
   "source": [
    "You have two options here:\n",
    "\n",
    "Firstly (**recommended**), you can already have a saved .csv file named as [Participant ID]_events.csv in your path with a shape similar to this:\n",
    "\n",
    "| timestamp | conditions | datapoint |\n",
    "|-----------|------------|-----------|\n",
    "| 11:48:28  | sitting    | start     |\n",
    "| 11:50:28  | sitting    | end       |\n",
    "| 11:53:00  | breathing  | start     |\n",
    "| 11:55:00  | breathing  | end       |\n",
    "\n",
    "You have already downloaded this in your path.Check out [test_event.csv](https://github.com/Aminsinichi/wearable-hrv/blob/develop/tests/test_individual/test_events.csv) for an example. If this is the case, in order to read this file, set `already_saved = True`. \n",
    "\n",
    "Secondly, you may not have such an events file. In this case, set `already_saved = False`. By running the code, a graphical user interface will pop up, where you can define your events.\n",
    "If you set `save_as_csv = True`, this event file will be saved in your path location. \n",
    "\n",
    "**WARNING:** Be careful not to overwrite your current event file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b85719cb",
   "metadata": {},
   "source": [
    "That's it! At this point, you should have been able to read all your data and import or define the experimental events.\n",
    "\n",
    "## Next Steps\n",
    "\n",
    "You're now ready to move on to the next notebook examples. \n",
    "\n",
    "Continue by consulting: \n",
    "\n",
    "- [Preprocess your data](https://github.com/Aminsinichi/wearable-hrv/blob/master/docs/examples/individual_pipeline/2.individual_data_preprocessing.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}