main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_____\n",
    "***\n",
    "\n",
    "<img width='700' src=\"https://user-images.githubusercontent.com/8030363/108961534-b9a66980-7634-11eb-96e2-cc46589dcb8c.png\" style=\"vertical-align:middle\">\n",
    "\n",
    "***\n",
    "***\n",
    "\n",
    "**Author:** [TJCallahan](https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=callahantiff@gmail.com)  \n",
    "**GitHub Repository:** [PheKnowLator](https://github.com/callahantiff/PheKnowLator/wiki)  \n",
    "**Current Release:** **[`v2.0.0`](https://github.com/callahantiff/PheKnowLator/wiki/v2.0.0)**\n",
    "\n",
    "<br>\n",
    "\n",
    "**Objective:** Knowledge graphs provide meaningful ways to integrate heterogeneous biological data and represent complex biological mechanisms. This work seeks to explore the utility of incorporating existing knowledge of molecular mechanism from ontologies, publicly available data, and the literature to construct a biomedical knowledge graph that models unbiased molecular mechanisms of human disease.\n",
    "\n",
    "<a target=\"_blank\" href=\"https://user-images.githubusercontent.com/8030363/103158881-11813b00-4780-11eb-8b45-5063765e7645.png\"> <img src=\"https://user-images.githubusercontent.com/8030363/103158881-11813b00-4780-11eb-8b45-5063765e7645.png\"></a> \n",
    "\n",
    "(*Click Figure to Enlarge Image in Current Browser Tab*)\n",
    "\n",
    "<br>\n",
    "\n",
    "***\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Purpose\n",
    "**Wiki Page:** **[`Release v2.0.0`](https://github.com/callahantiff/PheKnowLator/wiki/v2.0.0)**\n",
    "\n",
    "<br>\n",
    "\n",
    "**Purpose:** This notebook serves as a `main` file for the PheKnowLator project. This scripts walks through this program step-by-step and generates the knowledge graph shown above. There is also a command line version of this file ([`main.py`](https://github.com/callahantiff/PheKnowLator/blob/master/main.py)). Please see the [README](https://github.com/callahantiff/PheKnowLator/blob/master/README.md) for more information.\n",
    "\n",
    "<br>\n",
    "\n",
    "**Assumptions:**     \n",
    "1. Make sure that the following input documents have been constructed (see the [Dependencies Wiki](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies) for more information):  \n",
    "  - [`resource_info.txt`](https://github.com/callahantiff/PheKnowLator/blob/master/resources/resource_info.txt)\n",
    "  - [`ontology_source_list.txt`](https://github.com/callahantiff/PheKnowLator/blob/master/resources/ontology_source_list.txt)\n",
    "  - [`edge_source_list.txt`](https://github.com/callahantiff/PheKnowLator/blob/master/resources/edge_source_list.txt)   \n",
    "\n",
    "2. Prepare [relations](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#relations-data) and [node metadata](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#node-metadata) files prior to running the scripts.  \n",
    "\n",
    "3. Select a knowledge graph build type (i.e. `full`, `partial`, or `post-closure`) and construction method (i.e. `instance-based` or `subclass-based`).  \n",
    "\n",
    "<br>\n",
    "\n",
    "***\n",
    "### Table of Contents\n",
    "***\n",
    "The three primary steps involved in building a knowledge graph are `Downloading Data Sources`, `Creating Edge Lists`, and `Building the knowledge graphs`.\n",
    "\n",
    "* [Data Sources](#data-source)  \n",
    "* [Create Edge Lists](#create-edges)  \n",
    "* [Build Knowledge Graph](#build-kg)  \n",
    "\n",
    "***\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "is_executing": false
    }
   },
   "source": [
    "_____\n",
    "### Set-Up Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import needed libraries\n",
    "import glob\n",
    "import json\n",
    "import pandas\n",
    "import psutil\n",
    "import ray\n",
    "import time\n",
    "\n",
    "# import module\n",
    "from pkt_kg.downloads import OntData, LinkedData\n",
    "from pkt_kg.edge_list import CreatesEdgeList\n",
    "from pkt_kg.knowledge_graph import FullBuild, PartialBuild, PostClosureBuild"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "## Download Data Sources <a class=\"anchor\" id=\"data-source\"></a>\n",
    "\n",
    "**Wiki Page:** **[`Dependencies`](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies)**  \n",
    "\n",
    "**Purpose:**\n",
    "This portion of this portion of the algorithm is to download:\n",
    "1. [Download Ontology Data](#download-ontology-data)  \n",
    "2. [Download Edge Data](#download-edge-data)   \n",
    "\n",
    "<br>\n",
    "\n",
    "**Input Files:**\n",
    "  - [`resource_info.txt`](https://github.com/callahantiff/PheKnowLator/blob/master/resources/resource_info.txt)\n",
    "  - [`ontology_source_list.txt`](https://github.com/callahantiff/PheKnowLator/blob/master/resources/ontology_source_list.txt)\n",
    "  - [`edge_source_list.txt`](https://github.com/callahantiff/PheKnowLator/blob/master/resources/edge_source_list.txt)\n",
    "\n",
    "<br>\n",
    "\n",
    "**Assumptions:**  \n",
    "- All sources used to construct our knowledge graph need to be preprocessed and ready to download prior to running this code. All mapping, filtering, and label data have been generated prior to this step. For assistance with creating these datasets, see the [`Data_Preparation.ipynb`](https://github.com/callahantiff/PheKnowLator/blob/master/notebooks/Data_Preparation.ipynb) Jupyter Notebook.  \n",
    "- All downloaded and generated data sources for all PheKnowLator builds can be accessed through [Zenodo](https://zenodo.org/doi/10.5281/zenodo.7030039).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "***\n",
    "### Ontology Data  <a class=\"anchor\" id=\"download-ontology-data\"></a>\n",
    "Ontologies are the core data structure used when building PheKnowLator. In the figure above, ontology data are shown in yellow boxes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ont = OntData('resources/ontology_source_list.txt', 'resources/resource_info.txt')\n",
    "ont.downloads_data_from_url()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "### Edge Data   <a class=\"anchor\" id=\"download-edge-data\"></a>\n",
    "In PheKnowLator, classes are nodes that originate from ontologies. Class data sources are Linked Data sources that are used to create edges in the knowledge graph and thus can connect to other class data sources. Sometimes we want to add data that is not already part of an ontology. In that case, data either be added as an `instance` of an existing ontology class or as its own `owl:class` by being added to the knowledge graph as a `subclass` of an existing `owl:class`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "edges = LinkedData('resources/edge_source_list.txt', 'resources/resource_info.txt')\n",
    "edges.downloads_data_from_url()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Create Edge Lists <a class=\"anchor\" id=\"create-edges\"></a>\n",
    "\n",
    "**Wiki Page:** **[`Data Sources`](https://github.com/callahantiff/PheKnowLator/wiki/v2-Data-Sources)**\n",
    "\n",
    "<br>\n",
    "\n",
    "**Purpose:** The code below will take the dictionaries of processed data described above and use them to create edge lists for each of the edge types specified in the [`resource_info.txt`](https://github.com/callahantiff/PheKnowLator/blob/master/resources/resource_info.txt). Each edge list will be appended to a nested dictionary (see details below).\n",
    "\n",
    "<br>\n",
    "\n",
    "**Assumptions:**  \n",
    "1. All `ontology` and `edge` data sources have been downloaded.   \n",
    "\n",
    "2. All code in the [`Data_Preparation.ipynb`](https://github.com/callahantiff/PheKnowLator/blob/master/notebooks/Data_Preparation.ipynb) Jupyter Notebook has been run. This Notebook contains code needed to generate all mapping, filtering, and label data.\n",
    "\n",
    "<br>\n",
    "\n",
    "**Output:** `Master_Edge_List_Dict.json`. Below is an example of what the `Master Edge Dictionary` contains for each processed resource:  \n",
    "```python\n",
    "master_edges = {'chemical-disease'  :\n",
    "                {'source_labels'    : ';MESH_;',\n",
    "                 'data_type'        : 'class-class',\n",
    "                 'edge_relation'    : 'RO_0002606',\n",
    "                 'uri'              : ('http://purl.obolibrary.org/obo/',\n",
    "                                       'http://purl.obolibrary.org/obo/'),\n",
    "                 'delimiter'        : '#',\n",
    "                 'column_idx'       : '1;4',\n",
    "                 'identifier_maps'  : '0:./MESH_CHEBI_MAP.txt;1:disease-dbxref-map',\n",
    "                 'evidence_criteria': \"5;!=;' \",\n",
    "                 'filter_criteria'  : 'None',\n",
    "                 'edge_list'        : ['...']}\n",
    "```\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set-up environment for parallel processing -- even if running program serially these steps are needed\n",
    "cpus = psutil.cpu_count(logical=True)\n",
    "ray.init()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine data sources\n",
    "combined_edges = dict(edges.data_files, **ont.data_files)\n",
    "resource_info_loc = './resources/resource_info.txt'\n",
    "\n",
    "# initialize edge dictionary class\n",
    "master_edges = CreatesEdgeList(data_files=combined_edges, source_file=resource_info_loc)\n",
    "master_edges.runs_creates_knowledge_graph_edges(source_file=resource_info_loc, data_files=combined_edges, cpus=cpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Preview Master Edge Data**  \n",
    "Generate a table that includes each `edge-type`, its primary `relation`, example identifiers, and count of unique edges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # read in master edge dictionary\n",
    "master_edges = json.load(open('resources/Master_Edge_List_Dict.json', 'r'))\n",
    "\n",
    "# read in relation data\n",
    "relation_data = open('./resources/relations_data/RELATIONS_LABELS.txt').readlines()\n",
    "relation_dict = {x.split('\\t')[0]: x.split('\\t')[1].strip('\\n') for x in relation_data}\n",
    "\n",
    "# print basic stats on each resource\n",
    "edge_data = [[key, master_edges[key]['edge_relation'],\n",
    "              ', '.join(master_edges[key]['edge_list'][0]),\n",
    "              len(master_edges[key]['edge_list'])]\n",
    "             for key in master_edges.keys()]\n",
    "\n",
    "# convert dict to pandas df for nice printing\n",
    "df = pandas.DataFrame(edge_data, columns = ['Edge Type', 'Relation', 'Example Edge', 'Unique Edges']) \n",
    "df                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Build Knowledge Graph  <a class=\"anchor\" id=\"build-kg\"></a>\n",
    "**Wiki Pages:**  \n",
    "- **[`KG-Construction`](https://github.com/callahantiff/PheKnowLator/wiki/KG-Construction)**  \n",
    "- **[`relations-data`](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#relations-data)**  \n",
    "- **[`node-metadata`](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#node-metadata)** \n",
    "\n",
    "**Jupyter Notebooks:**  \n",
    "- [`Data_Preparation.ipynb`](https://github.com/callahantiff/PheKnowLator/blob/master/notebooks/Data_Preparation.ipynb)  \n",
    "[`Ontology_Cleaning.ipynb`](https://github.com/callahantiff/PheKnowLator/blob/master/notebooks/Ontology_Cleaning.ipynb)  \n",
    "\n",
    "\n",
    "<br>\n",
    "\n",
    "**Assumptions:**  \n",
    "- <u>Construction Approach</u>. If using the `subclass-based` construction approach, please make sure that a `pickled` dictionary mapping each non-ontology data node to an existing ontology class is created and added to the `./resources/knowledge_graph` directory (please see [here](https://github.com/callahantiff/PheKnowLator/tree/master/resources/knowledge_graphs#construction-method) for additional information).   \n",
    "- <u>Relations Data</u>. If inverse relation data is going to be used to build the knowledge graph, that it has been generated and added to the `./resources/relations_data` directory (please see [here](https://github.com/callahantiff/PheKnowLator/blob/master/resources/relations_data/README.md) for additional information).  \n",
    "- <u>Node Metadata</u>. If node metadata is going to be used to build the knowledge graph, that it has been generated and added to the `./resources/node_metadata` directory (please see [here](https://github.com/callahantiff/PheKnowLator/blob/master/resources/node_data/README.md) for additional information).  \n",
    "- <u>Decoding OWL Semantics</u>. If decoding OWL-Semantics, please make sure to provide a list of owl:Property types to keep is created and added to the `./resources/knowledge_graph` directory (please see [here](https://github.com/callahantiff/PheKnowLator/wiki/OWL-NETS-2.0) for additional information). \n",
    "\n",
    "<br>\n",
    "\n",
    "**Input:** \n",
    "- `Master_Edge_List_Dict.json`  \n",
    "- Directory of relations data sources - see [here](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#relations-data) for more information\n",
    "- Directory of node data sources - see [here](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#node-metadata) for more information\n",
    "\n",
    "<br>\n",
    "\n",
    "**Output:** Please see [`Release v2.0.0 Wiki`](https://github.com/callahantiff/PheKnowLator/wiki/v2.0.0) for access to all generated output files.   \n",
    "- `Knowledge Graph` (`.owl` and Networkx MultiDiGraph `.pkl`)  \n",
    "- `Class Instance URI-UUID Map` (if \"instance\" construction approach)   \n",
    "- `Triple List - Integer`  \n",
    "- `Triple List - Identifier`  \n",
    "- `Node Integer-Identifier Map`  \n",
    "- `Node Attribute Data`  \n",
    "\n",
    "<br>\n",
    "\n",
    "The process to build the knowledge graph is somewhat time consuming and can be broken into the following steps:  \n",
    "\n",
    "1. Merge Ontologies. See [here](https://github.com/callahantiff/PheKnowLator/blob/master/resources/ontologies/README.md) for additional information on how to preprocess the ontologies prior to merging them.    \n",
    "\n",
    "2. Create Edges. Add edge lists to merged ontologies.  \n",
    "\n",
    "3. Add Inverse Relations and Node Data. See the [Dependencies](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies) Wiki page for details on how to construct these resources.  \n",
    "\n",
    "4. Filter OWL Semantics. Filter the knowledge graph with the goal of removing all edges that contain entities that are needed to support owl semantics, but are not biologically meaningful (please see [here](https://github.com/callahantiff/PheKnowLator/wiki/OWL-NETS-2.0) for additional information).\n",
    "\n",
    "5. Save Edge Lists and Node Metadata. Several versions of the knowledge graph are saved, including: the full knowledge graph (`owl` or Networkx MultiDiGraph `pickle`), triple lists (i.e. integer index and identifier labeled edge lists with a dictionary that maps between the integer indices and node identifiers), and a file of metadata (i.e. identifiers, labels, synonyms, and descriptions) for all nodes in the knowledge graph.  \n",
    "\n",
    "<br>\n",
    "\n",
    "**‼ IMPORTANT:**  \n",
    "- The file containing the merged ontologies is quite large and can take up to 30 minutes to read in.  This is not a limitation of the code directly, but rather a function of the [`RDFLib Library`](https://github.com/RDFLib). While there are other ways to read in this data, we maintain reliance on this library as it is the most user-friendly for non-RDF users.   \n",
    "- If you'd like to include [node metadata](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#node-metadata) when building the knowledge graph, please hold off on building the knowledge graph until you have generated the node data. For details on how to do this see the [node metadata](https://github.com/callahantiff/PheKnowLator/wiki/Dependencies#node-metadata) section of the `Dependencies` Wiki Page or help for help with generating the data, please see the []() section of the [`Data_Preparation.ipynb`](https://github.com/callahantiff/PheKnowLator/blob/master/notebooks/Data_Preparation.ipynb) Jupyter Notebook.\n",
    "\n",
    "***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify input arguments\n",
    "build = 'full'\n",
    "construction_approach = 'instance'\n",
    "add_node_data_to_kg = 'no'\n",
    "add_inverse_relations_to_kg = 'no'\n",
    "decode_owl_semantics = 'yes'\n",
    "kg_directory_location = './resources/knowledge_graphs'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# construct knowledge graphs\n",
    "if build == 'partial':\n",
    "    kg = PartialBuild(construction=construction_approach,\n",
    "                      node_data=add_node_data_to_kg,\n",
    "                      inverse_relations=add_inverse_relations_to_kg,\n",
    "                      decode_owl=decode_owl_semantics,\n",
    "                      cpus=cpus,\n",
    "                      write_location=kg_directory_location)\n",
    "elif build == 'post-closure':\n",
    "    kg = PostClosureBuild(construction=construction_approach,\n",
    "                          node_data=add_node_data_to_kg,\n",
    "                          inverse_relations=add_inverse_relations_to_kg,\n",
    "                          decode_owl=decode_owl_semantics,\n",
    "                          cpus=cpus,\n",
    "                          write_location=kg_directory_location)\n",
    "else:\n",
    "    kg = FullBuild(construction=construction_approach,\n",
    "                   node_data=add_node_data_to_kg,\n",
    "                   inverse_relations=add_inverse_relations_to_kg,\n",
    "                   decode_owl=decode_owl_semantics,\n",
    "                   cpus=cpus,\n",
    "                   write_location=kg_directory_location)\n",
    "\n",
    "kg.construct_knowledge_graph()\n",
    "ray.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kg.ontologies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "***\n",
    "***\n",
    "\n",
    "```\n",
    "@misc{callahan_tj_2019_3401437,\n",
    "  author       = {Callahan, TJ},\n",
    "  title        = {PheKnowLator},\n",
    "  month        = mar,\n",
    "  year         = 2019,\n",
    "  doi          = {10.5281/zenodo.3401437},\n",
    "  url          = {https://doi.org/10.5281/zenodo.3401437}\n",
    "}\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PyCharm (PheKnowLator)",
   "language": "python",
   "name": "pycharm-5f7498df"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}