Lab_journal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Software info**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|Software     |Version|\n",
    "|-------------|-------|\n",
    "|python    |3.11.9|\n",
    "|ipykernel    |[6.28.0](https://anaconda.org/anaconda/ipykernel)|\n",
    "|Biopython    |[1.70](https://anaconda.org/bioconda/biopython)|\n",
    "|Entrez-direct|[21.6](https://anaconda.org/bioconda/entrez-direct)|\n",
    "|mafft        |[7.525](https://anaconda.org/bioconda/mafft)|\n",
    "|iq-tree2     |[2.3.0](https://anaconda.org/bioconda/iqtree)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conda envinronment: `dstu_hantavirus_phylo.yaml`<br>\n",
    "Install the envinronment with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! conda env create -f dstu_hantavirus_phylo.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload VS Code (close & open), then activate this envinronment as kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Hardware info**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- OS: Ubuntu 22.04 (Windows Subsystem for Linux)\n",
    "- CPU: Intel Xeon E5-2670v3\n",
    "- RAM: 32GB (16GB for WSL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Architecture:            x86_64\n",
      "  CPU op-mode(s):        32-bit, 64-bit\n",
      "  Address sizes:         46 bits physical, 48 bits virtual\n",
      "  Byte Order:            Little Endian\n",
      "CPU(s):                  24\n",
      "  On-line CPU(s) list:   0-23\n",
      "Vendor ID:               GenuineIntel\n",
      "  Model name:            Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz\n",
      "    CPU family:          6\n",
      "    Model:               63\n",
      "    Thread(s) per core:  2\n",
      "    Core(s) per socket:  12\n",
      "    Socket(s):           1\n",
      "    Stepping:            2\n",
      "    BogoMIPS:            4589.36\n",
      "    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mc\n",
      "                         a cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscal\n",
      "                         l nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopo\n",
      "                         logy cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1\n",
      "                          sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervis\n",
      "                         or lahf_lm abm invpcid_single pti ssbd ibrs ibpb stibp \n",
      "                         tpr_shadow vnmi ept vpid ept_ad fsgsbase bmi1 avx2 smep\n",
      "                          bmi2 erms invpcid xsaveopt md_clear flush_l1d arch_cap\n",
      "                         abilities\n",
      "Virtualization features: \n",
      "  Virtualization:        VT-x\n",
      "  Hypervisor vendor:     Microsoft\n",
      "  Virtualization type:   full\n",
      "Caches (sum of all):     \n",
      "  L1d:                   384 KiB (12 instances)\n",
      "  L1i:                   384 KiB (12 instances)\n",
      "  L2:                    3 MiB (12 instances)\n",
      "  L3:                    30 MiB (1 instance)\n",
      "Vulnerabilities:         \n",
      "  Itlb multihit:         KVM: Mitigation: VMX disabled\n",
      "  L1tf:                  Mitigation; PTE Inversion; VMX conditional cache flushe\n",
      "                         s, SMT vulnerable\n",
      "  Mds:                   Mitigation; Clear CPU buffers; SMT Host state unknown\n",
      "  Meltdown:              Mitigation; PTI\n",
      "  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl\n",
      "                          and seccomp\n",
      "  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer\n",
      "                          sanitization\n",
      "  Spectre v2:            Mitigation; Full generic retpoline, IBPB conditional, I\n",
      "                         BRS_FW, STIBP conditional, RSB filling\n",
      "  Srbds:                 Not affected\n",
      "  Tsx async abort:       Not affected\n"
     ]
    }
   ],
   "source": [
    "! lscpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 0. Install `phyloki` tool**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget https://raw.githubusercontent.com/iliapopov17/phyloki/refs/heads/v0.5/phyloki.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phyloki import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 1. Download sequences**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`accession_numbers.txt` file was created manually based on previously published papers:\n",
    "1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10025241/<br>\n",
    "2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7106157/<br>\n",
    "3. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10943075/<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded: NC_038515\n",
      "Downloaded: KT316176\n",
      "Downloaded: MN337866\n",
      "Downloaded: MG663536\n",
      "Downloaded: JX193700\n",
      "Downloaded: KX779125\n",
      "Downloaded: KU950715\n",
      "Downloaded: KR920360\n",
      "Downloaded: OM912841\n",
      "Downloaded: KX845680\n",
      "Downloaded: OM912842\n",
      "Downloaded: OM912840\n",
      "Downloaded: OM963009\n",
      "Downloaded: OM912844\n",
      "Downloaded: OM912843\n",
      "Downloaded: MK165653\n",
      "Downloaded: KJ000540\n",
      "Downloaded: KY040508\n",
      "Downloaded: KT899703\n",
      "Downloaded: MN850095\n",
      "Downloaded: AF005729\n",
      "Downloaded: NC_043407\n",
      "Downloaded: GQ200821\n",
      "Downloaded: EU788002\n",
      "Downloaded: GU997097\n",
      "Downloaded: FJ858378\n",
      "Downloaded: NC_038299\n",
      "Downloaded: AB620102\n",
      "Downloaded: AB620105\n",
      "Downloaded: HM015222\n",
      "Downloaded: MN639746\n",
      "Downloaded: AB677488\n",
      "Downloaded: JX028271\n",
      "Downloaded: KJ857315\n",
      "Downloaded: KJ857316\n",
      "Downloaded: MK883761\n",
      "Downloaded: KY751731\n",
      "Downloaded: MK542664\n",
      "Downloaded: MT441741\n",
      "Downloaded: HQ728461\n",
      "Downloaded: KC880348\n",
      "Downloaded: GU566021\n",
      "Downloaded: KJ857320\n",
      "Downloaded: KF974361\n",
      "Downloaded: JX990941\n",
      "Downloaded: MN006903\n",
      "Downloaded: JX990965\n",
      "Downloaded: KF705679\n",
      "Downloaded: EU929078\n",
      "Downloaded: NC_043175\n",
      "Downloaded: EF050454\n",
      "Downloaded: KC631784\n",
      "Downloaded: HQ834697\n",
      "Downloaded: MF683846\n",
      "Downloaded: KC490923\n",
      "Downloaded: AM998806\n",
      "Downloaded: LC553715\n",
      "Downloaded: GU904045\n",
      "Downloaded: AB620030\n",
      "Downloaded: AY675354\n",
      "Downloaded: MN608087\n",
      "Downloaded: DQ268652\n",
      "Downloaded: KF704716\n",
      "Downloaded: KF704717\n",
      "Downloaded: KF704715\n",
      "Downloaded: JX912953\n",
      "Downloaded: KF704713\n",
      "Downloaded: JN037851\n",
      "Downloaded: KT316176\n",
      "Downloaded: KM102249\n",
      "Downloaded: JX990946\n",
      "Downloaded: KR072623\n",
      "Downloaded: KF031355\n",
      "Downloaded: FJ593498\n",
      "Downloaded: KF010517\n",
      "Downloaded: KU950715\n",
      "Downloaded: JQ287716\n",
      "Downloaded: JX465369\n",
      "Downloaded: EU424341\n",
      "Downloaded: EF641806\n",
      "Downloaded: EF641807\n",
      "Downloaded: JX193700\n",
      "Downloaded: EU001330\n",
      "Downloaded: JX193697\n",
      "Downloaded: JX465379\n",
      "Downloaded: JX465383\n",
      "Downloaded: JX465381\n",
      "Downloaded: JX465382\n",
      "Downloaded: OR684449\n",
      "Downloaded: MG663536\n",
      "Downloaded: NC_038515\n",
      "Downloaded: NC_078262\n",
      "Downloaded: NC_055632\n",
      "Downloaded: NC_034401\n",
      "Downloaded: FJ593498\n",
      "Downloaded: KX512432\n",
      "Downloaded: NC_078485\n",
      "Downloaded: OR934973\n",
      "Downloaded: OR934974\n",
      "All downloads completed.\n"
     ]
    }
   ],
   "source": [
    "get_sequences('iljapopov17@gmail.com', 'data/accession_numbers.txt', 'genbank_sequences')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 1.1. Check downloaded sequences**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "93\n"
     ]
    }
   ],
   "source": [
    "! ls genbank_sequences/| wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of accession numbers is 99, but there are 93 downloaded sequences. There are non unique accession numbers extracted from one of these papers:\n",
    "1. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10025241/<br>\n",
    "2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7106157/<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_non_unique_strings(file_path):\n",
    "    counts = {}\n",
    "    non_unique_count = 0\n",
    "    with open(file_path, 'r') as file:\n",
    "        for line in file:\n",
    "            line = line.strip()\n",
    "            if line in counts:\n",
    "                if counts[line] == 1:\n",
    "                    non_unique_count += 1\n",
    "                counts[line] += 1\n",
    "            else:\n",
    "                counts[line] = 1\n",
    "\n",
    "    return non_unique_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of non-unique strings: 6\n"
     ]
    }
   ],
   "source": [
    "file = 'data/accession_numbers.txt'\n",
    "non_unique = count_non_unique_strings(file)\n",
    "print(f\"Number of non-unique strings: {non_unique}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above mentioned papers utilize 6 identical sequences for building the phylogenetic tree. 93 sequences downloaded is explained in that case. No data has been lost."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 1.2. 10 additional sequences**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "10 additional hantaviruses associated with human infections will be included in our analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Orthohantavirus seoulense**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/MG386252.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/29774860/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"MG386252\" | efetch -format fasta > genbank_sequences/MG386252.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"MG386252\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/OR047284.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/38147030/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"OR047284\" | efetch -format fasta > genbank_sequences/OR047284.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"OR047284\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Orthohantavirus tulaense**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/KU297981.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/26691901/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"KU297981\" | efetch -format fasta > genbank_sequences/KU297981.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"KU297981\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/MT993951.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/33754997/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"MT993951\" | efetch -format fasta > genbank_sequences/MT993951.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"MT993951\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Orthohantavirus dobravaense**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/MK605664.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/31625853/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"MK605664\" | efetch -format fasta > genbank_sequences/MK605664.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"MK605664\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/MK605665.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/31625853/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"MK605665\" | efetch -format fasta > genbank_sequences/MK605665.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"MK605665\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hantaan orthohantavirus**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/MW349026.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/34370707/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"MW349026\" | efetch -format fasta > genbank_sequences/MW349026.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"MW349026\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/MZ191082.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/34370707/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"MZ191082\" | efetch -format fasta > genbank_sequences/MZ191082.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"MZ191082\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Orthohantavirus sinnombreense**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/ON571586.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/37486231/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"ON571586\" | efetch -format fasta > genbank_sequences/ON571586.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"ON571586\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.ncbi.nlm.nih.gov/nuccore/ON571589.1<br>\n",
    "https://pubmed.ncbi.nlm.nih.gov/37486231/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "! esearch -db nucleotide -query \"ON571589\" | efetch -format fasta > genbank_sequences/ON571589.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "! echo \"ON571589\" >> data/accession_numbers.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 2. Combine all sequences to one file**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cat genbank_sequences/*.fasta > all_seqs.fa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of sequences in combined file: 103\n"
     ]
    }
   ],
   "source": [
    "with open(\"all_seqs.fa\", \"r\") as fasta_file:\n",
    "    content = fasta_file.read()\n",
    "    num_sequences = content.count(\">\")\n",
    "print(f\"The number of sequences in combined file: {num_sequences}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 3. Multiple sequence alignment**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mafft --auto data/all_seqs.fa > data/all_seqs_mafft.fa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 4. Launching `ModelFinder` to get the best substitution model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! iqtree2 -m MFP -s data/all_seqs_mafft.fa --prefix model-finder/tree_MF2 -T AUTO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best-fit model according to BIC: GTR+F+I+G4\n",
      "\n",
      "List of models sorted by BIC scores: \n",
      "\n",
      "Model                  LogL         AIC      w-AIC        AICc     w-AICc         BIC      w-BIC\n",
      "GTR+F+I+G4      -209288.601  419003.202 -  0.00514  419017.199 -   0.0154  420454.559 +        1\n"
     ]
    }
   ],
   "source": [
    "! head -42 model-finder/tree_MF2.iqtree | tail -6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 5. Building the final tree**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`iq-tree2` launch with the best substitution model & generating 1000 replicas of ultrafast bootstrap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! iqtree2 -s data/all_seqs_mafft.fa -m GTR+F+I+G4 -pre tree/tree_ufb -bb 1000 -nt AUTO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 6. First tree visualisation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this purpose [iTOL](https://itol.embl.de) software was used with followed tree annotation in Pixelmator Pro.<br>\n",
    "To visualise the tree `tree_ufb.treefile` must be uploaded to [iTOL](https://itol.embl.de)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 7. Annotating the tree**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 7.1. Returning organisms names to the tree**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request has been fulfilled.\n",
      "File saved to data/accession_organism.txt\n"
     ]
    }
   ],
   "source": [
    "get_organisms('iljapopov17@gmail.com', 'data/accession_numbers.txt', 'data/accession_organism.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NC_038515.1 Laibin virus\n",
      "KT316176.1 Makokou virus\n",
      "MN337866.1 Sarawak mobatvirus\n",
      "MG663536.1 Dakrong virus\n",
      "JX193700.1 Kilimanjaro virus\n"
     ]
    }
   ],
   "source": [
    "! head -5 data/accession_organism.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything worked well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 7.2. Updating the tree**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request has been fulfilled.\n",
      "File saved to tree/annotated_tree.treefile\n"
     ]
    }
   ],
   "source": [
    "update_tree('data/accession_organism.txt', 'tree/tree_ufb.treefile', 'tree/annotated_tree.treefile')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(AB620030.1 Amur virus:0.0771041878,((((((((((AB620102.1 Orthohantavirus montanoense:0.3586812403,AB620105.1 Carrizal virus:0.3164538911)100:0.0821256184,(ON571586.1 Orthohantavirus sinnombreense:0.0003496417,ON571589.1 Orthohantavirus sinnombreense:0.0000027792)100:0.3526346556)94:0.0679804387,(((((AF005729.1 Orthohantavirus negraense:0.3758037094,NC_043407.1 Necocli virus:0.3953678541)99:0.0611167013,MN850095.1 Orthohantavirus andesense:0.2115210053)99:0.1127135879,EU788002.1 Maporal virus:0.3285749060)99:0.0815653694,GQ200821.1 Orthohantavirus delgaditoense:0.3673061736)78:0.0440712496,(FJ858378.1 Catacamas virus:0.2198999425,(GU997097.1 Orthohantavirus nigrorivense:0.2424566531,NC_038299.1 Orthohantavirus bayoui:0.2143819459)97:0.0646432754)100:0.1905032908)100:0.0596284326)100:0.1483520267,((((AB677488.1 Ussuri virus:0.2315952341,JX028271.1 Muju virus:0.2947312110)57:0.0621199827,MN639746.1 Orthohantavirus puumalaense:0.2298150170)100:0.1004645997,(((((HQ728461.1 Orthohantavirus tulaense:0.0991419758,MT993951.1 Orthohantavirus tulaense:0.1198827908)100:0.0800585338,KU297981.1 Orthohantavirus tulaense:0.1618395760)100:0.1912227809,(((KY751731.1 Kielder hantavirus:0.1151455066,MK883761.1 Orthohantavirus tatenalense:0.1215848998)100:0.0990470187,MK542664.1 Traemmersee orthohantavirus:0.1986814928)100:0.0574368767,MT441741.1 Rusne orthohantavirus:0.1899661246)100:0.0618239866)98:0.0857654171,KJ857315.1 Orthohantavirus khabarovskense:0.2791373880)84:0.0473592676,KJ857316.1 Yuanjiang virus:0.2726077642)95:0.0542967810)100:0.1992801697,KT899703.1 Fugong virus:0.4700677119)100:0.1130002341)69:0.0567435563,HM015222.1 Rockport virus:0.6082931559)100:0.2677252605,(((((EF641806.1 Imjin virus:0.0146151276,KX779125.1 Imjin virus:0.0138818167)100:0.0769890463,EF641807.1 Imjin virus:0.0804300980)100:0.3346745369,EU001330.1 Thottapalayam virus:0.4228360394)100:0.2536585750,(JX193697.1 Uluguru virus:0.6692053051,JX193700.1 Kilimanjaro virus:0.4766759736)100:0.1298513865)100:0.4854320839,((((((EU424341.1 Altai virus:0.2654389245,NC_078485.1 Lena virus:0.2155769500)100:0.5582020229,JX990965.1 Boginia virus:0.5363602544)78:0.3823058529,((((FJ593498.1 Nova virus:0.1138267013,(KF010517.1 Nova virus:0.1090607191,KF031355.1 Nova virus:0.1938245416)98:0.0302351889)98:0.0252765011,KX512432.1 Nova virus:0.1341231385)99:0.0561960783,(JX990946.1 Nova virus:0.0340618990,KR072623.1 Nova virus:0.0079225951)100:0.0732037168)100:0.2867605796,KY040508.1 Asturias virus:0.2039299345)100:0.5707414930)58:0.0916991172,(((JN037851.1 Hantavirus MGB/1209:0.6361097514,KT316176.1 Makokou virus:0.3812644990)66:0.0456683944,((JX912953.2 Xuan son virus:0.1408219222,KF704713.1 Xuan son virus:0.1620798600)100:0.1150648055,((KF704715.1 Xuan son virus:0.0038421865,(KF704716.1 Xuan son virus:0.0062914844,KF704717.1 Xuan son virus:0.0145854887)100:0.0188894736)100:0.1802940875,NC_078262.1 Xuan son virus:0.1752778191)98:0.0584955771)100:0.2695940521)90:0.0551921386,(((KM102249.1 Laibin virus:0.0000010001,NC_038515.1 Laibin virus:0.0000010001)100:0.2426973377,MN337866.1 Sarawak mobatvirus:0.2544616999)98:0.1417640648,MG663536.1 Dakrong virus:0.4933778150)99:0.0875153991)98:0.2337792501)57:0.0683692102,((JX465369.1 Huangpi virus:0.5393286223,OR684449.1 Buritiense virus:0.5548160827)47:0.1175896767,((KU950715.1 Quezon virus:0.0000010001,NC_034401.1 Quezon virus:0.0000010001)100:0.5732888012,(MK165653.1 Orthohantavirus robinaense:0.0000010001,NC_055632.1 Orthohantavirus robinaense:0.0000010001)100:0.5309016345)100:0.2607446493)41:0.0898181287)42:0.0764483214,((JQ287716.2 Mouyassue virus:0.0000023848,KJ000540.1 Mouyassue virus:0.0000010001)100:0.4267644543,((JX465379.1 Longquan virus:0.0000020853,((JX465381.1 Longquan virus:0.0038141945,JX465382.1 Longquan virus:0.0077268601)100:0.0196543002,JX465383.1 Longquan virus:0.0000020681)96:0.0077421990)100:0.3703425331,(KR920360.1 Brno virus:0.0000021791,((((KX845680.1 Brno virus:0.0037963581,(OM912843.1 Loanvirus brunaense:0.0137485885,(OM912844.1 Loanvirus brunaense:0.0000010001,OM963009.1 Loanvirus brunaense:0.0000010001)69:0.0000020133)97:0.0033926390)55:0.0000026778,(OM912840.1 Loanvirus brunaense:0.0000010001,OM912842.1 Loanvirus brunaense:0.0000010001)99:0.0134331563)100:0.0542071098,(OR934973.1 Loanvirus brunaense:0.0049500299,OR934974.1 Loanvirus brunaense:0.0163547086)95:0.0197518443)95:0.0236773267,OM912841.1 Loanvirus brunaense:0.0180655310)39:0.0207196720)100:0.4180488781)90:0.1910070183)51:0.2445995949)76:0.1260525283)79:0.2957849738)100:0.2132251977,((((EF050454.1 Tanganya virus:0.4626176976,KC631784.1 Bowe virus:0.3150541234)99:0.1191717914,HQ834697.1 Jeju virus:0.4504119611)100:0.1997444220,(((EU929078.1 Asama virus:0.3651385985,((GU566021.1 Qian Hu Shan virus:0.2054115794,KC880348.1 Orthohantavirus asikkalaense:0.2583087016)80:0.0478747851,((JX990941.1 Seewis virus:0.2461045089,KF974361.1 Amga virus:0.2324359653)99:0.0505468662,KJ857320.1 Kenkeme virus:0.2844332666)86:0.0578482552)94:0.0883369661)99:0.0719271114,(KF705679.1 Xinyi virus:0.2888953092,MN006903.1 Jemez Springs virus:0.3470836938)100:0.0839674435)100:0.0520037869,NC_043175.1 Oxbow virus:0.5409916277)100:0.0713003641)100:0.0952947788,MF683846.1 Orthohantavirus brugesense:0.6818134739)59:0.0500300676)100:0.2567808467,((AM998806.1 Serang virus:0.1469951860,(KC490923.1 Anjozorobe virus:0.2429580339,LC553715.1 Orthohantavirus thailandense:0.2435300852)83:0.0474757758)100:0.1384906244,((GU904045.1 Orthohantavirus dobravaense:0.0902845208,MK605664.1 Orthohantavirus dobravaense:0.1225113023)98:0.0606089420,MK605665.1 Orthohantavirus dobravaense:0.0399085736)100:0.2020387586)89:0.0690533160)85:0.0190746178,(DQ268652.1 Orthohantavirus sangassouense:0.3466753654,(MG386252.1 Orthohantavirus seoulense:0.0137372170,OR047284.1 Orthohantavirus seoulense:0.0548624754)100:0.2848089750)91:0.0911279902)100:0.1716524032,(MN608087.1 Orthohantavirus hantanense:0.1744392228,(MW349026.1 Orthohantavirus hantanense:0.0248825313,MZ191082.1 Orthohantavirus hantanense:0.0110645270)100:0.1996850841)99:0.0404345463)100:0.1049070173,AY675354.1 Soochong virus-2:0.0948964879);\n"
     ]
    }
   ],
   "source": [
    "! head -1 tree/annotated_tree.treefile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now tree leaves have accession number + organism name (AB620030.1 Amur virus) instead of just an accession number (AB620030.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 7.3. Fetching information about viruses hosts**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request has been fulfilled.\n",
      "File saved to data/accession_host.txt\n"
     ]
    }
   ],
   "source": [
    "get_hosts('iljapopov17@gmail.com', 'data/accession_numbers.txt', 'data/accession_host.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NC_038515.1 Taphozous melanopogon\n",
      "KT316176.1 Hipposideros ruber\n",
      "MN337866.1 Murina aenea\n",
      "MG663536.1 Aselliscus stoliczkanus (Stoliczka's Asian trident bat)\n",
      "JX193700.1 Myosorex zinki\n"
     ]
    }
   ],
   "source": [
    "! head -5 data/accession_host.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything worked well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 7.4. Fetching information about viruses hosts's phylogenetic order**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The request has been fulfilled.\n",
      "File saved to data/accession_order.txt\n",
      "Please do not forget to edit the file manually.\n",
      "The query to NCBI database from this function is pretty difficult.\n",
      "Sometimes this function prints:\n",
      "\"Error - HTTP Error 400: Bad Request\" in case of bad connection or\n",
      "\"Note - False record\" in case there is no record about the host organism.\n"
     ]
    }
   ],
   "source": [
    "get_hosts_orders('iljapopov17@gmail.com', 'data/accession_host.txt', 'data/accession_order.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NC_038515.1\tChiroptera\n",
      "KT316176.1\tChiroptera\n",
      "MN337866.1\tChiroptera\n",
      "MG663536.1\tChiroptera\n",
      "JX193700.1\tEulipotyphla\n"
     ]
    }
   ],
   "source": [
    "! head -5 data/accession_order.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything worked well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Step 7.5. Setting up the color map for visualization in iTOL**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Chiroptera', 'Eulipotyphla', 'Rodentia', 'ND', 'Primates']\n"
     ]
    }
   ],
   "source": [
    "unique_orders = get_unique_orders(\"data/accession_order.txt\")\n",
    "print(unique_orders)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Chiroptera': '#32cd32', 'Eulipotyphla': '#ffd700', 'Rodentia': '#1e90ff', 'ND': '#FFFFFF', 'Primates': '#8a2be2'}\n"
     ]
    }
   ],
   "source": [
    "color_map = set_color_map(\"data/accession_order.txt\")\n",
    "print(color_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Colors were set by the user.\n",
      "The request has been fulfilled.\n"
     ]
    }
   ],
   "source": [
    "get_itol_dataset(\"data/accession_organism.txt\", \"data/accession_order.txt\", \"data/dataset_for_iTOL.txt\", color_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 8. Final tree visualization**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Visit [iTOL](https://itol.embl.de)\n",
    "2. Upload `tree/annotated_tree.treefile` as the tree\n",
    "3. Upload `data/dataset_for_iTOL.txt` as the annotation dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Step 9. Geographical map of _Brno loanvirus_ distribution**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download [`Natural Earth II with Shaded Relief, Water, and Drainages`](https://www.naturalearthdata.com/downloads/10m-natural-earth-2/10m-natural-earth-2-with-shaded-relief-water-and-drainages/) map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! wget https://naturalearth.s3.amazonaws.com/10m_raster/NE2_HR_LC_SR_W_DR.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unzip downloaded archive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Make sure `unzip` is installed in your system<br>\n",
    ">For `Ubuntu` please write in your terminal:\n",
    ">```\n",
    ">sudo apt-get install unzip\n",
    ">```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! unzip NE2_HR_LC_SR_W_DR.zip -d map/NE2_HR_LC_SR_W_DR && rm -rf NE2_HR_LC_SR_W_DR.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please open `RStudio` and go through `map/geo_map_journal.R` script.\n",
    "There are a lot of manual adjustments to the plots to make it executable."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "phylo4",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}