How to install the NCBI database manually

[Ronflatau]

I tried to use the automated installation for the database but I got the following error.
[08/03/2023 08:44:38 AM DEBUG] autometa.config.databases: MISSING: (ncbi,names)
[08/03/2023 08:44:38 AM DEBUG] autometa.config.databases: MISSING: (ncbi,merged)
[08/03/2023 08:44:38 AM DEBUG] autometa.config.databases: MISSING: (ncbi,nodes)
[08/03/2023 08:44:38 AM DEBUG] autometa.config.databases: MISSING: (ncbi,delnodes)
[08/03/2023 08:44:38 AM DEBUG] autometa.config.databases: UPDATE: (ncbi,taxdump): /work/kraken1/RonFlatau/Discovery/AutometaNCBI//taxdump.tar.gz
[08/03/2023 08:44:38 AM DEBUG] autometa.config.databases: starting taxdump download
Traceback (most recent call last):
File "/home/r.flatau/.conda/envs/autometa/bin/autometa-update-databases", line 10, in
sys.exit(main())
File "/home/r.flatau/.conda/envs/autometa/lib/python3.9/site-packages/autometa/config/databases.py", line 833, in main
config = dbs.configure(section=section, no_checksum=args.no_checksum)
File "/home/r.flatau/.conda/envs/autometa/lib/python3.9/site-packages/autometa/config/databases.py", line 707, in configure
self.download_missing(section=section)
File "/home/r.flatau/.conda/envs/autometa/lib/python3.9/site-packages/autometa/config/databases.py", line 575, in download_missing
dispatchersection
File "/home/r.flatau/.conda/envs/autometa/lib/python3.9/site-packages/autometa/config/databases.py", line 390, in download_ncbi_files
subprocess.run(
File "/home/r.flatau/.conda/envs/autometa/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rsync', '--quiet', '--archive', 'rsync://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz', '/work/kraken1/RonFlatau/Discovery/AutometaNCBI//taxdump.tar.gz']' returned non-zero exit status 10.

Therefore, I download the file manually but I don't find any instruction on how to install the database from that point.

Thanks for the help

[evanroyrees]

Hi Ron,

Can you share what files you have here? /work/kraken1/RonFlatau/Discovery/AutometaNCBI/

It looks like your config path may have an additional /? What does autometa-config --print look like?

[Ronflatau]

Thanks for the help this is the file I have there;

and this is how the autometa-config --print look likes

(autometa) [r.flatau@login-00 ~]$ autometa-config --print
[08/04/2023 03:26:19 PM DEBUG] root: environment dependencies satisifed: True
section option value
common home_dir /home/r.flatau/.conda/envs/autometa/lib/python3.9/site-packages
environ diamond /home/r.flatau/.conda/envs/autometa/bin/diamond
environ hmmsearch /home/r.flatau/.conda/envs/autometa/bin/hmmsearch
environ hmmpress /home/r.flatau/.conda/envs/autometa/bin/hmmpress
environ hmmscan /home/r.flatau/.conda/envs/autometa/bin/hmmscan
environ prodigal /home/r.flatau/.conda/envs/autometa/bin/prodigal
environ bowtie2 /home/r.flatau/.conda/envs/autometa/bin/bowtie2
environ samtools /home/r.flatau/.conda/envs/autometa/bin/samtools
environ bedtools /home/r.flatau/.conda/envs/autometa/bin/bedtools
versions diamond 2.1.8
versions hmmsearch 3.3.2
versions hmmpress 3.3.2
versions hmmscan 3.3.2
versions prodigal 2.6.3
versions bowtie2 2.5.1
versions samtools 1.17
versions bedtools 2.31.0
databases base /home/r.flatau/.conda/envs/autometa/lib/python3.9/site-packages/autometa/databases
databases ncbi /work/kraken1/RonFlatau/Discovery/AutometaNCBI
databases gtdb /work/kraken1/RonFlatau/Discovery/autometa/GTDBdatabases
databases markers /work/kraken1/RonFlatau/Discovery/Automematmarker
database_urls taxdump ftp://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz
database_urls accession2taxid ftp://ftp.ncbi.nlm.nih.gov/pub/taxonomy/accession2taxid/prot.accession2taxid.gz
database_urls nr ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/nr.gz
database_urls bacteria_single_copy https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/bacteria.single_copy.hmm
database_urls bacteria_single_copy_cutoffs https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/bacteria.single_copy.cutoffs
database_urls archaea_single_copy https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/archaea.single_copy.hmm
database_urls archaea_single_copy_cutoffs https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/archaea.single_copy.cutoffs
database_urls proteins_aa_reps https://data.gtdb.ecogenomic.org/releases/latest/genomic_files_reps/gtdb_proteins_aa_reps.tar.gz
database_urls gtdb_taxdmp https://github.com/shenwei356/gtdb-taxdump/releases/latest/download/gtdb-taxdump.tar.gz
checksums taxdump ftp://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz.md5
checksums accession2taxid ftp://ftp.ncbi.nlm.nih.gov/pub/taxonomy/accession2taxid/prot.accession2taxid.gz.md5
checksums nr ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/nr.gz.md5
checksums bacteria_single_copy https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/bacteria.single_copy.hmm.md5
checksums bacteria_single_copy_cutoffs https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/bacteria.single_copy.cutoffs.md5
checksums archaea_single_copy https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/archaea.single_copy.hmm.md5
checksums archaea_single_copy_cutoffs https://raw.githubusercontent.com/KwanLab/Autometa/main/autometa/databases/markers/archaea.single_copy.cutoffs.md5
ncbi host ftp.ncbi.nlm.nih.gov
ncbi taxdump /work/kraken1/RonFlatau/Discovery/AutometaNCBI/taxdump.tar.gz
ncbi nodes /work/kraken1/RonFlatau/Discovery/AutometaNCBI/nodes.dmp
ncbi names /work/kraken1/RonFlatau/Discovery/AutometaNCBI/names.dmp
ncbi merged /work/kraken1/RonFlatau/Discovery/AutometaNCBI/merged.dmp
ncbi delnodes /work/kraken1/RonFlatau/Discovery/AutometaNCBI/delnodes.dmp
ncbi accession2taxid /work/kraken1/RonFlatau/Discovery/AutometaNCBI/prot.accession2taxid.gz
ncbi nr /work/kraken1/RonFlatau/Discovery/AutometaNCBI/nr.gz
gtdb host data.gtdb.ecogenomic.org
gtdb release latest
gtdb proteins_aa_reps /work/kraken1/RonFlatau/Discovery/autometa/GTDBdatabases/gtdb_proteins_aa_reps.tar.gz
gtdb gtdb_taxdmp /work/kraken1/RonFlatau/Discovery/autometa/GTDBdatabases/gtdb-taxdump.tar.gz
markers host raw.githubusercontent.com
markers bacteria_single_copy /work/kraken1/RonFlatau/Discovery/Automematmarker/bacteria.single_copy.hmm
markers bacteria_single_copy_cutoffs /work/kraken1/RonFlatau/Discovery/Automematmarker/bacteria.single_copy.cutoffs
markers archaea_single_copy /work/kraken1/RonFlatau/Discovery/Automematmarker/archaea.single_copy.hmm
markers archaea_single_copy_cutoffs /work/kraken1/RonFlatau/Discovery/Automematmarker/archaea.single_copy.cutoffs
files metagenome metagenome.fna
files fwd_reads fwd_reads.fastq
files rev_reads rev_reads.fastq
files se_reads se_reads.fastq
files sam alignments.sam
files bam alignments.bam
files lengths lengths.tsv
files bed alignments.bed
files length_filtered metagenome.filtered.fna
files coverages coverages.tsv
files kmer_counts kmers.tsv
files kmer_normalized kmers.normalized.tsv
files kmer_embedded kmers.embedded.tsv
files nucleotide_orfs metagenome.filtered.orfs.fna
files amino_acid_orfs metagenome.filtered.orfs.faa
files blastp blastp.tsv
files blastp_hits blastp.hits.pkl.gz
files lca lca.tsv
files blastx blastx.tsv
files taxonomy taxonomy.tsv
files bacteria_hmmscan bacteria.hmmscan.tsv
files bacteria_markers bacteria.markers.tsv
files archaea_hmmscan archaea.hmmscan.tsv
files archaea_markers archaea.markers.tsv
files bacteria_binning bacteria.binning.tsv
files archaea_binning archaea.binning.tsv
files checkpoints checkpoints.tsv

[evanroyrees]

Hi Ron,

I apologize for the late reply. I am uncertain as to why you are encountering errors here. Could you try to configure your databases with a common database directory?

For example

NOTE: The trailing / should be missing when passing in these values

# This will set the base directory
autometa-config --section databases --option base --value /work/kraken1/RonFlatau/Discovery/AutometaDatabases

Now the ncbi and markers database directories should be located as sub-dirs of /work/kraken1/RonFlatau/Discovery/AutometaDatabases

I.e.

/work/kraken1/RonFlatau/Discovery/AutometaDatabases/{markers,ncbi}

However

Since you have already configured each of the component databases respectively, this will not set each of these automagically.

Therefore, in addition to the above command, you will need to update the component paths:

NCBI

# ncbi
autometa-config --section databases --option ncbi --value /work/kraken1/RonFlatau/Discovery/AutometaDatabases/ncbi

GTDB

# gtdb
autometa-config --section databases --option gtdb --value /work/kraken1/RonFlatau/Discovery/AutometaDatabases/gtdb

markers

# markers
autometa-config --section databases --option markers --value /work/kraken1/RonFlatau/Discovery/AutometaDatabases/markers

Also, we have released v2.2.1 with a GTDB database setup bug fix. This is not yet available on bioconda, but should be available in less than 24hours.

Can you give this a try and then re-perform the autometa-update-databases command? If you are still running in to errors can you reply with the updated config information and logs?

autometa-config --print