Combining individual circRNA read counts - error #68

mihinduk · 2019-09-16T21:13:59Z

Describe the bug
DCC quits when trying to combine individual circRNA read counts. This is only when I run it using a docker in an interactive queue:
bsub -Is -q research-hpc
-a 'docker(buddej/dcc:0.1.3)'
/bin/bash

I have run this locally without issue and am trying to run it on a larger server to run datasets that demand too much memory for my local machine (although I am testing on a small dataset of 83 samples)

Since I have to convert the wrapper I wrote from python3 to python2.17.16 to be compatible with DCC, I have isolated the actual DCC command (The previous steps of generating the infiles worked) and have been just running this inside the interactive queue:

To Reproduce
Steps to reproduce the behavior:

Command line used for the command:
DCC @/gscmnt/gc2645/wgs/km_test/dcc/gtex/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/Amygdala/samplesheet -mt1 @/gscmnt/gc2645/wgs/km_test/dcc/gtex/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/Amygdala/mate1 -mt2 @/gscmnt/gc2645/wgs/km_test/dcc/gtex/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/Amygdala/mate2 -T 20 -D -R /gscmnt/gc2645/wgs/resources/RNAseq/genome/hg19_Repeats_RepeatMasker_SimpleRepeats.gtf -an /gscmnt/gc2645/wgs/resources/RNAseq/genome/gencode.v19.annotation.spike-in.gtf -Pi -F -M -Nr 1 1 -fg -k -G -A /gscmnt/gc2645/wgs/resources/RNAseq/genome/GRCh37.p13.genome.lite.spike-in.fa -B @/gscmnt/gc2645/wgs/km_test/dcc/gtex/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/Amygdala/bam_files
Complete error message
finished circRNA detection from file _tmp_DCC/SRR818418_unified.Chimeric.out.junction.7MIX0L
Combining individual circRNA read counts
Traceback (most recent call last):
File "/usr/local/bin/DCC", line 11, in
load_entry_point('DCC==0.4.7', 'console_scripts', 'DCC')()
File "/usr/local/lib/python2.7/site-packages/DCC-0.4.7-py2.7.egg/DCC/main.py", line 287, in main
File "/usr/local/lib/python2.7/site-packages/DCC-0.4.7-py2.7.egg/DCC/circAnnotate.py", line 26, in selectGeneGtf
File "/usr/local/lib/python2.7/site-packages/HTSeq/init.py", line 197, in iter
for line in FileOrSequence.iter(self):
File "/usr/local/lib/python2.7/site-packages/HTSeq/init.py", line 50, in iter
for line in lines:
IOError: [Errno 14] Bad address

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. Debian 8.6, Ubuntu 18.04]
Python version [e.g. 3.6, 3.4]
Version [e.g. 1.1.0.1]

python = python2.17.16
Version = DCC 0.4.7
Dockerfile = buddej/dcc:0.1.3
https://github.com/buddej/mgi-hpc/blob/master/dcc/Dockerfile**

Any advice you can give would be greatly appreciated.

Thank you,
Kathie Mihindukulasuriya

mihinduk · 2019-10-02T19:13:43Z

I have solved the problem for PE data by setting -T 2 and requesting more memory based on the stats generated by running /usr/bin/time -v . For SE data, if I run the identical command on a local server it runs, but when I try to run on a remote server using Docker, I get the following error:

Command:
/usr/bin/time -v DCC @/gscmnt/gc2645/wgs/km_test/dcc/msbb/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/BM36/samplesheet -mt1 @/gscmnt/gc2645/wgs/km_test/dcc/msbb/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/BM36/read -T 2 -D -N -R /gscmnt/gc2645/wgs/resources/RNAseq/genome/hg19_Repeats_RepeatMasker_SimpleRepeats.gtf -an /gscmnt/gc2645/wgs/resources/RNAseq/genome/gencode.v19.annotation.spike-in.gtf -F -M -Nr 1 1 -fg -k -G -A /gscmnt/gc2645/wgs/resources/RNAseq/genome/GRCh37.p13.genome.lite.spike-in.fa -B @/gscmnt/gc2645/wgs/km_test/dcc/msbb/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/BM36/bam_files -O /gscmnt/gc2645/wgs/km_test/dcc/msbb/02.-ProcessedData/06.-circRNA/Hg19/DCC/BM36 -t /gscmnt/gc2645/wgs/km_test/dcc/msbb/02.-ProcessedData/06.-circRNA/Hg19/DCC/DCC_InputFiles/BM36/_tmp_DCC

Error:
Traceback (most recent call last):
File "/usr/local/bin/DCC", line 11, in
load_entry_point('DCC==0.4.7', 'console_scripts', 'DCC')()
File "/usr/local/lib/python2.7/site-packages/DCC-0.4.7-py2.7.egg/DCC/main.py", line 145, in main
File "/usr/local/lib/python2.7/site-packages/DCC-0.4.7-py2.7.egg/DCC/main.py", line 529, in remove_empty_lines
TypeError: 'NoneType' object is not iterable
Command exited with non-zero status 1

JunmingH · 2019-11-18T21:39:13Z

I met the same issue, Do you have any ideas to solve this?

mihinduk · 2019-11-18T21:44:17Z

I would try increasing the amount of memory you request:

Dataset | tissue | samples | Cores | time | Max mem
MSBB | BM10 | 325 | 2 | 56:43:04 | 282082312
MSBB | BM22 | 334 | 4 | 44:50:42 | 313739268
MSBB | BM36 | 315 | 4 | 37:54:52 | 296575084
MSBB | BM44 | 308 | 4 | 40:16:14 | 286656860

JunmingH · 2019-11-19T15:00:24Z

Thanks For your reply! Is this the solution for first issue or second issue?

tjakobi · 2019-11-19T15:24:15Z

Hi @mihinduk,
hi @JunmingH,

thank you for reporting the issues and for your patience.

increasing the memory should fix issue 1, since here the error message refers to a bad memory address: IOError: [Errno 14] Bad address. So requesting more memory on cluster scheduled environments will solve that issue.

The second issue looks familiar. Could it be, that you forgot to specify -mt2?

Cheers,
Tobias

mihinduk · 2019-11-19T15:26:38Z

The second error is on SE data. It takes more memory and time than the PE data. I was able to get this to run locally, but not yet on a server, using a Dockerfile. From: Tobias Jakobi <notifications@github.com> Sent: Tuesday, November 19, 2019 9:24 AM To: dieterich-lab/DCC <DCC@noreply.github.com> Cc: Mihindukulasuriya, Kathie <mihindu@wustl.edu>; Mention <mention@noreply.github.com> Subject: Re: [dieterich-lab/DCC] Combining individual circRNA read counts - error (#68) Hi @mihinduk<https://github.com/mihinduk>, hi @JunmingH<https://github.com/JunmingH>, increasing the memory should fix issue 1, since here the error message refers to a bad memory address: IOError: [Errno 14] Bad address. So requesting more memory on cluster scheduled environments will solve that issue. The second issue looks familiar. Could it be, that you forgot to specify -mt2? Cheers, Tobias — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#68?email_source=notifications&email_token=ANDVLDIZIALZTVRSZ66ERKLQUQAKDA5CNFSM4IXHJYDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEOSELA#issuecomment-555557420>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANDVLDI6VD3BH7RE4JNUVXDQUQAKDANCNFSM4IXHJYDA>.

…

________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

tjakobi · 2019-11-19T15:29:47Z

Hi @mihinduk,

for SE data you must not use -mt1. -mt1 and -mt2 are reserved for PE setups.

Cheers,
Tobias

mihinduk · 2019-11-19T15:36:09Z

Hi Tobias,

Do you have any idea why this would run locally:
/usr/bin/time -v DCC @/40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/DCC_InputFiles/samplesheet -mt1 @/40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/DCC_InputFiles/read -T 4 -D -N -R /40/pipelines/RNAseq/circRNA/hg19_Repeats_RepeatMasker_SimpleRepeats.gtf -an /40/pipelines/RNAseq/circRNA/Hg19_gencodev19_spikein/gencode.v19.annotation.spike-in.gtf -F -M -Nr 1 1 -fg -k -G -A /40/pipelines/RNAseq/circRNA/Hg19_gencodev19_spikein/GRCh37.p13.genome.lite.spike-in.fa -B @/40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/DCC_InputFiles/bam_files -O /40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/ -t /40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/_tmp_DCC

JunmingH · 2019-11-19T15:42:08Z

@tjakobi Hi Tobias,
I am also run SE data without specify the -mt1 and -mt2. But still have the same error message.
I try to integrate all the files with the location information in one files to run it. But same error.

JunmingH · 2019-11-19T17:03:15Z

@tjakobi Hi Tobias,

I have a question about combine the results. Since I could not run all files at same time. Therefore, I run it one by one and store in different directory. I was wondering how could I combine them together? since each subject have different results. How can I treat the missing circRNA?

Thanks!

tjakobi · 2019-11-24T19:15:51Z

Hi Tobias,

Do you have any idea why this would run locally:
/usr/bin/time -v DCC @/40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/DCC_InputFiles/samplesheet -mt1 @/40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/DCC_InputFiles/read -T 4 -D -N -R /40/pipelines/RNAseq/circRNA/hg19_Repeats_RepeatMasker_SimpleRepeats.gtf -an /40/pipelines/RNAseq/circRNA/Hg19_gencodev19_spikein/gencode.v19.annotation.spike-in.gtf -F -M -Nr 1 1 -fg -k -G -A /40/pipelines/RNAseq/circRNA/Hg19_gencodev19_spikein/GRCh37.p13.genome.lite.spike-in.fa -B @/40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/DCC_InputFiles/bam_files -O /40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/ -t /40/Public_Data/bulkRNASeq/201812_MSBB/Gene_Expression/02.-ProcessedData/06.-circRNA/Hg19/BM44/DCC/_tmp_DCC

I am not sure how DCC handles the case where only mate1 is supplied like in your example. Generally it's either both , -mt1 AND -mt2 or none of both, the I am pretty sure your command with only -mt1 is not behavign correctly. Anyway, this needs to be addressed in the code before DCC starts running.

tjakobi · 2019-11-24T19:18:17Z

@tjakobi Hi Tobias,
I am also run SE data without specify the -mt1 and -mt2. But still have the same error message.
I try to integrate all the files with the location information in one files to run it. But same error.

Hi @JunmingH,

are you referring to the

remove_empty_lines
TypeError: 'NoneType' object is not iterable
Command exited with non-zero status 1

Error?

Cheers,
Tobias

tjakobi · 2019-11-24T19:20:19Z

@tjakobi Hi Tobias,

I have a question about combine the results. Since I could not run all files at same time. Therefore, I run it one by one and store in different directory. I was wondering how could I combine them together? since each subject have different results. How can I treat the missing circRNA?

Thanks!

Hi @JunmingH,

please see my response in your new issue: #72 (comment)

Cheers,
Tobias

JunmingH · 2019-11-24T19:24:10Z

@tjakobi Yes same error with this
remove_empty_lines
TypeError: 'NoneType' object is not iterable
Command exited with non-zero status 1

tjakobi · 2019-11-25T12:07:48Z

Hi @JunmingH,

could you please make sure that your BAM input list file does not contain any empty lines?

Also: I would like to see your complete DCC call. Did you use @filename for specifying the input list?

Cheers,
Tobias

JunmingH · 2019-11-25T15:19:10Z

@tjakobi Sure

python2 ${app_dir}/main.py @samplesheet
-D -N -R ${gtf_dir}/GRCh38_Repeats_simpleRepeats_RepeatMasker.gtf
-an ref/GRCh38/annotation/Homo_sapiens.GRCh38.95.gtf
-F -M -Nr 1 1 -fg -G -A ref/Homo_sapiens.GRCh38.dna.primary_assembly.fa
-T 2 -O /dcc_all_results/
-B @bam_files

The format for the sample sheet and bam_files is like this:
/align/subject1.sort.coord.combined_Chimeric.out.junction
/align/subject2.sort.coord.combined_Chimeric.out.junction
/align/subject3.sort.coord.combined_Chimeric.out.junction

/align/subject1.sort.coord.combined_Aligned.sortedByCoord.out.bam
/align/subject2.sort.coord.combined_Aligned.sortedByCoord.out.bam
/align/subject3.sort.coord.combined_Aligned.sortedByCoord.out.bam

tjakobi · 2019-11-25T15:30:39Z

Hi @JunmingH,

looks good. Could you please attach the original bam_files and samplesheet files?

Cheers,
Tobias

JunmingH · 2019-11-25T15:38:47Z

Could you give me an email address? I can send it to you!

tjakobi · 2019-11-25T15:47:36Z

You can directly upload files here on GitHub via the area under the text field ("Attach files by dragging...")

tjakobi self-assigned this Sep 19, 2019

tjakobi added bug question labels Nov 24, 2019

tjakobi closed this as completed Nov 27, 2019

tjakobi reopened this Nov 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Combining individual circRNA read counts - error #68

Combining individual circRNA read counts - error #68

mihinduk commented Sep 16, 2019

mihinduk commented Oct 2, 2019

JunmingH commented Nov 18, 2019

mihinduk commented Nov 18, 2019

JunmingH commented Nov 19, 2019

tjakobi commented Nov 19, 2019 •

edited

Loading

mihinduk commented Nov 19, 2019 via email

tjakobi commented Nov 19, 2019

mihinduk commented Nov 19, 2019

JunmingH commented Nov 19, 2019

JunmingH commented Nov 19, 2019

tjakobi commented Nov 24, 2019 •

edited

Loading

tjakobi commented Nov 24, 2019

tjakobi commented Nov 24, 2019 •

edited

Loading

JunmingH commented Nov 24, 2019

tjakobi commented Nov 25, 2019 •

edited

Loading

JunmingH commented Nov 25, 2019

tjakobi commented Nov 25, 2019

JunmingH commented Nov 25, 2019

tjakobi commented Nov 25, 2019

Combining individual circRNA read counts - error #68

Combining individual circRNA read counts - error #68

Comments

mihinduk commented Sep 16, 2019

mihinduk commented Oct 2, 2019

JunmingH commented Nov 18, 2019

mihinduk commented Nov 18, 2019

JunmingH commented Nov 19, 2019

tjakobi commented Nov 19, 2019 • edited Loading

mihinduk commented Nov 19, 2019 via email

tjakobi commented Nov 19, 2019

mihinduk commented Nov 19, 2019

JunmingH commented Nov 19, 2019

JunmingH commented Nov 19, 2019

tjakobi commented Nov 24, 2019 • edited Loading

tjakobi commented Nov 24, 2019

tjakobi commented Nov 24, 2019 • edited Loading

JunmingH commented Nov 24, 2019

tjakobi commented Nov 25, 2019 • edited Loading

JunmingH commented Nov 25, 2019

tjakobi commented Nov 25, 2019

JunmingH commented Nov 25, 2019

tjakobi commented Nov 25, 2019

tjakobi commented Nov 19, 2019 •

edited

Loading

tjakobi commented Nov 24, 2019 •

edited

Loading

tjakobi commented Nov 24, 2019 •

edited

Loading

tjakobi commented Nov 25, 2019 •

edited

Loading