Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trace origin of crash when running multi-threaded with XROOTD #13081

Closed
Dr15Jones opened this issue Jan 26, 2016 · 12 comments
Closed

Trace origin of crash when running multi-threaded with XROOTD #13081

Dr15Jones opened this issue Jan 26, 2016 · 12 comments
Labels
core-pending pending-signatures

Comments

@Dr15Jones
Copy link
Contributor

The following traceback was found in the integration build after going to a multi-threaded mixing module which stresses the thread-safety fo XROOTD and out interface to it much harder:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc493/CMSSW_8_0_THREADED_X_2016-01-21-2300/pyRelValMatrixLogs/run/25205.0_ZTT_13+ZTT_13INPUT+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step3_ZTT_13+ZTT_13INPUT+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log

WIth traceback

Thread 10 (Thread 0x7f654be9d700 (LWP 24409)):
#0  0x000000387d60f32d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f6559c14b27 in edm::service::cmssw_stacktrace_fork() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginFWCoreServicesPlugins.so
#2  0x00007f6559c15462 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginFWCoreServicesPlugins.so
#3  0x00007f656336eac0 in execute_native_thread_routine () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/external/gcc/4.9.3/lib64/libstdc++.so.6
#4  0x000000387d607a51 in start_thread () from /lib64/libpthread.so.0
#5  0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f6518f7b700 (LWP 24481)):
#0  0x000000387d2e8f33 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f65190403e2 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdUtils.so.2
#2  0x00007f651903cc2d in XrdSys::IOEvents::BootStrap::Start(void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdUtils.so.2
#3  0x00007f651903b447 in XrdSysThread_Xeq () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdUtils.so.2
#4  0x000000387d607a51 in start_thread () from /lib64/libpthread.so.0
#5  0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f651857a700 (LWP 24482)):
#0  0x000000387d60efbd in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f651903bbdb in XrdSysTimer::Wait(int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdUtils.so.2
#2  0x00007f6519134f11 in XrdCl::TaskManager::RunTasks() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#3  0x00007f6519135079 in ?? () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#4  0x000000387d607a51 in start_thread () from /lib64/libpthread.so.0
#5  0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f6517b79700 (LWP 24483)):
#0  0x000000387d2e5199 in syscall () from /lib64/libc.so.6
#1  0x00007f651918dc63 in XrdCl::JobManager::RunJobs() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#2  0x00007f651918ddf9 in ?? () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#3  0x000000387d607a51 in start_thread () from /lib64/libpthread.so.0
#4  0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f6517178700 (LWP 24484)):
#0  0x000000387d2e5199 in syscall () from /lib64/libc.so.6
#1  0x00007f651918dc63 in XrdCl::JobManager::RunJobs() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#2  0x00007f651918ddf9 in ?? () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#3  0x000000387d607a51 in start_thread () from /lib64/libpthread.so.0
#4  0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f6516777700 (LWP 24485)):
#0  0x000000387d2df113 in poll () from /lib64/libc.so.6
#1  0x00007f6559c152c4 in full_read.constprop () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginFWCoreServicesPlugins.so
#2  0x00007f6559c1552c in edm::service::InitRootHandlers::stacktraceFromThread() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginFWCoreServicesPlugins.so
#3  0x00007f6559c156ab in sig_dostack_then_abort () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00000000004106b1 in std::_Rb_tree<std::string, std::pair<std::string const, std::string>, std::_Select1st<std::pair<std::string const, std::string> >, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, std::string> >*) ()
#6  0x00007f65190ac1bd in XrdCl::ResponseHandler::HandleResponseWithHosts(XrdCl::XRootDStatus*, XrdCl::AnyObject*, std::vector<XrdCl::HostInfo, std::allocator<XrdCl::HostInfo> >*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/libUtilitiesXrdAdaptor.so
#7  0x00007f65191672ae in ?? () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#8  0x00007f65191480cf in XrdCl::XRootDMsgHandler::HandleResponse() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#9  0x00007f6519149ffe in XrdCl::XRootDMsgHandler::Process(XrdCl::Message*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#10 0x00007f651912d20a in XrdCl::Stream::HandleIncMsgJob::Run(void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#11 0x00007f651918dc16 in XrdCl::JobManager::RunJobs() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#12 0x00007f651918ddf9 in ?? () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libXrdCl.so.2
#13 0x000000387d607a51 in start_thread () from /lib64/libpthread.so.0
#14 0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f6465aff700 (LWP 24550)):
#0  0x00007f65640bd1b6 in TBufferFile::ReadFastArray(int*, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#1  0x00007f6564098321 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#2  0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#3  0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#4  0x00007f65640bd6bf in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#5  0x00007f656447b5a4 in TBranch::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#6  0x00007f6563d4b3b8 in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#7  0x00007f656409aaf2 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#8  0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#9  0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#10 0x00007f65640bbe45 in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#11 0x00007f6564465a54 in TBranchElement::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#12 0x00007f65640c36ac in TBufferFile::ReadObjectAny(TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#13 0x00007f6563cf1660 in TObjArray::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#14 0x00007f6564099b92 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#15 0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#16 0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#17 0x00007f65640bd6bf in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#18 0x00007f656447b5a4 in TBranch::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#19 0x00007f6563d4b3b8 in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#20 0x00007f656409aaf2 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#21 0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#22 0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#23 0x00007f65640bbe45 in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#24 0x00007f6564465a54 in TBranchElement::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#25 0x00007f65640c36ac in TBufferFile::ReadObjectAny(TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#26 0x00007f6563cf1660 in TObjArray::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#27 0x00007f6564099b92 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#28 0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#29 0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#30 0x00007f65640bd6bf in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#31 0x00007f6564496ded in TTree::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#32 0x00007f6563fea155 in TKey::ReadObj() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#33 0x00007f6563fef0a1 in TDirectoryFile::Get(char const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#34 0x00007f65466eec89 in edm::RootTree::RootTree(std::shared_ptr<edm::InputFile>, edm::BranchType const&, unsigned int, unsigned int, unsigned int, unsigned int, bool, edm::InputType) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#35 0x00007f65466d9ed4 in edm::RootFile::RootFile(std::string const&, edm::ProcessConfiguration const&, std::string const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#36 0x00007f65466c990a in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#37 0x00007f65466e5bed in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#38 0x00007f65466e7369 in edm::RootInputFileSequence::skipToItemInNewFile(unsigned int, unsigned int, unsigned long long, unsigned long) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#39 0x00007f65466e76b6 in edm::RootInputFileSequence::skipToItem(unsigned int, unsigned int, unsigned long long, unsigned long, bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#40 0x00007f65466c93b1 in edm::RootEmbeddedFileSequence::readOneSpecified(edm::EventPrincipal&, unsigned long&, edm::SecondaryEventIDAndFileInfo const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#41 0x00007f651943a764 in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginSimGeneralMixingModulePlugins.so
#42 0x00007f651937141a in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/libMixingBase.so
[remove unimportant stuff]
#61 0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f64650fe700 (LWP 24551)):
#0  0x00007f6563d34610 in TClass::Size() const () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#1  0x00007f6563d47c3a in TStreamerSTL::GetSize() const () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#2  0x00007f6563f5ed5e in TStreamerInfoActions::TActionSequence::CreateReadMemberWiseActions(TVirtualStreamerInfo*, TVirtualCollectionProxy&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#3  0x00007f65640b5a29 in TGenCollectionProxy::GetReadMemberWiseActions(int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#4  0x00007f6563fbf270 in int TStreamerInfoActions::ReadSTL<&TStreamerInfoActions::ReadSTLMemberWiseSameClass, &TStreamerInfoActions::ReadSTLObjectWiseFastArray>(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#5  0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#6  0x00007f6564456e99 in TBranchElement::ReadLeavesMember(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#7  0x00007f6564477af2 in TBranch::GetEntry(long long, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#8  0x00007f6564462871 in TBranchElement::GetEntry(long long, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#9  0x00007f6564490123 in TTree::GetEntry(long long, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#10 0x00007f65466ec481 in edm::roottree::getEntry(TTree*, long long) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#11 0x00007f65466daba7 in edm::RootFile::RootFile(std::string const&, edm::ProcessConfiguration const&, std::string const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#12 0x00007f65466c990a in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#13 0x00007f65466e5bed in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#14 0x00007f65466e7369 in edm::RootInputFileSequence::skipToItemInNewFile(unsigned int, unsigned int, unsigned long long, unsigned long) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#15 0x00007f65466e76b6 in edm::RootInputFileSequence::skipToItem(unsigned int, unsigned int, unsigned long long, unsigned long, bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#16 0x00007f65466c93b1 in edm::RootEmbeddedFileSequence::readOneSpecified(edm::EventPrincipal&, unsigned long&, edm::SecondaryEventIDAndFileInfo const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#17 0x00007f651943a764 in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginSimGeneralMixingModulePlugins.so
#18 0x00007f651937141a in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/libMixingBase.so
[unimportant stuff cut]
#36 0x000000387d607a51 in start_thread () from /lib64/libpthread.so.0
#37 0x000000387d2e893d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f64646fd700 (LWP 24552)):
#0  0x000000387d60b63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f6563369758 in __gthread_cond_wait(pthread_cond_t*, pthread_mutex_t*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/external/gcc/4.9.3/lib64/libstdc++.so.6
#2  0x00007f65633697f6 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/external/gcc/4.9.3/lib64/libstdc++.so.6
#3  0x00007f65190a5f1b in XrdFile::read(void*, unsigned long) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/libUtilitiesXrdAdaptor.so
#4  0x00007f6559cacf29 in StorageAccountProxy::read(void*, unsigned long) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/libUtilitiesStorageFactory.so
#5  0x00007f6559c9193a in IOInput::xread(void*, unsigned long) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/libUtilitiesStorageFactory.so
#6  0x00007f65592efd1e in TStorageFactoryFile::ReadBuffer(char*, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/libIOPoolTFileAdaptor.so
#7  0x00007f65640a5e29 in TFile::Init(bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#8  0x00007f65592f1939 in TStorageFactoryFile::Initialize(char const*, char const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/libIOPoolTFileAdaptor.so
#9  0x00007f65592f20c4 in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int, int, bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/libIOPoolTFileAdaptor.so
#10 0x00007f651a7450a9 in ?? ()
#11 0x00007f6500000000 in ?? ()
#12 0x00007f65194fc9a0 in ?? ()
#13 0x00007f656037d6c0 in ?? ()
#14 0x00007f64646f8f70 in ?? ()
#15 0x00007f64646f8c28 in ?? ()
#16 0x00000005545e5cdd in ?? ()
#17 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f6563091ba0 (LWP 24337)):
#0  0x00007f65640bebee in TBufferFile::ReadFastArray(long long*, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#1  0x00007f6564098a81 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#2  0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#3  0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#4  0x00007f65640bd6bf in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#5  0x00007f656447b5a4 in TBranch::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#6  0x00007f6563d4b3b8 in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#7  0x00007f656409aaf2 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#8  0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#9  0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#10 0x00007f65640bbe45 in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#11 0x00007f6564465a54 in TBranchElement::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#12 0x00007f65640c36ac in TBufferFile::ReadObjectAny(TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#13 0x00007f6563cf1660 in TObjArray::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#14 0x00007f6564099b92 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#15 0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#16 0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#17 0x00007f65640bd6bf in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#18 0x00007f656447b5a4 in TBranch::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#19 0x00007f6563d4b3b8 in TStreamerBase::ReadBuffer(TBuffer&, char*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#20 0x00007f656409aaf2 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#21 0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#22 0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#23 0x00007f65640bbe45 in TBufferFile::ReadClassBuffer(TClass const*, void*, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#24 0x00007f6564465a54 in TBranchElement::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#25 0x00007f65640c36ac in TBufferFile::ReadObjectAny(TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#26 0x00007f6563cf1660 in TObjArray::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libCore.so
#27 0x00007f6564099b92 in int TStreamerInfo::ReadBuffer<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#28 0x00007f6563f5101d in TStreamerInfoActions::GenericReadAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#29 0x00007f65640bb725 in TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#30 0x00007f65640bd6bf in TBufferFile::ReadClassBuffer(TClass const*, void*, int, unsigned int, unsigned int, TClass const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#31 0x00007f6564496ded in TTree::Streamer(TBuffer&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libTree.so
#32 0x00007f6563fea155 in TKey::ReadObj() () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#33 0x00007f6563fef0a1 in TDirectoryFile::Get(char const*) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/external/slc6_amd64_gcc493/lib/libRIO.so
#34 0x00007f65466eec89 in edm::RootTree::RootTree(std::shared_ptr<edm::InputFile>, edm::BranchType const&, unsigned int, unsigned int, unsigned int, unsigned int, bool, edm::InputType) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#35 0x00007f65466d9ed4 in edm::RootFile::RootFile(std::string const&, edm::ProcessConfiguration const&, std::string const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#36 0x00007f65466c990a in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#37 0x00007f65466e5bed in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#38 0x00007f65466e7369 in edm::RootInputFileSequence::skipToItemInNewFile(unsigned int, unsigned int, unsigned long long, unsigned long) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#39 0x00007f65466e76b6 in edm::RootInputFileSequence::skipToItem(unsigned int, unsigned int, unsigned long long, unsigned long, bool) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#40 0x00007f65466c93b1 in edm::RootEmbeddedFileSequence::readOneSpecified(edm::EventPrincipal&, unsigned long&, edm::SecondaryEventIDAndFileInfo const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_0_THREADED_X_2016-01-20-1100/lib/slc6_amd64_gcc493/pluginIOPoolInput.so
#41 0x00007f651943a764 in edm::MixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/pluginSimGeneralMixingModulePlugins.so
#42 0x00007f651937141a in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /afs/cern.ch/cms/sw/ReleaseCandidates/vol0/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_THREADED_X_2016-01-21-2300/lib/slc6_amd64_gcc493/libMixingBase.so
[stuff cut]

A fatal system signal has occurred: segmentation violation
@cmsbuild
Copy link
Contributor

A new Issue was created by @Dr15Jones (Chris Jones).

@davidlange6, @smuzaffar, @Degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are list here #13029

@Dr15Jones
Copy link
Contributor Author

assign core

@cmsbuild
Copy link
Contributor

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

@Dr15Jones
Copy link
Contributor Author

@bbockelm looks like a new crash in the CMS xrootd interface.

@bbockelm
Copy link
Contributor

What's the best way to re-run the above IB by hand?

Just guessing from the stdout and lack of other details, I would guess there's an issue with the delayed close callback.

@davidlt
Copy link
Contributor

davidlt commented Jan 26, 2016

RelVals? runTheMatrix.py -i all -j <core_count> This would replay all ~350 workflows. You can also add -l <num>,<num>,.. just to run specific workflows.

@Dr15Jones
Copy link
Contributor Author

I doubt the problem is easily repeatable. To rerun the job do

runTheMatrix.py -l 25205.0 -t 4

@davidlt
Copy link
Contributor

davidlt commented Jan 26, 2016

There are 72-virtual-core x86_64 machines (sadly only with 64GB of RAM) and we could hammer some workflows very hard (high number of threads and multiple cmsRun processes) just reveal threading issues faster.

@bbockelm
Copy link
Contributor

Bah. No, looking closer at the traceback, it's probably XrdAdaptor::ClientRequest::HandleResponse.

Not entirely sure though; tough to tell with inlining. Any way to recover symbols?

@bbockelm
Copy link
Contributor

Discussing with Chris, we now think the problem is in QualityMetricFactory::get(). We think this is corrupting memory (particularly, the stack) as the stacktrace above shouldn't be possible.

This singleton isn't thread-safe and is used when there are concurrent opens (i.e., something we don't do often except for the new threaded mixing module). So, it should be fixed regardless of whether it caused the problem here.

@bbockelm bbockelm mentioned this issue Jan 27, 2016
Merged
@bbockelm
Copy link
Contributor

I believe #13085 may fix this issue.

@bbockelm
Copy link
Contributor

Fix was merged, we think. Can this ticket be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core-pending pending-signatures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@davidlt @bbockelm @Dr15Jones @cmsbuild @davidlange6