Build failure with ROOT 6.22 new python structure

[lastCent]

DD4hep fails to build in our nightly builds, probably because of the way in which multiple Python versions now are organized in the lib/ directory, and the splitting of PyROOT from TPython. ROOT 6.20 changelog: https://root.cern/doc/v620/release-notes.html

• OS version: e.g. SLC6

  • SLC6, Centos7

• Compiler version: e.g GCC 6.2

  • gcc7, gcc8, gcc9, clang8

• DD4hep version: tag or commit ID, or GitHub branch

  • Master, commit: 84cf31e

• Reproduced by: exact steps to reproduce the problem: checkout, setup environment (Geant, ROOT versions, iLCSoft release), cmake, build, run...

  • The build was run with LCGCmake as part of the LCG stacks.
  • Geant4 version: 10.6
  • ROOT version: Master (1796471234a85b7d6272e371d586b58a7e1287e7)

• Input: link to input files if applicable

• Output: full build and run log output

  • Current build logs are available in the dev3 and dev3python3 section here.
  • logs.tar.gz

• Goal: A short description of what you are trying to achieve

  • Build DD4hep as part of our nightly builds.

[petricm]

Interesting, we are testing in our CI against dev3 and dev4 and we did not observe this. How can we test this particular ROOT configuration of yours?

[petricm]

But just for clarification dev3 has ROOT built with both python2 and 3 or is there dev3(py2) and dev3python3(py3)?

[andresailer]

@lastCent I couldn't find the cmake command and output in the logs or the cdash website

[petricm]

So this is related to this

Subject:	Experimental PyROOT: switch to default TODAY
Mon, 17 Feb 2020 10:14:44 +0100
Enric Tejedor etejedor@cern.ch
root-planning@cern.ch, rootdev root-dev@cern.ch

Dear All,

As discussed in the ROOT planning meeting of last week (https://indico.cern.ch/event/884189/), we are going to do the switch of experimental PyROOT to default in master today. Therefore, nightly builds with ROOT master will start running with experimental PyROOT tonight.
Please communicate to us any issues you may observe and we will work on them right away.

Best,

Enric

and the relevant commit root-project/root@71c42ac

[petricm]

@lastCent but this affects only root master and not the 6-20 branch, so this change is for ROOT 6.22?

[lastCent]

Hello,

How can we test this particular ROOT configuration of yours?

You can source the complete view of a particular nightly from any CERN CVMFS-enabled machine. To do so run something like:
source /cvmfs/sft.cern.ch/lcg/views/dev3/<Weekday>/<platform>/setup.sh
Example: source /cvmfs/sft.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc8-opt/setup.sh

But just for clarification dev3 has ROOT built with both python2 and 3 or is there dev3(py2) and dev3python3(py3)?

We aim to build ROOT with python2 in the dev3 and dev4 builds, and python3 in dev3python3. We have used to specify this with the -Dpython3 flag, which is set to ON for dev3python3 (not for dev3/dev4). But I understand that ROOT changes the python versioning system with the 6.20 release, such that more than one version can be specified. Could this be the issue perhaps?

I couldn't find the cmake command and output in the logs or the cdash website

Apologies for the bad UI. The logs can be accessed by clicking the yellow package icons next to the name of a failing build (which is a build where the error column contains errors). So for example, in the section dev3 press one of the package icons, then DD4hep-master-install.log.

I'm unsure how widespread access rights are. If some of you don't have those, let me know and I'll fetch some of those logs for you.

The build uses our LCGCMake build system, which is built on top of CMake. This makes the exact command a it difficult to read, but you can see the flags+environment used here https://gitlab.cern.ch/sft/lcgcmake/blob/master/projects/CMakeLists.txt#L73 . Let me know which parts are unclear, or if nothing about it is clear at all.

@lastCent but this affects only root master and not the 6-20 branch, so this change is for ROOT 6.22?

We build every night, so the exact version of ROOT may change. For yesterday's (Wednesday) build root --version output:

ROOT Version: 6.21/01
Built for lunuxx8664gcc on Feb 19 2020, 03:15:00
From heads/master@v6-19-01-2986-gfae2fe2

[petricm]

The error I get on wednesday dev3 for configure

[2020-02-20 12:58 mpetric@pclcdgpu:build (master)$ source /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc9-opt/setup.sh 
[2020-02-20 12:59 mpetric@pclcdgpu:build (master)$ cmake -DDD4HEP_USE_GEANT4=ON -DBoost_NO_BOOST_CMAKE=ON -DDD4HEP_USE_LCIO=ON -DBUILD_TESTING=ON -DGeant4_DIR=$G4INSTALL/lib/Geant4-10.6.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 ..
-- CMAKE_INSTALL_PREFIX is /home/mpetric/DD4hep - overwrite with -D CMAKE_INSTALL_PREFIX
-- Found Doxygen: /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc9-opt/bin/doxygen (found version "1.8.15") found components:  doxygen dot 
-- Could NOT find LATEX (missing: LATEX_COMPILER) 
-- No LaTeX/Biber found, cannot compile user manual.
-- The CXX compiler identification is GNU 9.2.0
-- Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/g++
-- Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Will be building these packages: DDRec;DDDetectors;DDCond;DDAlign;DDDigi;DDG4;DDEve;UtilityApps
-- Including DD4hepBuild.cmake
-- The C compiler identification is GNU 9.2.0
-- Check for working C compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gcc
-- Check for working C compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/9.2.0-afc57/x86_64-centos7/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- DD4hep_configure_output: set CMAKE_INSTALL_PREFIX to /home/mpetric/DD4hep
-- Found Python: /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc9-opt/lib/libpython2.7.so (found version "2.7.16") found components:  Development 
CMake Error at /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc9-opt/cmake/ROOTConfig-targets.cmake:1276 (message):
  The imported target "ROOT::cppyy_backend" references the file

     "/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc9-opt/lib/python2.7/site-packages/libcppyy_backend.so"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc9-opt/cmake/ROOTConfig-targets.cmake"

  but not all the files it references.

Call Stack (most recent call first):
  /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc9-opt/cmake/ROOTConfig.cmake:92 (include)
  CMakeLists.txt:109 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/mpetric/DD4hep/build/CMakeFiles/CMakeOutput.log".
2020-02-20 12:59 mpetric@pclcdgpu:build (master)$ 

This does not seem like a DD4hep error?

[lastCent]

Oh sorry, I should have given you the CMake command we usually use for local testing:
cmake -DCMAKE_INSTALL_PREFIX=../install -DLCG_VERSION=dev3 ..
I suspect LCG_VERSION=dev3 makes the difference, the above command works for me if I add that flag.

Additionally you can take prebuilt packages from the stack by setting -DLCG_INSTALL_PREFIX=/cvmfs/sft.cern.ch/lcg/releases and setting packages you want rebuild locally with -DLCG_IGNORE. This will save you from rebuilding the whole stack, unless you want to.

Edit: The views may not be a proper build environment for views, especially if they don't finish. Please use this script to set up a local environment for building instead:
build_setup.tar.gz

[petricm]

Assuming that the build of ROOT in Wed dev3 is I do this

source /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc8-opt/setup.sh 

and then I create a nearly empty CMakeList.txt with only 3 lines

cmake_minimum_required(VERSION 3.12 FATAL_ERROR)
PROJECT( DD4hep LANGUAGES CXX)

find_package (ROOT)

and when invoking cmake I get this

2020-02-20 14:55 mpetric@pclcdgpu:build (master *)$ cmake ..
-- The CXX compiler identification is GNU 8.3.0
-- Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/g++
-- Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc8-opt/cmake/ROOTConfig-targets.cmake:1276 (message):
  The imported target "ROOT::cppyy_backend" references the file

     "/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/libcppyy_backend.so"

  but this file does not exist.  Possible reasons include:

  * The file was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and contained

     "/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc8-opt/cmake/ROOTConfig-targets.cmake"

  but not all the files it references.

Call Stack (most recent call first):
  /cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc8-opt/cmake/ROOTConfig.cmake:92 (include)
  CMakeLists.txt:4 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/mpetric/DD4hep/build/CMakeFiles/CMakeOutput.log".

[iarspider]

Hi all,

sorry for the confusion. The issue with cppyy in views is something we need to fix, but the overall problem is still there.

You can, for now, use the following script to set up environment: https://stikked.web.cern.ch/stikked/view/5a6cc1fa (save it as setup.sh and source it).

I have done some debugging on my own, and would like to share some results. First, here is the error message we observe (full log ):

-- Installing: /build/jenkins/workspace/install/DD4hep/master/x86_64-centos7-gcc8-opt/lib/libDDG4Plugins.so
CMake Error at DDG4/cmake_install.cmake:133 (file):
  file INSTALL cannot find
  "/mnt/build/jenkins/workspace/build/projects/DD4hep-master/src/DD4hep-master-build/DDG4/../lib/G__DDG4Python_rdict.pcm".
Call Stack (most recent call first):
  cmake_install.cmake:105 (include)


make[3]: *** [install] Error 1

Then, I compared CMake configuration logs (/cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Mon/DD4hep/master/x86_64-centos7-gcc8-opt/logs/DD4hep-master-configure.log ) from last successful build with the failing build (copy ):

Was

-- |> Building DDCore
-- |++> Will be Building DDCore
-- |++++> Building dictionary ... G__DD4hep
-- |++++> Building dictionary ... G__DD4hepGeo
-- |++++> Building dictionary ... G__DD4hepProperties
-- |++++> Building dictionary ... G__DD4hepSegmentations
-- |++> Boost available, building Json Parsers
-- |++> Building Plugin: DDCorePlugins
-- |++> Found Root::GDML: Creating DDGDMLPlugins
-- |++> Building Plugin: DDGDMLPlugins
-- |++> Building Plugin: DDPythonPlugins
-- |> Building DDRec

Now:

-- |> Building DDCore
-- |++> Will be Building DDCore
-- |++++> Building dictionary ... G__DD4hep
-- |++++> Building dictionary ... G__DD4hepGeo
-- |++++> Building dictionary ... G__DD4hepProperties
-- |++++> Building dictionary ... G__DD4hepSegmentations
-- |++> Boost available, building Json Parsers
-- |++> Building Plugin: DDCorePlugins
-- |++> Found Root::GDML: Creating DDGDMLPlugins
-- |++> Building Plugin: DDGDMLPlugins
-- ROOT does not include PyRoot, not building DDCore DDPython Plugins
-- |> Building DDRec

DD4Hep detects PyRoot by looking for libPyROOT.so, which is no longer created by ROOT. We don't know (waiting for feedback from ROOT people) what is the replacement for this library (probably lib/python2.7/site-packages/libROOTPythonizations.so).

Another question is why does DD4Hep attempt to install dictionaries that are not built?

Thanks,
Ivan

[andresailer]

For clarification, we detect PyROOT by looking for the target ROOT::PyROOT, so if that target changes names, we need to adapt, but it doesn't matter which libraries sit behind the target, as long as they exist.

There is a bug in the DDG4 that would ask for dictionary creation, but not actually call the command unless a library is build that uses the dictionary.cxx file, thus also the pcm file was not created. (Should be fixed with #631 )

The relevant part of the log is actually this part

-- |++> Python found, creating DDG4Python Dictionary
-- |++++> Building dictionary ... G__DDG4Python
-- |++++> Building dictionary ... G__DDPython
-- |++> Building Plugin: DDG4LCIO

There is no message if pyroot is not found.

[andresailer]

The target for the new Root Python is called ROOT::ROOTTPython, attempt in #632 .

I think we should merge #631 and then see if we can already compile DD4hep against the new root python thing

[andresailer]

@iarspider @lastCent can you confirm that the nightly builds worked over the weekend?

[iarspider]

@andresailer yes, the nightly builds (of DD4Hep master) worked.

[andresailer]

@iarspider
is there some issue regarding this problem with ROOT / cppyy that we can follow ?

  * The installation package was faulty and contained
     "/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Wed/x86_64-centos7-gcc8-opt/cmake/ROOTConfig-targets.cmake"

Or will you let us know once that is fixed?

[iarspider]

@andresailer the views were not recreated due to build failures. You can use the other solution (dedicated setup script) for now.

[MarkusFrankATcernch]

I have added a JIRA issue, but relaxed the importance, since it is the ROOT HEAD
https://sft.its.cern.ch/jira/browse/ROOT-10669