Skip to content

Continuous Integration and packaging with Jenkins

Jump to bottom
Nikhil Biyani edited this page Feb 21, 2017 · 7 revisions

Continuous integration of 2DX Software is done using Jenkins and can be found here: bs-ci02.ethz.ch:8082/

Continuous Integration Jobs

There are two main builds which are triggered for the master branch of 2DX:

  • 2dx_trunk_ci : Triggers as soon a commit is pushed to the master branch of 2DX. This job would just check if the code compiles on various machines

  • 2dx_nightly_build : Checks if a new commit was pushed each hour, compiles the new 2DX code on various machines. It also builds packages of the respective architecture which it was built on. These packages are uploaded to the 2DX website.

Master and Slaves

Master node controls the slaves and automatically triggers build jobs on them. The master node is currently maintained on (bs-ci02). The slaves are as follows:

  • cina-buildtst02: Ubuntu Machine (in one of the server rooms)
  • cina-zenon01: Ubuntu Machine (near Titan)
  • cina-buildtst03: Fedora Machine
  • bs-mm-compile09: OSX 10.10
  • bs-mm-compile10: OSX 10.11
  • bs-mm-compile11: OSX 10.12

Note that bs-hudson has the admin right in all these machines.

Creating a new slave node

First of all to create a slave node go to: http://bs-ci02.ethz.ch:8082/computer/ and use the link "New Node" on left menu. If possible use the option to copy settings from the nearest architecture possible and then change few options like description, name. Finally when done, launch the slave agent. If java is properly installed it should work. Otherwise try to add the following file in home folder: ~/.ssh/authorize_keys. The contents of this file should be copied from existing node. The should create a new slave node.

Sometimes jenkins complains: Authentication method password not supported by the server at this stage

What is to be done in this case is to edit the file /etc/ssh/sshd_config and change PasswordAuthentication yes

Installing new software

A bunch of new software would have to be installed for everything to work smoothly. The list goes as below:

Qt5 installation is identified using qmake. So qmake should be in path.

Setting up webDAV

Apart from the installation one needs to create a file named .netrc on home path which can be used by webDAV client cadaver to automatically login while uploading to the server. This file contains the following lines (make sure that the machine does not contain any https:// or x.uni.ch/files

machine www.2dx.unibas.ch
login night
passwd <>

Linux specific settings

The folder /opt should be writable by all users. So follow the following command:

sudo chmod a+w /opt

OSX specific settings

One would need to probably install packages like gcc/g++/gfortran using homebrew or macports. If done so, the following should be added to path:

/usr/local/bin:/opt/bin:/opt/local/bin:/usr/bin/:/usr/sbin

Make sure that the gcc/g++ from the clag is not used to build otherwise it will fail. When building a package change_dylibs.sh expects the following library paths. Make sure they exist, if not please link them correctly.

/usr/local/lib/libfftw3.3.dylib
/usr/local/lib/libfftw3f.3.dylib
/usr/local/lib/libfftw3_threads.3.dylib
/usr/local/lib/libfftw3f_threads.3.dylib
/usr/local/lib/libgfortran.3.dylib
/usr/local/lib/libquadmath.0.dylib
/usr/local/lib/libstdc++.6.dylib
/usr/local/lib/libgcc_s.1.dylib
/usr/local/lib/libgomp.1.dylib
/usr/lib/libSystem.B.dylib

OSX High DPI Settings

To make sure that on Retina screens of OSX 2DX does not look pixelated, some properties have to be added to the info.plist file of the package. We use CMAKE to build the packages. There is no way to introduce these properties using the CMakeLists.txt, as they are missing in the template. So we will have to change the template to introduce these properties. Following file is to be edited:

MacOSXBundleInfo.plist.in

which should be located in the CMAKE installation dir: /Applications/CMake.app/Contents/share/cmake-3.5/Modules

The file should need the following new properties:

<key>NSPrincipalClass</key>
<string>NSApplication</string>
<key>NSHighResolutionCapable</key>
<string>True</string>
<key>NSSupportsAutomaticGraphicsSwitching</key>
<true/>

and the file should look like:

?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>NSPrincipalClass</key>
    <string>NSApplication</string>
    <key>NSHighResolutionCapable</key>
    <string>True</string>
    <key>NSSupportsAutomaticGraphicsSwitching</key>
    <true/>
    <key>CFBundleDevelopmentRegion</key>
    <string>English</string>
    <key>CFBundleExecutable</key>
    <string>${MACOSX_BUNDLE_EXECUTABLE_NAME}</string>
    <key>CFBundleGetInfoString</key>
    <string>${MACOSX_BUNDLE_INFO_STRING}</string>
    <key>CFBundleIconFile</key>
    <string>${MACOSX_BUNDLE_ICON_FILE}</string>
    <key>CFBundleIdentifier</key>
    <string>${MACOSX_BUNDLE_GUI_IDENTIFIER}</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>CFBundleLongVersionString</key>
    <string>${MACOSX_BUNDLE_LONG_VERSION_STRING}</string>
    <key>CFBundleName</key>
    <string>${MACOSX_BUNDLE_BUNDLE_NAME}</string>
    <key>CFBundlePackageType</key>
    <string>APPL</string>
    <key>CFBundleShortVersionString</key>
    <string>${MACOSX_BUNDLE_SHORT_VERSION_STRING}</string>
    <key>CFBundleSignature</key>
    <string>????</string>
    <key>CFBundleVersion</key>
    <string>${MACOSX_BUNDLE_BUNDLE_VERSION}</string>
    <key>CSResourcesFileMapped</key>
    <true/>
    <key>LSRequiresCarbon</key>
    <true/>
    <key>NSHumanReadableCopyright</key>
    <string>${MACOSX_BUNDLE_COPYRIGHT}</string>
</dict>
</plist>

URLs and uploading to the server

There are two main files which needs to be uploaded to the server.

Build Job Script Examples

Some examples of the build scripts that can be specified in CI jobs

Ubuntu

if [ "$NODE_NAME" = "cina-zenon01" ]
then
    echo I am cina-zenon01 
    echo My os is:
    echo ubuntu 14.04
    export PATH=/home/bs-hudson/Qt/5.5/gcc_64/bin:$PATH
    rm -rf "/home/bs-hudson/hudson/jenkins_builds/trunk_nb_bin/$NODE_NAME"
    rm -rf "/home/bs-hudson/hudson/jenkins_builds/trunk_nb_inst/$NODE_NAME"
   ./build_all "/home/bs-hudson/hudson/jenkins_builds/trunk_nb_bin/$NODE_NAME" "/home/bs-hudson/hudson/jenkins_builds/trunk_nb_inst/$NODE_NAME"

    cd "/home/bs-hudson/hudson/jenkins_builds/trunk_nb_bin/$NODE_NAME"

    cpack -G DEB -C CPackConfig.cmake
    ./packaging/upload_to_server.sh `find ./ -maxdepth 1 -name "*.deb"` "http://www.2dx.unibas.ch/download/linux/deb/developmental"
    ./packaging/upload_to_server.sh ./changes.htm "http://www.2dx.unibas.ch/documentation/2dx-software/version-change-log/"
fi

OSX

if [ "$NODE_NAME" = "bs-vmsvr04" ]
then 
    echo My os is:
    echo osx 10.10
    source "/Users/bs-hudson/.profile"
    echo "My path is: $PATH"
    rm -rf "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_build"
    rm -rf "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_inst"
    ./build_all "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_build" "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_inst" ON

    cd "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_build/"

    sh packaging/change_dylibs.sh "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_inst"
    sh packaging/osx_package_builder.sh "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_inst/" "/Users/bs-hudson/jenkins/jenkins_builds/trunk_nb_build/packages" `cat current.version | tr -d "\n"`

   dmg_name=`find ./packages -maxdepth 1 -name "*.dmg"`
   dmg_basename=`basename ${dmg_name} .dmg`
   cp ./packages/${dmg_basename}.dmg ./packages/${dmg_basename}-OSX_10.10.dmg

    sh ./packaging/upload_to_server.sh ./packages/${dmg_basename}-OSX_10.10.dmg "http://www.2dx.unibas.ch/download/osx/osx-10.10/developmental"
fi
Clone this wiki locally