Minimal working layer to re-use existing nlopt

[eddelbuettel]

Per the earlier discussion here is (minimal) layer reusing an existing nlopt installation (and not requiring) cmake.

On Linux (Ubuntu 21.04) with nlopt present (and sufficiently recent)

edd@rob:~/git/nloptr(feature/build_options)$ install.r     ## install.r is a shell wrapper around install.packages()
* installing *source* package found in current working directory ...
* installing *source* package ‘nloptr’ ...
** using staged installation
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether ccache g++-11 accepts -g... yes
checking how to run the C++ preprocessor... ccache g++-11 -E
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether ccache g++-11 accepts -g... (cached) yes
checking for pkg-config... /usr/bin/pkg-config
checking if pkg-config knows NLopt... yes
checking for pkg-config checking NLopt version... >= 2.7.0
configure: creating ./config.status
config.status: creating src/Makevars
** libs
ccache gcc-11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include  -I'/usr/local/lib/R/site-library/testthat/include'    -fpic  -g -O3 -Wall -pipe   -std=gnu99 -c init_nloptr.c -o init_nloptr.o
ccache gcc-11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include  -I'/usr/local/lib/R/site-library/testthat/include'    -fpic  -g -O3 -Wall -pipe   -std=gnu99 -c nloptr.c -o nloptr.o
ccache g++-11  -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include  -I'/usr/local/lib/R/site-library/testthat/include'    -fpic  -g -O3 -Wall -pipe   -c test-C-API.cpp -o test-C-API.o
ccache g++-11  -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include  -I'/usr/local/lib/R/site-library/testthat/include'    -fpic  -g -O3 -Wall -pipe   -c test-runner.cpp -o test-runner.o
ccache g++-11 -std=gnu++11 -Wl,-S -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -Wl,-z,relro -o nloptr.so init_nloptr.o nloptr.o test-C-API.o test-runner.o -llapack -lblas -lgfortran -lm -lquadmath -lnlopt -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-nloptr/00new/nloptr/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (nloptr)
edd@rob:~/git/nloptr(feature/build_options)$

On Linux (Docker container with Debian testing, cmake and testthat added, no nlopt present

root@7f35403ba549:/work# install.r

• installing source package found in current working directory ...
• installing source package ‘nloptr’ ...
  ** using staged installation
  checking whether the C++ compiler works... yes
  checking for C++ compiler default output file name... a.out checking for suffix of executables...
  checking whether we are cross compiling... no checking for suffix of object files... o
  checking whether we are using the GNU C++ compiler... yes checking whether g++ -std=gnu++14 accepts -g... yes
  checking how to run the C++ preprocessor... g++ -std=gnu++14 -E checking whether we are using the GNU C++ compiler... (cached) yes
  checking whether g++ -std=gnu++14 accepts -g... (cached) yes
  checking for pkg-config... /usr/bin/pkg-config
  checking if pkg-config knows NLopt... no
  using NLopt via local build...
  set CMAKE_BIN=/usr/bin/cmake
  set CC=gcc -std=gnu99 -std=gnu11
  set CFLAGS= -fpic -g -O2 -ffile-prefix-map=/build/r-base-PT7Nxy/r-base-4.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
  set CXX=g++
  set CXXFLAGS=-std=gnu++11 -fpic -g -O2 -ffile-prefix-map=/build/r-base-PT7Nxy/r-base-4.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g
  set LDFLAGS=-Wl,-z,relro
  CMake Deprecation Warning at CMakeLists.txt:15 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.
  Update the VERSION argument value or use a ... suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- NLopt version 2.7.0
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for getopt.h
-- Looking for getopt.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for strings.h
-- Looking for strings.h - found -- Looking for inttypes.h
-- Looking for inttypes.h - found -- Looking for memory.h
-- Looking for memory.h - found -- Looking for stdlib.h
-- Looking for stdlib.h - found -- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for time.h
-- Looking for time.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for getpid
-- Looking for getpid - found
-- Looking for syscall
-- Looking for syscall - found
-- Looking for isinf
-- Looking for isinf - found
-- Looking for isnan
-- Looking for isnan - found
-- Looking for gettimeofday
-- Looking for gettimeofday - found
-- Looking for qsort_r
-- Looking for qsort_r - found
-- Looking for time
-- Looking for time - found
-- Looking for copysign
-- Looking for copysign - found
-- Looking for getopt
-- Looking for getopt - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of uint32_t
-- Check size of uint32_t - done
-- Check size of unsigned int
-- Check size of unsigned int - done
-- Check size of unsigned long
-- Check size of unsigned long - done
-- Looking for sqrt in m
-- Looking for sqrt in m - found
-- Looking for fpclassify
-- Looking for fpclassify - TRUE
-- Performing Test HAVE_THREAD_LOCAL_STORAGE
-- Performing Test HAVE_THREAD_LOCAL_STORAGE - Success -- Performing Test HAVE_THREAD_LOCAL_STORAGE
-- Performing Test HAVE_THREAD_LOCAL_STORAGE - Failed -- Looking for __cplusplus
-- Looking for __cplusplus - found -- Performing Test SUPPORTS_STDCXX11
-- Performing Test SUPPORTS_STDCXX11 - Success -- Performing Test HAS_FPIC
-- Performing Test HAS_FPIC - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /work/src/nlopt-build
[ 3%] Generating nlopt.f
[ 3%] Generating nlopt.hpp
CMake Deprecation Warning at /work/src/nlopt-src/cmake/generate-fortran.cmake:1 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a ... suffix to tell
CMake that the project does not need compatibility with older versions.

[ 3%] Built target generate-fortran
[ 3%] Built target generate-cpp
[ 5%] Building C object CMakeFiles/nlopt.dir/src/algs/cdirect/cdirect.c.o
[ 9%] Building C object CMakeFiles/nlopt.dir/src/algs/direct/direct_wrap.c.o
[ 9%] Building C object CMakeFiles/nlopt.dir/src/algs/cdirect/hybrid.c.o
[ 11%] Building C object CMakeFiles/nlopt.dir/src/algs/direct/DIRserial.c.o
[ 13%] Building C object CMakeFiles/nlopt.dir/src/algs/direct/DIRsubrout.c.o
[ 15%] Building C object CMakeFiles/nlopt.dir/src/algs/praxis/praxis.c.o
[ 19%] Building C object CMakeFiles/nlopt.dir/src/algs/direct/DIRect.c.o
[ 21%] Building C object CMakeFiles/nlopt.dir/src/algs/luksan/plip.c.o
[ 19%] Building C object CMakeFiles/nlopt.dir/src/algs/luksan/plis.c.o
[ 23%] Building C object CMakeFiles/nlopt.dir/src/algs/luksan/pnet.c.o
[ 25%] Building C object CMakeFiles/nlopt.dir/src/algs/luksan/mssubs.c.o
[ 27%] Building C object CMakeFiles/nlopt.dir/src/algs/luksan/pssubs.c.o
[ 29%] Building C object CMakeFiles/nlopt.dir/src/algs/crs/crs.c.o
[ 31%] Building C object CMakeFiles/nlopt.dir/src/algs/mlsl/mlsl.c.o
[ 33%] Building C object CMakeFiles/nlopt.dir/src/algs/mma/mma.c.o
[ 35%] Building C object CMakeFiles/nlopt.dir/src/algs/mma/ccsa_quadratic.c.o
[ 37%] Building C object CMakeFiles/nlopt.dir/src/algs/cobyla/cobyla.c.o [ 39%] Building C object CMakeFiles/nlopt.dir/src/algs/newuoa/newuoa.c.o
[ 41%] Building C object CMakeFiles/nlopt.dir/src/algs/neldermead/sbplx.c.o [ 43%] Building C object CMakeFiles/nlopt.dir/src/algs/neldermead/nldrmd.c.o
[ 45%] Building C object CMakeFiles/nlopt.dir/src/algs/auglag/auglag.c.o [ 47%] Building C object CMakeFiles/nlopt.dir/src/algs/bobyqa/bobyqa.c.o
[ 49%] Building C object CMakeFiles/nlopt.dir/src/algs/isres/isres.c.o [ 50%] Building C object CMakeFiles/nlopt.dir/src/algs/slsqp/slsqp.c.o
[ 52%] Building C object CMakeFiles/nlopt.dir/src/algs/esch/esch.c.o
[ 54%] Building C object CMakeFiles/nlopt.dir/src/api/general.c.o
[ 56%] Building C object CMakeFiles/nlopt.dir/src/api/options.c.o
[ 58%] Building C object CMakeFiles/nlopt.dir/src/api/optimize.c.o
[ 60%] Building C object CMakeFiles/nlopt.dir/src/api/deprecated.c.o
[ 62%] Building C object CMakeFiles/nlopt.dir/src/api/f77api.c.o
[ 64%] Building C object CMakeFiles/nlopt.dir/src/util/mt19937ar.c.o
[ 66%] Building C object CMakeFiles/nlopt.dir/src/util/sobolseq.c.o
[ 68%] Building C object CMakeFiles/nlopt.dir/src/util/timer.c.o
[ 70%] Building C object CMakeFiles/nlopt.dir/src/util/stop.c.o
[ 72%] Building C object CMakeFiles/nlopt.dir/src/util/redblack.c.o
[ 74%] Building C object CMakeFiles/nlopt.dir/src/util/qsort_r.c.o
[ 76%] Building C object CMakeFiles/nlopt.dir/src/util/rescale.c.o
[ 78%] Building CXX object CMakeFiles/nlopt.dir/src/algs/stogo/global.cc.o
[ 80%] Building CXX object CMakeFiles/nlopt.dir/src/algs/stogo/linalg.cc.o
[ 82%] Building CXX object CMakeFiles/nlopt.dir/src/algs/stogo/stogo.cc.o
[ 84%] Building CXX object CMakeFiles/nlopt.dir/src/algs/stogo/local.cc.o
[ 86%] Building CXX object CMakeFiles/nlopt.dir/src/algs/stogo/tools.cc.o
[ 88%] Building CXX object CMakeFiles/nlopt.dir/src/algs/ags/evolvent.cc.o
[ 90%] Building CXX object CMakeFiles/nlopt.dir/src/algs/ags/solver.cc.o
[ 92%] Building CXX object CMakeFiles/nlopt.dir/src/algs/ags/local_optimizer.cc.o
[ 94%] Building CXX object CMakeFiles/nlopt.dir/src/algs/ags/ags.cc.o
[ 96%] Linking CXX static library libnlopt.a
[100%] Built target nlopt
-- Install configuration: "Release"
-- Installing: /work/src/nlopt/lib/pkgconfig/nlopt.pc
-- Installing: /work/src/nlopt/include/nlopt.h
-- Installing: /work/src/nlopt/include/nlopt.hpp
-- Installing: /work/src/nlopt/include/nlopt.f
-- Installing: /work/src/nlopt/lib/libnlopt.a
-- Installing: /work/src/nlopt/lib/cmake/nlopt/NLoptLibraryDepends.cmake
-- Installing: /work/src/nlopt/lib/cmake/nlopt/NLoptLibraryDepends-release.cmake
-- Installing: /work/src/nlopt/lib/cmake/nlopt/NLoptConfig.cmake
-- Installing: /work/src/nlopt/lib/cmake/nlopt/NLoptConfigVersion.cmake
-- Installing: /work/src/nlopt/share/man/man3/nlopt.3
-- Installing: /work/src/nlopt/share/man/man3/nlopt_minimize.3
-- Installing: /work/src/nlopt/share/man/man3/nlopt_minimize_constrained.3
configure: creating ./config.status
config.status: creating src/Makevars
** libs
gcc -std=gnu99 -std=gnu11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/lib/R/site-library/testthat/include' -fpic -g -O2 -ffile-prefix-map=/build/r-base-PT7Nxy/r-base-4.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c init_nloptr.c -o init_nloptr.o
gcc -std=gnu99 -std=gnu11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/lib/R/site-library/testthat/include' -fpic -g -O2 -ffile-prefix-map=/build/r-base-PT7Nxy/r-base-4.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c nloptr.c -o nloptr.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/lib/R/site-library/testthat/include' -fpic -g -O2 -ffile-prefix-map=/build/r-base-PT7Nxy/r-base-4.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c test-C-API.cpp -o test-C-API.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/lib/R/site-library/testthat/include' -fpic -g -O2 -ffile-prefix-map=/build/r-base-PT7Nxy/r-base-4.1.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c test-runner.cpp -o test-runner.o
g++ -std=gnu++11 -shared -L/usr/lib/R/lib -Wl,-z,relro -o nloptr.so init_nloptr.o nloptr.o test-C-API.o test-runner.o -llapack -lblas -lgfortran -lm -lquadmath -Lnlopt/lib -lnlopt -L/usr/lib/R/lib -lR
installing to /usr/local/lib/R/site-library/00LOCK-work/00new/nloptr/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path

• DONE (nloptr)
  root@7f35403ba549:/work#

I can probably try a RHub test build on a few Linux or macOS variants there. They should all reliably call cmake.

[eddelbuettel]

Some follow-up:

• I am not a fan of overly involved GitHub Actions setups, your systematic failure here appears to provide a useful illustration of why (and I frankly also do not understand why it has to spam my forked repo with a new packagedown or whateverdown commit from the bot but het, if you as maintainer prefer that it's all cool
• to asses if this works or not I sent it to RHub for two linux and two macos builds:
  https://builder.r-hub.io/status/nloptr_2.0.0.1.tar.gz-604963a3e5114a3ab61a1a2cba5c4ac2 (debian)
  https://builder.r-hub.io/status/nloptr_2.0.0.1.tar.gz-764c618224834c70b15d150621215fda (ubuntu)
  https://builder.r-hub.io/status/nloptr_2.0.0.1.tar.gz-8b2eb0f0c18b455c895b8b940fd9c1e0 (macOS i7)
  https://builder.r-hub.io/status/nloptr_2.0.0.1.tar.gz-6519a18790de4fc6b746e8ad5930f083 (macOS m1)
• these are back now in email too (I updated DESCRIPTION locally to make the version distinct and make me receive the emails) and show
  • error on macOS M1 as I do not properly set the ARCH subdirectory, easiest to adjust if I copy what you use in cmake
  • OK on plaon macOS (two minutes)
  • NOTE of large subdir (to be ignored) on Debian Linux (twelve minutes) and Ubuntu Linux (twelve minutes)

so this looks to pretty close. I apparently just need one uname call to R_ARCH to arm64.

[eddelbuettel]

Looks like I was wrong -- this is actually an error in your scripts which does not seem to propage ARCH as third argument but rather sets "" consistently. I guess that needs a fix in cmake_call.sh.

Also, nlopt_cleanup.sh uses R_ARCH, others use ARCH. Hm.

[astamm]

Looks like I was wrong -- this is actually an error in your scripts which does not seem to propage ARCH as third argument but rather sets "" consistently. I guess that needs a fix in cmake_call.sh.
Also, nlopt_cleanup.sh uses R_ARCH, others use ARCH. Hm.

I use the $ARCH variable (which I create myself) only in configure.win because I need to loop over the two architectures on Windows until next R release.
This is why in the configure file which is used on Linux and macOS, I call the script with ARCH="", because there is no bi-architecture there.
This is a different environment variable w.r.t. $R_ARCH which is defined in the config.site of R itself.
But maybe I had it wrong in some places, causing install to fail on some platforms.

[astamm]

Following up on the use of GH actions, I like to setup continuous integration this way but I agree that it should not spam others' forks. This is annoying. I will look into it.
As for the automatic R CMD check, it is triggered by the PR. But it should not have failed.
Looking at the details it seems the set up phase of downloading some other GH action script failed.
I will look at that later once the PR is merged.
Thanks for pointing that out too.

[eddelbuettel]

That makes sense as Windows (for a few more months til R 4.2.0 which will drop i386 at long last) is the only bi-arch platform.

It may have gotten side-tracked by the arm fail.

[astamm]

Good. It will indeed be easier when next R release will be out.
Do you still intend to push some more commits or should I merge ?

[eddelbuettel]

If you have a moment, please review this PR as it stands. I have not checked if it bites Windows (should not, has its own file), and I was taken aback when the CI checks failed. If you find it suitable, merge ahead.

[eddelbuettel]

For completeness, my tarball that failed at RHub's M1 machine also passes in NZ on the mac-builder: https://mac.r-project.org/macbuilder/results/1642018536-d00f808ac7702ef6/

[eddelbuettel]

It passes. The supposed Windows failure is an 'WARNING as ERRORS' setting I would never turn on.

[eddelbuettel]

But I added src/nlopt/ to .Rbuildignore which I should have caught earlier.

[eddelbuettel]

Hah -- misses the final comment :) as "resolved" hides the conversation. Coming right up, and indeed easiest in an added YAML. I will permit myself to use my form there.

[eddelbuettel]

I rushed this -- it runs Ubuntu 20.04 where we do not yet have NLopt 2.7, and for symmetry with your approach I made that a minimum requirement. 2.6 is likely good enough, but I will pivot the yaml to Debian testing instead. One sec and now I can dry-running in Docker here before committing. make sure I have NLopt 2.7 available for ubuntu-20.04. It'll take a moment.

[eddelbuettel]

Weeee -- that turned into a comedy of errors but here we are now. "Proof" of it working on all platforms we care about in all deployment forms.

[astamm]

That seems quite neat, thank you. I will merge this.

[eddelbuettel]

Thanks for bearing with me and working through the changes. We are in a good spot now.

[astamm]

Thanks to you.
Since CRAN seems not to answer me about my previous submission, I'll just resubmit this one and see if it can unlock the situation.

[eddelbuettel]

Communicating with them is "challenging" -- but then again there are only a handful or so of them, and literally over a thousand of us bugging the,. Patience is good but we may now be getting there.

[astamm]

I ran further checks on win-builder release and devel. It succeeds on the release version but cannot find cmake on the win-builder devel.
I also ran checks on some r-hub platforms again and all is good except Windows that came back with the following warning that was already reported in #85.
I might have to go there and tweak the source file. Or file an issue in the nlopt repo but we need a reply soon.
I will make another submission maybe as it is now in the meantime because windows checks by CRAN are made on win-builder which goes well except on the devel version but it's out of my hands that cmake cannot be found there. I do not know who to ask to install cmake on the win-builder devel machine. It seems quite strange that it is found on the release machine and not the devel one.

[eddelbuettel]

but cannot find cmake

Well I tried to explain at the very beginning. Few CRAN packages do what you do. CRAN may not be ready, but if you ask nicely they may move along and install it. Re-read Writing R Extentions. Just because you declare something a system dependency does not mean it will be provided.

I do not know who to ask to install cmake on the win-builder devel machine

Email cran@r-project.org and/or for Windows issues Dr Uwe Ligges at U Dortmund.

It seems quite strange that it is found on the release machine and not the devel one.

See above. You may live in a cmake world, but CRAN and most of the R ecosystem do not. It's a risk you took, knowingly or not.

[bbolker]

FWIW, Dirk, in 2017 you commented that you could find 47 packages that were on CRAN and used cmake by searching for CMakelists.txt. Now there seems to be only one, https://github.com/davidcsterratt/geometry/tree/5e6fa7e88f9b93a9841c6da2b180442b43afb8ef ... ?

[eddelbuettel]

Definitely more. From the top of my head Joe's httpuv does the same (of launching into a cmake build of a library). Back then when I had shell access to a directory of unpackaged CRAN packages for grepping, I no longer do,

Expanding the search to cmake gets 943 hits, number of repos surely a subset.