Fix hardcoding of paths in R binary

[kcrisman]

See this thread on sage-support.

Here is how I got the optional package automap to install into a 
binary sage R. 
Go into the sage directory and edit the following files: 
local/bin/R and local/lib/R/bin/R 
and change all the hard-set user variables "/scratch/...." to the true 
locations of R_HOME_DIR, R_HOME, R_INCLUDE_DIR, R_SHARE_DIR and for 
good measure, R_DOC_DIR. Replace the default string EVERYWHERE in the 
file. 
I then exported SAGE_HOME as well (Not sure that this is needed.), and 
run local/bin/R 
Inside R, install.packages("automap") 
No more build errors, and when I restart R, automap loads using 
library. Just have to try it out from sage now. 
Any chance there's a script to find all of these hard-set strings and 
change them to correct values? 

---

Related (R package):

• Move the RPy package outside of the R package #9906
• Handle preset R_PROFILE or R_HOME variables #9847
• R's spkg-install needs a good overhaul. #8274
• R breaks if SAGE_LOCAL undefined #10967

---

New spkg: http://boxen.math.washington.edu/home/palmieri/SPKG/r-2.15.2.p2.spkg

CC: @nexttime @jhpalmieri

Component: packages: standard

Keywords: R spkg R.sh.in libR.pc pkg-config hard-coded package installation R_HOME_DIR sd32 r-project

Author: John Palmieri

Reviewer: Karl-Dieter Crisman, Jeroen Demeyer

Merged: sage-5.9.rc0

Issue created by migration from https://trac.sagemath.org/ticket/9668

[nexttime]

comment:2

IIRC I have some "almost" ready R spkg for #9906, which also does a lot of clean-up in spkg-install...

[kcrisman]

comment:3

Well, I would certainly welcome and help review this. Anything which cleans up parts of #8274 could be commented there, and perhaps even a new ticket opened if there isn't much left.

But what about #9668 (this ticket) itself? Does the stuff that is "almost" ready at #9906 address that?

[nexttime]

comment:4

Replying to @kcrisman:

But what about #9668 (this ticket) itself? Does the stuff that is "almost" ready at #9906 address that?

Not yet, i.e. not all of it. See this comment there.

[nexttime]

comment:5

Replying to @nexttime:

Replying to @kcrisman:

But what about #9668 (this ticket) itself? Does the stuff that is "almost" ready at #9906 address that?

Not yet, i.e. not all of it. See this comment there.

Ok, a bit more subtle than I expected (R is quite weird, failed to build inbetween), but I now have a p6 that fixes both the R scripts and the pkg-config file (libR.pc).

---

There was another issue with R someone reported on sage-devel which was caused by a "complicated" setting of an environment variable, in this case PAGER, but that's certainly an upstream problem and rather exotic.

[kcrisman]

comment:6

Replying to @nexttime:

Replying to @nexttime:

Replying to @kcrisman:

But what about #9668 (this ticket) itself? Does the stuff that is "almost" ready at #9906 address that?

Not yet, i.e. not all of it. See this comment there.

Ok, a bit more subtle than I expected (R is quite weird, failed to build inbetween), but I now have a p6 that fixes both the R scripts and the pkg-config file (libR.pc).

Okay, I'll watch this space.

There was another issue with R someone reported on sage-devel which was caused by a "complicated" setting of an environment variable, in this case PAGER, but that's certainly an upstream problem and rather exotic.

Agreed.

[kcrisman]

comment:7

See also #10967. We should probably either incorporate that here, or make a followup spkg there.

[nexttime]

comment:8

Replying to @kcrisman:

See also #10967. We should probably either incorporate that here, or make a followup spkg there.

Almost certainly the former.

[Note to myself:]

The problem is that removing any reference to SAGE_ROOT and SAGE_LOCAL disables the script being "automatically" relocated in the first place.

I'll then have to guess SAGE_ROOT if none of Sage's variables are defined. We also have two copies of the R script, one in local/bin/, and one in local/lib/R/bin/.

[nexttime]

Dependencies: #9906

[nexttime]

Changed keywords from none to R spkg R.sh.in libR.pc pkg-config hard-coded package installation R_HOME_DIR

[nexttime]

comment:9

Replying to @nexttime:

[Note to myself:]

The problem is that removing any reference to SAGE_ROOT and SAGE_LOCAL disables the script being "automatically" relocated in the first place.

I'll then have to guess SAGE_ROOT if none of Sage's variables are defined. We also have two copies of the R script, one in local/bin/, and one in local/lib/R/bin/.

Guessing SAGE_ROOT there is simply bullshit. Simply add a sanity check, pointing to sage --R, install_scripts() and perhaps sage --sh in case SAGE_LOCAL isn't defined.

Furthermore:

• Address / check the following (copied from [comment:ticket:10967:14]):

The problem was with hard-coded paths, not the 
permissions.  Anyway, the fix was easy.  I opened all the files listed 
above by George: 
sage/local/lib/R/bin/R 
sage/local/lib/R/bin/libtool 
sage/local/lib/R/etc/Makeconf 
sage/local/lib/R/etc/ldpath 
sage/local/lib/R/etc/Renviron 
sage/local/bin/R 
and edited obvious lines containing hardcoded paths (using find- 
replace-all at once). 

• Change libR.pc to use either ${pc_top_builddir} or $${SAGE_ROOT}. (In the former case, perhaps also set PC_TOP_BUILD_DIR to $SAGE_ROOT, or prepend $SAGE_ROOT:, unless it is already contained.)

[nexttime]

Work Issues: Provide an R 2.10.1.p6 spkg.

[nexttime]

comment:10

Replying to @nexttime:

• Change libR.pc to use either ${pc_top_builddir} or $${SAGE_ROOT}. (In the former case, perhaps also set PC_TOP_BUILD_DIR to $SAGE_ROOT, or prepend $SAGE_ROOT:, unless it is already contained.)

Should read:

• Change libR.pc to use either ${pc_top_builddir} or $${SAGE_ROOT}. (In the former case, set PKG_CONFIG_TOP_BUILD_DIR to $SAGE_ROOT unless it is already, perhaps issuing a warning in case it isn't, or especially if its setting differs.)

  Also prepend $SAGE_ROOT/local/lib/pkgconfig to PKG_CONFIG_PATH, unless it is already contained. Warn if PKG_CONFIG_PATH is empty, then set it to $SAGE_ROOT/local/lib/pkgconfig.

[williamstein]

Changed keywords from R spkg R.sh.in libR.pc pkg-config hard-coded package installation R_HOME_DIR to R spkg R.sh.in libR.pc pkg-config hard-coded package installation R_HOME_DIR sd32

[kcrisman]

Changed keywords from R spkg R.sh.in libR.pc pkg-config hard-coded package installation R_HOME_DIR sd32 to R spkg R.sh.in libR.pc pkg-config hard-coded package installation R_HOME_DIR sd32 r-project

[jhpalmieri]

comment:14

This problem prevents Sage from being relocatable on Solaris, or at least on the skynet machines mark and mark2: if I build Sage, then move the entire Sage directory, then run sage so the relocation scripts get executed, and then run sage -R, I get an error:

$ ./sage -R
ld.so.1: R: fatal: libgcc_s.so.1: version `GCC_4.3.0' not found (required by file /usr/local/gcc-4.7.0/sparc-SunOS-ultrasparc3/lib/libgomp.so.1)
ld.so.1: R: fatal: libgcc_s.so.1: open failed: No such file or directory
/home/palmieri/mark2/sage-5.4.rc2-7797/spkg/bin/sage: line 457: 28710 Killed                  "$SAGE_LOCAL/bin/R" "$@"

I tried just modifying local/bin/R and local/lib/R/bin/R, replacing the hard-coded paths with $SAGE_ROOT, but I still got an error.

[nexttime]

comment:15

I doubt this is related to the original topic at all. The setup on skynet's Solaris machines is (or used to be) quite broken, as there are outdated versions of shared libraries left around in typical paths, and some R scripts insist on messing up your paths such that the former gets relevant.

Unfortunately I don't recall what the exact problem was, and how I managed to work around it, just that I had to [and somehow successfully did]; I think this problem appeared with, or was related to, [the installation of] GCC 4.7.0. In case I am right, changing PATH and/or LD_LIBRARY_PATH should "solve" the issue.

Sorry for not being of much help here, at least right now... ;-)

[jhpalmieri]

comment:16

Well, if the problems on mark are not related, then for this ticket, we could just add some lines at the end of spkg-install:

# Make R relocatable by using "$SAGE_ROOT" instead of the hardcoded path.
sed -e "s|$SAGE_ROOT|\$SAGE_ROOT/|" "$SAGE_LOCAL/bin/R" > "$SAGE_LOCAL/bin/R.tmp" && mv "$SAGE_LOCAL/bin/R.tmp" "$SAGE_LOCAL/bin/R" && chmod a+x "$SAGE_LOCAL/bin/R"
sed -e "s|$SAGE_ROOT|\$SAGE_ROOT/|" "$SAGE_LOCAL/lib/R/bin/R" > "$SAGE_LOCAL/lib/R/bin/R.tmp" && mv "$SAGE_LOCAL/lib/R/bin/R.tmp" "$SAGE_LOCAL/lib/R/bin/R" && chmod a+x "$SAGE_LOCAL/lib/R/bin/R"

(It's too bad that the -i flag for sed is not portable.) libR.pc already uses SAGE_ROOT. What else needs to be done?

[jhpalmieri]

Author: John Palmieri

[jhpalmieri]

comment:17

I've posted a new spkg, along with the corresponding patch.

[jhpalmieri]

Changed work issues from Provide an R 2.10.1.p6 spkg. to none

[jhpalmieri]

comment:19

Replying to @nexttime:

• Address / check the following (copied from [comment:ticket:10967:14]):

The problem was with hard-coded paths, not the 
permissions.  Anyway, the fix was easy.  I opened all the files listed 
above by George: 
sage/local/lib/R/bin/R 
sage/local/lib/R/bin/libtool 
sage/local/lib/R/etc/Makeconf 
sage/local/lib/R/etc/ldpath 
sage/local/lib/R/etc/Renviron 
sage/local/bin/R 
and edited obvious lines containing hardcoded paths (using find- 
replace-all at once). 

It's easy enough to add these to the "for" loop that I added to spkg-install, so I might as well do that. For what it's worth, the binary files Rscript in both local/bin and local/lib/R/bin also have the path hard-coded in them, but I don't know how to fix this, or whether it's important. When are those files used?

[jdemeyer]

comment:36

Replying to @jhpalmieri:

If you look at src/scripts/R.sh.in, it has lines like

        R_HOME_DIR="@prefix@/${libnn}/R"

This line is completely irrelevant, as we already patch R.sh.in to overwrite R_HOME_DIR.

[jhpalmieri]

Attachment: trac_9668-r.v3.patch.gz

[jhpalmieri]

comment:37

I'll post a new version of the spkg when sage.math is back up. I've attached the patch. This has the effect of changing lines in local/bin/R (and the corresponding file local/lib/R/bin/R) from

R_SHARE_DIR=.../sage-5.8.beta4/local/lib/R/share
export R_SHARE_DIR
R_INCLUDE_DIR=.../sage-5.8.beta4/local/lib/R/include
export R_INCLUDE_DIR
R_DOC_DIR=.../sage-5.8.beta4/local/lib/R/doc
export R_DOC_DIR

to

R_SHARE_DIR="${R_HOME_DIR}/share"
export R_SHARE_DIR
R_INCLUDE_DIR="${R_HOME_DIR}/include"
export R_INCLUDE_DIR
R_DOC_DIR="${R_HOME_DIR}/doc"
export R_DOC_DIR

With this change, running sage -R and then install.packages("automap") seems to work.

[jhpalmieri]

comment:38

New spkg posted.

[nexttime]

comment:39

Replying to @jhpalmieri:

I'll post a new version of the spkg when sage.math is back up.

Replying to @jhpalmieri:

New spkg posted.

I still cannot log into sage.math... ;-)

[jhpalmieri]

comment:40

Replying to @nexttime:

Replying to @jhpalmieri:

I'll post a new version of the spkg when sage.math is back up.

Replying to @jhpalmieri:

New spkg posted.

I still cannot log into sage.math... ;-)

I did an scp to boxen. If you want, I can try to keep my promise by posting the new version (again) when sage.math comes back up...

[jhpalmieri]

comment:41

For what it's worth, someone reported on ask.sagemath.org that this spkg fixed their problem with installing an R package.

[kcrisman]

comment:42

I think that sounds like a positive review, combined with Jeroen's good comments... What do you think?

[jhpalmieri]

comment:43

I think that someone should confirm that with the new spkg, in the script local/bin/R, the variables R_SHARE_DIR, R_INCLUDE_DIR, and R_DOC_DIR are now defined in terms of R_HOME_DIR rather than being hard-coded paths as they are with the current spkg.

[kcrisman]

comment:44

I think that someone should confirm that with the new spkg, in the script local/bin/R, the variables R_SHARE_DIR, R_INCLUDE_DIR, and R_DOC_DIR are now defined in terms of R_HOME_DIR rather than being hard-coded paths as they are with the current spkg.

I can do this.

[kcrisman]

Changed reviewer from Karl-Dieter Crisman to Karl-Dieter Crisman, Jeroen Demeyer

[kcrisman]

comment:45

I confirmed that moving a Sage install (not just binary) caused installing an R package to fail with precisely the problems one would expect if these were incorrectly defined (e.g.

Warning: R include directory is empty -- perhaps need to install R-devel.rpm or similar

with appropriate nonexistent directory referenced). The script local/bin/R had the (now incorrect) paths.

Then installing this spkg and retrying caused success, and local/bin/R looks right now too. Nice work!

[kcrisman]

comment:46

One question, though... when I move Sage back and run Sage, it doesn't change R_HOME_DIR back.

R_HOME_DIR=/Users/.../Downloads/tempR/sage-5.9.beta5/local/lib/R

when it should be

R_HOME_DIR=/Users/.../Downloads/sage-5.9.beta5/local/lib/R

So this was changed, presumably, when I reinstalled the spkg. It doesn't impact installing new R packages, by the way, nor functionality of R.

In particular, moving a different Sage installation and starting Sage changes some things, but doesn't change the location of R_HOME_DIR in local/bin/R. I'm not sure why that doesn't affect functionality, but presumably this should somehow be taken care of. On this ticket?

[jhpalmieri]

comment:47

It took me a little while to understand this, too. Right before the lines defining R_SHARE_DIR, etc., there are lines

if test x$SAGE_BUILDING_R = x; then
    R_HOME_DIR="$SAGE_LOCAL/lib/R/"
fi

If you're not building the R spkg, then this will be executed, overriding the hard-coded path earlier in the script, and setting R_HOME_DIR to the desired portable setting. I don't know R, so I don't know how to test this: if you run sage -R, can you execute some R command to tell you the current setting of R_HOME_DIR?

[kcrisman]

comment:48

Well, that is a Sage-specific thing, I think, but we have

R_HOME="${R_HOME_DIR}"

later on and also

$ ./sage -R RHOME
/Users/.../Downloads/sage-5.9.beta5/local/lib/R/

So all is well, I think.

[jdemeyer]

Merged: sage-5.9.rc0