[lsst-dm-stack-users] Feedback on preparing a relocatable binary distribution of the stack

[Fabio Hernandez]

Hello,

I have been working recently on getting familiar with the LSST stack as an end user, in particular with the aim of producing a binary package from sources. My goal is to use that binary to evaluate CernVM FS [1] as a mechanism to share a centralised installation of the stack, so that individuals and compute nodes can use exactly the same version. This would be particularly useful to provide end users a way to mount and use the stack from the comfort of their personal computers, across wide area networks, without requiring them to explicitly build the software.

This is a report on my progress with this task. (Apologies for the length of this message.)

*) Testing environment: Scientific Linux v6.5 (64 bits)

*) I pre-installed the current version of Anaconda (v.2.0.1 64 bits), so that the bootstrap script does not have to. (This Anaconda uses Python v2.7.7). This is to separate the concerns of relocating Anaconda and relocating the LSST-specific packages. Git v1.8.4.2 was already installed on my system.

*) I built the stack from sources following the instructions in [2].

*) I tested my installation by running the demo, following the instructions in [3] (without the step of visualisation of the output, though).

Everything went well, as far as I can tell looking at the text results produced by the demo. However, I didn't verify the accuracy of the results. I mention this because newinstall.sh installs an older version of Anaconda (so an older version of numpy and friends) than the one I used for my test.

Now, here is what I found trying to relocate the LSST stack (not Anaconda) to another directory. By relocating I mean moving all the contents of $LSST_HOME to another directory with different path in the same or in another machine:

1) The bootstrap script 'newinstall.sh' creates 'loadLSST.sh' (among other scripts) to prepare the environment for installing the stack. loadLSST.sh sources the script for setting up EUPS, i.e.  $LSST_HOME/eups/bin/setups.sh. Unfortunately, it uses the absolute path of this script. This means that when relocating to another $LSST_HOME you have to edit (or automatically process) this file. I see two possibilities of improvement here:

a) loadLSST.sh uses the value of the environmental variable LSST_HOME to locate the EUPS setup script, or better

b) loadLSST.sh determines at execution time what the value of LSST_HOME is and locates the path of EUPS setup script based on that value. The value of LSST_HOME can be determined based on the location of the script itself. The supported flavors of the shells (bash, ksh, csh) all provide mechanisms for determining at execution time the exact directory where a script is located.


2) The script $LSST_HOME/eups/bin/setups.sh sets the environmental variables EUPS_PATH  and EUPS_DIR at configuration time. The value of these variables are $LSST_HOME  and $LSST_HOME/eups respectively. As in the previous case, the EUPS setup script could determine the correct value of these two variables at execution time based on the location of the script itself.

3) The eupspkg file for some of the packages installed are symbolic links which point to $LSST_HOME/eups/lib/eupspkg.sh. For some other packages, it is not a symbolic link but a copy of the same file. My understanding is that this is not relevant at execution time, but I found curious that not all packages are similar in that respect. For instance, mysqlclient package uses a symbolic link whereas skypix does not. Again, the symbolic links point to the absolute path of the $LSST_HOME/eups/lib/eupspkg.sh script. From my test, it is not necessary to relocate these symbolic links for the stack to execute correctly, but I mention this feature in case it may be the symptom of something not intended.


The good news is that not too much seems missing for having a relocatable installation of the stack. Solving this would allow for having several installed versions of the stack on the same machine and let the user choose whichever s/he wants. Also, it seems that the stack is compatible with the current version of Anacoda, so you may want to consider upgrading.

I hope this feedback may help other people with similar goals. I'm certainly interested in the feedback that the LSST stack experts would provide. I also volunteer to help modifying the shell scripts to implement my proposed modifications and 1) and 2). If the developers are interested in a contribution on this, please let me know how to proceed.

In the coming days, I will report on my progress evaluating of CernVM FS, for those interested.

Cheers,


[1] http://cernvm.cern.ch/portal/sites/cernvm.cern.ch/files/cvmfstech-2.1-4.pdf
[2] https://confluence.lsstcorp.org/display/LSWUG/Building+the+LSST+Stack+from+Source
[3] https://confluence.lsstcorp.org/display/LSWUG/Testing+the+Installation

Fabio Hernandez

CNRS – IN2P3 Computing Centre · Lyon (France)     ·     e-mail: fabio at in2p3.fr     ·     tel: +33 4 78 93 08 80


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://listserv.lsstcorp.org/pipermail/dm-users/attachments/20140924/a1f7ec6d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1870 bytes
Desc: not available
URL: <https://listserv.lsstcorp.org/pipermail/dm-users/attachments/20140924/a1f7ec6d/attachment.p7s>