[lsst-dm-stack-users] Information and good practices about rules which define if a package has to managed by eups or by the system.

[Mario Juric]

On 4/3/14, 8:28 , Fabrice Jammes wrote:
> Hello,
> 

Hi Fabrice,
	I'm cross-posting to lsst-data, since that may be a better list for
these kinds of technical/development details. Let's try to move future
replies there.

> For Qserv, i packaged several dependencies with eups, and  i would like
> to understand better the rules which define if a package has to managed
> by eups or by the system.
> 

....[snip]....

(disclaimer: everything below is open for discussion!)

(note: these are all good questions, and they recently arised in the
"Dependencies for MAF" thread on lsst-data; take a look at that one as
well).

I think we're trying to satisfy two (conflicting) desiderata:

* Make it easy for someone w/o root access to install LSST software
* Make our software integrate well with the system packages.

The former would be easiest if we essentially packaged and distributed
everything through EUPS. The latter would be easiest by packaging
nothing other than LSST-written code with EUPS.

I think we're beginning to adopt a middle-of-the-road solution.
LSST-written code will be packaged and distributed with EUPS. Some less
common external packages may be packaged and distributed that way as
well. But common external packages (python, numpy, Java, etc.), won't;
we will require the user to provide them. However, since this may be
hard for an average user (as well as for us, when we deploy to machines
we don't fully control (think XSEDE or OSG)), we should strive to
provide EUPS-packaged versions of those external packages as well.

Importantly, LSST packages wouldn't explicitly depend on these (i.e.,
through setupRequired/setupOptional). Instead, the check for their
compatibility and/or existence would be made in newinstall.sh.

Here's a concrete example: the stack depends on Python 2.7, numpy and
matplotlib. But we don't depend on them, explicitly (*). Instead, we
check for their existence in newinstall.sh and warn the user if they're
incompatible. We also offer to install them from EUPS, for convenience
(essentially, this just runs 'eups distrib install anaconda').
Ultimately, I'd like more and more packages to move into this category,
rather than be explicit build-time dependencies.

(*) We actually do depend on them (there are still stub python,
matplotlib, and numpy packages that do nothing other than check that all
these are compatible). I now see this as a historical artifact and would
like to make them go away completely in the next release.

> 
> A concrete example : zookeeper relies on java. So do we have to package
> java in eups or can we use the system version ? If we use the system
> version, do we perform both  the eups declare product system, and the
> version checking in newinstall.sh ? In this case Qserv and LSST-stack
> may have a different newinstall.sh file, depending on their system
> dependencies.
> 

Java should definitely be an external package, i.e., no qserv packages
should explicitly depend on it in their table files. Also, Java
shouldn't be declared (in general, no end-user should *ever* need to run
'eups declare' unless they're hacking).

For convenience, we should package Java to let those who don't have the
system version installed do something like:

	eups distrib install java

to get it.

I see your point about newinstall.sh. What you're saying is that
different prerequisites may be requried for different pieces of the
stack. A quick-and-dirty workaround would be to advise user they don't
require Java unless they're trying to install qserv. But that's not a
good long-term solution; we should think about what needs to be done
with EUPS to make it more aware of system packages.

> Is there some rules and good practices in order to choose a packaging
> method (system or eups) for a given product ? I suppose that stable
> products (like kernel, gcc, libc) could be system-based, so that
> newinstall.sh would change only exceptionally, and that products moving
> quickly could be eups-based, but is it possible to set more accurately
> the cursor ?
> 

As I said in the MAF thread, there are general guidelines, but I think
we'll want to decide on a case-by-case basis. It's probably nothing more
involved than a quick e-mail or chat via HipChat.

Hope this helps,
-- 
Mario Juric,
Data Mgmt. Project Scientist, Large Synoptic Survey Telescope
Web : http://research.majuric.org     Phone : +1 617 744 9003