[lsst-dm-stack-users] problem installing sconsUtils on ubuntu 12.04

[Adrian Pope]

On Aug 21, 2014, at 7:41 PM, Kian-Tat Lim <ktl at slac.stanford.edu> wrote:

> Adrian,
> 
>> I think I finally understand something about what’s going on with the LSST DM stack install at Argonne. On some of the networks here I end up with an uncompressed sconsUtils-8.0.0.0.eupspkg file after download.
> 
> 	Wow.  Are you forced to use an HTTP proxy on the ArgonneA-Auth
> network?  

The network folks at Argonne tracked it down to a problem with how sw.lsstcorp.org is identifying file types and what the proxy (which must operate on the wired networks in addition to the auth wireless, but not on guest wireless) wants to do to different file types to scan for viruses, etc. Here’s the relevant part of that conversation:

2014-08-22 17:21 CDT - Eugene Rackow	Customer Comments
It appears to be a combination of things causing this problem.
Primarly the difference between auth and guest is that the auth nets are going through a proxy.
The proxy checks the data for malware and virus before it getting to your desktop.
On the wireless side it does not get those checks.
 
Next, the fault appears to lie with the remote site, sw.lsstcorp.org.
Their web server is improperly identifying the files being sent.
 
On the files that appear to work, their server claims them as application type "troff", which is clearly wrong, but the web proxy just passes these through.
On the files that fail, their server claims the file is type "x-tar". The file IS a gzip tar file, so the proxy unpacks the file to match the application type.
Web servers have the capability of compressing the data stream as it's sending it, so the proxy assumes the server is doing compression as part of
transport and delivers the application file as defined by the server transport system, uncompressed.
This uncompressed version isn't what the application is expecting, but....
 
Since you have a copy of the files that will work, I recommend you use those
If you have a contact for this project and/or server, they should be alerted to the fact their server is doing
the wrong thing, inconsistently which will cause problems for some of their people attempting to download the software.

> Do things change if you use this command?
> 
> curl -k -O https://sw.lsstcorp.org/eupspkg/products/sconsUtils-8.0.0.0.eupspkg

Yes - the file shows up still compressed if I use that command.

> If so, we can move a little faster at making the "-k" unnecessary.
> 	If your network is not end-to-end transparent, it's not clear
> that patching the installer for this particular case wouldn't just lead
> to worse behavior later (e.g. when we start checking checksums or
> signatures).

I’m not sure what you mean by end-to-end transparent.

It seems like we have the information to fix this now. For workstations at Argonne I don’t think there’s a convenient way to circumvent the proxy scrubbing. It's probably possible to tunnel a connection to the outside world, but this would depend on someone else’s resources and doesn’t strike me as a good solution. Though the way the Argonne proxy deals with compressed files is annoying I would guess it is not unique. Is there some way of refactoring the curl/gzip/tar commands that would be more robust and also support checksums and such later?

Cheers,
Adrian