[lsst-dm-stack-users] problem installing sconsUtils on ubuntu 12.04

[Adrian Pope]

I sent another email to the Argonne network folks to see if they consider it a bug that the proxy delivers files in an altered state (or not) depending on mime type. It seems like this is inconsistent/undesirable behavior.

In the meantime I also chatted with a colleague here who wondered whether the application/gzip or application/octet-stream mime types would be more appropriate for the LSST DM package files.

Thanks,
Adrian

On Aug 26, 2014, at 2:20 PM, Dustin Lang <dstndstn at gmail.com> wrote:

> 
> It sounds like your proxy server is too smart by half, as well as not very 
> smart.
> 
> It's scanning for malicious content, yet demands that the server (---the 
> server that may be serving malicious content!---) correctly identify the 
> content-type.  Hmm... ok...
> 
> --dstn
> 
> 
> On Tue, 26 Aug 2014, Adrian Pope wrote:
> 
>> On Aug 21, 2014, at 7:41 PM, Kian-Tat Lim <ktl at slac.stanford.edu> wrote:
>> 
>>> Adrian,
>>> 
>>>> I think I finally understand something about what’s going on with the LSST DM stack install at Argonne. On some of the networks here I end up with an uncompressed sconsUtils-8.0.0.0.eupspkg file after download.
>>> 
>>> 	Wow.  Are you forced to use an HTTP proxy on the ArgonneA-Auth
>>> network?  
>> 
>> The network folks at Argonne tracked it down to a problem with how sw.lsstcorp.org is identifying file types and what the proxy (which must operate on the wired networks in addition to the auth wireless, but not on guest wireless) wants to do to different file types to scan for viruses, etc. Here’s the relevant part of that conversation:
>> 
>> 2014-08-22 17:21 CDT - Eugene Rackow	Customer Comments
>> It appears to be a combination of things causing this problem.
>> Primarly the difference between auth and guest is that the auth nets are going through a proxy.
>> The proxy checks the data for malware and virus before it getting to your desktop.
>> On the wireless side it does not get those checks.
>> 
>> Next, the fault appears to lie with the remote site, sw.lsstcorp.org.
>> Their web server is improperly identifying the files being sent.
>> 
>> On the files that appear to work, their server claims them as application type "troff", which is clearly wrong, but the web proxy just passes these through.
>> On the files that fail, their server claims the file is type "x-tar". The file IS a gzip tar file, so the proxy unpacks the file to match the application type.
>> Web servers have the capability of compressing the data stream as it's sending it, so the proxy assumes the server is doing compression as part of
>> transport and delivers the application file as defined by the server transport system, uncompressed.
>> This uncompressed version isn't what the application is expecting, but....
>> 
>> Since you have a copy of the files that will work, I recommend you use those
>> If you have a contact for this project and/or server, they should be alerted to the fact their server is doing
>> the wrong thing, inconsistently which will cause problems for some of their people attempting to download the software.
>> 
>>> Do things change if you use this command?
>>> 
>>> curl -k -O https://sw.lsstcorp.org/eupspkg/products/sconsUtils-8.0.0.0.eupspkg
>> 
>> Yes - the file shows up still compressed if I use that command.
>> 
>>> If so, we can move a little faster at making the "-k" unnecessary.
>>> 	If your network is not end-to-end transparent, it's not clear
>>> that patching the installer for this particular case wouldn't just lead
>>> to worse behavior later (e.g. when we start checking checksums or
>>> signatures).
>> 
>> I’m not sure what you mean by end-to-end transparent.
>> 
>> It seems like we have the information to fix this now. For workstations at Argonne I don’t think there’s a convenient way to circumvent the proxy scrubbing. It's probably possible to tunnel a connection to the outside world, but this would depend on someone else’s resources and doesn’t strike me as a good solution. Though the way the Argonne proxy deals with compressed files is annoying I would guess it is not unique. Is there some way of refactoring the curl/gzip/tar commands that would be more robust and also support checksums and such later?
>> 
>> Cheers,
>> Adrian
>> _______________________________________________
>> lsst-dm-stack-users mailing list
>> lsst-dm-stack-users at listserv.lsstcorp.org
>> https://listserv.lsstcorp.org/mailman/listinfo/lsst-dm-stack-users