[lsst-dm-stack-users] Using obs_decam
Simon Krughoff
simon.krughoff at gmail.com
Tue Apr 29 09:19:39 PDT 2014
O.K. I'm not sure exactly what offsetUnit does. The way I usually debug
this kind of thing is to put a pdb.set_trace just before the sql query in
daf.butlerUtils.registry. Then you can print the query and see why the
registry is returning more than one data unit.
Out of curiosity, there is usually a filter with each visit. Does it help
to specify that?
Simon
On Tue, Apr 29, 2014 at 8:44 AM, James Chiang <jchiang at slac.stanford.edu>wrote:
> Hi Simon,
>
> Thanks. When I give a unique dataId, I still get the same outcome:
>
> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293 ccd=1 side='N'
>
> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
> 0.0.0.0/config/processCcd.py'
> : Config override file does not exist:
> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/0.0.0.0/config/decam/processCcd.py'
> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
> : calib=None
> : output=None
> CameraMapper: Loading registry registry from
> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
> CameraMapper: Loading calibRegistry registry from
> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
> processCcd FATAL: Failed in task initialization: No unique lookup for
> ['object', 'visit', 'date', 'ccd', 'proposal', 'side'] from {}: 62 matches
> Traceback (most recent call last):
> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
> line 172, in precall
> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
> line 306, in writeConfig
> elif butler.datasetExists(configName):
> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
> line 187, in datasetExists
> location = self.mapper.map(datasetType, dataId)
> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
> line 116, in map
> return func(self.validate(dataId), write)
> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
> line 282, in mapClosure
> return mapping.map(mapper, dataId, write)
> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
> line 118, in map
> actualId = self.need(self.keyDict.iterkeys(), dataId)
> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
> line 201, in need
> raise RuntimeError, "No unique lookup for %s from %s: %d matches" %
> (newProps, newId, len(lookups))
> RuntimeError: No unique lookup for ['object', 'visit', 'date', 'ccd',
> 'proposal', 'side'] from {}: 62 matches
> [obs_decam]$
>
>
> FWIW, I can get obs_decam/processCcd.py to work fine (i.e., it loops over
> all 62 sensors with a partial dataId) using DM stack v6_2 (after commenting
> out the offsetUnit entry in decam/decam_geom.paf).
>
> -Jim
>
>
> On Tue, Apr 29, 2014 at 8:27 AM, Simon Krughoff <simon.krughoff at gmail.com>wrote:
>
>> Hi JIm,
>>
>> This sort of thing happens when the butler tries to figure out a data id
>> from a partial data id. You are only giving the visit, so the butler has
>> to figure out the rest of the data id. Ideally it would loop over all
>> chips in that visit, but somewhere there is a degeneracy that is not being
>> broken by the query the butler is doing.
>>
>> One thing you can try is to give a fully qualified data id (all the keys
>> it needs to find a raw exposure). Without looking more closely I can't say
>> why it can't fill in the rest of your data id. One thing to note is that
>> you can qualify most of the data id and use the range (..) and
>> concatenation (^) operators to specify the ccds you want to process.
>>
>> Simon
>>
>>
>> On Mon, Apr 28, 2014 at 6:14 PM, James Chiang <jchiang at slac.stanford.edu>wrote:
>>
>>> Hi All,
>>>
>>> I'm trying to use obs_decam with DM stack v7_3 to analyze DECam data.
>>> I'm able to run ingestImages.py to get my data into a repository, but when
>>> I try to run processCcd.py, I get this:
>>>
>>> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293
>>> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>> 0.0.0.0/config/processCcd.py'
>>> : Config override file does not exist:
>>> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>> 0.0.0.0/config/decam/processCcd.py'
>>> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
>>> : calib=None
>>> : output=None
>>> CameraMapper: Loading registry registry from
>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
>>> CameraMapper: Loading calibRegistry registry from
>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
>>> processCcd FATAL: Failed in task initialization: no such table:
>>> raw_skyTile
>>> Traceback (most recent call last):
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>> line 172, in precall
>>> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>> line 306, in writeConfig
>>> elif butler.datasetExists(configName):
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
>>> line 187, in datasetExists
>>> location = self.mapper.map(datasetType, dataId)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
>>> line 116, in map
>>> return func(self.validate(dataId), write)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
>>> line 282, in mapClosure
>>> return mapping.map(mapper, dataId, write)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>> line 118, in map
>>> actualId = self.need(self.keyDict.iterkeys(), dataId)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>> line 199, in need
>>> lookups = self.lookup(newProps, newId)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>> line 168, in lookup
>>> where, self.range, values)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/registries.py",
>>> line 120, in executeQuery
>>> c = self.conn.execute(cmd, values)
>>> OperationalError: no such table: raw_skyTile
>>> [obs_decam]$
>>>
>>>
>>> There is, of course, no raw_skyTile table in registry.sqlite3, so I
>>> commented out the corresponding lines in policy/DecamMapper.paf and get the
>>> following:
>>>
>>> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293
>>> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>> 0.0.0.0/config/processCcd.py'
>>> : Config override file does not exist:
>>> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>> 0.0.0.0/config/decam/processCcd.py'
>>> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
>>> : calib=None
>>> : output=None
>>> CameraMapper: Loading registry registry from
>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
>>> CameraMapper: Loading calibRegistry registry from
>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
>>> processCcd FATAL: Failed in task initialization: No unique lookup for
>>> ['object', 'visit', 'date', 'ccd', 'proposal', 'side'] from {}: 62 matches
>>> Traceback (most recent call last):
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>> line 172, in precall
>>> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>> line 306, in writeConfig
>>> elif butler.datasetExists(configName):
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
>>> line 187, in datasetExists
>>> location = self.mapper.map(datasetType, dataId)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
>>> line 116, in map
>>> return func(self.validate(dataId), write)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
>>> line 282, in mapClosure
>>> return mapping.map(mapper, dataId, write)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>> line 118, in map
>>> actualId = self.need(self.keyDict.iterkeys(), dataId)
>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>> line 201, in need
>>> raise RuntimeError, "No unique lookup for %s from %s: %d matches" %
>>> (newProps, newId, len(lookups))
>>> RuntimeError: No unique lookup for ['object', 'visit', 'date', 'ccd',
>>> 'proposal', 'side'] from {}: 62 matches
>>> [obs_decam]$
>>>
>>>
>>> Any help or suggestions on how to proceed would be most welcome.
>>>
>>> Thanks,
>>>
>>> -Jim
>>>
>>> --
>>> James Chiang SLAC, MS 29 home: (650) 605-3346
>>> Fermi ISOC 2575 Sand Hill Rd office: (650) 926-2930
>>> Menlo Park CA 94025 FAX: (650) 926-5566
>>>
>>> _______________________________________________
>>> lsst-dm-stack-users mailing list
>>> lsst-dm-stack-users at lsstcorp.org
>>> http://listserv.lsstcorp.org/mailman/listinfo/lsst-dm-stack-users
>>>
>>>
>>
>>
>> --
>> K. Simon Krughoff
>> University of Washington
>> Department of Astronomy
>>
>> Physics-Astronomy Bldg.
>> 3910 15th Ave NE
>> Room C319
>> Seattle WA 98195-0002
>> USA
>>
>> (206)543-9487
>>
>
>
>
> --
> James Chiang SLAC, MS 29 home: (650) 605-3346
> Fermi ISOC 2575 Sand Hill Rd office: (650) 926-2930
> Menlo Park CA 94025 FAX: (650) 926-5566
>
--
K. Simon Krughoff
University of Washington
Department of Astronomy
Physics-Astronomy Bldg.
3910 15th Ave NE
Room C319
Seattle WA 98195-0002
USA
(206)543-9487
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://listserv.lsstcorp.org/pipermail/dm-users/attachments/20140429/f325fb74/attachment.html>
More information about the dm-users
mailing list