[lsst-dm-stack-users] Using obs_decam
James Chiang
jchiang at slac.stanford.edu
Tue Apr 29 09:34:34 PDT 2014
For anyone interested, we've moved this conversation to hipchat in the
obs_decam room. -Jim
On Tue, Apr 29, 2014 at 9:19 AM, Simon Krughoff <simon.krughoff at gmail.com>wrote:
> O.K. I'm not sure exactly what offsetUnit does. The way I usually debug
> this kind of thing is to put a pdb.set_trace just before the sql query in
> daf.butlerUtils.registry. Then you can print the query and see why the
> registry is returning more than one data unit.
>
> Out of curiosity, there is usually a filter with each visit. Does it help
> to specify that?
>
> Simon
>
>
> On Tue, Apr 29, 2014 at 8:44 AM, James Chiang <jchiang at slac.stanford.edu>wrote:
>
>> Hi Simon,
>>
>> Thanks. When I give a unique dataId, I still get the same outcome:
>>
>> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293 ccd=1 side='N'
>>
>> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>> 0.0.0.0/config/processCcd.py'
>> : Config override file does not exist:
>> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/0.0.0.0/config/decam/processCcd.py
>> '
>> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
>> : calib=None
>> : output=None
>> CameraMapper: Loading registry registry from
>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
>> CameraMapper: Loading calibRegistry registry from
>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
>> processCcd FATAL: Failed in task initialization: No unique lookup for
>> ['object', 'visit', 'date', 'ccd', 'proposal', 'side'] from {}: 62 matches
>> Traceback (most recent call last):
>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>> line 172, in precall
>> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>> line 306, in writeConfig
>> elif butler.datasetExists(configName):
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
>> line 187, in datasetExists
>> location = self.mapper.map(datasetType, dataId)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
>> line 116, in map
>> return func(self.validate(dataId), write)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
>> line 282, in mapClosure
>> return mapping.map(mapper, dataId, write)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>> line 118, in map
>> actualId = self.need(self.keyDict.iterkeys(), dataId)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>> line 201, in need
>> raise RuntimeError, "No unique lookup for %s from %s: %d matches" %
>> (newProps, newId, len(lookups))
>> RuntimeError: No unique lookup for ['object', 'visit', 'date', 'ccd',
>> 'proposal', 'side'] from {}: 62 matches
>> [obs_decam]$
>>
>>
>> FWIW, I can get obs_decam/processCcd.py to work fine (i.e., it loops over
>> all 62 sensors with a partial dataId) using DM stack v6_2 (after commenting
>> out the offsetUnit entry in decam/decam_geom.paf).
>>
>> -Jim
>>
>>
>> On Tue, Apr 29, 2014 at 8:27 AM, Simon Krughoff <simon.krughoff at gmail.com
>> > wrote:
>>
>>> Hi JIm,
>>>
>>> This sort of thing happens when the butler tries to figure out a data id
>>> from a partial data id. You are only giving the visit, so the butler has
>>> to figure out the rest of the data id. Ideally it would loop over all
>>> chips in that visit, but somewhere there is a degeneracy that is not being
>>> broken by the query the butler is doing.
>>>
>>> One thing you can try is to give a fully qualified data id (all the keys
>>> it needs to find a raw exposure). Without looking more closely I can't say
>>> why it can't fill in the rest of your data id. One thing to note is that
>>> you can qualify most of the data id and use the range (..) and
>>> concatenation (^) operators to specify the ccds you want to process.
>>>
>>> Simon
>>>
>>>
>>> On Mon, Apr 28, 2014 at 6:14 PM, James Chiang <jchiang at slac.stanford.edu
>>> > wrote:
>>>
>>>> Hi All,
>>>>
>>>> I'm trying to use obs_decam with DM stack v7_3 to analyze DECam data.
>>>> I'm able to run ingestImages.py to get my data into a repository, but when
>>>> I try to run processCcd.py, I get this:
>>>>
>>>> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293
>>>> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>>> 0.0.0.0/config/processCcd.py'
>>>> : Config override file does not exist:
>>>> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>>> 0.0.0.0/config/decam/processCcd.py'
>>>> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
>>>> : calib=None
>>>> : output=None
>>>> CameraMapper: Loading registry registry from
>>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
>>>> CameraMapper: Loading calibRegistry registry from
>>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
>>>> processCcd FATAL: Failed in task initialization: no such table:
>>>> raw_skyTile
>>>> Traceback (most recent call last):
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>>> line 172, in precall
>>>> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>>> line 306, in writeConfig
>>>> elif butler.datasetExists(configName):
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
>>>> line 187, in datasetExists
>>>> location = self.mapper.map(datasetType, dataId)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
>>>> line 116, in map
>>>> return func(self.validate(dataId), write)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
>>>> line 282, in mapClosure
>>>> return mapping.map(mapper, dataId, write)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>>> line 118, in map
>>>> actualId = self.need(self.keyDict.iterkeys(), dataId)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>>> line 199, in need
>>>> lookups = self.lookup(newProps, newId)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>>> line 168, in lookup
>>>> where, self.range, values)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/registries.py",
>>>> line 120, in executeQuery
>>>> c = self.conn.execute(cmd, values)
>>>> OperationalError: no such table: raw_skyTile
>>>> [obs_decam]$
>>>>
>>>>
>>>> There is, of course, no raw_skyTile table in registry.sqlite3, so I
>>>> commented out the corresponding lines in policy/DecamMapper.paf and get the
>>>> following:
>>>>
>>>> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293
>>>> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>>> 0.0.0.0/config/processCcd.py'
>>>> : Config override file does not exist:
>>>> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>>>> 0.0.0.0/config/decam/processCcd.py'
>>>> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
>>>> : calib=None
>>>> : output=None
>>>> CameraMapper: Loading registry registry from
>>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
>>>> CameraMapper: Loading calibRegistry registry from
>>>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
>>>> processCcd FATAL: Failed in task initialization: No unique lookup for
>>>> ['object', 'visit', 'date', 'ccd', 'proposal', 'side'] from {}: 62 matches
>>>> Traceback (most recent call last):
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>>> line 172, in precall
>>>> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>>>> line 306, in writeConfig
>>>> elif butler.datasetExists(configName):
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
>>>> line 187, in datasetExists
>>>> location = self.mapper.map(datasetType, dataId)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
>>>> line 116, in map
>>>> return func(self.validate(dataId), write)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
>>>> line 282, in mapClosure
>>>> return mapping.map(mapper, dataId, write)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>>> line 118, in map
>>>> actualId = self.need(self.keyDict.iterkeys(), dataId)
>>>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>>>> line 201, in need
>>>> raise RuntimeError, "No unique lookup for %s from %s: %d matches" %
>>>> (newProps, newId, len(lookups))
>>>> RuntimeError: No unique lookup for ['object', 'visit', 'date', 'ccd',
>>>> 'proposal', 'side'] from {}: 62 matches
>>>> [obs_decam]$
>>>>
>>>>
>>>> Any help or suggestions on how to proceed would be most welcome.
>>>>
>>>> Thanks,
>>>>
>>>> -Jim
>>>>
>>>> --
>>>> James Chiang SLAC, MS 29 home: (650) 605-3346
>>>> Fermi ISOC 2575 Sand Hill Rd office: (650) 926-2930
>>>> Menlo Park CA 94025 FAX: (650) 926-5566
>>>>
>>>> _______________________________________________
>>>> lsst-dm-stack-users mailing list
>>>> lsst-dm-stack-users at lsstcorp.org
>>>> http://listserv.lsstcorp.org/mailman/listinfo/lsst-dm-stack-users
>>>>
>>>>
>>>
>>>
>>> --
>>> K. Simon Krughoff
>>> University of Washington
>>> Department of Astronomy
>>>
>>> Physics-Astronomy Bldg.
>>> 3910 15th Ave NE
>>> Room C319
>>> Seattle WA 98195-0002
>>> USA
>>>
>>> (206)543-9487
>>>
>>
>>
>>
>> --
>> James Chiang SLAC, MS 29 home: (650) 605-3346
>> Fermi ISOC 2575 Sand Hill Rd office: (650) 926-2930
>> Menlo Park CA 94025 FAX: (650) 926-5566
>>
>
>
>
> --
> K. Simon Krughoff
> University of Washington
> Department of Astronomy
>
> Physics-Astronomy Bldg.
> 3910 15th Ave NE
> Room C319
> Seattle WA 98195-0002
> USA
>
> (206)543-9487
>
--
James Chiang SLAC, MS 29 home: (650) 605-3346
Fermi ISOC 2575 Sand Hill Rd office: (650) 926-2930
Menlo Park CA 94025 FAX: (650) 926-5566
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://listserv.lsstcorp.org/pipermail/dm-users/attachments/20140429/5d55ba17/attachment.html>
More information about the dm-users
mailing list