[lsst-dm-stack-users] Using obs_decam
James Chiang
jchiang at slac.stanford.edu
Tue Apr 29 08:44:41 PDT 2014
Hi Simon,
Thanks. When I give a unique dataId, I still get the same outcome:
[obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293 ccd=1 side='N'
: Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
0.0.0.0/config/processCcd.py'
: Config override file does not exist:
'/u/gl/jchiang/ki18/DESC/DES/obs_decam/0.0.0.0/config/decam/processCcd.py'
: input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
: calib=None
: output=None
CameraMapper: Loading registry registry from
/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
CameraMapper: Loading calibRegistry registry from
/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
processCcd FATAL: Failed in task initialization: No unique lookup for
['object', 'visit', 'date', 'ccd', 'proposal', 'side'] from {}: 62 matches
Traceback (most recent call last):
File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
line 172, in precall
task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
line 306, in writeConfig
elif butler.datasetExists(configName):
File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
line 187, in datasetExists
location = self.mapper.map(datasetType, dataId)
File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
line 116, in map
return func(self.validate(dataId), write)
File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
line 282, in mapClosure
return mapping.map(mapper, dataId, write)
File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
line 118, in map
actualId = self.need(self.keyDict.iterkeys(), dataId)
File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
line 201, in need
raise RuntimeError, "No unique lookup for %s from %s: %d matches" %
(newProps, newId, len(lookups))
RuntimeError: No unique lookup for ['object', 'visit', 'date', 'ccd',
'proposal', 'side'] from {}: 62 matches
[obs_decam]$
FWIW, I can get obs_decam/processCcd.py to work fine (i.e., it loops over
all 62 sensors with a partial dataId) using DM stack v6_2 (after commenting
out the offsetUnit entry in decam/decam_geom.paf).
-Jim
On Tue, Apr 29, 2014 at 8:27 AM, Simon Krughoff <simon.krughoff at gmail.com>wrote:
> Hi JIm,
>
> This sort of thing happens when the butler tries to figure out a data id
> from a partial data id. You are only giving the visit, so the butler has
> to figure out the rest of the data id. Ideally it would loop over all
> chips in that visit, but somewhere there is a degeneracy that is not being
> broken by the query the butler is doing.
>
> One thing you can try is to give a fully qualified data id (all the keys
> it needs to find a raw exposure). Without looking more closely I can't say
> why it can't fill in the rest of your data id. One thing to note is that
> you can qualify most of the data id and use the range (..) and
> concatenation (^) operators to specify the ccds you want to process.
>
> Simon
>
>
> On Mon, Apr 28, 2014 at 6:14 PM, James Chiang <jchiang at slac.stanford.edu>wrote:
>
>> Hi All,
>>
>> I'm trying to use obs_decam with DM stack v7_3 to analyze DECam data.
>> I'm able to run ingestImages.py to get my data into a repository, but when
>> I try to run processCcd.py, I get this:
>>
>> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293
>> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>> 0.0.0.0/config/processCcd.py'
>> : Config override file does not exist:
>> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/0.0.0.0/config/decam/processCcd.py
>> '
>> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
>> : calib=None
>> : output=None
>> CameraMapper: Loading registry registry from
>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
>> CameraMapper: Loading calibRegistry registry from
>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
>> processCcd FATAL: Failed in task initialization: no such table:
>> raw_skyTile
>> Traceback (most recent call last):
>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>> line 172, in precall
>> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>> line 306, in writeConfig
>> elif butler.datasetExists(configName):
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
>> line 187, in datasetExists
>> location = self.mapper.map(datasetType, dataId)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
>> line 116, in map
>> return func(self.validate(dataId), write)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
>> line 282, in mapClosure
>> return mapping.map(mapper, dataId, write)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>> line 118, in map
>> actualId = self.need(self.keyDict.iterkeys(), dataId)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>> line 199, in need
>> lookups = self.lookup(newProps, newId)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>> line 168, in lookup
>> where, self.range, values)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/registries.py",
>> line 120, in executeQuery
>> c = self.conn.execute(cmd, values)
>> OperationalError: no such table: raw_skyTile
>> [obs_decam]$
>>
>>
>> There is, of course, no raw_skyTile table in registry.sqlite3, so I
>> commented out the corresponding lines in policy/DecamMapper.paf and get the
>> following:
>>
>> [obs_decam]$ processCcd.py data_repo_v7_3 --id visit=155293
>> : Loading config overrride file '/u/gl/jchiang/ki18/DESC/DES/obs_decam/
>> 0.0.0.0/config/processCcd.py'
>> : Config override file does not exist:
>> '/u/gl/jchiang/ki18/DESC/DES/obs_decam/0.0.0.0/config/decam/processCcd.py
>> '
>> : input=/nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3
>> : calib=None
>> : output=None
>> CameraMapper: Loading registry registry from
>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/registry.sqlite3
>> CameraMapper: Loading calibRegistry registry from
>> /nfs/slac/g/ki/ki18/jchiang/DESC/DES/data_repo_v7_3/calibRegistry.sqlite3
>> processCcd FATAL: Failed in task initialization: No unique lookup for
>> ['object', 'visit', 'date', 'ccd', 'proposal', 'side'] from {}: 62 matches
>> Traceback (most recent call last):
>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>> line 172, in precall
>> task.writeConfig(parsedCmd.butler, clobber=self.clobberConfig)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/pipe_base/7.3.1.0+1/python/lsst/pipe/base/cmdLineTask.py",
>> line 306, in writeConfig
>> elif butler.datasetExists(configName):
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/butler.py",
>> line 187, in datasetExists
>> location = self.mapper.map(datasetType, dataId)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_persistence/7.3.1.0+1/python/lsst/daf/persistence/mapper.py",
>> line 116, in map
>> return func(self.validate(dataId), write)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/cameraMapper.py",
>> line 282, in mapClosure
>> return mapping.map(mapper, dataId, write)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>> line 118, in map
>> actualId = self.need(self.keyDict.iterkeys(), dataId)
>> File "/opt/lsst/rhel-6-x86_64/Linux64/daf_butlerUtils/7.3.1.1+1/python/lsst/daf/butlerUtils/mapping.py",
>> line 201, in need
>> raise RuntimeError, "No unique lookup for %s from %s: %d matches" %
>> (newProps, newId, len(lookups))
>> RuntimeError: No unique lookup for ['object', 'visit', 'date', 'ccd',
>> 'proposal', 'side'] from {}: 62 matches
>> [obs_decam]$
>>
>>
>> Any help or suggestions on how to proceed would be most welcome.
>>
>> Thanks,
>>
>> -Jim
>>
>> --
>> James Chiang SLAC, MS 29 home: (650) 605-3346
>> Fermi ISOC 2575 Sand Hill Rd office: (650) 926-2930
>> Menlo Park CA 94025 FAX: (650) 926-5566
>>
>> _______________________________________________
>> lsst-dm-stack-users mailing list
>> lsst-dm-stack-users at lsstcorp.org
>> http://listserv.lsstcorp.org/mailman/listinfo/lsst-dm-stack-users
>>
>>
>
>
> --
> K. Simon Krughoff
> University of Washington
> Department of Astronomy
>
> Physics-Astronomy Bldg.
> 3910 15th Ave NE
> Room C319
> Seattle WA 98195-0002
> USA
>
> (206)543-9487
>
--
James Chiang SLAC, MS 29 home: (650) 605-3346
Fermi ISOC 2575 Sand Hill Rd office: (650) 926-2930
Menlo Park CA 94025 FAX: (650) 926-5566
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://listserv.lsstcorp.org/pipermail/dm-users/attachments/20140429/24a6a8b4/attachment.html>
More information about the dm-users
mailing list