[LSST|dm-users 632] Re: Error when Running Chris Walter's: Introduction to the DM AFW packages ipython notebook
Will Dawson
will at dawsonresearch.com
Fri May 8 09:57:18 PDT 2015
Chris,
Thanks to your notebooks I knew that I could customize a scheme and get a reduction to complete (and your email is helpful in know exactly what to tweak). Still it would be nice if DM stack:
1) had consistent default config/task setups for the makeMinimalSchema function (perhaps v10_1 fixes this)
2) if there were a clear warning for depreciated functions like lsst.meas.algorithms.SourceMeasurement[Config/Task] with a note of the recommended function (perhaps it was there and I just missed it)
Thanks,
Will
> On May 8, 2015, at 6:19 AM, Chris Walter <chris.walter at duke.edu> wrote:
>
> Hi Will,
>
> Sorry.. your note came as I was in the air traveling to Japan. I’ve just arrived here now and will go to sleep soon, but there is a quick fix you can use with 10_0.
>
> Add these first (like in the previous example in the notebook). I noticed this was now necessary in 10_0.
>
> #Aliases only
> measureSourcesConfig.slots.psfFlux = None
> measureSourcesConfig.slots.apFlux = None
> measureSourcesConfig.slots.modelFlux = None
> measureSourcesConfig.slots.instFlux = None
> measureSourcesConfig.validate()
>
> As a reference this code:
>
> https://github.com/cwwalter/brighter-fatter/blob/master/processSpots.py
>
> which uses the same code in the notebook works. You can see I fixed it here:
>
> https://github.com/cwwalter/brighter-fatter/commit/1b0dda84f293306521b3ef6d5cbc7d2d6966019b
>
> We should update the notebooks too (I forgot that this would also be an issue there). When 10_1 is released we should change it to whatever the “recommended method” is.
>
> BTW the way I knew to fix this by looking at the error message is that there are the lines:
>
> "if self.apFlux is not None:”
>
> it is complaining about. So clearly we are not setting that ourselves in the simple example and they are not being set to None when initialized. I didn’t look into exactly what had changed.
>
> -Chris
>
>> On May 7, 2015, at 7:44 PM, Simon Krughoff <simon.krughoff at gmail.com> wrote:
>>
>> That's what it's supposed to look like. Maybe we can set something up tomorrow. Just ping me when you are free and we can set something up.
>>
>> Simon
>>
>> On Thu, May 7, 2015 at 5:07 PM, Will Dawson <will at dawsonresearch.com> wrote:
>> Simon,
>>
>> I am happy to do a screen share session sometime, although I am about to sign-off for a few hours.
>>
>> I do have the lsst64defs.py file (see attached). Looks pretty simple to me.
>>
>> Thanks,
>> Will
>>
>>
>>
>>> On May 7, 2015, at 5:03 PM, Simon Krughoff <simon.krughoff at gmail.com> wrote:
>>>
>>> Very strange. It looks like the lsst64def module was not built. It may work better to solve this in a session where we can share screens, but let me know what you see with the following.
>>>
>>> Do you have a file called lsst64defs.py in /Users/dawson/lsst/EupsBuildDir/DarwinX86/base-10.0-5-g766159b/base-10.0-5-g766159b/python? If so, what is in it?
>>>
>>> If not, what does the following give you if you run it in the python directory?
>>> m4 -Dm4_RTLD_GLOBAL=8 -Dm4_RTLD_NOW=2 < lsst64defs.py.m4 > /Users/dawson/lsst/EupsBuildDir/DarwinX86/base-10.0-5-g766159b/base-10.0-5-g766159b/python/lsst64defs.py
>>>
>>> Simon
>>>
>>> On Thu, May 7, 2015 at 4:49 PM, Will Dawson <will at dawsonresearch.com> wrote:
>>> Simon,
>>>
>>> Here is the .failed file.
>>>
>>> Thanks for your time,
>>> Will
>>>
>>>
>>>
>>>> On May 7, 2015, at 4:43 PM, Simon Krughoff <simon.krughoff at gmail.com> wrote:
>>>>
>>>> Will,
>>>>
>>>> Actually, it looks like a test failed when you were building. The failed test log should be in:
>>>> /Users/dawson/lsst/EupsBuildDir/DarwinX86/base-10.0-5-g766159b/base-10.0-5-g766159b/tests/.tests
>>>>
>>>> There will be a file with the .failed suffix. In there will be the messages about why that test failed.
>>>>
>>>> Can you send that along?
>>>>
>>>> Simon
>>>>
>>>> On Thu, May 7, 2015 at 4:37 PM, Will Dawson <will at dawsonresearch.com> wrote:
>>>> Simon,
>>>>
>>>> I’ve attached the build.log. apparently the error is related to the CC compiler (note I am running Yosemite).
>>>>
>>>> Thanks,
>>>> Will
>>>>
>>>>
>>>>
>>>>> On May 7, 2015, at 4:29 PM, Simon Krughoff <simon.krughoff at gmail.com> wrote:
>>>>>
>>>>> Hi Will,
>>>>>
>>>>> If you can attach the contents of /Users/dawson/lsst/EupsBuildDir/DarwinX86/base-10.0-5-g766159b/build.log. We might be able to diagnose.
>>>>>
>>>>> Simon
>>>>>
>>>>> On Thu, May 7, 2015 at 4:11 PM, Will Dawson <will at dawsonresearch.com> wrote:
>>>>> Jim / Dominique,
>>>>>
>>>>> Thank you for the suggestions. Apparently the SingleFrameMeasurementConfig/Task are not in v10.0, as I get the following error when instantiating it
>>>>>
>>>>> AttributeError: 'module' object has no attribute 'SingleFrameMeasurementConfig'
>>>>>
>>>>> I tried installing 10_1_rc3 and got through 23/55 programs before encountering the following error:
>>>>>
>>>>> eups distrib: Failed to build base-10.0-5-g766159b.eupspkg: Command:
>>>>> source /Users/dawson/lsst/eups/bin/setups.sh; export EUPS_PATH=/Users/dawson/lsst; (/Users/dawson/lsst/EupsBuildDir/DarwinX86/base-10.0-5-g766159b/build.sh) >> /Users/dawson/lsst/EupsBuildDir/DarwinX86/base-10.0-5-g766159b/build.log 2>&1 4>/Users/dawson/lsst/EupsBuildDir/DarwinX86/base-10.0-5-g766159b/build.msg
>>>>> exited with code 252
>>>>>
>>>>> I am burnt out on switching between different distributions (in part motivated by dm stack and galsim I recently switched from macports to homebrew and canopy to anaconda) and fixing the things that will inevitably break so I don’t feel like trying the CernVMFS distribution. I will just wait for v10_1.
>>>>>
>>>>> Will
>>>>>
>>>>>> On May 7, 2015, at 2:33 PM, Dominique Boutigny <boutigny at in2p3.fr> wrote:
>>>>>>
>>>>>> Hi Will,
>>>>>>
>>>>>> As far as I know, v10_1 will be released in a couple of days so I don't think that you should invest too much time in v10_0. In the mean time you can use v10_1_rc3 which is very close to v10_1, either by installing it directly (https://confluence.lsstcorp.org/display/LSWUG/LSST+Software+User+Guide) or even simpler using the CernVMFS distrib : see https://github.com/airnandez/lsst-cvmfs
>>>>>> With the later, once CernVMFS is installed you have nothing to build, the packages are loaded on demand from the network, so this takes a little bit of time the first time, but then it is cached on your local disk and it is as fast as a local install.
>>>>>>
>>>>>> Dominique
>>>>>>
>>>>>> On 05/07/2015 01:35 PM, Will Dawson wrote:
>>>>>>> Dominique,
>>>>>>>
>>>>>>> Thank you for translating the error. The problem then seems to be with an inconsistent DM functions default parameter settings. From the notebook the schema and measurement configuration is set by
>>>>>>>
>>>>>>> schema = lsst.afw.table.SourceTable.makeMinimalSchema()
>>>>>>> measureSourcesConfig = lsst.meas.algorithms.SourceMeasurementConfig()
>>>>>>>
>>>>>>> So it appears that makeMinimialSchema() and SourceMeasurementConfig() need to be made compatible (perhaps is has been in v10_1). Should I be using v10_1 instead of v10_0?
>>>>>>>
>>>>>>> Thank you for the very helpful CFHTLS references.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Will
>>>>>>>
>>>>>>>> On May 7, 2015, at 12:09 PM, Dominique Boutigny <boutigny at in2p3.fr> wrote:
>>>>>>>>
>>>>>>>> Hi Will,
>>>>>>>>
>>>>>>>> Yes, the schema has changed in the latest stack versions and "flux.sinc_flux" is not any more in the source table schema. Something directly replacing sinc_flux was present in the v10_0 but this has been completely replaced in the latest stack_v10_1. There are now 10 Aperture flux corresponding to 10 different radii.
>>>>>>>>
>>>>>>>> You can have a look to http://nbviewer.ipython.org/github/DarkEnergyScienceCollaboration/CFHTLS-Reprocess/tree/master/notebooks/ to get notebook examples with the most recent stack.
>>>>>>>>
>>>>>>>> I can have a look to Chris' notebooks if you wish.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Dominique
>>>>>>>>
>>>>>>>> On 05/07/2015 11:52 AM, Will Dawson wrote:
>>>>>>>>> I encountered an error when running Chris Walter’s Introduction to the DM AFW packages ipython notebook. In the second to last step when I run the command:
>>>>>>>>>
>>>>>>>>> # Apply the measurement routines to the exposure using the sources as input
>>>>>>>>> measure.run(exposure, sources)
>>>>>>>>>
>>>>>>>>> I get the following error:
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>>> NotFoundError
>>>>>>>>> Traceback (most recent call last)
>>>>>>>>>
>>>>>>>>> <ipython-input-19-77c6eca8bda0> in <module>()
>>>>>>>>> 16
>>>>>>>>> 17 # Apply the measurement routines to the exposure using the sources as input
>>>>>>>>> ---> 18 measure.run(exposure, sources)
>>>>>>>>> 19
>>>>>>>>> 20 # Now let's look at the output from some of the measurment algorithms.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /Users/dawson/lsst/DarwinX86/pipe_base/10.0+2/python/lsst/pipe/base/timer.pyc in wrapper(self, *args, **keyArgs)
>>>>>>>>> 116 logInfo(obj = self, prefix = func.__name__ + "Start")
>>>>>>>>> 117 try:
>>>>>>>>> --> 118 res = func(self, *args, **keyArgs)
>>>>>>>>> 119 finally:
>>>>>>>>> 120 logInfo(obj = self, prefix = func.__name__ + "End")
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /Users/dawson/lsst/DarwinX86/meas_algorithms/10.0+2/python/lsst/meas/algorithms/measurement.pyc in measure(self, exposure, sources, noiseImage, noiseMeanVar, references, refWcs)
>>>>>>>>> 422
>>>>>>>>> 423 self.log.info("Measuring %d sources" % len(sources))
>>>>>>>>> --> 424 self.config.slots.setupTable(sources.table, prefix=self.config.prefix)
>>>>>>>>> 425
>>>>>>>>> 426 self.preMeasureHook(exposure, sources)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /Users/dawson/lsst/DarwinX86/meas_algorithms/10.0+2/python/lsst/meas/algorithms/measurement.pyc in setupTable(self, table, prefix)
>>>>>>>>> 57 if self.centroid is not None: table.defineCentroid(prefix + self.centroid)
>>>>>>>>> 58 if self.shape is not None: table.defineShape(prefix + self.shape)
>>>>>>>>> ---> 59 if self.apFlux is not None: table.defineApFlux(prefix + self.apFlux)
>>>>>>>>> 60 if self.modelFlux is not None: table.defineModelFlux(prefix + self.modelFlux)
>>>>>>>>> 61 if self.psfFlux is not None: table.definePsfFlux(prefix + self.psfFlux)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /Users/dawson/lsst/DarwinX86/afw/10.0+3/python/lsst/afw/table/tableLib.pyc in defineApFlux(self, *args)
>>>>>>>>> 9869 def defineApFlux(self, *args):
>>>>>>>>> 9870 """defineApFlux(SourceTable self, std::string const & name)"""
>>>>>>>>> -> 9871 return _tableLib.SourceTable_defineApFlux(self, *args)
>>>>>>>>> 9872
>>>>>>>>> 9873 def getApFluxDefinition(self):
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> NotFoundError
>>>>>>>>> :
>>>>>>>>> File "src/table/
>>>>>>>>> Schema.cc
>>>>>>>>> ", line 239, in SchemaItem<T> lsst::afw::table::detail::SchemaImpl::find(const std::string &) const [T = double]
>>>>>>>>> Field or subfield withname 'flux.sinc_flux' not found with type 'D'. {0}
>>>>>>>>> lsst::pex::exceptions::NotFoundError: 'Field or subfield withname 'flux.sinc_flux' not found with type 'D'.'
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Perhaps this is related to the fact that the notebook was written in 2013 and I am using v10.0 of the stack. As I am new to the stack, and given the surprisingly complex error message for such a seemingly simple command, I am hoping that the experts can shed some light on the matter.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Will Dawson
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> dm-users mailing list
>>>>>>>>>
>>>>>>>>> dm-users at lists.lsst.org
>>>>>>>>> https://lists.lsst.org/mailman/listinfo/dm-users
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dominique Boutigny - CNRS / CC-IN2P3
>>>>>>>> Now at SLAC National Accelerator Laboratory
>>>>>>>>
>>>>>>>> Kavli Building - Room 221
>>>>>>>> Mail :
>>>>>>>> boutigny at in2p3.fr - boutigny at slac.stanford.edu
>>>>>>>>
>>>>>>>> Office :
>>>>>>>> +1 650-926-5759 - Cellular : +1 774-232-0912
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> dm-users mailing list
>>>>>>>> dm-users at lists.lsst.org
>>>>>>>> https://lists.lsst.org/mailman/listinfo/dm-users
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dominique Boutigny - CNRS / CC-IN2P3
>>>>>> Now at SLAC National Accelerator Laboratory
>>>>>>
>>>>>> Kavli Building - Room 221
>>>>>> Mail :
>>>>>> boutigny at in2p3.fr - boutigny at slac.stanford.edu
>>>>>>
>>>>>> Office :
>>>>>> +1 650-926-5759 - Cellular : +1 774-232-0912
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> dm-users mailing list
>>>>>> dm-users at lists.lsst.org
>>>>>> https://lists.lsst.org/mailman/listinfo/dm-users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> dm-users mailing list
>>>>> dm-users at lists.lsst.org
>>>>> https://lists.lsst.org/mailman/listinfo/dm-users
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> K. Simon Krughoff
>>>>> University of Washington
>>>>> Department of Astronomy
>>>>>
>>>>> Physics-Astronomy Bldg.
>>>>> 3910 15th Ave NE
>>>>> Room C319
>>>>> Seattle WA 98195-0002
>>>>> USA
>>>>>
>>>>> (206)543-9487
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> K. Simon Krughoff
>>>> University of Washington
>>>> Department of Astronomy
>>>>
>>>> Physics-Astronomy Bldg.
>>>> 3910 15th Ave NE
>>>> Room C319
>>>> Seattle WA 98195-0002
>>>> USA
>>>>
>>>> (206)543-9487
>>>
>>>
>>>
>>>
>>>
>>> --
>>> K. Simon Krughoff
>>> University of Washington
>>> Department of Astronomy
>>>
>>> Physics-Astronomy Bldg.
>>> 3910 15th Ave NE
>>> Room C319
>>> Seattle WA 98195-0002
>>> USA
>>>
>>> (206)543-9487
>>
>>
>>
>>
>>
>> --
>> K. Simon Krughoff
>> University of Washington
>> Department of Astronomy
>>
>> Physics-Astronomy Bldg.
>> 3910 15th Ave NE
>> Room C319
>> Seattle WA 98195-0002
>> USA
>>
>> (206)543-9487
>> _______________________________________________
>> dm-users mailing list
>> dm-users at lists.lsst.org
>> https://lists.lsst.org/mailman/listinfo/dm-users
>
More information about the dm-users
mailing list