[lsst-dm-stack-users] How to calculate RMS, sigma-clipped RMS on DM Winter2013 Stripe82 forced photometry

[Simon Krughoff]

Michael,

To follow up with this, as you know we have already calculated the mean and
inner quartile range for all the lightcurves in the database.  We are
planning on adding a clipped mean and standard deviation within the next
week or so.  It's possible that all you'll have to do is wait.  If there
are other quantities you'd like materialized, we can potentially add those
as well.

If you are going to be at the DESC meeting, we can talk about this in
person.

Simon

On Wed, Jan 9, 2013 at 8:58 AM, Michael Wood-Vasey <wmwv at pitt.edu> wrote:

> Hi KT,
>
> Thanks.  I'll give these a try and report back.
>
>  - Michael
>
> On Jan 8, 2013, at 22:48 , Kian-Tat Lim <ktl at slac.stanford.edu> wrote:
>
> > Michael,
> >
> >> I'd like to calculate the RMS, the chisq, and a sigma-clipped RMS on
> >> the DM Winter2013 Stripe 82 forced photometry
> >
> >       Do you mean across all the measurements for each object?  Or
> > across the entire dataset?  In either case, a full table scan on
> > DeepForcedSource will take some time, but it's not impossible.
> >
> >> But what would be the most efficient way to calculate the above
> >> aggregate quantities?
> >
> >       Generally, GROUP BY, if necessary, and aggregate functions.
> >
> >> Should I implement a cursor in my SQL query (easy to write, probably
> >> painfully slow to run),
> >
> >       You mean as a stored procedure?  It might not be that slow,
> > actually, but it's probably better to avoid it.
> >
> >> is there some support for such aggregate functions in the MySQL server
> >> (I checked and STDEV wasn't available),
> >
> >       MySQL spells it "STD()" or "STDDEV_POP()" or "STDDEV_SAMP()" or
> > "STDDEV()" (with all but "_SAMP()" being population).  You also have the
> > other standard aggregation functions.  For a sigma-clipped RMS, you'll
> > probably have to do two passes (as you would with a program).
> >
> >> or should I talk to you about efficient ways to reprocess the raw
> >> files that fed these tables?
> >
> >       I'd at least like to see that querying the database is too
> > complex or too slow before going to the raw files (which still need to
> > be joined).
> >
> > --
> > Kian-Tat Lim, LSST Data Management, ktl at slac.stanford.edu
>
> _______________________________________________
> lsst-dm-stack-users mailing list
> lsst-dm-stack-users at lsstcorp.org
> http://listserv.lsstcorp.org/mailman/listinfo/lsst-dm-stack-users
>



-- 
K. Simon Krughoff
University of Washington
Department of Astronomy

Physics-Astronomy Bldg.
3910 15th Ave NE
Room C319
Seattle WA 98195-0002
USA

(206)543-9487
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://listserv.lsstcorp.org/pipermail/dm-users/attachments/20130109/5e97b3ff/attachment.html>