[lsst-dm-stack-users] How to calculate RMS, sigma-clipped RMS on DM Winter2013 Stripe82 forced photometry

[Kian-Tat Lim]

Michael,

> I'd like to calculate the RMS, the chisq, and a sigma-clipped RMS on
> the DM Winter2013 Stripe 82 forced photometry

	Do you mean across all the measurements for each object?  Or
across the entire dataset?  In either case, a full table scan on
DeepForcedSource will take some time, but it's not impossible.

> But what would be the most efficient way to calculate the above
> aggregate quantities?

	Generally, GROUP BY, if necessary, and aggregate functions.

> Should I implement a cursor in my SQL query (easy to write, probably
> painfully slow to run),

	You mean as a stored procedure?  It might not be that slow,
actually, but it's probably better to avoid it.

> is there some support for such aggregate functions in the MySQL server
> (I checked and STDEV wasn't available),

	MySQL spells it "STD()" or "STDDEV_POP()" or "STDDEV_SAMP()" or
"STDDEV()" (with all but "_SAMP()" being population).  You also have the
other standard aggregation functions.  For a sigma-clipped RMS, you'll
probably have to do two passes (as you would with a program).

> or should I talk to you about efficient ways to reprocess the raw
> files that fed these tables?

	I'd at least like to see that querying the database is too
complex or too slow before going to the raw files (which still need to
be joined).

-- 
Kian-Tat Lim, LSST Data Management, ktl at slac.stanford.edu