[lsst-dm-stack-users] How to calculate RMS, sigma-clipped RMS on DM Winter2013 Stripe82 forced photometry

[Michael Wood-Vasey]

Hi KT,

Thanks.  I'll give these a try and report back.

 - Michael

On Jan 8, 2013, at 22:48 , Kian-Tat Lim <ktl at slac.stanford.edu> wrote:

> Michael,
> 
>> I'd like to calculate the RMS, the chisq, and a sigma-clipped RMS on
>> the DM Winter2013 Stripe 82 forced photometry
> 
> 	Do you mean across all the measurements for each object?  Or
> across the entire dataset?  In either case, a full table scan on
> DeepForcedSource will take some time, but it's not impossible.
> 
>> But what would be the most efficient way to calculate the above
>> aggregate quantities?
> 
> 	Generally, GROUP BY, if necessary, and aggregate functions.
> 
>> Should I implement a cursor in my SQL query (easy to write, probably
>> painfully slow to run),
> 
> 	You mean as a stored procedure?  It might not be that slow,
> actually, but it's probably better to avoid it.
> 
>> is there some support for such aggregate functions in the MySQL server
>> (I checked and STDEV wasn't available),
> 
> 	MySQL spells it "STD()" or "STDDEV_POP()" or "STDDEV_SAMP()" or
> "STDDEV()" (with all but "_SAMP()" being population).  You also have the
> other standard aggregation functions.  For a sigma-clipped RMS, you'll
> probably have to do two passes (as you would with a program).
> 
>> or should I talk to you about efficient ways to reprocess the raw
>> files that fed these tables?
> 
> 	I'd at least like to see that querying the database is too
> complex or too slow before going to the raw files (which still need to
> be joined).
> 
> -- 
> Kian-Tat Lim, LSST Data Management, ktl at slac.stanford.edu