Choose visit image IQ window for successful template generation

[drphilmarshall]

@SimonKrughoff Sounded like a minimum IQ of 2.5 pixels might work - keep us posted! When you find a set up that works, I guess we'll need the cookbook updated for @tony-johnson to work off. Good luck!

PS. Ill add this to the Run 3 DM Workflow epic #421

[SimonKrughoff]

Some background. I thought that u-band was not subject to the issue we were seeing. I.e. in r-band:

But it also happens in u-band.

It turns out that there are far fewer images in u-band with IQ in the range 2. - 2.5 pixels.

[SimonKrughoff]

Here is an r-band image with reported FWHM = 2.5:

It seems to be right on the hairy edge. E.g. for another image with FWHM = 2.45:

To be conservative, I'm going to suggest we make the window 2.6 < FWHM < 3.0. Here is an example of an image with FWHM = 2.63:

I am wondering if this has something to do with a minimum size for the PSF basis set.

[drphilmarshall]

Could be. 2.6 pixels seems good for now - will that give us enough frames to make a decent DR1 co-add (for example)? How are you planning to communicate this to the rest of the DM team?

[SimonKrughoff]

I'll file a ticket. The problem is putting together a minimal "how to repeat".

[SimonKrughoff]

And I think it will have enough frames, but I don't have a histogram on hand to show that.

[SimonKrughoff]

O.K. I ran a coadd generation with makeCoaddTempExp.py /global/cscratch1/sd/desc/twinkles/work/6/output --output $SCRATCH/test_out --selectId filter=r --id filter=r --config modelPsf.defaultFwhm=3.00 select.minPsfFwhm=2.60 select.maxPsfFwhm=3.00 --clobber-config --no-versions. That seemed to work pretty well. It produces a coadd with 166 visits (over all 10 years) as opposed to 295 visits.

Here's a cutout:

There are still a few artifacts I don't understand, but we should look at some diffims to see how they look.

I'll update the cookbook later today.

[drphilmarshall]

Splendid! Looks like you've cracked it. For DR1 I guess we can expect 10-20 images. Think that'll be enough? How many years of data should we try to analyze then, before the DESC meeting? At the meeting we should discuss the DR schedule for Twinkles 1.

[SimonKrughoff]

Yes. 10-20 should be plenty. Of course, the fewer images that go into the template, the less sensitivity we have, but we'll still get something useful out.

[SimonKrughoff]

FYI, I didn't get the cookbook out yesterday, because I'm trying to update both at once since they are related.

[drphilmarshall]

No worries. I'm going to wait until I see what they both look like before deciding how to Note-ify them. Could be they just make a pair of Notes.

…

On Fri, Jan 27, 2017 at 9:41 AM, SimonKrughoff ***@***.***> wrote: FYI, I didn't get the cookbook out yesterday, because I'm trying to update both at once since they are related. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#435 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AArY955HiqFKQk8uiAon3xNAqbi-94D8ks5rWixXgaJpZM4Lt8xE> .

[SimonKrughoff]

O.K. so this ended up being a the place where I did a bunch of things. I now have a cookbook that sketches out both level 1 and level 2 processing using appropriate configs for each. There are still some things I'd like to try out: e.g. a different PSF estimation algorithm.

• Cookbook
• Updated obs_lsstSim

I don't know how to review this and get it merged, but my suggestion is that we use this cookbook to do run 3. I also suggest merging twinkles_#435 to twinkles_3 instead of merging it with twinkle_1, but I'm can certainly be talked out of that.

[drphilmarshall]

Great! I see that the diaObjectMaker doesn't take a filter as input - does this mean that it now makes multi-filter objects with a single centroid (as opposed to one centroid per filter)? Have you managed to make any such DIAObjects? I am excited :-)

…

On Fri, Feb 3, 2017 at 10:21 AM, SimonKrughoff ***@***.***> wrote: O.K. so this ended up being a the place where I did a bunch of things. I now have a cookbook that sketches out both level 1 and level 2 processing using appropriate configs for each. There are still some things I'd like to try out: e.g. a different PSF estimation algorithm. - Cookbook <https://github.com/LSSTDESC/Twinkles/blob/Issue%23435/updatecookbook/doc/Cookbook/DiaObjectGen.md> - Updated obs_lsstSim <https://github.com/lsst/obs_lsstSim/tree/twinkles_%23435> I don't know how to review this and get it merged, but my suggestion is that we use this cookbook to do run 3. I also suggest merging twinkles_#435 to twinkles_3 instead of merging it with twinkle_1, but I'm can certainly be talked out of that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#435 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AArY94q31d_GkFoGX9RKCCuT7ybAT6Jfks5rY3BAgaJpZM4Lt8xE> .

[SimonKrughoff]

So the diaObjectMaker.py can take any valid dataId, so you can build per filter DiaObjects, but if you don't specify a filter it will aggregate over all of them.

Just to be clear, I am not doing any sort of averaging over the source properties. That's hard enough to get right that I want to do that in the issue that covers the forced photometry. @wmwv may have ideas about that.

I have made a few single band DiaObjects, but one remaining issue is how to reject the epochs from the DiaObject that went into the coadd.

[drphilmarshall]

What happens if you don't reject them? Could you simply flag them instead of rejecting them, please? I think we are probably all OK with the DIAObjects having just forced photometry at this point. As long as the Monitor can pull out all the DIASources that belong to each DIAObject then we can do some interesting multi-epoch analysis. Also I am pretty sure that we will want multi-filter aggregation as long as the inter-band astrometric registration is good. @jbkalmbach @rbiswas4 @wmwv PLease chip in with comments and feedback for Simon - he is enabling marvellous things here :-)

[SimonKrughoff]

I just updated the cookbook again. I'm considering this a fairly complete list of example command lines except forced phot for diaObjects.

[drphilmarshall]

@SimonKrughoff Looks like the cookbook recipe at https://github.com/LSSTDESC/Twinkles/blob/master/doc/Cookbook/Coadd_Diffim_Recipe.md needs finalizing to reflect the final scheme (including using Year 1 for the template and Years 2-10 for science).

[SimonKrughoff]

O.K. I will try to get to this today.

[rbiswas4]

Since we are looking into this now again to understand what needs to be done.

@SimonKrughoff : Is this still the recommended way to go?

If I understand the last comment right, we need to build the template for coadd using the visits in year 1. Is that correct @drphilmarshall ?
We have those obsHistIDs. Is the way to implement these here in adding the list of obsHistIDs to the makeDiscreteSkyMap flag --id @SimonKrughoff ?

Thanks!