PREOPS-5245: Add support for reading visits from ConsDB #80

ehneilsen · 2024-08-06T17:37:42Z

There is (or will be shortly) a corresponding pull request in schedview.

rhiannonlynne

Looks fine. I was wondering about the benefit of making the consdb values cached quantities, instead of calculating everything.
And if it would be good to allow a range of dates instead of a single dayobs.
If there are things missing from the consdb, we should also be moving to get them included.

rhiannonlynne · 2024-08-07T00:04:36Z

rubin_scheduler/utils/consdb.py

+import numpy as np
+import pandas as pd
+
+# import rubin_sim


Remove this commented out line, please.

rhiannonlynne · 2024-08-07T00:16:19Z

rubin_scheduler/utils/consdb.py

+GAUSSIAN_FWHM_OVER_SIGMA: float = 2.0 * sqrt(2.0 * log(2.0))
+
+
+def query_consdb(query: str, url: str = "postgresql://[email protected]:5432/exposurelog"):


Merging this as-is is fine, but I don't think accessing via Postgres is probably the best option long-term. (maybe via TAP would be better, so that this would be available anywhere not just on the RSP)

I'm guessing that's why this is pulled out separately.

rhiannonlynne · 2024-08-07T00:22:35Z

rubin_scheduler/utils/consdb.py

+        raise NotImplementedError
+
+    @abstractmethod
+    def instrumental_zeropoint_for_band(self, band: str) -> float:


I don't understand instrumental zero point for band. Is this from the consdb, as is reported per visit? While there can be an instrumental zero point per band, I think the values reported in the consdb are per visit and represent -- this is the magnitude that a point source would have to produce 1 adu (per second? per exposure? dm have said per exposure, but are moving toward per second I think) in that particular visit. Or is this representing the calculated instrumental zero point for the particular bandpass at a given airmass, as we calculate it with rubin_sim?

As above - I see how these are being used later, but am not sure if this is how we should be doing this now.

Or is this representing the calculated instrumental zero point for the particular bandpass at a given airmass, as we calculate it with rubin_sim?

If it's "at a given airmass," the it isn't an instrumental zero point, but an overall zeropoint: what "instrumental" is supposed to indicate is that it includes only contributions from the instrument itself, and not the atmosphere. The overall zeropoint is in a different property (zeropoint_e). We want both.

I had thought "instrumental zero point" reflected the zero point from the particular instrument configuration, for a particular visit -- and thus did include atmosphere. Isn't that what you're looking for here though, given that it's used with only the "shut_time" to scale the zero point to the zero point for the exposure? Although, I guess I don't know what you're doing with the resulting value scaled to the exposure time afterwards, I'd assumed it was a comparison to the zero point measured for the exposure from DM.

It is not used for the zero point for the exposure. That's zeropoint_e, which reads its value from zero_point_median in the consdb.
instrumental_zero_point is only used for as the zero point for converting sky magnitude from counts in the consdb to mag/asec^2. As I understand it from Alex, one of the issues he originally had matching sky brightness is that the zero point measured from the consdb incorporated extinction, but the sky brightness magnitudes should not include this, as they do not arise from the top of the atmosphere.

If I had a value for the extinction, I could instrumental_zero_point from airmass, extinction, and zero point, and that would be better. But, there's no value for the extinction in consdb.

I'm getting a little confused about the various zero points floating around - let's consolidate to a comment in the "Conversation"? -- (asked on slack instead).

rhiannonlynne · 2024-08-07T00:25:25Z

rubin_scheduler/utils/consdb.py

+
+    @property
+    @abstractmethod
+    def gain(self) -> pd.Series:


I'm not sure I understand this either .. but maybe I'm not understanding the point of this class. I thought it was maybe to indicate which of the databases to access and something about the schema. The gain can change in different images (I don't think it does in practice, but it could .. and the reported gain is an average over the different amplifiers as well). Isn't this a value in the consdb per visit or exposure? (It might only be in the per-amp tables though, in which case we should get it rolled up as an average into the exposure table).

Nevermind, I see how this and some of the following properties are being used.
I suspect there is some thinking to be done about what to use from DM measurements (and then access them in the consdb) vs. what to use "from predictions".

Gain is not available in the consdb. If it becomes available, it will be easy to update the class.
For some of what we need, this needs to work properly before DM processes the data (but after quicklook). We may want to have separate subclasses for when the "real" DM outputs are available, and when only quicklook ones are.

Prompt processing should also be very fast, so ought to be also available. I think there is currently a question running around about how to keep parallel but not identical outputs between RA and PP.

rhiannonlynne · 2024-08-07T00:26:37Z

rubin_scheduler/utils/consdb.py

+
+    @property
+    @abstractmethod
+    def pixel_scale(self) -> pd.Series:


I think maybe I'm starting to understand why this, gain and zero point is specified here .. are these attempts to hard-code these values so that they are available for converting values to opsim equivalents? They should all be present in the exposure table instead, even if they are averages over CCD or amp.

As above - I see how these are being used later, but am not sure if this is how we should be doing this now.

Yes, at present they are hard-coded because they are not in consdb. I separated them out into their own properties rather than just using constants to make it easy to update the class if they ever become available, or even to support a mix of instruments for which they are available for some and not others.

rhiannonlynne · 2024-08-07T00:37:43Z

rubin_scheduler/utils/consdb.py

+        zp : `pd.Series`
+            Instrumental zero point for each visit, with "counts" in electons.
+        """
+        return self.band.apply(self.instrumental_zeropoint_for_band) + 2.5 * np.log10(self.shut_time)


Ooh ok, so maybe I see what the instrumental zero point above was for.
If the point of this is to give a "predicted zero point" to compare to, let's call it that. Instrumental zero point should come from the instrument/pipelines. Also, we can calculate a better value than just one per band, as we could add an airmass correction term.
Also, is there any reason that both this and the DM zero point have to be per visit, and not per second as is much more rational for zeropoints?

No, it's not the "predicted zero point", but the zero point due to the instrument, excluding factors outside the instrument (e.g. atmospheric extinction). It's only "predicted" because I have now way to get at "measured."

Also, is there any reason that both this and the DM zero point have to be per visit

The zero points are per visit because that's how they are reported in the consdb (as near as I could figure out), and I didn't want to introduce an unnecessary inconsistency.

(seems like this is used just to calculate the skybrightness in mag/sq arc second?)

rhiannonlynne · 2024-08-07T00:38:23Z

rubin_scheduler/utils/consdb.py

+        sky : `pd.Series`
+            Median sky background in each visit, in mags asec^-2
+        """
+        return self.instrumental_zeropoint_e - 2.5 * np.log10(self.sky_e_per_pixel / (self.pixel_scale**2))


Why not use the DM zero points here, and instead of our predicted zero point?

Ok, this was explained above - in order to turn the skybackground into sky mag/arcsecond, the atmospheric contribution should be excluded, so the DM zero points aren't quite right.

rhiannonlynne · 2024-08-07T00:40:47Z

rubin_scheduler/utils/consdb.py

+        Returns
+        -------
+        hpix : `pd.Series`
+            The heaplix of the pointing for each visit, for a given nside.


"The healpixels of the pointing for each visit, for a given nside."
(avoid any confusion about single healpixel vs many heal pixels).
Unless the plural of healpixel is healpix, which I suppose it could be.

I am only returning one healpix value per visit, that of the (central) pointing, so singular. So, it's the healpix of the pointing, not the healpixels of the image footprint.
It's used for getting the pre-computed model sky brightness for comparison with the measured one, and I'm only looking at the median, so just one value.
There might be some merit in looking at the max and min as well (there are columns for these in the consdb), so maybe the full set might be useful. But, I think that's not necessary for the immediate purpose (generating an opsim-like output from the consdb).

rhiannonlynne · 2024-08-07T00:42:37Z

rubin_scheduler/utils/consdb.py

+        fwhm : `pd.Series`
+            FWHM at zenith and 500nm, in arcseconds, at the time of each visit.
+        """
+        return self.consdb_visits["seeing_zenith_500nm_median"] * GAUSSIAN_FWHM_OVER_SIGMA * self.pixel_scale


This might not need to be converted from DIMM value to fwhm.
In the sims, we use the "fwhm500" values in the database .. but I don't know exactly what the DIMM is reporting. I believe "seeing_zenith_500nm_median" would be DIMM? (we should check).

The consdb documentation says:

Measured PSF sigma, corrected to 500nm and an airmass of 1 (median across all detectors).

So, it is sigma. But, not clear if it's arcseconds or pixels.

rhiannonlynne · 2024-08-07T00:59:37Z

rubin_scheduler/utils/consdb.py

+# fine; the user should just instantiate another ConsDBVisits instance if
+# they want a different database or date. But, what if someone does?
+# To avoid this, make it a frozen dataclass, which prevents members from
+# being updated.


Are we not using all of the consdb columns or things calculated from the consdb columns when generating an opsim-like set of outputs?

Yes, we are, and that's the intention. But, I can imagine someone playing in a notebook and manually setting, say, the gain, and being surprised if sky brightness doesn't change accordingly. This is really not what this classes in intended for, so using a construction that just blocks the attempt seems like the right things to do.

ehneilsen requested a review from rhiannonlynne August 6, 2024 17:37

rhiannonlynne approved these changes Aug 7, 2024

View reviewed changes

ehneilsen added 11 commits August 12, 2024 12:53

initial partial implemenation of loading from the consdb

7e9e28e

type annotation fixes

6fff4cf

add sky brightness and psd fwhm

78efd12

add limiting magnitude

3969baf

fix m0 scale

6814b53

starting to add docstrings

66d85e4

fix rot_tel_pos units

ddab6e1

Support returning obs arrays

e2965d7

black

4157b43

Avoid requiring access to consdb for tests

426547f

drop sky_model_per calculation

1fd7294

ehneilsen force-pushed the tickets/PREOPS-5245 branch from d41948b to 1fd7294 Compare August 12, 2024 20:01

black

c0d8d4a

ehneilsen merged commit f035ad3 into main Aug 12, 2024
7 checks passed

rhiannonlynne changed the title ~~tickets/PREOPS-5245: Add support for reading visits from ConsDB~~ PREOPS-5245: Add support for reading visits from ConsDB Sep 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PREOPS-5245: Add support for reading visits from ConsDB #80

PREOPS-5245: Add support for reading visits from ConsDB #80

ehneilsen commented Aug 6, 2024

rhiannonlynne left a comment

rhiannonlynne Aug 7, 2024

rhiannonlynne Aug 7, 2024

rhiannonlynne Aug 7, 2024

rhiannonlynne Aug 7, 2024

rhiannonlynne Aug 7, 2024

ehneilsen Aug 8, 2024

rhiannonlynne Aug 8, 2024

ehneilsen Aug 8, 2024

ehneilsen Aug 8, 2024

rhiannonlynne Aug 8, 2024 •

edited

Loading

rhiannonlynne Aug 7, 2024

rhiannonlynne Aug 7, 2024

ehneilsen Aug 8, 2024

rhiannonlynne Aug 8, 2024

rhiannonlynne Aug 7, 2024

rhiannonlynne Aug 7, 2024

ehneilsen Aug 8, 2024

rhiannonlynne Aug 7, 2024

ehneilsen Aug 8, 2024

rhiannonlynne Aug 8, 2024

rhiannonlynne Aug 7, 2024

rhiannonlynne Aug 8, 2024 •

edited

Loading

rhiannonlynne Aug 7, 2024

ehneilsen Aug 8, 2024

rhiannonlynne Aug 7, 2024

ehneilsen Aug 8, 2024

rhiannonlynne Aug 7, 2024

ehneilsen Aug 8, 2024

		GAUSSIAN_FWHM_OVER_SIGMA: float = 2.0 * sqrt(2.0 * log(2.0))


		def query_consdb(query: str, url: str = "postgresql://[email protected]:5432/exposurelog"):

PREOPS-5245: Add support for reading visits from ConsDB #80

PREOPS-5245: Add support for reading visits from ConsDB #80

Conversation

ehneilsen commented Aug 6, 2024

rhiannonlynne left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhiannonlynne Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhiannonlynne Aug 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhiannonlynne Aug 8, 2024 •

edited

Loading

rhiannonlynne Aug 8, 2024 •

edited

Loading