Skip to content

Commit 0b7098d

Browse files
rileythairosteen
rileythai
and
rosteen
authored
add: new SDSS V datatype loaders (#1107)
* add: sdss_v.py working on loaders * add: test.py for testing things, fix ordered method for apVisit * feat: apStar/apVisit functionality, new helper funcs new helper funcs: - _fetch_metadata to perform grab of common metadata - _fetch_flux_unit to get flux unit from the given HDU and convert it to an astropy unit object * add: astra_readers.py pulled from https://raw.githubusercontent.com/sdss/astra/298a73ce600db428cf0a2ed8a707a56c2182ae57/python/astra/tools/spectrum/readers.py * feat: BOSS specFull loaders can fully load boss files now other changes: - commented out abstract @ things to make it so ipython autoreloads notes: - flux unit where for BOSS files? - what the heck are spAll, zAll, and zList HDUs? - does the InverseVariance need a unit? * feat: mwmVisit, mwmStar loaders added mwmVisit and mwmStar loaders. updated demonstration notebook accordingly and output to PDF del: test.pdf and test.ipynb del: secret SDSS-V data * feat: specLite and other BOSS REDUX loader functionality able to now load all BOSS spec directly with the same underlying code. required refactoring methods into BOSS_spec loaders * chore: identifier + documentation * add: test_implementation.py going to now write implementation test add * feat: partial implementation of loaders mwm confirmed working still todo: - add HDU not-specified message - merge the mwm types into a single 2 loaders - confirm all other loaders work and add to __all__ * feat: functioning loaders (+refactoring & chore: docs) - refactored BOSS spec methods and mwm spec methods into single functions for simplicity - all loaders WORKING!! (except apStar multi) - all the documentation + type hinting (excluding outputs) - changed variable names to standard types used in specutils - TODO: the apStar multi-loader is confusing, so it remains unimplemented for now. - CHECK: do I need to clean the files of zero vals? - TODO: BUNIT pulls for spec and mwm files - TODO: check with data team what mwm files are needed * add: test_implementation jupyter notebook - currently non-functional because of zero values in x-axis - deleted test_implementation.ipynb for policy reasons * fix: jdaviz nan and zero flux bug - jdaviz hates nan and zero flux, so they have to be removed - TODO: open issue on jdaviz repo about nan and zero flux bug the bug originates in the x_min x_max values used for the redshift slider for line_lists (somehow) on nan and zero flux values in the Spectrum object. * feat: all multiloaders functional apStar loader not yet tested because file is of length 1 (no visits) mwm loaders will SKIP any DATASUM=0 because Spectrum1D cannot be instantiated with zero data * fix: Astropy units warning + warning format -> print * ignore: demonstration and test files * fix: jdaviz specviz 1D in 2D array handling fixes a jdaviz issue regarding a 1D flux in a 2D object, where it gets confused and explodes i will put an issue in for it this fix is different from the previous as it keeps all zero and NaN flux points * fix: header method -> .get() + other minor fixes * feat: unit tests on dummy data (excl. MWM) need someone to help me write a BinTableHDU for mwm files... * feat: unit tests with assertions still need to write mwm dummy file for the tests there's also a foobar variable check for the metadata * chore: docustrings * fix: header fetch method -> specutils standard now obtains header from PrimaryHDU in the HDUList, any data that was previously accessed through it has been removed too * del: mpl_preamble.py * feat: individual identifiers + unit tests update * fix: .gitignore list keeping .jukit incase anyone else uses vim-jukit during dev * add: bitmasks to Spectrum object outputs Spectrum1D intializer converts any 0 to valid values. I'm assuming that zeroes in the bitmask means that its valid, as per manga.py * fix: spec mask, AND_MASK -> OR_MASK fix as per @Sean-Morrison 's suggestion in astropy pull req [#1107](#1107) could be reverted in future, in which case this commit can just be deleted * fix: spec file identify OBSERVAT column OBSERVAT column not in everything so i changed it, also adding another LOGLAM check to the coadd HDU check. * fix: hdu spec -> find 1st hdu with data instead of specifying a hdu on Spectrum1D loaders for spec and mwm types, it will not find the first HDU with data, or in the case of spec, just use the coadd. this means that it works directly with jdaviz for those two datatypes correctly now. there are no user facing methods, and I don't want to break anything, but it should be noted that these datafiles can contain several spectra, which inherently limits this. in theory, I could put everything as a Spectrum1D nD flux object, but I'm pretty sure that breaks sometimes for jdaviz. * add: mwm dummy file tests + mask fixes - force masks to be boolean prior to entering initializer - add mwm file tests based on dummy file (credit to @andycasey for those dummy file generators) - add more mwm file tests for failures - added checks to see if file is empty for mwm files based on datasum (failsafe) * Fix codestyle errors --------- Co-authored-by: Ricky O'Steen <[email protected]>
1 parent c646007 commit 0b7098d

File tree

4 files changed

+1401
-49
lines changed

4 files changed

+1401
-49
lines changed

‎.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,4 +66,9 @@ pip-wheel-metadata/
6666
# Mac OSX
6767
.DS_Store
6868
.vscode
69+
70+
# Other
6971
.pytest_cache
72+
.jukit
73+
venv
74+
.venv

‎specutils/io/default_loaders/sdss.py

Lines changed: 67 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -47,10 +47,10 @@ def _sdss_wcs_to_log_wcs(old_wcs):
4747
This function does the conversion from the SDSS WCS to FITS WCS.
4848
"""
4949
w0 = old_wcs.wcs.crval[0]
50-
w1 = old_wcs.wcs.cd[0,0]
51-
crval = 10 ** w0
50+
w1 = old_wcs.wcs.cd[0, 0]
51+
crval = 10**w0
5252
cdelt = crval * w1 * np.log(10)
53-
cunit = Unit('Angstrom')
53+
cunit = Unit("Angstrom")
5454
ctype = "WAVE-LOG"
5555

5656
w = WCS(naxis=1)
@@ -70,11 +70,11 @@ def spec_identify(origin, *args, **kwargs):
7070
"""
7171
# Test if fits has extension of type BinTable and check for spec-specific keys
7272
with read_fileobj_or_hdulist(*args, **kwargs) as hdulist:
73-
return (hdulist[0].header.get('TELESCOP') == 'SDSS 2.5-M' and
74-
hdulist[0].header.get('FIBERID', 0) > 0 and
75-
len(hdulist) > 1 and
76-
(isinstance(hdulist[1], fits.BinTableHDU) and
77-
hdulist[1].header.get('TTYPE3').lower() == 'ivar'))
73+
return (hdulist[0].header.get("TELESCOP") == "SDSS 2.5-M"
74+
and hdulist[0].header.get("FIBERID", 0) > 0
75+
and len(hdulist) > 1
76+
and (isinstance(hdulist[1], fits.BinTableHDU)
77+
and hdulist[1].header.get("TTYPE3").lower() == "ivar"))
7878

7979

8080
def spSpec_identify(origin, *args, **kwargs):
@@ -85,10 +85,10 @@ def spSpec_identify(origin, *args, **kwargs):
8585
# Test telescope keyword and check if primary HDU contains data
8686
# consistent with spSpec format
8787
with read_fileobj_or_hdulist(*args, **kwargs) as hdulist:
88-
return (hdulist[0].header.get('TELESCOP') == 'SDSS 2.5-M' and
89-
hdulist[0].header.get('FIBERID', 0) > 0 and
90-
isinstance(hdulist[0].data, np.ndarray) and
91-
hdulist[0].data.shape[0] == 5)
88+
return (hdulist[0].header.get("TELESCOP") == "SDSS 2.5-M"
89+
and hdulist[0].header.get("FIBERID", 0) > 0
90+
and isinstance(hdulist[0].data, np.ndarray)
91+
and hdulist[0].data.shape[0] == 5)
9292

9393

9494
def spPlate_identify(origin, *args, **kwargs):
@@ -99,14 +99,16 @@ def spPlate_identify(origin, *args, **kwargs):
9999
# Test telescope keyword and check if primary HDU contains data
100100
# consistent with spSpec format
101101
with read_fileobj_or_hdulist(*args, **kwargs) as hdulist:
102-
return (hdulist[0].header.get('TELESCOP') == 'SDSS 2.5-M' and
103-
hdulist[0].header.get('FIBERID', 0) <= 0 and
104-
isinstance(hdulist[0].data, np.ndarray) and
105-
hdulist[0].data.shape[0] > 5)
102+
return (hdulist[0].header.get("TELESCOP") == "SDSS 2.5-M"
103+
and hdulist[0].header.get("FIBERID", 0) <= 0
104+
and isinstance(hdulist[0].data, np.ndarray)
105+
and hdulist[0].data.shape[0] > 5)
106106

107107

108108
@data_loader(
109-
label="SDSS-III/IV spec", identifier=spec_identify, extensions=['fits'],
109+
label="SDSS-III/IV spec",
110+
identifier=spec_identify,
111+
extensions=["fits"],
110112
priority=10,
111113
)
112114
def spec_loader(file_obj, **kwargs):
@@ -127,30 +129,37 @@ def spec_loader(file_obj, **kwargs):
127129
"""
128130
with read_fileobj_or_hdulist(file_obj, **kwargs) as hdulist:
129131
header = hdulist[0].header
130-
meta = {'header': header}
132+
meta = {"header": header}
131133

132-
bunit = header.get('BUNIT', '1e-17 erg / (Angstrom cm2 s)')
133-
if 'Ang' in bunit and 'strom' not in bunit:
134-
bunit = bunit.replace('Ang', 'Angstrom')
134+
bunit = header.get("BUNIT", "1e-17 erg / (Angstrom cm2 s)")
135+
if "Ang" in bunit and "strom" not in bunit:
136+
bunit = bunit.replace("Ang", "Angstrom")
135137
flux_unit = Unit(bunit)
136138

137139
# spectrum is in HDU 1
138-
flux = hdulist[1].data['flux'] * flux_unit
140+
flux = hdulist[1].data["flux"] * flux_unit
139141

140-
uncertainty = InverseVariance(hdulist[1].data['ivar'] / flux_unit**2)
142+
uncertainty = InverseVariance(hdulist[1].data["ivar"] / flux_unit**2)
141143

142-
dispersion = 10**hdulist[1].data['loglam']
143-
dispersion_unit = Unit('Angstrom')
144+
dispersion = 10**hdulist[1].data["loglam"]
145+
dispersion_unit = Unit("Angstrom")
144146

145-
mask = hdulist[1].data['and_mask'] != 0
147+
mask = hdulist[1].data["and_mask"] != 0
146148

147-
return Spectrum1D(flux=flux, spectral_axis=dispersion * dispersion_unit,
148-
uncertainty=uncertainty, meta=meta, mask=mask)
149+
return Spectrum1D(
150+
flux=flux,
151+
spectral_axis=dispersion * dispersion_unit,
152+
uncertainty=uncertainty,
153+
meta=meta,
154+
mask=mask,
155+
)
149156

150157

151158
@data_loader(
152-
label="SDSS-I/II spSpec", identifier=spSpec_identify,
153-
extensions=['fit', 'fits'], priority=10,
159+
label="SDSS-I/II spSpec",
160+
identifier=spSpec_identify,
161+
extensions=["fit", "fits"],
162+
priority=10,
154163
)
155164
def spSpec_loader(file_obj, **kwargs):
156165
"""
@@ -176,32 +185,37 @@ def spSpec_loader(file_obj, **kwargs):
176185
with read_fileobj_or_hdulist(file_obj, **kwargs) as hdulist:
177186
header = hdulist[0].header
178187
# name = header.get('NAME')
179-
meta = {'header': header}
188+
meta = {"header": header}
180189
wcs = WCS(header).dropaxis(1)
181190

182-
bunit = header.get('BUNIT', '1e-17 erg / (Angstrom cm2 s)')
191+
bunit = header.get("BUNIT", "1e-17 erg / (Angstrom cm2 s)")
183192
# fix mutilated flux unit
184-
bunit = bunit.replace('/cm/s/Ang', '/ (Angstrom cm2 s)')
185-
if 'Ang' in bunit and 'strom' not in bunit:
186-
bunit = bunit.replace('Ang', 'Angstrom')
193+
bunit = bunit.replace("/cm/s/Ang", "/ (Angstrom cm2 s)")
194+
if "Ang" in bunit and "strom" not in bunit:
195+
bunit = bunit.replace("Ang", "Angstrom")
187196
flux_unit = Unit(bunit)
188197
flux = hdulist[0].data[0, :] * flux_unit
189198

190199
uncertainty = StdDevUncertainty(hdulist[0].data[2, :] * flux_unit)
191200

192201
# Fix the WCS if it is claimed to be linear
193-
if header.get('DC-Flag', 1) == 1:
202+
if header.get("DC-Flag", 1) == 1:
194203
fixed_wcs = _sdss_wcs_to_log_wcs(wcs)
195204
else:
196205
fixed_wcs = wcs
197206

198207
mask = hdulist[0].data[3, :] != 0
199208

200-
return Spectrum1D(flux=flux, wcs=fixed_wcs,
201-
uncertainty=uncertainty, meta=meta, mask=mask)
209+
return Spectrum1D(flux=flux,
210+
wcs=fixed_wcs,
211+
uncertainty=uncertainty,
212+
meta=meta,
213+
mask=mask)
202214

203215

204-
@data_loader(label="SDSS spPlate", identifier=spPlate_identify, extensions=['fits'])
216+
@data_loader(label="SDSS spPlate",
217+
identifier=spPlate_identify,
218+
extensions=["fits"])
205219
def spPlate_loader(file_obj, limit=None, **kwargs):
206220
"""
207221
Loader for SDSS spPlate files, reading flux spectra from all fibres into single array.
@@ -223,26 +237,30 @@ def spPlate_loader(file_obj, limit=None, **kwargs):
223237
"""
224238
with read_fileobj_or_hdulist(file_obj, **kwargs) as hdulist:
225239
header = hdulist[0].header
226-
meta = {'header': header}
240+
meta = {"header": header}
227241
wcs = WCS(header).dropaxis(1)
228242
if limit is None:
229-
limit = header['NAXIS2']
243+
limit = header["NAXIS2"]
230244

231-
bunit = header.get('BUNIT', '1e-17 erg / (Angstrom cm2 s)')
232-
if 'Ang' in bunit and 'strom' not in bunit:
233-
bunit = bunit.replace('Ang', 'Angstrom')
245+
bunit = header.get("BUNIT", "1e-17 erg / (Angstrom cm2 s)")
246+
if "Ang" in bunit and "strom" not in bunit:
247+
bunit = bunit.replace("Ang", "Angstrom")
234248
flux_unit = Unit(bunit)
235249
flux = hdulist[0].data[0:limit, :] * flux_unit
236-
uncertainty = InverseVariance(hdulist[1].data[0:limit, :] / flux_unit**2)
250+
uncertainty = InverseVariance(hdulist[1].data[0:limit, :] /
251+
flux_unit**2)
237252

238253
# Fix the WCS if it is claimed to be linear
239-
if header.get('DC-Flag', 1) == 1:
254+
if header.get("DC-Flag", 1) == 1:
240255
fixed_wcs = _sdss_wcs_to_log_wcs(wcs)
241256
else:
242257
fixed_wcs = wcs
243258

244259
mask = hdulist[2].data[0:limit, :] != 0
245-
meta['plugmap'] = Table.read(hdulist[5])[0:limit]
260+
meta["plugmap"] = Table.read(hdulist[5])[0:limit]
246261

247-
return Spectrum1D(flux=flux, wcs=fixed_wcs,
248-
uncertainty=uncertainty, meta=meta, mask=mask)
262+
return Spectrum1D(flux=flux,
263+
wcs=fixed_wcs,
264+
uncertainty=uncertainty,
265+
meta=meta,
266+
mask=mask)

0 commit comments

Comments
 (0)