diff --git a/notebooks/COS/AsnFile/AsnFile.ipynb b/notebooks/COS/AsnFile/AsnFile.ipynb index d0b64040..57109cf0 100644 --- a/notebooks/COS/AsnFile/AsnFile.ipynb +++ b/notebooks/COS/AsnFile/AsnFile.ipynb @@ -16,6 +16,10 @@ "**2. [Editing an existing association file](#editAF)**\n", "\n", "\\- 2.1. [Removing an exposure](#subAF)\n", + "\n", + "\\+ 2.1.1. [Removing a bad exposure](#removebadAF)\n", + "\n", + "\\+ 2.1.2. [Filtering to a single exposure (e.g. for LP6 data)](#removefiltAF)\n", " \n", "\\- 2.2. [Adding an exposure](#addAF)\n", " \n", @@ -24,7 +28,12 @@ "\\- 3.1. [Simplest method](#simpleAF)\n", " \n", "\\- 3.2. [With fits header metadata](#metaAF)\n", - " \n" + " \n", + "\\- 3.3. [Association files for non-TAGFLASH datasets](#nontagAF)\n", + "\n", + "\\+ 3.3.1. [Gathering the exposure information](#331-gathering-the-exposure-informationAF)\n", + "\n", + "\\+ 3.3.2. [Creating the `SPLIT` wavecal association file](#332-creating-the-split-wavecal-association-fileAF)\n" ] }, { @@ -39,7 +48,7 @@ "- For an in-depth manual to working with COS data and a discussion of caveats and user tips, see the [COS Data Handbook](https://hst-docs.stsci.edu/display/COSDHB/).\n", "- For a detailed overview of the COS instrument, see the [COS Instrument Handbook](https://hst-docs.stsci.edu/display/COSIHB/).\n", "\n", - "We'll demonstrate creating an asn file in two ways: First, we'll demonstrate editing an existing asn file to add or remove an exposure. Second, we'll show how to create an entirely new asn file." + "We'll demonstrate creating an `asn` file in three ways: First, we'll demonstrate [editing an existing `asn` file to add or remove an exposure](#2-editing-an-existing-association-file). Second, we'll show how to [create an entirely new `asn` file](#newAF). Finally, we'll show an example of [creating an `asn` file for use with SPLIT wavecal data](#332-example-for-split-wavecal-dataAF), such as that obtained at COS lifetime position 6 (LP6)." ] }, { @@ -69,6 +78,9 @@ "# Import for: array manipulation\n", "import numpy as np\n", "\n", + "# Import for: data table manipulatioon\n", + "import pandas as pd\n", + "\n", "# Import for: reading fits files\n", "from astropy.io import fits \n", "from astropy.table import Table\n", @@ -149,7 +161,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Above, we downloaded two association files and their rawtag data files. We will begin by searching for the association files and reading one of them (`LDIF01010`). We could just as easily pick `ldif02010`." + "Above, we downloaded two association files and their rawtag data files. We will begin by searching for the association files and reading one of them (with observation ID `LDIF01010`). We could just as easily pick `ldif02010`." ] }, { @@ -158,7 +170,7 @@ "metadata": {}, "outputs": [], "source": [ - "asnfiles = glob.glob(\"**/*ldif*asn*\", recursive=True) # There will be two (ldif01010_asn.fits and ldif02010_asn.fits)\n", + "asnfiles = sorted(glob.glob(\"**/*ldif*asn*\", recursive=True)) # There will be two (ldif01010_asn.fits and ldif02010_asn.fits)\n", "asnfile = asnfiles[0] # We want to work primarily with ldif01010_asn.fits\n", "\n", "asn_contents = Table.read(asnfile) # Gets the contents of the asn file\n", @@ -169,7 +181,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We see that the association file has five rows: four exposures denoted with the `MEMTYPE` = `EXP-FP`, and a product with `MEMTYPE` = `PROD-FP`.\n", + "The association file has three columns: `MEMNAME`, `MEMTYPE`, `MEMPRSNT` which we describe below.\n", + "\n", + "1. `MEMNAME`: The rootname of the file, e.g. `ldif01u0q` for the file `ldif01u0q_rawtag_a.fits`\n", + "2. `MEMTYPE`: Whether the item is a science exposure (`EXP-FP`), a wavecal exposure (`EXP-SWAVE`/`EXP-GWAVE`/`EXP-AWAVE`), or the output product (`PROD-FP`)\n", + "3. `MEMPRSNT`: Whether to include the file in `CalCOS`' processing\n", + "\n", + "We also see that this association file has five rows: four science exposures denoted with the `MEMTYPE` = `EXP-FP`, and an output product with `MEMTYPE` = `PROD-FP`.\n", "\n", "In the cell below, we examine a bit about each of the exposures as a diagnostic:" ] @@ -196,10 +214,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**We notice that something seems amiss with exposure LDIF01TYQ**:\n", + "**We notice that something seems amiss with the science exposure LDIF01TYQ**:\n", "This file has an exposure time of 0.0 seconds - something has gone wrong. In this case, there was a guide star acquisition failure as described on the [data preview page](http://archive.stsci.edu/cgi-bin/mastpreview?mission=hst&dataid=LDIF01010).\n", "\n", - "In the next section, we will correct this lack of data by replacing the bad exposure with an exposure from the other association group." + "In the next section, we will correct this lack of data by removing the bad exposure and combining in exposures from the other association group." ] }, { @@ -212,7 +230,10 @@ "\n", "## 2.1. Removing an exposure\n", "\n", - "We know that at least one of our exposures - `ldif01tyq` - is not suited for combination into the final product. It has an exposure time of 0.0 seconds, in this case from a guide star acquisition failure. This is a generalizable issue, as you may often know an exposure is \"*bad*\" for many reasons: perhaps it was taken with the shutter closed, or with anomolously high background noise, or any number of reasons we may wish to exclude an exposure from our data. To do this, we will need to alter our existing association file before we re-run `CalCOS`." + "\n", + "#### 2.1.1. Removing a bad exposure\n", + "\n", + "We know that at least one of our exposures - `ldif01tyq` - is not suited for combination into the final product. It has an exposure time of 0.0 seconds, in this case from a guide star acquisition failure. This is a generalizable issue, as you may often know an exposure is \"*bad*\" for many reasons: perhaps it was taken with the shutter closed, or with anomolously high background noise, or any number of reasons we may wish to exclude an exposure from our data. To do this, we will need to alter our existing association file before we re-run `CalCOS`. Afterwards we will compare the flux of the resulting spectrum made without the bad exposure to the same dataset with the bad exposure. The flux levels should match closely." ] }, { @@ -235,7 +256,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can set the `MEMPRSNT` value to `False` or `0` for our bad exposure:" + "We can set the `MEMPRSNT` value to `False` or `0` for our bad exposure. If we were to run the `CalCOS` pipeline on the edited association file, it would not be processed. `CalCOS` can take a long time to run. To avoid slowing down this Notebook, the code to run `CalCOS` on the association files we have created has been spun out into `Python` files in this directory with the names `test_.py` (in this case, `test_removed_exposure_asn.py`). If you wish to re-run the analysis within, you will need to specify a valid cache of CRDS reference files (see our [Setup Notebook](https://github.com/spacetelescope/notebooks/blob/master/notebooks/COS/Setup/Setup.ipynb) for details on downloading reference files) in the `Python` file." ] }, { @@ -249,8 +270,107 @@ " for expfile in tbdata: # Check if each file is one of the bad ones\n", " if expfile['MEMNAME'] in ['LDIF01TYQ']:\n", " expfile['MEMPRSNT'] = False # If so, set MEMPRSNT to False AKA 0\n", - " \n", - "Table.read(asnfile) # Re-read the table to see the change" + "\n", + "# Copy this file, which we will go on to edit, in case we want to run CalCOS on it.\n", + "shutil.copy(asnfile, datadir / \"removed_badfile_asn.fits\")\n", + "# Re-read the table to see the change\n", + "Table.read(asnfile)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To show what would happen if we run `CalCOS` on this association file, we include Figure 1. This figure shows that the flux level of the spectrum retrieved from MAST matches that which was processed after removing the bad exposure. Slight differences in the two spectra may arise from MAST using a different random seed value. What matters for this comparison is that the fluxes match well, as they should because the empty exposure adds 0 counts over 0 time. Thus, for the purposes of flux calibration, it should be ignored. Other types of bad exposures, for instance those taken with the source at the edge of COS's aperture, would impact the fluxes.\n", + "\n", + "### Figure 1. Comparison of fluxes between the data retrieved from MAST, and the same data reprocessed after removing the bad exposure\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "#### 2.1.2. Filtering to a single exposure (e.g. for LP6 data)\n", + "\n", + "Another situation when users often wish to remove exposures from an association file is when re-processing individual exposures taken at COS's lifetime position 6 (LP6) or other modes at which the wavelength calibration data is not contained by the science exposures' `rawtag` files. \n", + "\n", + "Often users wish to process a single exposure's file with `CalCOS`. With the [`TAGFLASH` wavelength calibration method](https://hst-docs.stsci.edu/cosihb/chapter-5-spectroscopy-with-cos/5-7-internal-wavelength-calibration-exposures#id-5.7InternalWavelengthCalibrationExposures-Section5.7.15.7.1ConcurrentWavelengthCalibrationwithTAGFLASH) standard at all lifetime positions prior to LP6, the wavelength calibration data was taken concurrently with the science data and was contained by the same `rawtag` data files. However, as described in this [COS 2030 plan poster](https://aas240-aas.ipostersessions.com/default.aspx?s=06-68-AE-28-CC-4A-12-7A-B0-4F-02-46-BC-D1-BE-7E), this is not possible at LP6. Instead, separate wavelength calibration exposures are taken at separate lifetime positions in a process known as [SPLIT wavecals](https://hst-docs.stsci.edu/cosihb/chapter-5-spectroscopy-with-cos/5-7-internal-wavelength-calibration-exposures#id-5.7InternalWavelengthCalibrationExposures-Section5.7.65.7.6SPLITWavecals(defaultnon-concurrentwavelengthcalibrationatLP6)). As a result, the wavelength calibration data no longer exists in the same file as the science data for such exposures.\n", + "\n", + "For `TAGFLASH` data, users could run `CalCOS` directly on their `rawtag` file of interest (though using an association file was always recommended). However, doing so with LP6 data would not allow the proper wavelength calibration. Instead, users should create a custom association file created by removing exposures from the default association file. We demonstrate this below." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We'll alter an LP6 association file with the observation ID `letc01010`.\n", + "Let's begin by downloading the default LP6 association file from MAST and displaying it:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Search for the correct obs_ids and get the list of data products\n", + "lp6_observation_products = Observations.get_product_list( \n", + " Observations.query_criteria(obs_id='letc01010'),\n", + ")\n", + "\n", + "# Download the asnfile product list, then get the 0th element of the resulting path list, since there's only 1 asn file\n", + "lp6_original_asnfile = Path(\n", + " Observations.download_products(\n", + " Observations.filter_products(\n", + " lp6_observation_products,\n", + " productSubGroupDescription=['ASN'],\n", + " ),\n", + " download_dir=str(datadir)\n", + " )['Local Path'][0]\n", + ")\n", + "\n", + "# Show the default association file for the LP6 dataset.\n", + "Table.read(lp6_original_asnfile)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As expected, we see 4 `EXP-FP` members, which are the science exposures at COS's 4 [fixed-pattern noise positions (`FP-POS`)](https://hst-docs.stsci.edu/cosdhb/chapter-1-cos-overview/1-1-instrument-capabilities-and-design). Each is bracketed by a SPLIT wavecal wavelength calibration exposure on each side (`EXP-SWAVE`).\n", + "\n", + "If we only wish to process one of the science exposures, (along with its wavelength calibration exposures,) we turn off all the other exposures by setting their `MEMPRSNT` values to 0/`False`. The product file (`PROD-FP`) must also be turned on, and should be renamed to show that it will only include data from a single exposure. Below, we do so, assuming we wish to process only the FP3 science exposure: `LETC01MTQ` (and its wavelength calibration exposures: `LETC01M6Q` and `LETC01MVQ`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "exposures_to_use = ['LETC01M6Q','LETC01MTQ','LETC01MVQ'] # WAVECAL -> SCIENCE -> WAVECAL\n", + "print(f\"We will turn off all the input exposures except for: {exposures_to_use}.\" +\\\n", + " \" We will also leave the output product file turned on.\")\n", + "# Make a copy of the original asn file with a name to indicate it will only process a single chosen exposure\n", + "lp6_1exposure_asnfile = shutil.copy(lp6_original_asnfile, datadir / 'letc01mtq_only_asn.fits')\n", + "\n", + "# Turn off all the other exposures in the copy\n", + "with fits.open(lp6_1exposure_asnfile, mode='update') as hdulist:\n", + " tbdata = hdulist[1].data\n", + " for expfile in tbdata:\n", + " # Turn off files except those listed here\n", + " if expfile['MEMNAME'] not in exposures_to_use:\n", + " expfile['MEMPRSNT'] = False # If so, set MEMPRSNT to False AKA 0\n", + " # Turn on and rename the product file to indicate it will only include the chosen exposure\n", + " if expfile['MEMTYPE'] == 'PROD-FP':\n", + " expfile['MEMPRSNT'] = True # Turn on the product file\n", + " expfile['MEMNAME'] = \"LETC01MTQ_only\" # Rename the product file\n", + "\n", + "# Re-read the table to see the change\n", + "Table.read(lp6_1exposure_asnfile)" ] }, { @@ -259,11 +379,11 @@ "source": [ "\n", "## 2.2. Adding an exposure\n", - "We removed the failed exposure taken with `FP-POS = 1`. Usually we want to combine one of each of the four [*fixed-pattern noise positions* (`FP-POS`)](https://hst-docs.stsci.edu/cosdhb/chapter-1-cos-overview/1-1-instrument-capabilities-and-design), so let's add the `FP-POS = 1` exposure from the other association group.\n", + "In section 2.1.1, we removed the failed exposure taken with `FP-POS = 1` from our `ldif01010` association file. Usually we want to combine one or more of each of the four FP-POS. In this case, let's add the FP1 exposure from the other association group. Please note that combining data from separate visits with different target acquisitions can result in true errors which are higher than those calculated by `CalCOS`. Also note that you should only combine files taken using the same grating and central wavelength settings in this manner.\n", "\n", "In the cell below, we determine which exposure from `LDIF02010` was taken with `FP-POS = 1`.\n", "- *It does this by looping through the files listed in `LDIF02010`'s association file, and then reading in that file's header to check if its `FPPOS` value equals 1.*\n", - "- *It also prints some diagnostic information about all of the esxposure files.*" + "- *It also prints some diagnostic information about all of the exposure files.*" ] }, { @@ -292,9 +412,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It's a slightly different procedure to add a new exposure to the list rather than remove one. \n", + "There's a slightly different procedure to add a new exposure to the list rather than remove one. \n", "\n", - "Here we want to read the table in the fits association file into an `astropy` Table. We can then add a row into the right spot, filling it with the new file's `MEMNAME`, `MEMTYPE`, and `MEMPRSNT`. Finally, we have to save this table into the existing fits association file." + "Here we will read the table in the fits association file into an `astropy` Table. We can then add a row into the right spot, filling it with the new file's `MEMNAME`, `MEMTYPE`, and `MEMPRSNT`. Finally, we have to save this table into the existing fits association file." ] }, { @@ -341,6 +461,8 @@ "\n", "# 3. Creating an entirely new association file\n", "\n", + "Users will rarely need to create an entirely new association file. Much more often, they should alter an existing one found on MAST. However, a user may wish to create a new association file if they are combining altered exposures, perhaps after using the [`costools.splittag` tool](https://github.com/spacetelescope/notebooks/blob/master/notebooks/COS/SplitTag/SplitTag.ipynb). This section describes how to make one from scratch.\n", + "\n", "For the sake of demonstration, we will generate a new association file with four exposure members: even-numbered `FP-POS` (2,4) from the first original association (`LDIF01010`), and odd-numbered `FP-POS` (1,3) from from the second original association (`LDIF02010`).\n", "\n", "From section 2, we see that this corresponds to :\n", @@ -359,10 +481,7 @@ "source": [ "\n", "## 3.1. Simplest method\n", - "Below, we manually build up an association file from the three necessary columns:\n", - "1. `MEMNAME`\n", - "2. `MEMTYPE`\n", - "3. `MEMPRSNT`" + "Below, we manually build up an association file from the three necessary columns (`MEMNAME`, `MEMTYPE`, `MEMPRSNT`)." ] }, { @@ -392,7 +511,7 @@ "# Writing the fits table\n", "asn_table.writeto(outputdir / 'ldifcombo_asn.fits', overwrite=True)\n", "\n", - "print('Saved: '+ 'ldifcombo_asn.fits'+ f\" in the output directory: {outputdir}\")" + "print('Saved '+ 'ldifcombo_asn.fits'+ f\" in the output directory: {outputdir}\")" ] }, { @@ -417,7 +536,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "**However, the 0th and 1st fits headers no longer contain useful information about the data:**" + "**However, the 0th and 1st fits headers no longer contain useful information about the data.**\n", + "While `CalCOS` will process these files, vital header keys may not be propagated. If it's important to preserve the header information, you may follow the steps in the next section." ] }, { @@ -469,7 +589,7 @@ "new_HDUlist = fits.HDUList([hdu0,hdu1]) # New HDUList from old HDU 0 and new combined HDU 1\n", "new_HDUlist.writeto(outputdir / 'ldifcombo_2_asn.fits', overwrite=True) # Write this out to a new file\n", "new_asnfile = outputdir / 'ldifcombo_2_asn.fits' # Path to this new file\n", - "print('\\nSaved: '+ 'ldifcombo_2_asn.fits'+ f\"in the output directory: {outputdir}\")" + "print('\\nSaved '+ 'ldifcombo_2_asn.fits'+ f\"in the output directory: {outputdir}\")" ] }, { @@ -520,6 +640,254 @@ "If the test runs successfully, it will create a plot in the subdirectory `./output/plots/` ." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## 3.3. Association files for non-TAGFLASH datasets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As touched upon in [Section 2.1.2.](#removefiltAF), most COS exposures contain the calibration lamp data necessary to perform wavelength calibration; however, exposures taken at LP6 and those taken with [AUTO and GO wavecals](https://hst-docs.stsci.edu/cosihb/chapter-5-spectroscopy-with-cos/5-7-internal-wavelength-calibration-exposures#id-5.7InternalWavelengthCalibrationExposures-Section5.7.25.7.2AUTOWavecals(whenTAGFLASHisnotused)) (including BOA and ACCUM mode exposures) have separate exposures for wavelength calibration. Table 1 below summarizes the types of wavelength calibration modes. More information may be found in [Section 5.7 of the COS Instrument Handbook](https://hst-docs.stsci.edu/cosihb/chapter-5-spectroscopy-with-cos/5-7-internal-wavelength-calibration-exposures).\n", + "\n", + "#### Table 1. Comparison of wavelength calibration modes\n", + "\n", + "|**Wavelength calibration mode**||`TAGFLASH`|`SPLIT` Wavecal|`AUTO` Wavecal|GO Wavecal|\n", + "|-|-|-|-|-|-|\n", + "|**When used**||Default for LP1-LP5|Default for LP6|Used for LP1-LP5 non-TAGFLASH exposures|User specified (rare)|\n", + "|**Description**||Wavelength calibration lamp flashes concurrently with science exposure|Wavelength calibration lamp flashes at another LP|Wavelength calibration lamp flashes separately from science exposure|Wavelength calibration lamp flashes separately from science exposure|\n", + "|**Additional wavelength calibration file in the association table**||None - wavelength calibration data included in the `EXP-FP`s of the science exposures|`EXP-SWAVE`|`EXP-AWAVE`|`EXP-GWAVE`|" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Non-`TAGFLASH` datasets must be processed from a single association file which includes all the science and wavelength calibration exposures, properly labeled with the correct `MEMTYPE`s.\n", + "\n", + "In this section we will create an association file using the `SPLIT` wavecal data from the same dataset whose association file we altered as part of [Section 2.1.2](#removefiltAF). You could just as easily do so with `AUTO` or GO wavecal data. We will first select and examine the relevant exposures, then we will combine them into an association file." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 3.3.1. Gathering the exposure information\n", + "We begin by downloading the data from a visit which utilized COS' LP6 split wavecal mode (proposal ID: `16907`, observation ID: `letc01010`). We previously downloaded the association file for this dataset in [Section 2.1.2](#removefiltAF), but here we download the rest of the data products we will need." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Search for the correct obs_ids and get the list of data products\n", + "lp6_observation_products = Observations.get_product_list( \n", + " Observations.query_criteria(obs_id='letc01010'),\n", + ")\n", + " \n", + "# Filter and download the rawtag files and asn file in the product list from MAST\n", + "# First filter to just the rawtag products \n", + "lp6_rawtags_pl = Observations.filter_products(\n", + " lp6_observation_products,\n", + " productSubGroupDescription=['RAWTAG_A', 'RAWTAG_B'],\n", + ")\n", + "# Download these chosen rawtag products\n", + "lp6_rawtags = Observations.download_products(lp6_rawtags_pl, download_dir=str(datadir))['Local Path']" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We'll begin by taking a look at the original association file from MAST in the cell below. Note that the exposures are ordered in groups of a single `EXP-FP` (also known as science or `EXTERNAL/SCI`) exposure bracketed on both sides by an `EXP-SWAVE` (SPLIT wavecal) exposure (i.e. wavecal-->science-->wavecal). This is often the case for non-TAGFLASH files, especially for exposures longer than ~600 seconds." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Table.read(lp6_original_asnfile)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We'll simulate an occasion on which we would need to create our own association file by supposing we want to process only data from this observation taken in the FP1 or FP3 configuration. We could achieve the same result by taking the default association file from MAST and turning the `MEMPRSNT` value to False for all the FP2 and FP4 files, similarly to in [Section 2.1.2](#removefiltAF).\n", + "\n", + "Below, we filter to such exposures and check whether they meet our expectations for `SPLIT` wavecal data. Namely, we check that:\n", + "1. The files are listed in chronological order.\n", + "2. All science (`EXTERNAL/SCI`) exposures are bracketed by a `WAVECAL` exposure on either side.\n", + "3. The data is organized into groups of 3 exposures with the grouping described in test 2 (`WAVECAL`-->`EXTERNAL/SCI`-->`WAVECAL`).\n", + "\n", + "`CalCOS` does not strictly need the files to be arranged in chronological order; however, it is a helpful exercise to make sure we include all the files we need to calibrate this dataset. We can investigate the files manually and diagnose any failures in the table printed by the following cell." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "notebookRunGroups": { + "groupValue": "" + } + }, + "outputs": [], + "source": [ + "# Filter the rawtag files to exposures at FP1 and FP3\n", + "lp6_rawtags_fp13 = [rt for rt in lp6_rawtags if fits.getval(rt, \"FPPOS\", ext=0) in [1,3]]\n", + "# Gather information on the exposures\n", + "rawtag_a_exptypes_dict = {rt : fits.getval(rt, \"EXPTYPE\", ext=0) for rt in lp6_rawtags_fp13 if \"rawtag_a\" in rt}\n", + "rawtag_a_rootnames = [fits.getval(rt, \"ROOTNAME\", ext=0) for rt in lp6_rawtags_fp13 if \"rawtag_a\" in rt]\n", + "rawtag_a_exptypes = [fits.getval(rt, \"EXPTYPE\", ext=0) for rt in lp6_rawtags_fp13 if \"rawtag_a\" in rt]\n", + "rawtag_a_expstart_times = [fits.getval(rt, \"EXPSTART\", ext=1) for rt in lp6_rawtags_fp13 if \"rawtag_a\" in rt]\n", + "# Test 1 - Check chronological sorting\n", + "assert all(np.diff(rawtag_a_expstart_times) >= 0),\"These exposures are not ordered by start time.\"\n", + "print(\"Passed Test #1: The exposures are in chronological order.\")\n", + "# Test 2 - Check science exposures are bracketed by wavecals\n", + "for i, rta_type in enumerate(rawtag_a_exptypes):\n", + " if rta_type == \"EXTERNAL/SCI\":\n", + " assert rawtag_a_exptypes[i-1] == \"WAVECAL\", \"EXTERNAL/SCI exposure not preceded by a WAVECAL exposure\"\n", + " assert rawtag_a_exptypes[i+1] == \"WAVECAL\", \"EXTERNAL/SCI exposure not followed by a WAVECAL exposure\"\n", + "print(\"Passed Test #2: Each EXTERNAL/SCI is bracketted on both sides by a WAVECAL exposure.\")\n", + "# Test 3 - Check that only these groups of exposures exist (WAVECAL-->EXTERNAL/SCI-->WAVECAL)\n", + "for group_of_3 in [rawtag_a_exptypes[3*i:3*i+3] for i in range(int(len(rawtag_a_exptypes)/3))]:\n", + " assert group_of_3 == ['WAVECAL', 'EXTERNAL/SCI', 'WAVECAL'], \"Incorrect groupings of exposures\"\n", + "print(\"Passed Test #3: No unexpected groupings of files were found.\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Below, we may inspect these files by eye, and see that they are indeed ordered correctly." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pd.DataFrame({\n", + " \"Rootname\":[name.upper() for name in rawtag_a_rootnames], \n", + " \"Exposure_type\":rawtag_a_exptypes, \n", + " \"Exposure_start_date\":rawtag_a_expstart_times, # Date in MJD\n", + " \"Seconds_since_first_exposure\":\\\n", + " # convert time since the first exposure into seconds\n", + " 86400*np.subtract(rawtag_a_expstart_times, min(rawtag_a_expstart_times))\n", + "})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once we are sure that the correct exposures are selected, we gather the information we need for an association table:\n", + "* The exposure ID (`ROOTNAME`) of each exposure.\n", + " * When capitalized, this is the same as the `MEMNAME` we find in association files.\n", + "* The exposure type (`EXPTYPE`) of each exposure.\n", + " * We will need to convert from the way that exposure types are written in the FITS header to the way `CalCOS` will recognize them in the association file.\n", + "\n", + "To prevent duplication, we only gather this information from either the `rawtag_a` or `rawtag_b` files." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Choose either rawtag_a (default) of rawtag_b files if no rawtag_a files found\n", + "if any([\"rawtag_a\" in rt for rt in lp6_rawtags_fp13]):\n", + " segment_found=\"a\"\n", + "elif any([\"rawtag_b\" in rt for rt in lp6_rawtags_fp13]):\n", + " segment_found=\"b\"\n", + "else:\n", + " print(\"Neither rawtag_a nor rawtag_b found.\")\n", + "\n", + "lp6_fp13_memnames = [fits.getval(rt, \"ROOTNAME\", ext=0).upper() for rt in lp6_rawtags_fp13 if f\"rawtag_{segment_found}\" in rt]\n", + "lp6_fp13_exptypes = [fits.getval(rt, \"EXPTYPE\", ext=0) for rt in lp6_rawtags_fp13 if f\"rawtag_{segment_found}\" in rt]\n", + "# We need to change the wavecals' MEMTYPE to \"EXP-SWAVE\", and the sciences' to \"EXP-FP\"\n", + "# If these were GO or AUTO wavecals, we would instead convert to \"EXP-AWAVE\" or \"EXP-GWAVE\"\n", + "lp6_fp13_exptypes = [\n", + " \"EXP-SWAVE\" if etype == \"WAVECAL\" \n", + " else \"EXP-FP\" if etype == \"EXTERNAL/SCI\" \n", + " else None for etype in lp6_fp13_exptypes\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 3.3.2. Creating the `SPLIT` wavecal association file\n", + "Now that we've gathered `MEMNAME`s and `MEMTYPE`s of our exposures, we can create the association file. This very closely mirrors [Section 3.2](#32-with-fits-header-metadata)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Adding the exposure file details to the association table\n", + "new_asn_memnames = lp6_fp13_memnames # MEMNAME\n", + "types = lp6_fp13_exptypes # MEMTYPE\n", + "included = [True] * len(lp6_fp13_memnames) # MEMPRSNT\n", + "\n", + "# Adding the output science product details to the end of the association table columns\n", + "new_asn_memnames.append('splitwave'.upper()) # MEMNAME column\n", + "types.append('PROD-FP') # MEMTYPE column\n", + "included.append(True) # MEMPRSNT column\n", + "\n", + "# Convert the columns into a the necessary form for a fits file\n", + "c1 = fits.Column(name='MEMNAME', array=np.array(new_asn_memnames), format='40A') \n", + "c2 = fits.Column(name='MEMTYPE', array=np.array(types), format='14A')\n", + "c3 = fits.Column(name='MEMPRSNT', format='L', array=included)\n", + "\n", + "with fits.open(lp6_original_asnfile, mode='readonly') as hdulist: # Open up the old asn file\n", + " hdulist.info() # Shows the first hdu is empty except for the header we want\n", + " hdu0 = hdulist[0] # We want to directly copy over the old 0th header/data-unit\n", + " d0 = hdulist[0].data # gather the data from the header/data unit to allow the readout\n", + " h1 = hdulist[1].header # gather the header from the 1st header/data unit to copy to our new file\n", + " \n", + "hdu1 = fits.BinTableHDU.from_columns([c1, c2, c3], header=h1) # Put together new 1st hdu from old header and new data\n", + "\n", + "new_HDUlist = fits.HDUList([hdu0,hdu1]) # New HDUList from old HDU 0 and new combined HDU 1\n", + "new_HDUlist.writeto(outputdir / 'splitwave_asn.fits', overwrite=True) # Write this out to a new file\n", + "new_asnfile = outputdir / 'splitwave_asn.fits' # Path to this new file\n", + "print('\\nSaved '+ 'splitwave_asn.fits'+ f\" in the output directory: {outputdir}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "Table.read(outputdir/\"splitwave_asn.fits\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Well done - you have now successfully created the association file you need to process your `SPLIT` wavecal data. If you wish to test it by running the `CalCOS` data calibration pipeline on it, you may run the file `test_splitwave_asn.py` in this directory. Note the following requirements: \n", + "1. `test_splitwave_asn.py` can only be run after creating the `asn` files in Section 3.3 of this Notebook.\n", + "2. Your version of `CalCOS` must be ≥ 3.4. This version introduced the ability to process SPLIT wavecal data to the pipeline. You can check your version with `calcos --version` from the command line.\n", + "3. You must first update the `lref` environment variable set in `test_splitwave_asn.py` to the path to a directory containing all of your reference files. For more information on setting up such a directory, see [our Notebook on setting up an environment for COS data analysis](https://github.com/spacetelescope/notebooks/blob/master/notebooks/COS/Setup/Setup.ipynb)." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -538,7 +906,7 @@ "\n", "**Contributors:** Elaine Mae Frazer\n", "\n", - "**Updated On:** 2021-7-06\n", + "**Updated On:** 2022-08-17\n", "\n", "> *This tutorial was generated to be in compliance with the [STScI style guides](https://github.com/spacetelescope/style-guides) and would like to cite the [Jupyter guide](https://github.com/spacetelescope/style-guides/blob/master/templates/example_notebook.ipynb) in particular.*\n", "\n", @@ -563,7 +931,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3.9.5 ('fs2')", "language": "python", "name": "python3" }, @@ -577,7 +945,12 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.10" + "version": "3.9.5" + }, + "vscode": { + "interpreter": { + "hash": "a3aa8de4418ad0524928fefd3f2f3f8d89d7ea9dca3e47b348ac10cdea72900f" + } } }, "nbformat": 4, diff --git a/notebooks/COS/AsnFile/figures/compare_fluxes_after_removing_badfile.png b/notebooks/COS/AsnFile/figures/compare_fluxes_after_removing_badfile.png new file mode 100644 index 00000000..ad202560 Binary files /dev/null and b/notebooks/COS/AsnFile/figures/compare_fluxes_after_removing_badfile.png differ diff --git a/notebooks/COS/AsnFile/test_asn.py b/notebooks/COS/AsnFile/test_asn.py index 1c959c54..1a769e7c 100644 --- a/notebooks/COS/AsnFile/test_asn.py +++ b/notebooks/COS/AsnFile/test_asn.py @@ -16,12 +16,23 @@ outputdir = Path('./output/') plotsdir = Path('./output/plots/') # %% -######### SETTING THE lref VARIABLE: +######### Setting the lref environment variable: ### YOU LIKELY NEED TO CHANGE THIS LOCATION ! where_i_keep_my_ref_files = "/grp/hst/cdbs/lref/" os.environ['lref'] = where_i_keep_my_ref_files +assert Path(os.environ['lref']).exists(), "Make sure to set the 'lref' environment variable to a valid path with all of your reference files." +# Copy the asn file we made into the directory where we moved all of our ldif* exposures' data +shutil.copy("./output/ldifcombo_2_asn.fits", "./data/ldifcombo_2_asn.fits") + +# Run the CalCOS pipeline on our ldifcombo asn file +calcos.calcos('./data/ldifcombo_2_asn.fits', verbosity=0, outdir=str(outputdir/"./calcos_processed_1")) + +# %% +# Read in the processed data processed_data_tab = Table.read(str(outputdir/'calcos_processed_1/')+'/ldifcombo_x1dsum.fits') + +# Plot the processed data for segment in processed_data_tab: wvln, flux = segment["WAVELENGTH", "FLUX"] plt.plot(wvln, flux) @@ -29,7 +40,9 @@ plt.xlabel('Wavelength [$\AA$]') plt.ylabel('Flux [ergs/s/$cm^2$/$\AA$]') -plt.title("If this graph looks reasonable, your ASN file seems to have worked\n") +plt.title("If this graph looks at all reasonable, your ASN file seems to have worked\n") plt.tight_layout() -plt.savefig(str(plotsdir/"AsnFile_test.png"), bbox_inches = 'tight', dpi = 200) +plot_path = str(plotsdir/"AsnFile_test.png") +plt.savefig(plot_path, bbox_inches = 'tight', dpi = 200) +print(f"Saved plot to: {plot_path}") # %% diff --git a/notebooks/COS/AsnFile/test_removed_exposure_asn.py b/notebooks/COS/AsnFile/test_removed_exposure_asn.py new file mode 100644 index 00000000..2135ec41 --- /dev/null +++ b/notebooks/COS/AsnFile/test_removed_exposure_asn.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +#%%[markdown] +### From here, you can run the `CalCOS` pipeline on your association file where we've removed the empty exposure file. +##### Running `CalCOS` is explained in *much* more detail in our [Notebook on running the pipeline](https://github.com/spacetelescope/notebooks/blob/master/notebooks/COS/CalCOS/CalCOS.ipynb) + +##### In short, to run the `CalCOS` pipeline, you will need the relavent reference files. These will need be hosted in the directory assigned the environmetn variable `lref`. +##### No matter *where* you place these files, you *must* create the lref environment variable. +# %% +import os, shutil +import calcos +from astropy.table import Table +import matplotlib.pyplot as plt +from pathlib import Path +from astropy.io import fits +# %% +datadir = Path('./data/') +mast_download_dir = datadir / "mastDownload" +outputdir = Path('./output/') +plotsdir = Path('./output/plots/') +# %% +######### Setting the lref environment variable: +### YOU LIKELY NEED TO CHANGE THIS LOCATION ! +where_i_keep_my_ref_files = "/grp/hst/cdbs/lref/" +os.environ['lref'] = where_i_keep_my_ref_files +assert Path(os.environ['lref']).exists(), "Make sure to set the 'lref' environment variable to a valid path with all of your reference files." +# %% +remove_badfile_asn = "data/removed_badfile_asn.fits" +# Run the CalCOS pipeline on our asn file +calcos.calcos(remove_badfile_asn, verbosity=0, outdir=str(outputdir/"calcos_processed_remove_badfile/")) +# %% +# Read in the processed data +processed_data_tab = Table.read(str(outputdir/'calcos_processed_remove_badfile/')+'/ldif01010_x1dsum.fits') + +# Plot the processed data +for segment in processed_data_tab: + wvln, flux = segment["WAVELENGTH", "FLUX"] + plt.plot(wvln, flux) + +plt.xlabel('Wavelength [$\AA$]') +plt.ylabel('Flux [ergs/s/$cm^2$/$\AA$]') + +plt.title("If this plot looks reasonable, your ASN file seems to have worked\n") +plt.tight_layout() +plot_path = str(plotsdir/"remove_badfile_AsnFile_test.png") +plt.savefig(plot_path, bbox_inches = 'tight', dpi = 200) +print(f"Saved plot to: {plot_path}") +# %% diff --git a/notebooks/COS/AsnFile/test_splitwave_asn.py b/notebooks/COS/AsnFile/test_splitwave_asn.py new file mode 100644 index 00000000..1d3fdeda --- /dev/null +++ b/notebooks/COS/AsnFile/test_splitwave_asn.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +#%%[markdown] +### From here, you can run the `CalCOS` pipeline on your new association SPLIT wavecal file. +##### Running `CalCOS` is explained in *much* more detail in our [Notebook on running the pipeline](https://github.com/spacetelescope/notebooks/blob/master/notebooks/COS/CalCOS/CalCOS.ipynb) + +##### In short, to run the `CalCOS` pipeline, you will need the relavent reference files. These will need be hosted in the directory assigned the environmetn variable `lref`. +##### No matter *where* you place these files, you *must* create the lref environment variable. +# %% +import os, shutil +import calcos +from astropy.table import Table +import matplotlib.pyplot as plt +from pathlib import Path +from astropy.io import fits +# %% +datadir = Path('./data/') +mast_download_dir = datadir / "mastDownload" +lp6_1exp_datadir = datadir / "lp6_1exp_datadir" +lp6_1exp_datadir.mkdir(exist_ok=True) +outputdir = Path('./output/') +plotsdir = Path('./output/plots/') +# %% +# Copy all of the relevant LP6 data files into the lp6_1exp_datadir +splitwave_rawtags = [ + shutil.copy(rt, lp6_1exp_datadir / rt.name) \ + for rt in mast_download_dir.glob("**/*rawtag*fits") \ + # Only grab the necessary files + if fits.getval(rt, "ROOTNAME").upper() in ['LETC01M6Q','LETC01MTQ','LETC01MVQ'] +] +# Copy the LP6 single exposure asn file we made into the directory where we moved all of the relevant exposures' data +lp6_1exp_asn = shutil.copy('data/letc01mtq_only_asn.fits', lp6_1exp_datadir) +# %% +######### Setting the lref environment variable: +### YOU LIKELY NEED TO CHANGE THIS LOCATION ! +where_i_keep_my_ref_files = "/grp/hst/cdbs/lref/" +os.environ['lref'] = where_i_keep_my_ref_files +assert Path(os.environ['lref']).exists(), "Make sure to set the 'lref' environment variable to a valid path with all of your reference files." +# %% +# Run the CalCOS pipeline on our SPLIT wavecal asn file +calcos.calcos(lp6_1exp_asn, verbosity=0, outdir=str(outputdir/"calcos_processed_lp6_1exp/")) +# %% +# Read in the processed data +processed_data_tab = Table.read(str(outputdir/'calcos_processed_lp6_1exp/')+'/letc01mtq_only_x1dsum.fits') + +# Plot the processed data +for segment in processed_data_tab: + wvln, flux = segment["WAVELENGTH", "FLUX"] + plt.plot(wvln, flux) + +plt.xlabel('Wavelength [$\AA$]') +plt.ylabel('Flux [ergs/s/$cm^2$/$\AA$]') + +plt.title("If this plot looks reasonable, your ASN file seems to have worked\n") +plt.tight_layout() +plot_path = str(plotsdir/"LP6_1exposure_AsnFile_test.png") +plt.savefig(plot_path, bbox_inches = 'tight', dpi = 200) +print(f"Saved plot to: {plot_path}") +# %% diff --git a/notebooks/COS/DataDl/DataDl.ipynb b/notebooks/COS/DataDl/DataDl.ipynb index 5bb11df7..169c8aeb 100644 --- a/notebooks/COS/DataDl/DataDl.ipynb +++ b/notebooks/COS/DataDl/DataDl.ipynb @@ -818,8 +818,9 @@ "---\n", "## About this Notebook\n", "**Author:** Nat Kerman \n", + "**Edited by:** Travis Fischer \n", "\n", - "**Updated On:** 2022-02-25\n", + "**Updated On:** 2022-09-09\n", "\n", "> *This tutorial was generated to be in compliance with the [STScI style guides](https://github.com/spacetelescope/style-guides) and would like to cite the [Jupyter guide](https://github.com/spacetelescope/style-guides/blob/master/templates/example_notebook.ipynb) in particular.*\n", "\n", @@ -913,7 +914,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3.9.5 ('fs2')", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -927,7 +928,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.5" + "version": "3.9.7" }, "vscode": { "interpreter": {