How_to_use_SPARCL_with_DESI_DR1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b20ea4e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = 'Alice Jacques <alice.jacques@noirlab.edu>, SPARCL team <datalab-spectro@noirlab.edu>'\n",
    "__version__ = '20250128' # yyyymmdd\n",
    "__datasets__ = ['desi_dr1']  \n",
    "__keywords__ = ['sparcl', 'spectroscopy', 'HowTo', 'desi spectra', 'tutorial']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4162486",
   "metadata": {},
   "source": [
    "# How to use SPARCL to access DESI DR1 data\n",
    "## SPectra Analysis and Retrievable Catalog Lab (SPARCL)\n",
    "Alice Jacques (NOIRLab), Stéphanie Juneau (NOIRLab), Benjamin Weaver (NOIRLab), Steve Pothier (NOIRLab), Adam Bolton (SLAC) and the SPARCL team"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45030923",
   "metadata": {},
   "source": [
    "### Table of contents\n",
    "* [Goals & Summary](#goalssummary)\n",
    "* [Disclaimer & attribution](#disclaimer)\n",
    "* [If necessary, install the most recent version of the SPARCL Client](#install)\n",
    "* [Imports and setup](#imports)\n",
    "* [Authentication for SPARCL](#auth_sparcl)\n",
    "* [Data discovery: using SPARCL's <tt>client.find()</tt> method](#datadiscovery)\n",
    "* [Retrieve records by <tt>sparcl_id</tt> using <tt>client.retrieve()</tt>](#retrieve)\n",
    "* [Retrieve records by <tt>specid</tt> using <tt>client.retrieve_by_specid()</tt>](#retrieve_specid)\n",
    "* [Access fields in records](#access)\n",
    "* [Convert retrieved output to Pandas DataFrame or Spectrum1D object](#convert)\n",
    "* [Plot spectra](#plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acda2d5d",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"goalssummary\"></a>\n",
    "## Goals & Summary \n",
    "\n",
    "SPARCL (SPectra Analysis and Retrievable Catalog Lab) is an online service for discovery and retrieval of one-dimensional optical-infrared spectra. SPARCL is designed to work for large survey datasets containing many millions of spectra, and to provide access to multiple different data sets through common methods. For more information, see the [SPARCL User Guide](https://astrosparcl.datalab.noirlab.edu/static/SPARCLUserManual.pdf).\n",
    "\n",
    "This notebook provides a basic introduction to using the SPARCL client (or sparclclient) to find and retrieve DESI spectroscopic data within a Python notebook context. The sparclclient connects to the SPARCL server at [NSF NOIRLab](https://noirlab.edu/public/) and provides access to the contents of the SPARCL database.\n",
    "\n",
    "To see the current data sets and number of spectra available in the SPARCL database, please visit the [SPARCL summary page](https://astrosparcl.datalab.noirlab.edu/sparc/).\n",
    "\n",
    "To get the latest sparclclient documentation, visit the [sparclclient readthedocs site](https://sparclclient.readthedocs.io/en/latest/).\n",
    "\n",
    "Feedback on SPARCL (questions, comments, science use cases, feature requests, bug reports, confusing error messages, etc.) can be submitted to datalab-spectro@noirlab.edu. For bug reports and confusing error messages, it's helpful if you include: a log of *what you did*, *the result you got*, and *the result you expected*.\n",
    "\n",
    "See our science use-case notebooks that use SPARCL:\n",
    "- [Introduction to DESI Early Data Release (EDR) at the Astro Data Lab](https://github.com/astro-datalab/notebooks-latest/blob/master/03_ScienceExamples/DESI/01_Intro_to_DESI_EDR.ipynb)\n",
    "- [Comparing SDSS and DESI spectra using SPARCL](https://github.com/astro-datalab/notebooks-latest/blob/master/03_ScienceExamples/DESI/02_DESI_EDR_SDSS_Comparison.ipynb)\n",
    "- [Stacking SDSS Spectra of Galaxies Selected from the BPT Diagram](https://github.com/astro-datalab/notebooks-latest/blob/master/03_ScienceExamples/EmLineGalaxies/01_EmLineGalaxies_SpectraStack.ipynb)\n",
    "- [Multi-wavelength Image Cutouts and SDSS Spectra of Active Galaxies with Extreme Emission-Line Ratios](https://github.com/astro-datalab/notebooks-latest/blob/master/03_ScienceExamples/EmLineGalaxies/02_EmLineGalaxies_Outliers.ipynb)\n",
    "\n",
    "And our other How-To notebooks that use SPARCL:\n",
    "- [Obtain spectra with SPARCL and plot them with Jdaviz](https://github.com/astro-datalab/notebooks-latest/blob/master/04_HowTos/SPARCL/Plot_SPARCL_Spectra_with_Jdaviz.ipynb)\n",
    "- [Obtain spectra with SPARCL and plot them with prospect](https://github.com/astro-datalab/notebooks-latest/blob/master/04_HowTos/SPARCL/Plot_SPARCL_Spectra_with_Prospect.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12173d5-9849-4113-808a-c920d912e244",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"attribution\"></a>\n",
    "# Disclaimer & attribution\n",
    "\n",
    "Disclaimers\n",
    "-----------\n",
    "Note that using the Astro Data Lab constitutes your agreement with our minimal [Disclaimers](https://datalab.noirlab.edu/disclaimers.php).\n",
    "\n",
    "Acknowledgments\n",
    "---------------\n",
    "If you use **Astro Data Lab** in your published research, please include the text in your paper's Acknowledgments section:\n",
    "\n",
    "_This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation._\n",
    "\n",
    "If you use **SPARCL jointly with the Astro Data Lab platform** (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:\n",
    "\n",
    "_This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation._\n",
    "\n",
    "In either case **please cite the following papers**:\n",
    "\n",
    "* Data Lab concept paper: Fitzpatrick et al., \"The NOAO Data Laboratory: a conceptual overview\", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445\n",
    "\n",
    "* Astro Data Lab overview: Nikutta et al., \"Data Lab - A Community Science Platform\", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411\n",
    "\n",
    "If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:\n",
    "\n",
    "* Juneau et al., \"Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms\", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097\n",
    "\n",
    "If publishing in a AAS journal, also add the keyword: `\\facility{Astro Data Lab}`\n",
    "\n",
    "And if you are using SPARCL, please also add `\\software{SPARCL}` and cite:\n",
    "\n",
    "* Juneau et al., \"SPARCL: SPectra Analysis and Retrievable Catalog Lab\", Conference Proceedings for ADASS XXXIII, 2024\n",
    "https://doi.org/10.48550/arXiv.2401.05576\n",
    "\n",
    "The NOIRLab Library maintains [lists of proper acknowledgments](https://noirlab.edu/science/about/scientific-acknowledgments) to use when publishing papers using the Lab's facilities, data, or services."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c39b96f",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"install\"></a>\n",
    "### If necessary, install the most recent version of the SPARCL Client:\n",
    "If you are using the Astro Data Lab Jupyter notebook server, you do not need to run this cell.\n",
    "### NOTE: After installing the most recent version, please restart your kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cb3b11f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Uncomment the following only if SPARCL client is not already installed\n",
    "#!pip install --upgrade sparclclient"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25a609e2",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"imports\"></a>\n",
    "## Imports and setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d913a9b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# SPARCL imports\n",
    "from sparcl.client import SparclClient\n",
    "\n",
    "# 3rd party imports\n",
    "import numpy as np\n",
    "import astropy.units as u\n",
    "from specutils import Spectrum1D\n",
    "from astropy.nddata import InverseVariance\n",
    "from astropy.convolution import convolve, Gaussian1DKernel\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "from getpass import getpass\n",
    "\n",
    "# plots default setup\n",
    "plt.rcParams['font.size'] = 14\n",
    "plt.rcParams['figure.figsize'] = (14,8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5647c893",
   "metadata": {},
   "source": [
    "#### Create a SPARCL client instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed2bfd68-6a5a-486f-84ee-c9f1f7d1db38",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "client = SparclClient()\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43e8678f-21a6-4246-8061-358c19f8ff5a",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"auth_sparcl\"></a>\n",
    "# Authentication for SPARCL\n",
    "All public SPARCL data sets can be accessed without explicitly logging in. However, some data sets are private and can only be accessed by authorized users. If you are an authorized user and wish to login to SPARCL, un-comment the cell below and enter your NOIRLab CSDC SSO user name and password. If you need to create an account, sign-up at https://sso.csdc.noirlab.edu/account/signup/. If you encounter an issue, email datalab-spectro@noirlab.edu with your First Name, Last Name, and Email Address (the same one you used to create your NOIRLab CSDC SSO account).\n",
    "\n",
    "To logout of SPARCL after a session, use:\n",
    "```\n",
    "client.logout()\n",
    "```\n",
    "\n",
    "For assistance with SPARCL authentication/authorization please contact us at datalab-spectro@noirlab.edu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c33c4445-7ea4-42fb-9c0c-549dcf8180dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.login(input(\"Enter SSO user name: (+ENTER) \"),getpass(\"Enter password: (+ENTER) \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4e92c80-c8bb-434a-acb5-9973634ba2f2",
   "metadata": {},
   "source": [
    "#### View which data sets you have access to:\n",
    "**Note:** if you are not logged in or if your SSO user name is not in the authorized list of SPARCL users, you will only see public data sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "944b18e6-3d8b-4239-9514-89103da393fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.authorized"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd9cc611",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"datadiscovery\"></a>\n",
    "## Data discovery: using SPARCL's `client.find()` method\n",
    "The first way you can discover your data is by using SPARCL's `client.find()` method, which allows you to find records in the SPARCL database based on certain parameters passed to the function. Only Core fields may be in the `outfields` and `constraints` parameters. The descriptions for all fields, including Core fields, is located [here](https://astrosparcl.datalab.noirlab.edu/sparc/sfc/). The SPARCL Core fields constraint types are:\n",
    "\n",
    "\n",
    "| Field name       | Constraint type | Example |\n",
    "|:----------------|:---------------|:-------|\n",
    "| data_release     | List of allowed values<br>from [SPARCL Categoricals](https://astrosparcl.datalab.noirlab.edu/sparc/cats/) | ['DESI-EDR', 'BOSS-DR16', 'SDSS-DR16']\n",
    "| datasetgroup     | List of allowed values<br>from [SPARCL Categoricals](https://astrosparcl.datalab.noirlab.edu/sparc/cats/) | ['DESI', 'SDSS_BOSS']\n",
    "| dateobs_center   | Range of values | ['2013-03-14T10:16:17Z',<br>'2014-05-24T12:10:00Z']\n",
    "| dec              | Range of values | [2.03, 7.76]\n",
    "| exptime          | Range of values | [3603.46, 3810.12]\n",
    "| instrument       | List of allowed values<br>from [SPARCL Categoricals](https://astrosparcl.datalab.noirlab.edu/sparc/cats/) | ['SDSS', 'BOSS', 'DESI']\n",
    "| ra               | Range of values (may not<br>\"wrap\" around RA=0) | [44.53, 47.96]\n",
    "| redshift         | Range of values | [0.5, 0.9]\n",
    "| redshift_err     | Range of values | [0.000225, 0.000516]\n",
    "| redshift_warning | List of values  | [0, 3, 5]\n",
    "| sparcl_id               | List of values (but not<br>intended for data discovery) | ['00001658-460c-4da1-987d-e493d8c9b89b',<br>'000017b6-56a2-4f87-8828-3a3409ba1083']\n",
    "| site             | List of allowed values<br>from [SPARCL Categoricals](https://astrosparcl.datalab.noirlab.edu/sparc/cats/) |  ['apo', 'kpno']\n",
    "| specid           | List of values | [6988698046080241664, 6971782884823945216]\n",
    "| spectype         | List of allowed values<br>from [SPARCL Categoricals](https://astrosparcl.datalab.noirlab.edu/sparc/cats/) | ['GALAXY', 'STAR', 'QSO']\n",
    "| specprimary      | List of values (but typically<br>would only include 1 if<br>being used for data<br>discovery constraints) | [1]\n",
    "| targetid         | List of values | [1237679502171374316, 1237678619584692841]\n",
    "| telescope        | List of allowed values<br>from [SPARCL Categoricals](https://astrosparcl.datalab.noirlab.edu/sparc/cats/) | ['sloan25m', 'kp4m']\n",
    "| wavemin          | Range of values | [3607, 3608]\n",
    "| wavemax          | Range of values | [10363, 10364]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ecf1aa4",
   "metadata": {},
   "source": [
    "#### Define the fields we want returned (`outfields`) and the constraints (`constraints`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1a3fd21",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "out = ['sparcl_id', 'specid', 'data_release', 'spectype', 'ra', 'dec', 'redshift']\n",
    "cons = {'data_release': ['DESI-DR1'],\n",
    "        'spectype': ['GALAXY'],\n",
    "        'redshift': [0.1, 0.3]}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c423b17d",
   "metadata": {},
   "source": [
    "#### Execute the `client.find()` method with our parameters:\n",
    "The `limit` argument here is being used for demonstration purposes only, and simply returns only the first 20 results here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c55933c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "found = client.find(outfields=out, constraints=cons, limit=20)\n",
    "\n",
    "# Convert to a dataframe\n",
    "df_found = pd.json_normalize(found.records)\n",
    "\n",
    "# Print length and a few example rows\n",
    "print('N results:', len(df_found))\n",
    "print('N(unique specids):', len(np.unique(df_found['specid'])))\n",
    "\n",
    "df_found[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cd77794-e1e5-4005-8cdf-97072a5e0e0b",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a class=\"anchor\" id=\"retrieve\"></a>\n",
    "## Retrieve records by `sparcl_id` using `client.retrieve()`\n",
    "In order to retrieve spectra records from SPARCL by `sparcl_id`, pass the following to the `client.retrieve()` method:\n",
    "\n",
    "`uuid_list` : List of SPARCL IDs.  \n",
    "`dataset_list` : List of data sets to search for the SPARCL IDs in (default: None, which will search all available data sets).  \n",
    "`include` : List of field names to include in each record (default: 'DEFAULT').  \n",
    "`limit` : Maximum number of records to return (default: 500). Max allowed is 24,000.\n",
    "\n",
    "\n",
    "**NOTE: A reasonable amount of records to request retrieval of is about 10,000. Exceeding this value may cause the retrieval to timeout or fail.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40036a51",
   "metadata": {},
   "source": [
    "#### Use the sparcl_ids from the output of using `client.find()` to retrieve records from SPARCL:\n",
    "Note that `ids` in `found_I.ids` is a property name of the Found class. It is a list of records from all records, not a field name of a record."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c93923fb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the fields to include in the retrieve function\n",
    "inc = ['sparcl_id', 'specid', 'data_release', 'spectype', 'ra', 'dec', 'redshift', 'survey', 'program', 'specprimary', \n",
    "       'flux', 'wavelength', 'model', 'ivar', 'mask']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b86047a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sparcl_ids = found.ids\n",
    "results = client.retrieve(uuid_list=sparcl_ids, dataset_list=['DESI-DR1'], include=inc)\n",
    "results.info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "316c6f45",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"retrieve_specid\"></a>\n",
    "## Retrieve records by `specid` using `client.retrieve_by_specid()`\n",
    "In order to retrieve spectra records from SPARCL by `specid`, pass the following to the `client.retrieve_by_specid()` method:\n",
    "\n",
    "`specid_list` : List of specIDs.  \n",
    "`dataset_list` : List of data sets to search for the specIDs in (default: None, which will search all available data sets).  \n",
    "`include` : List of field names to include in each record (default: 'DEFAULT').  \n",
    "`limit` : Maximum number of records to return (default: 500). Max allowed is 24,000.\n",
    "\n",
    "\n",
    "**NOTE: A reasonable amount of records to request retrieval of is about 10,000. Exceeding this value may cause the retrieval to timeout or fail.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f73a2d7",
   "metadata": {},
   "source": [
    "#### Use the specIDs from the output of using `client.find()` to retrieve records from SPARCL:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "424617cf-0a22-4df2-b3cd-6d466561344d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the fields to include in the retrieve_by_specid function\n",
    "inc = ['sparcl_id', 'specid', 'data_release', 'spectype', 'ra', 'dec', 'redshift',\n",
    "       'flux', 'wavelength', 'model', 'ivar', 'mask']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b751a29e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "specids = [f.specid for f in found.records]\n",
    "results_specids = client.retrieve_by_specid(specid_list=specids, dataset_list=['DESI-DR1'], include=inc)\n",
    "results_specids.info"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3332ef24",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"access\"></a>\n",
    "## Access fields in records\n",
    "You can access the fields from records by using dot notation and dictionary indexing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81856081",
   "metadata": {},
   "source": [
    "#### Accessing a record from our example using `client.find()` and `client.retrieve()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "869c4f66",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "records = results.records[0]\n",
    "\n",
    "sparcl_id = records.sparcl_id\n",
    "specid = records.specid\n",
    "spectype = records.spectype\n",
    "redshift = records.redshift\n",
    "flux = records.flux\n",
    "wavelength = records.wavelength\n",
    "model = records.model\n",
    "ivar = records.ivar\n",
    "mask = records.mask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "020f7d50",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"convert\"></a>\n",
    "## Convert retrieved output to Pandas DataFrame or Spectrum1D object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cbb5458",
   "metadata": {},
   "source": [
    "#### Pandas DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e7e06ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.json_normalize(results_specids.records)\n",
    "\n",
    "print(len(df))\n",
    "\n",
    "# Check the first few rows\n",
    "df[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de3baaf9",
   "metadata": {},
   "source": [
    "#### Spectrum1D:\n",
    "When applicable, the units for each field are documented on the [Fields tab of the astrosparcl website](https://astrosparcl.datalab.noirlab.edu/sparc/sfc/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fe8cde6-b66e-43b0-bbb0-916c324d6db0",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Get one record for the wl\n",
    "wl = df.wavelength[0]\n",
    "\n",
    "## Alternatively, could define manually for DESI as it's a common wave grid\n",
    "#wl = np.arange(3600, 9824.8, 0.8)\n",
    "wl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fc03e7e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Convert to a Spectrum1D object\n",
    "# turning the dataframe into Nspec x Npix arrays for flux, ivar, mask\n",
    "ivar = u.Quantity(np.stack(df.ivar), u.Unit('1e34 erg-2 cm4 s2 AA2'))\n",
    "specs = Spectrum1D(spectral_axis = wl*u.AA,\n",
    "                   flux = u.Quantity(np.stack(df.flux), u.Unit('1e-17 erg cm-2 s-1 AA-1')),\n",
    "                   uncertainty = InverseVariance(ivar),\n",
    "                   redshift = df.redshift, \n",
    "                   mask = np.stack(df['mask']), \n",
    "                   meta = {'ra': df.ra, 'dec': df.dec, 'z': df.redshift})\n",
    "\n",
    "# Added a few columns in the meta attribute (including the redshift in case one wants \n",
    "# to save to a FITS file as the redshift attribute does not appear to get saved)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee58cd93",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Plot a couple\n",
    "f, ax = plt.subplots()  \n",
    "ax.step(specs[0].spectral_axis, specs[0].flux) \n",
    "ax.step(specs[1].spectral_axis, specs[1].flux)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2800c6b7-7730-4fc8-b665-26802053b431",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-01-27T23:56:24.541595Z",
     "iopub.status.busy": "2025-01-27T23:56:24.541199Z",
     "iopub.status.idle": "2025-01-27T23:56:24.544819Z",
     "shell.execute_reply": "2025-01-27T23:56:24.544213Z",
     "shell.execute_reply.started": "2025-01-27T23:56:24.541565Z"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "Plot the S/N-per-pixel using the flux and ivar from Spectrum1D object. This can be done for one or two example objects such as the ones plotted above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce3ebae5",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"plot\"></a>\n",
    "## Plot spectra\n",
    "\n",
    "Function using the SPARCL records as an input. **Exercise**: one can create a similar function to take an input dataframe or Spectrum1D object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35e2607b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def plot_spec(index, res):\n",
    "    \"\"\"\n",
    "    Pass an index value and the output from using client.retrieve()\n",
    "    to plot the spectrum at the specified index.\n",
    "    \"\"\"\n",
    "    \n",
    "    record = res.records[index]\n",
    "\n",
    "    sparcl_id = record.sparcl_id\n",
    "    data_release = record.data_release\n",
    "    spectype = record.spectype\n",
    "    flux = record.flux\n",
    "    wavelength = record.wavelength\n",
    "    model = record.model\n",
    "    mask = record.mask\n",
    "\n",
    "    # Round the redshift, RA, Dec to a reasonable number of significant digits\n",
    "    redshift = np.round(record.redshift, 4)\n",
    "    ra = np.round(record.ra, 7)\n",
    "    dec = np.round(record.dec, 7)\n",
    "\n",
    "    # Define unmasked pixels as the valid range\n",
    "    valid = mask==0\n",
    "\n",
    "    plt.title(f\"SPARCL ID = {sparcl_id}\\n\"\n",
    "              f\"Data Set = {data_release}\\n\"\n",
    "              f\"Type = {spectype}\\n\"\n",
    "              f\"RA = {ra}\\n\"\n",
    "              f\"Dec = {dec}\\n\"\n",
    "              f\"Redshift = {redshift}\\n\", loc='left')\n",
    "    plt.xlabel('$\\lambda\\ [\\AA]$')\n",
    "    plt.ylabel('$f_{\\lambda}$ $(10^{-17}$ $erg$ $s^{-1}$ $cm^{-2}$ $\\AA^{-1})$')\n",
    "    \n",
    "    # Plot unsmoothed spectrum in grey\n",
    "    plt.plot(wavelength, flux, color='k', alpha=0.2, label='Unsmoothed spectrum')\n",
    "    \n",
    "    # Overplot spectrum smoothed using a 1-D Gaussian Kernel in black\n",
    "    plt.plot(wavelength[valid], convolve(flux[valid], Gaussian1DKernel(5)), color='k', label='Smoothed spectrum')\n",
    "    \n",
    "    # Overplot the model spectrum in red\n",
    "    plt.plot(wavelength[valid], model[valid], color='r', label='Model spectrum')\n",
    "    \n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2545c6e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plot_spec(index=5, res=results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "874c148e-ab3e-48b7-a2d2-39030eba10b6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-01-28T00:02:49.874060Z",
     "iopub.status.busy": "2025-01-28T00:02:49.873692Z",
     "iopub.status.idle": "2025-01-28T00:02:49.881855Z",
     "shell.execute_reply": "2025-01-28T00:02:49.880859Z",
     "shell.execute_reply.started": "2025-01-28T00:02:49.874031Z"
    }
   },
   "source": [
    "### Interactive spectra visualization\n",
    "\n",
    "Now that you have seen how to create static visualizations of the spectra with the commonly-used matplotlib library, you can explore this notebook that shows [how to obtain spectra with SPARCL and plot them with prospect](https://github.com/astro-datalab/notebooks-latest/blob/master/04_HowTos/SPARCL/Plot_SPARCL_Spectra_with_Prospect.ipynb). The [prospect](https://github.com/desihub/prospect) tool has been used for a number of DESI visual inspection campains and allows users to pan, zoom, adjust the smoothing level and the redshift value. It also features buttons to easily navigate through a pre-loaded set of spectra, and displays some catalog information. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "DESI 23.1 (DL,Py3.10.13)",
   "language": "python",
   "name": "desi_23.1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}