README.md

KPIC Data Reduction Pipeline

Install

Run in the top level of this repo:

> pip install -r requirements.txt -e .

This will create a configuration file at ~/.kpicdrp that will specify the path to where the KPIC DRP calibration databases live. By default, it is where the source code for the KPIC DRP lives. The following calibration databases will be defined:

• caldb_detector.csv
• caldb_traces.csv
• caldb_wavecal.csv

caldb_detector.csv defines the location of the thermal background as a function of exposure time as well as the bad pixel map. caldb_traces.csv defines the location of the trace normally using bright A0V stars. caldb_wavecal.csv defines the wavelength solution, typically using bright early M giant star spectra. Note that when you rerun the trace or wavelength calibration using a different target or model, you will need to remove the earlier entry to ensure the most recent files are properly used in the following steps.

To make use of our wavelength calibration routine, you will need to download initial guesses from our Public KPIC Data if you have not reduced any KPIC data yet. There are also examples of raw and reduced KPIC spectra on the drive for you to compare.

Pipelines to extract 1D spectrum

There are two instructions to reduce the observed KPIC data into 1D spectra. One is "Instructions to Reduce Individual Datasets", which reduces only one given science target. The other is "Instructions to Reduce Nightly Data", which reduces all data observed within the same night altogether. Users who would like to reduce the full nightly data using the latter method are encouraged to try the method to reduce individual targets to understand how the KPIC DRP is reducing the spectra.

Instructions to Reduce Individual Datasets

DRP Steps  

Outline 

• Background/Bad Pixel Finding 
• Telluric/RV Trace Finding 
• RV Standard Flux Extraction 
• RV Standard Wavelength Calibration 
• Target Flux Extraction 

Background/Bad Pixel Finding 

Select a location on your machine to store, process, and reduce data

• Output locations are flexible in data reduction scripts
• Identify which nspec files are Background exposures
• Make a folder and copy valid background files for the night

cd /mykpicdatadir/

mkdir calibs;cd calibs

mkdir 20221009;cd 20221009 (replace with date of your data)

mkdir bkgd_bdmap;cd bkgd_bdmap

mkdir raw

• Move the pertinent files to this directory 
• Make a copy of the background_demo.py script from /kpic_pipeline/examples and change the paths to match where the data is  

  Output should give files like: [bkgd_fileprefix]_background_med_nobars_tint[integration_time]_coadds1.fits and [bkgd_fileprefix] _persistent_badpix_nobars_tint[integration_time]_coadds1.fits for each true integration time (tint) used  

Result should look like 


 

Trace Finding 

• Identify which nspec files are telluric calibrator (A0V star) exposures
• Make a folder for trace and copy valid data files

cd ~/mykpicdatadir/20221009/  (replace with your path and date)

mkdir trace  

mkdir raw

• Make a copy of the trace_demo.py script from /kpic_pipeline/examples and change the paths to match where the data is

• Output should give files like: nspec20200928_[last_filenumber]_trace.fits 
• Output should look something like: 

The star fibers 1-4 are labeled in s1, s2, s3, s4, respectively, whereas the background/dark traces are in between these fibers are labeled as b0, b1, b2, b3, or d0, d1, d2, d3.

Flux Extraction 

• Start with the wavelength calibrator/RV standard (e.g. HIP 81497 on 20220721) 
• Find/Make a KPIC Campaign Science folder for the RV standard and copy valid data files

cd ~/mykpicdatadir

mkdir science;cd science

mkdir HIP81497 (replace target name with your wavelength calibrator or RV standard)

mkdir 20220721;cd 20220721 (swap date with your observation date)

mkdir raw

• Make a copy of the extraction_demo.py script from /kpic_pipeline/examples
• There are a few options to note in this script:
  • A choice of a box extraction or optimal extraction
  • A choice of the background subtraction method, either using the main background file or using a nod subtraction if your dataset uses fiber bouncing during observation

# define the extraction methods
box = False # box or optimal extraction

subtract_method = 'bkgd' # bkgd (background) or nod (nod subtraction/pair subtraction)

• and change the paths to match where the data is
 

• Output should give files like: nspec220721_[filenumbers]_bkgdsub_spectra.fits 

• Output should look something like, though it will depend on your exposure time and throughput for the night: 

*To check throughput on calibrators, modify and run kpic_pipeline/examples/throughput_demo.py 
 

Wavelength Calibration 

KPIC uses a bright early M giant star which utilizes CO lines as well as earth telluric absorption to extract the wavelength solutions. The three best orders in K band for science analyses are orders 31-33 (2.29-2.49 microns), but KPIC wavelength calibration attempts to provide the wavelength solutions for all nine NIRSPEC orders 31-39. To enable this routine, make sure that you have downloaded the initial guesses and earth atmosphere models from the utils folder in the Public KPIC Data. The initial guesses of the wavelength solutions are 'first_guess_wvs_20200607_HIP_81497.fits' or 'first_guess_wvs_20200928_HIP_81497.fits'. To model the M giant stellar atmosphere, the KPIC DRP uses PHOENIX-ACES-AGSS-COND-2011 models at a given effective and surface gravity (e.g. HIP81497_lte03600-1.00-0.0.PHOENIX-ACES-AGSS-COND-2011-HiRes.fits) and requires an input radial velocity for the M giant spectra.

• Make a calib folder for wavelength calibration in KPIC_Campaign

cd /mykpicdatadir/calibs/20220721 (replace with your date and directory)

mkdir wave;cd wave

• Make a copy of the wavecal_demo.py script from /kpic_pipeline/examples and change the paths to match where the data is (Be careful here, there is a lot to change)

• Be sure to change the RV value to match the wavecal star you are using
 

If the wavelength solution is bad, you can try to use another initial wavelength solution file as a starting point, or you can widen the grid search delta wavelength (grid_dwv) from 1e-4 to 3e-4 microns. These should in principle provide a reasonable wavelength calibration.

• File outputs should look like: 20220721_HIP81497_psg_wvs.fits  
• The script will plot images for each fiber to show how the wavelength solution fits and save the plots; make sure these are in the wave directory as well 
• Solution should look something like this for each fiber:



Rinse and Repeat Flux Extraction for all Host/Companions/Targets that you wish to analyze 

Instructions to Reduce Nightly Data

Before running, you need to generate the following. Example scripts are provided in the examples folder.

1. calib_info file which specifies which frames are background, wavecal, and trace (calib_info.py)
2. Bad pixel maps and backgrounds (calc_background.py)
3. Trace file (calc_trace.py)

The nightly pipeline is called run_extraction_night.py. You run it with:

> python run_extraction_night.py

To run in real time (during observing), you need to constantly read the new files. Nodding is recommended since you might not have backgrounds. Run:

> python run_extraction_night.py --nod_only=y --new_files=y --plot=y