SofI Pipeline
This page describes the SofI Pipeline. It is a clone of the ISAAC, adopted to fit the SofI features, i.e. crosstalk, orientation of the slit, etc. Contributions and suggestions are encouraged.
Update 2017-10-04: The latest version of the pipeline (1.5.8 from 2017-05-15) has a problem with the 1D spectrum extraction. The sofi_spc_jitter recipe fails before writing saving the 1D and 2D products. The pipeline is at the end of its support and the fix may take a while. At the end of this page we suggest a workaround: to comment out the 1D extraction, so the 2D spectrum is saved and the users are given the opportunity to carry out the 1D extraction by other means, e.g. IRAF.
What does the pipeline do?
Quick-look on-line data reduction. This means that the pipeline automatically classifies every file produced by SofI, and reduces it according to pre-defined rules, with one of the specific SofI recipes. The recipes are specialized data reduction tools, similar to the IRAF task but more comprehensive. They are written in C. The SofI pipeline recipes are available upon request starting the beginning of P78.
The data product is transferred to the off-line workstation wg5off where the user can review the file. Note: to view the newly arrived files on wg5off with Gasgano, one has to "refresh" the content of the Gasgano file list.
All file transfers are transparent to the user.
Purpose of the pipeline
- To provide reasonably well reduced data that can be reviewd by the observer in nearly real time to facilitate making decisions about the quality of the acquired data.
- To provide data quality control parameters measured from the data itself, such as: zero points, central wavelength for the given spectroscopic setup, readout noise, gain, illumination level for the flats, dark/bias level, etc. These parameters are used for both data quality control and for instrument health check.
What type of data does the pipeline reduce?
As a rule of thumb, the pipeline fares well on good, high signal-to-noise data and tends to fail on poor quality data. The parameters of the recipes are tuned to encompass as larger variety of scientific observations as possible. Of course, it is not possible to address all observing strategies, so do not expect the pipeline to reduce your complicated imaging program that alternates between four objects and a sky field. The pipeline is likely to fail if you do spectroscopy of faint objects, or objects with weak continuum, or a galaxy without sharp central peak. Polarimetry is not supported.
Most likely the pipeline will reduce AutoJitter-taken imaging and spectroscopy of reasonably bright stars obtained with nodding along the slit.
Consult your support astronomer to determine whether you need the pipeline products, what are the chances of getting good reduction or any other pipeline related issues and all imaging and spectroscopic daytime calibrations.
What recipes are available?
Here is a short description of the recipes, with some data quality parameters that they calculate:
- sofi_img_dark - combines dark frames, calculates bias/dark level, RoN and gain
- sofi_img_domeflat - reduces SofI imaging special dome flats, calculates the illumination level
- sofi_img_illum - imaging illumination correction
- sofi_img_jitter - reduces imaging taken with jittering templates, calculates the sky level
- sofi_img_zpoint - calculates the photometric zero points for observations of standard stars produced with the dedicated standard star template (if other imaging templates are used the recipe will ignore the standards and treat them as any science imaging data)
- sofi_spc_arc - wavl. calibrations for arcs, calculates the central wavl. for the given setting
- sofi_spc_flat - combines the spectrosocpic flats, calculates the illuminaiton level
- sofi_spc_jitter - combines 2-dimensional spectra taken with nodding along the slit after geometic correction for the distortions, extracts wavl. calibrated 1-dimensional spectrum for the brightest point source in the field
Where can I get a detailed description of the recipes? -- The SofI pipeline manual can be found at the ESO pipeline page here . In case of problems contact the User Support Department (usd-help@eso.org)
How to review the pipeline products?
The easiest way is to use Gasgano on the astro@wg5off. The raw data are in sub-dir /data/raw/2006-04-19 or similar, and the reduced data are in sub-dir /data/reduced/2006-06-24 or similar. The quality control parameters, incl. the zero points are written in *.tfits or *.paf files. The names of the reduced files starts with "r.*". Remmeber to "refresh" the Gasgano content to see the latest data products.
How to get the pipeline products? -- The pipeline products are not part of the visitor's back up. The users can ftp the reduced data from wg5off or use the CD/DVD burner on that machine to back them up by themselves. Talk to your support astronomer about this.
Workaround to save the 2D spectra in the SofI pipeline version 1.5.8 (released on 2017-05-15)
First, download the pipeline kit sofi-kit-1.5.8.tar.gz from the ESO pipeline page (ww.eso.org/sci/software/pipelines/), unzip and untar it. This will create a number of *tar.gz files in the sub-dir where you want to make the installatoin. One of these files will be sofi-1.5.8.tar.gz
At this point stop following the instructions. Instead, unzip and untar this file. It will create a sub-dir sofi-1.5.8 with a number of sub-sub-dirs. Go to sofi-1.5.8/recipes, find the file sofi_spc_jitter.c and make a back up copy, e.g.
cp sofi_spc_jitter.c sofi_spc_jitter_orig.c
open the file sofi_spc_jitter.c with any editor, and comment lines 404-410, so if you run the diff command, you will have the folling output:
diff sofi_spc_jitter.c sofi_spc_jitter_orig.c
404,410c404,410
< // cpl_msg_info(cpl_func, "Extract the spectrum");
< // cpl_msg_indent_more();
< // extracted = sofi_spc_jitter_extract(combined[0]);
< // cpl_msg_indent_less();
< //
< // error_if (extracted == NULL, CPL_ERROR_DATA_NOT_FOUND,
< // "Cannot extract the spectrum");
---
> cpl_msg_info(cpl_func, "Extract the spectrum");
> cpl_msg_indent_more();
> extracted = sofi_spc_jitter_extract(combined[0]);
> cpl_msg_indent_less();
>
> error_if (extracted == NULL, CPL_ERROR_DATA_NOT_FOUND,
> "Cannot extract the spectrum");
Go back up to the directory where you will be installing the pipeline, rename or remove the sofi-1.5.8.tar.gz and create a new one with the modified recipe:
tar cvf sofi-1.5.8.tar sofi-1.5.8
gzip sofi-1.5.8.tar
This "trick" will allow you to proceed with the standard pipeline installation procedure, and indeed, after this step you can go on by rinning
./install_pipeline
This will create a clone of the SofI pipleine that will not extract 1D spectra, instead it will save the geometrically corrected (e.g. ortogonalized) wavelength calibrated 2D spectrum.
Unfortunately, the 2D product is not easily interpreted by some widely used data reduction pieces of software.
Bellow we describe the changes that need to be made in the fits headers, so IRAF (and PyRAF) will "understand" the wavelength calibration and extract wavelength calibrated 1D spectra. It is a good idea to apply the changes described here on a working copy of the data product:
cl> imcopy sofi_spc_jitter_combined sofi_spc_jitter_combined_2
keeping a copy of the original, so in case of errors one does not have to reprocess the data.
Keywords CRVAL1, CRVAL2, CRPIX1, CRPIX2, CDELT1, CDELT, CD1_1 and CD2_2 must be edited (this is easily done with the IRAF task hedit); it is a simple swap of the value of keywords *1 and *2, in some cases with a sign change; CRPIX2 is an exception - it must be set to 1024. Bellow we give example of before and after the editing:
cl> imhead sofi_spc_jitter_combined l+ | grep CR
CRVAL1 = 19377.55859375
CRVAL2 = 1
CRPIX1 = 1
CRPIX2 = 1.
cl> imhead sofi_spc_jitter_combined_2 l+ | grep CR
CRVAL1 = 1
CRVAL2 = 19377.55859375
CRPIX1 = 1
CRPIX2 = 1024
cl> imhead sofi_spc_jitter_combined l+ | grep CD
CDELT1 = 4.78515625000007
CDELT2 = 1
CD1_1 = 4.78515625000007
CD2_2 = 1
cl> imhead sofi_spc_jitter_combined_2 l+ | grep CD
CDELT1 = 1
CDELT2 = -4.78515625000007
CD1_1 = 1
CD2_2 = -4.78515625000007
To make sure the tasks in the IRAF packages ondspec and twodspec interpret the units right, set:
cl> hedit sofi_spc_jitter_combined_2 CUNIT1='Angstrom' add+ addonly+
cl> hedit sofi_spc_jitter_combined_2 CUNIT1='Angstrom' add+ addonly+
Before proceeding to extract 1D spectra from the edited 2D spectrum with the apall task, make sure to set up
dispaxis=2
Disclaimer: This is not an official release, but a temporary workaround, to be used until the problem is solved! We take no responsibility for incorrectly reduced data.