PyDL

Introduction

This package consists of Python replacements for functions that are part of the IDL® built-in library or part of astronomical IDL® libraries. The emphasis is on reproducing results of the astronomical library functions. Only the bare minimum of IDL® built-in functions are implemented to support this.

There are four astronomical libraries targeted:

This package affiliated with the astropy project and is registered with PyPI.

IDL® is a registered trademark of Harris Geospatial Solutions.

Components

Most of the functionality of PyDL is in sub-packages.

SDSS Utilities (pydl.pydlutils)

Introduction

This package provides functionality similar to idlutils, a general suite of tools heavily used by SDSS.

idlutils is itself divided into a number of subpackages. Below we list the subpackages and the usability of the PyDL equivalent. The readiness levels are defined as:

Obsolete
No point in implementing because the purpose of the code lapsed many years ago.
Not Applicable (NA)
No point in implementing because another built-in or numpy/scipy/astropy package completely replaces this.
None
Not (yet) implemented at all.
Rudimentary
Only a few functions are implemented.
Fair
Enough functions are implemented to be useful, but some are missing.
Good
Pretty much anything you could do with the idlutils code you can do with the equivalent here.
Subpackage Readiness Level Comments
2mass None For use with matching 2MASS catalogs to SDSS data.
astrom None For use with SDSS astrometric data structures. Largely superseded by WCS.
bspline Good Fitting B-splines to data, especially for resampling.
cooling Good See pydl.pydlutils.cooling.read_ds_cooling().
coord Fair Some functionality already provided by astropy.coordinates.
cosmography NA Tools for computing lookback time, angular sizes at cosmological distances, etc. Use astropy.cosmology.
dimage None Interface to C code used for sky subtraction.
djsphot None A simple aperture photometry code.
dust None For use with the SFD galactic dust map.
first None For use with matching FIRST catalogs to SDSS data.
fits NA Use astropy.io.fits.
healpix NA Interact with HEALPix data. Use healpy.
image Rudimentary Image manipulation functions.
json NA Use json or other packages.
mangle Fair Some work still required on polygon area calculations.
math Fair Generic mathematical functions. Many are implemented in numpy or scipy.
mcmc None But there are plenty of good Python MCMC packages out there.
mglib Obsolete An IDL object-oriented configuration file reader.
misc Fair General purpose utility functions.
mpeg None Wrapper for ppmtompeg, makes movies from data.
mpfit None Appears to be an out-of-sync copy of the “markwardt” package in the The IDL® Astronomy User’s Libary.
physics None Implementation of physical formulas, e.g. free-free scattering.
plot None Much functionality already exists in matplotlib.
psf Obsolete Point-spread function fitting.
rgbcolor Good Some functionality is duplicated in astropy.visualization, especially make_lupton_rgb().
rosat None For use with matching ROSAT catalogs to SDSS data.
sdss Good Most important functionalities are bitmasks and reading sweep files.
slatec None Fit B-splines using C code.
spheregroup Good Used for matching arbitrary RA, Dec coordinates to other arbitrary RA, Dec coordinates.
TeXtoIDL NA This package is for including TeX in IDL plots. Since matplotlib understands TeX natively, this is not needed.
trace Fair Used for fitting orthogonal functions to spectroscopic wavelength solutions.
ukidss None Used for matching UKIDSS catalogs to SDSS data.
wise None Used for matching WISE catalogs to SDSS data.
yanny Good Tools for manipulating SDSS parameter files.

API

pydl.pydlutils Package

This subpackage implements functions from the idlutils package.

Classes
PydlutilsException Exceptions raised by pydl.pydlutils that don’t fit into a standard exception class like ValueError.
PydlutilsUserWarning Class for warnings issued by pydl.pydlutils.
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.PydlutilsException, pydl.pydlutils.PydlutilsUserWarning

pydl.pydlutils.bspline Module

This module corresponds to the bspline directory in idlutils.

Functions
cholesky_band(l[, mininf]) Compute lower Cholesky decomposition of a banded matrix.
cholesky_solve(a, bb) Solve the equation \(A x = b\) where a is a lower Cholesky-banded matrix.
iterfit(xdata, ydata[, invvar, upper, …]) Iteratively fit a B-spline set to data, with rejection.
Classes
bspline(x[, nord, npoly, bkpt, bkspread]) B-spline class.
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.bspline.bspline

pydl.pydlutils.cooling Module

This module corresponds to the cooling directory in idlutils.

Functions
read_ds_cooling(fname[, logT]) Read in the Sutherland & Dopita (1993) cooling function.
pydl.pydlutils.coord Module

This module corresponds to the coord directory in idlutils.

Functions
current_mjd() Return the current MJD.
munu_to_radec(munu, icrs_frame) Convert from SDSS great circle coordinates to equatorial coordinates.
radec_to_munu(icrs_frame, munu) Convert from equatorial coordinates to SDSS great circle coordinates.
stripe_to_eta(stripe) Convert from SDSS great circle coordinates to equatorial coordinates.
stripe_to_incl(stripe) Convert from SDSS stripe number to an inclination relative to the equator.
Classes
SDSSMuNu(*args[, copy, representation_type, …]) SDSS Great Circle Coordinates
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.coord.SDSSMuNu

pydl.pydlutils.image Module

This module corresponds to the image directory in idlutils.

Functions
djs_maskinterp(yval, mask[, xval, axis, const]) Interpolate over masked pixels in a vector, image or 3-D array.
djs_maskinterp1(yval, mask[, xval, const]) Interpolate over a masked, 1-d array.
pydl.pydlutils.mangle Module

This module corresponds to the mangle directory in idlutils.

Mangle [1] is a software suite that supports the concept of polygons on a sphere, and is used to, for example, describe the window function of the Sloan Digital Sky Survey.

This implementation is intended to support the portions of Mangle that are included in idlutils. To simplify the coding somewhat, unlike idlutils, the caps information is accessed through polygon.x and polygon.cm, not polygon.caps.x or polygon.caps.cm.

Window function operations are already supported by is_in_polygon() and is_in_window(). However, calculation of polygon area (solid angle) from first principles is not yet implemented.

Note that in traditional geometry “spherical polygon” means a figure bounded by great circles. Mangle allows polygons to be bounded by any circle, great or not. Use care when comparing formulas in this module to formulas in the mathematical literature.

Functions
angles_to_x(points[, latitude]) Convert spherical angles to unit Cartesian vectors.
cap_distance(x, cm, points) Compute the distance from a point to a cap, and also determine whether the point is inside or outside the cap.
circle_cap(radius, points) Construct caps based on input points (directions on the unit sphere).
is_cap_used(use_caps, i) Returns True if a cap is used.
is_in_cap(x, cm, points) Are the points in a (single) given cap?
is_in_polygon(polygon, points[, ncaps]) Are the points in a given (single) polygon?
is_in_window(polygons, points[, ncaps]) Check to see if points lie within a set of polygons.
read_fits_polygons(filename[, convert]) Read a “polygon” format FITS file.
read_mangle_polygons(filename) Read a “polygon” format ASCII file in Mangle’s own format.
set_use_caps(polygon, index_list[, add, …]) Set the bits in use_caps for a polygon.
x_to_angles(points[, latitude]) Convert unit Cartesian vectors to spherical angles.
Classes
FITS_polygon Handle polygons read in from a FITS file.
ManglePolygon(*args, **kwargs) Simple object to represent a polygon.
PolygonList(*args, **kwargs) A list that contains ManglePolygon objects and possibly some metadata.
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.mangle.FITS_polygon, pydl.pydlutils.mangle.ManglePolygon, pydl.pydlutils.mangle.PolygonList

pydl.pydlutils.math Module

This module corresponds to the math directory in idlutils.

Functions
djs_median(array[, dimension, width, boundary]) Compute the median of an array.
djs_reject(data, model[, outmask, inmask, …]) Routine to reject points when doing an iterative fit to data.
find_contiguous(x) Find the longest sequence of contiguous non-zero array elements.
Classes
computechi2(bvec, sqivar, amatrix) Solve the linear set of equations \(A x = b\) using SVD.
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.math.computechi2

pydl.pydlutils.misc Module

This module corresponds to the misc directory in idlutils.

Functions
decode_mixed(x) Convert bytes in Numpy arrays into strings.
djs_laxisgen(dims[, iaxis]) Returns an integer array where each element of the array is set equal to its index number along the specified axis.
djs_laxisnum(dims[, iaxis]) Returns an integer array where each element of the array is set equal to its index number in the specified axis.
hogg_iau_name(ra, dec[, prefix, precision]) Properly format astronomical source names to the IAU convention.
hogg_iau_name_main()
struct_print(array[, filename, formatcodes, …]) Print a NumPy record array (analogous to an IDL structure) in a nice way.
pydl.pydlutils.rgbcolor Module

This module corresponds to the rgbcolor directory in idlutils.

This code reproduces the algorithms of Lupton et al. (2004) [2]. The purpose is to produce nice color (RGB, JPEG, PNG, etc.) from FITS images in astronomical filter bands.

Functions
nw_arcsinh(colors[, nonlinearity]) Scales colors by a degree of nonlinearity specified by user.
nw_cut_to_box(colors[, origin]) Limits the pixel values of the image to a ‘box’, so that the colors do not saturate to white but to a specific color.
nw_float_to_byte(image[, bits]) Converts an array of floats in [0.0, 1.0] to bytes in [0, 255].
nw_scale_rgb(colors[, scales]) Multiply RGB image by color-dependent scale factor.
pydl.pydlutils.sdss Module

This module corresponds to the sdss directory in idlutils.

Functions
default_skyversion() Returns skyversion number to use for photoop imaging.
sdss_astrombad(run, camcol, field[, …]) For a list of RUN, CAMCOL, FIELD, return whether each field has bad astrometry.
sdss_flagexist(flagname, bitname[, …]) Check for the existence of flags.
sdss_flagname(flagname, flagvalue[, concat]) Return a list of flag names corresponding to the values.
sdss_flagval(flagname, bitname) Convert bitmask names into values.
sdss_objid(run, camcol, field, objnum[, …]) Convert SDSS photometric identifiers into CAS-style ObjID.
sdss_specobjid(plate, fiber, mjd, run2d[, …]) Convert SDSS spectrum identifiers into CAS-style specObjID.
sdss_sweep_circle(ra, dec, radius[, stype, …]) Read the SDSS datasweep files and return objects around a location.
set_maskbits([idlutils_version, maskbits_file]) Populate the maskbits cache.
unwrap_specobjid(specObjID[, run2d_integer, …]) Unwrap CAS-style specObjID into plate, fiber, mjd, run2d.
pydl.pydlutils.spheregroup Module

This module corresponds to the spheregroup directory in idlutils.

Functions
spheregroup(ra, dec, linklength[, chunksize]) Perform friends-of-friends grouping given ra/dec coordinates.
spherematch(ra1, dec1, ra2, dec2, matchlength) Match points on a sphere.
Classes
chunks(ra, dec, minSize) chunks class
groups(coordinates, distance[, separation]) Group a set of objects (a list of coordinates in some space) based on a friends-of-friends algorithm
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.spheregroup.chunks, pydl.pydlutils.spheregroup.groups

pydl.pydlutils.trace Module

This module corresponds to the trace directory in idlutils.

Functions
fchebyshev(x, m) Compute the first m Chebyshev polynomials.
fchebyshev_split(x, m) Compute the first m Chebyshev polynomials, but modified to allow a split in the baseline at \(x=0\).
fpoly(x, m) Compute the first m simple polynomials.
func_fit(x, y, ncoeff[, invvar, …]) Fit x, y positions to a functional form.
traceset2xy(tset[, xpos, ignore_jump]) Convert from a trace set to an array of x,y positions.
xy2traceset(xpos, ypos, **kwargs) Convert from x,y positions to a trace set.
Classes
TraceSet(*args, **kwargs) Implements the idea of a trace set.
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.trace.TraceSet

pydl.pydlutils.yanny Module

This module corresponds to the yanny directory in idlutils.

This is a Python library for reading & writing yanny files.

yanny is an object-oriented interface to SDSS Parameter files (a.k.a. “FTCL” or “yanny” files) following these specifications. Parameter files typically have and can be recognized by the extension .par. These files may also be recognized if the first line of the file is:

#%yanny

This is not part of the standard specification, but it suffices to identify, e.g., files that have not yet been written to disk, but only exist as file objects.

Because Parameter files can contain multiple tables, as well as metadata, there is no simple, one-to-one correspondence between these files and, say, an astropy Table object. Thus, yanny objects are implemented as a subclass of OrderedDict (to remember the order of keyword-value pairs), and the actual data are values accessed by keyword. Still, it is certainly possible to write a table-like object to a yanny file.

Given the caveats above, we have introduced experimental support for reading and writing yanny files directly to/from Table objects. Because of the experimental nature of this support, it must be activated “by hand” by including this snippet in your code:

from astropy.table import Table
from astropy.io.registry import (register_identifier, register_reader,
                                 register_writer)
from pydl.pydlutils.yanny import (is_yanny, read_table_yanny,
                                  write_table_yanny)

register_identifier('yanny', Table, is_yanny)
register_reader('yanny', Table, read_table_yanny)
register_writer('yanny', Table, write_table_yanny)

Currently multidimensional arrays are only supported for type char, and a close reading of the specifications indicates that multidimensional arrays were only ever intended to be supported for type char. So no multidimensional arrays, sorry.

Functions
is_yanny(origin, path, fileobj, *args, **kwargs) Identifies Yanny files or objects.
read_table_yanny(filename[, tablename]) Read a yanny file into a Table.
write_ndarray_to_yanny(filename, datatables) Converts a NumPy record array into a new FTCL/yanny file.
write_table_yanny(table, filename[, …]) Write a Table to a yanny file.
Classes
yanny([filename, raw]) An object interface to a yanny file.
Class Inheritance Diagram

Inheritance diagram of pydl.pydlutils.yanny.yanny

Goddard Utilities (pydl.goddard)

Introduction

This package provides functionality similar to the The IDL® Astronomy User’s Libary, sometimes called the “Goddard Utilities”, maintained by Wayne Landsman and distributed with idlutils.

In general, functions that are needed by pydl.pydlutils or pydl.pydlspec2d are implemented, while functions that are not needed have much lower priority.

The Goddard package is itself divided into a number of subpackages. Below we list the subpackages and the usability of the PyDL equivalent. The readiness levels are defined as:

Obsolete
No point in implementing because the purpose of the code lapsed many years ago.
Not Applicable (NA)
No point in implementing because another built-in or numpy/scipy/astropy package completely replaces this.
None
Not (yet) implemented at all.
Rudimentary
Only a few functions are implemented.
Fair
Enough functions are implemented to be useful, but some are missing.
Good
Pretty much anything you could do with the Goddard code you can do with the equivalent here.
Subpackage Readiness Level Comments
astro Rudimentary General astronomical utility functions.
astrom NA Tools for manipulating WCS data in FITS headers. Use astropy.io.fits and astropy.wcs.
coyote NA The Coyote library for plotting and graphics developed by David Fanning.
database None Allows access to IDL-specific databases.
disk_io None Provides access to IRAF image (.imh) files and AJ/ApJ-style tables.
fits NA Use astropy.io.fits.
fits_bintable NA Use astropy.io.fits.
fits_table NA Use astropy.io.fits.
idlphot None Adapted from an early version of DAOPHOT.
image None Generic image processing functions, including convolution/deconvolution.
jhuapl None Functions from the JHU Applied Physics Lab.
markwardt None Levenberg-Marquardt least-squares minimization.
math Rudimentary Generic mathematical functions. Many are implemented in numpy or scipy.
misc Rudimentary General utility functions that do not involve astronomy specifically.
plot NA Functions that supplement the built-in IDL plotting capabilities.
robust None Robust statistical fitting procedures.
sdas None Provides access to STDAS/GEIS image files.
sockets NA Functions for performing web queries in IDL. Use astroquery.
structure NA Tools for manipulating IDL data structures. Use numpy.recarray.
tv NA Functions for manipulating IDL image displays.

API

pydl.goddard Package

This subpackage contains the Goddard utilities.

pydl.goddard.astro Module

This module corresponds to the goddard/astro directory in idlutils.

Functions
airtovac(air) Convert air wavelengths to wavelengths in vacuum.
gcirc(ra1, dec1, ra2, dec2[, units]) Computes rigorous great circle arc distances.
get_juldate([seconds]) Returns the current Julian date.
get_juldate_main() Entry point for the get_juldate command-line script.
vactoair(vacuum) Convert vacuum wavelengths to wavelengths in air.
pydl.goddard.math Module

This module corresponds to the goddard/math directory in idlutils.

Functions
flegendre(x, m) Compute the first m Legendre polynomials.
pydl.goddard.misc Module

This module corresponds to the goddard/misc directory in idlutils.

Functions
cirrange(ang[, radians]) Convert an angle larger than 360 degrees to one less than 360 degrees.

SDSS Spectroscopic Data (pydl.pydlspec2d)

Introduction

This package provides functionality in the SDSS idlspec2d package. This package is used for processing and analyzing data from the SDSS optical spectrographs. The code is thus relevant to the SDSS Legacy, BOSS and eBOSS surveys. This package does not work with any infrared spectrograph data associated with the APOGEE-2 survey.

The primary technical focus of this particular implementation is the function combine1fiber(). This function is responsible for resampling 1D spectra onto a new wavelength solution. This allows for:

  1. Shifting a spectrum from observed redshift to rest frame.
  2. Coaddition of spectra of the same object, after resampling all spectra onto the same wavelength solution.

The primary scientific motivation of implementing combine1fiber() is to create template spectra based on curated spectra of, e.g., luminous red galaxies (LRGs). Principal Component Analysis (PCA) or other techniques may be used to construct template spectra, but putting all spectra on the same rest-frame wavelength solution is the first step.

The idlspec2d package is itself divided into a number of subpackages. Below we list the subpackages and the usability of the PyDL equivalent. The readiness levels are defined as:

Obsolete
No point in implementing because the purpose of the code lapsed many years ago.
Not Applicable (NA)
No point in implementing because another built-in or numpy/scipy/astropy package completely replaces this.
None
Not (yet) implemented at all.
Rudimentary
Only a few functions are implemented.
Fair
Enough functions are implemented to be useful, but some are missing.
Good
Pretty much anything you could do with the idlspec2d code you can do with the equivalent here.
Subpackage Readiness Level Comments
apo2d None Quick extraction code for quality assurance at observation time.
config None Extraction pipeline configuration parameters in object-oriented IDL.
fluxfix None Flux calibration
guider None Interface to guider camera.
inspect None Tools for manual inspection of spectra.
photoz Obsolete Photometric redshifts for SDSS objects using spectral templates.
plan None Tools for planning exposures and recordings summaries of exposures.
plate Obsolete Tools for designing SDSS spectroscopic plates, especially for star clusters.
science None Code for science analysis of sets of 1D spectra.
spec1d Fair Tools for processing 1D spectra, including redshift fitting.
spec2d Fair Tools for extracting spectra from 2D images.
specdb Obsolete Tests on storing spectroscopic results in SQL databases.
specflat None Flat-fielding of spectroscopic 2D images.
templates None Tools for constructing spectroscopic templates.
testsuite None Tools for high-level quality assurance, e.g. comparing two reductions of the same data.

API

pydl.pydlspec2d Package

This subpackage implements functions from the idlspec2d package.

Classes
Pydlspec2dException Exceptions raised by pydl.pydlspec2d that don’t fit into a standard exception class like ValueError.
Pydlspec2dUserWarning Class for warnings issued by pydl.pydlspec2d.
Class Inheritance Diagram

Inheritance diagram of pydl.pydlspec2d.Pydlspec2dException, pydl.pydlspec2d.Pydlspec2dUserWarning

pydl.pydlspec2d.spec1d Module

This module corresponds to the spec1d directory in idlspec2d.

Functions
findspec(*args, **kwargs) Find SDSS/BOSS spectra that match a given RA, Dec.
latest_mjd(plate, **kwargs) Find the most recent MJD associated with a plate.
number_of_fibers(plate, **kwargs) Returns the total number of fibers per plate.
pca_solve(newflux, newivar[, maxiter, …]) Replacement for idlspec2d pca_solve.pro.
plot_eig(filename[, title]) Plot spectra from an eigenspectra/template file.
preprocess_spectra(flux, ivar[, loglam, …]) Handle the processing of input spectra through the combine1fiber() stage.
readspec(platein[, mjd, fiber]) Read SDSS/BOSS spec2d & spec1d files.
skymask(invvar, andmask[, ormask, ngrow]) Mask regions where sky-subtraction errors are expected to dominate.
spec_append(spec1, spec2[, pixshift]) Append the array spec2 to the array spec1 & return a new array.
spec_path(plate[, path, topdir, run2d]) Return the directory containing spPlate files.
template_input(inputfile, dumpfile[, flux, …]) Collect spectra and pass them to PCA or HMF solvers to compute spectral templates.
template_input_main() Entry point for the compute_templates script.
template_metadata(inputfile[, verbose]) Read template metadata from file.
template_qso(metadata, newflux, newivar[, …]) Run PCA or HMF on QSO spectra.
template_star(metadata, newloglam, newflux, …) Run PCA or HMF on stellar spectra of various classes.
wavevector(minfullwave, maxfullwave[, …]) Return an array of wavelengths.
Classes
HMF(spectra, invvar[, K, n_iter, seed, …]) Class used to manage data for Heteroscedastic Matrix Factorization (HMF).
Class Inheritance Diagram

Inheritance diagram of pydl.pydlspec2d.spec1d.HMF

pydl.pydlspec2d.spec2d Module

This module corresponds to the spec2d directory in idlspec2d.

Functions
aesthetics(flux, invvar[, method]) Add nice values to a spectrum where it is masked.
combine1fiber(inloglam, objflux, newloglam) Combine several spectra of the same object, or resample a single spectrum.
filter_thru(flux[, waveimg, wset, mask, …]) Compute throughput in SDSS filters.

SDSS Imaging Data (pydl.photoop)

Introduction

The photoop package is used to process SDSS imaging data. This package is used to reduce, resolve and flux-calibrate the SDSS raw data, resulting in both flux-calibrated images and catalogs.

SDSS ceased taking imaging data in 2009, and there has only been one full processing of the imaging data since then, although adjustments have been made to the astrometry and flux calibration.

The primary emphasis of this implementation is:

  1. Functions related to the SDSS photometric objID, which is a unique integer used in SDSS databases, constructed from quantities that specify a particular astronomical object on a particular image.
  2. Functions related to finding SDSS photometric data on disk.
  3. The SDSS “window” function, which defines what parts of the sky are covered by SDSS images. This can be used in conjunction with the mangle module to find points and regions that have SDSS imaging.
  4. In general, functions that work with existing imaging data, rather than functions to reduce the data.

The photoop package is itself divided into a number of subpackages. Below we list the subpackages and the usability of the PyDL equivalent. The readiness levels are defined as:

Obsolete
No point in implementing because the purpose of the code lapsed many years ago.
Not Applicable (NA)
No point in implementing because another built-in or numpy/scipy/astropy package completely replaces this.
None
Not (yet) implemented at all.
Rudimentary
Only a few functions are implemented.
Fair
Enough functions are implemented to be useful, but some are missing.
Good
Pretty much anything you could do with the photoop code you can do with the equivalent here.
Subpackage Readiness Level Comments
apache Obsolete Processing of “Apache Wheel” images, used for calibration.
astrom None Astrometry for SDSS images.
atlas None Construction of “atlas” images, small cutouts of individual objects.
bluetip None Tools for “blue-tip” photometry and extinction estimation.
compare Obsolete Compare the same images in two different data reductions.
database Obsolete Experimental database loading code.
flats None Analysis of flat-field files.
hoggpipe Obsolete Another version of “Apache Wheel” processing code.
image None Tools for creating “corrected frame” images. These are flux-calibrated and sky-subtracted images with physical flux units.
ircam Obsolete Tools for processing all-sky “cloud camera” images, used to establish photometricity.
match None Tools for matching SDSS spectra to corresponding photometric objects.
misc None Code with no obvious home in any other category.
pcalib None Tools related to “ubercalibration”.
photoobj Rudimentary Tools for creating calibrated catalogs from images.
plan Obsolete Tools for planning photometric reductions.
plots None Plots for high-level quality assurance.
psf None Analysis of point-spread functions.
ptcalib Obsolete Processing of “photometric telescope” data, an obsolete technique for flux-calibration.
resolve None Code for the resolve stage of image processing.
sdss3_runqa Obsolete Quality assurance tests from the most recent photometric reduction.
sdssio Rudimentary Tools for reading and writing various data files produced by the photometric reductions.
window Rudimentary Tools for determining the sky coverage of the survey.

API

pydl.photoop Package

This subpackage implements functions from the photoop package.

Classes
PhotoopException Exceptions raised by pydl.photoop that don’t fit into a standard exception class like ValueError.
Class Inheritance Diagram

Inheritance diagram of pydl.photoop.PhotoopException

pydl.photoop.photoobj Module

This module corresponds to the photoobj directory in photoop.

Functions
sdss_calibv() Return calibration for velocities from pix/frame to deg/day.
unwrap_objid(objid) Unwrap CAS-style objID into run, camcol, field, id, rerun.
pydl.photoop.sdssio Module

This module corresponds to the sdssio directory of photoop.

Functions
filtername(f) Return the name of a filter given its number.
filternum([filt]) Return index number for SDSS filters either from a number or name.
sdss_calib(run, camcol, field[, rerun]) Read photometric calibration solutions from calibPhotom or calibPhotomGlobal files.
sdss_name(ftype, run, camcol, field[, …]) Return the name of an SDSS data file including path.
sdss_path(ftype, run[, camcol, rerun]) Return the path name for SDSS data assuming SAS directory structure.
sdssflux2ab(flux[, magnitude, ivar]) Convert the SDSS calibrated fluxes (magnitudes) into AB fluxes (magnitudes).
pydl.photoop.window Module

This module corresponds to the window directory in photoop.

Functions
sdss_score(flist[, silent]) Score a list of imaging fields from zero to one.
window_read(**kwargs) Read window files in $PHOTO_RESOLVE.
window_score(**kwargs) For uber-resolve, score all the fields from zero to one.

Other Notes

SDSS Spectroscopic Templates with PyDL

One of the original motivations for creating the PyDL package was to reproduce and, ultimately, improve the method for generating spectroscopic templates for the Sloan Digital Sky Survey (SDSS). The spectroscopic templates are used by the “1D” portion of the SDSS spectroscopic pipeline (Bolton et al. 2012 [1]; See also pydl.pydlspec2d). Historically, those were generated by:

  1. Selecting individual objects (galaxies, QSOs, stars, white dwarfs, etc.) with good SDSS spectra and reliable classifications.
  2. For a given set of objects, resample the spectra onto a common wavelength grid and redshift to rest frame.
  3. Perform an iterative PCA analysis on the flux versus wavelength matrix.
  4. Select the most significant eigenspectra as the templates. Typically the first four eigenspectra were chosen.

For SDSS-I/II, the input spectra were all high signal-to-noise, but as the BOSS survey in SDSS-III started to target more distant galaxies, there was interest in improving the robustness of the PCA fitting to handle low signal-to-noise inputs. In addition, full alternatives to PCA, such has Heteroscedastic Matrix Factorization (HMF) [2], [3] could be compared.

Here are the algorithmic steps taken in the code to implement the procedure described above.

  1. template_input() reads a configuration file that includes a list of individual spectra. There is one configuration file for each type of object (galaxy, QSO, etc.).
  2. readspec() reads each spectrum. Masked areas are identified by setting the per-spectrum inverse variance to zero as needed.
  3. A new wavelength grid is created. SDSS typically uses wavelength grids that are evenly spaced in \(\log_{10}\) wavelength.
  4. preprocess_spectra() calls combine1fiber() to perform the resampling and rest frame shift.
  5. pca_solve() or HMF are used to obtain the eigenspectra.
  6. The eigenspectra are written to a FITS file.

At least historically, there was no test data set on which to (unit) test these algorithms. The best one could do was to take the inputs and compare them, function-by-function, to the equivalent IDL® code, ensuring that the results were the same, to some numerical precision.

[1]Bolton, Adam, et al. 2012 AJ 144, 144.
[2]Tsalmantza, P., Decarli, R., Dotti, M., Hogg, D. W., 2011 ApJ 738, 20
[3]Tsalmantza, P., Hogg, D. W., 2012 ApJ 753, 122

PyDL Changelog

1.0.0 (unreleased)

This version will only support Python 3 and Astropy 3. This release is planned for early 2019.

0.7.0 (2019-02-22)

This version is planned to be the last version with Python 2 support.

  • Support the firstField bit in ObjIDs from DR7 and earlier (Issue #37).
  • Change tests of Astropy development version from Python 2 to Python 3.
  • Update to astropy_helpers/v2.0.6 (PR #40).
  • Add astropy.units support to airtovac() and vactoair() (PR #41).
  • Change Exelis to Harris Geospatial (PR #42).
  • Fix FutureWarning in re in Python 3.7 due to nested sets (PR #44).
  • Use six instead of astropy.extern.six (PR #48).
  • Update _astropy_init.py (PR #47 via PR #48).
  • Update astropy_helpers/v2.0.8 (PR #45 via PR #48).

0.6.0 (2017-09-19)

0.5.4 (2017-05-04)

  • Added sdss_specobjid() to compute SDSS specObjIDs, and its inverse function unwrap_specobjid().
  • Update to astropy_helpers/v1.3.1.
  • Refactor HMF code into an object to contain the data and methods.
  • Use functions from astropy.utils.data where possible.
  • Fix an integer division error encountered when using Numpy 1.12 (Issue #19).
  • Fixed tests that were failing on 32-bit platforms and Python 3.5 (Issue #20).

0.5.3 (2016-12-03)

  • Fixed formatting of TODO document.
  • Fixed tests that were failing on 32-bit platforms (Issue #14).
  • Use temporary files so that tests can run when astropy is installed read-only (e.g., with pip; Issue #16)

0.5.2 (2016-08-04)

  • Changes in how Mangle-polygon containing FITS files are handled, related to Issue #11.
  • Fixed memory leak in combine1fiber(), see Issue #12.
  • Added is_in_window().
  • Allow polygon area functions to deal with negative caps and use_caps.
  • Update docs/conf.py for Python 3.5 compatibility (PR #13).

0.5.1 (2016-06-22)

  • Removed unnecessary from __future__ import in pydl.pydlspec2d.spec1d.
  • Ongoing documentation upgrades.
  • Update some links that needed to be transitioned from SDSS-III to SDSS-IV.
  • Upgrade to astropy_helpers/v1.2.
  • Update to latest version of package-template.
  • Disabled tests on Python 3.3; enabled tests on Python 3.5
  • Fix Issue #8; Issue #9.
  • Add warnings about incomplete Mangle functions.

0.5.0 (2016-05-01)

  • Dropped support for Python 2.6. Python 2.6 does not contain collections.OrderedDict, which is needed to support yanny objects, and at this point it is not worth going to the trouble to support this with an external package.
  • Ongoing review and upgrade of docstrings.
  • Yanny files can now be converted into genuine NumPy record arrays; previously, the conversion was only to numpy.ndarray with named columns, which is a slightly different thing.
  • Added additional tests on yanny objects.
  • Experimental support for interconversion of yanny objects and Table objects.
  • Improving PEP 8 compliance
  • Restructuing sub-packages to reduce the number of files.
  • Improvements to spectral template processing code, deduplicated some code.
  • Support platform-independent home directory (PR #7).
  • Uppercase the package name (in documentation only).
  • Upgrade to astropy_helpers/v1.1.1.
  • Add functions from the idlutils rgbcolor directory.
  • spec_path() can now find SDSS spectra, not just BOSS.

0.4.1 (2015-09-22)

  • No changes at all from 0.4.0. This tag only exists because of a botched PyPI upload.

0.4.0 (2015-09-22)

  • Use astropy_helpers/v1.0.3, package-template/v1.0.
  • Remove some old FITS code that astropy.io.fits makes moot.
  • Remove code for command-line scripts. These are now auto-generated by the “entry_point” method.
  • Remove Python/3.2 tests.
  • Improved test coverage.
  • Fixed problem with the spheregroup code.
  • Removed some code that is 100% redundant with astropy (e.g. angles_to_xyz()).
  • Fixed bug in set_use_caps() that was discovered on the IDL side.
  • Updated documentation of read_fits_polygons().
  • Added cross-references to classes, functions, etc.

0.3.0 (2015-02-20)

0.2.3 (2014-07-22)

0.2.2 (2014-05-07)

0.2.1 (2013-10-06)

0.2.0 (2013-04-22)

  • Using the astropy package-template to bring pydl into astropy-compatible form.
  • Some but not all tests are re-implemented.

0.1.1 (2013-03-06)

  • Creating a tag representing the state immediately after creation of the git repository.

0.1 (2010-11-10)

TODO

Authors and Credits

Thank you to everyone who has contributed to PyDL. If we have neglected to include you, please let us know right away!

Licenses

PyDL License

PyDL is licensed under a 3-clause BSD style license:

Copyright (c) 2010-2018, Benjamin Alan Weaver <baweaver@lbl.gov> All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
  • Neither the name of the Astropy Team nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Base API

pydl Package

Python replacements for functions that are part of the IDL built-in library, or part of astronomical IDL libraries. The emphasis is on reproducing results of the astronomical library functions. Only the bare minimum of IDL built-in functions are implemented to support this.

Functions

file_lines(path[, compress]) Replicates the IDL FILE_LINES() function.
median(array[, width, axis, even]) Replicate the IDL MEDIAN() function.
rebin(x, d[, sample]) Resize x to new dimensions given by d.
smooth(signal, owidth[, edge_truncate]) Replicates the IDL SMOOTH() function.
uniq(x[, index]) Replicates the IDL UNIQ() function.

Classes

pcomp(x[, standardize, covariance]) Replicates the IDL PCOMP() function.
PydlException Base class for exceptions raised in PyDL functions.