hetdex_api modules

hetdex_api.detections module

Initiates the Detections class.

Created on 2019/01/28

@author: Erin Mentuch Cooper

class hetdex_api.detections.Detections(survey='hdr4', catalog_type='lines', curated_version=None, loadtable=False, verbose=False, searchable=False, recentbadamp=False)

Bases: object

add_hetdex_gmag(loadpickle=True, picklefile='gmags.pickle'): Calculates g-band magnitude from spec1d for each detectid in the Detections class instance If the gmags.pickle file and pickle=True is given then it will just load from previous computation

close()

find_match(coord, radius=None, wave=None, dwave=5.0, shotid=None)

Function to cross match another line detection

Parameters

coord: an astropy coordinates object
wave: central wavelength in AA you want to search. If nothing is given, it will search without any wavelength contraint
radius: search radius. An astropy quantity
dwave: delta wavelength to search
shotid: optional shotid for a specific observation

Returns

match_index: index of matches

get_coord(detectid_i)

Return SkyCoord object for a single detection

Parameters

detectid: int: detectid (integer ID) of the detection you want to get SkyCoord object for

Return

coords: SkyCoord Class object

get_detection_flags(detectid, F=None): Parameters

detectid int

F: Fibers class object Optional to pass through the Fibers class so that it does not have to be reopen for each detectid

Returns

flag diction keys are flag_pixmask, flag_badamp, flag_badpix, flag_badfib, flag_meteor, flag_largegal, flag_chi2fib 1 if good, 0 if flagged bad

get_detection_info(detectid_i, rawh5=False, verbose=False)

Returns Detections table information from H5 file Applies relevent corrections such as noise model fix and spectral adjustment if rawh5 is False

Parameters

detectid: int: detectid (integer ID) of the detection you want to get information for
rawh5: bool: if True, this will simply return the row from the detecth5 file. If False (the default), any relevent correcctions are applied.
verbose: bool: provide info statements if set to True. Default is False

Returns

det_info: ndarray object: returns detection info column info given by det_info.dtype attribute

get_fiber_info(detectid_i)

Return Fibers table for a single detectid

Parameters

detectid: int: detectid (integer ID) of the detection you want to get the fiber information

Returns

fiber_info: ndarray object: table of fiber information for the specific detection. Use fiber_info.dtype to get column info

get_gband_mag(detectid_i): Calculates the gband magnitude from the 1D spectrum

get_hetdex_mag(detectid_i, filter='sdss2010-g')

filter = can be any filter used in the speclite: package https://speclite.readthedocs.io/en/latest/api.html

get_spectrum(detectid_i, deredden=False, apply_extinction_fix=True, add_apcor=False, rawh5=False, verbose=False, ffsky=False)

Grabs the 1D spectrum used to measure fitted parameters.

Parameters

detectid: int: detectid (integer ID) of the detection you want to grab the spectrum for
deredden: bool: flag to return dust corrected spectrum
apply_extinction_fix: bool: HDR2 correction. Does not apply to any other release
add_apcor: bool: add a column with the applied aperture correction
rawh5: bool: if False, this will convert to 1AA binning and apply any relevent spectral correcctions. This is the default if True, it will only pull the spetral data from the detections h5 file
verbose: bool: provide info statements if set to True. Default is False
ffsky: bool: option to use full frame sky calibration. Default is False

Returns

spec_table: astropy table: 2 or 3 column astropy table with spectral data

get_survey_info(detectid_i)

Return Survey class info for detectid

Parameters

detectid: int: detectid (integer ID) of the detection you want to grab the spectrum for

Returns

survey_info: ndarray object: row of information from the survey_hdrX.h5 table column info given by survey_info.dtype attribute

plot_spectrum(detectid_i, xlim=None, ylim=None)

query_by_coord(coord, radius=None, astropy=True)

Returns list of detecitds or astropy table object based on a coordinate search

Paramters

self: Detections Class object. If catalog_type==’index’, healpix searching is done and is most efficient. Can also be applied to other catalog_types with the searchable==True or loadtable==True option set
coord - astropy coordinate object: only works for a single coordinate objects
radius: an astropy Quantity object
astropy bool: if True will return an astropy table of detection info if False will return detectid list

Returns

Detectid or Table

query_by_dictionary(limits)

Note: This function is pretty obsolete. You should use the source catalog

Takes a dictionary of query limits and reduces the detections database. This can either be taken from the dictionary saved by detwidgets.py GUI or can take the following form:

# limits to be returned to query detections

self.wave_low = None self.wave_high = None self.flux_low = None self.flux_high = None self.linewidth_low = None self.linewidth_high = None self.sn_low = None self.sn_high = None self.chi2_low = None self.chi2_high = None self.cont_low = None self.cont_high = None self.aperture_flag = False self.ra = None self.dec = None self.rad = None self.field = None

Dictionary Description aperture_flat = when True, will query for defined aperture ra = right ascension for aperture in degrees dec = declination of aperture in degrees rad = radius of aperture in arcmin field = (‘all’, ‘dex-spring’, ‘dex-fall’, ‘cosmos’, ‘egs’, ‘goods-n’, ‘other’)

others should be obvious

query_by_pickle(picklefile)

this function queries the Detections class object based on query made with detwidgets.py

Input

self = a detections class object picklefile = string filename for a pickle created in detwidget.py

refine(gmagcut=None, remove_large_gal=True, d25scale=1.5)

Masks out bad and bright detections and returns a refined Detections class object

gmagcut = mag limit to exclude everything: brighter, defaults to None

remove_bad_amps(): Reads in the bad amp list from config.py and creates a mask to remove those detections. It will also assign a 0 to signify an artifact in vis_class

remove_bad_detects(): Reads in the bad detect list from config.py and removes those detectids Set these to -2 (they are software errors) Don’t use for machine learning or other classifying algorithms tests.

remove_bad_pix(): Refines catalog based on badpixel lists found for the release in hetdex-api: https://github.com/HETDEX/hetdex_api/tree/master/known_issues/hdr4/badpix.list

remove_balmerdip_stars()

Applies a cut to the databased to remove all stars that show a false emission feature around 3780 Also assigns a star classification to each detectid

This is obsolete as gmag cuts get rid of these easily

remove_bright_stuff(gmagcut): Applies a cut to remove bright stars based on gmag attribute. Assigns a star classification to each detectid, although some may be nearby galaxies. Will want to improve this if you are looking to study nearby galaxies.

remove_ccd_features(): Remove all objects with very at the edges of the detectors and denotes them as artifacts in vis_class

remove_large_gal(d25scale=1.5)

Returns boolean mask with detections landing within galaxy defined by d25scale flagged as False.

Based on check_all_large_gal from hetdex_tools/galmask.py written by John Feldmeier

remove_meteors(): Returns boolean mask with detections landing on meteor streaks masked. Use np.invert(mask) to find meteors

remove_shots(): Takes a list of bad shots and removes them. Assigns -2 to detections in these shots so they are not used in any MLing analysis

return_astropy_table()

Return an astropy table version of the Detections that can easily be saved

Returns

tableastropy.table:Table: an astropy table you can save

save_spectrum(detectid_i, outfile=None)

hetdex_api.extinction module

Scripts to deal with extinction.

This includes HDR2.1 flat extinction fix

hetdex_api.extinction.deredden_spectra(wave, coords)

Apply S&F 2011 Extinction from SFD Map https://iopscience.iop.org/article/10.1088/0004-637X/737/2/103#apj398709t6

Paramters

wave array: wavelength to apply correction
coords SkyCoord object: sky coordinates

hetdex_api.extinction.dustmaps_setup()

hetdex_api.extinction.get_2pt1_extinction_fix(pad=True)

This is to fix the E(B-V)=0.02 flat extinction applied to 2.1

Curve from Karl /work/00115/gebhardt/fluxcor/extinction

Paramaters

pad bool This is to pad curve to 3470 and 5400 to match wave_rect

hetdex_api.extract module

Created on Mon Mar 11 11:48:55 2019

Note: this uses nway and pandas

@author: gregz

hetdex_api.extract.sclean(waves, fiber_data, fiber_error, mask, npix=5, error_scale=4.0)

A version of sclean from karlspipe/fitrsdecsp.f. Bad data is replaced with the average of a +/- npix window around the nearest element. The error is replaced with error_scale times the error of the nearest element. These value where tuned by K. Gebhardt to optimise the detections and minimise false detections.

Parameters

fiber_data, fiber_error1D array: the data and error arrays of the fiber
mask1D array: the mask for this fiber (so the bad data can be unmasked after replacement)
npixint: the size of the +/- pixel range to average to find the replacement value
error_scalefloat: scale the error by this when replacing

Returns

fiber_data_out, fiber_error_out, mask1D array: the input data with bad values replaced and unmasked

hetdex_api.mask module

API to masking HETDEX data products

Created on 2020/09/08 Update on 2020/11/10: Updating to create galaxy regions from rc3 catalog

@author: Erin Mentuch Cooper

hetdex_api.mask.amp_flag_from_closest_fiber(coords, FibIndex, bad_amps_table, shotid=None, maxdistance=None)

Function to retrieve the amp flag for the closest fiberid in a shot

Parameters

self: the FiberIndex class for a specific survey
coords: coordinate you want to search for the closest fiber. This should be an astropy SkyCoord object
FibIndex: a hetdex_api.survey FiberIndex class object
bad_amps_table: astropy table containing the bad amp flag values. This can be retrieved from config.badamp
shotid: Specific shotid (dtype=int) you want
maxdistance: The max distance you want to search for a nearby fiber. Default is 8.*u.arcsec

Returns

bool: None if no matching fiber is found to the coords True if amp is usable False if any fiber in the defined region is flagged in a bad amp

hetdex_api.mask.amp_flag_from_coords(coords, FibIndex, bad_amps_table, radius=None, shotid=None)

Returns a boolean flag whether the amp has been flagged usable

Parameters

coords: an astropy.coordinates SkyCoord object
FibIndex: a hetdex_api.survey FiberIndex class object
bad_amps_table: astropy table containing the bad amp flag values. This can be retrieved from config.badamp
radius: radius to search for fibers
shotid: shotid to search. If none it will search all shots at once. If any are flagged bad then it will return False for all.

Returns

None if no matching fiber is found in the aperture

True if all fibers in aperture are on good amps

False if any fiber in the defined region is flagged in a bad amp

Examples

from hetdex_api.config import HDRconfig from hetdex_api.mask import *

config = HDRconfig() bad_amps_table = Table.read(config.badamp) FibIndex = FiberIndex()

coords = SkyCoord(11.628530 * u.deg, 0.081790 * u.deg, frame=’icrs’)

flag = amp_flag_from_coords(coords, FibIndex, bad_amps_table,: radius=1.5*u.arcsec, shotid=20181003009)

hetdex_api.mask.amp_flag_from_fiberid(fiberid, bad_amps_table)

Returns a boolean flag whether the amp has been flagged usable

Parameters

fiberid: string with the unique fiber_id info
bad_amps_table: astropy table containing the bad amp flag values. This can be retrieved from config.badamp

Returns

None if no matching fiber is found to the coords

True if amp is usable

False if any fiber in the defined region is flagged in a bad amp

hetdex_api.mask.create_dummy_wcs(coords, pixscale=None, imsize=None)

Create a simple fake WCS in order to use the regions subroutine. Adapted from John Feldmeiers galmask.py

Parameters

coords: a SkyCoord object: center coordinates of WCS
pixscale: astropy quantity: pixel scale of WCS in astropy angle quantity units
imsize: astropy quantity: size of WCS in astropy angle quanity units

hetdex_api.mask.create_gal_ellipse(galaxy_cat, row_index=None, pgcname=None, d25scale=1.5)

Similar to galmask.py/ellreg but can take a galaxy name as input.

Create galaxy ellipse region using an input galaxy catalog_table (likely need to change this API as it heavily follows the RC3 catalog format)

Parameters

galaxy_catalog_table: an astropy table: table of nearby galaxy positions and sizes. Must have a central coordinate and the SemiMajorAxis, SemiMinorAxis, Position Angle info as table column names
row_index: int: a row index in the galaxy catalog to create the ellipse for
pgcname: str: the PGCNAME in the RC3 cat. This is a string. eg. “PGC 43255” for NGC 4707
d25scale: float: how many times D25 should to scale the region. Default is 1.5 based on visual exploration

hetdex_api.mask.gal_flag_from_coords(coords, galaxy_cat, d25scale=1.5)

Returns a boolean flag value to mask sources near large galaxies

Adapted from John Feldmeier’s hetdex_tools/galmask.py

Parameters

coords: an astropy.coordinates SkyCoord object
galaxy_cat: an astropy table containing the large galaxy parameters. This is catered for the RC3 catalog stored in config.rc3cat
d25scale: The scaling of ellipses. 1.0 means use the ellipse for D25. Experimentation shows value of 1.5 works well

Returns

flag - boolean: True if the source is not in the galaxy mask False if the source is within the scaling of the galaxy mask

hetdex_api.mask.meteor_flag_from_coords(coords, shotid=None, streaksize=None)

Returns a boolean flag value to mask out meteors

Parameters

coords: an astropy.coordinates SkyCoord object
shotid: shotid to search. If none it will search all shots at once. If any are flagged bad then it will return False for all.
streaksize: an astropy quantity object defining how far off the perpendicular line of the meteor streak to mask out. Default is 12*u.arcsec

Returns

bool: True if no meteors fall in the aperture False if a meteor falls in the aperture

Example

hetdex_api.shot module

Functions to interact with the shot HDF5 file and the Fibers Class. Requires a shotid (interger: 20180123009) or datevobs (str: eg. ‘20180123v009’)

author = Erin Mentuch Cooper

class hetdex_api.shot.CALFIB_DQ(*args: Any, **kwargs: Any)

Bases: BitFlagNameMap

BADAMP = 16

BADFIB = 512

BADPIX = 8

BADSHOT = 128

CHI2FIB = 4

FTF = 2

LARGEGAL = 32

MAIN = 1

METEOR = 64

SAT = 1024

THROUGHPUT = 256

class hetdex_api.shot.Fibers(shot, survey='hdr4', add_rescor=False, add_mask=False, mask_version=None, args=None)

Bases: object

close(): Close the H5 files related to the Fibers call

get_closest_fiber(coords, exp=None)

Returns index to closest fiber to an RA/DEC

Parameters

coords: an astropy SkyCoords object
exp: exposure number.

get_fib_image2D(wave_obj=None, fiber_id=None, fibnum_obj=None, multiframe_obj=None, expnum_obj=None, width=60, height=40, imtype='clean_image')

Returns an image from the 2D data for a specific fiber centered at a specific wavelength

Parameters

self: fibers class object
wave_obj: astropy wavelength object
fiber_id: fiber_id for a fiber: shotid_exp_multiframe_fibnum
expnum: dither exposure number [1,2,3]
fibnum: fiber number (This comes from the detections table and relates to the fiber index by fibnum = fibidx + 1)
multiframe: amp multiframe ID
imtype: image option to display
width: pixel width to be cutout (image size is 1032 pix)
height: pixel height to be cutout (image size is 1032 pix)

Returns

A 2D image of the specified fiber

get_image_xy(idx, wave_obj)

Finds the X,Y image position from a row in the fibers table

Note: this returns x,y image coords from 2D image arrays produced by Panacea.

plot_fiber_spectrum(idx, type='calfib', xlim=None, ylim=None)

query_region(coords, radius=None)

Returns an indexed fiber table for a defined aperture.

Parameters

self: Fibers class object
coords: astropy coordinate object
radiusfloat or ~astropy.units.Quantity: astropy quantity. If no quantity given, assume arcsec

query_region_idx(coords, radius=3.0)

Returns an index for a Fibers class object to retrieve all fibers in the defined aperture

self - Fibers class object coords - astropy coordinate object radius - astropy quantity object or value in arcsec

return_astropy_table()

Return an astropy table version of the Fibers table that can easily be saved

Returns

tableastropy.table:Table: an astropy table you can save

save_fiber_spectrum(idx, type='calfib', file='spec.dat')

Saves a fiber spectrum at a defined index

Parameters

self: Fibers class object
idx: index of the fiber in the Fibers Table
types: The fiber array type. Options are ‘calfib’, ‘spectrum’, ‘sky_spectrum’, ‘twi_spectrum’,’error1Dfib’, fiber_to_fiber’
file: output file name

Returns

An astropy table of the fiber spectra with columns of wavelength, spectrum data, error.

hetdex_api.shot.get_fibers_table(shot, coords=None, ifuslot=None, multiframe=None, expnum=None, radius=None, survey='hdr4', astropy=True, verbose=False, rawh5=False, F=None, fiber_flux_offset=None, add_rescor=False, add_mask=False, mask_options=None, mask_version=None, ignore_mask=None, mask_in_place=False, mask_value=nan)

Returns fiber specta for a given shot.

Parameters

shot: either shotid or datevobs
coords: astropy coordinate object
radius: an astropy quantity object
astropy: flag to make it an astropy table. Deprecated. Output is always an astropy table
survey: data release you want to access
rawh5: bool: if True, this will simply return the fibers from the specified shoth5 file. If False (the default), any relevent correcctions are applied.
verbose: print out warnings. Default is False
F Fibers class object: a pre-intiated fibers class object. This is used to limit I/O.
fiber_flux_offset: 1036 array: array of values in units of 10**-17 ergs/s/cm2/AA to add to each fiber spectrum used in the extraction. Defaults to None
add_rescor: option to add calfib_ffsky_rescor column generated by Maja Lujan Niemeyer. Defaults to False for now
add_mask: option to add mask column. Defaults to False for now.
mask_options: string array options to select to mask. Default None will select all flags. Set this to ‘BITMASK’ to return the full bitmask array Options are ‘MAIN’, ‘FTF’, ‘CHI2FIB’, ‘BADPIX’, ‘BADAMP’, ‘LARGEGAL’, ‘METEOR’, ‘BADSHOT’, ‘THROUGHPUT’, ‘BADFIB’, ‘SAT’
ignore_mask: Option to provide a list of flag names that should be ingored in the mask model
mask_in_place: Set to True to apply mask to calfib, calfibe, calfib_ffsky and calfib_ffsky_rescor
mask_value: value to fill masked values. Default is np.nan

Returns

A table of fibers within the defined aperture. Will be an astropy table object if astropy=True is set

hetdex_api.shot.get_image2D_amp(shot, multiframe=None, specid=None, amp=None, ifuslot=None, imtype='clean_image', expnum=1, survey='hdr4')

Returns an image from the 2D data based on a multiframe or a ifuslot/specid and amp combo

Parameters

multiframe: unique amp identifier to display
specid: instead of multiframe, you can provide both specid and amp or ifuslot and amp
ifuslot: provide ifuslot id and amp in place of multiframe
amp: inplace of amp you can provide the ifuslot and amp or the specid and amp
imtype: image option to display options are:[‘spectrum’, ‘error, clean_image’]
expnum_obj: integer for the dither/exposure

Returns

a 2D numpy array for the specified amp

hetdex_api.shot.get_image2D_cutout(shot, coords, wave_obj, width=40, height=40, imtype='clean_image', survey='hdr4')

Returns an image from the 2D data based on ra/dec/wave.

Parameters

self: HDF5 file called from open_shot_file(‘shot’)
coords: astropy coordinate object
wave_obj: astropy wavelength object
imtype: image option to display. Options are ‘clean_image’, ‘image’, and ‘error’.
width: pixel width to be cutout (image size is 1032 pix) this is the width X in the wavelength dimension
height: pixel height to be cutout (image size is 1032 pix) this is height Y in the fiber dimension
survey: string identifying data release

Returns

hetdex_api.shot.open_shot_file(shotid, survey='hdr4')

Open the H5 file for a shot. This is a global function that allows you to open an H5 file based on its shotid and data release.

Parameters

shotid: either in the form of integer shotid (eg. 20180123009) or string datevobs (eg. ‘20180123v009’)
surveystring: Data release you would like to load, i.e., ‘HDR1’, ‘hdr2’, This is case insensitive.

Example

>>> from hetdex_api.shot import open_shot_file
>>> fileh = open_shot_file(20180123009)

or

>>> fileh = open_shot_file('20180123v009')

hetdex_api.sqlite_utils module

Utility functions for SQLite3 ELiXer Report imaging databases

Each HETDEX Data release will contain multiple serverless ELiXer report imaging databsaes with approx 100,000 images

per database file with each file containing one type of report image.

The database files should be named as: elixer_reports_<detectid prefix>_<option report type>.db

For example: “elixer_reports_20003.db” contains HDR2 (from the leading ‘2’) main report images that all begin with 20003

… so, images 2000300000.png to 2000399999.png

“elixer_reports_10006_nei.db” contain the Neighborhood report images for HDR1 that fit the 10006*nei.png pattern

“elixer_reports_20000_mini.db” contain the mini report images (e.g. for smart phone app) for HDR2, with: detectIDs 2000000000 to 2000099999

class hetdex_api.sqlite_utils.ConnMgr

Bases: object

Primitive container for managing SQLite connection (to avoid repeated path search and connection building) Just for reading

close_conns()

fetch_image(detectid, report_type='report')

wrapper just to make code cleaner

Parameters:

detectid
report_type

Returns:

get_connection(detectid, report_type='report')

hetdex_api.sqlite_utils.build_elixer_report_image_db(db_name, img_dir, img_regex)

Not for general use. Should normally be called once per image type per grouping of (100,000) per data release.

If db already exists, this will insert new images and replace existing images (if detectIDs match)

This DOES NOT validate the images or check that they are appropriate for the db_name.

Progress is reported per 100 inserts

Insert speed does not depend on database size, but does depend on disk speed (local vs network, SSD vs HD, etc)

Parameters:

db_name – name (with path) of the SQLite output db
img_dir – location of all images to be imported (is NOT recursive)
img_regex – generally simple wildcard, like “.png” or “*nei.png” or “.jpg”, etc

Returns:

hetdex_api.sqlite_utils.fetch_elixer_report_image(conn, detectid)

Return a single image (image type (png, jpg) and report type (report, neighborhood, mini) depend on the database connection

Parameters:

conn – a sqlite3.Connection object or a path to a database
detectid – HETDEX detectid (int64 or string)

Returns:

None or single image

hetdex_api.sqlite_utils.get_db_connection(fn, readonly=True)

return a SQLite3 databse connection object for the provide databse filename

assumes file exists (will trap exception if not and return None) :param fn: :return: None or connection object

hetdex_api.sqlite_utils.get_elixer_report_db_path(detectid, report_type='report')

Return the top (first found) path to database file based on the detectid (assumes the HDR version is part of the prefix, i.e. HDR1 files are 1000*, HDR2 are 2000*, and so on)

Parameters:

detectid
report_type – choose one of “report” (normal ELiXer report image) [default] “nei” (ELiXer neighborhood image) “mini” (ELiXer mini-report image for phone app)

Returns:

None or database filename

hetdex_api.survey module

Initiates survey class and provides an API to query the survey class based on astropy coordinates

Added 2020/03/29 = FiberIndex class to query all fibers in the survey

Created on Tue Jan 22 11:02:53 2019 @author: gregz/Erin Mentuch Cooper

class hetdex_api.survey.FiberIndex(survey='hdr4', load_fiber_table=False, loadall=False)

Bases: object

close(): Close the hdfile when done

get_amp_flag(): Generate amp flag for each fiber in the survey

Parameters

FiberIndex Class

Returns

fiber_id: str unique fiber identifier string amp_flag: bool True if fiber is on a good quality amplifier

get_closest_fiberid(coords, shotid=None, maxdistance=None)

Function to retrieve the closest fiberid in a shot

Parameters

self: the FiberIndex class for a specific survey
coords: center coordinate you want to search. This should be an astropy SkyCoord object
shotid: Specific shotid (dtype=int) you want
maxdistance: The max distance (an angle quantity) you want to search for a nearby fiber. Default is 8.*u.arcsec

Returns

fiberid: unique fiber identifier for the shot

get_fib_from_hp(hp, shotid=None, astropy=True, return_index=False)

Return rows with corresponding healpix value

Parameters

hp:int: healpix integer you want to search
shotid:int: optional shotid to search. Default is none.
astropy: bool: returned table is an astropy table object. Defaults to True.
return_index: bool: option to return fiber row index values for additional slicing

Returns

table: table with fiber information including coordinates, healpix, fiber_id name and more
table_index: row indices to access h5 table coordinates

get_fiber_flags(coord=None, shotid=None)

Returns boolean mask values for a coord/shotid combo

Parameters

coord: SkyCoord object: sky coordinates
shotid: int: shotid

Returns

meteor_flag: bool: False if location is on meteor streak
gal_flag: bool: False if location is within a large galaxy
amp_flag: bool: False if location is on a bad quality amplifier
flag: bool: False if any of the above flags are False

get_gal_flag(d25scale=3.0)

Returns boolean mask with detections landing within galaxy defined by d25scale flagged as False.

Based on check_all_large_gal from hetdex_tools/galmask.py written by John Feldmeier

get_meteor_flag(streaksize=None): Returns boolean mask with detections landing on meteor streaks masked. Use np.invert(mask) to find meteors

get_shot_flag()

Generate shot flag for each fiber in the survey

Parameters

FiberIndex Class

Returns

shot_flag: bool: True if fiber is on a good shot

get_throughput_flag(throughput_threshold=0.08)

Generate throughput flag for each fiber in the survey

Parameters

FiberIndex Class

Returns

throughput_flag: bool: True if fiber observation is above throughput_threshold

query_region(coords, radius=None, shotid=None, return_index=False, return_flags=True)

Function to retrieve the indexes of the FiberIndex table for a specific region

Parameters

self: the FiberIndex class for a specific survey
coords: center coordinate you want to search. This should be an astropy SkyCoord object
radius: radius you want to search. An astropy quantity object
shotid: Specific shotid (dtype=int) you want
return_index: bool: Option to return row index values for slicing. Default is False
return_flags: bool: Option to return mask info. Default is True

Returns

table: astropy table: An astropy table of Fiber infomation in queried aperture
table_index: optional: an optional array of row coordinates corresponding to the retrieved fiber table

class hetdex_api.survey.Survey(survey='hdr4')

Bases: object

close(): Close the HDF5 file when you are done using it to release anything that might be in memory

Example

S.close()

get_shotlist(coords, radius=None, width=None, height=None)

Returns a list of shots for defined region of sky.

Parameters

self: Survey Class object
coords: astropy coordinate object
radius: an astropy Quantity object, or a string that can be parsed into one. e.g., ‘1 degree’ or 1*u.degree. If radius is specified, the shape is assumed to be a circle
width: in degress. Specifies the edge length of a square box. If a radius is not specified a width and height of a region must be given.
height: in degrees. Specifies the height of a rectangular box. Must be passed with width.

Examples

S = Survey(‘hdr1’) coords = SkyCoord(150.025513 * u.deg, 2.087767 * u.deg, frame=’icrs’) shotlist = S.get_shotlist(coords, radius=0.5*u.deg)

or for a rectangular region around the point defined in coords

shotlist = S.get_shotlist(coords, width=0.5, height=0.1)

remove_shots()

Function to remove shots that should be exluded from most analysis.

Parameters

self: Survey class object

Returns

ind_good: Boolean mask of good shots

Examples

S = Survey(‘hdr2’) ind_good_shots = S.remove_shots() S_good = S[ind_good_shots]

return_astropy_table(return_good=True)

Function to return an astropy table that is machine readable

Parameters

self: Survey Class object
return_good: Boolean flag to only exclude shots in config.badshot

Returns

survey_table: astropy table of the Survey class object

slice(): This will slice the survey class upon initialization to remove any bad shots from the survey so that only one class variable is used.

Example

survey = Survey(‘hdr1’).slice()

hetdex_api.survey.create_dummy_wcs(coords, pixscale=None, imsize=None)

Create a simple fake WCS in order to use the regions subroutine. Adapted from John Feldmeiers galmask.py

Parameters

coords: a SkyCoord object: center coordinates of WCS
pixscale: astropy quantity: pixel scale of WCS in astropy angle quantity units. Will default to 0.5 arcsec if not set
imsize: astropy quantity: size of WCS in astropy angle quanity units. Will default to 60 arcsec if not set

hetdex_api.survey.create_gal_ellipse(galaxy_cat, row_index=None, pgcname=None, d25scale=3.0)

Similar to galmask.py/ellreg but can take a galaxy name as input.

Create galaxy ellipse region using an input galaxy catalog_table (likely need to change this API as it heavily follows the RC3 catalog format)

Parameters

galaxy_catalog_table: an astropy table: table of nearby galaxy positions and sizes. Must have a central coordinate and the SemiMajorAxis, SemiMinorAxis, Position Angle info as table column names
row_index: int: a row index in the galaxy catalog to create the ellipse for
pgcname: str: the PGCNAME in the RC3 cat. This is a string. eg. “PGC 43255” for NGC 4707
d25scale: float: how many times D25 should to scale the region

hetdex-tools modules

hetdex_tools.interpolate

Created on October 8 2019 @author: Erin Mentuch Cooper

Extracts 1D spectrum at specified RA/DEC.

Provide either an ID/RA/DEC as input or a file with a list of ID, RA, DEC. Can be in the format of an astropy table (fits, ascii and no_header are currently supported) provided columns are appropriately marked ‘ID’,’ra’, ‘dec’ or ‘RA’, ‘DEC’. Coordinates can be in any astropy unit coordinate in the input table.

Option can be to provide a single shotid/datevobs to work on for batch processing, or can let program figure out which shots to extract spectra on.

Parameters

ra: RA for single source in an astropy unit, or assumed as deg
dec: DEC for single source in an astropy unit, or assumed as deg
rad: radius size for aperture in arcsec or an astropy unit, defaults to 3arcsec
ID: object ID if using a single source
shotid: use if you are running on just a single shotid (or datevobs)
input: path to input catalog of ID/RA/DEC, can use any astropy units or program will assume degree
output: name of output pickle file
multiprocess: flag to use python multiprocess, don’t use in a slurm job
single: flag to write out several astropy tables for each ID/shot spectra
merge: will combine all .pkl files in a directory, for use in slurm job cleanup
mergepath: use if you want to combine pickle files in another directory
survey: str: Survey you want to access. User note that HDR1 extractions are much slower compared to HDR2.
tpmin: float: Include only shots above tpmin. Default is None, we recommend 0.08
keep_bad_shots: bool: Set this to True if you want to include fibers from bad shots. This is dangerous as it can include fibers with bad astrometry, bad calibration. Default is False.
ffsky: bool: Use the full frame 2D sky subtraction model. Default is to use the local sky subtracted, flux calibrated fibers.
fiberweights: bool: Boolean flag to include fiber_weights tuple in source dictionary. This is used in Elixer, but is slow when used on large source lists.
fiber_info: bool: returns the fiber_info and weights of the fibers used in the extraction

Examples

For a single coordinate, you can search and extract all shots in HDR1

python3 get_spec.py –ra 150.02548 –dec 2.087987 –ID cosmos_LAE –outfile cosmos_LAE

To speed up, you can use the –multiprocess option. This works for interactive use on a node either through a jupyter notebook or an interactive node called by idev

python3 get_spec.py –multiprocess –ra 150.02548 –dec 2.087987 –ID cosmos_LAE –outfile cosmos_LAE

For working on a single shot only you can use the -s option. It is recommended to batch process each SHOTID individually and provide the same source list in each call.

python3 get_spec.py -ra 8.86535 -dec 0.59352 -s 20190104008

This can also work on an input file. It should either have the format of 3 columns = ID/RA/DEC or be an astropy table with columns labeled ‘ID’, ‘ra’ or ‘RA’, ‘dec’ or ‘DEC’

python3 get_spec.py –multiprocess -i ‘3dhst_input.cat’ -o ‘3dhst’

hetdex_tools.get_spec.get_flags(fiber_info): Get flags from fiber_info

hetdex_tools.get_spec.get_parser(): function that returns a parser from argparse

hetdex_tools.get_spec.get_source_spectra(shotid, args)

hetdex_tools.get_spec.get_source_spectra_mp(source_dict, shotid, manager, args)

hetdex_tools.get_spec.get_spectra(coords, ID=None, rad=3.5, multiprocess=True, shotid=None, survey='hdr4', tpmin=0.08, keep_bad_shots=False, ffsky=False, ffsky_rescor=False, fiberweights=False, return_fiber_info=False, loglevel='INFO', apply_update=True, fiber_flux_offset=None, apply_mask=False)

Function to retrieve PSF-weighted, ADR and aperture corrected spectral extractions of HETDEX fibers. It will search all shots within a specific HETDEX Data Release and return a table of spectra for each extraction per shot in which more than 7 fibers are found in order to generate an extracted spectrum.

Parameters

coords: list astropy coordinates
ID: list of ID names (must be same length as coords). Will generate a running index if no ID is given
rad: radius of circular aperture to be extracted in arcsec. Default is 3.5
multiprocess: bool: boolean flag to use multiprocessing. This will greatly speed up its operation as it will extract on 32 shots at time. But only use this when on a compute node. Use idev, a jupyter notebook, or submit the job as a single python slurm job. Default is True
shotid: int: list of integer shotids to do extractions on. By default it will search the whole survey except for shots located in the bad.shotlist file
survey: str: Survey you want to access. User note that HDR1 extractions are much slower compared to HDR2.
tpmin: float: Include only shots above tpmin. Default is 0.08
ffsky: bool: Use the full frame 2D sky subtraction model. Default is to use the local sky subtracted, flux calibrated fibers.
fiberweights: bool: Boolean flag to include fiber_weights tuple in source dictionary. This is used in Elixer, but is slow when used on large source lists.
fiber_info: bool: returns the fiber_info and weights of the fibers used in the extraction
keep_bad_shots: bool: Set this to True if you want to include fibers from bad shots. This is dangerous as it can include fibers with bad astrometry, bad calibration. Default is False.
loglevel: str: Level to set logging. Options are ERROR, WARNING, INFO, DEBUG. Defaults to INFO
apply_update: bool: Apply relevent HDR calibration updates. Default is True.
fiber_flux_offset: 1036 array: array of values in units of 10**-17 ergs/s/cm2/AA to add to each fiber spectrum used in the extraction. Defaults to None
ffsky_rescor: bool: Flag to use updated spectra for ffsky with residual correction applied. Corrected fiber arrays generated by Maja Lujan Niemeyer 2023-12-14. This is a post-HDR4 correction. Default is False
apply_mask: bool: Apply HETDEX fiber mask model. This will mask all fibers contributing to the spectral extraction before summation. Masked in place as NaNs

Returns

sources: an astropy table object of source spectra for all input coords/ID that have spectra in the survey shots. There is one row per source ID/shotid observation.

hetdex_tools.get_spec.get_spectra_dictionary(args)

hetdex_tools.get_spec.main(argv=None): Main Function

hetdex_tools.get_spec.merge(dict1, dict2): Return a new dictionary by merging two dictionaries recursively.

hetdex_tools.get_spec.return_astropy_table(Source_dict, fiberweights=False, return_fiber_info=False): Returns an astropy table fom a source dictionary

hetdex_tools.interpolate.make_data_cube(detectid=None, coords=None, shotid=None, pixscale=None, imsize=None, wave_range=[3470, 5540], dwave=2.0, dcont=50.0, convolve_image=False, ffsky=False, subcont=False, survey='hdr4', fiber_flux_offset=None, interp_kind='linear', apply_mask=False, fill_value=nan)

Function to make a datacube from either a detectid or from a coordinate/shotid combination.

Paramaters

detectid: int: detectid from the continuum or lines catalog. Default is None. Provide a coords/shotid combo if this isn’t given
coords: SkyCoords object: coordinates to define the centre of the data cube
pixscale: astropy angle quantity: plate scale
imsize: astropy angle quantity: spatial length of cube (equal dims is only option)
wave_range: list: start and stop value for the wavelength range in Angstrom
dwave: step in wavelength range in Angstrom
convolve_image: bool: option to convolve image with shotid seeing
ffsky: bool: option to use full frame calibrated fibers. Default is True.
subcont: bool: option to subtract continuum. Default is False. This will measure the continuum 50AA below and above the input wave_range
dcont: float: width in angstrom to measure the continuum. Default is to measure 50 AA wide regions on either side of the line
fiber_flux_offset: 1036 array: array of values in units of 10**-17 ergs/s/cm2/AA to add to each fiber spectrum used in the extraction. Defaults to None
interp_kind: str: Kind of interpolation to pixelated grid from fiber intensity. Options are ‘linear’, ‘cubic’, ‘nearest’. Default is linear.
apply_mask: bool Apply HETDEX fiber mask model. This will mask all fibers contributing: to the spectral extraction before summation. Masked in place as NaNs fill_value: float, optional Value used to fill in for requested points outside of coverage or in a mask region. If not provided, then the default is nan.

Returns

hdu: PrimaryHDU object: the data cube 3D array and associated 3d header Units are ‘10^-17 erg cm-2 s-1 per spaxel’

Examples

Can either pass in a detectid:

>>> detectid_obj=2101602788
>>> hdu = make_data_cube( detectid=detectid_obj)
>>> hdu.writeto( str(detectid_obj) + '.fits', overwrite=True)

or can put in an SkyCoord object:

>>> star_coords = SkyCoord(9.625181, -0.043587, unit='deg')
>>> hdu = make_data_cube( coords=star_coords[0], shotid=20171016108, dwave=2.0)
>>> hdu.writeto( 'star.fits', overwrite=True)

hetdex_tools.interpolate.make_narrowband_image(detectid=None, coords=None, shotid=None, pixscale=None, imsize=None, wave_range=None, convolve_image=False, ffsky=False, subcont=False, dcont=50.0, include_error=False, survey='hdr4', extract_class=None, fiber_flux_offset=None, interp_kind='linear', apply_mask=False, fill_value=nan)

Function to make narrowband image from either a detectid or from a coordinate/shotid combination.

Paramaters

detectid: int: detectid from the continuum or lines catalog. Default is None. Provide a coords/shotid combo if this isn’t given
coords: SkyCoords object: coordinates to define the centre of the data cube
pixscale: astropy angle quantity: plate scale
imsize: astropy angle quantity: image size
wave_range: list or None: start and stop value for the wavelength range in Angstrom. If not given, the detectid linewidth is used
convolve_image: bool: option to convolve image with shotid seeing
ffsky: bool: option to use full frame calibrated fibers. Default is True.
subcont: bool: option to subtract continuum. Default is False. This will measure the continuum 50AA below and above the input wave_range
dcont: width in angstrom to measure the continuum. Default is to measure 50 AA wide regions on either side of the line
include_error bool: option to include error array
extract Extract class object: option to include a preloaded Extract class object. Default is to load extract class according to detection info
fiber_flux_offset: 1036 array: array of values in units of 10**-17 ergs/s/cm2/AA to add to each fiber spectrum used in the extraction. Defaults to None
interp_kind: str: Kind of interpolation to pixelated grid from fiber intensity. Options are ‘linear’, ‘cubic’, ‘nearest’. Default is linear.
apply_mask: bool: Apply HETDEX fiber mask model. This will mask all fibers contributing to the spectral extraction before summation. Masked in place as NaNs
fill_value: float, optional: Value used to fill in for requested points outside of coverage or in a mask region. If not provided, then the default is nan.

Returns

hdu: PrimaryHDU object: the 2D summed data array and associated 2d header Units are ‘10^-17 erg cm-2 s-1’ If include_error=True will include addiional hdu

Examples

For a specific detectid: >>> hdu = make_narrowband_image(detectid=2101046271)

For a SkyCoords object. You must provide shotid and wavelength range

>>> coords = SkyCoord(188.79312, 50.855747, unit='deg')
>>> wave_obj = 4235.84 #in Angstrom
>>> hdu = make_narrowband_image(coords=coords,
                                shotid=20190524021,
                                wave_range=[wave_obj-10, wave_obj+10])