Utility functions for skypac

This module provides utility functions for use by stsci.skypac module.

Authors

Mihai Cara

License

LICENSE

class stsci.skypac.utils.ImageRef(hdulist_refcnt=None)

A lightweight class that supports reference counting for FITS images and holds a ResourceRefCount object. It provides several attributes that describe main characteristics of the image (file) and essential functions for manipulating reference count.

Parameters

hdulist_refcnt: ResourceRefCount

A ResourceRefCount object holding a astropy.io.fits.HDUList object.

Attributes

filename: str

Name of the opened file. Can be None for in-memory created astropy.io.fits.HDUList objects.

can_reload_data: bool

True for files attached to a physical file, False for in-memory astropy.io.fits.HDUList objects.

original_fname: str

Original name of the file as requested by the user. Note: may be different from filename if the orininal file was in GEIS or WAIVER FITS format and subsequently was converted to a MEF FITS format. In that case this attribute will show the name of the original GEIS or WAIVER FITS file.

original_ftype: str

Type of the original file. Can take one of the following values: ‘MEF’, ‘SIMPLE’, ‘GEIS’, ‘WAIVER’, or ‘UNKNOWN’.

original_exists: bool

Indicates if the physical file exists. It is False for in-memory images.

mef_fname: str, None

Name of the MEF FITS file if exists, None otherwise.

mef_exists: bool

Indicates whether the MEF FITS file exists.

DQ_model: str, None

Type of the DQ model: ‘external’ for WFPC, WFPC2, and FOC instruments (or non-HST data if set so) that have DQ data in a separate (from image) file and ‘intrinsic’ for ACS, etc. images that have DQ extensions in the image file. It is None if the image does not have DQ data.

telescope: str

Telescope that acquired the image.

instrument: str

Instrument used to acquire data.

fmode: str

File mode used to open FITS file. See astropy.io.fits.open for more details.

memmap: bool

Is the astropy.io.fits.HDUList memory mapped?

property closed

Is the ImageRef object closed?

property extname

Extension name of the first extension.

property hdu

astropy.io.fits.HDUList of the attached image.

hold()

Increment the reference count of the attached resource.

info(self, fh=sys.stdout)

Print a summary of the object attributes.

property refcount

Reference count of the attached astropy.io.fits.HDUList.

release()

Decrement reference count of the attached resource. If the reference count reaches zero, the attached ResourceRefCount object will be closed and set to None.

set_HDUList_RefCount(hdulist_refcnt=None)

Set (attach) a new ResourceRefCount object that holds a astropy.io.fits.HDUList object. This is allowed only if the already attached ResourceRefCount can be closed. The reference count of the ResourceRefCount being attached will be incremented.

Parameters

hdulist_refcnt: ResourceRefCount, None

A ResourceRefCount object containing a astropy.io.fits.HDUList object. If it is None,it will release and close the attached ResourceRefCount object.

Raises

ValueError

The (already) attached ResourceRefCount must have reference count <= 1 so that it can be closed before being replaced with a new resource.

ValueError

The resource being attached must be in an open state.

class stsci.skypac.utils.MultiFileLog(console=True, enableBold=True, flog=None, append=True, autoflush=True, appendEOL=True)

This is a class that facilitates writting to multiple files. MultiFileLog stores multiple file objects and can write the same log entry to all of them. It also facilitates controlling when a function can close a log file. Finally, it provides some utility functions that automate such things as appending EOL at the end of the log entry, flushing the files (to avoid losing log entries in case of uncaught exceptions), displaying WARNING, ERROR, etc. in bold on standard streams (sys.stdout, etc.)

Parameters

console: bool, optional

Enables writting to the standard output.

enableBold: bool, optional

Enable or disable writing bold text to console, e.g., 'WARNING:' 'ERROR:', etc.

flog: str, file, None, list of str or file objects, optional

File name or file object to be added to MultiFileLog during the initialization. More files can be added lated with add_logfile().

append: bool, optional

Default open mode for the files that need to be opened (e.g., that are passed as file names). If append is True, new files added to a MultiFileLog object will be opened in the “append” mode: new log entries will be appended to existing files – same as mode ‘a’ of standard function open(). When append is False, any existing files will be overwritten.

autoflush: bool, optional

Indicates whether or not to flush files after each log entry.

appendEOL: bool, optional

Indicates whether or not to add EOL at the end of each log entry.

add_logfile(flog, initial_skip=2, can_close=None, mode=None)

Add (and open, if necessary) a log file to the MultiFileLog object.

Parameters

flog: str, file, None, list of str or file objects, optional

File name or file object to be added.

initial_skip: int, optional

The number of blank line to be written to the file if not at the beginning of the file.

can_close: bool, None, optional

Indicates whether the file object can be closed by the close() function:

• can_close is True – file can be closed by the close() function;

• can_close is False – file will not be closed by the close() function;

• can_close is None – automatic selection based on the type of the flog argument: True if flog is a string (e.g., file name), False otherwise.

mode: str, None, optional

File open mode: same meaning as the mode parameter of the Python’s built-in open() function. If None, the mode will be inherited from file open mode set during initialization.

Returns

flog: file object

File object of newly opened (or attached) file.

close()

Close all files opened by MultiFileLog.

It will close all files opened by MultiFileLog - essentially, all files added as file name. It will not close files added as file handles.

property count

Return the number of files attached to the MultiFileLog object excluding the sys.stdout file.

enable_console(enable=True, enableBold=True)

Enable output to the standard console – sys.stdout.

Parameters

enable: bool, optional

Enable or disable output to sys.stdout.

enableBold: bool, optional

Enable or disable writing bold text to console, e.g., 'WARNING:', 'ERROR:', etc.

error(msg, *fmt)

Prints an error message. The word ‘ERROR:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.

flush()

Flush all files.

important(msg, *fmt)

Prints an “important” message. The word ‘IMPORTANT:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.

logentry(msg, *fmt, **skip)

Write a log entry to the log files.

Parameters

msg: str

String to be printed. Can contain replacement fields delimited by braces {}.

fmt

Parameters to be passed to str.format() method for formatting the string msg.

skip: int

Number of empty lines that should follow the log message.

Examples

>>> logentry('Sky background for chip {} is {}', 'SCI1', 10.0, skip=2)
will print:
Sky background for chip SCI1 is 10.0
followed by two blank lines.

print_endlog_msg()

Print a message (“Log written to…”) indicating the end of the log.

set_close_flag(fh, can_close=True)

Modify the “can be closed” status of a given file.

Parameters

fh: file object

can_close: bool, optional

Indicates if the file can be closed by the close() function.

Note

File object fh must have been already added to the MultiFileLog object.

skip(nlines=1)

Skip a specified number of blank lines.

Parameters

nlines: int, optional

unclose_copy()

Return a copy of the MultiFileLog object with all the attached files marked as “keep open”, that is, the close() will not close these files.

This is useful before passing the MultiFileLog object to a function that may add its own log files, and then attempt to close the files with the close() method. Thus, by passing an “unclose copy”, one can be sure that the files opened at the top level will not be closed by other functions to which the MultiFileLog object may be passed.

warning(msg, *fmt)

Prints a warning message. The word ‘WARNING:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.

class stsci.skypac.utils.ResourceRefCount(resource, *close_args, resource_close_fnct=None, **close_kwargs)

A class that implements reference counting for various resources: file objects, FITS HDU lists, etc. It is intended to be used as a mechanism of controlling the “lifespan” of resources that can be used in different parts of the code “indipendently”. The resource is kept open as long as the reference count is larger than zero. Once the reference count is decreased to 0, the resource is automatically closed.

Note

The reference count of the newly created ResourceRefCount object is set to 0. It is user’s responsibility to increment the reference count of this object through the call to hold() function.

Parameters

resource

An object who must be kept open or closed based on reference count.

close_args: tuple

Positional arguments to be passed to the resource_close_fnct() function.

resource_close_fnct: function, optional

The function (usually a method of the attached resource), that can “close” the resource.

close_kwargs: dict

Keyword arguments to be passed to the resource_close_fnct() function.

property closed

Indicates if the resource is “closed”.

hold()

Increment the reference count of the attached resource.

is_subscribed_on_close(obj)

Check if the object is subscribed to on close notifications.

property refcount

Reference count.

release()

Decrement reference count of the attached resource. If the reference count reaches zero, call all registered “on close” notify callbacks, and then call the “close” method on the resource which was set at initialization of the ResourceRefCount object. Finally, the resource property of the ResourceRefCount object will be set to None.

property resource

Resource attached to a ResourceRefCount object.

subscribe_close_notify(obj, callback=None)

Set the object and its method that need to be called when the resource is about to be closed.

unsubscribe_close_notify(obj)

Remove the object (and its method) from the list of callbacks that need to be notified of impending closing of the resource.

stsci.skypac.utils.almost_equal(arr1, arr2, fp_accuracy=None, fp_precision=None)

Compares two values, or values of numpy.ndarray and verifies that these values are close to each other. For exact type (integers and boolean) the comparison is exact. For inexact types (float, numpy.float32, etc.) it checks that the values (or all values in a numpy.ndarray) satisfy the following inequality:

\[|v1-v2| \leq a + 10^{-p} \cdot |v2|,\]

where a is the accuracy (“absolute error”) and p is precision (“relative error”).

Parameters

arr1: float, int, bool, str, numpy.ndarray, None, etc.

First value or array of values to be compared.

arr2: float, int, bool, str, numpy.ndarray, None, etc.

Second value or array of values to be compared.

fp_accuracy: int, float, None, optional

Accuracy to withing values should be close. Default value will use twice the value of the machine accuracy (machine epsilon) for the input type. This parameter has effect only when the values to be compared are of inexact type (e.g., float).

fp_precision: int, float, None, optional

Accuracy to withing values should be close. Default value will use twice the value of the machine precision (resolution) for the input type. This parameter has effect only when the values to be compared are of inexact type (e.g., float).

Returns

almost_equal: bool

Returns True if input values are close enough to each other or False otherwise.

Raises

ValueError

If fp_accuracy is negative.

TypeError

If fp_precision is not int, float, or None.

stsci.skypac.utils.count_extensions(img, extname='SCI')

Return the number of extname extensions in the input image img. If extname is None, return the number of all image-like extensions.

Input parameters are identical to those of get_extver_list().

See also

get_extver_list, get_ext_list

Examples

>>> count_extensions('j9irw1rqq_flt.fits',extname='SCI')
2
>>> count_extensions('j9irw1rqq_flt.fits',extname=None)
10

stsci.skypac.utils.ext2str(ext, compact=False, default_extver=1)

Return a string representation of an extension specification.

Parameters

ext: tuple, int, str

Extension specification can be a tuple of the form (str,int), e.g., (‘sci’,1), an integer (extension number), or a string (extension name).

compact: bool, optional

If compact is True the returned string will have extension name quoted and separated by a comma from the extension number, e.g., "'sci',1". If compact is False the returned string will have extension version immediately follow the extension name, e.g., 'sci1'.

default_extver: int, optional

Specifies the extension version to be used when the ext parameter is a string (extension name).

Returns

strext: str

String representation of extension specification ext.

Raises

TypeError

Unexpected extension type.

Examples

>>> ext2str('sci',compact=False,default_extver=6)
"'sci',6"
>>> ext2str(('sci',2))
"'sci',2"
>>> ext2str(4)
'4'
>>> ext2str('dq')
"'dq',1"
>>> ext2str('dq',default_extver=2)
"'dq',2"
>>> ext2str('sci',compact=True,default_extver=2)
'sci2'

stsci.skypac.utils.file_name_components(fname, detect_HST_FITS_suffix=True)

Splits base file name into a root, suffix, and extension. Given a full path, this function extracts the base name, and splits it into three components: root name, suffix, and file extension.

Parameters

fname: str

file name

detect_HST_FITS_suffix: bool, optional

If True, detects the suffix of most HST files by looking for the rightmost occurence of the underscore (‘_’) in the file name.

Returns

root: str

Root name of the file. When detect_HST_FITS_suffix is True, this is the part of the file name preceding the rightmost suffix separator (‘_’). Otherwise, it is the base file name without file extension.

suffix: str

If detect_HST_FITS_suffix is True, this field will contain the suffix of most HST files, i.e., the part of the file name contained between the rightmost suffix separator (‘_’) and file extension separator. This return value will be an empty string if If detect_HST_FITS_suffix is False or if the file name has no extension separator.

fext: str

File extension

Examples

>>> file_name_components('/data/m87/ua0x5001m_c0f.fits')
('ua0x5001m', 'c0f', '.fits')
>>> file_name_components('/data/m87/ua0x5001m_c0f.fits',False)
('ua0x5001m_c0f', '', '.fits')

stsci.skypac.utils.get_ext_list(img, extname='SCI')

Return a list of all extension versions of extname extensions. img can be either a file name or a astropy.io.fits.HDUList object.

This function is similar to get_extver_list(), the main difference being that it returns a list of fully qualified extensions: either tuples of the form (extname, extver) or integer extension numbers (when extname is None).

See also

get_extver_list, count_extensions

Examples

>>> get_ext_list('j9irw1rqq_flt.fits',extname='SCI')
[('SCI', 1), ('SCI', 2)]
>>> get_ext_list('j9irw1rqq_flt.fits',extname=None)
[1, 2, 3, 4, 5, 6, 8, 9, 10, 11]

stsci.skypac.utils.get_extver_list(img, extname='SCI')

Return a list of all extension versions with extname extension names. If extname is None, return extension numbers of all image-like extensions.

Note

If input image is a ImageRef, this function will not modify its reference count.

Parameters

img: str, `astropy.io.fits.HDUList`, or `~skypac.utils.ImageRef`

Input image object. If img is a string object (file name) then that file will be opened. If the file pointed to by the file name is a GEIS or WAIVER FITS file, it will be converted to a simple/MEF FITS format if clobber is True.

extname: str, optional

Indicates extension name for which all existing extension versions should be found. If extname is None, then get_extver_list will return a list of extension numbers of all image-like extensions.

Returns

extver: list

List of extension versions corresponding to the input extname. If extname is None, it will return a list of extension numbers of all image-like extensions.

Raises

IOError

Unable to open input image file.

TypeError

Argument img must be either a file name (str), an ImageRef, or a astropy.io.fits.HDUList object.

TypeError

Argument extname must be either a string or None.

See also

get_ext_list, count_extensions

Examples

>>> get_extver_list('j9irw1rqq_flt.fits',extname='sci')
[1, 2]
>>> get_extver_list('j9irw1rqq_flt.fits',extname=None)
[1, 2, 3, 4, 5, 6, 8, 9, 10, 11

stsci.skypac.utils.is_countrate(hdulist, ext, units_kwd='BUNIT', guess_if_missing=True, telescope=None, instrument=None, verbose=True, flog=None)

Infer the units of the data of the input image from the input image. Specifically, it tries to infer whether the units are counts (or count-like) or if the units are count-rate.

The units of data are determined from the BUNIT header keyword by searching its value for the division sign '/'. If the division sign is not found, then the units are assumed to be “counts”. If the division sign is found in the BUNIT value and if the numerator is one of the following: ‘ELECTRONS’,’COUNTS’, or ‘DN’, and denumerator is either ‘S’,’SEC’, or ‘SECOND’, then the units are assumed to be count-rate.

Parameters

hdulist: `astropy.io.fits.HDUList`

astropy.io.fits.HDUList of the image.

ext: tuple, int, str

Extension specification for whose data the units need to be inferred. An int ext specifies extension number. A tuple in the form (str, int) specifies extension name and number. A string ext specifies extension name and the extension version is assumed to be 1. See documentation for astropy.io.fits.getData for examples.

units_kwd: str, optional

FITS header keyword describing data units of the image. This keyword is assumed to be in the header of the extension specified by the ext parameter.

guess_if_missing: bool, optional

Instructs to try make best guess on image units when the keyword specified by units_kwd is not found in the image header. The first action will be to look for this keyword in the primary header, and if not found, infer the units based on the telescope, instrument, and detector information.

telescope: str, None, optional

Specifies the telescope from which the data came. If not specified, the value specified in the 'TELESCOP' keyword in the primary header will be used.

instrument: str, None, optional

Specifies the instrument used for acquiring data. If not specified, the value specified in the 'INSTRUME' keyword in the primary header will be used.

verbose: bool, optional

Specifies whether to print warning messages.

flog: str, file, MultiFileLog, None, optional

Specifies the log file to which the messages should be printed. It can be a file name, a file object, a MultiFileLog object, or None.

Returns

bool, None

Returns True if the units of the input image for a given extension are count-rate-like and False if the units are count-like. Returns None if the units cannot be inferred from the header.

stsci.skypac.utils.openImageEx(filename, mode='readonly', dqmode='readonly', memmap=True, saveAsMEF=True, output_base_fitsname=None, clobber=True, imageOnly=False, openImageHDU=True, openDQHDU=False, preferMEF=True, verbose=False)

Open an image file and (if requested) the associated DQ file and return corresponding ImageRef objects.

This function is an enhanced version of stsci.tools.fileutil.openImage() function in that it can open both the image file and the associated DQ image. It also provides additional inormation about the opened files: file type, original file name, DQ model (“intrinsic”, where DQ data are placed in the same file as the science data, or “extrinsic” when DQ data are in a separate file from the science data), etc. Because of the way it was implemented, it requires half of the number of calls to astropy.io.fits.open thus making it almost twice as fast as the openImage() function.

Parameters

filename: str

File name of the file to be opened. The image file formats are recognized: simple/MEF FITS, HST GEIS format, or WAIVER FITS format.

mode: str, optional

File mode used to open main image FITS file. See astropy.io.fits.open for more details.

dqmode: str, optional

File mode used to open DQ image FITS file. See parameter mode above for more details.

memmap: bool, optional

Should memory mapping to be used whe opening simple/MEF FITS?

saveAsMEF: bool, optional

Should an input GEIS or WAIVER FITS be converted to simple/MEF FITS?

output_base_fitsname: str, None, optional

The base name of the output simple/MEF FITS when saveAsMEF is True. If it is None, the file name of the converted file will be determined according to HST file naming conventions.

clobber: bool, optional

If saveAsMEF is True, should any existing output files be overwritten?

imageOnly: bool, optional

Should this function open the image file only? If True, then the DQ-related attributes will not be valid.

openImageHDU: bool, optional

Indicates whether the returned ImageRef object corresponding to the science image file should be in an open or closed state.

openDQHDU: bool, optional

Indicates whether the returned ImageRef object corresponding to the DQ image file should be in an open or closed state.

preferMEF: bool, optional

Should this function open an existing MEF file that complies with HST naming convention when the input file is in GEIS or WAIVER FITS format, even when saveAsMEF is False or clobber is False?

verbose: bool, optional

If True, some addition information will be printed out to standard output.

Returns

(img, dqimg): (ImageRef, ImageRef)

A tuple of ImageRef objects corresponding to the science image and to the DQ image. Each object in the returned tuple open or closed depending on the input arguments.

Note

If the returned object is open, then its reference count will be at least 1. The caller is responsible for “releasing” the object when it is no longer needed.

Note

If the DQ model of the opened file is “intrinsic”, then the dqimg component of the returned tuple will hold a reference counter to the same image. Thus, for “intrinsic” DQ data models, the reference count of the returned objects may be 2 (if both science openImageHDU and openDQHDU are True).

Raises

ValueError

If input file is neither a GEIS file nor a FITS file.

ValueError

Errors occured while accessing/reading the file possibly due to corrupted file, non-compliant file format, etc.s

stsci.skypac.utils.skyval2txt(files='*_flt.fits', skyfile='skyfile.txt', skykwd='SKYUSER', default_ext=('SCI', '*'))

A convenience function that allows retrieving computed sky background values from image headers and storing them in a text file that can be re-used by drizzlepac.astrodrizzle.AstroDrizzle(). This is particularly useful when performing sky matching on a large number of images which takes considerable time. Saving computed sky values to a text file allows re-running AstroDrizzle() without re-computing sky values.

Warning

The file specified by skyfile is overwritten without warning.

Note

Images that do not have specified extensions will be ignored.

Parameters

files: str

File name(s), including extension specification if necessary, from which sky values should be retrieved:

• a comma-separated list of valid science image file names (see note below) and (optionally) extension specifications, e.g.: 'j1234567q_flt.fits[1], j1234568q_flt.fits[sci,2]';

• an @-file name, e.g., '@fits_files_with_skyvals.txt'.

Note

Valid science image file names are:

• file names of existing FITS, GEIS, or WAIVER FITS files;

• partial file names containing wildcard characters, e.g., '*_flt.fits';

• Association (ASN) tables (must have _asn, or _asc suffix), e.g., 'j12345670_asn.fits'.

Warning

@-file names MAY NOT be followed by an extension specification.

Warning

If an association table or a partial file name with wildcard characters is followed by an extension specification, it will be considered that this extension specification applies to each file name in the association table or each file name obtained after wildcard expansion of the partial file name.

skyfile: str, optional

Output “skyfile” to which sky values from the image headers should be written out.

skykwd: str, optional

Header keyword holding the value of the computed sky background.

default_ext: int, tuple, optional

Default extension to be used with image files that to not have an extension specified.

stsci.skypac.utils.temp_mask_file(rootname, suffix, ext, data, dir=os.path.curdir, fnameOnly=False)

Saves 2D data array to temporary simple FITS file. The name of the emporary file is generated based on the input parameters.

Parameters

data: numpy array

Data to be written to the temporary FITS file. Data will be written in the primary HDU.

rootname: str

Root name of the file.

prefix: str, optional

Prefix to be added in front of the root name. If randomize_prefix is True, then a random string will be added to the right of the string specified by prefix (with no separator between them). Prefix (or the randomized prefix) will be separated from the root name by the string specified in sep. If prefix is an empty string ('') then no prefix will be prepended to the root file name.

suffix: str, optional

Suffix to be added to the root name. Suffix will be separated from the root name by the string specified in sep.

ext: int, str, or tuple of the form (str, int), optional

Extention to be added to the temporary file after the suffix. Extension name string will be separated from the suffix by the string specified in sep.

sep: str, optional

Separator string to be inserted between (randomized) prefix and root name, root name and suffix, and suffix and extension.

randomize_prefix: bool, optional

Specifies whether to add (postpend) a random string to string specified by prefix.

dir: str, optional

Directory to which the temporary file should be written. If directory dir is None then the file will be written to the default (for more details, see the explanation for argument dir to the tempfile.mstemp function).

fnameOnly: bool, optional

Specifies what should temp_mask_file return: file name of the created file (if fnameOnly is True), or a tuple with the file name of the created file and an open ImageRef object of that file.

Returns

fname: str

File name of the temporary file.

mask: ImageRef

An open ImageRef object of the temporary FITS file. This is returned as a tuple together with the file name only when fnameOnly is False.

Note

Mask data will be in the Primary HDU.

Raises

TypeError

Extension specifier must be either an integer, a string, or a tuple of the form (str, int).

Examples

>>> import numpy as np
>>> from stsci import skypac
>>> mask=np.ones((800,800),dtype=np.uint8)
>>> skypac.utils.temp_mask_file(mask, 'ua0x5001m',
...     suffix='skymatch_mask', ext=('sci',4), dir='/data/m87',
...     fnameOnly=True)
'/data/m87/tmp39gCpw_ua0x5001m_skymatch_mask_sci4.fits'
>>> skypac.utils.temp_mask_file(mask, 'ua0x5001m',
...     suffix='skymatch_mask', ext=('sci',4), dir='.', fnameOnly=True)
'tmpxl7LTO_ua0x5001m_skymatch_mask_sci4.fits'
>>> skypac.utils.temp_mask_file(mask, 'ua0x5001m',
...     suffix='skymatch_mask', ext=('sci',4), dir='.',fnameOnly=False)
('tmpxMcL5g_ua0x5001m_skymatch_mask_sci4.fits', <skypac.utils.ImageRef object at 0x101f5a3d0>)