Utility functions for skypac

This module provides utility functions for use by stsci.skypac module.

Authors:: Mihai Cara
License:: LICENSE

class stsci.skypac.utils.ImageRef(hdulist_refcnt=None)[source]

A lightweight class that supports reference counting for FITS images and holds a ResourceRefCount object. It provides several attributes that describe main characteristics of the image (file) and essential functions for manipulating reference count.

Parameters:

hdulist_refcnt: ResourceRefCount: A ResourceRefCount object holding a astropy.io.fits.HDUList object.

Attributes:

filename: str: Name of the opened file. Can be None for in-memory created astropy.io.fits.HDUList objects.
can_reload_data: bool: True for files attached to a physical file, False for in-memory astropy.io.fits.HDUList objects.
original_fname: str: Original name of the file as requested by the user. Note: may be different from filename if the orininal file was in GEIS or WAIVER FITS format and subsequently was converted to a MEF FITS format. In that case this attribute will show the name of the original GEIS or WAIVER FITS file.
original_ftype: str: Type of the original file. Can take one of the following values: ‘MEF’, ‘SIMPLE’, ‘GEIS’, ‘WAIVER’, or ‘UNKNOWN’.
original_exists: bool: Indicates if the physical file exists. It is False for in-memory images.
mef_fname: str, None: Name of the MEF FITS file if exists, None otherwise.
mef_exists: bool: Indicates whether the MEF FITS file exists.
DQ_model: str, None: Type of the DQ model: ‘external’ for WFPC, WFPC2, and FOC instruments (or non-HST data if set so) that have DQ data in a separate (from image) file and ‘intrinsic’ for ACS, etc. images that have DQ extensions in the image file. It is None if the image does not have DQ data.
telescope: str: Telescope that acquired the image.
instrument: str: Instrument used to acquire data.
fmode: str: File mode used to open FITS file. See astropy.io.fits.open for more details.
memmap: bool: Is the astropy.io.fits.HDUList memory mapped?

property closed: Is the ImageRef object closed?

property extname: Extension name of the first extension.

property hdu: astropy.io.fits.HDUList of the attached image.

hold()[source]: Increment the reference count of the attached resource.

info(self, fh=sys.stdout)[source]: Print a summary of the object attributes.

property refcount: Reference count of the attached astropy.io.fits.HDUList.

release()[source]: Decrement reference count of the attached resource. If the reference count reaches zero, the attached ResourceRefCount object will be closed and set to None.

set_HDUList_RefCount(hdulist_refcnt=None)[source]

Set (attach) a new ResourceRefCount object that holds a astropy.io.fits.HDUList object. This is allowed only if the already attached ResourceRefCount can be closed. The reference count of the ResourceRefCount being attached will be incremented.

Parameters:

hdulist_refcnt: ResourceRefCount, None: A ResourceRefCount object containing a astropy.io.fits.HDUList object. If it is None,it will release and close the attached ResourceRefCount object.

Raises:

ValueError: The (already) attached ResourceRefCount must have reference count <= 1 so that it can be closed before being replaced with a new resource.
ValueError: The resource being attached must be in an open state.

class stsci.skypac.utils.MultiFileLog(console=True, enableBold=True, flog=None, append=True, autoflush=True, appendEOL=True)[source]

This is a class that facilitates writting to multiple files. MultiFileLog stores multiple file objects and can write the same log entry to all of them. It also facilitates controlling when a function can close a log file. Finally, it provides some utility functions that automate such things as appending EOL at the end of the log entry, flushing the files (to avoid losing log entries in case of uncaught exceptions), displaying WARNING, ERROR, etc. in bold on standard streams (sys.stdout, etc.)

Parameters:

console: bool, optional: Enables writting to the standard output.
enableBold: bool, optional: Enable or disable writing bold text to console, e.g., 'WARNING:' 'ERROR:', etc.
flog: str, file, None, list of str or file objects, optional: File name or file object to be added to MultiFileLog during the initialization. More files can be added lated with add_logfile().
append: bool, optional: Default open mode for the files that need to be opened (e.g., that are passed as file names). If append is True, new files added to a MultiFileLog object will be opened in the “append” mode: new log entries will be appended to existing files – same as mode ‘a’ of standard function open(). When append is False, any existing files will be overwritten.
autoflush: bool, optional: Indicates whether or not to flush files after each log entry.
appendEOL: bool, optional: Indicates whether or not to add EOL at the end of each log entry.

add_logfile(flog, initial_skip=2, can_close=None, mode=None)[source]

Add (and open, if necessary) a log file to the MultiFileLog object.

Parameters:

flog: str, file, None, list of str or file objects, optional

File name or file object to be added.

initial_skip: int, optional

The number of blank line to be written to the file if not at the beginning of the file.

can_close: bool, None, optional

Indicates whether the file object can be closed by the close() function:

can_close is True – file can be closed by the close() function;
can_close is False – file will not be closed by the close() function;
can_close is None – automatic selection based on the type of the flog argument: True if flog is a string (e.g., file name), False otherwise.

mode: str, None, optional

File open mode: same meaning as the mode parameter of the Python’s built-in open() function. If None, the mode will be inherited from file open mode set during initialization.

Returns:

flog: file object: File object of newly opened (or attached) file.

close()[source]

Close all files opened by MultiFileLog.

It will close all files opened by MultiFileLog - essentially, all files added as file name. It will not close files added as file handles.

property count: Return the number of files attached to the MultiFileLog object excluding the sys.stdout file.

enable_console(enable=True, enableBold=True)[source]

Enable output to the standard console – sys.stdout.

Parameters:

enable: bool, optional: Enable or disable output to sys.stdout.
enableBold: bool, optional: Enable or disable writing bold text to console, e.g., 'WARNING:', 'ERROR:', etc.

error(msg, *fmt)[source]: Prints an error message. The word ‘ERROR:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.

flush()[source]: Flush all files.

important(msg, *fmt)[source]: Prints an “important” message. The word ‘IMPORTANT:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.

logentry(msg, *fmt, **skip)[source]

Write a log entry to the log files.

Parameters:

msg: str: String to be printed. Can contain replacement fields delimited by braces {}.
fmt: Parameters to be passed to str.format() method for formatting the string msg.
skip: int: Number of empty lines that should follow the log message.

Examples

>>> logentry('Sky background for chip {} is {}', 'SCI1', 10.0, skip=2)
will print:
Sky background for chip SCI1 is 10.0
followed by two blank lines.

print_endlog_msg()[source]: Print a message (“Log written to…”) indicating the end of the log.

set_close_flag(fh, can_close=True)[source]

Modify the “can be closed” status of a given file.

Parameters:

fh: file object
can_close: bool, optional: Indicates if the file can be closed by the close() function.

Note

File object fh must have been already added to the MultiFileLog object.

skip(nlines=1)[source]

Skip a specified number of blank lines.

Parameters:

nlines: int, optional

unclose_copy()[source]

Return a copy of the MultiFileLog object with all the attached files marked as “keep open”, that is, the close() will not close these files.

This is useful before passing the MultiFileLog object to a function that may add its own log files, and then attempt to close the files with the close() method. Thus, by passing an “unclose copy”, one can be sure that the files opened at the top level will not be closed by other functions to which the MultiFileLog object may be passed.

warning(msg, *fmt)[source]: Prints a warning message. The word ‘WARNING:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.

class stsci.skypac.utils.ResourceRefCount(resource, *close_args, resource_close_fnct=None, **close_kwargs)[source]

A class that implements reference counting for various resources: file objects, FITS HDU lists, etc. It is intended to be used as a mechanism of controlling the “lifespan” of resources that can be used in different parts of the code “indipendently”. The resource is kept open as long as the reference count is larger than zero. Once the reference count is decreased to 0, the resource is automatically closed.

Note

The reference count of the newly created ResourceRefCount object is set to 0. It is user’s responsibility to increment the reference count of this object through the call to hold() function.

Parameters:

resource: An object who must be kept open or closed based on reference count.
close_args: tuple: Positional arguments to be passed to the resource_close_fnct() function.
resource_close_fnct: function, optional: The function (usually a method of the attached resource), that can “close” the resource.
close_kwargs: dict: Keyword arguments to be passed to the resource_close_fnct() function.

property closed: Indicates if the resource is “closed”.

hold()[source]: Increment the reference count of the attached resource.

is_subscribed_on_close(obj)[source]: Check if the object is subscribed to on close notifications.

property refcount: Reference count.

release()[source]: Decrement reference count of the attached resource. If the reference count reaches zero, call all registered “on close” notify callbacks, and then call the “close” method on the resource which was set at initialization of the ResourceRefCount object. Finally, the resource property of the ResourceRefCount object will be set to None.

property resource: Resource attached to a ResourceRefCount object.

subscribe_close_notify(obj, callback=None)[source]: Set the object and its method that need to be called when the resource is about to be closed.

unsubscribe_close_notify(obj)[source]: Remove the object (and its method) from the list of callbacks that need to be notified of impending closing of the resource.

stsci.skypac.utils.almost_equal(arr1, arr2, fp_accuracy=None, fp_precision=None)[source]

Compares two values, or values of numpy.ndarray and verifies that these values are close to each other. For exact type (integers and boolean) the comparison is exact. For inexact types (float, numpy.float32, etc.) it checks that the values (or all values in a numpy.ndarray) satisfy the following inequality:

\[|v1-v2| \leq a + 10^{-p} \cdot |v2|,\]

where a is the accuracy (“absolute error”) and p is precision (“relative error”).

Parameters:

arr1: float, int, bool, str, numpy.ndarray, None, etc.: First value or array of values to be compared.
arr2: float, int, bool, str, numpy.ndarray, None, etc.: Second value or array of values to be compared.
fp_accuracy: int, float, None, optional: Accuracy to withing values should be close. Default value will use twice the value of the machine accuracy (machine epsilon) for the input type. This parameter has effect only when the values to be compared are of inexact type (e.g., float).
fp_precision: int, float, None, optional: Accuracy to withing values should be close. Default value will use twice the value of the machine precision (resolution) for the input type. This parameter has effect only when the values to be compared are of inexact type (e.g., float).

Returns:

almost_equal: bool: Returns True if input values are close enough to each other or False otherwise.

Raises:

ValueError: If fp_accuracy is negative.
TypeError: If fp_precision is not int, float, or None.

stsci.skypac.utils.count_extensions(img, extname='SCI')[source]

Return the number of extname extensions in the input image img. If extname is None, return the number of all image-like extensions.

Input parameters are identical to those of get_extver_list().

See also

get_ext_list, count_extensions

Examples

>>> get_extver_list('j9irw1rqq_flt.fits',extname='sci')
[1, 2]
>>> get_extver_list('j9irw1rqq_flt.fits',extname=None)
[1, 2, 3, 4, 5, 6, 8, 9, 10, 11

stsci.skypac.utils.is_countrate(hdulist, ext, units_kwd='BUNIT', guess_if_missing=True, telescope=None, instrument=None, verbose=True, flog=None)[source]

Infer the units of the data of the input image from the input image. Specifically, it tries to infer whether the units are counts (or count-like) or if the units are count-rate.

The units of data are determined from the BUNIT header keyword by searching its value for the division sign '/'. If the division sign is not found, then the units are assumed to be “counts”. If the division sign is found in the BUNIT value and if the numerator is one of the following: ‘ELECTRONS’,’COUNTS’, or ‘DN’, and denumerator is either ‘S’,’SEC’, or ‘SECOND’, then the units are assumed to be count-rate.

Parameters:

hdulist: `astropy.io.fits.HDUList`: astropy.io.fits.HDUList of the image.
ext: tuple, int, str: Extension specification for whose data the units need to be inferred. An int ext specifies extension number. A tuple in the form (str, int) specifies extension name and number. A string ext specifies extension name and the extension version is assumed to be 1. See documentation for astropy.io.fits.getData for examples.
units_kwd: str, optional: FITS header keyword describing data units of the image. This keyword is assumed to be in the header of the extension specified by the ext parameter.
guess_if_missing: bool, optional: Instructs to try make best guess on image units when the keyword specified by units_kwd is not found in the image header. The first action will be to look for this keyword in the primary header, and if not found, infer the units based on the telescope, instrument, and detector information.
telescope: str, None, optional: Specifies the telescope from which the data came. If not specified, the value specified in the 'TELESCOP' keyword in the primary header will be used.
instrument: str, None, optional: Specifies the instrument used for acquiring data. If not specified, the value specified in the 'INSTRUME' keyword in the primary header will be used.
verbose: bool, optional: Specifies whether to print warning messages.
flog: str, file, MultiFileLog, None, optional: Specifies the log file to which the messages should be printed. It can be a file name, a file object, a MultiFileLog object, or None.

Returns:

bool, None: Returns True if the units of the input image for a given extension are count-rate-like and False if the units are count-like. Returns None if the units cannot be inferred from the header.

stsci.skypac.utils.openImageEx(filename, mode='readonly', dqmode='readonly', memmap=True, saveAsMEF=True, output_base_fitsname=None, clobber=True, imageOnly=False, openImageHDU=True, openDQHDU=False, preferMEF=True, verbose=False)[source]

Open an image file and (if requested) the associated DQ file and return corresponding ImageRef objects.

This function is an enhanced version of stsci.tools.fileutil.openImage() function in that it can open both the image file and the associated DQ image. It also provides additional inormation about the opened files: file type, original file name, DQ model (“intrinsic”, where DQ data are placed in the same file as the science data, or “extrinsic” when DQ data are in a separate file from the science data), etc. Because of the way it was implemented, it requires half of the number of calls to astropy.io.fits.open thus making it almost twice as fast as the openImage() function.

Parameters:

filename: str: File name of the file to be opened. The image file formats are recognized: simple/MEF FITS, HST GEIS format, or WAIVER FITS format.
mode: str, optional: File mode used to open main image FITS file. See astropy.io.fits.open for more details.
dqmode: str, optional: File mode used to open DQ image FITS file. See parameter mode above for more details.
memmap: bool, optional: Should memory mapping to be used whe opening simple/MEF FITS?
saveAsMEF: bool, optional: Should an input GEIS or WAIVER FITS be converted to simple/MEF FITS?
output_base_fitsname: str, None, optional: The base name of the output simple/MEF FITS when saveAsMEF is True. If it is None, the file name of the converted file will be determined according to HST file naming conventions.
clobber: bool, optional: If saveAsMEF is True, should any existing output files be overwritten?
imageOnly: bool, optional: Should this function open the image file only? If True, then the DQ-related attributes will not be valid.
openImageHDU: bool, optional: Indicates whether the returned ImageRef object corresponding to the science image file should be in an open or closed state.
openDQHDU: bool, optional: Indicates whether the returned ImageRef object corresponding to the DQ image file should be in an open or closed state.
preferMEF: bool, optional: Should this function open an existing MEF file that complies with HST naming convention when the input file is in GEIS or WAIVER FITS format, even when saveAsMEF is False or clobber is False?
verbose: bool, optional: If True, some addition information will be printed out to standard output.

Returns:

(img, dqimg): (ImageRef, ImageRef): A tuple of ImageRef objects corresponding to the science image and to the DQ image. Each object in the returned tuple open or closed depending on the input arguments.

Note

If the returned object is open, then its reference count will be at least 1. The caller is responsible for “releasing” the object when it is no longer needed.

Note

If the DQ model of the opened file is “intrinsic”, then the dqimg component of the returned tuple will hold a reference counter to the same image. Thus, for “intrinsic” DQ data models, the reference count of the returned objects may be 2 (if both science openImageHDU and openDQHDU are True).

Raises:

ValueError: If input file is neither a GEIS file nor a FITS file.
ValueError: Errors occured while accessing/reading the file possibly due to corrupted file, non-compliant file format, etc.s

stsci.skypac.utils.skyval2txt(files='*_flt.fits', skyfile='skyfile.txt', skykwd='SKYUSER', default_ext=('SCI', '*'))[source]

A convenience function that allows retrieving computed sky background values from image headers and storing them in a text file that can be re-used by drizzlepac.astrodrizzle.AstroDrizzle(). This is particularly useful when performing sky matching on a large number of images which takes considerable time. Saving computed sky values to a text file allows re-running AstroDrizzle() without re-computing sky values.

Warning

The file specified by skyfile is overwritten without warning.

Note

Images that do not have specified extensions will be ignored.

Parameters:

files: str

File name(s), including extension specification if necessary, from which sky values should be retrieved:

a comma-separated list of valid science image file names (see note below) and (optionally) extension specifications, e.g.: 'j1234567q_flt.fits[1], j1234568q_flt.fits[sci,2]';

an @-file name, e.g., '@fits_files_with_skyvals.txt'.

Note

Valid science image file names are:

file names of existing FITS, GEIS, or WAIVER FITS files;
partial file names containing wildcard characters, e.g., '*_flt.fits';
Association (ASN) tables (must have _asn, or _asc suffix), e.g., 'j12345670_asn.fits'.

Warning

@-file names MAY NOT be followed by an extension specification.

Warning

If an association table or a partial file name with wildcard characters is followed by an extension specification, it will be considered that this extension specification applies to each file name in the association table or each file name obtained after wildcard expansion of the partial file name.

skyfile: str, optional

Output “skyfile” to which sky values from the image headers should be written out.

skykwd: str, optional

Header keyword holding the value of the computed sky background.

default_ext: int, tuple, optional

Default extension to be used with image files that to not have an extension specified.

stsci.skypac.utils.temp_mask_file(rootname, suffix, ext, data, dir=os.path.curdir, fnameOnly=False)[source]

Saves 2D data array to temporary simple FITS file. The name of the emporary file is generated based on the input parameters.

Parameters:

data: numpy array: Data to be written to the temporary FITS file. Data will be written in the primary HDU.
rootname: str: Root name of the file.
prefix: str, optional: Prefix to be added in front of the root name. If randomize_prefix is True, then a random string will be added to the right of the string specified by prefix (with no separator between them). Prefix (or the randomized prefix) will be separated from the root name by the string specified in sep. If prefix is an empty string ('') then no prefix will be prepended to the root file name.
suffix: str, optional: Suffix to be added to the root name. Suffix will be separated from the root name by the string specified in sep.
ext: int, str, or tuple of the form (str, int), optional: Extention to be added to the temporary file after the suffix. Extension name string will be separated from the suffix by the string specified in sep.
sep: str, optional: Separator string to be inserted between (randomized) prefix and root name, root name and suffix, and suffix and extension.
randomize_prefix: bool, optional: Specifies whether to add (postpend) a random string to string specified by prefix.
dir: str, optional: Directory to which the temporary file should be written. If directory dir is None then the file will be written to the default (for more details, see the explanation for argument dir to the tempfile.mstemp function).
fnameOnly: bool, optional: Specifies what should temp_mask_file return: file name of the created file (if fnameOnly is True), or a tuple with the file name of the created file and an open ImageRef object of that file.

Returns:

fname: str: File name of the temporary file.
mask: ImageRef: An open ImageRef object of the temporary FITS file. This is returned as a tuple together with the file name only when fnameOnly is False.

Note

Mask data will be in the Primary HDU.

Raises:

TypeError: Extension specifier must be either an integer, a string, or a tuple of the form (str, int).

Examples

>>> import numpy as np
>>> from stsci import skypac
>>> mask=np.ones((800,800),dtype=np.uint8)
>>> skypac.utils.temp_mask_file(mask, 'ua0x5001m',
...     suffix='skymatch_mask', ext=('sci',4), dir='/data/m87',
...     fnameOnly=True)
'/data/m87/tmp39gCpw_ua0x5001m_skymatch_mask_sci4.fits'
>>> skypac.utils.temp_mask_file(mask, 'ua0x5001m',
...     suffix='skymatch_mask', ext=('sci',4), dir='.', fnameOnly=True)
'tmpxl7LTO_ua0x5001m_skymatch_mask_sci4.fits'
>>> skypac.utils.temp_mask_file(mask, 'ua0x5001m',
...     suffix='skymatch_mask', ext=('sci',4), dir='.',fnameOnly=False)
('tmpxMcL5g_ua0x5001m_skymatch_mask_sci4.fits', <skypac.utils.ImageRef object at 0x101f5a3d0>)