Utility functions for skypac¶
This module provides utility functions for use
by stsci.skypac
module.
- Authors
Mihai Cara
- License
- class stsci.skypac.utils.ImageRef(hdulist_refcnt=None)[source]¶
A lightweight class that supports reference counting for FITS images and holds a
ResourceRefCount
object. It provides several attributes that describe main characteristics of the image (file) and essential functions for manipulating reference count.- Parameters
- hdulist_refcnt: ResourceRefCount
A
ResourceRefCount
object holding aastropy.io.fits.HDUList
object.
- Attributes
- filename: str
Name of the opened file. Can be
None
for in-memory createdastropy.io.fits.HDUList
objects.- can_reload_data: bool
True
for files attached to a physical file,False
for in-memoryastropy.io.fits.HDUList
objects.- original_fname: str
Original name of the file as requested by the user. Note: may be different from
filename
if the orininal file was in GEIS or WAIVER FITS format and subsequently was converted to a MEF FITS format. In that case this attribute will show the name of the original GEIS or WAIVER FITS file.- original_ftype: str
Type of the original file. Can take one of the following values: ‘MEF’, ‘SIMPLE’, ‘GEIS’, ‘WAIVER’, or ‘UNKNOWN’.
- original_exists: bool
Indicates if the physical file exists. It is
False
for in-memory images.- mef_fname: str, None
Name of the MEF FITS file if exists,
None
otherwise.- mef_exists: bool
Indicates whether the MEF FITS file exists.
- DQ_model: str, None
Type of the DQ model: ‘external’ for WFPC, WFPC2, and FOC instruments (or non-HST data if set so) that have DQ data in a separate (from image) file and ‘intrinsic’ for ACS, etc. images that have DQ extensions in the image file. It is
None
if the image does not have DQ data.- telescope: str
Telescope that acquired the image.
- instrument: str
Instrument used to acquire data.
- fmode: str
File mode used to open FITS file. See
astropy.io.fits.open
for more details.- memmap: bool
Is the
astropy.io.fits.HDUList
memory mapped?
- property extname¶
Extension name of the first extension.
- property hdu¶
astropy.io.fits.HDUList
of the attached image.
- property refcount¶
Reference count of the attached
astropy.io.fits.HDUList
.
- release()[source]¶
Decrement reference count of the attached resource. If the reference count reaches zero, the attached
ResourceRefCount
object will be closed and set toNone
.
- set_HDUList_RefCount(hdulist_refcnt=None)[source]¶
Set (attach) a new
ResourceRefCount
object that holds aastropy.io.fits.HDUList
object. This is allowed only if the already attachedResourceRefCount
can be closed. The reference count of theResourceRefCount
being attached will be incremented.- Parameters
- hdulist_refcnt: ResourceRefCount, None
A
ResourceRefCount
object containing aastropy.io.fits.HDUList
object. If it isNone
,it will release and close the attachedResourceRefCount
object.
- Raises
- ValueError
The (already) attached
ResourceRefCount
must have reference count <= 1 so that it can be closed before being replaced with a new resource.- ValueError
The resource being attached must be in an open state.
- class stsci.skypac.utils.MultiFileLog(console=True, enableBold=True, flog=None, append=True, autoflush=True, appendEOL=True)[source]¶
This is a class that facilitates writting to multiple files.
MultiFileLog
stores multiple file objects and can write the same log entry to all of them. It also facilitates controlling when a function can close a log file. Finally, it provides some utility functions that automate such things as appending EOL at the end of the log entry, flushing the files (to avoid losing log entries in case of uncaught exceptions), displayingWARNING
,ERROR
, etc. in bold on standard streams (sys.stdout
, etc.)- Parameters
- console: bool, optional
Enables writting to the standard output.
- enableBold: bool, optional
Enable or disable writing bold text to console, e.g.,
'WARNING:'
'ERROR:'
, etc.- flog: str, file, None, list of str or file objects, optional
File name or file object to be added to
MultiFileLog
during the initialization. More files can be added lated withadd_logfile()
.- append: bool, optional
Default open mode for the files that need to be opened (e.g., that are passed as file names). If
append
isTrue
, new files added to aMultiFileLog
object will be opened in the “append” mode: new log entries will be appended to existing files – same as mode ‘a’ of standard functionopen()
. Whenappend
isFalse
, any existing files will be overwritten.- autoflush: bool, optional
Indicates whether or not to flush files after each log entry.
- appendEOL: bool, optional
Indicates whether or not to add EOL at the end of each log entry.
- add_logfile(flog, initial_skip=2, can_close=None, mode=None)[source]¶
Add (and open, if necessary) a log file to the
MultiFileLog
object.- Parameters
- flog: str, file, None, list of str or file objects, optional
File name or file object to be added.
- initial_skip: int, optional
The number of blank line to be written to the file if not at the beginning of the file.
- can_close: bool, None, optional
Indicates whether the file object can be closed by the
close()
function:- mode: str, None, optional
File open mode: same meaning as the
mode
parameter of the Python’s built-inopen()
function. IfNone
, the mode will be inherited from file open mode set during initialization.
- Returns
- flog: file object
File object of newly opened (or attached) file.
- close()[source]¶
Close all files opened by
MultiFileLog
.It will close all files opened by
MultiFileLog
- essentially, all files added as file name. It will not close files added as file handles.
- property count¶
Return the number of files attached to the
MultiFileLog
object excluding thesys.stdout
file.
- enable_console(enable=True, enableBold=True)[source]¶
Enable output to the standard console –
sys.stdout
.- Parameters
- enable: bool, optional
Enable or disable output to
sys.stdout
.- enableBold: bool, optional
Enable or disable writing bold text to console, e.g.,
'WARNING:'
,'ERROR:'
, etc.
- error(msg, *fmt)[source]¶
Prints an error message. The word ‘ERROR:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.
- important(msg, *fmt)[source]¶
Prints an “important” message. The word ‘IMPORTANT:’ will be printed as bold on the console and enclosed in asterisks (*) in a disk file.
- logentry(msg, *fmt, **skip)[source]¶
Write a log entry to the log files.
- Parameters
- msg: str
String to be printed. Can contain replacement fields delimited by braces
{}
.- fmt
Parameters to be passed to
str.format()
method for formatting the stringmsg
.- skip: int
Number of empty lines that should follow the log message.
Examples
>>> logentry('Sky background for chip {} is {}', 'SCI1', 10.0, skip=2) will print: Sky background for chip SCI1 is 10.0 followed by two blank lines.
- set_close_flag(fh, can_close=True)[source]¶
Modify the “can be closed” status of a given file.
- Parameters
- fh: file object
- can_close: bool, optional
Indicates if the file can be closed by the
close()
function.Note
File object
fh
must have been already added to theMultiFileLog
object.
- unclose_copy()[source]¶
Return a copy of the
MultiFileLog
object with all the attached files marked as “keep open”, that is, theclose()
will not close these files.This is useful before passing the
MultiFileLog
object to a function that may add its own log files, and then attempt to close the files with theclose()
method. Thus, by passing an “unclose copy”, one can be sure that the files opened at the top level will not be closed by other functions to which theMultiFileLog
object may be passed.
- class stsci.skypac.utils.ResourceRefCount(resource, *close_args, resource_close_fnct=None, **close_kwargs)[source]¶
A class that implements reference counting for various resources: file objects, FITS HDU lists, etc. It is intended to be used as a mechanism of controlling the “lifespan” of resources that can be used in different parts of the code “indipendently”. The resource is kept open as long as the reference count is larger than zero. Once the reference count is decreased to 0, the resource is automatically closed.
Note
The reference count of the newly created
ResourceRefCount
object is set to 0. It is user’s responsibility to increment the reference count of this object through the call tohold()
function.- Parameters
- resource
An object who must be kept open or closed based on reference count.
- close_args: tuple
Positional arguments to be passed to the
resource_close_fnct()
function.- resource_close_fnct: function, optional
The function (usually a method of the attached resource), that can “close” the resource.
- close_kwargs: dict
Keyword arguments to be passed to the
resource_close_fnct()
function.
- property closed¶
Indicates if the resource is “closed”.
- property refcount¶
Reference count.
- release()[source]¶
Decrement reference count of the attached resource. If the reference count reaches zero, call all registered “on close” notify callbacks, and then call the “close” method on the resource which was set at initialization of the
ResourceRefCount
object. Finally, theresource
property of theResourceRefCount
object will be set toNone
.
- property resource¶
Resource attached to a
ResourceRefCount
object.
- stsci.skypac.utils.almost_equal(arr1, arr2, fp_accuracy=None, fp_precision=None)[source]¶
Compares two values, or values of
numpy.ndarray
and verifies that these values are close to each other. For exact type (integers and boolean) the comparison is exact. For inexact types (float
,numpy.float32
, etc.) it checks that the values (or all values in anumpy.ndarray
) satisfy the following inequality:\[|v1-v2| \leq a + 10^{-p} \cdot |v2|,\]where
a
is the accuracy (“absolute error”) andp
is precision (“relative error”).- Parameters
- arr1: float, int, bool, str, numpy.ndarray, None, etc.
First value or array of values to be compared.
- arr2: float, int, bool, str, numpy.ndarray, None, etc.
Second value or array of values to be compared.
- fp_accuracy: int, float, None, optional
Accuracy to withing values should be close. Default value will use twice the value of the machine accuracy (machine epsilon) for the input type. This parameter has effect only when the values to be compared are of inexact type (e.g.,
float
).- fp_precision: int, float, None, optional
Accuracy to withing values should be close. Default value will use twice the value of the machine precision (resolution) for the input type. This parameter has effect only when the values to be compared are of inexact type (e.g.,
float
).
- Returns
- Raises
- stsci.skypac.utils.count_extensions(img, extname='SCI')[source]¶
Return the number of
extname
extensions in the input imageimg
. Ifextname
isNone
, return the number of all image-like extensions.Input parameters are identical to those of
get_extver_list()
.See also
Examples
>>> count_extensions('j9irw1rqq_flt.fits',extname='SCI') 2 >>> count_extensions('j9irw1rqq_flt.fits',extname=None) 10
- stsci.skypac.utils.ext2str(ext, compact=False, default_extver=1)[source]¶
Return a string representation of an extension specification.
- Parameters
- ext: tuple, int, str
Extension specification can be a tuple of the form (str,int), e.g., (‘sci’,1), an integer (extension number), or a string (extension name).
- compact: bool, optional
If
compact
isTrue
the returned string will have extension name quoted and separated by a comma from the extension number, e.g.,"'sci',1"
. Ifcompact
isFalse
the returned string will have extension version immediately follow the extension name, e.g.,'sci1'
.- default_extver: int, optional
Specifies the extension version to be used when the
ext
parameter is a string (extension name).
- Returns
- strext: str
String representation of extension specification
ext
.
- Raises
- TypeError
Unexpected extension type.
Examples
>>> ext2str('sci',compact=False,default_extver=6) "'sci',6" >>> ext2str(('sci',2)) "'sci',2" >>> ext2str(4) '4' >>> ext2str('dq') "'dq',1" >>> ext2str('dq',default_extver=2) "'dq',2" >>> ext2str('sci',compact=True,default_extver=2) 'sci2'
- stsci.skypac.utils.file_name_components(fname, detect_HST_FITS_suffix=True)[source]¶
Splits base file name into a root, suffix, and extension. Given a full path, this function extracts the base name, and splits it into three components: root name, suffix, and file extension.
- Parameters
- fname: str
file name
- detect_HST_FITS_suffix: bool, optional
If True, detects the suffix of most HST files by looking for the rightmost occurence of the underscore (‘_’) in the file name.
- Returns
- root: str
Root name of the file. When
detect_HST_FITS_suffix
isTrue
, this is the part of the file name preceding the rightmost suffix separator (‘_’). Otherwise, it is the base file name without file extension.- suffix: str
If
detect_HST_FITS_suffix
isTrue
, this field will contain the suffix of most HST files, i.e., the part of the file name contained between the rightmost suffix separator (‘_’) and file extension separator. This return value will be an empty string if Ifdetect_HST_FITS_suffix
isFalse
or if the file name has no extension separator.- fext: str
File extension
Examples
>>> file_name_components('/data/m87/ua0x5001m_c0f.fits') ('ua0x5001m', 'c0f', '.fits') >>> file_name_components('/data/m87/ua0x5001m_c0f.fits',False) ('ua0x5001m_c0f', '', '.fits')
- stsci.skypac.utils.get_ext_list(img, extname='SCI')[source]¶
Return a list of all extension versions of
extname
extensions.img
can be either a file name or aastropy.io.fits.HDUList
object.This function is similar to
get_extver_list()
, the main difference being that it returns a list of fully qualified extensions: either tuples of the form(extname, extver)
or integer extension numbers (whenextname
isNone
).See also
Examples
>>> get_ext_list('j9irw1rqq_flt.fits',extname='SCI') [('SCI', 1), ('SCI', 2)] >>> get_ext_list('j9irw1rqq_flt.fits',extname=None) [1, 2, 3, 4, 5, 6, 8, 9, 10, 11]
- stsci.skypac.utils.get_extver_list(img, extname='SCI')[source]¶
Return a list of all extension versions with
extname
extension names. Ifextname
isNone
, return extension numbers of all image-like extensions.Note
If input image is a
ImageRef
, this function will not modify its reference count.- Parameters
- img: str, `astropy.io.fits.HDUList`, or `~skypac.utils.ImageRef`
Input image object. If
img
is a string object (file name) then that file will be opened. If the file pointed to by the file name is a GEIS or WAIVER FITS file, it will be converted to a simple/MEF FITS format ifclobber
isTrue
.- extname: str, optional
Indicates extension name for which all existing extension versions should be found. If
extname
isNone
, thenget_extver_list
will return a list of extension numbers of all image-like extensions.
- Returns
- extver: list
List of extension versions corresponding to the input
extname
. Ifextname
isNone
, it will return a list of extension numbers of all image-like extensions.
- Raises
- IOError
Unable to open input image file.
- TypeError
Argument
img
must be either a file name (str), anImageRef
, or aastropy.io.fits.HDUList
object.- TypeError
Argument
extname
must be either a string orNone
.
See also
Examples
>>> get_extver_list('j9irw1rqq_flt.fits',extname='sci') [1, 2] >>> get_extver_list('j9irw1rqq_flt.fits',extname=None) [1, 2, 3, 4, 5, 6, 8, 9, 10, 11
- stsci.skypac.utils.is_countrate(hdulist, ext, units_kwd='BUNIT', guess_if_missing=True, telescope=None, instrument=None, verbose=True, flog=None)[source]¶
Infer the units of the data of the input image from the input image. Specifically, it tries to infer whether the units are counts (or count-like) or if the units are count-
rate
.The units of data are determined from the
BUNIT
header keyword by searching its value for the division sign'/'
. If the division sign is not found, then the units are assumed to be “counts”. If the division sign is found in theBUNIT
value and if the numerator is one of the following: ‘ELECTRONS’,’COUNTS’, or ‘DN’, and denumerator is either ‘S’,’SEC’, or ‘SECOND’, then the units are assumed to be count-rate.- Parameters
- hdulist: `astropy.io.fits.HDUList`
astropy.io.fits.HDUList
of the image.- ext: tuple, int, str
Extension specification for whose data the units need to be inferred. An int
ext
specifies extension number. A tuple in the form (str, int) specifies extension name and number. A stringext
specifies extension name and the extension version is assumed to be 1. See documentation forastropy.io.fits.getData
for examples.- units_kwd: str, optional
FITS header keyword describing data units of the image. This keyword is assumed to be in the header of the extension specified by the
ext
parameter.- guess_if_missing: bool, optional
Instructs to try make best guess on image units when the keyword specified by
units_kwd
is not found in the image header. The first action will be to look for this keyword in the primary header, and if not found, infer the units based on the telescope, instrument, and detector information.- telescope: str, None, optional
Specifies the telescope from which the data came. If not specified, the value specified in the
'TELESCOP'
keyword in the primary header will be used.- instrument: str, None, optional
Specifies the instrument used for acquiring data. If not specified, the value specified in the
'INSTRUME'
keyword in the primary header will be used.- verbose: bool, optional
Specifies whether to print warning messages.
- flog: str, file, MultiFileLog, None, optional
Specifies the log file to which the messages should be printed. It can be a file name, a file object, a
MultiFileLog
object, orNone
.
- Returns
- stsci.skypac.utils.openImageEx(filename, mode='readonly', dqmode='readonly', memmap=True, saveAsMEF=True, output_base_fitsname=None, clobber=True, imageOnly=False, openImageHDU=True, openDQHDU=False, preferMEF=True, verbose=False)[source]¶
Open an image file and (if requested) the associated DQ file and return corresponding
ImageRef
objects.This function is an enhanced version of
stsci.tools.fileutil.openImage()
function in that it can open both the image file and the associated DQ image. It also provides additional inormation about the opened files: file type, original file name, DQ model (“intrinsic”, where DQ data are placed in the same file as the science data, or “extrinsic” when DQ data are in a separate file from the science data), etc. Because of the way it was implemented, it requires half of the number of calls toastropy.io.fits.open
thus making it almost twice as fast as theopenImage()
function.- Parameters
- filename: str
File name of the file to be opened. The image file formats are recognized: simple/MEF FITS, HST GEIS format, or WAIVER FITS format.
- mode: str, optional
File mode used to open main image FITS file. See
astropy.io.fits.open
for more details.- dqmode: str, optional
File mode used to open DQ image FITS file. See parameter
mode
above for more details.- memmap: bool, optional
Should memory mapping to be used whe opening simple/MEF FITS?
- saveAsMEF: bool, optional
Should an input GEIS or WAIVER FITS be converted to simple/MEF FITS?
- output_base_fitsname: str, None, optional
The base name of the output simple/MEF FITS when
saveAsMEF
isTrue
. If it isNone
, the file name of the converted file will be determined according to HST file naming conventions.- clobber: bool, optional
If
saveAsMEF
isTrue
, should any existing output files be overwritten?- imageOnly: bool, optional
Should this function open the image file only? If
True
, then the DQ-related attributes will not be valid.- openImageHDU: bool, optional
Indicates whether the returned
ImageRef
object corresponding to the science image file should be in an open or closed state.- openDQHDU: bool, optional
Indicates whether the returned
ImageRef
object corresponding to the DQ image file should be in an open or closed state.- preferMEF: bool, optional
Should this function open an existing MEF file that complies with HST naming convention when the input file is in GEIS or WAIVER FITS format, even when
saveAsMEF
isFalse
orclobber
isFalse
?- verbose: bool, optional
If
True
, some addition information will be printed out to standard output.
- Returns
- (img, dqimg): (ImageRef, ImageRef)
A tuple of
ImageRef
objects corresponding to the science image and to the DQ image. Each object in the returned tuple open or closed depending on the input arguments.Note
If the returned object is open, then its reference count will be at least 1. The caller is responsible for “releasing” the object when it is no longer needed.
Note
If the DQ model of the opened file is “intrinsic”, then the
dqimg
component of the returned tuple will hold a reference counter to the same image. Thus, for “intrinsic” DQ data models, the reference count of the returned objects may be 2 (if both scienceopenImageHDU
andopenDQHDU
areTrue
).
- Raises
- ValueError
If input file is neither a GEIS file nor a FITS file.
- ValueError
Errors occured while accessing/reading the file possibly due to corrupted file, non-compliant file format, etc.s
- stsci.skypac.utils.skyval2txt(files='*_flt.fits', skyfile='skyfile.txt', skykwd='SKYUSER', default_ext=('SCI', '*'))[source]¶
A convenience function that allows retrieving computed sky background values from image headers and storing them in a text file that can be re-used by
drizzlepac.astrodrizzle.AstroDrizzle()
. This is particularly useful when performing sky matching on a large number of images which takes considerable time. Saving computed sky values to a text file allows re-runningAstroDrizzle()
without re-computing sky values.Warning
The file specified by
skyfile
is overwritten without warning.Note
Images that do not have specified extensions will be ignored.
- Parameters
- files: str
File name(s), including extension specification if necessary, from which sky values should be retrieved:
a comma-separated list of valid science image file names (see note below) and (optionally) extension specifications, e.g.:
'j1234567q_flt.fits[1], j1234568q_flt.fits[sci,2]'
;an @-file name, e.g.,
'@fits_files_with_skyvals.txt'
.
Note
Valid science image file names are:
file names of existing FITS, GEIS, or WAIVER FITS files;
partial file names containing wildcard characters, e.g.,
'*_flt.fits'
;Association (ASN) tables (must have
_asn
, or_asc
suffix), e.g.,'j12345670_asn.fits'
.
Warning
@-file names MAY NOT be followed by an extension specification.
Warning
If an association table or a partial file name with wildcard characters is followed by an extension specification, it will be considered that this extension specification applies to each file name in the association table or each file name obtained after wildcard expansion of the partial file name.
- skyfile: str, optional
Output “skyfile” to which sky values from the image headers should be written out.
- skykwd: str, optional
Header keyword holding the value of the computed sky background.
- default_ext: int, tuple, optional
Default extension to be used with image files that to not have an extension specified.
- stsci.skypac.utils.temp_mask_file(rootname, suffix, ext, data, dir=os.path.curdir, fnameOnly=False)[source]¶
Saves 2D data array to temporary simple FITS file. The name of the emporary file is generated based on the input parameters.
- Parameters
- data: numpy array
Data to be written to the temporary FITS file. Data will be written in the primary HDU.
- rootname: str
Root name of the file.
- prefix: str, optional
Prefix to be added in front of the root name. If
randomize_prefix
isTrue
, then a random string will be added to the right of the string specified byprefix
(with no separator between them). Prefix (or the randomized prefix) will be separated from the root name by the string specified insep
. Ifprefix
is an empty string (''
) then no prefix will be prepended to the root file name.- suffix: str, optional
Suffix to be added to the root name. Suffix will be separated from the root name by the string specified in
sep
.- ext: int, str, or tuple of the form (str, int), optional
Extention to be added to the temporary file after the suffix. Extension name string will be separated from the suffix by the string specified in
sep
.- sep: str, optional
Separator string to be inserted between (randomized) prefix and root name, root name and suffix, and suffix and extension.
- randomize_prefix: bool, optional
Specifies whether to add (postpend) a random string to string specified by
prefix
.- dir: str, optional
Directory to which the temporary file should be written. If directory
dir
isNone
then the file will be written to the default (for more details, see the explanation for argumentdir
to the tempfile.mstemp function).- fnameOnly: bool, optional
Specifies what should
temp_mask_file
return: file name of the created file (iffnameOnly
isTrue
), or a tuple with the file name of the created file and an openImageRef
object of that file.
- Returns
- fname: str
File name of the temporary file.
- mask: ImageRef
An open
ImageRef
object of the temporary FITS file. This is returned as a tuple together with the file name only whenfnameOnly
isFalse
.Note
Mask data will be in the Primary HDU.
- Raises
- TypeError
Extension specifier must be either an integer, a string, or a tuple of the form (str, int).
Examples
>>> import numpy as np >>> from stsci import skypac >>> mask=np.ones((800,800),dtype=np.uint8) >>> skypac.utils.temp_mask_file(mask, 'ua0x5001m', ... suffix='skymatch_mask', ext=('sci',4), dir='/data/m87', ... fnameOnly=True) '/data/m87/tmp39gCpw_ua0x5001m_skymatch_mask_sci4.fits' >>> skypac.utils.temp_mask_file(mask, 'ua0x5001m', ... suffix='skymatch_mask', ext=('sci',4), dir='.', fnameOnly=True) 'tmpxl7LTO_ua0x5001m_skymatch_mask_sci4.fits' >>> skypac.utils.temp_mask_file(mask, 'ua0x5001m', ... suffix='skymatch_mask', ext=('sci',4), dir='.',fnameOnly=False) ('tmpxMcL5g_ua0x5001m_skymatch_mask_sci4.fits', <skypac.utils.ImageRef object at 0x101f5a3d0>)