Utility functions for parsing user catalog files for skypac

Module for parsing @-files or user input strings for use by stsci.skypac module.

Authors:: Mihai Cara
License:: LICENSE

class stsci.skypac.parseat.FileExtMaskInfo(default_ext=('SCI', '*'), default_mask_ext=0, clobber=False, doNotOpenDQ=False, fnamesOnly=False, im_fmode='update', dq_fmode='readonly', msk_fmode='readonly')[source]

A class that holds image, dq, user masks, and extensions to be used with these files. It is designed to facilitate keeping track of user input in catalog files.

This class is intended to be used primarily for functions such as parse_at_line() and other related functions as a return value. It is also used to initialize skypac.skyline.SkyLine objects.

FileExtMaskInfo was designed to be used in a specific ordered workflow. Below is a typical use of this class:

Initialize the object with the desired settings for default extensions to be used with the files (when a specific extension for a file is not provided) and the open modes for the files;
Add image file using image();
Add image’s extension(s) using append_ext();
[Optional; can be performed at any later stage] Add DQ file and extensions using DQimage() and dqext() methods;
Append mask files and extensions using append_mask();
[Optional] Finalize the FileExtMaskInfo() object.

Parameters:

default_ext: int, tuple, optional: Default extension to be used with image files that to not have an extension specified.
default_mask_ext: int, tuple, optional: Default extension to be used with image mask files that to not have an extension specified.
clobber: bool, optional: If a file being appended is in GEIS or WAIVER FITS format, should any existing MEF files be overwritten?
doNotOpenDQ: bool, optional: Should the DQ files be oppened when simultaneously with the image files?
fnamesOnly: bool, optional: Return file names only, or open the files and return ImageRef objects?
im_fmode: str, optional: File mode to be used to open image FITS file. See astropy.io.fits.open for more details.
dq_fmode: str, optional: File mode to be used to open DQ FITS file. This is valid only if the DQ model of the image file is ‘external’ (see documentation for ImageRef for more details). For ‘intrinsic’ DQ model the DQ files will use the same setting as for im_fmode.
msk_fmode: str, optional: File mode to be used to open mask files.

Attributes:

clobber: bool: If a file being appended is in GEIS or WAIVER FITS format, should any existing MEF files be overwritten?
dq_bits: int: Bitmask specifying what pixels in the mask should be removed (or kept) with the precise interpretation being left to the user. This flag is not used by this class but was designed to be passed to other functions that will use FileExtMaskInfo.

property DQimage

DQ image (file or ImageRef object depending on the fnamesOnly value).

Getter:: Get the ImageRef DQ image object.
Setter:: Set the DQ file.
Type:: str, ImageRef

append_ext(ext)[source]

Append extensions to the list of “selected” extensions for the image file.

Note

This function appends the extensions. If it is desired to set the extensions, use replace_ext() instead.

Parameters:

ext: int, tuple, None, list: Extension specification: None, an integer extension number, a tuple (extension number, extension version) where extension version can be '*' which will be replaced with the extension versions of all extensions having given extension name. If ext is None, it will be replaced with the default extension specification set during the initialization of the FileExtMaskInfo object.

append_mask(mask, ext, mask_stat=None)[source]

Append a mask image and its extension(s).

Note

Mask files and extensions are kept in ordered lists and their order is significant: the first mask file-extension pair is associated with the first extension of the science image file set with append_ext() and so on.

Parameters:

mask: str, ImageRef, None: Mask image file. Can be a string file name, an ImageRef object (only if fnamesOnly is False), or None (to act as a place holder in the ordered list of extensions).
ext: int, tuple, None, list: Extension specification: None, an integer extension number, a tuple (extension number, extension version) where extension version can be '*' which will be replaced with the extension versions of all extensions having given extension name. If ext is None, it will be replaced with the default extension specification for mask images set during the initialization of the FileExtMaskInfo object.
mask_stat: `os.stat_result`, optional: An os.stat_result structure for the input mask file. If None, then append_mask() will compute stat for the input mask file.

Raises:

RuntimeError: Raised if attempting to add masks when the science image was not yet set.
AssertionError: Raised if finalized is True.
TypeError: Raised if mask is an ImageRef object but fnamesOnly is True or if mask argument is of incorrect type.
ValueError: If mask is an ImageRef, it must not be closed.

clear_masks()[source]: Remove all attached mask files and extensions.

convert2ImageRef()[source]

Replace any existing file names with opened ImageRef objects and change the fnamesOnly property to False.

Note

The finalized property will not be modified.

Warning

The FileExtMaskInfo must not have been finalized (finalized is False) and must contain file names only (fnamesOnly is True).

Raises:

AssertionError: Raised if finalized is True or fnamesOnly is False.

See also

append_ext

stsci.skypac.parseat.parse_at_file(fname, default_ext=('SCI', '*'), default_mask_ext=0, clobber=False, fnamesOnly=False, doNotOpenDQ=False, match2Images=None, im_fmode='update', dq_fmode='readonly', msk_fmode='readonly', logfile=None, verbose=False)[source]

This function is similar to parse_at_line(), the main difference being that is can parse multiple (EOL terminated) lines of the format described in the documentation for parse_at_line().

Below we describe only differences bewtween this function and parse_at_line().

Parameters:

fname: str: File name of the catalog file.
match2Images: list of str, list of ImageRef, None, optional: List of file names or ImageRef objects whose mask specifications are to be read from the catalog file. Mask specifications for other files in the catalog that do not match the files in the match2Images list will be ignored. If match2Images is None, then all files from the catalog will be read.
logfile: str, file, MultiFileLog, None, optional: Specifies the log file to which the messages should be printed. It can be a file name, a file object, a MultiFileLog object, or None.

Returns:

list: Returns a list of filenames if fnamesOnly is True or a list of FileExtMaskInfo objects if fnamesOnly is False.

stsci.skypac.parseat.parse_at_line(fstring, default_ext=('SCI', '*'), default_mask_ext=0, \ clobber=False, fnamesOnly=False, doNotOpenDQ=False, \ im_fmode='update', dq_fmode='readonly', msk_fmode='readonly', verbose=False)[source]

Parse a line from a catalog file containing a science image file and extension specifications and optionally followed by a comma-separated list of mask files and extension specifications (or None).

File names will be stripped of leading and trailing white spaces. If it is essential to keep these spaces, file names may be enclosed in single or double quotation marks. Quotation marks may also be required when file names contain special characters used to separate file names and extension specifications: ,[]{}

Extension specifications must follow the file name and must be delimited by either square or curly brackets. Curly brackets allow specifying multiple comma-separated extensions: integer extension numbers and/or tuples (‘ext name’, ext version).

Some possible ways of specifying extensions:
[1] – extension number

[‘sci’,2] – extension name and version

{1,4,(‘sci’,3)} – multiple extension specifications, including tuples

{(‘sci’,*)} – wildcard extension versions (i.e., all extensions with extension name ‘sci’)

[‘sci’] – equivalent to [‘sci’, 1]

{‘sci’} – equivalent to {(‘sci’,*)}

For extensions in the science image for which no mask file is provided, the corresponding mask file names may be omitted (but a comma must still be used to show that no mask is provided in that position) or None can be used in place of the file name. NOTE: 'None' (in quotation marks) will be interpreted as a file named 'None'.

Some examples of possible user input:
image1.fits{1,2,('sci',3)},mask1.fits,,mask3.fits[0]

In this case:

image1.fits[1] is associated with mask1.fits[0];

image1.fits[2] does not have an associated mask;

image1.fits['sci',3] is associated with mask3.fits[0].

– Assume image2.fits has 4 ‘SCI’ extensions:

image2.fits{'sci'},None,,mask3.fits

In this case:

image2.fits['sci', 1] and image2.fits['sci', 2] and image2.fits['sci', 4] do not have an associated mask; image2.fits['sci', 3] is associated with mask3.fits[0]

Note

Wildcard extension version in extension specification can be expanded only when fnamesOnly is False.

Parameters:

fstring: str

A comma-separated string describing the image file name and (optionally) followed by the extension specifier (e.g., [sci,1,2], or [sci]). The image file name may be followed (comma-separated) by optional mask file names (and their extension specifier).

File and extension names may NOT contain leading and/or trailing spaces, commas, and/or square or curly brakets.

default_ext: int, tuple, optional

Default extension to be used with image files that to not have an extension specified.

default_mask_ext: int, tuple, optional

Default extension to be used with image mask files that to not have an extension specified.

fnamesOnly: bool, optional

Return file names only, or open the files and return ImageRef objects?

doNotOpenDQ: bool, optional

Should the DQ files be oppened when simultaneously with the image files?

im_fmode: str, optional

File mode to be used to open image FITS file. See astropy.io.fits.open for more details.

dq_fmode: str, optional

File mode to be used to open DQ FITS file. This is valid only if the DQ model of the image file is ‘external’ (see documentation for ImageRef for more details). For ‘intrinsic’ DQ model the DQ files will use the same setting as for im_fmode.

msk_fmode: str, optional

File mode to be used to open mask files.

verbose: bool, optional

Specifies whether to print warning messages.

Returns:

FileExtMaskInfo: A FileExtMaskInfo object.

Raises:

ValueError

Input argument ‘fstring’ must be a Python string.
Input argument ‘fstring’ contains either unbalanced or nested square brackets.
Extension specification must be preceeded by a valid image file name.

stsci.skypac.parseat.parse_cs_line(csline, default_ext=('SCI', '*'), clobber=False, fnamesOnly=False, doNotOpenDQ=False, im_fmode='update', dq_fmode='readonly', msk_fmode='readonly', logfile=None, verbose=False)[source]

This function is similar to parse_at_line(), the main difference being the content of the input string: a list of comma-separated science image file names. No masks can be specified and file names must be valid (i.e., None is not allowed). Extension specifications are allowed and must folow the same sintax as described for parse_at_line().

Below we describe only differences bewtween this function and parse_at_line().

Parameters:

csline: str

User input string that needs to be parsed containing one of the following:

a comma-separated list of valid science image file names (see note below) and (optionally) extension specifications, e.g.: 'j1234567q_flt.fits[1], j1234568q_flt.fits[sci,2]';

an @-file name, e.g., '@files_to_match.txt'.

Note

Valid science image file names are:

file names of existing FITS, GEIS, or WAIVER FITS files;
partial file names containing wildcard characters, e.g., '*_flt.fits';
Association (ASN) tables (must have _asn, or _asc suffix), e.g., 'j12345670_asn.fits'.

Warning

@-file names MAY NOT be followed by an extension specification.

Warning

If an association table or a partial file name with wildcard characters is followed by an extension specification, it will be considered that this extension specification applies to each file name in the association table or each file name obtained after wildcard expansion of the partial file name.

logfile: str, file, MultiFileLog, None, optional

Specifies the log file to which the messages should be printed. It can be a file name, a file object, a MultiFileLog object, or None.

Returns:

list: Returns a list of filenames if fnamesOnly is True or a list of FileExtMaskInfo objects if fnamesOnly is False.