Utility functions for parsing user catalog files for skypac

Module for parsing @-files or user input strings for use by stsci.skypac module.

Authors

Mihai Cara

License

LICENSE

class stsci.skypac.parseat.FileExtMaskInfo(default_ext=('SCI', '*'), default_mask_ext=0, clobber=False, doNotOpenDQ=False, fnamesOnly=False, im_fmode='update', dq_fmode='readonly', msk_fmode='readonly')[source]

A class that holds image, dq, user masks, and extensions to be used with these files. It is designed to facilitate keeping track of user input in catalog files.

This class is intended to be used primarily for functions such as parse_at_line() and other related functions as a return value. It is also used to initialize skypac.skyline.SkyLine objects.

FileExtMaskInfo was designed to be used in a specific ordered workflow. Below is a typical use of this class:

  1. Initialize the object with the desired settings for default extensions to be used with the files (when a specific extension for a file is not provided) and the open modes for the files;

  2. Add image file using image();

  3. Add image’s extension(s) using append_ext();

  4. [Optional; can be performed at any later stage] Add DQ file and extensions using DQimage() and dqext() methods;

  5. Append mask files and extensions using append_mask();

  6. [Optional] Finalize the FileExtMaskInfo() object.

Parameters
default_ext: int, tuple, optional

Default extension to be used with image files that to not have an extension specified.

default_mask_ext: int, tuple, optional

Default extension to be used with image mask files that to not have an extension specified.

clobber: bool, optional

If a file being appended is in GEIS or WAIVER FITS format, should any existing MEF files be overwritten?

doNotOpenDQ: bool, optional

Should the DQ files be oppened when simultaneously with the image files?

fnamesOnly: bool, optional

Return file names only, or open the files and return ImageRef objects?

im_fmode: str, optional

File mode to be used to open image FITS file. See astropy.io.fits.open for more details.

dq_fmode: str, optional

File mode to be used to open DQ FITS file. This is valid only if the DQ model of the image file is ‘external’ (see documentation for ImageRef for more details). For ‘intrinsic’ DQ model the DQ files will use the same setting as for im_fmode.

msk_fmode: str, optional

File mode to be used to open mask files.

Attributes
clobber: bool

If a file being appended is in GEIS or WAIVER FITS format, should any existing MEF files be overwritten?

dq_bits: int

Bitmask specifying what pixels in the mask should be removed (or kept) with the precise interpretation being left to the user. This flag is not used by this class but was designed to be passed to other functions that will use FileExtMaskInfo.

property DQimage

DQ image (file or ImageRef object depending on the fnamesOnly value).

Getter

Get the ImageRef DQ image object.

Setter

Set the DQ file.

Type

str, ImageRef

append_ext(ext)[source]

Append extensions to the list of “selected” extensions for the image file.

Note

This function appends the extensions. If it is desired to set the extensions, use replace_ext() instead.

Parameters
ext: int, tuple, None, list

Extension specification: None, an integer extension number, a tuple (extension number, extension version) where extension version can be '*' which will be replaced with the extension versions of all extensions having given extension name. If ext is None, it will be replaced with the default extension specification set during the initialization of the FileExtMaskInfo object.

append_mask(mask, ext, mask_stat=None)[source]

Append a mask image and its extension(s).

Note

Mask files and extensions are kept in ordered lists and their order is significant: the first mask file-extension pair is associated with the first extension of the science image file set with append_ext() and so on.

Parameters
mask: str, ImageRef, None

Mask image file. Can be a string file name, an ImageRef object (only if fnamesOnly is False), or None (to act as a place holder in the ordered list of extensions).

ext: int, tuple, None, list

Extension specification: None, an integer extension number, a tuple (extension number, extension version) where extension version can be '*' which will be replaced with the extension versions of all extensions having given extension name. If ext is None, it will be replaced with the default extension specification for mask images set during the initialization of the FileExtMaskInfo object.

mask_stat: `os.stat_result`, optional

An os.stat_result structure for the input mask file. If None, then append_mask() will compute stat for the input mask file.

Raises
RuntimeError

Raised if attempting to add masks when the science image was not yet set.

AssertionError

Raised if finalized is True.

TypeError

Raised if mask is an ImageRef object but fnamesOnly is True or if mask argument is of incorrect type.

ValueError

If mask is an ImageRef, it must not be closed.

clear_masks()[source]

Remove all attached mask files and extensions.

convert2ImageRef()[source]

Replace any existing file names with opened ImageRef objects and change the fnamesOnly property to False.

Note

The finalized property will not be modified.

Warning

The FileExtMaskInfo must not have been finalized (finalized is False) and must contain file names only (fnamesOnly is True).

Raises
AssertionError

Raised if finalized is True or fnamesOnly is False.

property count

Number of extensions associated with the image file.

property dqext

FITS extensions associated with the DQ file.

property fext

FITS extensions associated with the image file.

finalize(toImageRef=False)[source]

Finalize the object by trimming or extending mask image lists to match the number of science image extensions.

In principle, the number of mask files and their extensions need not be equal to the number of extensions specified for the science image. If the number of masks/extensions is smaller than the number of science extensions, the list of mask extensions will be appended with None (if fnamesOnly is True) or dummy ImageRef (if fnamesOnly is False) until the number of mask extensions is equal to the number of science image extensions. If the number of mask extensions is larger than the number of science image extensions, the list of mask extensions will be trimmed to match the number of science image extensions. The trimmed out mask files (if represented by ImageRef) will be “released”.

property finalized

Is the FileExtMaskInfo object finalized?

property fnamesOnly

Was the FileExtMaskInfo initialized to return file names or the ImageRef objects?

property image

Image file name or the associated ImageRef object (depending on the fnamesOnly value).

Getter

Get the ImageRef image object.

Setter

Set the image file.

Type

str, ImageRef, None

Note

Setting the image will re-initialize FileExtMaskInfo. All previous settings will be lost and previously attached files will be released/deleted.

info()[source]

Print information about the state of the object.

property mask_images

Mask image file names or ImageRef object depending on the fnamesOnly value.

property maskext

FITS extensions associated with the mask files.

release_all_images()[source]

Release all images if fnamesOnly is False and replace any existing ImageRef with their original file names.

Note

This will set the fnamesOnly property to True and the finalized property to False.

See also

convert2ImageRef
replace_fext(ext)[source]

Replace/set image file extension list.

See also

append_ext
stsci.skypac.parseat.parse_at_file(fname, default_ext=('SCI', '*'), default_mask_ext=0, clobber=False, fnamesOnly=False, doNotOpenDQ=False, match2Images=None, im_fmode='update', dq_fmode='readonly', msk_fmode='readonly', logfile=None, verbose=False)[source]

This function is similar to parse_at_line(), the main difference being that is can parse multiple (EOL terminated) lines of the format described in the documentation for parse_at_line().

Below we describe only differences bewtween this function and parse_at_line().

Parameters
fname: str

File name of the catalog file.

match2Images: list of str, list of ImageRef, None, optional

List of file names or ImageRef objects whose mask specifications are to be read from the catalog file. Mask specifications for other files in the catalog that do not match the files in the match2Images list will be ignored. If match2Images is None, then all files from the catalog will be read.

logfile: str, file, MultiFileLog, None, optional

Specifies the log file to which the messages should be printed. It can be a file name, a file object, a MultiFileLog object, or None.

Returns
list

Returns a list of filenames if fnamesOnly is True or a list of FileExtMaskInfo objects if fnamesOnly is False.

stsci.skypac.parseat.parse_at_line(fstring, default_ext=('SCI', '*'), default_mask_ext=0, \ clobber=False, fnamesOnly=False, doNotOpenDQ=False, \ im_fmode='update', dq_fmode='readonly', msk_fmode='readonly', verbose=False)[source]

Parse a line from a catalog file containing a science image file and extension specifications and optionally followed by a comma-separated list of mask files and extension specifications (or None).

File names will be stripped of leading and trailing white spaces. If it is essential to keep these spaces, file names may be enclosed in single or double quotation marks. Quotation marks may also be required when file names contain special characters used to separate file names and extension specifications: ,[]{}

Extension specifications must follow the file name and must be delimited by either square or curly brackets. Curly brackets allow specifying multiple comma-separated extensions: integer extension numbers and/or tuples (‘ext name’, ext version).

Some possible ways of specifying extensions:

[1] – extension number

[‘sci’,2] – extension name and version

{1,4,(‘sci’,3)} – multiple extension specifications, including tuples

{(‘sci’,*)} – wildcard extension versions (i.e., all extensions with extension name ‘sci’)

[‘sci’] – equivalent to [‘sci’, 1]

{‘sci’} – equivalent to {(‘sci’,*)}

For extensions in the science image for which no mask file is provided, the corresponding mask file names may be omitted (but a comma must still be used to show that no mask is provided in that position) or None can be used in place of the file name. NOTE: 'None' (in quotation marks) will be interpreted as a file named 'None'.

Some examples of possible user input:

image1.fits{1,2,('sci',3)},mask1.fits,,mask3.fits[0]

In this case:

image1.fits[1] is associated with mask1.fits[0];

image1.fits[2] does not have an associated mask;

image1.fits['sci',3] is associated with mask3.fits[0].

– Assume image2.fits has 4 ‘SCI’ extensions:

image2.fits{'sci'},None,,mask3.fits

In this case:

image2.fits['sci', 1] and image2.fits['sci', 2] and image2.fits['sci', 4] do not have an associated mask; image2.fits['sci', 3] is associated with mask3.fits[0]

Note

Wildcard extension version in extension specification can be expanded only when fnamesOnly is False.

Parameters
fstring: str

A comma-separated string describing the image file name and (optionally) followed by the extension specifier (e.g., [sci,1,2], or [sci]). The image file name may be followed (comma-separated) by optional mask file names (and their extension specifier).

File and extension names may NOT contain leading and/or trailing spaces, commas, and/or square or curly brakets.

default_ext: int, tuple, optional

Default extension to be used with image files that to not have an extension specified.

default_mask_ext: int, tuple, optional

Default extension to be used with image mask files that to not have an extension specified.

fnamesOnly: bool, optional

Return file names only, or open the files and return ImageRef objects?

doNotOpenDQ: bool, optional

Should the DQ files be oppened when simultaneously with the image files?

im_fmode: str, optional

File mode to be used to open image FITS file. See astropy.io.fits.open for more details.

dq_fmode: str, optional

File mode to be used to open DQ FITS file. This is valid only if the DQ model of the image file is ‘external’ (see documentation for ImageRef for more details). For ‘intrinsic’ DQ model the DQ files will use the same setting as for im_fmode.

msk_fmode: str, optional

File mode to be used to open mask files.

verbose: bool, optional

Specifies whether to print warning messages.

Returns
FileExtMaskInfo

A FileExtMaskInfo object.

Raises
ValueError
  • Input argument ‘fstring’ must be a Python string.

  • Input argument ‘fstring’ contains either unbalanced or nested square brackets.

  • Extension specification must be preceeded by a valid image file name.

stsci.skypac.parseat.parse_cs_line(csline, default_ext=('SCI', '*'), clobber=False, fnamesOnly=False, doNotOpenDQ=False, im_fmode='update', dq_fmode='readonly', msk_fmode='readonly', logfile=None, verbose=False)[source]

This function is similar to parse_at_line(), the main difference being the content of the input string: a list of comma-separated science image file names. No masks can be specified and file names must be valid (i.e., None is not allowed). Extension specifications are allowed and must folow the same sintax as described for parse_at_line().

Below we describe only differences bewtween this function and parse_at_line().

Parameters
csline: str

User input string that needs to be parsed containing one of the following:

  • a comma-separated list of valid science image file names (see note below) and (optionally) extension specifications, e.g.: 'j1234567q_flt.fits[1], j1234568q_flt.fits[sci,2]';

  • an @-file name, e.g., '@files_to_match.txt'.

Note

Valid science image file names are:

  • file names of existing FITS, GEIS, or WAIVER FITS files;

  • partial file names containing wildcard characters, e.g., '*_flt.fits';

  • Association (ASN) tables (must have _asn, or _asc suffix), e.g., 'j12345670_asn.fits'.

Warning

@-file names MAY NOT be followed by an extension specification.

Warning

If an association table or a partial file name with wildcard characters is followed by an extension specification, it will be considered that this extension specification applies to each file name in the association table or each file name obtained after wildcard expansion of the partial file name.

logfile: str, file, MultiFileLog, None, optional

Specifies the log file to which the messages should be printed. It can be a file name, a file object, a MultiFileLog object, or None.

Returns
list

Returns a list of filenames if fnamesOnly is True or a list of FileExtMaskInfo objects if fnamesOnly is False.