SkyView in a Jar

 

Using SkyView as a Standalone Java Application

 

 

Thomas McGlynn and Laura McDonald

High Energy Astrophysics Science Archive Research Center

Goddard Space Flight Center

Greenbelt, MD 20771

 

Version 1.01

December 22, 2005

 

 


 

Table of Contents

 

Introduction. 3

Setup. 3

Command Summary. 4

Default arguments. 9

Example requests. 10

SkyView-in-a-Jar Operations. 11

Getting defaults and user arguments. 12

Locating surveys. 12

Looping over the surveys. 13

Getting candidate survey images. 13

Building images. 13

Finding the best survey image for each output pixel. 14

Preprocessing. 15

Validating the image. 15

Resampling setup. 15

Sampling in energy. 16

Sampling in space. 16

Post-Processing. 16

De-edging. 16

Data conversion and writing the FITS file. 17

Cache cleanup. 17

Survey Definition File. 17

Survey. 17

ShortName. 17

Name. 18

Description. 18

Settings. 18

MetaTable. 18

OnlineText 19

FITS. 20

The Images Element 20

Explicit Image Lists. 20

Image Generators. 21

Examples: 22

 


 

Introduction

The SkyView-in-a-Jar application provides users with local access to the SkyView.  Users can generate images from major surveys in any requested geometry, resample and mosaic their own data and create their own surveys.  SkyView-in-a-Jar comes completely ready to use as a single file.  In this default configuration many major surveys are available.  It is  easy to add new surveys using local files or the Virtual Observatory Simple Image Access protocol.

The code itself is also extremely extensible.  Users can add new samplers, projections, and coordinate systems.  Even the classes which define the basic algorithms for finding surveys, selecting images within the survey, and mosaicking the images can be overridden by the sophisticated user.  Users can insert their own pre- and post- processing functions to modify the input data or the resulting image.

This document provides a comprehensive User’s Guide to SkyView-in-a-Jar and an introduction to many of the constituent Java classes.

Setup.

 

The SkyView-in-a-Jar is self-contained within a single executable JAR file.   JAR files are a modified version of ZIP files used to store Java code.  However JAR’s can include data and program control files as well as the Java code.  The SkyView-in-a-Jar contains all of the control files needed to access almost all of the surveys available from SkyView’s standard Web interface.   Simply download the JAR file from

http://skyview.gsfc.nasa.gov/jar/skyview.jar

and then run the file using the command

java –jar skyview.jar key1=arg1 key2=arg2 …

The same JAR file is used for all operating systems.  This program requires at least a Java 1.5 JRE to run.  Java distributions for most major operating systems are available at

http://www.java.sun.com.

 

E.g., to create a DSS and FIRST  images of 3c273 simply enter

java –jar skyview.jar survey=DSS,FIRST position=3c273

The program creates output1.fits and output2.fits. The first contains an image resampled from the DSS and the second contains data from FIRST. 

There are many options controlling how to resample the image.  These are discussed briefly in the command summary section and then in more detail in the context in the context of how SkyView works.

 

The SkyView Java classes in the JAR file can be used in other applications as well.  Simply include the JAR file in your CLASSPATH.   The only difference between a regular and executable JAR file is that the later contains a manifest that indicates how to start up when the –jar option is used in the java command. The SkyView Java application could also be started up with

java –cp pathto/skyview.jar skyview.executive.Imager key1=arg1 …

The Imager class can be instantiated within other Java applications as well as being invoked from the command line. 

Later versions of the JAR files may start a different class, but this method of using the Imager class should still work, and we anticipate that the command line arguments will be upwardly compatible.

Command Summary

This section describes the command line arguments to the SkyView-in-a-Jar application.  Each command line argument consists of a key=value pair.  The command keys are case-insensitive but the case of the value can matter.   Generally, to associate more than one value with a key, separate the values by commas. 

 

Command Line Arguments

Key

Value

Survey

The survey or surveys indicate which surveys the user wishes to generate an image for.  If more than one survey is desired, the values should be separated by commas.  The list of all surveys known (by default) to the program can be generated by invoking the application with no arguments.  This will give a quick description of the arguments and a list of known surveys.  Survey names are case insensitive.

 

The survey user is special.  It indicates that one or more files indicated in the userfile argument are to be used as the survey.  Using the survey=user and userfile=file1,file2,… arguments, users can resample and mosaic local data files.

 

Survey=DSS,2MASS-K

 

Arguments that define the output image geometry

 

Position

The position argument gives the center position for which images are requested.  It may be either a coordinate pair, or a target name.  The position argument or the Lon/Lat pair is required.

 

Position=3c273

Position=’10 20 30, -15 16 22’   

Note that the quotes are not part of the actual value: they are needed to ensure that the value is treated as a single string.  Your operating system may have other methods for enclosing embedded spaces in an argument.

Lon

This argument gives the Longitude or right ascension of the center of the image.  Lon/Lat arguments supersede the position if both are specified.

Lat

This argument gives the Latitude or declination of the center of the image.

Coordinates

This argument gives the coordinate system used for the output image.  The first letter in the value gives the type of coordinates.  If a number is appended then this is the epoch of the equinox of the coordinate system. 

 

Recognized values include

    J  -- Julian equatorial coordinates

   B  -- Besselian equatorial coordinates

   ICRS – International Coordinate Reference System coordinates.

   Gal – Galactic coordinates

   H – Helioecliptic coordinates

   E  -- Ecliptic coordinates.

 

The single letter values may be combined with an epoch, e.g.,

J2000, B1950, E2020.123

 

The default coordinate system is J2000.

 

Coordinates=J2000

Coordinates=Galactic

Projection

The three letter designation used to specify the projection of the output image.  This uses the suffix in the FITS CTYPE keywords.  Supported projections include Car, Tan, Ait, Zea, and Sin.

 

The default is Tan.

 

Projection=Tan

Equinox

If the coordinate system only specified a coordinate type, the equinox can be specified separately.

 

Equinox=1980

Scale

Scale gives the size of pixels in the output image and corresponds to the CDELT keywords.  Scale may be specified as a scalar, in which case the pixels are square, or two values may be given if rectangular pixels are desired. The scale is always positive.  Output images follow the astronomical convention where the longitude decreases along a horizontal line.  The scale is specified in degrees per pixel. 

 

The default scale depends upon the survey.  If you do not specify an explicit scale, and you choose more than one survey, then the scale may differ among surveys.

 

If there is no survey defined default, then the default scale is 1” per pixel.   Here the Java application differs from the SkyView web interfaces where the size of the entire output image is given rather than the size of an individual pixel.

 

Scale=0.0025

Scale=0.0002777777

Scale=0.0002777777,0.0005

 

Resolver

When a target name is specified rather than specific coordinates, the name must be ‘resolved’ into coordinates.  There are two widely used name resolvers available: NED and SIMBAD.   The application invokes these through a HEASARC web service.  This argument can be used to specify which resolver or resolvers are to be used.  If both resolvers are specified, then the second is checked only if the object is not found at the first.  Values are: SIMBAD, NED, NED-SIMBAD and SIMBAD-NED.  The default is SIMBAD-NED.

 

Resolver=NED

Pixels

This gives the number of pixels in the output image.  If specified as a single value, the number of pixels will be the same in both dimensions.

 

The default is 300.

 

Pixels=500

Pixels=1280,640

Rotation

The argument indicates that the output image is rotated by the specified angle (in degrees).

 

By default there is no rotation of the image.

 

Rotation=45.0

 

Arguments that control resampling of the input data

 

Ebins

This argument is used only when sampling three-dimensional images which have a set of ‘energy’ values at each pixel.  It must have three comma separated numeric values.  The first gives the offset of the first output energy bin, the second gives the size of the output bins relative to the original bins, and the third gives the number of output bins.  E.g., suppose we have 10 channels in the input data.  We wish to eliminate the top and bottom channels and rebin the remaining 8 channels into 4 output  bins in the output image.  Then we have

 

Ebins=1,2,4

 

Note that the offset is 0-based.  The bin size does not need to be an integer, the output bins will be integrated over the input bins assuming flux to be uniformly distributed over the energy bin.

 

In SkyView  3-D surveys normally specify a default for the Ebins value which gives a 2-D output image.

Sampler

This argument specifies the sampling method to be used. The value consists of an algorithm designator which may in some cases be suffixed with an integer giving the order of the sampler.  Currently supported samplers include:  Nearest Neighbor (NN), Bi-linear interpolation (LI), Splines from 2-5 order (Spline or Spline3, Spline2, Spline4, Spline5), Lanczos interpolation with a specified kernel size

(Lanczos, or LanczosN where N >= 2, the default is 3), and an exact-area resampling using Sutherland-Hodges clipping (Clip).  The default is nearest neighbor.

 

Sampler=Lanczos5

Sampler=Spline

Sampler=Clip

 

Arguments that control the output file and format

 

Output

The name (or stem) of the output image or images.  The user may specify the full path of the output image x.fits or just a stem x.  If more than one image is being produced, then ordinal numbers will be added so that the output files will be x1.fits, x2.fits.  If compression is requested the .gz extension should not be specified, it will be added automatically.

 

The default is output.

 

Output=~/myoutputs/test.fits

Output=c:\datafiles\3c273

Compression

This flag indicates that the output files are to be gzip compressed.  The default is not to compress the data.   The content of the value is ignored and the value can be omitted.

 

Compression=T

Compression   

Float

This flag indicates the output files should be 4 byte reals.  The default is 8 byte reals.  The content of the value is ignored and can be omitted.

 

Float=T

Float

 

Arguments that control the definition of surveys

 

Userfile

This argument is used to specify one or more files that are treated as the constituent image files of the ‘user’ survey.  Users can resample and mosaic local files easily using this argument and the user survey.

 

There is no default.

 

Userfile=myfile1.fits,myfile2.fits

SurveyXML

This argument allows the user to specify one or more survey XML descriptors that users can query from (see the appendix A for details of the format).  This allows more sophisticated use of locally defined data, including default values, data to be incorporated into the FITS files and metadata descriptions.  User this argument to specify individual files and the XmlRoot argument if you wish to specify an entire directory.

XmlRoot

This argument  gives a directory where survey XML files are to be found.  Each ‘.xml’ file in the directory will be treated as a survey descriptor file.  Currently only one directory is permitted.

 

XMLRoot=/survey/docs/xml

SurveyManifest

The Survey manifest is a file (or Java resource) that lists survey XML files.  By default the Survey Manifest is given in the system resource surveys/survey.manifest that is included in the JAR.

 

SurveyManifest=mysurvey/docs/mysurvey.manifest

 

Arguments that control the caching of survey files

 

Cache

When a survey’s data is retrieved from a remote location, it is cached in the local file store.  The cache argument specifies one or more directories in which the application should look for the remote data before attempting to retrieve it.  If a file matching the desired name is found, it is used directly.  The first directory in the cache is used to save any files that need to be retrieved.  If no cache is specified the current directory is used as the cache.

 

Each survey can be cached into a different directory by specifying a unique cache value in the Settings element of the Survey descriptor file or a common cache can be used for multiple surveys.  To have the survey definition for the cache location have priority over a user specified cache location, set the cache location in the

Images area of the survey definition rather than in the Settings area.

 

Cache=/surveycache/cache1,/surveycache/cache2

PurgeCache

This argument indicates that cache files are to be purged after the processing of that survey is completed.  Only survey files retrieved in this request will be deleted.  Other files in the cache are left unchanged even files used in processing.  The value of the argument is not used.

 

PurgeCache=T

PurgeCache

 

Arguments defining the classes used in processing

 

SurveyFinder

The class used to find surveys.   This class must implement the skyview.survey.SurveyFinder interface.

 

By default this is the class skyview.survey.XMLSurveyFinder but it may be overridden by knowledgeable users.

ImageFinder

The class used to find the image to sample for a given image in an input pixel.    This class must inherit from the skyview.geometry.ImageFinder abstract class.

 

The default is skyview.geometry.imagefinder.RecursiveImageFinder

PreProcessor

One or more classes implementing the skyview.geometry.Processor interface that are to be used in processing (presumably of the input images) before mosaicking takes place.  If  a user needs to communicate parameters to a pre (or post) processor, additional Settings can be defined which are used to transmit the information.

 

No preprocessing is performed by default and no classes appropriate for preprocessing are currently provided.  An input image smoother might be an appropriate preprocessor class.

Mosaicker

A class implementing the Processor interface that fills in the pixel values in the output image.  The default mosaicker is the class skyview.geometry.Mosaicker.

PostProcessor

One or more classes defining the Processor interface which process the image after mosaicking. 

 

Currently the skyview.geometry.Deedger and other deedging classes can be used in post-processing.  Users can write their own classes.

Available classes are:

   skyview.geometry.Deedger which de-edges using the default algorithm (current using the BoundaryMedian)

   skyview.geometry.deedger.BoundaryMedian  sets the median change across boundaries is 0

   skyview.geometry.deedger.BoundaryAverage sets the average change across boundaries to 0

   skyview.geometry.deedger.ImageMedian sets the medians pixels derived from each image to a  common value.

 

When survey images are mosaicked together there may be artifacts at the edges of the images where the different backgrounds of the images merge.  The deedger attempts to minimize these artificats.  Deedging is turned on by default for some surveys (e.g., the DSS).  It can be turned off for those surveys with the value ‘null’ for post-processing.

 

PostProcessor=skyview.geometry.Deedger

PostProcessor=null

 

 

Default arguments.

 

When the Java application begins, default settings are loaded.  These default settings are normally read in through the system resource skyview.settings.  If users wish to provide their own default settings, they can override the settings provided in the JAR file by creating a skyview.settings file in the current directory, or by pointing to a file in any location (and of any name) using the environment variable SKYVIEW_SETTINGS.

 

The settings file is a simple ASCII file.  Each line can be either blank, a comment line

(first character of the line is ‘#’) or specify a setting.  Here is what an example file looks like:

 

# An example skyview.settings file.

 

Pixels=500

Projection=Sin

Coordinates=J2000

Sampler=Lanczos

Equinox=2000

 

Resolver=NED-SIMBAD

XMLroot=$SKYVIEW_XMLROOT

 

Mosaicker=skyview.geometry.Mosaicker

SurveyFinder=skyview.survey.XMLSurveyFinder

ImageFinder=skyview.geometry.imagefinder.RecursiveImageFinder

 

Note the definition for the XMLRoot (the directory where survey XML files may be found). When a value begins with a $ the value of the corresponding environment variable will be used but the line will be ignored if the environment variable is not defined.  This allows users to override the default settings using environment variables using two lines in the settings file, e.g.,

Projection=Sin

Projection=$PROJ

will use a Sine projection unless the user overrides it by specifying the PROJ environment variable.

 

Example requests.

 

1.

 

java –jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-i

This returns three 300x300 pixel images named output1.fits, output2.fits, and output3.fits with the survey default  pixel size..

 

2.

java –jar skyview.jar position=3c273 survey=2mass-j \

scale=.0005 pixels=1200

This returns a single file 1200x1200 pixel file named output.fits with pixels of 1.8”.

 

3.

java –jar skyview.jar position=3c273 survey=2mass-j \

cache=/local/data/2masscache

This returns a 300x300 file but looks in the cache directory /local/data/2masscache to see if needed survey files have been downloaded there first.  In the previous examples all survey files were placed in the ‘current’ directory.  Images are generated much faster if needed survey data is already present in the cache.

 

4.

java –jar skyview.jar position=3c273 survey=2mass-j \

cache=/local/data/2masscache purgecache

This returns the same image as the previous request, but deletes any survey files downloaded from the 2MASS site for this request.  However any files that might have been downloaded in previous requests are left untouched.

 

5.

java –jar skyview.jar position=3c273 survey=user \

userfile=ff1.fits,ff2.fits,ff3.fits   scale=.003 \

sampler=Lanczos4 postprocessor=skyview.geometry.Deedger

This request looks at the three local files ff[1-3].fits and tries to create a mosaic from them using a fourth order Lanczos resampler. Boundaries between the input images are to be masked using the standard deedger class. 

 

6.

java –jar skyview.jar position=’horsehead nebula’ \

postprocessor=null output=horsehead \

float coordinates=B1950

 

This asks for a DSS image of the Horsehead nebula but turns de-edging off.  The output image should use 4-byte reals and be in B1950 coordinates.

 

The Cache

 

When a user requests an image from a survey whose source files are on another computer, the application first downloads the files onto the local computer and then resamples the data.  By default the downloaded files are copied into the current directory and are left there when the application is complete and serves as a cache for subsequent requests.   Often users make several requests near a given location, perhaps changing the pixel size or resampling method used.  When these subsequent requests are made, the application notes that the needed files are already downloaded and immediately begins resampling.

 

If a user is sure that a survey file is not going to be needed again, or if space is at a premium, the user can request that the survey files be deleted by specifying the PurgeCache setting.  This will delete all files downloaded for a given survey in the current request.  However if multiple files are needed for a given survey, then all files for the survey will be downloaded and the output image produced before the downloaded files are deleted.

 

While the advantages of keeping the survey files available can be substantial, having them in the current directory can be a nuisance if more than a few requests are made.  In this case the user will likely wish to specify a cache directory to save these files, so they do not obscure other files in the current directory.  A cache may be specified in the settings file or in each command using the Cache setting to specify one or more cache directories.

 

Users who create their own survey data sets (or who modify the XML survey files distributed with the application) can create different cache directories for each survey.  By setting the cache directory in the <Settings> area of the survey description the user specifies a default cache directory which can be overridden on the command line.  If the user specifies a cache in the <Images> area, then this setting cannot be overridden by the user.  Note that more than one cache directory can be specified.  All of the cache directories are searched when looking to see if a file is already available, however any files that are downloaded will be placed in the first directory.

 

Caching the Digitized Sky Survey.

 

The digitized sky survey data is handled somewhat differently than most surveys.  Rather than trying to download entire plates of data, which would take forever even if it were permitted by the STScI servers, SkyView requests small tiles from these images.  It then builds the image the user requests by mosaicking the tiles together.  The tiles are cached like any other survey.  The headers for each tile indicate the original image from which it was derived.

Dependencies on Remote Systems

 

While all data processing occurs in the local system, the SkyView-in-a-Jar application may fail if remote resources are not available.  The underlying data for a few surveys are included inside the JAR file (e.g., 408 MHz and EGRET).  These are generally low resolution surveys.  The application will fail if it is unable to retrieve data for other surveys.  For many surveys that data is fetched from the SkyView server at skyview.gsfc.nasa.gov. However the DSS, DSS2 and FIRST surveys link to archives that the Space Telescope Science Institute, and the 2MASS and NEAT surveys use archives at JPL.  A number of other surveys, e.g., the SDSS, now use the VO Simple Image Access protocol, and link to URLs described therein.  The <Images> area of the survey description will generally indicate the dependencies. 

 

The only other remote dependency for the application is in the resolution of target names.  If the user specifies a target rather than a pair of coordinates, the Name resolver/coordinate converter service at the HEASARC is used to resolve this name into coordinates.  The HEASARC service in turn calls the SIMBAD and NED name resolvers.  The HEASARC service may also be used if the user enters the coordinates using characters other than numerics, signs, colons, commas and decimal points (e.g,

10h30m10.3s). 

 

 

Compatibility with the SkyView Batch Interface

 

For the convenience of users of the old SkyView batch interface, the SkyView-in-a-Jar application supports most of the arguments of that interface insofar as they related to the generation of FITS files. The following are supported as input settings:

 

VCOORD

This is converted to the Position setting

SCOORD

This is converted to an appropriate value for the Coordinates setting

MAPROJ

This is converted to an appropriate value for the Projection setting

PIXELX,PIXELY

If both are specified they are joined and converted to a Pixels setting

EQUINX

This is converted to the Equinox setting

SFACTR

This is converted to the Size setting

file

This is converted to the Output setting

 

The current Java application does not support generation of non-FITS images, so that the brightness scaling and color table arguments are not meaningful.

 

Generally results from the new interface should be very similar to the old SkyView interface.  There is one major exception.  The IDL code which the old interface used ran almost entirely in single precision arithmetic.  The Java classes use double precision arithmetic exclusively with an option for converting the final result to single precision. Users must specify the Float setting to get single precision results.  The difference in precision and other small differences in the algorithms mean that there can be small differences in the values returned.  E.g., when using a near-neighbor resampling, positions near the edge of one pixel might move onto an adjacent pixel.

 

There format of the FITS headers differs in considerably between the two versions.  Substantially more information about the processing is included in the new version including the files that were used in generating the output mosaic.

SkyView-in-a-Jar Operations.

 

In this section we describe how the SkyView Java application works.  This includes short descriptions of some of the algorithms used.   

 

The basic goal of SkyView is relatively simple.  Given a set of one or more existing images comprising a survey, SkyView generates a new image in a user-specified geometry.  The output user image may overlap one or more of the survey images, and may overlap different images in different locations.  This section discusses how the application accomplishes this task. 

Getting defaults and user arguments.

When the application starts, a set of default parameters is established.  The java.executive.Settings class is used to convey the current parameter settings.  The default parameters are read in from a configuration file or system resource.  User arguments are then processed and override any defaults.  System parameters are stored in a HashMap but both the put and get methods convert the key to lower case, so that the keys are not case sensitive.

Locating surveys.

Once the user arguments have been parsed, the application looks for surveys.  Using the default SurveyFinder there are four possible sources of surveys:

·        The application searches for a survey manifest resource.  This manifest gives the names of other resources or XML files where each file defines one survey.  In this initial pass the XML is only parsed far enough to find the names of the surveys.  Surveys can have several names.  When the system tries to find the surveys the user is looking for the match will be done in a case-insensitive fashion.  The source manifest is where the default surveys are normally found when the executable JAR file is used.

·        The second location where surveys may be found is determined by the setting for the SourceXML setting.  This may be set on the command line, or if the user sets the environment variable SKYVIEW_SOURCEXML then the value of that variable is used. When the SourceXML is set, that directory is searched and each file of type .xml is treated as a survey definition file.  Users can use this method to add a set of surveys to their SkyView implementation.  It is also used when the SkyView application is run outside of the JAR file.

·        Users can specify one or more survey files directly using the SurveyXML setting.  Each field in the SurveyXML is assumed to be a Survey definition file and is parsed.

·        Finally, the user can specify a ‘User’ survey by setting the SurveyFile setting.  Each field of the SurveyFile is treated as a survey image file.  Users can generate an image from this survey by using the survey name ‘User’.

 

Looping over the surveys.

Once the set of available surveys has been determined, the application loops over the surveys the user specified in the Survey setting.  If the name the user has specified is not found, then an error is signaled and processing continues with the next survey.

When a survey is found, the corresponding survey definition file is parsed.  Part of the survey definition file is a set of survey default settings, e.g., the default scale for the image, or whether the image will be de-edged.  These settings do not override settings already made, so that survey settings do not override the overall defaults or user defined settings.

Getting candidate survey images

The first step in processing a survey is determining a list of candidate survey images that may be used in generating the output image.  If a survey uses the VO Simple Image Access Protocol (SIAP, see http://us-vo.org) , then the candidates are all images returned in the initial SIAP request.  If the survey explicitly lists the images, then the application passes through these images and uses the ImageSize characteristic to find images that might overlap part of the user’s image.  If this is a request for the User survey, then all of the images the user specified are candidates.

Building images.

At this point, a survey image is only a candidate for processing.  Often there will be many more candidates than will actually be used.  If the images are present locally this is not an issue, but it may seriously delay processing if we download a number of images that are not actually needed.  To ensure that only needed data are retrieved from remote sites, remote images are normally represented initially as proxy images.  Proxy images have the same geometry (or a very good approximation) as the real image, but the image data is not filled in.  If and when the application decides to sample the data in the image, it downloads the remote data and converts the proxy into a real image.

The SIA protocol allows for fields that define the geometry of the images returned which makes it easy to build proxies without actually retrieving data.  For other surveys we find that the geometry of the images is largely constant with only the center position changing.  This allows us to build ‘spells’ to enable the creation of proxy images and their transformation to real images. 

The Survey XML descriptor file controls this process.  It can specify either an explicit list of image files or spells that define the content of the survey, or an ImageGenerator class that will create this list dynamically.  E.g., if the IRAS100 survey uses local data, then its survey XML file lists 430 files explicitly.  In the SkyView-in-a-Jar distribution, the IRAS data is retrieved over the net, so the survey XML file includes sufficient information to generate proxy images for the survey.  These proxies are used to find out which images are to be resampled and which pixels in the output image are to be resampled from which input image.  Only when we actually begin resampling does the program actually download the data.  

The SDSS-I survey uses an ImageGenerator class that in turn uses the SIA protocol to find candidate images around the user’s requested location.  The SIA call returns sufficient information to generate proxy images so we only actually retrieve the SDSS images we are going to use.

An ImageFactory class is also defined in the survey XML file. It shows how to convert the file name/spell String into an actual image object.  The FitsImageFactory just reads in a FITS file, but the CachingImageFactory first looks to see if files are present in the cache, and if not creates a proxy image as described above.

Users can create their own ImageGenerator’s and ImageFactory’s to meet special requirements for new surveys they might want to include.

Finding the best survey image for each output pixel.

Once the set of candidate images is determined, the application determines the image to sample for each pixel in the output image.  This uses an object of the ImageFinder class.  The default, BorderImageFinder is described here.  The position of the four corners of the output image is determined for each of the candidate input images.  The best candidate image for each of the four pixels is then determined.  The ‘best’ input image is where the output pixel location is located in the image but furthest from the edge of the input image.

E.g., consider the output pixel at 0,0 (the lower left corner) of the output image and assume there are three candidate images each of which is a 500x500 image.  The position corresponding to the center of output pixel 0,0 is transformed into the pixel coordinates of the three candidate images.  If the pixel coordinates for the three candidates are (-80, 300), (40,40) and (250,35), then the second image is the best.  The output pixel location is not within the first image, and is only 35 pixels from the edge of the third, but 40 pixels from the edge of the second. 

If all four corners match the same input image, then that image is assumed to be the best image for all of the interior pixels.  If different images are best (or perhaps there is no corresponding image for one of the pixels), then the rectangle is broken up into four sub-rectangles and the process is repeated recursively until we have either checked each pixel individually, or we get common input images for each corner of some sub-rectangle.

In the case where the output image is comparable to or larger than the input images, the initial checking will be done on rectangles no larger than half the size of the input images. When the input image selection splits into sub-rectangles, any candidate image that was not found on any of the four corners is not passed down to the next level of the recursion. 

 

Consider generating an all sky map from the IRAS100 survey which has 430 12x12 degree square images.  The application first breaks the output image into 6x6 degree squares and checks each of these rectangles against all 430 images.  In most cases it will not find the same best image for all four corners of the 6x6 degree squares, so it looks at  3x3 degree squares.  However, in this second step it only looks at candidate images on which at least one of the four pixels at the previous step was on the image (though not necessarily the best image).  So in this second stage we are typically looking at only three or four candidates making the search much more efficient.

The end-point of this process is an integer array with the same dimensions as the user output image but where the value at each pixel is the best input image.  Special flag values are used to indicate that this output pixel is not matched by any survey image.  An output pixel may also be outside of the standard projection coverage, e.g., outside the oval of an all-sky Aitoff projection, or the circle defined in the Orthographic (SIN) projection.

In many cases the user may have selected images from a set of surveys which have identical or near identical geometry (e.g., IRAS100 and IRAS25, or 2MASS-J and 2MASS-K).  These are called ‘geometry twins’ and are identified in the survey descriptor files.  If two geometry twins are processed in succession, then the calculation of the best image from the first twin processed is re-used for the later twin (or triplets, or …).

Preprocessing

At this point the program allows for image pre-processing.  The user can define a class (or classes) that can modify the data that are used for sampling.  A preprocessor class has access to the input images, the index associating images with output pixels and the output image.  Only the geometry of the output image has been specified, the data so far are nil.

No preprocessing classes are currently provided.  One useful class might be a smoother that replaces the input images, with smoothed versions.

Validating the image.

Up to this point in the processing of the image, only the coordinate system of the input image has been needed.  To avoid downloading images that may not actually be used, remote images use proxy images that do not have data values associated.   Even if the image is available locally, only the FITS header is read rather than the data values if this can be done efficiently.  As we begin to process each survey image, the image is validated, i.e., we download the image if it is a remote URL and then populate the actual input data array.

While the proxy image should have a geometry very similar to the real image, it may not be quite identical.  If a request for a region is repeated and the survey images were cached on the first request, then the actual image geometry will be used at all stages in the second request which may result is slightly different boundaries between the regions sampled from the various input images.

 

Resampling setup.

Once the ‘best’ image for each output pixel has been selected, the program begins to process each input image.  For some samplers there is some processing of the input image done before any resampling occurs.  E.g., the spline resampler needs to calculate the spline coefficients to be used.  In this case if the input image is much larger than the output image, then spline coefficients may only be computed on a sub-region of the input.  The minimum and maximum x and y values for the output image that are being sampled in this input image are computed.  These bounds are projected into the input coordinate system and the spline is computed only for this sub-region.  Currently this behavior is enabled only for tiled surveys.

Sampling in energy.

If the input survey has depth in the energy dimension, the data is first sampled in energy according to the Ebin’s specification before spatial sampling.  A default specification is normally included in the survey default to produce a 2-d output image.  Thus unless the user specifies that they want a 3-d output image, they get a 2-d image.

If there is no energy dimension to the data, this step is skipped.

Sampling in space.

The program then loops over the output pixels associated with the current input image and computes the value using the selected resampler.  For most resamplers the center of the output pixel is projected into the plane of the input image.  For the clipping sampler, the corners of the pixel are projected.  The output pixel is then populated with the resampled value according to the requested algorithm.  If the output image has depth in the energy dimension, the sampling is performed at each output energy channel.

Currently the spline sampler cannot process 3-d images.

Post-Processing

After the data have been resampled, post-processing may be performed.  The interface for post-processing is exactly the same as for pre-processing, however at this point the output image has been populated and we would anticipate that most post-processing will perform some filtering of the output image while preprocessing is expected to primarily affect the input image or images.  One example of a post-processor is de-edging.

De-edging

Since survey data may have different background values in the different survey images, the edges of the image may show up as an unsightly artifact.  If the user wishes to minimize the edges, then a de-edger may be run on the output image.  .

The algorithm used by the default deedger attempts to minimize discontinuities along borders.  The output image is divided into regions defined by the input image from which the pixels were sampled.  These regions are placed into adjusted and unadjusted sets.  The region with maximum number of pixels is placed in the adjusted set and all other images are initially unadjusted.  The program then looks for the longest border between an adjusted and unadjusted region.  Borders are defined by looking at each pixel and seeing if any of the four neighboring pixels is from another region (diagonals are not considered).  The median of the shifts as we move across the border from the adjusted to the unadjusted regions is computed. An offset is applied to the unadjusted region to set the median shift across the border to 0.  The unadjusted region is moved into the adjusted set, and the process iterates until all regions have been adjusted.  Other classes use somewhat different algorithms to compute the offsets for images: use the average rather than the median shift at the border, adjusting the medians of all pixels that come from an image, not just the boundary pixels.

The FITS headers include information on the adjustments made when de-edging is applied.

Data conversion and writing the FITS file.

All of the internal computations are done with the data as a one-dimensional double array of values.  Just before writing the data out, the application checks to see if the user wished to have only 4-byte reals (float’s in C or Java-speak) rather than the default 8-byte data (double’s).  If so the data is converted.  The program then ‘curls’ the data into the appropriate dimensionality and uses the nom.tam.fits library to write the FITS image.   The survey descriptor defines survey-dependent information that is included in the FITS header.

Cache cleanup.

If a user requested data that was not present in local files, this data is first downloaded into the first cache directory (the current directory by default).  By default downloaded survey files are left for possible use in subsequent requests, but if PurgeCache has been set, then these files are deleted as the processing for each survey is completed.  However only the files actually downloaded during this request will be purged.  Files already present in the cache will be left untouched.

 

 


Survey Definition File

Each survey (other than the ‘User’ survey) used in SkyView is defined by a survey definition file.  This is an XML file that pulls together all information needed to use the survey.  This section describes the structure of the file.  [Note: If a user defines their own SurveyFinder object, the behavior can be entirely different.]

There is no formal schema for the survey file format but we suggest the following order for the top elements of the file.

Survey

ShortName

Name

Description

Settings

MetaTable

OnlineText

FITS

Images

 

ShortName

The short name element gives one or more short names for the survey.  Short names are used as the values of the Survey keyword in the SkyView-in-a-Jar application and therefore it is recommended that they not contain embedded blanks.  If multiple short names are to be specified they are separated by commas.  All survey description files are read at program startup until the short name is found, so putting this first can speed up the program a little.

Examples:

<ShortName>DSS2R</ShortName>

<ShortName>EGREThard,EGRET100-10000</ShortName>

 

 

Name

The name element gives a longer name for the survey generally spelling out acronyms and such that may have been used in the short name.  

Example:

<Name>Two Micron All-Sky Survey: K band </Name>

Description

This gives a text description of the survey.  This may include HTML and if so should be coded as a CDATA element.

Example

<Description>

<![CDATA[

This LISA Galactic survey is a low-resolution image

of the gravitational background in    

the Galactic plane as described in <a href=someurl>Lisa

Galactic Plane Handbook    

</a>

  ]]>

</Description>

Settings

The settings element is used to specify survey specific settings.  Within the settings element there may be 0 or more sub-elements of the form

<key>value</key>

When processing of a specific survey begins, the Settings element is parsed and the Settings hash is updated with any key/value pairs found.  The case of the key is not distinguished but case may be important in the value.  Note that XML parsers require that the opening and closing case of the keys be consistent within the file.

Common survey specific settings include the default image scale (Scale), other surveys that have the same geometry (GeometryTwin), the default energy binning for 3-d surveys (EBins), and post-processing options, notably the de-edger (PostProcessor).

Survey settings will not override existing user specified settings. 

 

<Settings>

<Scale> 0.003 </Scale>

<GeometryTwin>SurveyA,SurveyB,SurveyC</GeometryTwin>

<PostProcessor> skyview.geometry.Deedger </PostProcessor>

</Settings>

 

MetaTable

The metatable element includes a set of metadata elements that give an overall description of the survey.  SkyView attempts to give some common information about all surveys, energy range, resolution, … and these are grouped in the metatable.

The metatable information is normally included in any FITS file produced and elements may be displayed in interface SkyView interfaces.

Example:

 

<MetaTable>

  <Provenance>

    NASA IPAC/Jet Propulsion Laboratory

  </Provenance>

    <Copyright>   Public Domain  </Copyright>

    <Regime>      Infrared       </Regime>

    <NSurvey>     4              </NSurvey>

    <Frequency>   3-30 THz       </Frequency>

    <Coverage>    All-sky        </Coverage>

    <Scale>       0.025 deg/pix  </Scale>

    <Units>       MJy/sr         </Units>

    <Resolution>  2’             </Resolution>

    <Coordinates> Equatorial     </Coordinates>

 <Projection>  Gnomonic(TAN)  </Projection>

    <Equinox>     1950           </Equinox>

    <Epoch>       1983           </Epoch>

    <Reference>

      <![CDATA[

       Wheelock, et al., 1991, IRAS Sky Survey Atlas Explanatory

       Supplement.

       ]]>

</Reference>

</MetaTable>

 

OnlineText

The online text element is used to include text that is to be shown on Web pages associated with a generated image.  The online text may include HTML tags and if so should be enclosed a CDATA element.  Variables prefixed by $ will be substituted for in the text by the appropriate values.  These include:

$ra      The right ascension of the center of the image in decimal degrees

$raH   The hours of right ascension

$raM    The minutes of right ascension

$raS     The seconds of right ascension (including fraction)

$dec     The declination of the center of the image in decimal degrees

$decD The degrees (and sign) of declination

$decM The minutes of declination

$decS  The seconds of declination (including fraction)

$sizeD  The requested size of the image in degrees.

$sizeM The requested size of the image in minutes.

$sizeS  The requested size of the image in seconds.

 

Example:

<OnlineText>

<![CDATA[

<a href=http://somewhere/cgi-bin/query?POS=$ra,$dec&SIZE=$sizeD>Link     </a> to the associated source catalog.

   ]]>

</OnlineText>

 

No software currently uses this element and no data is supplied in these elements in the survey descriptions included in the Jar.

FITS

The FITS element includes information to be included in the FITS header generated with the image.  

Example:

<FITS>

<![CDATA[

SURVEY  = ‘IRAS 100 micron’

BUNIT   = ‘MJY/SR                    / INTENSITY                       

COMMENT   Note that the values are not adjusted for any change in pixel

COMMENT   size from the original pixel size of 90 x 90 arcseconds             

BLANK   =          -2000000000        / TAPE VALUE FOR EMPTY PIXEL      

ORIGIN  = ‘JPL-IPAC’                  /  INSTITUTION                    

TELESCOP= ‘IRAS                      /                                 

INSTRUME= ‘ISSA-FLD’                  /  IRAS SKY SURVEY ATLAS          

EDITED  =                    T        /  SCANS EDITED                   

APPCAL  =                    T        /  CALIBRATION CORRECTION 25 MICRON

DE-ZODY =                    T        /  DE-ZODIED IMAGE                

GLBL-D  =                    T        /  APPLIED GLOBAL PARAMETERS      

LOCAL-D =                    T        /  LOCAL DESTRIPER                

ASBLANK =                    T        /  ASTEROID BLANKING              

]]>

</FITS>

 

 

The Images Element 

The main business of the survey file is in the Images element and so we give it a whole section to itself.  There are two alternate configurations:  an explicit list of the images in the survey, or a link to a class that will generate a list of images for a given user request.   Most of the fields in the Images element (with the exception of Image elements) are parsed when survey processing begins and used to update or set entries in the Settings hash.  However, unlike the fields in the Settings element, these settings override any existing settings.

Explicit Image Lists

In this case, the Images element will contain a set of Image elements.  Each element has a text value with four blank delimited tokens: the filename or spell, a longitude and latitude, and the epoch of the image.  The coordinate system for the longitude and latitude can be specified in the “SurveyCoordinateSystem” setting and defaults to J2000.  In addition to the Image elements there should at a minimum be an ImageFactory element and an ImageSize element.  The image size gives the size in degrees of the smaller dimension  (if not square) of the images.  This is used in the process of determining the best image for each output pixel. 

The ImageFactory gives the class which given the appropriate spell or file name creates the image.  The skyview.survey.FitsImageFactory takes a filename and simply reads the file to create the input image.  If a <FileNamePrefix> element is included Images element, then this prefix is prepended to the file name in the Image element before being passed to the FitsImageFactory.

The skyview.survey.CachingImageFactory assumes that it is being given a spell that tells it both how to retrieve the data as a URL as well as sufficient metadata to create a proxy image.   The spell consists of  10 or 12 comma delimited fields.  These are:

1.      The URL to retrieve the data from

2.      The file name to store the cached file as

3.      The longitude of the center of the image

4.      The latitude of the center of the image

5.      The projection used in the image

6.      The coordinate system used in the image

7.      The width of the image in pixels

8.      The height of the image in pixels

9.      The width of a pixel in degrees

10.  The height of a pixel in degrees

..or..

9-12. The CD matrix elements for the image.

 

For projections (Aitoff, Cartesian and CSC) where the reference value is normally fixed, the third and fourth tokens are the pixel offsets for the image, while for other projections the reference pixel is assumed to be at the center of the image.

The user familiar with FITS will note the similarity of fields 3-10 (or 12) to standard FITS WCS keywords.  Note that the spell does not need to use the same coordinate system as the other fields in the Image elements.

If the SpellPrefix or SpellSuffix values are specified in the Images element, then the Image spell string is appropriately extended.

Image Generators

Rather than specifying each individual image, the survey file can describe an ImageGenerator class that will generate the images.  An Image generator can generate either files or spells.

There are specialized image generators for the DSS and 2MASS surveys.  Users may wish to examine these classes for extension to other special surveys, but the most useful generator is likely to be the SIAPGenerator.  The SIAP generator uses the Virtual Observatory Simple Image Access protocol and can be adapted to use many of the existing SIAP services with appropriate entries in the Images element.

The key fields are:

·        SiapURL: defines the base URL for the Simple Image Access service.

·        SiapProjection: The projection used in images returned by the service

·        SiapCoordinates: The coordinate system used in images returned by the service

·        SiapNaxis: The size (in pixels) of image returned by the service

·        SiapScaling: The size (in degrees) of the pixels returned by the service

·        SiapMaxImages: The number of images returned to include as candidates.

 

Alert readers will note the correspondence of most of the fields to the spell parameters and to the FITS WCS fields.  SIAP services can return WCS information for each image explicitly and to the extent that they do so (e.g., almost all return the size of the image in pixels), the user need not supply this information.  However many (most) SIAP services do not supply all needed information.  The fields above  remedy any deficit.

 

Note that not all SIAP services are good candidates for immediate inclusion as a service.  SkyView will use all images returned as candidates even if the images are quite heterogeneous.  Only SIAP services that return relatively homogeneous images can be used with some kind of initial filtering.  However it is often easy to modify the SIAPGenerator to do such filtering and we anticipate that this class will have filtering capabilities added.

The third example below shows the power of the SIAPGenerator.  The entire SDSS survey is included as a survey with just a few lines of text.

 

Examples:

Listing local files:

<Images>

  <FilePrefix>

    /surveydata/iras/iras100/

  </FilePrefix>

  <ImageFactory>

    skyview.survey.FitsImageFactory

   </ImageFactory>

<ImageSize> 12.5 </ImageSize>

<Image>  i001b4h0.fits    0.   -90.   1980.00 </Image>

<Image>  i002b4h0.fits    0.   -80.   1980.00 </Image>

<Image>  i003b4h0.fits   45.   -80.   1980.00 </Image>

<Image>  i004b4h0.fits   90.   -80.   1980.00 </Image>

<Image>  i005b4h0.fits  135.   -80.   1980.00 </Image>

… 425 more Image elements …

</Images>

Listing remote URLs:

<Images>

<SpellPrefix>  http://skyview.gsfc.nasa.gov/survey/iras100/images/

</SpellPrefix>

<SpellSuffix> ,Tan,B1950,500,500,0.025,0.025      </SpellSuffix>

<ImageFactory> skyview.survey.CachingImageFactory </ImageFactory>

<ImageSize> 12.5 </ImageSize>

<Image>  i001b4h0.fits,i001b4h0.fits,0.,-90.   0.   -90.  1980. </Image>

<Image>  i002b4h0.fits,i002b4h0.fits,0.,-80.   0.   -80.  1980. </Image>

<Image>  i003b4h0.fits,i003b4h0.fits,45.,-80.  45.  -80.  1980. </Image>

<Image>  i004b4h0.fits,i004b4h0.fits,90.,-80.  90.  -80.  1980. </Image>

<Image>  i005b4h0.fits,i005b4h0.fits,135.,-80. 135. -80.  1980. </Image>

… 425 more images …

</Images>

 

 

 

 

 

Using an Image generator.

<Images>

  <SiapURL>

<![CDATA[

http://casjobs.sdss.org/vo/DR3SIAP/siap.asmx/getSiapInfo?FORMAT=image/fits&BANDPASS=I&

]]>

  </SiapURL>

  <SiapProjection>  Tan   </SiapProjection>

  <SiapCoordinates> J2000 </SiapCoordinates>

   

   <ImageFactory>   skyview.survey.CachingImageFactory </ImageFactory>

 

  <ImageSize>       0.25 </ImageSize>

  <ImageGenerator>  skyview.survey.SIAPGenerator</ImageGenerator>

</Images>

 

A Simple Survey Definition File

While a survey XML definition file can be complex, most data is optional.  A minimal survey definition need only include the short name of the survey, a default scaling, the image size and the names and locations of the included files.

 

<Survey>

  <ShortName> MySimpleSurvey </ShortName>

  <Settings>

       <Scale> .1 </Scale>

  </Settings>

  <Images>

    <ImageSize> 10 </ImageSize>

    <ImageFactory> skyview.survey.FitsImageFactory</ImageFactory>

    <Image> file1.fits  0. 0. 2000 </Image>

    <Image> file2.fits 10. 0. 2000 </Image>

    <Image> file3.fits 20. 0. 2000 </Image>

  </Image>

</Survey>

 

This describes a survey with three images along the equator (at RAs of 0, 10 and 20 degrees) which have a minimum dimension of 10 degrees and  0.1 x 0.1 degree pixels.