SkyView in a Jar

 

Using SkyView as a Standalone Java Application

 

 

Thomas McGlynn and Laura McDonald

High Energy Astrophysics Science Archive Research Center

Goddard Space Flight Center

Greenbelt, MD 20771

 

Version 2.00

March 10, 2006

 

Three color all sky image in Radio, X-ray and γ-ray

(Equatorial coordinates  with Galactic coordinate grid overlay)

[survey=408mhz,heao1a,egrethard position=0,0 size=360,180 pixels=1200,600 grid=g gridlabels projection=Ait]

Table of Contents

 

1. Introduction. 3

2. Setup. 3

3. SkyView-in-a-Jar and ImageJ. 5

4. Command Summary. 6

4.1 Settings and Defaults. 25

5. Example requests. 27

6. Catalogs. 28

7. The Cache. 29

7.1 Caching the Digitized Sky Survey. 30

8. Dependencies on Remote Systems. 30

9. Compatibility with the SkyView Batch Interface. 31

10. SkyView-in-a-Jar Operations. 32

10.1 Getting defaults and user arguments. 32

10.2 Updating Settings. 32

10,3 Locating surveys. 32

10.4 Looping over the surveys. 33

10.5 Getting candidate survey images. 33

10.6 Building images. 33

10.7 Finding the best survey image for each output pixel. 34

10.8 Preprocessing. 35

10.9 Validating the image. 36

10.10 Resampling setup. 36

10.11 Sampling in energy. 36

10.12 Sampling in space. 36

10.13 Post-Processing. 37

10.13.1 De-edging. 37

10.13.2 Graphics Processing. 37

10.14 Data conversion and writing the FITS file. 38

10.15 Cache cleanup. 38

11. Survey Definition File. 39

11.1 ShortName. 39

11.2 Name. 39

11,3 Description. 39

11.4 Settings. 40

11.5 MetaTable. 40

11.6 OnlineText 41

11.7 FITS. 41

12. The < Images> Element 42

12.1 Explicit Image Lists. 42

12.2 Image Generators. 43

12.3 Example <Images> elements. 44

13. A Simple Survey Definition File. 45

14. Contents of the JAR file. 45

14.1 The Class Files. 45

14.2 Survey Data and Metadata. 46

14.3 Control Files. 46

14.4 Source code. 46


 

1. Introduction

SkyView-in-a-Jar provides users with a local SkyView system on their own machines.  Users can generate FITS, GIF, JPEG, … images from major surveys in any requested geometry, resample and mosaic their own data, overlay grids and catalog positions, and create their own surveys.  SkyView-in-a-Jar comes completely ready to use as a single file.  In this default configuration almost all surveys accessible through the SkyView web pages are available.  It is easy to add new surveys using local files or the Virtual Observatory Simple Image Access protocol and any catalog that users the VO Cone Search protocol can be queried.

The code itself is itself extensible.  Users can add new samplers, projections, and coordinate systems.  Even the classes which define the basic algorithms for finding surveys, selecting images within the survey, and mosaicking the images can be overridden by the sophisticated user.  Users can insert their own pre- and post- processing functions to modify the input data or the resulting image.

Although there are many options available to the user, there are very few required inputs.  Users need only specify the position around which they would like an image, and the survey from which the image is to be extracted.  SkyView will take care of the rest.

This document provides a comprehensive User’s Guide to SkyView-in-a-Jar.

2. Setup.

SkyView-in-a-Jar is self-contained within a single executable JAR file.   JAR files are a modified version of ZIP files used to store Java code.  However JAR’s can include data and program control files as well as the Java code.  SkyView-in-a-Jar contains all of the control files needed to access the surveys available from SkyView’s standard Web interface.  For a few surveys the survey data itself is included.  For these surveys not even network connections are required.  The other surveys are accessed through standard Web protocols.  Even if the basic survey data is accessed over the Web, all processing and resampling is done on the local machine.

Users can download the JAR file from

http://skyview.gsfc.nasa.gov/jar/skyview.jar

and then run the file using the command

java -jar skyview.jar key1=arg1 key2=arg2 …

The same JAR file is used for all operating systems.  This program requires at least a Java 1.5 (sometimes called Java 5.0) environment to run.  Java distributions for most major operating systems are available at

http://www.java.sun.com.

Text Box: java -jar skyview.jar survey=DSS,FIRST position=3c273 quicklook=jpg
Processing survey:dss
Number of candidate source images is 7.
Number of source images to be resampled is 1.
Processing image #3
Creating quicklook image: output1.jpg
Opening output file output1.fits.

Processing survey:first
Number of candidate source images is 5.
Number of source images to be resampled is 1.
Processing image #3
Creating quicklook image: output2.jpg
Opening output file output2.fits.

Sample command and output.

For example to create a DSS and FIRST  images of 3c273  enter the bolded text above.  The output of the program shows that the program looks at 7 candidate source files for the DSS but uses only one of them in generating the output.  It uses 1 of 5 for the FIRST survey. A FITS file and JPEG are produced for each survey.   If the quicklook argument had been omitted, no JPEG would have been produced, just a FITS file.  The JPEGs are shown below.

 

Text Box:                           
Output images

 

There are many options controlling how to generate and resample the image. All command settings are described in the command summary section and then discussed in the context of how SkyView works.


 

The SkyView Java classes in the JAR file can be used in other applications as well.  Simply include the JAR file in your CLASSPATH.   The only difference between a regular and executable JAR file is that the later contains a manifest file that indicates how to start up when the -jar option is used in the java command. The SkyView Java application could also be started up with

java –cp pathto/skyview.jar skyview.executive.Imager key1=arg1 …

If you want to use the Imager class in other Java applications go right ahead.

Later versions of the JAR files may use a different main class, but the command line arguments will be upwardly compatible.

3. SkyView-in-a-Jar and ImageJ

Starting with version 1.9, the SkyView JAR also contains a complete distribution of the ImageJ image display and analysis tool.  ImageJ is a very powerful public domain software package developed by Wayne Rasband (wsr@nih.gov) at the National Institutes of Health.  SkyView currently uses ImageJ functions for creating quicklook images in GIF, JPEG and other formats, including various scaling and color table operations.  ImageJ has many other capabilities for users may to analyze images created by SkyView.   Hundreds of plug-in modules are also available that provide powerful functionality in diverse areas.

 

Users should be able to start ImageJ itself with the SkyView JAR using the command

    java –cp skyview.jar ij.ImageJ [other arguments]

or they can download the ImageJ JAR file at http://rsb.info.nih.info/ij.  Much more information on ImageJ is available at that site.   Future versions of the SkyView are likely to use additional ImageJ functionality.   The SkyView JAR includes both the ImageJ

class and source code.  Note however that the class files are copied directly from the ImageJ distribution version v1.35r and have not been recompiled from the source.

 

You can start an ImageJ session to analyze SkyView images generated by this application by specifying the ImageJ setting.

 

4. Command Summary

This section describes the command line arguments to the SkyView-in-a-Jar application.  Each command line argument consists of a key=value pair.  The command keys are case-insensitive.  In some cases, particularly when the value is a file or class name, the values may be case-sensitive.   Generally, to associate more than one value with a key, the values are separated by commas. 


 

 

SkyView-in-a-Jar Command Line Arguments

 

Key

Value

Survey

The survey or surveys indicate which surveys the user wishes to generate an image for.  If more than one survey is desired, the values should be separated by commas.  The list of all surveys known (by default) to the program can be generated by invoking the application with no arguments.  This will give a quick description of the arguments and a list of known surveys.  Survey names are case insensitive.

 

The survey user is special.  It indicates that one or more files indicated in the userfile argument are to be used as the survey.  Using the survey=user and userfile=file1,file2,… arguments, users can resample and mosaic local data files.

 

Survey=DSS,2MASS-K

Settings

This setting may be used to specify one or more files that are to be read for settings.  The files specified here are processed after the system setting file (if any) but before any survey specific settings are extracted.  If the same setting is set in multiple locations, only the last setting is used.  The

format of Settings files is discussed in section 4.1.

 

settings=dir/mydefaults.set,thisrequest.set

 

Output image geometry

 

Position

The position argument gives the center position for which images are requested.  It may be either a coordinate pair, or a target name.  The position argument or the Lon/Lat pair is required.

 

Position=3c273

Position='10 20 30, -15 16 22'   

Note that the quotes are not part of the actual value: they are needed to ensure that the value is treated as a single string.  Your operating system may have other methods for enclosing embedded spaces in an argument.

Lon

This argument gives the Longitude or right ascension of the center of the image.  Lon/Lat arguments supersede the position if both are specified.

Lat

This argument gives the Latitude or declination of the center of the image.


 

Coordinates

This argument gives the coordinate system used for the output image.  The first letter in the value gives the type of coordinates.  If a number is appended then this is the epoch of the equinox of the coordinate system. 

 

Recognized values include

    J  -- Julian equatorial coordinates

   B  -- Besselian equatorial coordinates

   ICRS – International Coordinate Reference System coordinates.

   Gal – Galactic coordinates

   H – Helioecliptic coordinates

   E  -- Ecliptic coordinates.

 

The single letter values may be combined with an epoch, e.g.,

J2000, B1950, E2020.123

 

The default coordinate system is J2000.

 

Coordinates=J2000

Coordinates=Galactic

Projection

The three letter designation used to specify the projection of the output image.  This uses the suffix in the FITS CTYPE keywords.  Supported projections include Car, Tan, Ait, Zea,  Sin and Csc.  The Tan, Sin and Zea projections are typically used for small regions, including most high-resolution astronomical images.  The Car, Ait and Csc projections are generally used for all sky images – though it is not too uncommon to see the Car projection used for smaller images.  The Csc projection first projects the sphere to the six surfaces of a cube, and then unwraps these surfaces into a sideways T.

 

The default Tan is fine for images smaller than a few degrees.

 

Projection=Tan

 

Global geometry of projections

Tan

Sin

Ait

 

 

 

Zea

Car

Csc

 

(Images generated with:

survey=408mhz Projection=Xxx position=0,0 size=360,180 pixels=100,50 coordinates=g grid)

 

 

Equinox

If the coordinate system only specified a coordinate type, the equinox can be specified separately.

 

Equinox=1980

 

Scale

Scale gives the size of pixels in the output image and corresponds to the CDELT keywords.  Scale may be specified as a scalar, in which case the pixels are square, or two values may be given if rectangular pixels are desired. The scale is always positive.  Output images follow the astronomical convention where the longitude decreases along a horizontal line.  The scale is specified in degrees per pixel. 

 

The default scale depends upon the survey.  If you do not specify an explicit scale, and you choose more than one survey, then the scale may differ among surveys.

 

If there is no survey defined default, then the default scale is 1” per pixel.   A user may specify the scale of individual pixels, or the size of the image as a whole, but not both.

 

Scale=0.0025

Scale=0.0002777777

Scale=0.0002777777,0.0005

 

Using scale on DSS cutouts near 3c273

scale=0.0003

scale=0.0003,0.0001

Resolver

When a target name is specified rather than specific coordinates, the name must be ‘resolved’ into coordinates.  There are two widely used name resolvers available: NED and SIMBAD.   The application invokes these through a HEASARC web service.  This argument can be used to specify which resolver or resolvers are to be used.  If both resolvers are specified, then the second is checked only if the object is not found at the first.  Values are: SIMBAD, NED, NED-SIMBAD and SIMBAD-NED.  The default is SIMBAD-NED.

 

Resolver=NED

 

Pixels

This gives the number of pixels in the output image.  If specified as a single value, the number of pixels will be the same in both dimensions.

 

The default is 300.

 

Pixels=500

Pixels=1280,640

 

Rotation

The argument indicates that the output image is rotated by the specified angle (in degrees).

 

By default there is no rotation of the image.

 

Rotation=45.0

 

Rotating 3c273

No rotation

Rotation=30

 


 

 

Graphic outputs

 

Quicklook

This argument is used to give the format for quicklook/graphic outputs.  The supported formats are: GIF, JPEG (or JPG), TIFF and BMP.  A value of JPG is assumed if there are other graphic keywords and this argument is not specified.  The output format for three color images (see RGB below) is always JPEG.  The argument value is not case sensitive.

 

Quicklook=gif

ImageJ

This option starts an ImageJ application and feeds the survey images being produced into it.    The main ImageJ control window is a simple menubar.  For each survey there will also be an window.  You can click on a survey window to make it active and use the functions in the ImageJ control window to manipulate it.  ImageJ has dozens of powerful functions and is described in detail at http://rsb.info.nih.gov/ij.  To leave the ImageJ application use the Quit option of the File menu.

 

ImageJ

 

ImageJ Windows

ImageJ Control Window

Typical ImageJ image window

(survey=sdssr pixels=600 position=m101 lut=fire deedger=skyview.geometry.deedger strictgeometry imagej)

 

 


 

Scaling

This argument controls the scaling between the data and the displayed images.  Recognized values include

Linear – The output pixels are linearly scaled from the input.

Sqrt  -- Output scaled as the square root of the input pixels.  Negative pixels are scaled to the minimum.

Log – Output scaled logarithmically from the input.  Negative pixels scaled to the minimum.  This is the default.

HistEq – Output scaled so that each output bin has the same number of pixels.

 

scaling=Sqrt

 

Image scaling for 3c273

Linear/default

Sqrt

Log

Hist


 

LUT

The argument allows the user to read in a lookup table which will be used to translate pixels values to color.

There are three sources for look up tables.  ImageJ defines the following lookup tables internally:

Fire, Ice, Red, Yellow, Green, Blue, Cyan, Magenta, Grays, Spectrum, Red/Green, and 3-3-2 RGB.

There are also a set of lookup tables distributed with the JAR file to mimic standard IDL color tables.  These may be specified using the file name colortables/xxx.bin where xxx is the IDL color table name with spaces replaced by hyphens.   (These are also available using the syntax as the old SkyView Batch interface using the Batch Compatibility feature).

 

Users may also create their own color tables. The only format currently supported is a 768 byte file comprising the 256 red values followed by 256 green and 256 blue.

 

When using the internally defined color tables, the case does not matter, but for many operating systems if mixed case files names are used, the argument must be in the correct case.  Note that the program will always check for files with the argument in lower-case first.  If you have two LUTs  xxx.lut and xXx.lut, then the program will always choose the former even when the user specifies lut=xXx.lut.

 

If no color table is specified a gray scale image is produced.

 

lut=’3-3-2 RGB’

lut=colortables/Stern-Special.bin

lut=mylutDirectory/myLUT.lut

 

DSS image of M81 (size=0.5 lut=Fire)

Invert

Invert the output.  The value of this argument is ignored and may be elided.  The output image is given with the color table used in inverse order (or with a white rather than black background if no color table is specified).

 

Invert

Inverted Gray Scale DSS image of M81 (size=0.5 invert)

Min

Set the minimum value for the output image.  Before scaling, all values below the minimum will be set to the minimum.  This affects only the graphic image, not any FITS data.  In the example below a few very small positive values make a simple log-scaled image look peculiar.

 

min=0.

 

Galactic center in hard X-rays

(survey=integralspi_gc size=50 pixels=400 scaling=log projection=Car coordinates=G position=0,0)

No minimum specified

min=0.1

 

 

Max

Set the maximum value of the output image.  Before scaling, all values above the maximum will be set to the maximum.  This affects only the graphic image, not any FITS data.  This can be useful if a few very bright pixels making it difficult to scale the rest of the image properly.

 

max=256

 

RGB

Create a color image from three surveys using the first as the red color, the second as green and the third as blue.  The output image is always created as a jpg, e.g. output_rgb.jpg.  The value of the argument is ignored and may be elided.

 

RGB

 

The IRAS Galactic center (R=100μm, G=25 μm, B=12 μm)

(survey=iras100,iras25,iras12 position=0,0 coordinates=g scaling=log size=15,7.5 pixels=600,300 rgb)

Grid

This option requests a coordinate grid to be overlaid on the quicklook image.  The user may request a coordinate grid in a different coordinate system than that used in the image by specifying the value for this setting.

 

Grid        Overlay with a grid in the image's own coordinate system.

Grid=G               Overlay with a Galactic coordinate grid.

 

Gridding the 1420 Mhz survey

(survey =1420mhz position=0,0 size=360,180 projection=Ait pixels=400,400)

grid

grid gridlabels

grid=g

grid=g gridlabels

GridLabels

Label the grid lines with coordinate values.  Equatorial coordinates use sexagesimal values, but other coordinate systems use labels in decimal degrees.


 

 

Catalog Options

 

Catalog

Specify one or more catalogs to query for objects in field of the image.

The names used for catalogs are as described in the text or a full URL may be specified.  This is assumed to return a VOTable matching the Cone Search protocol description (though it need not be a cone search query).

 

If a quicklook image is requested, then the positions of the catalog objects in the field will be overlaid on the image.  However the catalog option does not in itself imply graphics ouyput.

 

Catalog=rosmaster,ascamaster

Catalog=ned,I/246

Catalog=http://some.web.edu/aFile.xml

 

The HEAO1A X-ray sky overlaid with the 4U survey.

(survey=heao1a position=0,0 catalog=uhuru4 pixels=600,300 projection=Ait size=360,180 scaling=log quicklook=jpg)

 

after adding catalogIDs

after adding catalogfilter=count_rate>20

 

 

CatalogFile

This setting indicates that a file listing the catalog objects found in the field should be created.  The file name may be given as the argument, or if no argument is specified a ".tab" suffix is appended to the stem used for the first file image.  It is possible for users to request a series of images with different sizes (since the default pixel scale varies by survey).  The catalog file will give the objects found in the region of the first image requested.

 

CatalogFile

CatalogFile=objects.list

 

Example output (adding catalogFile=out.cat to HEAO1A request)

  N  | Cat    | ID/Name             |   RA/Lon |  Dec/Lat |     X |     Y

   1 | uhuru4 | 4U0042+32           |   11.1745|   33.0484|  283.7|  204.4

   2 | uhuru4 | 4U2142+38           |  326.1659|   38.3175|  346.6|  213.3

   3 | uhuru4 | 4U0336+01           |   54.6950|    1.1822|  209.7|  152.0

   4 | uhuru4 | 4U2321+58           |  350.8689|   58.8326|  309.0|  243.9

   5 | uhuru4 | 4U0316+41           |   49.9727|   41.5335|  234.1|  219.1

   6 | uhuru4 | 4U2030+40           |  308.0896|   40.9559|  368.9|  218.3

....

 

CatalogRadius

Normally all catalog objects found in the image will be shown.  If the catalog radius is specified, then only those objects that are also within the specified radius (in degrees) from the center of the image will be included.

 

CatalogRadius=0.2

 

CatalogFilter

A catalog filter filters the results according to the specified criteria.  A filter string comprise a field name, operation, and value.  Multiple filters on the same field may be specified.  If the returned data has no column which matches the field name, then no rows will be returned.  CatalogFields may be set to show the fields returned by requested catalogs. 

 

CatalogFilter=vmag>12,vmag<15

 

CatalogFields

If a CatalogFile has been specified, then include a listing of all of the fields in the catalog.  This can be useful when a user wishes to know the fields available for catalog filters.

Example catalog output generated by CatalogFields:

 

Table: uhuru4 Field:unique_id

      type:char[*] UCD:ID_MAIN

Table: uhuru4 Field:name

      type:char[*]

Table: uhuru4 Field:ra

      type:double UCD:POS_EQ_RA_MAIN units:degree

Table: uhuru4 Field:dec

      type:double UCD:POS_EQ_DEC_MAIN units:degree

Table: uhuru4 Field:count_rate

      type:double units:ct/s

CatalogIDs

Indicates that both symbols and ID strings should be overlaid on images.


 

 

Resampling

 

Ebins

This argument is used only when sampling three-dimensional images which have a set of ‘energy’ values at each pixel.  It must have three comma separated numeric values.  The first gives the offset of the first output energy bin, the second gives the width of the output bins in terms of the original bins, and the third gives the number of output bins.  E.g., suppose we have 10 channels in the input data.  We wish to skip the bottom 2 channels and the top half channel and rebin the remaining 7.5 channels into 3 output  bins in the output image.  Then we have

 

Ebins=2,2.5,3

 

Note that the offset is 0-based.  The bin size does not need to be an integer, the output bins will be integrated over the input bins assuming flux to be uniformly distributed over the energy bin.  In the example given, the first bin in the output will be the total of the third and fourth and half the fifth bin of the input.

 

In SkyView, 3-D surveys normally specify a default for the Ebins value which gives a 2-D output image.

 

Sampler

This argument specifies the sampling method to be used. The value consists of an algorithm designator which may in some cases be suffixed with an integer giving the order of the sampler.  Currently supported samplers include:  Nearest Neighbor (NN), Bi-linear interpolation (LI), Splines from 2-5 order (Spline or Spline3, Spline2, Spline4, Spline5), Lanczos interpolation with a specified kernel size (Lanczos, or LanczosN where N >= 2, the default is 3), and an exact-area resampling using Sutherland-Hodges clipping (Clip).  The default is nearest neighbor.

 

Sampler=Lanczos5

Sampler=Spline

Sampler=Clip

StrictGeometry

This flag can be set to indicate that the best pixel should be determined for each pixel individually rather than using any approximations.  If an image appears to have missing segments due to resampling the wrong input, then using StrictGeometry may help.

 

Outputs

 

Output

The name (or stem) of the output image or images.  The user may specify the full path of the output image x.fits or just a stem x.  If more than one image is being produced, then ordinal numbers will be added so that the output files will be x1.fits, x2.fits.  If compression is requested the .gz extension should not be specified, it will be added automatically.

 

The default is output.

 

Output=~/myoutputs/test.fits

Output=c:\datafiles\3c273

Compression

This flag indicates that the output files are to be gzip compressed.  The default is not to compress the data.   The content of the value is ignored and the value can be omitted.

 

Compression=T

Compression   

Float

This flag indicates the output files should be 4 byte reals.  The default is 8 byte reals.  The content of the value is ignored and can be omitted.

 

Float=T

Float

NoFITS

Suppress the FITS output.  The value of the argument is ignored and may be elided.  This argument may be used in conjunction with graphics arguments if only a quicklook (e.g., GIF or JPEG) is desired.

 

nofits

 

Survey definition

 

Userfile

This argument is used to specify one or more files that are treated as the constituent image files of the ‘user’ survey.  Users can resample and mosaic local files easily using this argument and the user survey.

 

There is no default.

 

Userfile=myfile1.fits,myfile2.fits

 

SurveyXML

This argument allows the user to specify one or more survey XML descriptors that users can generate images from  (see below for details of the format).  Use this argument to specify individual files and the XmlRoot argument to specify an entire directory.

 

SurveyXML=surveys/mysurv1.xml,surveys/mysurv2.xml

 

 

XmlRoot

This argument  gives a directory where survey XML files are to be found.  Each ‘.xml’ file in the directory will be treated as a survey descriptor file.  Currently only one directory is permitted.

 

XMLRoot=/survey/docs/xml

SurveyManifest

The Survey manifest is a file (or Java resource) that lists survey XML files.  By default the Survey Manifest is given in the system resource surveys/survey.manifest that is included in the JAR.

 

SurveyManifest=mysurvey/docs/mysurvey.manifest

 

 

Caching of survey files

 

Cache

When a survey’s data is retrieved from a remote location, it is cached in the local file store.  The cache argument specifies one or more directories in which the application should look for the remote data before attempting to retrieve it.  If a file matching the desired name is found, it is used directly.  The first directory in the cache is used to save any files that need to be retrieved.  If no cache is specified the current directory is used as the cache.

 

Each survey can be cached into a different directory by specifying a unique cache value in the Settings element of the Survey descriptor file or a common cache can be used for multiple surveys

 

Cache=/surveycache/cache1,/surveycache/cache2

PurgeCache

This argument indicates that cache files are to be purged after the processing of that survey is completed.  Only survey files actually downloaded in this request will be deleted.  Other files in the cache are left unchanged even files used in processing.  The value of the argument is not used. All files needed for a given image will be downloaded before any files are purged.

 

PurgeCache=T

PurgeCache

 

Classes used in processing

 

SurveyFinder

The class used to find surveys.   This class must implement the skyview.survey.SurveyFinder interface.

 

By default this is the class skyview.survey.XMLSurveyFinder but it may be overridden by knowledgeable users.

ImageFinder

The class that determines which survey image to sample for a given output image pixel.    This class must inherit from the skyview.geometry.ImageFinder abstract class.

 

The default is skyview.geometry.imagefinder.RecursiveImageFinder


 

PreProcessor

One or more classes implementing the skyview.geometry.Processor interface that are to be used in processing before mosaicking takes place.  If a user needs to communicate parameters to a pre (or post) processor, additional Settings can be defined which are used to transmit the information.   A preprocessor is used when Catalog is set so that catalog retrieval can occur in parallel with image construction.  If filtering of the input images were desired, a preprocessor could be used for this.

Mosaicker

A class implementing the Processor interface that fills in the pixel values in the output image.  The default mosaicker is the class skyview.geometry.Mosaicker.

Deedger

A de-edger is a specific type of Post-Processor which allows the user to do additional processing on the image after it has been mosaicked. 

Currently the skyview.geometry.Deedger and several other deedging classes can be used or the user can write their own class.

Supplied classes are:

   skyview.geometry.Deedger which de-edges using the default algorithm (currently using the BoundaryMedian)

   skyview.geometry.deedger.BoundaryMedian  sets the median change across the boundary of two image to 0

   skyview.geometry.deedger.BoundaryAverage sets the average change across boundaries to 0

   skyview.geometry.deedger.ImageMedian sets the median pixel value for the regions derived from different input images to a  common value.

All of these de-edgers add a constant to each of the images.  More complex algorithms which add a slope or more complex flat-fielding could be coded.

 

When survey images are mosaicked together there may be artifacts at the edges of the images where the different backgrounds of the images merge.  The deedger attempts to minimize these artificats.  Deedging is turned on by default for some surveys (e.g., the DSS).  It can be turned off for those surveys by using deedger=null.

 

The de-edger is a kind of PostProcessor in many cases a user could use the PostProcessor setting equivalently.  However using the Deedger setting is preferred since it allows survey defaults to set de-edging on or off without confusion with other post-processors.

 

PostProcessor

One or more classes defining the Processor interface which process the image after mosaicking.   De-edgers (see above) and the graphics processor are included in the JAR..

 

The class skyview.ij.IJProcessor handles the interface to ImageJ functionality.  A call to this processor will normally be added whenever one of the graphic arguments is specified so that users do not need to specify this setting explicitly.

SettingsUpdaters

This argument is used to indicate classes that might wish to look over the arguments that have been specified before image processing begins.  E.g., one could build a class that translated arguments from French to the English values expected in the program. 

 

Threee classes are specified in the default settings: skyview.executive.BatchCompatibility, skyview.vo.CatalogProcessor, and skyview.ij.IJProcessor.  The first translates arguments from the old SkyView batch interface to comparable values for this program.  The second ensures that the catalog pre- and post-processors are called if catalog arguments are found.  The last fills in default values when the user specifies certain graphics keywords.  Classes specified in this setting should implement the skyview.executive.SettingsUpdater interface.

 

SettingsUpdaters=skyview.executive.BatchCompatibility,skyview.ij.IJProcessor,skyview.vo.CatalogProcessor

 

 

Settings used in image generation

 

ImageGenerator

An image generator class can be specified in lieu of a list of <Image> elements in the survey description file.  It should return a set of strings that can be used by an image factor to generate candidate images.  The

skyview.survey.SIAPImageGenerator can use the VO SIA protocol.

ImageFactory

An image factory must be specified in the <Images> area of the survey description file.  It converts a string from an <Image> element or from an image generator into an image the can be used by the program.  The two primary image factories are the skyview.survey.FitsImageFactory which is given the name of a file and simply reads it in, and the skyview.survey.CachingImageFactory which is used to access remote data and supports caching.

 

4.1 Settings and Defaults.

 

The SkyView settings can be set in several ways:

 

  • At startup the application attempts to find a system settings file.  This establishes base defaults for the system.  The application also ensures that values are set for critical system settings (e.g., the class names for key processing objects). These default settings are normally read in through the system resource skyview.settings.  If users wish to provide their own default settings, they can override the settings provided in the JAR file by creating a skyview.settings file in the current directory, or by pointing to a file in any location (and of any name) using the environment variable SKYVIEW_SETTINGS.

 

  • Users can specify settings on the command line.

 

  • After command line processing, if the setting 'settings' is specified, then the specified files are parsed for settings.

 

  • When surveys are processed, the Settings element in the survey definition file is parsed for survey specific settings.  These settings will not override any settings already in place.  These are used for survey defaults like image scaling, or de-edging.

 

  • When surveys are processed, Settings may also be set in the Images element.  These will override any user specified settings.  Any element other than the <Image> elements specifies a setting.  These are intended to define and communicate with the classes that are used to build survey images but users are free to exploit the differences in processing between how settings are processed inside the Images to set settings that the user cannot override.

 

  • Settings are also specified internally by various classes but normally this is invisible to the user. 

 

Both the system and user settings files are simple ASCII files.  Each line can be a blank, a comment line (first character of the line is ‘#’) or specify a single setting.  This example might be used as the systems settings:

 

# An example skyview.settings file.

 

Pixels=500

Projection=Sin

Coordinates=J2000

Sampler=Lanczos

Equinox=2000

 

Resolver=NED-SIMBAD

XMLroot=$SKYVIEW_XMLROOT

 

Mosaicker=skyview.geometry.Mosaicker

SurveyFinder=skyview.survey.XMLSurveyFinder

ImageFinder=skyview.geometry.imagefinder.RecursiveImageFinder

SettingsUpdaters=skyview.executive.BatchCompatibility,skyview.ij.IJProcessor

 

System environment variables can be used within settings.   In this example an environment variable is used to set the XMLRoot (the directory where survey XML files may be found). When a value begins with a $ the value of the corresponding environment variable will be used but the line will be ignored if the environment variable is not defined.  This allows users to override the default settings using environment variables using two lines in the settings file, e.g.,

Projection=Sin

Projection=$PROJ

will use a Sine projection unless the user overrides it by specifying the PROJ environment variable.

 

While a user can specify any number of settings on the command line, only one setting may be given on a line in a Settings file.

 

If a user is making a complex request it may be convenient to put the request parameters in a file.  E.g., the user might try:

   java -jar skyview.jar settings=request.settings

where the file request.settings has the contents

 

            survey=dss

      position=3c273

      size=.5

      catalog=rosmaster,ascamaster,chanmaster

      catalogIDs

      grid

      gridlabels

      quicklook=gif

 

User settings files can have comments and use logical names just like system files.  More than one settings file can be specified.  If a setting is specified in multiple locations, the last value is used.  The idiom settings=default,specific lets the user override default settings with values for a specific request.

 

5. Example requests.

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h

This returns three 300x300 pixel images named output1.fits, output2.fits, and output3.fits with the survey default  pixel size..

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h   \

    quicklook=jpg nofits

 

Get the same images but only as JPEGs

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h \ 

    quicklook=jpg scaling=log

 

Get the same images but only as both FITS and JPEGs.  Scale the JPEGs logarithmically.

 

java -jar skyview.jar position=3c273 survey=2mass-j,2mass-k,2mass-h rgb

Get the FITS images along with a 3-color JPEG.

 

java -jar skyview.jar position=3c273 survey=2mass-j \

scale=.0005 pixels=1200

 

This returns a single file 1200x1200 pixel file named output.fits with pixels of 1.8”.

 

 

 

java -jar skyview.jar position=3c273 survey=2mass-j \

cache=/local/data/2masscache

 

This returns a 300x300 file but looks in the cache directory /local/data/2masscache to see if needed survey files have been downloaded there first.  In the previous examples all survey files were placed in the ‘current’ directory.  Images are generated much faster if needed survey data is already present in the cache.  If the underlying data is not already in the cache it will be placed in the specified cache location.

 

java -jar skyview.jar position=3c273 survey=2mass-j \

cache=/local/data/2masscache purgecache

 

This returns the same image as the previous request, but deletes any survey files downloaded from the 2MASS site for this request.  However any files that might have been downloaded in previous requests are left untouched.

 

java -jar skyview.jar position=3c273 survey=user \

   userfile=ff1.fits,ff2.fits,ff3.fits   scale=.003 \

   sampler=Lanczos4 deedger=skyview.geometry.Deedger

 

This request looks at the three local files ff[1-3].fits and tries to create a mosaic from them using a fourth order Lanczos resampler. Boundaries between the input images are to be masked using the standard deedger class. 

 

java -jar skyview.jar position='horsehead nebula' \

postprocessor=null output=horsehead \

float coordinates=B1950

 

This asks for a DSS image of the Horsehead nebula but turns de-edging off.  The output image should use 4-byte reals and be in B1950 coordinates.

 

6. Catalogs

 

Users can find objects and observations that are within the fields of generated images from thousands of catalogs at a variety of sites.  The application uses the simple Virtual Observatory Cone Search protocol to query catalogs.  All HEASARC and Vizier catalogs  that can be queried by position are available.  A number of other special catalogs (e.g., NED) are also available. 

 

Catalogs are requested using the Catalog setting.  Multiple catalog requests can be separated by commas.  The catalog names are of three types.  The following names are recognized specially: NED and SIMBAD.  All names that include a "/" are assumed to be Vizier tables.   All other tables are assumed to be HEASARC tables.  You can query the Vizier system at http://vizier.u-strasbg.fr or the HEASARC at http://heasarc.gsfc.nasa.gov/cgi-bin/W3Browse/catindex.pl to get a complete list of catalogs available from these systems.  The name used for HEASARC catalogs is the short single token name of the catalog not the longer descriptive name.

 

Catalog queries run in parallel with the generation of images with each query in a separate thread.  With a bit of luck the catalog queries will be done before any image is ready.

 

The catalog queries are cached by the system, so that if the user asks for more than one image, the query need not be re-run for each image.  However, if the second image uses larger pixels than the previous image, then the query will need to be re-run to ensure that the entire image  is included (unless the CatalogRadius setting is specified).  For maximum efficiency the surveys with the largest scale should be specified first.  If the CatalogRadius is specified, then an object must be both within the field of the observation and within the specified radius of the center of the field to be included in the results.

 

If the CatalogFile setting is specified a file is produced which contains a simple ASCII table with all of the returned catalog data.  Each row gives the catalog from which the row was extracted, the name of the catalog object (using a column selected by the catalog provider), the J2000 RA and Dec of the object, and the X and Y pixel locations.  These locations use the corner of the image as 0,0 (offset by 0.5 from the standard FITS convention).  While the data for most columns in the input table are not included in the catalog file, all of the column names for the input column will be listed if CatalogFields is set. 

 

The results of the catalog query can be filtered using the CatalogFilter setting.  The user specifies a simple criterion on the value of a given column.  E.g., if a user knows that a catalog has a column named vmag, then

   CatalogFilter=vmag<10

will ensure that only the brightest objects are displayed.  If a specific range were desired one could try

   CatalogFilter=vmag<10,vmag>6

E.g., suppose a user wished to know the location of archived Chandra observation, then  they might try

   Catalog=chanmaster CatalogFilter=status=Archived

 

 

Valid operators include =, >, <, >=, <=.  Note that the > and < characters may need to be escaped if specified on the command line.

 

The catalog options do not automatically turn on graphics display since a user may just be interested in the catalog output file.  However if graphics is selected by some other option, then the positions of each catalog object will be overlaid. If the CatalogIDs field is specified, then in addition to displaying a symbol at the location of the object, an identifier string will also be displayed.

 

 

7. The Cache

 

When a user requests an image from a survey whose source files are on another computer, the application first downloads source files onto the local computer and then resamples the data.  By default the downloaded files are copied into the current directory and are left there when the application is complete.  They serve as a cache for subsequent requests.   Often users make several requests near a given location, perhaps changing the pixel size, resampling method or some aspect of the graphic overlay.  When these subsequent requests are made, the application notes that the needed files are already downloaded and immediately begins resampling.

 

If a user is sure that a survey file is not going to be needed again, or if space is at a premium, the user can request that the survey files be deleted by specifying the PurgeCache setting.  This will delete all files downloaded for a given survey in the current request.  However if multiple files are needed for a given survey, then all files for the survey will be downloaded and the output image produced before the downloaded files are deleted.

 

While the advantages of keeping the survey files available can be substantial, having them in the current directory can be a nuisance if more than a few requests are made.  In this case the user will likely wish to specify a cache directory to save these files, so they do not obscure other files in the current directory.  A cache may be specified in the settings file or in each command using the Cache setting to specify one or more cache directories.

 

Users who create their own survey data sets (or who modify the XML survey files distributed with the application) can create different cache directories for each survey.  By setting the cache directory in the <Settings> area of the survey description the user specifies a default cache directory which can be overridden on the command line.  If the user specifies a cache in the <Images> area, then this setting cannot be overridden by the user.  Note that more than one cache directory can be specified.  All of the cache directories are searched when looking to see if a file is already available, however any files that are downloaded will be placed in the first directory.

 

7.1 Caching the Digitized Sky Survey.

 

The Digitized Sky Survey data is handled somewhat differently than most surveys.  Rather than trying to download entire plates of data, which would take forever even if it were permitted by the STScI servers, SkyView requests small tiles from these images.  It then builds the image the user requests by mosaicking the tiles together.  The tiles are cached like any other survey.  The headers for each tile indicate the original image whence it was derived.

 

 

8. Dependencies on Remote Systems

 

While all data processing occurs in the local system, the SkyView-in-a-Jar application may fail if remote resources are not available.  The underlying data for a few surveys are included inside the JAR file (e.g., 408 MHz and EGRET).  These are generally low resolution surveys.  The application will fail if it is unable to retrieve data for other surveys.  For many surveys that data is fetched from the SkyView server at skyview.gsfc.nasa.gov. However the DSS, DSS2 and FIRST surveys link to archives at the Space Telescope Science Institute.  The 2MASS and NEAT surveys use archives at JPL.  A number of other surveys, e.g., the SDSS, now use the VO Simple Image Access protocol, and link to URLs described therein.  The <Images> area of the survey description will generally indicate the dependencies.

 

Catalog queries normally involve links to remote catalog sites and will fail if those sites are unavailable. 

 

The only other remote dependency for the application is in the resolution of target names.  If the user specifies a target rather than a pair of coordinates, the name-resolver/coordinate converter service at the HEASARC is used to resolve this name into coordinates.  The HEASARC service in turn calls the SIMBAD and NED name resolvers.  The HEASARC service may also be used if the user enters the coordinates using characters other than numerics, signs, colons, commas and decimal points (e.g,

10h30m10.3s). 

 

9. Compatibility with the SkyView Batch Interface

 

For the convenience of users of the old SkyView batch interface, the SkyView-in-a-Jar application supports most of the arguments of that interface insofar as they related to the generation of FITS files. The following are supported as input settings:

 

SkyView Batch keyword

SkyView-in-a-Jar setting

VCOORD

Position

SCOORD

Coordinates

MAPROJ

Projection

PIXELX,PIXELY

If both specified, converted to Pixels

EQUINX

Equinox

SFACTR

Size

COLTAB

LUT

ISCALE

Scaling

file

Output

 

The values are also converted appropriately.  E.g., MAPROJ=Gnomonic is translated to Projection=Tan.

 

Generally image results from the new interface should be very similar to the old SkyView interface.  There is one major exception.  The IDL code which the old interface used ran almost entirely in single precision arithmetic.  The Java classes use double precision arithmetic exclusively with an option for converting the final result to single precision. Users must specify the Float setting to get single precision results.  The difference in precision and other small differences in the algorithms mean that there can be small differences in the values returned.  E.g., when using a near-neighbor resampling, positions near the edge of one pixel might move onto an adjacent pixel.

 

The format of the FITS headers differs in considerably between the two versions.  Substantially more information about the processing is included in the new version including the files that were used in generating the output mosaic, all settings used in the command, and outputs from various processors.

10. SkyView-in-a-Jar Operations.

 

In this section we describe how the SkyView Java application works.  This includes short descriptions of some of the algorithms used.  

 

The basic goal of SkyView is relatively simple.  Given a set of one or more existing images comprising a survey, SkyView generates a new image in a user-specified geometry.  The output user image may overlap one or more of the survey images, and may overlap different images in different locations.  This section discusses how the application accomplishes this task. 

10.1 Getting defaults and user arguments.

When the application starts, a set of default parameters is established.  The java.executive.Settings class keeps track of the current parameters of the request.  The default parameters are read in from a configuration file or system resource.  User arguments are then processed and override any defaults.  System parameters are stored in a HashMap but both the put and get methods convert the key to lower case, so that the keys are not case sensitive.

10.2 Updating Settings.

One or more classes may wish to look at the settings before processing begins to fill in further defaults, to allow synonyms or for other reasons.  These classes should be specified in the SettingsUpdaters setting.  Currently the BatchCompability class allows syntax similar to the old Batch interface while the IJProcessor fills in missing settings when graphics have been requested.

10,3 Locating surveys.

Once the user arguments have been parsed, the application looks for surveys.  Using the default SurveyFinder there are four possible sources of surveys:

·        A survey manifest resource.  This manifest gives the names of other resources or XML files where each entry defines one survey.  In this initial pass the XML is only parsed far enough to find the names of the surveys.  Surveys can have several names.  When the system tries to find the surveys the user is looking for the match will be done in a case-insensitive fashion.  The source manifest is where the default surveys are normally found when the executable JAR file is used.

·        The SourceXML settings.  This may be set on the command line, or if the user sets the environment variable SKYVIEW_SOURCEXML then the value of that variable is used. When the SourceXML is set, that directory is searched and each file of type .xml is treated as a survey definition file.  Users can use this method to add a set of surveys to their SkyView implementation.  

·        The SurveyXML setting.  Each field in the SurveyXML is assumed to be a Survey definition file and is parsed.

·        The ‘User’ survey created by the SurveyFile setting.  Each field of the SurveyFile is treated as a survey image file.  Users can generate an image from this survey by requesting an image from the survey User.

 

10.4 Looping over the surveys.

Once the set of available surveys has been determined, the application loops over the surveys the user has requested.  If a survey is not found, then an error is signaled and processing continues with the next survey.

When a survey is found, the corresponding survey definition file is parsed.  Part of the survey definition file is a set of survey default settings, e.g., the default scale for the image, or whether the image will be de-edged.  These settings do not override settings already made, so that survey settings do not supersede anything the user specified.

10.5 Getting candidate survey images

The first step in processing a survey is determining a list of candidate survey images that may be used in generating the output image.  If a survey uses the VO Simple Image Access Protocol (SIAP, see http://us-vo.org), then the candidates are all images returned in the initial SIAP request.  If the survey explicitly lists the images, then the application passes through these images and uses the ImageSize characteristic to find images that might overlap part of the user’s image.  If this is a request for the User survey, then all of the images the user specified are candidates.

10.6 Building images.

At this point, a survey image is only a candidate for processing.  Often there will be many more candidates than will actually be used.  If the images are present locally this is not an issue, but it may seriously delay processing if we download a number of images that are not actually needed.  To ensure that only needed data are retrieved from remote sites, remote images are normally represented initially as proxy images.  Proxy images have the same geometry (or a very good approximation) as the real image, but the image data is not filled in.  If and when the application decides to sample the data in the image, it downloads the remote data and converts the proxy into a real image.

The SIA protocol allows for fields that define the geometry of the images returned which makes it easy to build proxies without actually retrieving data.  For other surveys we find that the geometry of the images is largely constant with only the center position changing.  This allows us to build ‘spells’ to enable the creation of proxy images and their transformation to real images. 

The Survey XML descriptor file controls this process.  It can specify either an explicit list of image files or spells that define the content of the survey, or an ImageGenerator class that will create this list dynamically.  E.g., if the IRAS100 survey uses local data, then its survey XML file lists 430 files explicitly.  In the SkyView-in-a-Jar distribution, the IRAS data is retrieved over the net, so the survey XML file includes sufficient information to generate proxy images for the survey.  These proxies are used to find out which images are to be resampled and which pixels in the output image are to be resampled from which input image.  Only when we actually begin resampling does the program actually download the data.  

The SDSS-I survey uses an ImageGenerator class that in turn uses the SIA protocol to find candidate images around the user’s requested location.  The SIA call returns sufficient information to generate proxy images so we only actually retrieve the SDSS images we are going to use.

An ImageFactory class is also defined in the survey XML file. It shows how to convert the file name/spell String into an actual image object.  The FitsImageFactory just reads in a FITS file, but the CachingImageFactory first looks to see if files are present in the cache, and if not creates a proxy image as described above.

Users can create their own ImageGenerator’s and ImageFactory’s to meet special requirements for new surveys they might want to include.

10.7 Finding the best survey image for each output pixel.

 

 

Once the set of candidate images is determined, the application determines the image to sample for each pixel in the output image. 

This source image finding uses an object of the ImageFinder class.  The algorithm used by the default RecursiveImageFinder is described here.  The finder first looks at the corners of the output image and the positions of these four corners of the output image are transformed to the frame for each of the candidate input images.  The best candidate image for each of the four pixels is then determined.  The ‘best’ input image is where the output pixel location is located in the image but furthest from the edge of the input image.

E.g., consider a corner in the output image and assume there are three candidate images each of which is a 500x500 image.  The corner position is transformed into the pixel coordinates of the three candidate images.  If the pixel coordinates corresponding to the output pixel in three candidate inputs are (-80, 300), (40,40) and (250,35), then the second image is the best.  The output pixel location is not within the first image, and is only 35 pixels from the edge of the third, but 40 pixels from the edge of the second. 

If all four corners match the same input image, then that image is assumed to be the best image for all of the interior pixels.  If different images are best (or perhaps there is no corresponding image for one of the pixels), then the rectangle is broken up into four sub-rectangles and the process is repeated recursively until we have either checked each pixel individually, or we get common input images for each corner of some sub-rectangle.

In the case where the output image is comparable to or larger than the input images, the initial checking will be done on rectangles no larger than half the size of the input images. When the input image selection splits into sub-rectangles, any candidate image that was not found on any of the four corners is not passed down to the next level of the recursion. 

 

Consider generating an all sky map from the IRAS100 survey which has 430 12x12 degree square images.  The application first breaks the output image into 6x6 degree squares and checks each of these rectangles against all 430 images.  In most cases it will not find the same best image for all four corners of the 6x6 degree squares, so it next looks at 3x3 degree squares.  However, in this second step it only looks at candidate images on which at least one of the four pixels at the previous step was on the image (though not necessarily the best image).  So in this second stage we are typically looking at only three or four candidates making the search much more efficient.

The end-point of this process is an integer array with the same dimensions as the user output image but where the value at each pixel is the best input image.  Special flag values are used to indicate that this output pixel is not matched by any survey image.  An output pixel may also be outside of the standard projection coverage, e.g., outside the oval of an all-sky Aitoff projection, or the circle defined in the Orthographic (SIN) projection.

In many cases the user may have selected images from a set of surveys which have identical or near identical geometry (e.g., IRAS100 and IRAS25, or 2MASS-J and 2MASS-K).  These are called ‘geometry twins’ and are identified in the survey descriptor files.  If two geometry twins are processed in succession, then the calculation of the best image from the first twin processed is re-used for the later twin (or triplets, or …).

In some cases it may be necessary to find the best image for every pixel in an image.  This can be requested by setting the StrictGeometry setting.  This tends to occur occur when input images have peculiar geometries and small overlap between adjacent images.

10.8 Preprocessing

At this point the program allows for image pre-processing.  The user can define a class (or classes) that can modify the data that are used for sampling.  A preprocessor class has access to the input images, the index associating images with output pixels and the output image.  Only the geometry of the output image has been specified, the data so far are nil.  Preprocessor can filter the input images however.

No real preprocessing classes are currently provided, but a preprocessor is used if catalog queries are to be done.  The preprocessor initiates these queries so that they can run in parallel with the image generation process.  As soon as the first image is ready for the catalog results, the process will wait for the query threads to complete if they have not already.  Often the catalog queries will finish before image calculations.

10.9 Validating the image.

Up to this point in the processing of the image, only the coordinate system of the input image has been needed.  To avoid downloading images that may not actually be used, remote images use proxy images that do not have data values associated.   Even if the image is available locally, only the FITS header is read rather than the data values if this can be done efficiently.  As we begin to process each survey image, the image is validated, i.e., we download the image if it is a remote URL and then populate the actual input data array.

While the proxy image should have a geometry very similar to the real image, it may not be quite identical.  If a request for a region is repeated and the survey images were cached on the first request, then the actual image geometry will be used at all stages in the second request which may result is slightly different boundaries between the regions sampled from the various input images.

10.10 Resampling setup.

Once the ‘best’ image for each output pixel has been selected, and all of the input data has been downloaded, the program begins to process each input image.  For some samplers there is some processing of the input image done before any resampling occurs.  E.g., the spline resampler needs to calculate the spline coefficients to be used.  In this case if the input image is much larger than the output image, then spline coefficients may only be computed on a sub-region of the input.  The minimum and maximum x and y values for the output image that are being sampled in this input image are computed.  These bounds are projected into the input coordinate system and the spline is computed only for this sub-region.  Currently this behavior is enabled only for tiled surveys.

10.11 Sampling in energy.

If the input survey has depth in the energy dimension, the data is first sampled in energy according to the Ebin’s specification before spatial sampling.  A default specification is normally included in the survey default to produce a 2-d output image.  Thus unless the user specifies that they want a 3-d output image, they get a 2-d image.

If there is no energy dimension to the data, this step is skipped.

10.12 Sampling in space.

The program then loops over the output pixels associated with the current input image and computes the value using the selected resampler.  For most resamplers the center of the output pixel is projected into the plane of the input image.  For the clipping sampler, the corners of the pixel are projected.  The output pixel is then populated with the resampled value according to the requested algorithm.  If the output image has depth in the energy dimension, the sampling is performed at each output energy channel.

Currently the spline sampler cannot process 3-d images.

10.13 Post-Processing

After the data have been resampled, post-processing may be performed.  The interface for post-processing is exactly the same as for pre-processing, however at this point the output image has been populated and we would anticipate that most post-processing will perform some filtering of the output image while preprocessing is expected to primarily affect the input image or images.  Examples of  post-processor are de-edging and graphics processing

10.13.1 De-edging

Since survey data may have different background values in the different survey images, the edges of the image may show up as an unsightly artifact.  If the user wishes to minimize the edges, then a de-edger may be run on the output image.  .

The algorithm used by the default deedger attempts to minimize discontinuities along borders.  The output image is divided into regions defined by the input image from which the pixels were sampled.  These regions are placed into adjusted and unadjusted sets.  The region with maximum number of pixels is placed in the adjusted set and all other images are initially unadjusted.  The program then looks for the longest border between an adjusted and unadjusted region.  Borders are defined by looking at each pixel and seeing if any of the four neighboring pixels is from another region (diagonals are not considered).  The median of the shifts as we move across the border from the adjusted to the unadjusted regions is computed. An offset is applied to the unadjusted region to set the median shift across the border to 0.  The unadjusted region is moved into the adjusted set, and the process iterates until all regions have been adjusted.  Other classes use somewhat different algorithms to compute the offsets for images: use the average rather than the median shift at the border, adjusting the medians of all pixels that come from an image, not just the boundary pixels.

The FITS headers include information on the adjustments made when de-edging is applied.

10.13.2 Graphics Processing

All graphics outputs are also produced in post-processing.  Appropriate functions in the ImageJ library are called to produce the desired data.  Graphics procdssing does not affect the FITS data.  If Min or Max settings were given, then any values below or above the limits are replaced by the limit value.  The data is then scaled using the appropriate scaling method.  Logarithmic (Log) and Square-root (Sqrt) scaling scale negative values to 0 (or the minimum positive value for log).  The result are images with values ranging from 0 to 255.  These data are then converted to the requested graphics output.

 

If users request an RGB image from three input surveys, then each of these surveys is processed in turn and the scaled results are stored until the last survey is processed.  Then the three images are combined to create a single RGB JPEG file.

10.14 Data conversion and writing the FITS file.

All of the internal computations are done with the data as a one-dimensional double array of values.  Just before writing the data out, the application checks to see if the user wished to have only 4-byte reals (float’s in C or Java-speak) rather than the default 8-byte data (double’s).  If so the data is converted.  The program then ‘curls’ the data into the appropriate dimensionality and uses the nom.tam.fits library to write the FITS image.   The survey descriptor defines survey-dependent information that is included in the FITS header.  The various processors used in generating the image may also write history records into the header.

The user can elect not to create a FITS image by specifying the NoFITS setting.

10.15 Cache cleanup.

If a user requested data that was not present in local files, this data is first downloaded into the first cache directory (the current directory by default).  By default downloaded survey files are left for possible use in subsequent requests, but if PurgeCache has been set, then these files are deleted as the processing for each survey is completed.  However only the files actually downloaded during this request will be purged.  Files already present in the cache will be left untouched.


11. Survey Definition File

Each survey (other than the ‘User’ survey) used in SkyView is defined by a survey definition file.  This is an XML file that pulls together all information needed to use the survey.  This section describes the structure of the file.  [Note: This assumes the default SurveyFinder object.  A user can define SurveyFinder object with behavior that they specify.]

There is no formal schema for the survey file format but we suggest the following order for the top elements of the file.

Survey

ShortName

Name

Description

Settings

MetaTable

OnlineText

FITS

Images

 

11.1 ShortName

The short name element gives one or more short names for the survey.  Short names are used as the values of the Survey keyword in the SkyView-in-a-Jar application and therefore it is recommended that they not contain embedded blanks.  If multiple short names are to be specified they are separated by commas.  All survey description files are read at program startup until the short name is found, so putting this first can speed up the program a little.

Examples:

<ShortName>DSS2R</ShortName>

<ShortName>EGREThard,EGRET100-10000</ShortName>

 

11.2 Name

The name element gives a longer name for the survey generally spelling out acronyms and such that may have been used in the short name. 

Example:

<Name>Two Micron All-Sky Survey: K band </Name>

11,3 Description

This gives a text description of the survey.  This may include HTML and if so should be coded as a CDATA element.

 

 

Example:

 

<Description>

<![CDATA[

This LISA Galactic survey is a low-resolution image

of the gravitational background in    

the Galactic plane as described in <a href=someurl>Lisa

Galactic Plane Handbook    

</a>

  ]]>

</Description>

11.4 Settings

The settings element is used to specify survey specific settings.  Within the settings element there may be 0 or more sub-elements of the form

<key>value</key>

When processing of a specific survey begins, the Settings element is parsed and the Settings hash is updated with any key/value pairs found.  The case of the key is not distinguished but case may be important in the value.  Note that XML parsers require that the opening and closing case of the keys be consistent within the file.

Common survey specific settings include the default image scale (Scale), other surveys that have the same geometry (GeometryTwin), the default energy binning for 3-d surveys (EBins), and post-processing options, notably the de-edger.

Survey settings will not override existing user specified settings. 

 

<Settings>

<Scale> 0.003 </Scale>

<GeometryTwin>SurveyA,SurveyB,SurveyC</GeometryTwin>

<Deedger> skyview.geometry.Deedger </Deedger>

</Settings>

 

 

Since the survey settings do not override user specified settings, the default de-edging of this survey will be turned off when the user specifies Deedger=null.

11.5 MetaTable

The metatable element includes a set of metadata elements that give an overall description of the survey.  SkyView attempts to give some common information about all surveys, energy range, resolution, … and these are grouped in the metatable.

The metatable information is normally included in any FITS file produced and elements may be displayed in other interfaces.

Example:

 

<MetaTable>

  <Provenance>

    NASA IPAC/Jet Propulsion Laboratory

  </Provenance>

    <Copyright>   Public Domain  </Copyright>

    <Regime>      Infrared       </Regime>

    <NSurvey>     4              </NSurvey>

    <Frequency>   3-30 THz       </Frequency>

    <Coverage>    All-sky        </Coverage>

    <Scale>       0.025 deg/pix  </Scale>

    <Units>       MJy/sr         </Units>

    <Resolution>  2’             </Resolution>

    <Coordinates> Equatorial     </Coordinates>

 <Projection>  Gnomonic(TAN)  </Projection>

    <Equinox>     1950           </Equinox>

    <Epoch>       1983           </Epoch>

    <Reference>

      <![CDATA[

       Wheelock, et al., 1991, IRAS Sky Survey Atlas Explanatory

       Supplement.

       ]]>

</Reference>

</MetaTable>

 

11.6 OnlineText

The online text element is used to include text that is to be shown on Web pages associated with a generated image.  The online text may include HTML tags and if so should be enclosed a CDATA element.  Variables prefixed by $ will be substituted for in the text by the appropriate values.  These include:

$ra      The right ascension of the center of the image in decimal degrees

$raH   The hours of right ascension

$raM    The minutes of right ascension

$raS     The seconds of right ascension (including fraction)

$dec     The declination of the center of the image in decimal degrees

$decD The degrees (and sign) of declination

$decM The minutes of declination

$decS  The seconds of declination (including fraction)

$sizeD  The requested size of the image in degrees.

$sizeM The requested size of the image in minutes.

$sizeS  The requested size of the image in seconds.

 

Example:

<OnlineText>

<![CDATA[

<a href=http://somewhere/cgi-bin/query?POS=$ra,$dec&SIZE=$sizeD>Link     </a> to the associated source catalog.

   ]]>

</OnlineText>

 

No software currently uses this element and no data is supplied in these elements in the survey descriptions included in the JAR file.

11.7 FITS

The FITS element includes information to be included in the FITS header generated with the image. 

Example:

<FITS>

<![CDATA[

SURVEY  = ‘IRAS 100 micron’

BUNIT   = ‘MJY/SR                    / INTENSITY                       

COMMENT   Note that the values are not adjusted for any change in pixel

COMMENT   size from the original pixel size of 90 x 90 arcseconds             

BLANK   =          -2000000000        / TAPE VALUE FOR EMPTY PIXEL      

ORIGIN  = ‘JPL-IPAC’                  /  INSTITUTION                    

TELESCOP= ‘IRAS                      /                                 

INSTRUME= ‘ISSA-FLD’                  /  IRAS SKY SURVEY ATLAS          

EDITED  =                    T        /  SCANS EDITED                   

APPCAL  =                    T        /  CALIBRATION CORRECTION 25 MICRON

DE-ZODY =                    T        /  DE-ZODIED IMAGE                

GLBL-D  =                    T        /  APPLIED GLOBAL PARAMETERS      

LOCAL-D =                    T        /  LOCAL DESTRIPER                

ASBLANK =                    T        /  ASTEROID BLANKING              

]]>

</FITS>

 

 

12. The < Images> Element 

The main business of the survey file is in the Images element.  There are two main configurations:  an explicit list of the images in the survey, or a link to a class that will generate a list of images for a given user request.   Most of the fields in the Images element (with the exception of Image elements) are parsed when survey processing begins and used to update or set entries in the Settings hash.  However, unlike the fields in the Settings element, these settings override any existing settings.

12.1 Explicit Image Lists

In this case, the Images element will contain a set of Image elements.  Each element has a text value with four blank delimited tokens: the filename or spell, a longitude and latitude, and the epoch of the image.  The coordinate system for the longitude and latitude can be specified in the “SurveyCoordinateSystem” setting and defaults to J2000.  In addition to the Image elements there should at a minimum be an ImageFactory element and an ImageSize element.  The image size gives the size in degrees of the smaller dimension  (if not square) of the images.  This is used in the process of determining the best image for each output pixel. 

The ImageFactory gives the class which given the appropriate spell or file name creates the image.  The skyview.survey.FitsImageFactory takes a filename and simply reads the file to create the input image.  If a <FileNamePrefix> element is included Images element, then this prefix is prepended to the file name in the Image element before being passed to the FitsImageFactory.

The skyview.survey.CachingImageFactory assumes that it is being given a spell that tells it both how to retrieve the data as a URL as well as sufficient metadata to create a proxy image.   The spell consists of 10 or 12 comma delimited fields.  These are:

1.      The URL to retrieve the data from

2.      The file name to store the cached file as

3.      The longitude of the center of the image

4.      The latitude of the center of the image

5.      The projection used in the image

6.      The coordinate system used in the image

7.      The width of the image in pixels

8.      The height of the image in pixels

9.      The width of a pixel in degrees

10.  The height of a pixel in degrees

..or..

9-12. The CD matrix elements for the image.

 

For projections (Aitoff, Cartesian and CSC) where the reference value is normally fixed, the third and fourth tokens are the pixel offsets for the image, while for other projections the reference pixel is assumed to be at the center of the image.

The user familiar with FITS will note the similarity of fields 3-10 (or 12) to standard FITS WCS keywords.  Note that the spell does not need to use the same coordinate system as the other fields in the Image elements.

If the SpellPrefix or SpellSuffix values are specified in the Images element, then the Image spell string is appropriately extended.  Usually much of a spell is constant for a given survey.

12.2 Image Generators

Rather than specifying each individual image, the survey file can describe an ImageGenerator class that will generate the images.  An Image generator can generate either files or spells.

There are specialized image generators for the DSS and 2MASS surveys.  Users may wish to examine these classes for extension to other special surveys, but the most useful generator is likely to be the SIAPGenerator.  The SIAP generator uses the Virtual Observatory Simple Image Access protocol and can be adapted to use many of the existing SIAP services with appropriate entries in the Images element.

The key fields are:

·        SiapURL: defines the base URL for the Simple Image Access service.

·        SiapProjection: The projection used in images returned by the service

·        SiapCoordinates: The coordinate system used in images returned by the service

·        SiapNaxis: The size (in pixels) of image returned by the service

·        SiapScaling: The size (in degrees) of the pixels returned by the service

·        SiapMaxImages: The number of images returned to include as candidates.

 

Alert readers will note the correspondence of most of the fields to the spell parameters and to the FITS WCS fields.  SIAP services can return WCS information for each image explicitly and to the extent that they do so (e.g., almost all return the size of the image in pixels), the user need not supply this information.  However many (most) SIAP services do not supply all needed information.  The fields above remedy any deficit.

 

Not all SIAP services are good candidates for immediate inclusion as a service.  SkyView will use all images returned as candidates even if the images are quite heterogeneous.  Only SIAP services that return relatively homogeneous images can be used with some kind of initial filtering.  However it is often easy to modify the SIAPGenerator to do such filtering and we anticipate that this class will have filtering capabilities added.

The third example below shows the power of the SIAPGenerator.  The entire SDSS survey is included as a survey with just a few lines of text.

 

12.3 Example <Images> elements

Listing local files:

<Images>

  <FilePrefix>

    /surveydata/iras/iras100/

  </FilePrefix>

  <ImageFactory>

    skyview.survey.FitsImageFactory

   </ImageFactory>

<ImageSize> 12.5 </ImageSize>

<Image>  i001b4h0.fits    0.   -90.   1980.00 </Image>

<Image>  i002b4h0.fits    0.   -80.   1980.00 </Image>

<Image>  i003b4h0.fits   45.   -80.   1980.00 </Image>

<Image>  i004b4h0.fits   90.   -80.   1980.00 </Image>

<Image>  i005b4h0.fits  135.   -80.   1980.00 </Image>

… 425 more Image elements …

</Images>

Listing remote URLs:

<Images>

<SpellPrefix>  http://skyview.gsfc.nasa.gov/survey/iras100/images/

</SpellPrefix>

<SpellSuffix> ,Tan,B1950,500,500,0.025,0.025      </SpellSuffix>

<ImageFactory> skyview.survey.CachingImageFactory </ImageFactory>

<ImageSize> 12.5 </ImageSize>

<Image>  i001b4h0.fits,i001b4h0.fits,0.,-90.   0.   -90.  1980. </Image>

<Image>  i002b4h0.fits,i002b4h0.fits,0.,-80.   0.   -80.  1980. </Image>

<Image>  i003b4h0.fits,i003b4h0.fits,45.,-80.  45.  -80.  1980. </Image>

<Image>  i004b4h0.fits,i004b4h0.fits,90.,-80.  90.  -80.  1980. </Image>

<Image>  i005b4h0.fits,i005b4h0.fits,135.,-80. 135. -80.  1980. </Image>

… 425 more images …

</Images>

 

 

Using an Image generator.

<Images>

  <SiapURL>

<![CDATA[

http://casjobs.sdss.org/vo/DR3SIAP/siap.asmx/getSiapInfo?FORMAT=image/fits&BANDPASS=I&

]]>

  </SiapURL>

  <SiapProjection>  Tan   </SiapProjection>

  <SiapCoordinates> J2000 </SiapCoordinates>

   

   <ImageFactory>   skyview.survey.CachingImageFactory </ImageFactory>

 

  <ImageSize>       0.25 </ImageSize>

  <ImageGenerator>  skyview.survey.SIAPGenerator</ImageGenerator>

</Images>

 

13. A Simple Survey Definition File

While a survey XML definition file can be complex, most data is optional.  A minimal survey definition need only include the short name of the survey, a default scaling, the image size and the names and locations of the included files.

 

<Survey>

  <ShortName> MySimpleSurvey </ShortName>

  <Settings>

       <Scale> .1 </Scale>

  </Settings>

  <Images>

    <ImageSize> 10 </ImageSize>

    <ImageFactory> skyview.survey.FitsImageFactory</ImageFactory>

    <Image> file1.fits  0. 0. 2000 </Image>

    <Image> file2.fits 10. 0. 2000 </Image>

    <Image> file3.fits 20. 0. 2000 </Image>

  </Image>

</Survey>

 

This describes a survey with three images along the equator (at RAs of 0, 10 and 20 degrees) which have a minimum dimension of 10 degrees and  0.1 x 0.1 degree pixels.

 

14. Contents of the JAR file

 

The SkyView JAR file contains files from a number of directories as well as indidual files used by the program.  This section discusses the content of the JAR for users who may wish to create their own customized versions of the JAR.  You can view the contents of the JAR file using the command  jar -tf skyview.jar.

 

14.1 The Class Files

 

Class files are found in three directory trees:

 

  • skyview/… contains the class files for the class files developed in SkyView.  This includes all the code relating to geometric transformations, image surveys and the main program for the JAR file.

 

  • nom/  contains the methods for reading and writing FITS files and utility I/O functions.

 

  • ij/… contains the classes and a few data files) for ImageJ.

 

14.2 Survey Data and Metadata

 

The JAR file contains metadata descriptions for many surveys and actual survey data for a few.  Metadata descriptions in the format described above are found in surveys/xml.  FITS data for a number of surveys is found in surveys/data.

 

The survey manifest file gives a list of all of the surveys included in the JAR and is found in surveys/survey.manifest.

 

14.3 Control Files

 

A number of control and miscellaneous files are found in the JAR.

 

  • The skyview.settings files contains the default settings for the program.

 

  • IJProps.txt contains settings used by ImageJ.

 

  • The MANIFEST/MANIFEST.MF file is included in the JAR using a command like jar fm skyview.jar manifest.fil  (the JAR command renames the file when it is inserted).  It contains a single keyword value pair that gives the class which is to be used when the JAR is executed (currently skyview.executive.Imager).

 

  • The onlineinfo.txt file contains help text which is displayed when the program is invoked without any arguments.

 

 

14.4 Source code

 

Source code is found in the source/… hierarchy for all Java classes defined in the JAR file.