SkyView Installation and Operations Guide

Introduction

This document describes the installation and operations of the SkyView Virtual Telescope facility. While this document is applicable specifically to the installation at NASA's HEASARC facility, it will also be useful in setting up this service at other locations.

Prerequisites

The operations scripts assume a Unix/Linux-like system supporting both Perl and Java. A Java version of at least 1.6 is expected. All of the actual image processing code is run in Java. In principle, a Windows installation of SkyView should be feasible, but some significant modification of the runtime scripts would be needed.

A Web server with CGI capabilities is assumed. Extensive use of .htaccess files is made to support redirection and to preclude external access to some directories. There is also some use of server side includes of standard files. The Apache HTTPD server is used in the HEASARC implementation of SkyView. However these Apache specific capabilities are not critical to the primary functionality of SkyView.

Organization

The SkyView code is installed as a single directory hierarchy within the Web environment of the server. E.g., the base directory might be /www/htdocs/skyview . All of the installed code runs within this directory and with the exception of specific system defined locations (discussed below) all SkyView code and data resides in these locations.

The directory tree looks like:

/www/htdocs/skyview
Top level of SkyView organization.
cgi
Primary directory for CGI scripts. This is where all user functionality is initiated.
cgifiles
Headers, footers and settings files used in standard SkyView operations.
data
Data used in building SkyView query forms: HEASARC table list, 3D surveys.
lib
Perl includes for SkyView CGI scripts including the SkyView_Env.pm which is where we define the locations used in the current installation and SurveyList.pm where we put in the list of surveys supported by SkyView.
scripts
Scripts which are invoked by the main CGI scripts.
vo
CGI script and associated data for the SkyView SIA service.
docsDocumentation of SkyView on- and off-line services.
helpExamples and FAQs for SkyView usage.
imagesImages used in SkyView web pages and example SkyView images.
includesCSS and server-side include HTML.
jarThe SkyView JAR files that are used in running requests.
javascript JavaScript files that are used in the SkyView web pages.
ops Operations scripts used in the routine SkyView operations. E.g., all cron jobs use scripts under this directory.
/www/htdocs/data Directory in which user files are to be created. This is defined in the runtime.settings file.
/www/htdocs/userimages Directory in which user gallery images and HTML is to be saved. This is defined in the the cgi/cgifiles/lib/SkyView_Env.pm file.

Installation

SkyView is checked in as a project in the HEASARC's SVN repository at file:///software/jira/svn/skyview . The user should check out the full SkyView hierarchy. We assume the user has a checked out copy of the full project under /proj/checkout/skyview. Our preferred approach has been to treat this as NetBeans project, but users should be able to build SkyView directly without using NetBeans and here we show what NetBeans does under the hood.

This installation process assumes that all customizations of the SkyView code have either been done, or will be performed after installation of SkyView into the appropriate directory. The kinds of customization that might be performed are discussed later in this document.

Dependencies

SkyView references four other Java libraries during the build process. A version of these libraries is included in the package, but more recent versions may be available. The libraries are:

Building the SkyView jar

The SkyView jar file contains source code, Java class files and data. To build the JAR the following steps are required.
  1. Create the build directory from which we'll generate the JAR
                        cd /proj/checkout/skyview
                        rm -rf build
                        mkdir build
                    
  2. Fill the build directory with the content of the JAR dependencies. We do this first, so that when we compile our local classes later it will overwrite the standard classes and we'll get our customized versions of a few ImageJ classes.
                        cd build
                        jar xf ../source/lib/ij_138k.jar
                        jar xf ../source/lib/jsamp-1.2.jar
                        jar xf ../source/lib/fits.jar      
                    
    Note that we do not include the JUnit classes in the Jar since they are only used for testing and not operationally. If you wish to run the test classes a Junit Jar should be added to the class path in some fashion.
  3. Compile all supplied Java files. You have to go six directories deep. Sigh.
                       cd /proj/checkout/skyview/source/java
                       javac -cp .:../../build:../lib/* -d ../../build  */*.java */*/*.java */*/*/*.java */*/*/*/*.java */*/*/*/*/*.java */*/*/*/*/*/*.java
                    
  4. Add in the data used in the Jar
                       cd /proj/checkout/skyview/data
                       cp -r * ../build                   
                    
  5. Create and index the jar. Note the manifest file is needed to make this an executable jar file.
                       cd /proj/checkout/skyview/build
                       jar cmf skyview.manifest ../skyvew.jar .
                       jar i ../skyview.jar 
                    
Several of these steps can be done more simply using Ant commands but are given explicitly here. E.g., Ant can combine the classes in the library Jar files without expanding them into the build area, or compile an entire directory hierarchy of arbitrary depth.

Building the JavaDocs.

If a copy of the javadocs for the SkyView classes is desired it can be easily created.
                 cd /proj/checkout/skyview
                 rm -rf javadoc
                 mkdir javadoc
                 cd source/java
                 javadoc -d ../../javadoc -subpackages ij -subpackages net -subpackages org -subpackages skyview
        

Installing the files

At this point all of the files needed are ready to be moved into the server. We assume that SkyView is to be installed at /www/htdocs/skyview on the same machine.
             cd /proj/checkout/skyview
             rm -rf /www/htdocs/skyview
             cp -r web /www/htdocs/skyview
             cp -r javadoc /www/htdocs/skyview/javadoc
             cp -skyview.jar /www/htdocs/skyview/jar/skyview.jar
        
While we could have copied the javadocs and jar into the web directory before we did the copy into the web directory we did these separately so that we didn't pollute the directories that we checked out from our SVN repository.

Customization

A number of features are available for customizing SkyView to a particular installation.

The SkyView_Env.pm file

The file SkyView_Env.pm is where the location of key elements of the SkyView interface is specified. It specifies a set of environment variables that are used by scripts. These include:
WWW_ROOT
The base of the HTDOCS tree for the web server, e.g., /www/htdocs. Note that we do not include the terminal '/'.
DATA_ROOT
The directory in which SkyView FITS and quicklook images will be created.
SKY_CACHE
The directory in which SkyView cache files will be placed.
PATH
The search path for executables to be used in looking for scripts.
WWWDIR_CANON
This is string which gives the canonicalized beginning of the path to files that are to be moved into the gallery. It is used in a security check to ensure that only appropriate files are being moved.
GALLERY_HTTP
The directory (relative to WWWDIR) in which gallery images are to be placed. It should not include a final '/'.
URL_TO_FILE
A translation between survey files described as URLs and as local file names.
BZIP_DECOMPRESSOR
A full path to a bzip2 decompression utility. This is used for SDSS data.
JARDIR
A full path to the SkyView jar file. E.g., in our example this would be /www/htdocs/skyview/jar/skyview.jar
SITE_nnn
The links to external sites may be collected here if desired so that they need be specified in only a single location.
SKYVIEW_BLOG
A URL to the SkyView blog.
CHECK_DIR
The full directory into which the external surveys check puts output.

SkyView settings files

The SkyView jar uses settings files to specify standard parameters. These settings are described in detail in the SkyView User's Guide but we highlight a few that are particularly relevant to the installation of a SkyView web service. The skyview.settings file found is bundled inside the jar. In we check out the SkyView project, it is found as /proj/checkout/skyview/data/skyview.settings. The other main settings file is the runtime settings which includes setting specifically for the Web service usage of SkyView. It is found in the distribution in /proj/checkout/skyview/web/cgi/cgifiles/runtime.settings. This ends up in the /www/htdocs/skyview/cgi/cgifiles directory when installed.

The skyview.settings specifies the URLs that are used to access external name resolver services at NED and SIMBAD. It also sets up the default values for the various geometry choices: projection, coordinates system and so forth.

The runtime.settings file includes a variety of settings that point to where information is to be placed. Several settings are of particular interest. Most of these settings inherit environment variables from the SkyView_Env.pm file discussed above.

OutputRoot
Gives the location where output files (generated images) are to be created. Generally it should be given as relative path to the script generating the image, since that conveniently allows both URLs and files to be generated.
Cache
Gives the location of a directory in which files that are retrieved remotely will be cached. This can add up to terabytes of data, but the cache may be trimmed at any time. Any data not found will be retrieved from the remote site.

For web services we recommend also including the SaveBySurvey setting. That splits up the cache so that each survey is saved in a separate subdirectory. If the web service runs in a chrooted environment then the cache must be in the chrooted area. However it does not need to be web-addressable.

WebRootPath
Indicates the root that needs to be added to web addresses to get file name addresses. E.g., if we see a URL http://host/a/b/c.html then we would prepend the value of this setting to a/b/c.html to get the path to the file on the web server.
LocalURL
This setting describes the translation between URLs and file locations for URLs used in survey descriptions. It should have two comma separated entries. The first is a URL prefix that will be found in the description of local data. The second is a file name prefix. When SkyView is about to retrieve a survey source image, it looks to see if the address is a URL. If so it checks to see if the URL begins with the URL prefix given in this setting. In that case, SkyView tries replacing the URL prefix with the file prefix to see if the resulting file is present on the local file system. If so that file is used.

The end result is that we can use exactly the same survey description for SkyView's web service and for distributed access to SkyView using the SkyView-in-a-Jar. When SkyView runs in its web service, the translation succeeds and the local file is used. When SkyView runs on some user machine, the translation fails and the data is accessed through its URL. In that case the user's cache will be checked before the retrieval is actually attempted.

There are many other settings which describe the location of various files and directories needed for processing. However most of these are given relative to the CGI location and need not be updated.

Standard headers and footers

A variety of standard headers and footers are to be found in the includes directory. The standard versions are appropriate for the HEASARC installation but should be carefully reviewed for other sites.

Adding surveys

There are three required elements to adding a survey to SkyView: providing access to the underlying survey data, creating a survey description file, and updating the SkyView web site. There is one additional step if the survey is three-dimensional, e.g., has energy resolution within each position pixel.

Providing access to the data

For many surveys SkyView uses a local copy of the data. This data must be in an area that is accessible to the CGI processes running in the Web server. I.e., if the web server runs in a chrooted environment, the survey data must be visible in that environment. SkyView generally requires that survey data be simple image FITS data with all data in the primary FITS extension. In some cases data must be transformed in some ways to meet the SkyView requirements. For example, data saved as several IMAGE extension might need to be broken out into multiple files.

Once the data are in the appropriate format, all data for a given survey are typically collected into a single directory. We generally provide a soft link from a web-addressable surveys directory to this new directory. E.g.,

              ln -s /dir/NEW_SURVEY/data /www/htdocs/surveys/NEW_SURVEY
        
Each new survey will have a link from our standard surveys directory. Different surveys may be in very different locations on our file system, but we can easily find them through the soft links.

This allows data to be accessed either as files -- which is how the local SkyView server will see it, or as a URL. This is needed if we want users to be able to access the data directly through SkyView-in-a-Jar. If we do not wish public access to the data except through our SkyView web service, then this link should not be made.

Creating the survey description

The survey description file is discussed in detail in the SkyView User's Guide . This file includes the precise statement of how users are to find the data. If data are not being stored locally, then this includes information on how the data will be found on the web. Typically this involves use of the VO Simple Image Access protocol. However it is possible to directly link to a set of remote files.

Updating the SkyView Web Site

There are three changes required to publish the new survey in the SkyView web site.
  1. The survey description file needs to be added to the SkyView jar file. This can be done by updating the jar directly but in the long run it is necessary to add the new file to the standard area ( /proj/checkout/skyview/data/surveys/xml and build a new Jar. The addition should be committed to the SVN repository.
  2. The survey manifest (surveys/survey.manifest) must be updated to reflect the new survey description file. As with the description file this can be done intially as an update to the jar but ultimately needs to be changed and committed in the base SkyView code.
  3. The survey needs to be added to web interface. The file cgi/cgifiles/SurveyList.pm needs to be updated to include the new survey. Much of the time this is just adding a single line in one of the survey categories. However we may wish to split an existing category or create a new category of surveys. The relevant portion of this file looks like ...['CategoryName', ['Survey1', 'Survey2', 'Survey3', ..]], ... where the category names are typical the wavelength regimes of for the survey but can in principle be arbitrary. The SurveyN are the names of the survey. Any of a survey's aliases may be used. When users are selecting surveys they see selection boxes labeled with the category names. A special category name of 'skip' is used to skip to start a new line of selection boxes.

    This change should be committed.

  4. If the new survey is a three-dimensional survey, then a new entry will be needed in the cgi/cgifiles/data/3dtable.txt file.

Operations

There are a number of routine operations in managing the SkyView web site. These activities use scripts in the ops area which is not visible to external users.

Cleaning up user data

Updating the list of HEASARC tables

Checking external data sources

Cron jobs

Web access

Access to the SkyView web site is controlled through .htaccess files in a number of directories. In general directories other than those in the cgi and ops areas are accessible to the public. The CGI scripts in the cgi area are available to be executed but may not be viewed directly and no user access is available to any subdirectory of cgi.

These .htaccess files also allow us to redirect users. This enables use to support old address with redirections to new locations.