SkyView Installation and Operations Guide
Introduction
This document describes the installation and operations of the SkyView Virtual Telescope
facility. While this document is applicable specifically to the installation
at NASA's HEASARC facility, it will also be useful in setting up this service
at other locations.
Prerequisites
The operations scripts assume a Unix/Linux-like system supporting both
Perl and Java. A Java version of at least 1.6 is expected. All of the
actual image processing code is run in Java. In principle, a Windows installation
of SkyView should be feasible, but some significant modification of the runtime
scripts would be needed.
A Web server with CGI capabilities is assumed. Extensive use of .htaccess files
is made to support redirection and to preclude external access to some directories.
There is also some use of server side includes of standard
files. The Apache HTTPD server is used in the HEASARC implementation of SkyView.
However these Apache specific capabilities are not critical to the primary functionality
of SkyView.
Organization
The SkyView code is installed as a single directory hierarchy within the Web
environment of the server. E.g., the base directory might be /www/htdocs/skyview .
All of the installed code runs within this directory and with the exception of specific
system defined locations (discussed below) all SkyView code and data resides in these locations.
The directory tree looks like:
/www/htdocs/skyview- Top level of SkyView organization.
cgi- Primary directory for CGI scripts. This is where
all user functionality is initiated.
cgifiles- Headers, footers and settings files used in standard SkyView operations.
data- Data used in building SkyView query forms: HEASARC table list, 3D surveys.
lib- Perl includes for SkyView CGI scripts including the
SkyView_Env.pm
which is where we define the locations used in the current installation and SurveyList.pm
where we put in the list of surveys supported by SkyView.
scripts- Scripts which are invoked by the main CGI scripts.
vo- CGI script and associated data for the SkyView SIA service.
docsDocumentation of SkyView on- and off-line services.
helpExamples and FAQs for SkyView usage.
imagesImages used in SkyView web pages and example SkyView images.
includesCSS and server-side include HTML.
jarThe SkyView JAR files that are used in running
requests.
javascript JavaScript files that are used in the SkyView web pages.
ops Operations scripts used in the routine SkyView operations.
E.g., all cron jobs use scripts under this directory.
/www/htdocs/data Directory in which user files are to be created. This is
defined in the runtime.settings file.
/www/htdocs/userimages Directory in which user gallery images and HTML
is to be saved. This is defined in the the cgi/cgifiles/lib/SkyView_Env.pm file.
Installation
SkyView is checked in as a project in the HEASARC's SVN repository at
file:///software/jira/svn/skyview . The user should check out the
full SkyView hierarchy. We assume the user has a checked out copy of the
full project under /proj/checkout/skyview.
Our preferred approach has been to treat this as NetBeans project, but
users should be able to build SkyView directly without using NetBeans and here
we show what NetBeans does under the hood.
This installation process assumes that all customizations of the SkyView code have either
been done, or will be performed after installation of SkyView into the appropriate
directory. The kinds of customization that might be performed are discussed
later in this document.
Dependencies
SkyView references four other Java libraries during the build process. A version of these
libraries is included in the package, but more recent versions may
be available. The libraries are:
- The nom.tam Java FITS library. See http://heasarc.gsfc.nasa.gov/docs/heasarc/fits/java/v1.0/fits.jar
for the latest version. This library is used for reading and writing
FITS data.
- The ImageJ imaging library. See http://rsb.info.nih.gov/ij/ for the latest download.
This library is for all quicklook image generation includes overlays and multicolor
images. SkyView can also initiate a full ImageJ interactive session.
SkyView overrides a few of the standard classes in this library so that
using a later version of the ImageJ library may not work. The version included
is ij138k.
- The Java SAMP library built by Mark Taylor. SAMP allows for the direct export of
images into other SAMP enabled applications. SkyView uses version 1.2. Later versions
are available at https://github.com/mbtaylor/jsamp.
- The JUNIT test library. SkyView uses JUnit 4.10. This is used
only in our regression testing.
Building the SkyView jar
The SkyView jar file contains source code, Java class files and data. To build
the JAR the following steps are required.
- Create the build directory from which we'll generate the JAR
cd /proj/checkout/skyview
rm -rf build
mkdir build
-
Fill the build directory with the content of the JAR dependencies.
We do this first, so that when we compile our local classes later
it will overwrite the standard classes and we'll get our
customized versions of a few ImageJ classes.
cd build
jar xf ../source/lib/ij_138k.jar
jar xf ../source/lib/jsamp-1.2.jar
jar xf ../source/lib/fits.jar
Note that we do not include the JUnit classes in the Jar since they
are only used for testing and not operationally. If you wish to
run the test classes a Junit Jar should be added to the class path
in some fashion.
- Compile all supplied Java files. You have
to go six directories deep. Sigh.
cd /proj/checkout/skyview/source/java
javac -cp .:../../build:../lib/* -d ../../build */*.java */*/*.java */*/*/*.java */*/*/*/*.java */*/*/*/*/*.java */*/*/*/*/*/*.java
- Add in the data used in the Jar
cd /proj/checkout/skyview/data
cp -r * ../build
-
Create and index the jar. Note the manifest
file is needed to make this an executable jar file.
cd /proj/checkout/skyview/build
jar cmf skyview.manifest ../skyvew.jar .
jar i ../skyview.jar
Several of these steps can be done more simply using Ant commands but
are given explicitly here. E.g., Ant can combine
the classes in the library Jar files without expanding them into
the build area, or compile an entire directory hierarchy of arbitrary
depth.
Building the JavaDocs.
If a copy of the javadocs for the SkyView classes is desired it can
be easily created.
cd /proj/checkout/skyview
rm -rf javadoc
mkdir javadoc
cd source/java
javadoc -d ../../javadoc -subpackages ij -subpackages net -subpackages org -subpackages skyview
Installing the files
At this point all of the files needed are ready to be moved into the
server. We assume that SkyView is to be installed at /www/htdocs/skyview
on the same machine.
cd /proj/checkout/skyview
rm -rf /www/htdocs/skyview
cp -r web /www/htdocs/skyview
cp -r javadoc /www/htdocs/skyview/javadoc
cp -skyview.jar /www/htdocs/skyview/jar/skyview.jar
While we could have copied the javadocs and jar into the
web directory before we did the copy into the web directory we
did these separately so that we didn't pollute the directories that we
checked out from our SVN repository.
Customization
A number of features are available for customizing SkyView to a particular
installation.
The SkyView_Env.pm file
The file SkyView_Env.pm is where the location of key elements
of the SkyView interface is specified. It specifies a set of environment
variables that are used by scripts. These include:
- WWW_ROOT
- The base of the HTDOCS tree for the web server, e.g.,
/www/htdocs. Note that we do not include the terminal '/'.
- DATA_ROOT
- The directory in which SkyView FITS and quicklook images will be
created.
- SKY_CACHE
- The directory in which SkyView cache files will be placed.
- PATH
- The search path for executables to be used in looking for scripts.
- WWWDIR_CANON
- This is string which gives the canonicalized beginning
of the path to files that are to be moved into the gallery. It is used
in a security check to ensure that only appropriate files are being moved.
- GALLERY_HTTP
- The directory (relative to WWWDIR) in which gallery images
are to be placed. It should not include a final '/'.
- URL_TO_FILE
- A translation between survey files described as URLs and as local
file names.
- BZIP_DECOMPRESSOR
- A full path to a bzip2 decompression utility. This is
used for SDSS data.
- JARDIR
- A full path to the SkyView jar file. E.g., in our example
this would be /www/htdocs/skyview/jar/skyview.jar
- SITE_nnn
- The links to external sites may be collected here if desired
so that they need be specified in only a single location.
- SKYVIEW_BLOG
- A URL to the SkyView blog.
- CHECK_DIR
- The full directory into which the external surveys check puts output.
SkyView settings files
The SkyView jar uses settings files to specify standard parameters. These
settings are described in detail in the SkyView User's Guide
but we highlight a few that are particularly relevant to the installation of
a SkyView web service. The skyview.settings file found is bundled inside
the jar. In we check out the SkyView project, it is found as
/proj/checkout/skyview/data/skyview.settings. The other main
settings file is the runtime settings which includes setting specifically
for the Web service usage of SkyView. It is found in the distribution in
/proj/checkout/skyview/web/cgi/cgifiles/runtime.settings. This ends up
in the /www/htdocs/skyview/cgi/cgifiles directory when installed.
The skyview.settings specifies the URLs that are used to access
external name resolver services at NED and SIMBAD. It also sets up the default
values for the various geometry choices: projection, coordinates system and so forth.
The runtime.settings file includes a variety of settings that
point to where information is to be placed. Several settings are of
particular interest. Most of these settings inherit environment variables
from the SkyView_Env.pm file discussed above.
- OutputRoot
- Gives the location where output
files (generated images) are to be created. Generally it should
be given as relative path to the script generating the image, since
that conveniently allows both URLs and files to be generated.
- Cache
- Gives the location of a directory in which
files that are retrieved remotely will be cached. This can
add up to terabytes of data, but the cache may be trimmed at any
time. Any data not found will be retrieved from the remote site.
For web
services we recommend also including the SaveBySurvey
setting. That splits up the cache so that each survey is saved
in a separate subdirectory. If the web service runs in a chrooted
environment then the cache must be in the chrooted area. However
it does not need to be web-addressable.
- WebRootPath
- Indicates the root that needs
to be added to web addresses to get file name addresses. E.g.,
if we see a URL http://host/a/b/c.html then we would
prepend the value of this setting to a/b/c.html to get the
path to the file on the web server.
- LocalURL
- This setting describes the translation between
URLs and file locations for URLs used in survey descriptions. It
should have two comma separated entries. The first is a URL prefix
that will be found in the description of local data. The second
is a file name prefix. When SkyView is about to retrieve a survey source
image, it looks to see if the address is a URL. If so it checks to
see if the URL begins with the URL prefix given in this setting. In
that case, SkyView tries replacing the URL prefix with the file prefix
to see if the resulting file is present on the local file system.
If so that file is used.
The end result is that we can use exactly the same survey description for
SkyView's web service and for distributed access to SkyView using
the SkyView-in-a-Jar. When SkyView runs in its web service, the translation
succeeds and the local file is used. When SkyView runs on some user
machine, the translation fails and the data is accessed through its
URL. In that case the user's cache will be checked before the
retrieval is actually attempted.
There are many other settings which describe the location of various
files and directories needed for processing. However most of these
are given relative to the CGI location and need not be updated.
Standard headers and footers
A variety of standard headers and footers are to be found in the includes directory.
The standard versions are appropriate for the HEASARC installation but should be carefully
reviewed for other sites.
Adding surveys
There are three required elements to adding a survey to SkyView: providing access
to the underlying survey data, creating a survey description file, and updating
the SkyView web site. There is one additional step if the survey is three-dimensional,
e.g., has energy resolution within each position pixel.
Providing access to the data
For many surveys SkyView uses a local copy of the data. This data
must be in an area that is accessible to the CGI processes running
in the Web server. I.e., if the web server runs in a chrooted environment, the
survey data must be visible in that environment. SkyView generally requires that
survey data be simple image FITS data with all data in the primary FITS extension.
In some cases data must be transformed in some ways to meet the SkyView
requirements. For example, data saved as several IMAGE extension might need to be
broken out into multiple files.
Once the data are in the appropriate format, all data for a given survey are
typically collected into a single directory. We generally provide a soft link from
a web-addressable surveys directory to this new directory. E.g.,
ln -s /dir/NEW_SURVEY/data /www/htdocs/surveys/NEW_SURVEY
Each new survey will have a link from our standard surveys directory. Different
surveys may be in very different locations on our file system, but we can
easily find them through the soft links.
This allows data to be accessed either as files -- which is how the local SkyView
server will see it, or as a URL. This is needed if we want users to be able
to access the data directly through SkyView-in-a-Jar. If we do not wish
public access to the data except through our SkyView web service, then
this link should not be made.
Creating the survey description
The survey description file is discussed in detail in the
SkyView User's Guide . This file
includes the precise statement of how users are to find the data. If data
are not being stored locally, then this includes information on how the data
will be found on the web. Typically this involves use of the VO Simple Image
Access protocol. However it is possible to directly link to a set of
remote files.
Updating the SkyView Web Site
There are three changes required to publish the new survey in the SkyView web
site.
- The survey description file needs to be added to the SkyView jar
file. This can be done by updating the jar directly but in the long
run it is necessary to add the new file to the standard area (
/proj/checkout/skyview/data/surveys/xml
and build a new Jar. The addition should be committed to the
SVN repository.
- The survey manifest (surveys/survey.manifest) must be updated
to reflect the new survey description
file. As with the description file this can be done intially as an
update to the jar but ultimately needs to be changed and committed in
the base SkyView code.
- The survey needs to be added to web interface.
The file cgi/cgifiles/SurveyList.pm
needs to be updated to include the new survey. Much of
the time this is just adding a single line in one of the survey
categories. However we may wish to split an existing category
or create a new category of surveys. The relevant portion of this
file looks like
...['CategoryName', ['Survey1', 'Survey2', 'Survey3', ..]], ...
where the category names are typical the wavelength regimes
of for the survey but can in principle be arbitrary.
The SurveyN are the names of the survey. Any of a survey's
aliases may be used. When users are selecting surveys
they see selection boxes labeled with the category names. A
special category name of 'skip' is used to skip to start a new line
of selection boxes.
This change should be committed.
- If the new survey is a three-dimensional survey, then
a new entry will be needed in the
cgi/cgifiles/data/3dtable.txt
file.
Operations
There are a number of routine operations in managing the SkyView web site.
These activities use scripts in the ops area which is
not visible to external users.
Cleaning up user data
Updating the list of HEASARC tables
Checking external data sources
Cron jobs
Web access
Access to the SkyView web site is controlled through .htaccess files
in a number of directories. In general directories other than
those in the cgi and ops areas are accessible to the public. The
CGI scripts in the cgi area are available to be executed but may not
be viewed directly and no user access is available to any subdirectory
of cgi.
These .htaccess files also allow us to redirect users. This enables
use to support old address with redirections to new locations.