Recent Heavy SkyView Usage

For the past several weeks there has been very intense usage of SkyView by a small number of users which has degraded responsiveness for other users. Most of the requests seem to be coming from virtual machines using Amazon Web Services so we are not clear who or what is causing the issue. The requests seem temporarily at least to have abated but we are looking at options for throttling or forbidding access should they start up again.

We apologize to our users for the recent poor performance and hope to be nimbler in addressing such concerns in the future.

If you are planning on making some large number of SkyView requests you may find that using SkyView-in-a-Jar is more convenient and less intrusive for other users. You download the jar file and run requests directly. This will often be more efficient — e.g., you can generate just FITS files if that is what you want — and allows many custom options not available through the web interface. With the exception of the Mellinger Optical survey, all surveys available on the Web interface can be accessed using SkyView in a Jar.

Posted in Discussion, Notices | Tagged , | Leave a comment

v3.4.2: SIA bug accessing Swift fixed

There had been a problem accessing Swift UVOT and XRT data through the Simple Image Access (SIA) service.  These data are stored in HIPS format, and this bug affected only those. It should now be fixed. Note that this did not affect the other SkyView interfaces such as the web page or SkyView-in-a-Jar.

Posted in Notices, releases | Tagged , | Leave a comment

SkyView V3.4.0: Handling Binary VOTables in SIAP data

Access to the TGSS ADR1 survey has taken a bit of a hit in the past couple of weeks. The TGSS data is served remotely by SkyView: we provide a resampling proxy to the underlying archive. As discussed in our previous blog post, the archive moved from http: to https: addresses which caused one problem. A few days after we fixed that issue, the format of the Simple Image Access (SIA) response from their server changed. We use SIA to find data at remote archives. This change had also broken our ability to serve the data. SkyView V3.4.0 is now available and should be released on August 22. It should fix this problem.

Simple Image Access is the primary protocol through which we access remote services. It’s used for many of our most popular surveys including the SDSS, 2MASS and the TGSS. Basically it allows us to send a query to the remote archives indicating that we’re looking for data in a given region and having the remote archive tell us about the images it has available nearby. We then download the files that seem useful (if we haven’t already done so in some earlier request) and do whatever resampling and mosaicking that seems appropriate.

The initial response to our SIA request is in a format called VOTable. There are several different structures that VOTable can use internally. The TGSS had been using the most common of these, the TABLEDATA/DATA structure. However they apparently switched to using BINARY/STREAM structure. This is often a little more efficient in bytes, though with most data transmission compressed it doesn’t usually make much difference.

Unfortunately earlier versions of SkyView could not support this structure, so we’ve added a couple of classes to enable SkyView to do so. Now it shouldn’t matter which of these two structures is used (though there are still other VOTable structures that we’ll worry about if we find a service the uses them). This reuses code that we’d originally developed for the HEASARC’s Xamin service so adding things in wasn’t too painful. We apologize to SkyView users for these interruptions.

Since this is a full release, once it is in place SkyView-in-a-Jar users should see their tgss.xml.gz files automatically updated (and will not need to manually delete the file as mentioned in the previous post).

Posted in Announce, releases | Tagged , | Leave a comment

TGSS issue

A couple of users noticed that requests for data from the TGSS ADR1 survey were not working. We’ve tracked this down to a change in the URLs used at the archive in the Netherlands used to store the TGSS data. They have moved to using secure HTTP (i.e., HTTPS urls), and were providing redirection from the old URLs to the new ones. While browsers and many programs will automatically follow such redirections, Java programs will not follow a redirect automatically if the scheme changes, as here where it goes from http to https. We’ve updated the URLs we were using and all should now be working.

Since we’ve fixed this without making a full formal release, if you use SkyView-in-a-Jar you may need to clear out the tgss.xml.gz file in your skycache but otherwise no action should be needed by users. We’ll be doing the full release soon at which point the tgss.xml.gz file will update automatically.

Our apologies to anyone inconvenienced by this and many thanks to those who pointed out the problem.

Posted in Notices | Tagged , | Leave a comment

SkyView V3.3.4: GLEAM Survey and new features

GLEAM image of 2 degree region at declination -80, RA 0

SkyView V3.3.4 has just been released. The new version includes a set of maps from the GaLactic and Extragalactic All-sky MWA Survey (GLEAM). The GLEAM survey was taken at the Murchison Widefield Array (MWA) and covers most of the sky south of 25 degrees north, about 25,000 square degrees.

The image above is an example field: two degrees around RA=0,Dec=-80. SkyView includes four wider bands from the GLEAM survey ranging from 71 to 231 MHz. These and many narrower bands are available from the GLEAM Cutout Service. More details are available from the survey documentation in SkyView or the GLEAM website itself.

There are a few new capabilities in SkyView as well. A new setting, TrackedInputs, can be used to transfer values from the input FITS files to the outputs. The value should be one or more comma-separated FITS keywords which are assumed to be real-valued. The pixel-weighted averages of these keywords are computed as the input files are read, and the averages are written out to the output file. The new keyword is used in the GLEAM survey to average the beam characteristics.

A new survey name, Average, may be used when generating RGB images. Previously, when one wished to create an RGB image from only two surveys, one could pick only two colors, or one could use one of the surveys in two of the RGB colors. Users now have a third choice, to use the average image of the two inputs as the third color. This can give a pleasing image while still evenly weighting the two input surveys. Only one Average survey can be specified and there should be two other surveys specified. The position of the Average survey indicates which color is to be averaged. E.g.,

     java skyview.jar survey=average,dss2red,dss2blue rgb

would use the average image for the red, the dss2red image for the green, and the dss2blue image for the blue.

A new smoother is also available in This smoother (which can be used as a postprocessor when using SkyView-in-a-Jar), does a median filtering of the image using some specified box size. However the median filtering is thresholded, so that a pixel is replaced with the median value only if it deviates from the median by more than some specified offset or ratio. This filter is essentially what is used in the popular Dust And Scratches filter in PhotoShop. One could, e.g., use this filter to get rid of point sources in an image while keeping a diffuse background. There are lots of options in this filter. E.g.,
one can set the threshold to 0 to do median filtering, request thresholding by offset or ratio, and threshold in only one direction,
i.e., for high values but not low as well as setting the box size for the median calculation.

Posted in Announce, releases | Tagged , , , | Leave a comment

SkyView V3.3.2: Links to Mosaicking Inputs

Version 3.3.2 of SkyView has been released with two significant changes: updated memory limits, and links to data inputs.

The parameters used to set the heap size limits for SkyView web requests have been updated so that users will generally be able to create larger images than before — although the exact size is a complex combination of he source data format, the requested output region and other traffic on the site. Images up to 10,000 x 10,000 pixels should be feasible though you may run out of time rather than memory — the connection will drop if nothing happens in 20 minutes. One particular problem was that SkyView ran out of memory if you requested multiple Planck surveys (which are very large single files). You can now retrieve as many Planck datasets as you like! Thanks for Ignacio Cisneros for alerting us to this problem.

The big change for users is a new button that will appear below every image. One of the defining characteristics of SkyView is that it always mosaicks and resamples the input survey data into whatever geometry the user requests. We now provide a button so users can easily see what input data was used for this resampling/mosaicking. This will usually give you a link to the source data that was used in generating the SkyView image. This isn’t necessarily the fundamental source data, it’s just whatever dataset SkyView uses for a particular survey. Currently there are five cases depending on how the data was derived:

  1. From the DSS (or DSS2) plate scans. Links to the directories in which the DSS tiles are found are given.
  2. From a HiPS image (see the recent releases). A link to the base HiPS directory and the properties file for the HiPS is given. Note however that the base directory is not required to be a working URL itself.
  3. From FITS files where SkyView keeps a complete copy of the survey locally. Links to the SkyView survey copy is given.
  4. From FITS files which have been copied from a remote archive into a SkyView cache. Links to the cached files are given.
  5. From FITS files which for copyright or technical reasons cannot be distributed to the public. No links are given.

Two caveats: Many of these datasets have some level of copyright. Please look at the documentation for the survey for more information. Also, to forestall a profusion of browser windows, we have defined a particular browser window in which the link information will be displayed. If you use a tabbed browser, then the second and subsequent times you click on one of the link buttons the tab may be updated, but probably will not be given focus. So it may look like nothing has happened….

You may also note that the detailed SkyView users guide is now available directly from the home page. You don’t need to go into the help pages to see it.

Internally this version considerably reorganizes the various Mosaicker classes minimizing duplication of code. The listing of which files were used in the FITS headers has been somewhat reorganized and now gives the number of pixels resampled from each image.

As always if you have questions or comments please let us know.

Posted in Announce, releases | Tagged , , | Leave a comment

SkyView V3.3.1: Swift UVOT, UltraVista and CFHT HiPS data

We are about to release SkyView version 3.3.1.  This new version includes extended support for HiPS data and has many new surveys.  The primary new datasets are counts, intensity and exposure maps for the Swift UVOT instrument in seven filters.  All data for the first 11.5 years of the mission have been combined into HiPS for each filter.  For the most observed filters a bit under 3% of the sky is observed.  There is a fair bit of overlap but also many differences in the coverage of the various filters.


M81 in UVOT U, UVW1 and UVW2 filters

We have also added in a number of other HIPS surveys from the UltraVista and CFHT surveys. These data are just a few of the many data in Hiearchical Progessive Survey (HiPS) format that are currently available through the web. We anticipate making more available in future data. HiPS joins the VO Simple Image Access Protocol in enabling SkyView to access data from remote

Caching of HiPS data is now supported and some some issues that disabled special access to a UVOT overlay capability have been resolved.

Posted in Announce, releases | Tagged , , , , | 1 Comment

SkyView v3.3.0: Swift XRT surveys

Versions 3.3.0 of SkyView was released today.  This includes an all-sky mosaic of the Swift XRT observations.  Over 129,000 XRT observations were combined to give coverage of just under 6% of the sky.  A set of three surveys were produced: the combined counts, the total exposure, and the intensity map that is the ratio of the first two.   This image ofM31 combines the XRT intensity map with the WISE 22 micron image — contrasting the very different sources of X-ray and infrared emission.

XRT and WISE image of M31

XRT Intensity (green and blue) and WISE (red) in a 1.2 degree field centered on M31.

A few other changes were made in the release, notably many references to HTTP  on our sites were changed to HTTPS.   Although there should be an automated redirection, we are trying to purge the old HTTP addresses since occasionally they cause problems.

Posted in Announce, Notices, releases | Tagged , , | Leave a comment

SkyView and HiPS

Next week we’ll be releasing version 3.3.0 of SkyView.  The big change from the user perspective will be the new Swift XRT combined image.  This is similar in spirit to the venerable PSPC surveys — we’ve added all of the XRT counts and exposures and created all sky maps based on the sums — and an intensity map as the ratio of the two.   From the programming side though the big change is that SkyView will be supporting a new survey format: the Hierarchical Progressive Survey or HiPS.

Traditionally SkyView simply takes a set of images which normally have overlaps and mosaics them to the users specifications.   So long as there is a little overlap between them, we don’t care too much about how the source image tiles are organized.  In HiPS the survey data is required to match a very specific tiling. The organization is strongly tied to the HEALPix pixelization.  This is a way of defining pixels in the sky such that pixels are all of equal area and the average latitude values of the pixels are restricted to a relatively small set — i.e., the pixels are centered on lines of constant latitude.  Both these characteristics are convenient in analysis of CMB datasets where HEALPix originated.  The pixelization starts with 12 top level (or level 0) pixels and then breaks them up recursively into higher level pixels.  So there are 48 level 1 pixels, 192 level 2 pixels, 768 level three pixels and so on.  In HiPS each tile corresponds to a specific HEALPix pixel but the tile also will have Nx and Ny that is some power of 2. So all of the pixels in the tile are also HEALPix pixels but of a higher order.

Levels 0-3 HEALpix pixels.

HEALPix pixelizaton

E.g., in the Swift XRT HiPS images the highest resolution tiles correspond to the 786,432 level 8 HEALPix pixels.  Each tile is a 512×512 image so that each pixel in a tile image  is a level 17 HEALPix pixel.  A client can find the level 8 pixels in their region of interest and download only the tiles of interest.

The other big feature of HiPS is that not only do we have the high-resolution tiles, we also precompute tiles of lower resolution.  E..g., we also have level 7 tiles where we’ve averaged the high-resolution pixels 2×2 and also combined four of the tiles into one tile that covers four times the area.   For the Swift data we’ve generated tiles down to level 3 and also a specially formatted all sky image with even lower resolution.  These choices mimic those of the  CDS Hipsgen tool (including with the latest versions of Aladin) which we used to generate the exposure maps.  We used our own software to build the counts and intensity maps.

A client reading a HiPS file can easily zoom in and out a la Google Earth and Sky or the WWT.  These technology is supported by Aladin and Aladin-lite and can be seen in the ESASky and the MAST data portal.  Our goal in SkyView is not so much providing this kind of dynamical interface as enabling the user to generate images to a clear specification, but we use the multiple resolutions too.  SkyView resamples the lowest resolution higher than the user requested resolution.

Many archives have generated HiPS of their datasets and provide them on the Web. We anticipate providing access to many of these datasets in upcoming versions of SkyView — and we’ll be publishing any HiPS we generate.  If you have a particular interest in getting some dataset into our system please let us know.


Posted in Discussion, Documentation | Tagged , , , | 3 Comments

Code 400, “Bad Request” error

Our system administrators have resolved a problem where users would occasionally get a code 400, “Bad Request” message when accessing SkyView (and other HEASARC sites). Tracking tools required by NASA policy were creating cookies that were too large for our servers to accept.  See  this earlier article ( for more background.  We apologize for any inconvenience caused. Please let us know if you are still seeing issues.

Posted in Discussion | Tagged , | Leave a comment