Problems with UKIDSS and SDSS remote surveys

Over the past couple of days we have discovered problems in accessing two of our remote survey data sets. The UKIDSS simple image access service has been non-responsive. We’re hoping it comes up again soon. You can see the red warning on the SkyView home page where we assess the aliveness of remote resources.

A user has alerted us to a more subtle and serious problem with the SDSS surveys. If they requested a large (0.5 degree) region they found that many of the pixels in their image were blank. If they requested the same image later most of those pixels were filled in. A few missing pixels might require three image requests to be filled in. It appears that the SDSS is supplying incorrect descriptions of the coverage of images in their SIA service. We have informed the SDSS of the issue but while it is being worked on we have created a version of SkyView, v3.2.1, which has a workaround for the issue. You can try it out now — and if you are using the SDSS for large images we strongly recommend that you do — and anticipate that it will be made the default version in the next few days.

The SDSS problem arises from how SkyView mosaics data from multiple images together. In the first stage an image generator defines a set of candidate images that are to be considered. For remote services, where we are using the SIA protocol to get data, all of the images returned by the SIA service are considered as candidates. We then consider each pixel in the output image to be produced and decide which of the candidates it should be sampled from. The usual approach — and the one used for the SDSS — is to pick the candidate where we will be furthest from the edge of the image.

To make this determination we need to know the sky coverage of each candidate. If we have already downloaded an image so that it exists in the cache we can and do use information in the FITS file itself. If we haven’t downloaded the image yet, we normally use the coverage information provided in the SIA metadata returned with the image URL. Typically there will be a lot of candidate images that aren’t going to be used in the mosaic and we don’t want to have to download them all.

Since the SIA metadata is wrong, when we chose the candidate image, it often turned out that that image could not be resampled at the given pixel. However the SDSS SIA metadata generally did point us to include the right images, just not the proper boundaries for each one. So after we downloaded the data we could recompute the best source image and that would work. Internally the class we use to find the image to resample a given pixel is called an ImageFinder. The default ImageFinder is the Border ImageFinder. For the SDSS we have created a special ImageFinder, LocalBorder, which iteratively reruns the image finding operation until all of the images that are used in the mosaic are downloaded before we do the image finding. Typically this requires three iterations for large images. However we still only download the images that we actually need, and only once. The image finding is generally pretty fast, so the time penalty for this kludge is modest. We only do the actual resampling with the final data.

However for this to work, the SIA information has to be at least in the correct ball park. Generally it appears that the SIA-specified center of the image is deep within the actual image. In this case
we anticipate that our workaround will at least greatly mitigate if not always cure the problem.

Note that the SDSS problem is particularly acute for those using SkyView-in-a-Jar. Since many popular regions are already available in the cache for the web site, the problem may not show up there. However jar users are building up their own caches from scratch.

The jar file that addresses these issues can be found at Note that to get the correct survey description file users will need to use the setting:
until this version is released since the default for this setting points to the current release which currently has the older version of the SDSS survey description.

Posted in Discussion, Notices | Tagged , , , | Leave a comment

SkyView v3.2.0: Displaying quantitative survey information

In the home page in the new v3.2.0 you may notice a new line under the Survey Documentation link:

Summary: CSV or Plot

You can click on CSV to download a table of the information on all SkyView surveys. If there are multiple ranges of epoch, the epoch given will be from the beginning of the first range till the end of the last.

If you click on Plot you can easily make plots of the information like

Plot of Resolution versus Frequency

You can just mouse around in the plot to see which survey corresponds to which point. Here the mouse happened to be near the GALEX NUV survey point. Or you can pick a different X or Y axis and see how SkyView surveys are distributed in time, energy, resolution and coverage.

These plots use the Plotly plotting package.

Posted in Discussion | Tagged , , , | Leave a comment

SkyView v3.2.0: Quantitative metadata

Version 3.2.0 of SkyView includes a new section, Quantitative, in the survey description file. This section is intended to capture machine-friendly quantitative characteristics of SkyView surveys. Diverse sources have been used for these and notes are included giving the source when it is not in the standard reference for the survey.

Currently all surveys include a central, minimum and maximum frequency in Hertz.

A central epoch is given and one or more ranges for the epoch are available. Note that the central epoch is a kind of average of the ranges and occasionally the central epoch is actually outside of the constituent ranges. Epochs are given in Modified Julian Days.

The coverage of each survey is given as a fraction of the sky. For some of the GOODS survey datasets these can be quite small.

Both a nominal resolution and scale are given. For surveys where the resolution may be variable a typical good value is given. Units for these quantities are arcseconds.

Lastly an estimated sensitivity of the survey in Janskies is given. Often this involves significant estimation and the numbers should be treated with caution. Note that a better gauge of the relative sensitivity of two surveys may be the product of frequency and sensitivity.

These data may be obtained as a CSV file or explored as a plot using new capabilties in v3.2.0.

Posted in Discussion | Tagged | Leave a comment

SkyView V3.2.0: Survey descriptions downloaded on demand

The biggest change in v3.2.0 of SkyView is that survey description files are no longer part of the SkyView jar. In v3.1.21 we added a small file that retained the mapping between survey names and the description files they referred to. In that version we only used the file to skip reading all of the survey description files — the user only needed to read the survey descriptions for the surveys they were interested in.

In v3.2.0 we took further advantage of this file by adding the information about when the survey description files were created to it. When a user requests a survey, SkyView looks to see if the survey is available as a local file (e.g., as it would be for the web site). If not it checks to see if there is a cached version of the file and if the cached version is newer than the version at the SkyView web site. If so this version is used. Otherwise the file is downloaded from the SkyView web site, placed in the cache and then processing continues. So users will always get have the latest version of the survey to work with.

There remains one possible problem. If a survey updates tile files without changing their names, then SkyView will not be able to tell that the cached data files are out of date. Fortunately most dynamic surveys change the names of the files (e.g., using a version identifier) when they update their tile data.

Posted in Discussion | Tagged , , | Leave a comment

SkyView v3.2.0: Some survey data removed from jar

SkyView-in-a-Jar users will note that the new v3.2.0 SkyView jar is much smaller than the earlier versions, under 4 megabytes compared to over 15 megabytes for v3.1.21. Part of the reason is that the way we access some older surveys has been changed.

The SkyView software can look for survey data in three distinct locales: over the web using URLs, in the local file system using file names, and in the SkyView jar using resource names.

Almost all surveys use URLs. There are two special features for these. First, SkyView can notice that the URL is pointing to data on the local file system. The SkyView web site uses this to translate SkyView URLs to local file references. Second, most URLs are cached. When a user or the SkyView web site asks for a URL from, say, the SDSS, it downloads it only once. The next time that URL is requested it is retrieved from the cache. Skyview-in-a-Jar users will also cache the data from surveys stored on the SkyView machines, since for them those data are not local.

In previous versions a few survey’s data (EGRET, CompTel, nH, 1420 Mhz, 408Mhz, HEA01A2) were actually included in the jar file directly. We’ve now moved these to use URLs as above. Since these will be cached, SkyView-in-a-Jar users will only download the data once. All of these surveys are quite small and we don’t anticipate anyone will really notice the difference. The one exception is that this did allow users to test out SkyView with no network connections. This is still easy to do using the UserSurvey settings. More sophisticated
users can also provide survey description files that point to local data.

A few surveys are still provided as references to files in the local file system. This includes the Mellinger survey where our agreement is that the underlying data will not be directly visible as a URL. The other survey is the GOODSIRAC data. These data are available as URLs but it is difficult to characterize the image WCSs in a way that works with our survey description files. When specifying image URLs SkyView tries to provide enough geometry information in the description to enable the program to decide which candidate images are needed for a mosaic before downloading them. That was hard to do for this survey. As local files rather than URLs, SkyView can always use the WCS in the actual FITS file when looking at their geometry.

Posted in Discussion | Tagged , , , , | Leave a comment

SkyView v3.2.0 Preview: Summary

Next week we will be releasing version 3.2.0 of SkyView. You can play with it right now if you specify the version v3.2.0 in the SkyView URL, but it’s still being tested. There are several major changes in this release and we’ll be putting out a series of posts describing them in detail. Major changes include:
– Survey description files are now downloaded (and cached) as needed from the SkyView web site. This means that SkyView-in-a-Jar users will no longer need to update the their local copy of the JAR file to take advantage of updates to surveys.
– The SkyView jar used to contain the data for a few smaller generally fairly old surveys. To further reduce the size of the jar, these have been taken out.
– Survey description files now contain a set of fields that give simple, uniform, quantitative information on surveys (frequency, epoch, resolution, sensitivity) in a format easily used by a machine.
– There are new capabilities to extract and display these quantities as CSV downloads or in interactive plots.

Posted in Announce, releases | Tagged , | Leave a comment

A Bit of a Speed Bump: V3.1.21

We have released a new version of SkyView (V3.1.21) which  includes changes in the way we gather meta data for image generation. When we build SkyView we  now create a single hash file which associates all survey names and aliases with the appropriate survey descriptions.  Thus SkyView only reads the survey descriptions of the surveys  the user asks for. Previously it needed to read the beginning of all ~100 files.  This seems to save about .2 seconds per image.  We have also added a data availability check for AKARI, our new remote survey.

Posted in Announce, releases | Tagged , | Leave a comment

AKARI and TGSS ADR1 in SkyView: V3.1.20

Two new surveys sets have been added to the next release of SkyView (v3.1.20): Four new infrared all-sky surveys taken by the AKARI FIS (wavelengths between 60 and 160 microns) are now available. We have also added the TGSS ADR1 survey, a near all-sky high-resolution survey at 150 MHz (2 meters). Breaking out the acronym this is the [Tata Institute for Fundamental Research (TIFR)] [Giant Metrewave Radio Telescope (GMRT)] Sky Survey First Alternate Data Release. Thanks to Huib Intema for help in adding this survey to SkyView.

These surveys use the VO Simple Image Access protocol to retrieve data from the archive centers at JAXA/JVO and Leiden respectively. Until we build up a local cache of data from popular regions access may be a little slower than it will ultimately be. The TGSS survey uses Lanczos resampling and Sqrt image scaling as a default, but these may be overriden by the user.

We have expanded the display of the infrared and radio surveys a little: breaking out various infrared missions and separating the radio data into GHz and MHz surveys. So there’s a bit of change in the appearance but no surveys should have been deleted.

The release also includes a few internal tweaks and bug fixes that should be invisible to users.
V3.1.20 is anticipated to be released on July 11 but is available now if you use the version explicitly.

If you have any surveys or other capabilities you would like to see added to SkyView please let us know.

Posted in Announce, releases | Tagged , , | 1 Comment

SkyView update to FIRST survey

We have just released V3.1.19 of SkyView. The only major change is an update to the FIRST survey which incorporates the latest data available at the MAST archive at the Space Telescope Science Institute. There is a bit more coverage and sometimes better processing in the new data. Thanks to Jean Tate for pointing out that our list of FIRST data was out of date.

V3.1.19 also includes a few tweaks to our scripts adapting to new internal locations after our move.

Posted in Notices, releases | Tagged , | Leave a comment

SkyView Hardware Update

Last week SkyView was updated with new hardware. The SkyView interface should look the same but behind the scenes this update may add a bit more speed to queries but will mostly provide a more robust computing environment, more space for caching query results files and better sharing of processing over multiple hosts.

So far the transition has been smooth. We did discover a couple of obsolete links that may have caused some problems but these should now resolve to current URLs.




Posted in Announce, releases | Tagged | Leave a comment