SkyView and HiPS

Next week we’ll be releasing version 3.3.0 of SkyView.  The big change from the user perspective will be the new Swift XRT combined image.  This is similar in spirit to the venerable PSPC surveys — we’ve added all of the XRT counts and exposures and created all sky maps based on the sums — and an intensity map as the ratio of the two.   From the programming side though the big change is that SkyView will be supporting a new survey format: the Hierarchical Progressive Survey or HiPS.

Traditionally SkyView simply takes a set of images which normally have overlaps and mosaics them to the users specifications.   So long as there is a little overlap between them, we don’t care too much about how the source image tiles are organized.  In HiPS the survey data is required to match a very specific tiling. The organization is strongly tied to the HEALPix pixelization.  This is a way of defining pixels in the sky such that pixels are all of equal area and the average latitude values of the pixels are restricted to a relatively small set — i.e., the pixels are centered on lines of constant latitude.  Both these characteristics are convenient in analysis of CMB datasets where HEALPix originated.  The pixelization starts with 12 top level (or level 0) pixels and then breaks them up recursively into higher level pixels.  So there are 48 level 1 pixels, 192 level 2 pixels, 768 level three pixels and so on.  In HiPS each tile corresponds to a specific HEALPix pixel but the tile also will have Nx and Ny that is some power of 2. So all of the pixels in the tile are also HEALPix pixels but of a higher order.

http://healpix.sourceforge.net/images/gorski_f1.jpg

Levels 0-3 HEALpix pixels.

HEALPix pixelizaton

E.g., in the Swift XRT HiPS images the highest resolution tiles correspond to the 786,432 level 8 HEALPix pixels.  Each tile is a 512×512 image so that each pixel in a tile image  is a level 17 HEALPix pixel.  A client can find the level 8 pixels in their region of interest and download only the tiles of interest.

The other big feature of HiPS is that not only do we have the high-resolution tiles, we also precompute tiles of lower resolution.  E..g., we also have level 7 tiles where we’ve averaged the high-resolution pixels 2×2 and also combined four of the tiles into one tile that covers four times the area.   For the Swift data we’ve generated tiles down to level 3 and also a specially formatted all sky image with even lower resolution.  These choices mimic those of the  CDS Hipsgen tool (including with the latest versions of Aladin) which we used to generate the exposure maps.  We used our own software to build the counts and intensity maps.

A client reading a HiPS file can easily zoom in and out a la Google Earth and Sky or the WWT.  These technology is supported by Aladin and Aladin-lite and can be seen in the ESASky and the MAST data portal.  Our goal in SkyView is not so much providing this kind of dynamical interface as enabling the user to generate images to a clear specification, but we use the multiple resolutions too.  SkyView resamples the lowest resolution higher than the user requested resolution.

Many archives have generated HiPS of their datasets and provide them on the Web. We anticipate providing access to many of these datasets in upcoming versions of SkyView — and we’ll be publishing any HiPS we generate.  If you have a particular interest in getting some dataset into our system please let us know.

 

Posted in Discussion, Documentation | Tagged , , , | 3 Comments

Code 400, “Bad Request” error

Our system administrators have resolved a problem where users would occasionally get a code 400, “Bad Request” message when accessing SkyView (and other HEASARC sites). Tracking tools required by NASA policy were creating cookies that were too large for our servers to accept.  See  this earlier article (https://skyview.gsfc.nasa.gov/blog/index.php/2014/07/16/v3-1-7-skyview-add-google-analytics/) for more background.  We apologize for any inconvenience caused. Please let us know if you are still seeing issues.

Posted in Discussion | Tagged , | Leave a comment

SkyView V3.2.3 Beware the Ides of March!

Over the next few days (as early as noon 3/15) the SkyView servers will be transitioning to requiring secure HTTP, i.e., HTTPS, URLs to access SkyView services.  This is part of a mandated transition of all NASA (and indeed all federal) web sites. Web users should see little difference.  If you come in using a non-HTTPS URL you will be automatically redirected to the correct URL.  However users who access SkyView through batch services may see a difference since the software that accesses SkyView may not automatically follow the redirect.  In particular, Java’s network library  will not automatically follow redirects which transition from HTTP to HTTPS.   Java (and any other framework that exhibits similar behavior) tools can be programmed to explicitly follow redirects, or the HTTPS URL can be given directly.

Since SkyView-in-a-Jar is written in Java it is sensitive to this change.  When you are using SkyView-in-a-Jar it retrieves survey data from SkyView using information in survey description files. Until version 3.2.3 those files had HTTP URLs.  With this version all the survey description files have been updated to use HTTPS URLs.  We have updated the survey XML files to use HTTPS addresses and it had been our thought that users of recent jar files would be OK since the program would detect the new survey description files and download them automatically.  However we have since recognized that the base URL for the survey description files needs to be updated.  So we recommend all SkyView-in-a-Jar users update to the latest version of the Jar (at least version 3.2.3).

Update: 3/17:  Small updates have been made to the Jar that may fix issues seen by Windows users of SkyView-in-a-Jar.  Our apologies for not catching these earlier.

Posted in Announce, releases | Tagged , | Leave a comment

SkyView update: v3.2.2 including HI4PI data

On Tuesday Dec. 13, 2016, we updated SkyView to v3.2.2 in order to include the new HI4PI radio survey. This dataset is the atomic neutral hydrogen (HI) column density map derived from HI4PI (|Vlsr| < 600 km/s). It includes the Effelsberg-Bonn HI Survey (EBHIS) of the northern sky but completes the sky coverage with the addition of southern sky data from Parkes.

Note that a previously-downloaded jar file of v3.2.1 should still pick up this new survey automatically. This update also includes some changes to the internal meta data handling, but this should be transparent to users.

Posted in Announce, releases, Uncategorized | 3 Comments

SkyView downtime October 4, 2016

SkyView was down from about 9:00am to 2:30pm today due to a problem encountered while updating the SkyView servers.  SkyView operations should now be back to normal. We apologize for the inconvenience.

Posted in Discussion, Notices | Leave a comment

NASA SkyView vs SkyView

We often get email requesting  help downloading and using SkyView on the iPhone and other devices.  We thought we would try to clear up some apparent confusion.  These questions refer to a smart phone app that is not associated with NASA SkyView but has the same name – SkyView (or SkyView Free).  Information about this app can be found at the Apple App Store and the Terminal Eleven site.

Posted in Discussion | 2 Comments

SkyView V3.2.1: SDSS and AKARI fixes

On Wednesday we released SkyView version 3.2.1 to fix the SDSS survey. It includes a new capability where when we are using an SIA service we can apply a regular expression transformation to the URLs being returned. We use this to transform SDSS image tile URLs from http://www.sdss3.org/… to https://www.sdss.org/…. The setting SiapUrlTransform expects two comma-separated fields which are interpreted as arguments to the Java String.replaceAll(String,String) method in modifying the input URL string.

We’ve had this fix for several days but were unable to release it. SkyView was unable to see remote HTTPS URLs due to GSFCs firewall settings. As soon as that was addressed by our system managers we made the release.

Shortly after we communicated some concerns with the publishers of the AKARI data regarding issues with their SIA service. They were using Ecliptic coordinates in places where ICRS coordinates were required. SkyView had implemented special code to address this. The AKARI team very quickly implemented fixes on their end, and this morning we removed the special handling we had been performing for AKARI. We delivered this as a modification to the v3.2.1 survey file rather than a new release to minimize the time that AKARI was off-line. SkyView in a Jar users who are using version 3.2.0 or later should get the new survey description next time they access AKARI data. Users of older versions of SkyView will need to upgrade to access AKARI data properly.

Users may have seen problems with accessing AKARI data for several hours this morning until we updated our software to recognize their fixes.

Posted in Notices, releases | Tagged , , | 2 Comments

Continued SDSS issues

While remote access to the UKIDSS server was restored shortly after our last message we are continuing to have problems with the SDSS. In addition to the problems with the metadata found in the SDSS SIA service, there has been some overloading of SDSS servers and it was recommended that we switch from http://data.sdss3.org/… urls to http://data.sdss.org urls which points to a different and more robust set of servers.

We enabled this capability and switched to the new servers in our test version 3.2.1. This failed initially. The new URLs were in turn being redirected to https://data.sdss.org. Java network connections do not follow redirects to a different scheme (i.e., from http to https) and so we weren’t getting the data. We put in some code to handle this kind of redirection and also tried simply making the initial request to the https addresses. All of this worked fine in our test environment and failed miserably when we tried to run it on the operational systems.

The current problem seems to be that the GSFC network firewall is quite restrictive and is not allowing SkyView general HTTPS access to the outside world. It does allow general outgoing HTTP. It may take a few days before we can get this switched.

In the meantime we have a dilemma. The SDSS3 links fail frequently when used from our web site, e.g., as I write this the initial SIA requests are failing due to some database error, but they do work some of the time. The SDSS links work very nicely when used in the JAR file but don’t currently work at all from the web site…

So for the nonce we are leaving SkyView at v3.2.0 but recommend that Jar users extract the jar from v3.2.1 (i.e.,
http://skyview.gsfc.nasa.gov/v3.2.1/jar/skyview.jar) where they are more likely to get a version of SkyView that addresses all of the problems we have seen.

Posted in Discussion, Notices | Tagged , | Leave a comment

Problems with UKIDSS and SDSS remote surveys

Over the past couple of days we have discovered problems in accessing two of our remote survey data sets. The UKIDSS simple image access service has been non-responsive. We’re hoping it comes up again soon. You can see the red warning on the SkyView home page where we assess the aliveness of remote resources.

A user has alerted us to a more subtle and serious problem with the SDSS surveys. If they requested a large (0.5 degree) region they found that many of the pixels in their image were blank. If they requested the same image later most of those pixels were filled in. A few missing pixels might require three image requests to be filled in. It appears that the SDSS is supplying incorrect descriptions of the coverage of images in their SIA service. We have informed the SDSS of the issue but while it is being worked on we have created a version of SkyView, v3.2.1, which has a workaround for the issue. You can try it out now — and if you are using the SDSS for large images we strongly recommend that you do — and anticipate that it will be made the default version in the next few days.

The SDSS problem arises from how SkyView mosaics data from multiple images together. In the first stage an image generator defines a set of candidate images that are to be considered. For remote services, where we are using the SIA protocol to get data, all of the images returned by the SIA service are considered as candidates. We then consider each pixel in the output image to be produced and decide which of the candidates it should be sampled from. The usual approach — and the one used for the SDSS — is to pick the candidate where we will be furthest from the edge of the image.

To make this determination we need to know the sky coverage of each candidate. If we have already downloaded an image so that it exists in the cache we can and do use information in the FITS file itself. If we haven’t downloaded the image yet, we normally use the coverage information provided in the SIA metadata returned with the image URL. Typically there will be a lot of candidate images that aren’t going to be used in the mosaic and we don’t want to have to download them all.

Since the SIA metadata is wrong, when we chose the candidate image, it often turned out that that image could not be resampled at the given pixel. However the SDSS SIA metadata generally did point us to include the right images, just not the proper boundaries for each one. So after we downloaded the data we could recompute the best source image and that would work. Internally the class we use to find the image to resample a given pixel is called an ImageFinder. The default ImageFinder is the Border ImageFinder. For the SDSS we have created a special ImageFinder, LocalBorder, which iteratively reruns the image finding operation until all of the images that are used in the mosaic are downloaded before we do the image finding. Typically this requires three iterations for large images. However we still only download the images that we actually need, and only once. The image finding is generally pretty fast, so the time penalty for this kludge is modest. We only do the actual resampling with the final data.

However for this to work, the SIA information has to be at least in the correct ball park. Generally it appears that the SIA-specified center of the image is deep within the actual image. In this case
we anticipate that our workaround will at least greatly mitigate if not always cure the problem.

Note that the SDSS problem is particularly acute for those using SkyView-in-a-Jar. Since many popular regions are already available in the cache for the web site, the problem may not show up there. However jar users are building up their own caches from scratch.

The jar file that addresses these issues can be found at http://skyview.gsfc.nasa.gov/v3.2.1/jar/skyview.jar. Note that to get the correct survey description file users will need to use the setting:
xmlurlprefix=http://skyview.gsfc.nasa.gov/v3.2.1/jar/
until this version is released since the default for this setting points to the current release which currently has the older version of the SDSS survey description.

Posted in Discussion, Notices | Tagged , , , | Leave a comment

SkyView v3.2.0: Displaying quantitative survey information

In the home page in the new v3.2.0 you may notice a new line under the Survey Documentation link:

Summary: CSV or Plot

You can click on CSV to download a table of the information on all SkyView surveys. If there are multiple ranges of epoch, the epoch given will be from the beginning of the first range till the end of the last.

If you click on Plot you can easily make plots of the information like

Plot of Resolution versus Frequency

You can just mouse around in the plot to see which survey corresponds to which point. Here the mouse happened to be near the GALEX NUV survey point. Or you can pick a different X or Y axis and see how SkyView surveys are distributed in time, energy, resolution and coverage.

These plots use the Plotly plotting package.

Posted in Discussion | Tagged , , , | Leave a comment