SkyView V3.4.5: New Surveys and Projection

We’re announcing a test release of SkyView V3.4.5. The new version includes the DRAO 22 MHz survey of most of the sky. This is now the lowest energy survey in SkyView. See Rogers et al, 1999 for details. In addition to missing much of the southern hemisphere the area around a few of the brightest radio sources has been excised since the survey was primarily intended to study diffuse emission.

We’ve also updated the Planck survey data and added the ability to get various polarization measures for most of the Planck frequencies. For all but the two highest energy bands users can get the total intensity, I, the Stokes polarization parameters, Q and U, the total polarized intensity, PI, the fractional polarization, PI/I, and the position angle, PA, of the polarization. Note that the position angle is currently with respect to Galactic coordinates even if you sample the survey in equatorial coordinates — we’re working on updates that will transform the angles appropriately. Future versions of SkyView should have new ways to display these data, e.g., to show positional angle overlays over other maps.

Finally, we also added the Mollweide (Mol) projection as one of SkyView‘s supported projections. This is a fixed projection similar to the Hammer-Aitoff projection in that it is an equal area projection that projects the entire sky into an ellipse. However, unlike the Aitoff, lines of constant latitude are projected into horizontal lines in the projection. Apparently this makes it popular in the CMB community.

Update: In testing out all of the new Planck image planes, we finally had to confront the known issue of the slow generation of Planck images — we’d mentioned this when we first added Planck data. Our test images are usually just a small fraction of the sky, but because of the way Planck data are stored, a single HEALPix file, SkyView was reading in the entire image. We’ve added a new feature to this release where a HEALPix image can be split into tiles and only the tiles that are needed for a given request will be read. We’re currently splitting the HEALPix image into 48 tiles. For a typical small image the new approach can be 10-20 times faster than reading the entire survey.

At least from the perspective of our software this is a bit different than the tiling used in many of our other surveys. We usually have separate FITS files covering regions typically with some overlap between the images. Internally in SkyView each of these files is treated as a separate Image object. SkyView mosaics these images together. In this new feature, there is only a single Image covering the entire sky but the data from that image is segmented and we read a segment only at the first request for data from it.

Posted in Announce, releases | Tagged , , , | Leave a comment

Updated survey documentation for SkyView v3.4.3

We have updated the survey documentation page. It had gotten out of date and was missing some survey information. The Contents listing the available surveys by wavelength regime had been incomplete but is now generated automatically from all of the meta data.  The figure summarizing the surveys in resolution-frequency space has been updated. (No core functionality of SkyView has changed.)

Posted in Documentation, Notices, Uncategorized | Comments Off on Updated survey documentation for SkyView v3.4.3

SkyView in a Jar file size fixed

The SkyView in a Jar file has been overly large lately. (We mistakenly included a bunch of unnecessary files when building the jar last time.) This has now been fixed so that the file size is back down to roughly 7 MB. The code has not changed.

Posted in Uncategorized | Leave a comment

Amazon AWS access to SkyView blocked.

One or more users running on Amazon Web Services accounts have been sending a steady and unsustainable stream of requests to SkyView, overloading the servers and severely degrading SkyView performance for others. We’ve temporarily blocked some SkyView access from any Amazon AWS doamin. We expect that this issue not an intentional attack so we have tried to limit the impact and will remove the block in few days.

Generally speaking we advise users to queue their requests, only submitting the next request when the previous one finishes. While one can gain a bit with two queues anything more than that can certainly affect overall SkyView performance. Alas, this is a pretty small site with just a couple of machines underlying everything.

Our apologies to those who have been affected.

Posted in Notices | Leave a comment

WISE and 2MASS move to HTTPS

Access to data at IRSA recently has moved to requiring HTTPS URLs. This has broken SkyView’s access to the WISE and 2MASS surveys. We have just updated the URLs in the current version of SkyView to re-enable access to these popular surveys.

Note that since this bypassed our regular release procedures users of SkyView in a Jar may wish to delete any wise.xml.gz or 2mass.xml.gz files in their skycache directories to ensure that they get the latest survey description files with the updated URLs.

Posted in Notices | Tagged , , | Leave a comment

Recent Heavy SkyView Usage

For the past several weeks there has been very intense usage of SkyView by a small number of users which has degraded responsiveness for other users. Most of the requests seem to be coming from virtual machines using Amazon Web Services so we are not clear who or what is causing the issue. The requests seem temporarily at least to have abated but we are looking at options for throttling or forbidding access should they start up again.

We apologize to our users for the recent poor performance and hope to be nimbler in addressing such concerns in the future.

If you are planning on making some large number of SkyView requests you may find that using SkyView-in-a-Jar is more convenient and less intrusive for other users. You download the jar file and run requests directly. This will often be more efficient — e.g., you can generate just FITS files if that is what you want — and allows many custom options not available through the web interface. With the exception of the Mellinger Optical survey, all surveys available on the Web interface can be accessed using SkyView in a Jar.

Posted in Discussion, Notices | Tagged , | Leave a comment

v3.4.2: SIA bug accessing Swift fixed

There had been a problem accessing Swift UVOT and XRT data through the Simple Image Access (SIA) service.  These data are stored in HIPS format, and this bug affected only those. It should now be fixed. Note that this did not affect the other SkyView interfaces such as the web page or SkyView-in-a-Jar.

Posted in Notices, releases | Tagged , | Leave a comment

SkyView V3.4.0: Handling Binary VOTables in SIAP data

Access to the TGSS ADR1 survey has taken a bit of a hit in the past couple of weeks. The TGSS data is served remotely by SkyView: we provide a resampling proxy to the underlying archive. As discussed in our previous blog post, the archive moved from http: to https: addresses which caused one problem. A few days after we fixed that issue, the format of the Simple Image Access (SIA) response from their server changed. We use SIA to find data at remote archives. This change had also broken our ability to serve the data. SkyView V3.4.0 is now available and should be released on August 22. It should fix this problem.

Simple Image Access is the primary protocol through which we access remote services. It’s used for many of our most popular surveys including the SDSS, 2MASS and the TGSS. Basically it allows us to send a query to the remote archives indicating that we’re looking for data in a given region and having the remote archive tell us about the images it has available nearby. We then download the files that seem useful (if we haven’t already done so in some earlier request) and do whatever resampling and mosaicking that seems appropriate.

The initial response to our SIA request is in a format called VOTable. There are several different structures that VOTable can use internally. The TGSS had been using the most common of these, the TABLEDATA/DATA structure. However they apparently switched to using BINARY/STREAM structure. This is often a little more efficient in bytes, though with most data transmission compressed it doesn’t usually make much difference.

Unfortunately earlier versions of SkyView could not support this structure, so we’ve added a couple of classes to enable SkyView to do so. Now it shouldn’t matter which of these two structures is used (though there are still other VOTable structures that we’ll worry about if we find a service the uses them). This reuses code that we’d originally developed for the HEASARC’s Xamin service so adding things in wasn’t too painful. We apologize to SkyView users for these interruptions.

Since this is a full release, once it is in place SkyView-in-a-Jar users should see their tgss.xml.gz files automatically updated (and will not need to manually delete the file as mentioned in the previous post).

Posted in Announce, releases | Tagged , | Leave a comment

TGSS issue

A couple of users noticed that requests for data from the TGSS ADR1 survey were not working. We’ve tracked this down to a change in the URLs used at the archive in the Netherlands used to store the TGSS data. They have moved to using secure HTTP (i.e., HTTPS urls), and were providing redirection from the old URLs to the new ones. While browsers and many programs will automatically follow such redirections, Java programs will not follow a redirect automatically if the scheme changes, as here where it goes from http to https. We’ve updated the URLs we were using and all should now be working.

Since we’ve fixed this without making a full formal release, if you use SkyView-in-a-Jar you may need to clear out the tgss.xml.gz file in your skycache but otherwise no action should be needed by users. We’ll be doing the full release soon at which point the tgss.xml.gz file will update automatically.

Our apologies to anyone inconvenienced by this and many thanks to those who pointed out the problem.

Posted in Notices | Tagged , | Leave a comment

SkyView V3.3.4: GLEAM Survey and new features

GLEAM image of 2 degree region at declination -80, RA 0

SkyView V3.3.4 has just been released. The new version includes a set of maps from the GaLactic and Extragalactic All-sky MWA Survey (GLEAM). The GLEAM survey was taken at the Murchison Widefield Array (MWA) and covers most of the sky south of 25 degrees north, about 25,000 square degrees.

The image above is an example field: two degrees around RA=0,Dec=-80. SkyView includes four wider bands from the GLEAM survey ranging from 71 to 231 MHz. These and many narrower bands are available from the GLEAM Cutout Service. More details are available from the survey documentation in SkyView or the GLEAM website itself.

There are a few new capabilities in SkyView as well. A new setting, TrackedInputs, can be used to transfer values from the input FITS files to the outputs. The value should be one or more comma-separated FITS keywords which are assumed to be real-valued. The pixel-weighted averages of these keywords are computed as the input files are read, and the averages are written out to the output file. The new keyword is used in the GLEAM survey to average the beam characteristics.

A new survey name, Average, may be used when generating RGB images. Previously, when one wished to create an RGB image from only two surveys, one could pick only two colors, or one could use one of the surveys in two of the RGB colors. Users now have a third choice, to use the average image of the two inputs as the third color. This can give a pleasing image while still evenly weighting the two input surveys. Only one Average survey can be specified and there should be two other surveys specified. The position of the Average survey indicates which color is to be averaged. E.g.,

     java skyview.jar survey=average,dss2red,dss2blue rgb

would use the average image for the red, the dss2red image for the green, and the dss2blue image for the blue.

A new smoother is also available in skyview.data.ThresholdMedianSmoother. This smoother (which can be used as a postprocessor when using SkyView-in-a-Jar), does a median filtering of the image using some specified box size. However the median filtering is thresholded, so that a pixel is replaced with the median value only if it deviates from the median by more than some specified offset or ratio. This filter is essentially what is used in the popular Dust And Scratches filter in PhotoShop. One could, e.g., use this filter to get rid of point sources in an image while keeping a diffuse background. There are lots of options in this filter. E.g.,
one can set the threshold to 0 to do median filtering, request thresholding by offset or ratio, and threshold in only one direction,
i.e., for high values but not low as well as setting the box size for the median calculation.

Posted in Announce, releases | Tagged , , , | Leave a comment