Amazon AWS access to SkyView blocked.

Posted on October 25, 2018 by Tess Jaffe

One or more users running on Amazon Web Services accounts have been sending a steady and unsustainable stream of requests to SkyView, overloading the servers and severely degrading SkyView performance for others. We’ve temporarily blocked some SkyView access from any Amazon AWS doamin. We expect that this issue not an intentional attack so we have tried to limit the impact and will remove the block in few days.

Generally speaking we advise users to queue their requests, only submitting the next request when the previous one finishes. While one can gain a bit with two queues anything more than that can certainly affect overall SkyView performance. Alas, this is a pretty small site with just a couple of machines underlying everything.

Our apologies to those who have been affected.

Posted in Notices | Leave a comment

WISE and 2MASS move to HTTPS

Posted on October 22, 2018 by Tess Jaffe

Access to data at IRSA recently has moved to requiring HTTPS URLs. This has broken SkyView’s access to the WISE and 2MASS surveys. We have just updated the URLs in the current version of SkyView to re-enable access to these popular surveys.

Note that since this bypassed our regular release procedures users of SkyView in a Jar may wish to delete any wise.xml.gz or 2mass.xml.gz files in their skycache directories to ensure that they get the latest survey description files with the updated URLs.

Posted in Notices | Tagged , , | Leave a comment

Recent Heavy SkyView Usage

Posted on October 22, 2018 by Tess Jaffe

For the past several weeks there has been very intense usage of SkyView by a small number of users which has degraded responsiveness for other users. Most of the requests seem to be coming from virtual machines using Amazon Web Services so we are not clear who or what is causing the issue. The requests seem temporarily at least to have abated but we are looking at options for throttling or forbidding access should they start up again.

We apologize to our users for the recent poor performance and hope to be nimbler in addressing such concerns in the future.

If you are planning on making some large number of SkyView requests you may find that using SkyView-in-a-Jar is more convenient and less intrusive for other users. You download the jar file and run requests directly. This will often be more efficient — e.g., you can generate just FITS files if that is what you want — and allows many custom options not available through the web interface. With the exception of the Mellinger Optical survey, all surveys available on the Web interface can be accessed using SkyView in a Jar.

Posted in Discussion, Notices | Tagged , | Leave a comment

v3.4.2: SIA bug accessing Swift fixed

Posted on October 11, 2018 by Tess Jaffe

There had been a problem accessing Swift UVOT and XRT data through the Simple Image Access (SIA) service.  These data are stored in HIPS format, and this bug affected only those. It should now be fixed. Note that this did not affect the other SkyView interfaces such as the web page or SkyView-in-a-Jar.

Posted in Notices, releases | Tagged , | Leave a comment

SkyView V3.4.0: Handling Binary VOTables in SIAP data

Posted on August 21, 2018 by Tess Jaffe

Access to the TGSS ADR1 survey has taken a bit of a hit in the past couple of weeks. The TGSS data is served remotely by SkyView: we provide a resampling proxy to the underlying archive. As discussed in our previous blog post, the archive moved from http: to https: addresses which caused one problem. A few days after we fixed that issue, the format of the Simple Image Access (SIA) response from their server changed. We use SIA to find data at remote archives. This change had also broken our ability to serve the data. SkyView V3.4.0 is now available and should be released on August 22. It should fix this problem.

Simple Image Access is the primary protocol through which we access remote services. It’s used for many of our most popular surveys including the SDSS, 2MASS and the TGSS. Basically it allows us to send a query to the remote archives indicating that we’re looking for data in a given region and having the remote archive tell us about the images it has available nearby. We then download the files that seem useful (if we haven’t already done so in some earlier request) and do whatever resampling and mosaicking that seems appropriate.

The initial response to our SIA request is in a format called VOTable. There are several different structures that VOTable can use internally. The TGSS had been using the most common of these, the TABLEDATA/DATA structure. However they apparently switched to using BINARY/STREAM structure. This is often a little more efficient in bytes, though with most data transmission compressed it doesn’t usually make much difference.

Unfortunately earlier versions of SkyView could not support this structure, so we’ve added a couple of classes to enable SkyView to do so. Now it shouldn’t matter which of these two structures is used (though there are still other VOTable structures that we’ll worry about if we find a service the uses them). This reuses code that we’d originally developed for the HEASARC’s Xamin service so adding things in wasn’t too painful. We apologize to SkyView users for these interruptions.

Since this is a full release, once it is in place SkyView-in-a-Jar users should see their tgss.xml.gz files automatically updated (and will not need to manually delete the file as mentioned in the previous post).

Posted in Announce, releases | Tagged , | Leave a comment

TGSS issue

Posted on August 7, 2018 by Tess Jaffe

A couple of users noticed that requests for data from the TGSS ADR1 survey were not working. We’ve tracked this down to a change in the URLs used at the archive in the Netherlands used to store the TGSS data. They have moved to using secure HTTP (i.e., HTTPS urls), and were providing redirection from the old URLs to the new ones. While browsers and many programs will automatically follow such redirections, Java programs will not follow a redirect automatically if the scheme changes, as here where it goes from http to https. We’ve updated the URLs we were using and all should now be working.

Since we’ve fixed this without making a full formal release, if you use SkyView-in-a-Jar you may need to clear out the tgss.xml.gz file in your skycache but otherwise no action should be needed by users. We’ll be doing the full release soon at which point the tgss.xml.gz file will update automatically.

Our apologies to anyone inconvenienced by this and many thanks to those who pointed out the problem.

Posted in Notices | Tagged , | Leave a comment

SkyView V3.3.4: GLEAM Survey and new features

Posted on May 2, 2018 by Tess Jaffe

GLEAM image of 2 degree region at declination -80, RA 0

SkyView V3.3.4 has just been released. The new version includes a set of maps from the GaLactic and Extragalactic All-sky MWA Survey (GLEAM). The GLEAM survey was taken at the Murchison Widefield Array (MWA) and covers most of the sky south of 25 degrees north, about 25,000 square degrees.

The image above is an example field: two degrees around RA=0,Dec=-80. SkyView includes four wider bands from the GLEAM survey ranging from 71 to 231 MHz. These and many narrower bands are available from the GLEAM Cutout Service. More details are available from the survey documentation in SkyView or the GLEAM website itself.

There are a few new capabilities in SkyView as well. A new setting, TrackedInputs, can be used to transfer values from the input FITS files to the outputs. The value should be one or more comma-separated FITS keywords which are assumed to be real-valued. The pixel-weighted averages of these keywords are computed as the input files are read, and the averages are written out to the output file. The new keyword is used in the GLEAM survey to average the beam characteristics.

A new survey name, Average, may be used when generating RGB images. Previously, when one wished to create an RGB image from only two surveys, one could pick only two colors, or one could use one of the surveys in two of the RGB colors. Users now have a third choice, to use the average image of the two inputs as the third color. This can give a pleasing image while still evenly weighting the two input surveys. Only one Average survey can be specified and there should be two other surveys specified. The position of the Average survey indicates which color is to be averaged. E.g.,

     java skyview.jar survey=average,dss2red,dss2blue rgb

would use the average image for the red, the dss2red image for the green, and the dss2blue image for the blue.

A new smoother is also available in skyview.data.ThresholdMedianSmoother. This smoother (which can be used as a postprocessor when using SkyView-in-a-Jar), does a median filtering of the image using some specified box size. However the median filtering is thresholded, so that a pixel is replaced with the median value only if it deviates from the median by more than some specified offset or ratio. This filter is essentially what is used in the popular Dust And Scratches filter in PhotoShop. One could, e.g., use this filter to get rid of point sources in an image while keeping a diffuse background. There are lots of options in this filter. E.g.,
one can set the threshold to 0 to do median filtering, request thresholding by offset or ratio, and threshold in only one direction,
i.e., for high values but not low as well as setting the box size for the median calculation.

Posted in Announce, releases | Tagged , , , | Leave a comment

SkyView V3.3.2: Links to Mosaicking Inputs

Posted on November 15, 2017 by Tess Jaffe

Version 3.3.2 of SkyView has been released with two significant changes: updated memory limits, and links to data inputs.

The parameters used to set the heap size limits for SkyView web requests have been updated so that users will generally be able to create larger images than before — although the exact size is a complex combination of he source data format, the requested output region and other traffic on the site. Images up to 10,000 x 10,000 pixels should be feasible though you may run out of time rather than memory — the connection will drop if nothing happens in 20 minutes. One particular problem was that SkyView ran out of memory if you requested multiple Planck surveys (which are very large single files). You can now retrieve as many Planck datasets as you like! Thanks for Ignacio Cisneros for alerting us to this problem.

The big change for users is a new button that will appear below every image. One of the defining characteristics of SkyView is that it always mosaicks and resamples the input survey data into whatever geometry the user requests. We now provide a button so users can easily see what input data was used for this resampling/mosaicking. This will usually give you a link to the source data that was used in generating the SkyView image. This isn’t necessarily the fundamental source data, it’s just whatever dataset SkyView uses for a particular survey. Currently there are five cases depending on how the data was derived:

  1. From the DSS (or DSS2) plate scans. Links to the directories in which the DSS tiles are found are given.
  2. From a HiPS image (see the recent releases). A link to the base HiPS directory and the properties file for the HiPS is given. Note however that the base directory is not required to be a working URL itself.
  3. From FITS files where SkyView keeps a complete copy of the survey locally. Links to the SkyView survey copy is given.
  4. From FITS files which have been copied from a remote archive into a SkyView cache. Links to the cached files are given.
  5. From FITS files which for copyright or technical reasons cannot be distributed to the public. No links are given.

Two caveats: Many of these datasets have some level of copyright. Please look at the documentation for the survey for more information. Also, to forestall a profusion of browser windows, we have defined a particular browser window in which the link information will be displayed. If you use a tabbed browser, then the second and subsequent times you click on one of the link buttons the tab may be updated, but probably will not be given focus. So it may look like nothing has happened….

You may also note that the detailed SkyView users guide is now available directly from the home page. You don’t need to go into the help pages to see it.

Internally this version considerably reorganizes the various Mosaicker classes minimizing duplication of code. The listing of which files were used in the FITS headers has been somewhat reorganized and now gives the number of pixels resampled from each image.

As always if you have questions or comments please let us know.

Posted in Announce, releases | Tagged , , | Leave a comment

SkyView V3.3.1: Swift UVOT, UltraVista and CFHT HiPS data

Posted on October 17, 2017 by Tess Jaffe

We are about to release SkyView version 3.3.1.  This new version includes extended support for HiPS data and has many new surveys.  The primary new datasets are counts, intensity and exposure maps for the Swift UVOT instrument in seven filters.  All data for the first 11.5 years of the mission have been combined into HiPS for each filter.  For the most observed filters a bit under 3% of the sky is observed.  There is a fair bit of overlap but also many differences in the coverage of the various filters.

m81uvot

M81 in UVOT U, UVW1 and UVW2 filters

We have also added in a number of other HIPS surveys from the UltraVista and CFHT surveys. These data are just a few of the many data in Hiearchical Progessive Survey (HiPS) format that are currently available through the web. We anticipate making more available in future data. HiPS joins the VO Simple Image Access Protocol in enabling SkyView to access data from remote
sites.

Caching of HiPS data is now supported and some some issues that disabled special access to a UVOT overlay capability have been resolved.

Posted in Announce, releases | Tagged , , , , | 1 Comment

SkyView v3.3.0: Swift XRT surveys

Posted on July 24, 2017 by Tess Jaffe

Versions 3.3.0 of SkyView was released today.  This includes an all-sky mosaic of the Swift XRT observations.  Over 129,000 XRT observations were combined to give coverage of just under 6% of the sky.  A set of three surveys were produced: the combined counts, the total exposure, and the intensity map that is the ratio of the first two.   This image ofM31 combines the XRT intensity map with the WISE 22 micron image — contrasting the very different sources of X-ray and infrared emission.

XRT and WISE image of M31

XRT Intensity (green and blue) and WISE (red) in a 1.2 degree field centered on M31.

A few other changes were made in the release, notably many references to HTTP  on our sites were changed to HTTPS.   Although there should be an automated redirection, we are trying to purge the old HTTP addresses since occasionally they cause problems.

Posted in Announce, Notices, releases | Tagged , , | Leave a comment