We’ve just released version 3.1.0 as the default version for SkyView. This includes 5 new GOODS surveys — a total of 13 new bands — and a little new infrastructure to support them as discussed in our previous articles. The new data include the highest resolution and deepest surveys in SkyView but cover only a very small fraction of the sly.
We are currently in the process of rolling out 5 new surveys including 13 separate bands to enable SkyView users to take advantage of some of the deepest surveys that have been made of the sky. This is the biggest single addition to SkyView’s surveys other than our initial setup twenty years ago. These are surveys made of the GOODS (Great Observatories Origins Deep Survey) regions centered on the Hubble and Chandra Deep Fields. Getting all of the metadata and regression tests set up is taking a while, but users can try out the surveys now at our Version 3.1.0 site. Soon this will be the default.
The surveys we are starting with include:
- A VLA 1.4 GHz survey of the GOODS north field including data from all 4 VLA configurations.
This data is combined to produce a single image of the field.
- Spitzer MIPS 24 micron data of both the north and south
- Four Spitzer IRAC surveys at 8.0, 5.8, 4.5, and 3.6 microns. Each of the north and south regions is split into two pieces for each wavelength. Only one image is available at 4.5 and 3.6 microns in the southern GOODS region, so that coverage is a little less in those bands.
These data are provided in FITS files where only a fraction of the file is actually populated. We generated metadata that describes the rectangle inset in the FITS file where there is data. This rectangle is normally rotated with respect to the FITS image axes. We’ve added a new image finder that uses this metadata,
For the moment this means that the IRAC data cannot be accessed through SkyView-in-a-Jar but only at our Web site.
- Four HST ACS observations in B, V, I and Z filters. These data are of extremely high resolution and are provided as a set of 18 ~0.25 GB image tiles.
The full-resolution image tiles are quite large and it takes a fair while to process the full set for a given filter. To make it easy to intercompare this ACS data with lower resolution datasets, we developed SkyView‘s first scheme for hierarchical images where we combined pixels in the original data by factors of two to produce lower resolution data. When a user requests an ACS image, the lowest resolution image that has higher resolution than the user’s request is sampled. SkyView keeps the original data and pixel-combined data at factors of two up to 32. I.e., in our lowest resolution data each pixel is the average of a 32×32 pixel region in the original data — but that’s still better than 1″ per pixel. The standard
XMLSurvey class was updated to support hierarchies generally, but so far this has only been used for the ACS.
The same tiles are used for each resolution. A consequence is that images of large regions (i.e., 0.1 degrees or more) generally render faster since they are typically of lower resolution and can use the smaller files.
- Three Chandra ACIS bands. This includes a soft 0.5-2 keV band, a hard 2-8 keV band and the full 0.5-8 keV band. This represents about 2 megaseconds exposure in the north and 4 megaseconds in the south.
These surveys include what is by far the highest resolution and deepest data in the far IR, optical and X-ray regimes available in SkyView. The resolution of VLA data is only modestly better than for the FIRST data, but the data is much deeper and better sampled. Of course nothing comes for free. The GOODS regions cover only about 0.01% of the sky.
We’ll be adding more GOODS surveys after this initial set — if you have a suggestion or preference please let us know. Currently HST NICMOS data and ground observations from Gemini and the VLT are in the queue.
To try out the new GOODS surveys just go to the V3.1.0 site. You’ll see all of the GOODS surveys listed at the end of the main survey list in their own box — we put them all together rather than breaking them out by regime. You can also use these surveys in the contours and they work very nicely in generating RGB images with each other. When we complete our metadata updates and checks on these data we’ll make 3.1.0 the current version.
Next week we hope to begin releasing a whole new set of surveys in SkyView. These surveys are from the GOODS regions — the Great Observatories Origins Deep Survey. Two regions of the sky (northern and southern hemispheres), relatively devoid of stars, were selected for very deep observations by HST, Chandra, Spitzer and many other space and ground observatories. To make the surveys as deep as possible, only a tiny fraction of the sky (a few hundred arcminutes) are in the official GOODS regions, but some instruments have larger fields of view.
The GOODS data reflect a new approach for SkyView. We’ll be providing access to smaller regions and perhaps soon to non-survey observational data.
You can get a taste of the new data by looking our next version of SkyView. Look for the group of surveys labeled GOODS. Remember to enter coordinates in the GOODS regions. We’re in the process of testing right now. Here’s a color image (with something like real color) using Hubble ACS data of the Hubble deep field. Galaxies near and far predominate, with only a relative scattering of stars.
This image below (from 2014-01-19) isn’t a mystery. We’re seeing a bright star through some secondary optical path. The star is out of focus — as secondary images normally are — and we see shadows from the optical and support structures.
Can we tell what star it is? The nearest really bright star is Spica, which has coordinates (in decimal degrees) of about (RA,Dec)=(201.298,-11.161). This image is centered at (200.01,-9.356). That’s 2 degrees, a pretty long way away–maybe 4000 pixels in the image. But the DSS plates from which these data are taken are a full 6 degrees on a side, so Spica could be in the field of view.
In fact if we look at the plate that seems to enclose this location (labeled s719), it has a center at (200.658,-10.261). That’s just about the center of the two locations. It’s pretty common that there will be an artifact at a location like this with some kind of symmetry with the real location. So Spica it is. Sometimes the obvious answer is the right one.
This release fixes a configuration error which made the VLSSr data unavailable.
This is a minor release mostly to fix the small glitches and omissions we found in our first operational use of our new release process.
The one substantive change allows DSSx requests to continue when a corrupt tile is found in the underlying data. DSSx data are compressed in 500×500 or 768×768 pixel tiles that are restored as needed when rendering a DSS image. There are about 1000 tiles for each plate scan and we have a total of about 7,000 plate scans. About six years ago we tried to decompress all data and found 19 bad tiles. We were able to get 9 of them updated (see the 2008/02/06 entry in http://skyview.gsfc.nasa.gov/current/help/whatsnew.html). However users could run into one of the 10 remaining bad tiles which heretofore caused the entire SkyView request to fail. Now the bad tile is rendered as zeroes but processing continues and a warning message is given in the results. Thanks to Jim M for noting the problem.
Note that the address for the SkyView home page is now
http://skyview.gsfc.nasa.gov/current/cgi/titlepage.pl. This gives the current version. You can get to earlier versions by replacing
current with a specific version, currently
v3.0.1. We anticipate keeping at least the last few versions around when possible.
During the past 24 hours we have released our new SkyView organization discussed in our prior article. All of the old URLs for queries should continue to work though most are now redirected. We did run into a couple of surprises so that it took us longer to switch all requests over to the new system than expected.
There are a few visible changes to our home page, one new catalog, the Stripe82VLA survey, and a couple of changes that may mostly affect SkyView-in-a-Jar users. While incremental changes in SkyView have often been made with limited or no visibility, we now anticipate that any permanent change will be reflected in the version visible on the home page.
Please let us know if you run into any issues.
Next week we will be making a major new release of SkyView. This is primarily a structural release where we have centralized the organization of SkyView‘s files. Previously the operational SkyView services used code scattered through a number of directories: CGI, jar, scripts, documents…. Now all of these are collected under a single directory and relative references are used. A number of scripts and documents that were used in the operational system were brought fully within our code management system. So it will be easy to have multiple, independent versions of SkyView available simultaneously. SkyView releases will be delivered into directories named after the version used, e.g., this new release is designated
v3.0.0. [Version 1 of SkyView was our original IDL-based system, Version 2 refers to the existing organization of the Java-based code. Typically the middle number in the version increments when new functionality is released, while the third number is updated when we add surveys or make minor bug fixes or updates.] The
current directory will be linked to whatever is the current operational release. Typically users will only ever reference this
current directory but when a new release is made users will still be able to run the old release by explicitly specifying the version in the SkyView URL they invoke (or they can get a specific version of SkyView-in-a-Jar by downloading the jar from the appropriate directory).
While there will be new URL addresses for most pages, all the old addresses for data requests will be linked to the new locations so existing scripts should not be affected.
There are a few changes: The code for reading survey data has been changed so that if the survey data is 4-byte floating point, it need not be converted to 8-byte floating point. This allows use of much larger input files without exceeding memory constraints. It should also speed things up a bit. We’ve also added in some support for hierarchical imaging where we store an image a multiple resolutions. Both of these changes are in preparation for supporting a set of surveys in the GOOOS regions where some of the source data is very high resolution HST data.
We’ve also added one new small survey, the VLA Stripe 82 survey, a VLA survey of a region where the SDSS coverage is especially deep.
You can try out the new version right away. Next week we’ll begin redirecting the standard URLs here. Some of the old code that we have distributed to access SkyView programmatically does not handle redirect requests so the
pskcall URL it invokes uses a server side redirect that is transparent to the client. Other URLs will redirected using HTTP 301 permanent redirects that any modern clients should handle.
Please let us know if you have any concerns or if you run into any problems.
A couple of years ago the Sloan Digital Sky Survey underwent a major reorganization and completely revamped its data. One of the changes that was made in the transition from SDSS DR7 to SDSS DR8 was that new releases were compressed in BZIP2 format which provided a better overall compression ratio.
Unfortunately, the BZIP2 format is not supported by Java native IO libraries in the same way that the GZIP format is. So we needed to make some changes to SkyView to accommodate the new compression. Our first idea was to incorporate a new BZIP2 library into the SkyView JAR. There is a JAR file available through the Apache Commons (there’s a JAR file in the binary downloads) and we tried using that. It worked, but it didn’t take us long to notice that it was quite slow. Much slower than we were used to with GZIP files. Also, the Apache library although very liberally licensed is under a license unlike the other public domain libraries that we have incorporated into the jar (notably the ImageJ library we use to generate quick look images).
When we investigated using the bunzip2 command that we found on our system we noted that this seemed to be much faster — fast enough the decompression no longer dominated the time required to generate images. However it was unclear how commonly users would have access to bunzip2.
Given this situation we decided to provide a dual path to enable users to use bzip2 files. If there is a bunzip2 executable available, then the user can define a logical name BZIP_DECOMPRESSOR to point to that executable. SkyView will spawn off a process to run the decompression using that command. If you don’t tell it, SkyView does not look for the bunzip2 command in any specific location — we certainly don’t know where it would be found on a Windows or Mac though on Unix machines it often seems to be in
If there is no executable available, then SkyView will look for the Apache commons classes. However to minimize the licensing issues we did not include these in the JAR. It’s up to the user of the JAR to add them into the class path. Normally users will execute SkyView as an executable JAR. The executable JAR defines the classpath such that it’s hard to override by a user. Suppose you have the Apache Commons JAR as bzip2.jar, then SkyView needs to be run as
java -cp skyview.jar:bzip2.jar skyview.executive.Imager [regular arguments] …
to get the new compression capabilities into the path. You can’t just do
java -cp bzip2.jar skyview.jar [regular arguments]
So you can use bzip2 files within SkyView though it requires a little work and we might have found better ways to do this.
However we do need to apologize to our users for never documenting these changes in the user’s guide or in the blog till now. It doesn’t do much good to add a capability if no one knows how to use it!
If you have any questions about BZIP2 files please let us know.
SkyView is back up and running! We appreciate your patience while we were offline.
Here’s a random image submitted by a SkyView user to our Image Gallery. Enjoy!