The Xamin Command Line Interface (CLI) and Data Products

We recently reviewed how data products are handled in the Xamin Command Line Interface and feel that some information on how to download data products might be helpful. (For help getting set up to use the CLI interface see our documentation.)

Let’s start with an example. The following command will return table results from the suzamaster table for object cen a:

java -jar users.jar table=suzamaster position="cen a" fields=name,ra,dec

The output is:

name |ra |dec |exposure
Cen A |13 25 28.68|-43 01 03.7|64675.90000
CENTAURUS_A|13 25 21.86|-43 04 19.2|62432.80000
CENTAURUS_A|13 25 19.73|-43 04 07.7|51278.70000
CENTAURUS_A|13 25 18.74|-43 03 53.6|55944.20000
CENTAURUS A|13 25 27.60|-43 01 09.0| -99.00000
CENTAURUS A|13 25 27.60|-43 01 09.0| -99.00000
Number of rows: 6
Number of columns: 4
—- Messages —-
Info: Search Radius
Search radius: 10′ from suzamaster

Info: Name resolution
Position cen a corresponds to J2000 decimal coordinates:201.365063 -43.019112
—————–

If you want to list the data products for each row add the parameter “products” to the command:

java -jar users.jar table=suzamaster position=”cen a” fields=name,ra,dec products

The output for this command is:

name |ra |dec |exposure
Cen A |13 25 28.68|-43 01 03.7|64675.90000

# Product suzaku.obs: Observation Directory
> /FTP/suzaku/data/obs/1/100005010/

# Product link.suzaxislog: Suzaku XIS Configuration Log
> query?table=suzaxislog&constraint=obsid=100005010

CENTAURUS_A|13 25 21.86|-43 04 19.2|62432.80000

# Product suzaku.obs: Observation Directory
> /FTP/suzaku/data/obs/7/704018010/

# Product link.suzaxislog: Suzaku XIS Configuration Log
> query?table=suzaxislog&constraint=obsid=704018010

CENTAURUS_A|13 25 19.73|-43 04 07.7|51278.70000

# Product suzaku.obs: Observation Directory
> /FTP/suzaku/data/obs/7/704018020/

# Product link.suzaxislog: Suzaku XIS Configuration Log
> query?table=suzaxislog&constraint=obsid=704018020

CENTAURUS_A|13 25 18.74|-43 03 53.6|55944.20000

# Product suzaku.obs: Observation Directory
> /FTP/suzaku/data/obs/7/704018030/

# Product link.suzaxislog: Suzaku XIS Configuration Log
> query?table=suzaxislog&constraint=obsid=704018030

CENTAURUS A|13 25 27.60|-43 01 09.0| -99.00000
CENTAURUS A|13 25 27.60|-43 01 09.0| -99.00000

Number of rows: 6
Number of columns: 8

—- Messages —-
Info: Search Radius
Search radius: 10′ from suzamaster

Info: Name resolution
Position cen a corresponds to J2000 decimal coordinates:201.365063 -43.019112
—————–

As you can see, if data products are available then product names and partial URLs strings are listed below each row of the table results. These URL strings can used with a command line data transfer tool (eg. wget, curl) to download data from the HEASARC server.

In this example there are two types of URLs – one type is used to download data products (eg., > /FTP/suzaku/data/obs/7/704018030/) and the other type (eg., > query?table=suzaxislog&constraint=obsid=100005010) is used to display data from related tables.

For data product downloads an example of a wget command for Unix type machines would be:

wget -q -nH -r -l0 -c -N -np -R ‘index*’ -erobots=off –-retr-symlinks –cut-dirs=1 http://heasarc.gsfc.nasa.gov/FTP/suzaku/data/obs/1//100005010/

If you are interested in what each of the wget options does take a look at this recent blog post.

If you would like to download data products at the same time that you run the java command the following Unix shell script example should work:

#!/bin/sh
java -jar users.jar table=swiftxrlog position=ngc4626 products | \
grep '>' | \
sort | \
uniq | \
sed -e "s/ > \/FTP/wget -q -nH --cut-dirs=6 -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks http\:\/\/heasarc.gsfc.nasa.gov\/FTP/" | \
sh

The URL string for Link Products (e.g., link.suzaxislog) can be used to view Xamin catalog results for related tables. An example of this type of command is:

wget -O suzaxislog_results “http://heasarc.gsfc.nasa.gov/xamin/query?table=ascamaster&radius=60&position=22+02+43.2,42+16+48″

The output is a list of rows from the ascamaster table for object cen a.

We will be creating a post soon with download instructions for users who may not have access to all the commands (eg, sed) used in our example shell script.

Xamin Data Products Download Commands

In addition to the option to download data products as a tar file Xamin will display a list of separate download commands. Users can then transfer individual data files from the command line or create a download script.

These download commands are listed under the Wget Commands Tab of the Products Cart as seen in this screen grab:

wgetcommands

wget is a very useful command with many, many options and it took some time to determine which options were needed to traverse the complicated HEASARC data products directory structure to find and transfer the products.

The following is one of the wget commands listed in the Products Cart.

wget -q -nH -r -l0 -c -N -np -R 'index*' -erobots=off --retr-symlinks --cut-dirs=7 http://heasarc.gsfc.nasa.gov/FTP/chandra/data/science/ao01/cat5//1828/primary/acisf01828_001N004_fov1.fits.gz

We thought some of our users might be interested in knowing exactly what these options do. We used GNU Wget online manual for most of our research.

-q: quiet mode, no output is displayed

-nH: directories created on the user’s system will not be named using the HEASARC host name

-r -10: directories will be searched recursively up to 10 levels down

-c: files only partially downloaded from a previous execution will continue to to downloaded

-N: a file will not be downloaded if a local file of the same name exists and is newer

-np: the search will not ascend up to the parent directory during recursion

-R ‘index*’: unnecessary files will not be downloaded

-erobots=off: do not observe directives from the robots.text file that might prevent downloads

–retr-symlinks: symbolic links are traversed and the pointed-to files are retrieved

–cut-dirs=N: skip N number of the components in the URL directory structure when the directory structure is created on user’s system

Querying Vizier From Xamin

The Vizier system hosted at the CDS is an unmatched source for astronomical tables with nearly 10,000 resources and over 20,000 tables. All public Vizier tables are queryable through Xamin.

 

First you have to select a table (or tables).

If you know the designation of the Vizier table you can just enter it directly. E.g., the USNO B catalog is catalog I/284 and you can just type that name in the ‘Enter table name’ box of the Tables Explorer. For the first couple of characters as you type, you may see a menu of HEASARC tables that start with the same letters. You can just ignore these.

Selecting tables with direct entry

Entering a Vizier table name directly

Hit return after you’ve entered the table name.

Or you can use the Available tables tree. All Vizier tables are listed within the External tables tree. We open the External tables. Vizier tables are the first entry. If we open this up we get the nine Vizier catalog type categories, with Roman numeral prefixes I through IX, mirrored catalogs, B, and many catalogs associated with specific journals. These have a prefix the starts with J and includes a contraction of the journal name, e.g., J/ApJ for Astrophysical Journal. We can open up any of these. If there are more than 50 resources included for that category, then these resources are broken up into subcategories in the appropriate order (e.g., the resource number for the numbered tables, or by journal volume for journal tables) into groups of 25. If we need to, we open up this level of the tree to see the actual Vizier resources.

Some resources include multiple tables so that they are still branch nodes in the table tree. They can be opened up to get the individual tables for the resource. Other Vizier resources have only a single table, so the resource is a leaf node.

Selecting tables with direct entry

Selecting a Vizier table using the Tables tree

 

Once you see the table you want you can double click it, click on the + icon or drag it into the selected tables area to select it.

If you know a bit about what you are looking for, or have some specific information like a bibcode, it may be more efficient to enter that in the Table Explorers, Search by Keywords box. That will look for matching Vizier entries, and you can select the table from the pane that come up when you click on the nearby Search button.

Selecting tables with direct entry Finding a Vizier table using the keyword search]

 

Just click on the + icon to select a table from the list of returned tables.

You can use all three of these mechanisms to select Vizier tables in the Tables Explorer. If you want to get documentation on a Vizier table, you can click on the resource name in either the Available or Selected tables trees. The documentation shows at the bottom of the query pane. You may want to enlarge that part by dragging the divider up to view the document.

Once you’ve selected the table you can, in principle, do any kind of query. However only positional constraints and relatively simple constraints on columns can be handled by Vizier itself. If you want to specify complex constraints or a cross-correlation, Xamin may try to download the table and use a local copy. For many Vizier tables with a few hundred of thousand rows, that will work fine, but Vizier has many very large tables and so a naive query may time out.

 

Let us know if you have any questions regarding Vizier queries in Xamin.