For the past several weeks we have been seeing extremely heavy usage (>~ 1 request per second) with several distinct users sending tens of thousands of cone search requests in rapid succession. Unfortunately, they generally seem to be sending a series of simultaneous requests until things back up, rather than doing one or two streams of requests waiting for one request to finish before starting another. Generally we’ve been able to respond to these requests, but the response time has been awful and the system load has made the interactive interface largely unusable.
To address this we’ve added in a filtering of input cone search requests. If we have more than two requests going on a given server for the same table, then the new request will immediately fail. Since we have two servers there may still be as many as four queries executing on the same table.
In the few hours since we’ve implemented this we’ve noted that the interactive interfaces are doing much better, though not as crisp as we might like. In the near future we will be moving the database to a much faster machine which should address these problems more definitively.
The vast majority of the queries that the HEASARC receives are through automated interfaces that use either the VO cone search protocol or the Batch protocol we developed for Browse (long before the VO). Starting this week cone searches are running the Xamin interfaces. Usually there are more batch interface queries than cone searches, but there is a lot of variation. Yesterday we had a few more cone search requests and as a result Xamin satisfied more query requests than Browse for the first time, just under 10,000 Xamin queries versus just over 7,000 in Browse.
We’re looking at moving the Batch requests to Xamin in the next couple of weeks. Preliminary tests show it is up to 4x faster. When we do that Xamin will be handling over 90% of the HEASARC’s query load.
Starting next week, queries to the HEASARC using the Virtual Observatory Cone Search protocol will use our Xamin interface rather than a script that invokes Browse. While the results for queries should be the same, the VOTables returned are not byte identical (e.g., they include differing white space). For a few tables changes have been made to column definitions to meet the requirements of the cone search protocol. In some cases ID columns that were returned as integers are now considered to be strings to meet the spec. Please get in touch with us if you have any problems. Our internal tests show the Xamin queries running about twice as fast.