ConeSearch usage filtering

For the past several weeks we have been seeing extremely heavy usage (>~ 1 request per second) with several distinct users sending tens of thousands of cone search requests in rapid succession. Unfortunately, they generally seem to be sending a series of simultaneous requests until things back up, rather than doing one or two streams of requests waiting for one request to finish before starting another. Generally we’ve been able to respond to these requests, but the response time has been awful and the system load has made the interactive interface largely unusable.

To address this we’ve added in a filtering of input cone search requests. If we have more than two requests going on a given server for the same table, then the new request will immediately fail. Since we have two servers there may still be as many as four queries executing on the same table.

In the few hours since we’ve implemented this we’ve noted that the interactive interfaces are doing much better, though not as crisp as we might like. In the near future we will be moving the database to a much faster machine which should address these problems more definitively.

Heavy Usage

This morning we had a user making very heavy usage of Xamin sending multiple cone search requests per second. Requests piled up in the database and this slowed down the system considerably. Requests that normally took a second could take several minutes. Metadata queries in the interactive versions of Xamin would often time out so that the Web interface was not really working.
We have temporarily disabled access for this IP address.

We’ll be looking at how we can meter usage more effectively in the future and apologize to those adversely affected.