For the past several weeks we have been seeing extremely heavy usage (>~ 1 request per second) with several distinct users sending tens of thousands of cone search requests in rapid succession. Unfortunately, they generally seem to be sending a series of simultaneous requests until things back up, rather than doing one or two streams of requests waiting for one request to finish before starting another. Generally we’ve been able to respond to these requests, but the response time has been awful and the system load has made the interactive interface largely unusable.
To address this we’ve added in a filtering of input cone search requests. If we have more than two requests going on a given server for the same table, then the new request will immediately fail. Since we have two servers there may still be as many as four queries executing on the same table.
In the few hours since we’ve implemented this we’ve noted that the interactive interfaces are doing much better, though not as crisp as we might like. In the near future we will be moving the database to a much faster machine which should address these problems more definitively.