Server slow ...

[whole diary]

[grahame]

There have been a couple of "abnormal loads" on the worker server today.   I *am* aware and will look through the logs overnight.  The server emails me when it's in trouble and typical recovers over a few minutes when these things happen.

Normal request load is around 100 to 150 requests per minute. That rose to over 3,000 requests in the minutes that our server load spiked - automated requests systematically looking for URLs which are admin accesses to standard web server software that webmasters can accidentally leave open.   I have characterised the requests and our server should now turn away such requests more quickly. 

It probably wasn't designed as a specific attack on our server, and it wasn't on one of the main domains served - rather more I would see it as a piece of rogue software from a "naughty-boy" trawling the web through web sites that owners may have forgotten about / left insecure.

[grahame]

Our worker server has been overworked again - not sure of the details.   I have done something which I hope has brought it back - a test post suggest it's running.   I'm on a train and will have better connectivity in an hour or two.

[grahame]

Our worker server has been overworked again - not sure of the details.   I have done something which I hope has brought it back - a test post suggest it's running.   I'm on a train and will have better connectivity in an hour or two.

OK - better connectivity here and I had cleared a problem.  No obvious pattern with the quick look I have had, so this may of may not have fixed it.  Server load dropping again as I write, so promising.

[grahame]

Our worker server has been overworked again - not sure of the details.   I have done something which I hope has brought it back - a test post suggest it's running.   I'm on a train and will have better connectivity in an hour or two.

OK - better connectivity here and I had cleared a problem.  No obvious pattern with the quick look I have had, so this may of may not have fixed it.  Server load dropping again as I write, so promising.

Ironically, whilst I have cleared a couple of naughty-boys that were loading the server, I have not found whatever it or who it is that is spiking a couple of times every day - and because that thing is starting from a lower base after my cleanup of other issues, it's getting more headroom and spiking worse.

I am home tomorrow afternoon, Eurostar and Great Western permitting (feels a bit like double jeopardy) and will find a quiet corner with some canine company to take another look.   In the meantime, I suggest you copy and paste a local copy of long posts before you submit them, just in case your post falls onto a spike.

[Chris from Nailsea]

  ... In the meantime, I suggest you copy and paste a local copy of long posts before you submit them, just in case your post falls onto a spike.

Yes, I've learned to do that - the hard way. 

Many thanks for your ongoing work here on the Coffee Shop forum, grahame.

CfN^▸ (Chris from Nailsea, an administrator on this forum).

[grahame]

Just "pinging" this topic to let members know that I *am* aware (and working on it).  The traffic (on the worker server) has hugely increased; clues suggest that the extras are automata. Typically, we have 500 or 600 "guests" around in a 15 minute period, last night we had 8000 in one such period.  And, yes, there is an extra zero on the end of that number.

[TaplowGreen]

Just "pinging" this topic to let members know that I *am* aware (and working on it).  The traffic (on the worker server) has hugely increased; clues suggest that the extras are automata. Typically, we have 500 or 600 "guests" around in a 15 minute period, last night we had 8000 in one such period.  And, yes, there is an extra zero on the end of that number.

More "guests" than usual needing repair at the same time? 

[bobm]

The number on line peaked at 8771 earlier today.  That’s a rather unwelcome new record for the most on line at anyone time in the forum’s history. 

[grahame]

The number on line peaked at 8771 earlier today.  That’s a rather unwelcome new record for the most on line at anyone time in the forum’s history. 

Indeed ... and I found a common theme in many of the excess visitors and diverted them away ... for a couple of minutes until I realised that I fell into the common theme too and I had diverted myself away.

I am looking for other themes / patterns ... watch this space.

[grahame]

The number on line peaked at 8771 earlier today.  That’s a rather unwelcome new record for the most on line at anyone time in the forum’s history. 

Indeed ... and I found a common theme in many of the excess visitors and diverted them away ... for a couple of minutes until I realised that I fell into the common theme too and I had diverted myself away.

I am looking for other themes / patterns ... watch this space.

I believe I have found something - a web page that resulted in a significant server load each time it was accessed.  It wasn't actually called up all that many times, but once there were several automata looking at it at the same time, the requests flooded in quicker than they were answered.    I still have my fingers crossed - the load spikes seemed to correlate with the page being accessed, and now that I have put something in to reduce their loading there have been no more spikes.   All circumstantial, eyes still open.  A bit of a surprise as to what it was / not related to vast numbers of visitors. Time will let me be more sure.

[grahame]

This is - and will continue to be - an ongoing thread of pressures put on our servers by rogue (accidental or intentional) clients.  The old "Save the Train" website at http://www.savethetrain.org.uk usually gets around 4,000 accesses per day. Since 04:30 this morning, we have had 200,000 - 197,000 from a single location, up to 1,500 per second and that has had a noticeable effect on other sites on the machine, including the Coffee Shop.

I have added the "attacking" IP address to the forbidden list ... will not stop it asking, but it will be told to go away (and perhaps in a language it understands) with much less call on resource per request