Coffee Shop forum Server load - ongoing issues, being resolved by grahame

[whole diary]

[grahame]

Here is where we were a week ago:

Server load (just one of a number of monitors) from last night - ideally it should be at or below 1 job queuing at any time

The hard black line is yesterday ... the coloured lines are previous days

Here we are now:



And I'm much happier with how that has settled.

[grahame]

During July, traffic on our web server rocketed. On 30th June there were 263,933 accesses and on 31st July that was up to 597,214.  And the extras were ones that take significant server resource - requests that carry on sessions (my technical choice of measure) up from 93,049 to 492,532.  Our server IS being robust over this at present but there are some slow responses - members just please be aware.  Work underway to characterise the changes more finely and take any necessary follow up actions.

[grahame]

The server load is getting silly again with so many guests it's ridiculous ... and there's a lack of commonality in them such that there is no quick and easy solution to identify and block those that are clearly not genuine.  I do have a cunning plan - however it will require some experimentation and may result in some breaks of service and funny (peculiar) results for genuine users from time to time while I tune.   Please don't panic for an hour or two.

[Chris from Nailsea]

Blimey!  There are apparently Luddites even worse than me, then!   

Full support to you, grahame, to do whatever is necessary. 

[bobm]

I noticed the other day we clocked up the most ever "visitors" at the same time on the 6th August with 1936 on line.   Shame when things like this skew the real picture.

[grahame]

I noticed the other day we clocked up the most ever "visitors" at the same time on the 6th August with 1936 on line.   Shame when things like this skew the real picture.

I think we're down to around 15% of that

If you're genuinely here and see this occasionally - please let me know as it would mean I have overdone the hard shell protecting us

[grahame]

If you're genuinely here and see this occasionally - please let me know as it would mean I have overdone the hard shell protecting us

I have seen the message about in some extreme testing I have done, but had no reports of it occurring for members to the extent that it irritates.  In any case in clears if you retry a few minutes later.

More "sillies" over the past few days, now fixed - sorry if the Coffee Shop had a blocked hot water nozzle for you earlier this week and was spluttering.



It appears that this was not a new problem - simply that a couple of the ongoing denial of service requests interfered with each other and confused our triage nurse code. I have put her straight.

[grahame]

There appears to be another DDOS (distributed denial of service) attack underway - some silly spikes on our server load over recent days. I doubt that it's specifically aimed at us but it has given our server a lot more to do that usual and resulted in some sluggish responses to members.

Here is "now" ... with up to 1,000 requests per minute versus under a hundred in a normal minute ... requests arriving in flotillas from various IP addresses ....



Here is what it should look like



Edit to add - I have put in a filter to help identify the requests but won't be able to confirm its operation until the next flotilla of requests arrives.  The requests do not identify themselves as being from a spider or crawler and have a whole range of headers and as such appear to be designed to sneak under the radar.

[grahame]

There appears to be another DDOS (distributed denial of service) attack underway - some silly spikes on our server load over recent days. I doubt that it's specifically aimed at us but it has given our server a lot more to do that usual and resulted in some sluggish responses to members.

[snip]

Edit to add - I have put in a filter to help identify the requests but won't be able to confirm its operation until the next flotilla of requests arrives.  The requests do not identify themselves as being from a spider or crawler and have a whole range of headers and as such appear to be designed to sneak under the radar.

Well - that did something  . ...



200 = good request
302 = redirection
403 = forbidden
404 = page not found (hardly any of those)

We do appear (last 24 hours) to have had an over-enthusiastic search engine - it has made 7,000 requests since 03:30 this morning and I have asked it to cut the rate.  I have also asked a visitor from Tehran who's looking at ticket flows every 2 seconds and has called up 1000 so far to desist! We'll see how it goes

[grahame]

To the charming "person" who is asking for 40 pages a second off our server - 22,000 so far this morning ... please go away.  You are misusing an open source piece of software to try to scrape our website.  I have written an electronic version of this "please go away" message into our configuration file so that the request to go away will be enforced.

To everyone else - good to see how well our server stood up to this brute force onslaught, and to see how our server emailed me to report the unusual activity.

[Richard Fairhurst]

You've almost certainly thought of this, but if not: have you considered putting Cloudflare in front of the site?

It stops the vast majority of bots, including AI crawlers. It's pretty simple to set up (just a bit of DNS^▸ (Domain name service) config) and the Coffee Shop would happily run on its free tier.

(I'm not a massive fan of centralising the internet behind a small number of big American companies, but Cloudflare seem much more benign than Google, and I've kind of made peace with the idea of letting them fight the bots freeing me up to do more productive things.)

[grahame]

You've almost certainly thought of this, but if not: have you considered putting Cloudflare in front of the site?

It may be an interesting solution to certain of the other elements I'm looking at - thanks for the nudge. With some of the other changes I'm working on, a direct change / solution would only be short term and because of that probably not worth the bother.

[Chris from Nailsea]

I've now updated the heading of this particular topic, simply to make it less historic date specific - and to record that grahame is continually dealing with these issues.

Thanks, grahame! 

CfN