Server slow ...

[whole diary]

[grahame]

Noting that our server is suffering a denial of service attack - 44 requests PER SECOND.   Dealing with it as best I can from where I am (I'm on a train - in Lithuania!)

[grahame]

Noting that our server is suffering a denial of service attack - 44 requests PER SECOND.   Dealing with it as best I can from where I am (I'm on a train - in Lithuania!)

OK - dealt with ... the heavy black line is today's worker server load.  Looks dramatic - there was (perhaps) a handful of proper requests that may not have been answered but the server did not fall over.



If you wonder how I find out there's a problem like this, the server sends me a message 

Now in Vilnius

[grahame]

"Server slow" but perhaps different reasons.  The number of spiders crawling our site has been creeping up - to some extent as expected once we moved from "http" to "https".   As I see individual spiders adding a significant load, I at taking a look at those that come onto my radar and asking (or forcing) some of the more noisy ones not to index our content. 

In terms of web searches, 19 out of 20 in the UK^▸ (United Kingdom) use Google and the Googlebot is allowed Coffee Shop access just as any human guest might be (so no indexing of things like Frequent posters) but others such as Yandex and Petal which are not widely used in the UK are turned away.  Ironically, Google does not put a big load at all on when indexing - I see it, but it's not effecting performance like (for example) the Petal crawler was earlier today.

It's not just web search indexers that crawl sites ... there are a variety of other companies indexing and selling data to their customers or making use of it within products - from plagiarism identifiers to AI feeds and Search Engine Optimisation tool and link reporters.  In some cases they may do us a bit of good, but as we're not driven by sales volume and income here, and we're very much UK based when they tend to index worldwide, that good if it's there tends to be tangential. 

It's a bit of an ongoing game - others play it too and have things like "Captcha" - "are you really human" test which we could do if we need to.   We do have some logic that spots aggressively or characteristically automated visiting without the need for users to tell us which boxes have motorcycles or stairs (or GWR^▸ (Great Western Railway) class 158s!) in them from time to time.   Please let me know if you get a "you are not really human" type message more than very rarely!

[grahame]

An update on this one - still being well and truly spidered and no single nasty culprit.  I have made a few ongoing adjustements while static here in Mosjoen - but on the road (or should I say rail?) again from tomorrow and I expect to be mobile for another week or so.

Receptionist and worker loads:





Views from just up the road - one looking forward and one looking back





And even here there are things that remind me of England, and of my home town

[grahame]

Our server load is going "through the roof".   But - it's a bit of a "careful what you wish for".  One of the hoped for consequences of going to https (six months ago!) was to make the pages more secure-looking and encourage crawlers ... and they are crawling.   I note Google and Alexa spiders have been especially active over the last couple of days and I am keeping an eye in others.  Here are the browsers that called for over 1000 Coffee Shop responses yesterday

Code:

14676 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
39550 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
5139 Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
7000 Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
3314 Mozilla/5.0 (compatible; BLEXBot/1.0; +https://help.seranking.com/en/blex-crawler)
1146 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
1290 meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
1806 Mozilla/5.0 (Macintosh; Intel Mac OS^▸ (Ordnance Survey) X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
1817 Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
1074 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36
3828 Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0
3575 newspaper/0.9.3.1

[grahame]

4th September 2025

A huge spike in traffic - described elsewhere - but it's starting to effect performance.   I *think* I can isolate a lot of the traffic involved over the next 24 hours - however, as I work on the system you may find that I over-trap some traffic.

If you get error messages for a few minutes over the next few days, or messages telling you ...

A quick answer from our receptionist

You have arrived in a spiderflood from a remote address. If you are a real user, and you keep getting this message, please get in touch with the admin team (address below).

please give me 30 minutes or so before your report it ....

[bobm]

For us armchair viewers you get a sense of the problem and how it is escalating when you see there have been 2497 "users" on line today - and the record since the forum started is 2526 and that was only a fortnight ago.

[grahame]

For us armchair viewers you get a sense of the problem and how it is escalating when you see there have been 2497 "users" on line today - and the record since the forum started is 2526 and that was only a fortnight ago.

To give you a comparison - this running traffic level in 10 times the maximum I would expect in any month. 

As a side issue with our (new, moved) hosting space provider,  I asked about adding resource to the virtual server as that would be something of a fix.    That would mean a move off the legacy contract which is in place and a rebuild - possible but prices quoted look like somewhere over (and perhaps well over) a doubling of monthly fees as well which is not an attractive solution, especially at it just raises the lid - potentially - until next time the pan overflows.   So just resizing is neither a long term nor a financial solution.

Edit to add - I will be following up with thoughts within members areas

[matth1j]

prices quoted look like somewhere over (and perhaps well over) a doubling of monthly fees

Who pays those fees - is it out of your own pocket Graham? I haven't see any adverts to fund the site.

[grahame]

prices quoted look like somewhere over (and perhaps well over) a doubling of monthly fees

Who pays those fees - is it out of your own pocket Graham? I haven't see any adverts to fund the site.

The Coffee Shop and other ran on spare capacity on the servers I used to use as "Well House Consultants" - my IT company that ran from 1996 until a few years ago, and those servers still host legacy sites though they become less relevant.  I love(d) doing the IT stuff and the transport stuff, so asnwe closed down the businesses I have taken on the server costs - for all the sites combined- as a hobby thing.     You can see all the various sites via https://www.sheepbingo.co.uk/error/errorpage.php and if things go almost totally belly-up, it's where you are taken.

My background / career was working with stuff like this, and in addition we have a fabulous voluntary team of moderators and admins, with a membership who are naturally helpful - so there is rarely need to buy extras in beyond coffee beans, and using open source software tends to employ more of my volunteer time than commercial software would,  but saves an awful lot of costs that would-be associated with a commercial and supported setup, and reduces our dependencies on others.

[Chris from Nailsea]

I continue to thank grahame for providing all of us with the Coffee Shop forum. 

... we have a fabulous voluntary team of moderators and admins, with a membership who are naturally helpful - so there is rarely need to buy extras in beyond coffee beans ...

As an aside, do you know how important coffee is?  Coffee is vital for survival.  Dinosaurs didn't have coffee, and look how that turned out.

CfN 

[grahame]

As an aside, do you know how important coffee is?  Coffee is vital for survival.  Dinosaurs didn't have coffee, and look how that turned out.

CfN 

Indeed - and as a more recent example, from Coffee Aid:

Largely through the efforts of the British East India Company and the Dutch East India Company, coffee became available in England no later than the 16th century according to Leonhard Rauwolf's 1583 account. The first coffeehouse in England was opened in St. Michael's Alley in Cornhill. The proprietor was Pasqua Rosée, the servant of Daniel Edwards, a trader in Turkish goods. Edwards imported the coffee and assisted Rosée in setting up the establishment. The Grand Cafe in Oxford is alleged to be the first Coffee House in England, opened in 1650 by a Jewish man named Jacob. It is still open today, but has since become a popular Wine Bar.

And I don't know anyone who was around before 1583 and is still around today - without cofeee they have all perished.

I note that the Grand Cafe has been open rather longer than our Coffee Shop - we will never catch them up.  However, they are now a wine bar and whilst we may do a bit of whining here too, I hope we always have a dominant positive vibe!

[grahame]

Information post - purely out of interest for members

The subject is "server slow" and not "server broken" - and it has been slow overnight ... such a classic case of an overenthusiastic automated crawler, from not one but a whole suite of hosts.



So that's up to 6 pages per second.  Where do they come from?  Let's look an example:

8.160.129.10 IP info & Whois online

Host Name: Undefined. Country: China. City: Beijing. Coordinates: lat: 39.911; lon: 116.395. Network: 8.160.128.0/22. Type: Hosting Spam rate: 28.00%

We have a wonderful resource at the Coffee Shop ... which means a wonderful range of pages published - and the word "public" is there in the very word "published".   I have tools which help me notice - often without but sometimes with manual intervention - these heavy automated uses, even if (as in this case) the requests pretend to be human / don't admit to being crawlers.   They may not be noticeable looking at each individual hit ... but I can see patterns



Note - the first report is BST and the second GMT - so they are at different times; the second report is after I referred 8.160 requests back to the receptionist to reduce automata load on our worker server.  The changes last December which added the receptionist have given me this capability 


EDIT to add - about an hour later - here is a "classic" denial of service 'attack' just flagged up

[ChrisB]

I got a non-responsive return when requesting this page just now....

[grahame]

I got a non-responsive return when requesting this page just now....

Since 03:30 this morning on our worker server:

on our worker server 56123 requests of which
38313 good requests
17708 planned diversions     
102 erroneous requests
0 server failures

on our receptionist server 126024 requests of which
106093 good requests
18940 planned diversions       
777 erroneous requests     
214 server failures including (ChrisB) your "server gone away" response

214 in over 100,000 is more than I like - but it is only about 1 in 500 requests.  I will echo - good idea to copy / paste locally long posts before you submit and that's general internet advice.