Server slow ...

[whole diary]

[grahame]

For us armchair viewers you get a sense of the problem and how it is escalating when you see there have been 2497 "users" on line today - and the record since the forum started is 2526 and that was only a fortnight ago.

Adjustments to our algorithms have reduced those numbers in recent days to about a third of what they were when you commented on the peak, BobM.   But it's an ongoing 'game', and care needs to be taken to ensure we don't throw out the baby with the bathwater - or in this case the member with the unwanted automated spiders, some of which are masters of disguise.

[grahame]

For us armchair viewers you get a sense of the problem and how it is escalating when you see there have been 2497 "users" on line today - and the record since the forum started is 2526 and that was only a fortnight ago.

Adjustments to our algorithms have reduced those numbers in recent days to about a third of what they were when you commented on the peak, BobM.   But it's an ongoing 'game', and care needs to be taken to ensure we don't throw out the baby with the bathwater - or in this case the member with the unwanted automated spiders, some of which are masters of disguise.

Some traffic levels up 3000% ... yes, 30 times what they were a week ago ... and no great surprise that the responses have been a bit slow over the weekend.  In fact the surprise, perhaps, is that the server has "toughed it out".   

[GBM]

You could perhaps offer your server to the various airlines currently struggling with an attack! 

[grahame]

Positive headine - 4,000 requests in the last 24 hours for pages from UK^▸ (United Kingdom) addresses which I can identify as probably being from real people looking at pages on via main server.  That's a fantastic ongoing number - it's actual pages to Coffee Shop content and is in addition to pages served "light" by the receptionist server which I will look to quantify in due course.

But that is 4,000 requests out of 200,000 (or out of 350,000 if you add in requests for style sheets, images, Jacascript file etc) and out of 700,000 requests to the receptionist!    I am posting this update on the "server slow" thread because it's all those extra numbers that are or have been flooding out the real responses for our substantial numbers of members and human guests - posting as something of an explanation.

For us armchair viewers you get a sense of the problem and how it is escalating when you see there have been 2497 "users" on line today - and the record since the forum started is 2526 and that was only a fortnight ago.

For the 4,000 requests I guesstimate we should be showing between 50 and 200 users online in that statistic - varying depending on time of day.  We have a few members and guests active in the early hours, and many more during the day.  We do need to add to that the visits from useful indexing crawlers such as the GoogleBot which are vital to us finding new people to look at our data.  Ideally, the numbers showing for "users" would have a running rate of between 350 and 700, and our servers "spec" should allow for - my guess - around 1,000 without serious response degradation.    Currently, numbers have returned to being in that ideal / target range with some filtering added in very recently.

[Witham Bobby]

Again, I thank you for all the effort you put in to keep this lively and interesting discussion site going.  Grand job!

[grahame]

Brief update - a few days ago (at the turn of the month) we hit 1 million requests of our receptionist server in a 24 hour period. Of those, over 350,000 were passed on to the worker server.   Requests yesterday were down to 750,000 of the receptionist with just over 250,000 passed on to the worker server.   

By taking requests that are clearly not in our interest to invests heavy resource into answering, and sending back brief responses politely saying "nothing much for you here", we have been able to persuade some of the requesters to limit their asking - hence the drop in traffic levels.   I am not resting on my laurels here and there remain occasions - particularly when the databases are backing up, and when an "overenthusiastic" automata gets going - that still push our worker server's capability to the extreme. 

[matth1j]

I guess the attached screenshot is a symptom of this sort of problem?

[grahame]

I guess the attached screenshot is a symptom of this sort of problem?

Yes ... some very interesting patterns at present; research ongoing.

[grahame]

I guess the attached screenshot is a symptom of this sort of problem?

Yes ... some very interesting patterns at present; research ongoing.

I have made a number of changes / tweaks over the past fortnight, and symptoms like that should now occur only very, very rarely.  Many of the changes made mean that our front of house receptionist / manager is now picking up loads more casual and repeated enquiries rather than passing them on though to our worker server.  This (Adobe, license, bought!) image is one I'll be using to help explain the various servers into the future.



But, sadly, I have failed to identify what it is that's causing the spikes and ironically because the worker server now has a lower general load the spike can climb further before it cuts off.  You will at times see slow responses.  With the worker so busy during these spikes, it's almost impossible to log in to it and look around, the problem being gone before I can actually have that look around.  There are *some* helpful patterns, but I'll need to shut down some segments of service to help isolate the problem.  Good news for members is that I can do that when the Coffee Shop is quiet - that is during the night - there may be occasions in the early hours in the next week or two when responses will be abnormal.

[CyclingSid]

I think I must have done something it didn't like:

Warning: mysqli_query(): MySQL server has gone away in /home/firstgreatwestern/include/ipfind.inc on line 41

Warning: mysqli_query(): Error reading result set's header in /home/firstgreatwestern/include/ipfind.inc on line 41

Warning: mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, boolean given in /home/firstgreatwestern/include/ipfind.inc on line 42

Warning: mysqli_fetch_assoc() expects parameter 1 to be mysqli_result, boolean given in /home/firstgreatwestern/include/ipfind.inc on line 89
Connection Problems
Sorry, SMF^▸ (Simple Machines Forum - The software we use on this site) was unable to connect to the database. This may be caused by the server being busy. Please try again later.

[grahame]

I think I must have done something it didn't like:

Not you ... heavy loading.  Ongoing experiments looking into the load spikes. In the meantime, I repeat my suggestion that LONG posts are copied before you press "post" or "preview" just in case

[grahame]

I have made a number of changes / tweaks over the past fortnight, and symptoms like that should now occur only very, very rarely.  Many of the changes made mean that our front of house receptionist / manager is now picking up loads more casual and repeated enquiries rather than passing them on though to our worker server. 


But, sadly, I have failed to identify what it is that's causing the spikes and ironically because the worker server now has a lower general load the spike can climb further before it cuts off.  You will at times see slow responses.  With the worker so busy during these spikes, it's almost impossible to log in to it and look around, the problem being gone before I can actually have that look around.  There are *some* helpful patterns ....

I *think* I have now identified what's been causing the problem load spike issue, and in the process also made some other efficiency changes.

Main change / issue: We have 9Gb of archived documents (and that will continue to grow) for our search engine to look through, and over 300,000 individual public posts to check too. And such searches are none-trivial. They are welcome - however, it would appear that at least one well known search engine (not Google) has been making lots of search requests and indeed with terms that are WAY off topic. A couple of lines of code mean that we now send a quick and polite reply when an automata searches for things like (and these are real searches from the hours before I made the change):
* MyFloridaMarketPlace abbreviation
* housing lawyer in penang
* Napoleon Dynamite Live reviews
* melinda breslin townsend louisville ky
* 2m tv documentaire region du sud
* wer ist emilio frazzoli unfall

While I was at it ... I have an image library containing 20,000 plus different images, a handful of which are called up frequently and consistently.  They are managed by our worker server which is the one that loads heavy, but I have now mirrored around 50 of them on the receptionist server to allow that machine to answer queries without reference to the worker.    Yesterday, with that change in place, our receptionist served 3,000 images of which perhaps a third were done without reference to the worker.

Here's the worker loading as of an hour ago - solid black line is today's



There will continue to be spikes, including server management and database backup operations but I don't want to see them going off the scale.

[stuving]

Not just this site, of course. I've mentioned OpenRail before; the landing page there now says:

Recently the website has suffered from high loads, believed to be due to AI crawlers. Regrettably I have therefore had to restrict access to registered users.

It's a free and open registration, so only for load limiting purposes. Further down, it says:

Security Advice

Security is mediocre at best on this website. When choosing a username and password, don't select anything you use for other sites.

Which is candid, at least.

[grahame]

Fingers and toes have all been crossed all day - and thus far it looks good.  Some fingers remain crossed into the evening which has been the problem time in recent days