Server issues

[whole diary]

[grahame]

1. Some URLs on our server, such as complex searches, take a lot of cpu resources, and traffic on one in particular rose from around 400 to 600 a day to up to 1700 a day recent days compared to the same days last month.   These requests are very database intensive.

2. The noise of spurious server requests has rocketed in the last days - up 3 times from a pretty stable normal on Wednesday, and a further doubling (so now six times the norm) yesterday (Thursday).

3. One of the 'back' services we use appears to have stopped responding and that's resulted / resulting in queues of requests blocking available capacity with traffic waiting for timeouts.

Messages "Server too busy" yesterday were indeed accurate - sorry, folks who are "proper" users - that you have had significant problems over recent days.

This post is an analysis and not a promise of a solution. However, I have made some changes which may provide some relief while I look further.   

* I am hampered by the risk of closing / blocking doors which block genuine use as well as unwanted use.

* I am further hampered by looking for a commonality in unwanted use - looking for source and cutting it off there, when there probably is a coming together of multiple sources each bubbling over.   

* And once infected URLs are being requested, the requesting sources (often distributed so I can't just cut off a rogue remote address) that keeps on asking.

* Incoming rouge traffic is not uniformly predictable - whilst some tends to come in clumps, it also comes at just about any time of day and with some comes is underlying sporadic noise.  And because it's unpredictable, I'm hampered in testing any changes I make - often just a feeling that I'm getting somewhere.

I am a scientist. I would take my original situation as 'control', make one change, and see how it works before moving on to the next change.  But that takes time - more time that I/we have - so I am making multiple changes, quick-testing each and moving on to doing more; far from ideal and it precludes me from any definitive "that will sort it" statements.

In summary - I'm working on it.  Should be better today, fingers crossed - but we're not out of the woods yet.

[grahame]

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

Edited to correct typo in URL

[grahame]

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

It has run well over the weekend ... and I suspect that's not entirely a co-incidence.  I have:
* Banned a couple of aggressive crawlers
* Retired some older heavy database URLs that crawlers loved (%)
* Ceased back bus service calls which were hanging processes (%)
* Widened SQL door to allow more parallel connections
* Cleaned out accumulated database logs including heavy context setters
Actions marked (%) could usefully have slimmer/better alternatives put in place - change is a bit of a sticking plaster. I have also enhanced a couple of logs - hoping to see if more can be learned about current running and help in future analysis.

Still a "watching brief'.  Our databases back up from time to time, and there are certain times bookkeeping is going on in the background, which may effect access. We rather crudely lock tables while we back them up and hold requests until the backup copy is completed and if the holding pen is full of excited requests ("Flying Scotsman spotted at Goonbarrow" or "First day of hourly Oxford to Fawley via Trowbridge and Eastleigh service - pictures") we may have an issue.

[grahame]

Overnight issues - server down from around 03:00 to around 06:00 - now restored and "night sleeper" readers can now pass by, read and write.  Engineers just tidying up for an hour or so may results in some slow running.

[grahame]

This issue has come up again ... love your pictures but PLEASE resize massive (4k x 3k) images before attaching them to posts, or sent them to me to do to  ... Thanks!

Members are allowed to attach files of certain types (such as .jpg images and .pdf documents) to their posts, subject to a maximum file size.  When displayed on the bottom of a post for the first time, certain of these file types generate a thumbnail so readers can quickly have an idea of what's in the file before they download it.  This thumbnailing process can have rather a large server memory requirement - especially when it's a large .jpg file involved.

The other day, a .jpg was uploaded which was within the maximum file size, but caused the forum software to require more than the maximum memory it's allowed when thumbnailling, causing the server request to crash "blank screen", without even an error log or message.  As the thumbnail hadn't been created, every time anyone called for the particular post to be displayed in the thread and with the attachment offered, the server tried again and crashed again.  It was, however, happy to show the post if users selected to print the thread or see it on recents, as neither of those viewing methods offers thumbnails.

The "user solution" is for me to ask members not to upload images larger than 2k x 1.5k pixels. The one the other day was over 4k x 3k.  It just contained a lot of flat grey, so the compression applied in the .jpg algorithm in the user's camera had resulted in quite a small file that had passed the server's file size limit test.

I could - but won't - limit the file upload size; the maximum is set where it is for other documents and all I would be doing is fixing one very rare problem and in the process creating a limit that would frustrate more often.  If it happens again (once in 12 years so far), please let me know - it's easy for me to resize the rogue image now that I know what I'm looking for.

[grahame]

A long time since I have posted here on this thread

Overnight - I'm investigating, but what looks like a denial of service attack of some sort - around 1500 requests from a single place in Texas, with over half of them taking (us) more than 20 seconds from request start to fulfilment completion.  Claiming to be a different model of browser on just about every request.  That IP now banned, server rebooted just to tidy up ... nothing lost ... steady as we go.

[GBM]

Ooh, exciting.
A new unannounced rail link opened up direct to the Scillies.
Well, looking at the map anyway 

[grahame]

Ooh, exciting.
A new unannounced rail link opened up direct to the Scillies.
Well, looking at the map anyway 

A relocation of Worcestershire Parkway High Level, which is abbreviated to Worcestershire Parkway Hl on certain data feeds and just as Worcestershire Parkway on others - only station with three different names in different places.