Great Western Coffee Shop

Just to let members know ... few funnies going on.  I am away with limited access / just note may be some service issues

Hi Grahame.  Just to let you know (not a criticism) that message still cropping up at odd times saying 'database gone away' or 'unable to connect to server'.  Usually solved by retrying a few times.

Totally known ... but thanks for confirming it's not just me.

From where I am this week,  I dare not try too much; very limited access (no shell access to server, no scp, Lisa's vpn left her dead in the water.  And that's on what's called "the works".  A grumble to the provider here, and I'm told "funny you should ask - someone else was asking just this morning".

Pinging this to the top ... normal service should resume from Friday, perhaps with some engineering overrun to Saturday.  Problems due to lack shortage of technical admins on one of the rare occasions we needed them!

I am now back with regular access ... and have been looking around at flakiness during the week.  And I note from our logs that the Ahrefs Bot (https://ahrefs.com/robot) has been making up to 42,000 requests of our server per day - that's an average rate of one request for a page every 2 seconds!

So what is this robot?


Looking around, I see we're not the only ones to have this issue:


And whilst it may be of value to the advertising community, I'm not sure what value if any it adds to our site.

They describe themselves as a "good bot":
Now - Googlebot is certainly most welcome ... but in our site is far less active than AhrefsBot - averaging about 6,000 requests per day, with a peak of just over 10,000.   Me thinks that the AHRefsBots folks describe themselves as a "good bot" in the hope that none-technical people will believe them.   As far as we're concerned, they're aggressive users of bandwidth for purposes of their own business profit - harvesting data the harvesting of which is no use to us.  And indeed they've been so aggressive that they caused a great deal of worry while I was supposed to be forgetting everything for a week's holiday!

Unscheduled outage from about 04:30 for about 3 and a half hours - sorry folks.  My alarms went off but I was sleeping like a log.     Since the Ahrefs bot problems described a week ago and the banning of that naughty bot, the server has returned to a much more stable state.   I have been watching but refraining from making other adjustments until convinced with a good degree of certainty that it really was that loading that gave the truly massive problems I had the previous week.  Once I did wake, fix was minutes rather than hours and I would have wasted time (and left debris) had I shifted to our backup status page on another server.  So decision was few more minutes of no data for a quicker return.

Also been watching things like Google Analytics to ensure that there were no indirect consequences of banning the bot, and I'm delighted to confirm that telling it to go away has - thus far - had zero effect on visits.   Big sigh of relief as it confirm that Analytics data is a meaningful statistic.

I will be rebuilding some tables overnight after the backup system has started to play up in last hour.  Tiny chance of losing late evening posts.

As ever, grahame, may we offer you our very grateful thanks for your sterling efforts - usually behind the scenes - in maintaining the Coffee Shop forum.  ;) :D

I feel a bit of a fraud tonight ... no need to to do the rebuild in the end.

No worries - you can credit our collective thanks to your previous sleepless nights instead. ;)

(http://gougleri.com/wp-content/uploads/2018/01/fresh-kitchen-colors-together-with-diy-dining-table-set.jpg)

My pet hate.  Always seem to end up with bits left over.

Pocket screws ... the ones you put in your pocket after you've rebuilt the table?

Noting server down for a while this afternoon ... I have been offline and now changig trains - think I have kicked it back into order but need to run for train 2000.  Will check in later and see if it's AOK.

Think it is!   ;D

Database playing up a bit this morning ... as ever, running for a train.  Backuped at 02:30; suggest members keep copy of posts today as a precaution

SIGNIFCANT SERVER ISSUES - Please keep backup copies of anything you post for a while. 
Update to follow.   This is a test post.

And this is another test post

---

I have updated the MySQL server as the previous one appeared to be damaged, but there are incompatibly issues ... "we changed how it behaves at [version no.] so that is more standard compliant" ... pity it become less old-SMF compliant.

Thanks for all your hard work on this Grahame and getting things running before your latest eta

Yes, at least you get a message telling you whats happening and an eta.  More than some other websites I frequent.  Thanks once again, Grahame.

Many thanks, Gentlemen.  Since I am both the person working to fix the issue and the person working to inform the customers, keeping you informed and with realistic timeframes is frankly good self defence, as well as polite customer service.  And as you're all bright people, I'm not frightened to say "don't know how long" at the start.

Let me say that I'm clearing the signals only to double amber and not yet to green.  The database switch was from MySQL 5.6 to MySQL 5.7.23 and there are incompatabilies between the two.  My first test broke and I had to put in fixes to overcome the issue, which the MySQL folks state as being to make their language more compatible with the ISO SQL standard.    There may be other issues in more obscure areas of the software that we haven't yet found ... and I also need to do some further testing which I'll do during the night when we can go down to single line working ...

Grahame, Double Yellow please...... :) ;D

Sorry - forgot I'm in GWR land now.

fast main
yellow amber
buffet trolley
restaurant pullman
hideous beautiful
slow relief

Please forgive lots of silly posts - bashing the server ...watching performance etc.

Graham priceless

Sorry for outage during the night ... looks like I didn't get one of the server triggers right on the re-install yesterday.   Up and running now ... need to do a look-see to see what it looks like I have missed.

Yesterday - major earthquake under the running lines and we were back in a couple of hours.   Early hours, looks like the windscreen washer bottle ran out of water and caused a major upset ... oh, the irony!

There are some issues on automatic error recovery at the moment ... means that if the server wobbles I've got to do some hand cranking;  just tried the automatic process and it needed to be poked (visitors may have noticed we were down for a couple of minutes).   Likely I'll have a further go in the morning - been a busy day writing a major proposal so tired and likely to make silly mistakes if I try clever sort-outs tonight.

I suspect a follow up of the same incident - rebooted the server and it was very slow back up.   I suggest members KEEP COPIES of their text as they post today ...

This post something of a test to see that posting is working ...

It is  ;D ;D

I am currently getting intermittent connection problems.  I assume that this is a related issue

Yes - I am working it.

Probably rebooting / may be down for an hour or two during the night.

Don’t fall asleep in the middle of it!

A clear restart / reboot during the night ... and I think we're running OK (famous last words!).   

Technical note: The combination of software versions we're running with - old (open source) code logic on a bed of more recent operating system and utilities  - is not as stable as I would wish and occasional restarts tidy things up.   Work done to clear an instability the other day had left the restart procedure calling for a service that wasn't available so the other night the restart locked up while it waited - it timed out / gave up after 30 minutes or so; I made pessimistic provision in last night's estimates that this could happen again - however, it seems that the fix I had applied to replace the service call actually worked, so that the gap in service caused by the restart was/is no longer noticeable.

In English:  I can fall asleep quickly, but not as quickly as the server can restart  ;D ;D and in any case it's all working automatically again once the process is kicked off.

Famous last words, indeed ... I need to take a further look.   Good news is that the fiddling should not involve significant downtime.

Ooh ... good to see
when posting on the thread ... however, I've had a couple of funnies while doing some admin work and notice there's been a gap in posts - so have rebooted the server.   To some extent, this is a test post - as well as an apology if others have had issues.

Graham,
I am seeing some posts with just a white page, eg second page of the charlbury platform extensions is completely blank (white, no banner etc) but the index page is indicating that there are posts there.

Ian

Odd.  Let me have a play.

And if you say "print" ... you see the posts.  Noting an earlier post reporting problems with at attachment ... I suspect that's a clue.

I'm going to give you no confidence in what I'm doing by saying "never seen it do that before" ... but I am pretty sure the problem is limited to a single thread and won't spread.

Just done the same with me 21:.33 pm.

Is that on the same thread? 
http://www.firstgreatwestern.info/coffeeshop/index.php?topic=19529 works
http://www.firstgreatwestern.info/coffeeshop/index.php?topic=19529.15 fails

Now noticing "e.g." - looking for further examples.

.   
Yes should have made it clear apologies.

Phew!  ... I have my suspicions / will look during the night when the forum is quiet and I can add test code and turn debug flags on.  If anyone has any further examples please let me know.

Only page I have seen is page 2 of Charlbury.  It did not load last night, but I left it to see if it loaded today.

I've had the same issue with that thread.  I was going to post that it looks like the end of March aspiration for the new platform extensions opening is heading towards an end of April one!

I *think* it's fixed ... certainly looks right for me now.  I'll leave it 24 hours to see if I get any more reports and if not consider it closed.

The page is loading fine for me today.

OK - issue sorted.

Members are allowed to attach files of certain types (such as .jpg images and .pdf documents) to their posts, subject to a maximum file size.  When displayed on the bottom of a post for the first time, certain of these file types generate a thumbnail so readers can quickly have an idea of what's in the file before they download it.  This thumbnailing process can have rather a large server memory requirement - especially when it's a large .jpf file involved.

The other day, a .jpg was uploaded which was within the maximum file size, but caused the forum software to require more than the maximum memory it's allowed when thumbnailling, causing the server request to crash "blank screen", without even an error log or message.  As the thumbnail hadn't been created, every time anyone called for the particular post to be displayed in the thread and with the attachment offered, the server tried again and crashed again.  It was, however, happy to show the post if users selected to print the thread or see it on recents, as neither of those viewing methods offers thumbnails.

The "user solution" is for me to ask members not to upload images larger than 2k x 1.5k pixels. The one the other day was over 4k x 3k.  It just contained a lot of flat grey, so the compression applied in the .jpg algorithm in the user's camera had resulted in quite a small file that had passed the server's file size limit test.

I could - but won't - limit the file upload size; the maximum is set where it is for other documents and all I would be doing is fixing one very rare problem and in the process creating a limit that would frustrate more often.  If it happens again (once in 12 years so far), please let me know - it's easy for me to resize the rogue image now that I know what I'm looking for.

Thanks Graham, All working for me now. I was confused as I was sure that I had looked at that page before without problems.

Ian

Just an alert that I'm doing some big backups and tidies ... you may see some drop in performance over the next few hours (and you may have noticed we were offline for a short while - sorry about that)

OK ... done.  41 Gbyte download ... major occasional backup, including the stuff that hardly ever changes  but needs "checkpointing" from time to time.

Server offline (unannounced and unplanned) for a couple of hours during the night - sorry.  I know what happened / straightforward to bring back up.

I am "playing" with some server settings ... a game of balancing buffers. Be aware that server may be down (for a few minutes) and back up on occasions over the next day of two. If you're writing LONG posts ... good idea to copy and paste a backup to your local machine just in case before you submit.

Sorry about the outage what was (now) late yesterday evening.  Up and running again - this is a test post to check it's working again as well as an apology.

We had a major server crash in the early hours

I am in the process of recovering and testing ... all postings to the backup at 22:30 last night should be recovered.

Anything you posted after that backup may have been lost - sorry. I am going to try to sort through the crash debris, but more in hope than expectation.  I am NOT going to look back for personal messages after the backup point for privacy reasons.

Please keep local copies of anything you post for the next few hours (always a good thing to do with major posts anyway!)

There is a significant risk in the installation of a tool I would need to use to look into the debris so discretion has me leaving things alone;  the system restored to 22:30 - posts, personal messages, likes, logs of what's new for you after that point to somewhat after midnight won't be back.

Thanks again for your hard work Graham

Hear, Hear.

Another sleepless night Graham but back in time for “start of service”.

Sorry about downtime - significant database crash. Coming back up.

Greetings from Newport, Rhode Island  ;D - and test post

Trickling back ... should be restores as was at 14:10 checkpoint.   Some images still to be restored.   Limited access prevents full analysis - just concentrating on getting it going (and not missing last tender back to ship!)

OK ... some issues remain but I will need to look at them tomorrow (in next port).   Deep issues on which I have not trained / instructed a backup person - sorry, folks.

I blame Brexit!
Or possibly a drone attack, maybe even a swarm of drones.

More seriously, thank you for all your work.

I am noting - and I'm sure other members are too - that our server has been staggering a bit over the last few days, with dropped connections and various other messages from time to time.  I'm taking a look but no magic wand.

If you are making a significant post (i.e. one that's taken a time to write and you really don't want to have to rewrite), take a copy on your local system before you post please - just in case the post goes wrong. And should you accidentally make a duplicate post, don't worry too much - one of the moderators / admins will understand and remove the second copy.

If it helps these are the messages I have been getting:

Database Error: MySQL server has gone away
File: /home/firstgreatwestern/httpdocs/coffeeshop/Sources/Display.php
Line: 145

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

Sadly I am getting similar messages.   Tried most of the normal things.   Will try some abnormal things this evening but working today.

Ah, it wasn't only me! I've just double-posted (again) in another thread after I got the above messages or nothing seemed to happen when I pressed "post".

Marlburian

I am experimenting - 17:50 on 10.12.2019 .... please take copies of everything you post.

Still testing - but looks promising.  I have cleaned up a couple of tables that appeared to have got a bit dirty, and the database engine sounds like its purring not grating now.  For those of you concerned as to our long term future, our database engine is electric and indeed has never run on fossil fuel, nor produced particulates.

Presuming that your electricity is obtained from the grid, then some fossil fuel IS consumed and some particulates are emitted by the production of the electricity used by the server.
UK electricity is greener than that in many other countries, but by no means completely green.

You are, of course, correct. You also correctly assume our server is in the UK; may move from dedicates to the server to the cloud over Christmas, which is suspect uses less power per page served than the old server.  And in any case (wherever we choose) we are joining in with the electricity industry as they reduce to the point of elimination the use of fossil fuel.

It’s all about striking a balance over climate change or we will all be back in the caves and burning fossil fuel to keep warm.

Seems to be running OK now (well, no messages like before being received now).....and the site seems to be running a bit faster than before.  Thanks for your hard work Grahame.

Just out of interest, what happened to the forum software upgrade that was talked about last year.

'tis planned for Christmas ...

Ooo Christmas Engineering Work does it appear on JourneyCheck yet? I suppose that is to avoid having RRS. 

Server down at around 15:45 ... database issues.  Reverted to 15:00 backups - sorry that I have lost about 45 minutes of posts.

Coming back up / images and Avatars may be a few minutes yet

Not quite server issues - looks like our hosting company had a network issue a few minutes back; back now - looks solid but can't be 100% sure.   This is something of a test post.

Oops - I had to restore backups ... posts from 09:50 to about 10:45 - if you managed to get the in - will need reposting. Sorry.  Some of my images from overnight  now yet back.

Things seem a bit slow this morning. Might be my end, or does the system need a tot of something?

There has been an issue for about 12 hours.  It is not at your end, nor at our server, but somewhere in between - someone's router or routing tables somewhere are playing up.

I am accessing the server presently through my phone and it's running blisteringly fast.   I can jump onto the server via an intermediate machine from my normal connection, but getting there directly is intermittent / a problem, even using the lowest level "ping" commands.   I wondered whether to report this to members, but looking at server loading, it is little changed to I think it only effects a very people and I did / do not want to scare the majority for a small issue which I would expect to be transient - cured without any inputs from me.

My PC ground to a halt at 2100 last night, just as I'd cocked up quoting and replying to a post. Lots more people online, playing games and streaming.

Advice from OFCOM
 (http://www.ofcom.org.uk/phones-telecoms-and-internet/advice-for-consumers/stay-connectedhttps://www.ofcom.org.uk/phones-telecoms-and-internet/advice-for-consumers/stay-connected)

All nice and fast this morning

Unfortunately you cocked up entering that link - it should be this (https://www.ofcom.org.uk/phones-telecoms-and-internet/advice-for-consumers/stay-connected).

Thanks for that. Usually I check my links after posting (and on other forums at the preview stage, but the Coffee shop preview has never worked for me). But at that stage my PC had ground to a halt and I was unable to access any website. So I went to bed.

Issue ongoing and intermittent - hit it again a few minutes ago.  But by the time I did a "traceroute" it was back. Any report at the moment to Internet companies would be along the lines of "somewhere between Melksham and London - not near either - there's a dry joint in a circuit that occasionally flips out for a few seconds".   I have had no trouble going from Melksham to London via Fremont.

But was your journey necessary?   ;D

Virtually vital  :D

A couple of issues with our server which I'm working on over the next few hours - there may be the occasional signal checks on this afternoon's services as I try and get more quarts that usual in our pint pot.

Server is not running very cleanly today ... having a look around but please excuse connection issues I'm aware of.

It looks rather like Friday night after lockdown in Trowbridge. All the unwanted hooligans are out on the street and knocking on lots of server doors where they're not wanted, including ours.  Some are clever, and know how to waste resources by having us answer complex questions and I'm sure taking little note of the answers.  And all the time we have to keep an eye on them to ensure they're not doing anything nasty while we get them the reply.

I have put a couple of automatic door answering machines in, so certain visitors can be quickly given a "go away", and put a bouncer at the gate to weed out a couple of others.   Whether that will be enough, I don't know - I will have a look at the full 24 hour logs in the morning.

1. Some URLs on our server, such as complex searches, take a lot of cpu resources, and traffic on one in particular rose from around 400 to 600 a day to up to 1700 a day recent days compared to the same days last month.   These requests are very database intensive.

2. The noise of spurious server requests has rocketed in the last days - up 3 times from a pretty stable normal on Wednesday, and a further doubling (so now six times the norm) yesterday (Thursday).

3. One of the 'back' services we use appears to have stopped responding and that's resulted / resulting in queues of requests blocking available capacity with traffic waiting for timeouts.

Messages "Server too busy" yesterday were indeed accurate - sorry, folks who are "proper" users - that you have had significant problems over recent days.

This post is an analysis and not a promise of a solution. However, I have made some changes which may provide some relief while I look further.   

* I am hampered by the risk of closing / blocking doors which block genuine use as well as unwanted use.

* I am further hampered by looking for a commonality in unwanted use - looking for source and cutting it off there, when there probably is a coming together of multiple sources each bubbling over.   

* And once infected URLs are being requested, the requesting sources (often distributed so I can't just cut off a rogue remote address) that keeps on asking.

* Incoming rouge traffic is not uniformly predictable - whilst some tends to come in clumps, it also comes at just about any time of day and with some comes is underlying sporadic noise.  And because it's unpredictable, I'm hampered in testing any changes I make - often just a feeling that I'm getting somewhere.

I am a scientist. I would take my original situation as 'control', make one change, and see how it works before moving on to the next change.  But that takes time - more time that I/we have - so I am making multiple changes, quick-testing each and moving on to doing more; far from ideal and it precludes me from any definitive "that will sort it" statements.

In summary - I'm working on it.  Should be better today, fingers crossed - but we're not out of the woods yet.

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

Edited to correct typo in URL

It has run well over the weekend ... and I suspect that's not entirely a co-incidence.  I have:
* Banned a couple of aggressive crawlers
* Retired some older heavy database URLs that crawlers loved (%)
* Ceased back bus service calls which were hanging processes (%)
* Widened SQL door to allow more parallel connections
* Cleaned out accumulated database logs including heavy context setters
Actions marked (%) could usefully have slimmer/better alternatives put in place - change is a bit of a sticking plaster. I have also enhanced a couple of logs - hoping to see if more can be learned about current running and help in future analysis.

Still a "watching brief'.  Our databases back up from time to time, and there are certain times bookkeeping is going on in the background, which may effect access. We rather crudely lock tables while we back them up and hold requests until the backup copy is completed and if the holding pen is full of excited requests ("Flying Scotsman spotted at Goonbarrow" or "First day of hourly Oxford to Fawley via Trowbridge and Eastleigh service - pictures") we may have an issue.

Overnight issues - server down from around 03:00 to around 06:00 - now restored and "night sleeper" readers can now pass by, read and write.  Engineers just tidying up for an hour or so may results in some slow running.

This issue has come up again ... love your pictures but PLEASE resize massive (4k x 3k) images before attaching them to posts, or sent them to me to do to  ... Thanks!

A long time since I have posted here on this thread

Overnight - I'm investigating, but what looks like a denial of service attack of some sort - around 1500 requests from a single place in Texas, with over half of them taking (us) more than 20 seconds from request start to fulfilment completion.  Claiming to be a different model of browser on just about every request.  That IP now banned, server rebooted just to tidy up ... nothing lost ... steady as we go.

Ooh, exciting.
A new unannounced rail link opened up direct to the Scillies.
Well, looking at the map anyway  ;)

A relocation of Worcestershire Parkway High Level, which is abbreviated to Worcestershire Parkway Hl on certain data feeds and just as Worcestershire Parkway on others - only station with three different names in different places.