Coffee is back - now to work on the cream and sugar

[whole diary]

[grahame]

Gentlemen and ladies,

At around 06:15 yesterday morning - 25th March 2025 - the network centre in London that houses our worker server suffered an unplanned power outage, and the server crashed.   Badly, it seems - with lots of stuff running at the time and no chance for it to do a controlled shutdown.

I knew fairly soon that databases were damaged as the systems came back up ... but then I found that the file system had been mounted read only - so stuff could be read, but nothing at all saved.  For Unix / Linux geeks, I had to do an "fsck" telling the operating system to fix things as best it could, and then I had to delete and re-install the database engine as well as the backup data files.    Heavy stuff, and I had to find/refind all those configuration tweaks with character set and security authorisations between the various versions of software we run.

TEXT of Coffee Shop back - this is a test post.   Backups restored back to 05:03 yesterday morning, and no posts losts.

IMAGES that I use in my posts - not yet back. Next stage.

Please save a copy of anything important you write locally on your own system for the next 48 hours just in case anything else crops up.  Having to "fsck" with fixes frightens me but I think I am saved a full server reload.

[GBM]

Thank you so much for all your efforts Graham. Greatly appreciated

[Witham Bobby]

Well done, and thank you, Graham

[CyclingSid]

Thank you for your hard work. You appear to be doing better than LHR.

[Mark A]

For good measure it's... national datacentre day.

Mark

https://www.theregister.com/2025/03/26/international_datacenter_day/

[grahame]

My image handling system with the Coffee Shop also back and working.  Some images from before last July still need to be restored - but I am watching mode at the moment, and headed for early sleep.

[bradshaw]

Thank you Grahame, Marion and I had a similar situation after one of last year’s storms. The power went off and after it returned her MacMini would not boot up. We took it to a computer specialist who declared it defunct, saying he had not come across a similar thing before.
Luckily, we were able to claim on the house insurance, who also allowed us to increase the RAM at our cost. The result was that we had a new MacMini for a couple of hundred pounds.
We have always run Time Machine on our Macs and this was not damaged enabling the restoration of the files on the new computer.
We did add a new surge protector just in case.

[stuving]

For good measure it's... national datacentre day.

Mark

https://www.theregister.com/2025/03/26/international_datacenter_day/

And if today's not a convenient day for you, don't worry - there's a backup day - World Backup Day is next Monday! Sez who, you may ask. A bunch of much the same companies as for International (you were a bit parochial there, Mark) Datacenter Day - operators and suppliers of the places, mostly.

PS: and as a backup for World Backup Day, the following day is ...

[Electric train]

Grahame thank you for all the hard work getting the coffee back good and strong 

Sounds like the Data Centre had a Hythe End moment

[grahame]

The last 48 hours have been ... frantic ... and I *think* that just about everything is back.  Missing are imaged from my posts from about summer 2023 to summer 2024 - I know why and I would expect them to return, largely unnoticed, in the next few days.

A really interesting exercise.   The "story" for backing up is that you should always assume that your server is going to crash DURING the taking of backups, leaving the backups incomplete at the same time as the system itself is broken. That did not happen, and in any case we have a multiple level backup scheme, and the local backups on the server itself were almost sufficient as I only had to restore part of the system. Lesson for me - take careful note of all the config files scattered around the system - those are the hardest to re-create with obscure but tiny changes made over the years.    Having said which, I do have twice weekly backups of the config files but in precise restored these scattered changes from "first principles".

The principle of all of this - just like with the railways - is that it's very complex behind the scenes but that the customer should not care / be worried / need to know how it's done. Cathartic for me to write it up and to learn lessons, but as far as you members and guests are concerned,  no need to know and the paragraph above may have been written in Akkadian.   And good that it provided me with a refresh course in the sort of thing I used to teach.

This sort of unexpected event WILL happen from time to time. We at the Coffee Shop do not employ a full time team (heck, we don't employ anyone) and the cost and time of a full time, fall over on fail second system with all the complexity of keeping them in sync and ensuring the fail-over actually works would be an awful lot of resource for very little return.   The pragmatic "solution" is to have (a) backups and an ability to rebuild if worst comes to worst and (b) if it falls over get the message out there however it can be done that we are on to it, with updates as we go along.

The Coffee Shop - yeah - is part of my life these days, and judging by the returning post volume yesterday afternoon (even when it came up a few hours earlier than my final estimate) and the utter understanding from people so much confirms how this place is important to others too and what a wonderful bunch of people you are.  Thank you.

[GBM]

Dare I say that WE are only here because of what YOU set up!
Again, thanks seems not enough to say

[grahame]

Quoting from the thread title - final 2000 images and older "Horse's Mouth" and other old content restored before dawn this morning.  There are still various extra log files and intermediate workings for me to remember what I have done in the last few days, but that should not interrupt service.   Post and message backup systems for the Coffee Shop have been running since soon after we came back; fresh offsite mirror to follow on Monday.

Coffee is back - now to work on the cream and sugar