Many thanks. I meant 6 hours ago, so Wednesday evening. So it looks like it was just the sheer amount of incoming traffic was stopping me getting through at the time. As you say, it is not a vital life support system, so I didn't wait to see if it would connect a bit later.

 

Posted at 03:55 on 2nd ... by yesterday, did you mean about 6 hours ago (late on 1st) or 30 hours ago (September 30th)?

The Coffee Shop's worker server was offline for a few hours over the turn of the month - late on 30th and early hours of 1st;  as far as analysis and digging back through history looks, shere volume of traffic filled our various queue buffers and something broke.  Worrisome, very rare indeed, but no damage to the infrastructure or databases and it was easy to restart and test in the early hours.   It would have taken longer to flag a problem than to just get it running again which took me seconds once I knew.

The deeper issue is the question as to how we handle all this traffic, which to a very great extent we encourage this year through our https rather that http protocol, and through various header settings made in response pages.  AI bots for services like Alexa have 300,000 messages to comb and update from time to time and are dripping away, as are a number of others.  I would rather not shout out about the ways I'm dealing with it - not that any of them will read, notice and act on the explanation ... we can cope with a request every second of the day but where one goes rogue (and we have had up to 20 requests joining the queue in a single second) it gets interesting

Long answer to a short question.

Ongoing advice is to keep a copy of the text of big posts until they are published when you refresh pages and see them.

The Coffee Shop is NOT a vital life support system or financial account site - no need for it to be as robust as a railway signalling system, a banking app, or a heart monitor,  with all the extra costs of going up from 99.5% operational to 99.999%.

What is now "yesterday evening" - and looking at it now - so that's late on 1st and into the early hours or 2nd October, traffic volumes have been silly-high again. I don't see the server actually being offline but the queues for service have been through the roof.   Our logging system notes the queue length every 4 minutes and it has been dramatic - though easing off over the last couple of hours.     You may have seen an indication that the worker was offline, but in fact it wasn't - just that the reponse you (failed to) get can be so slow that your browser gives up.

In real life, queues are regulated by the people joining them. Have you ever been to an attraction, seen a long queue, and decided not to bother?   That does not happen when accessing a web site, because you don't know how long the queue is ... and the net effect is that once incoming requests exceed capacity to handle them, queues build rapidly and we see this sort of thing.

The openness of the web, and our desire to be visible, means we are heavily crawled by automata that want to index us / make use of our valuable public data.  Most are "well mannered" and appreciate that asking for a page every few seconds makes sense and is polite, but just the odd one can be misprogrammed, and some twits out there are headstrong and make lots of urgent requests onto servers which in practise generate little of interest to them - they feel they are more important than they are to us at times. 

 
Did it go offline again yesterday evening? I tried to log in but couldn't.

 
Not directly related to the incident a month ago, but the worker server went offline late last night, but is now back - actually back about an hour ago but as it's still the early hours and we're quiet I terms of users from the UK, I have been watching it to check rather than rushing out an update.  Looks OK, but we do have severe loading issues which I am watching with a view to tweaking some settings.  Advice up this thread to take a copy of big posts before you submit them still applies.

 
I agree with grahame's advice to 'copy and keep' locally anything of substance you want to post out on the forum.

I have been caught out myself, posting what I personally thought was a rather witty topic - only to find that it was lost, due to the occasional vagaries of the internet provider we use.

Grahame continues to have my full support in his dealing with such issues. 

 
Backroom server has [been] rebooked twice today - not sure how or why;  logged question with our server space provider to see if it was their infrastructure that triggered it.   Advise to keep a local copy of what you post is repeated and emphasised until I know more.

 
Our main (and now back office) hosting has been with UK2 for several decades - the Coffee Shop was added as a side-site onto our the real (physical) server for Well House Consultants hosted there.   A long relationship and a good support team.  When I retired from the hotel and training we transferred to a cloud server with the same company, based on such things having come in and being a much reduced (monthly) price.  Not so much that my/our traffic had gone down, but that technology had moved forward, and that the cloud server was/is cost effective.

UK2 has moved on too.   It's now "a THG Company" ... and about this time last year they switched from their own email systems to Stackmail which was not entirely smooth.

They (and by which I think I mean THG) has move the server side of the business to "THG Ingenuity Cloud Services".  Previously, we were set up with a monthly payment made automatically with two emails - "we will be charging you" and a few days later "we have successfully renewed".  Looking back, these emails were not received late in July; an email from Ingenuity saying "your server has been transferred to us" at about that time appears to have gone into / been lost in spam, and there may have been one from the original UK2 control panel too.

Domains and email continue at UK2, and indeed a couple of annual charges have come though on some of those so the UK2 account is still live.   However, I hadn't noticed the lack of invoice(s) or payment for the cloud server, and the other evening that resulted in the hosting service "freezing" the server so that it became a read-only computer - effectively offline and only accessible to me as webmaster via a secure shell at a different IP address to the one we normally use. I became aware of the problem within minutes with both reports from members and from the receptionist server which is hosted elsewhere.

The UK2 / support ticket system no longer offered me a "help with your cloud server" option but I did put in a ticket and  member of the team there did try and find a fix, but had to "escalate to the data centre team" and told me that "I would need to allow them time".  And on that unhappy note I slept a bit, having posted a status report on the receptionist server.   And at 06:00 in the morning, I found the server was back up, and switched the receptionist to start to forward requests again.   A message in my status box told me that I hadn't paid for the service, and suggested I sign up with the new company to clear my bill and have the service continue.

---

I worry about signing up base on an email from a company / setup I have not dealt with before - however, I was assured that this was not a genuine setup / transfer, all be it not cleverly done.  And attempts to login and register with the new company failed because their verification email - as far as I can tell - was so much like spam that it never reached me.  So - in limbo with a server (temporarily?) up and running again and with a threat of being lost again when my bill remained unpaid.

Wind on to yesterday, and a phone call from "number withheld" while waiting inline at the pickup point in a local fast food takeaway ... to ask what I was doing about sorting out the account; bad line, and I am very poor of hearing - but it struck me that this MIGHT be genuine for once, and I asked the guy to call me back half an hour later - and it was at that point that the story of what had been going on became clear.  It seems that the part of our UK2 account relating to hosting could be and was transferred, but the payment details could not be and needed re-registering ...

---

Long chat with the new hosting support person - now from Manchester rather than docklands, and an invoice to pay with an invoice address in New York rather than in the UK - though in pounds and with UK VAT, and a system that's running again as I write and post but which was down for some eight hours and without warning that reached me (you could argue I should have been on the ball and noticed the lack of a request to pay).    I think we're OK for a week or two and I'll check back now that alerted to see if and how it settles.

I am inclined to stick with the new company for now rather than move in knee-jerk reaction.  I do have backups so that I can get back to where we were via a server elsewhere should I need to; that could take 24 to 48 hours.    We are promised "the same price as you have been paying" but it is not the exact equivalent to the new product being offered by Ingenuity, and I enquired about resizing - could also be useful to up certain specs.   However, the new company seems to lack quite the flexibility that would have needed - "your could purchase a bare server" but that would take us back to where we were in terms of extra cost and extra management we no longer need.   I am taking a look at the medium to long term and will throw options into the mix with the moderator / admin team over coming days.  No panic, I don't think, in the short term - but a need to plan for medium and long term.

Management summary - the Coffee Shop servers are still running.  We need to look at clouds on the horizon - as always, considering the medium and longer term in the moving IT field to ensure we remain operational and able to respond to incidents and changes.

 
I have been speaking with our web space provider trying to work out what happened the other evening. And it may happen again over the next few days.  Take copies of what you post, and I will tale regular offsite backups

It appears that the web space provider company has restructured and servers like ours are with a new sister outfit - however, payment details and systems have not been transferred. ...