Great Western Coffee Shop

We've had problems in the last few hours with the databases that contain the forum data - I am currently investigating and this is both a test post and an alert to our members.

OK - that posted correctly.

As things stand at the moment, I have restored databases to the state they were in at around 00:30 this morning and I am doing further investigation.

Overnight posts and personal messages may NOT have been restored - please resubmit.   And as I'm still investigating what happened, please keep copies of what you post locally / say in case we have further problems today.    I will be clearer later!

Detailed investigation underway but I also have to get to my paid / day job - so further report may not be for a while.   Please keep posting / sorry about any data loss from 00:30 to around 07:15

Many thanks for your hard work behind the scenes bringing the coffee shop back on line.

OK ... I'm still checking logs to see if I can work out how the database went wrong in the early hours - probability is that it was a rare "race" type condition; I've seen it once or twice before, and I suspect a combination of rather old forum software and more modern databases and programming language versions won't have been tested together, so we use these versions at our peril. Other forums such as Wnxx have gone to the extent of archiving their old forum and starting a new one due to upgrade issues. As a forum that was formed for the specific purpose of providing customer information for a specific franchise / period, scant attention was given here to longevity of what I had expected to be a limited life project, and there are time such as this morning (and such as issues with search and some others I'm aware of) where it's showing its age.  I am now minded to think - "mission creep" - that the forum's likely to be required even beyond future franchise changes, and options should be explored; unlikely to be small tasks, and interesting as to how we're going to go about them - we have huge moderator, admin and member support team, but we're running on spare capacity on a machine that I happen to have that's got space, and with zero income.

The forum ... looks like the problem is fixed, and we are restored to the backup taken 20 minutes before it occurred.  As the problem wasn't noticed during the night, posts and personal messages (and poll votes and "new" flags) updated during this time will have been lost.    Our usual target for rollback is to loose no more than 60 minutes of data, but in this case we overshot.

Happy to lend an extra pair of eyes or hands if need be. Not very experienced with php but happy to (try and) help out as and when.  :)

We have a handful of active "techie/programmer" members, and it may be worth at some point getting our heads together to look at Network Rail data (improving feeds through this site, for example) and also looking at what we can do to develop and more into the future. 

For the immediate "oh sh*t there's a problem" stuff, it's probaby easier for me to be doing it as some of the changes that I've made to the source aren't well documented and I would need to spend longer explaining what needed to be done rather than doing it.

A nice import of members and messages (and polls?) into a new forum - with personal messages and the old forum available for a month or two - might be a way to go. But don't expect it soon from me.

I have a note of the following active members who are a bit / lot techie ... and I expect there are more. Please raise a hand if you're one.  We may well have a code bash when the days get longer? 

http://www.firstgreatwestern.info/coffeeshop/index.php?action=profile;u=117911
http://www.firstgreatwestern.info/coffeeshop/index.php?action=profile;u=94156
http://www.firstgreatwestern.info/coffeeshop/index.php?action=profile;u=348
http://www.firstgreatwestern.info/coffeeshop/index.php?action=profile;u=1

GraHam, looking at the profiles I think you have it covered but if you need any more help I have experience in coding and database experience

Dave

I also work in IT with a techie background...

*raises hand*

Stuff to do with maps is my speciality (I do a lot in the OpenStreetMap project), and generally Ruby and Postgres are my weapons of choice.

I much appreciate the waving of those hand in the air - thank you;  as we look forward (and with questions of ageing software and providing what people want in the future) I'm very likely to come back to you, and I've noted this thread.  Other people - please add your names; it may look like there's a lot of expertise around here, and indeed there is - but it's a question of having people 1. ready, 2. willing and 3. able and that combination's needs to be AND not OR.  So a big pool quickly can turn to just a handful!

This thread, grown from the need to restore backups yesterday, wasn't preplanned at this time.  In fact I personally fall at the first hurdle - "ready" - if we were looking to move ahead at the moment.   Far too much to do, far too little time; plans are for that to improve at / after the TransWilts AGM - obviously it's up to members and directors how that goes. I need to find time for looking after my own [income generating] work / business rather than just running it, and there are so many people who will come out and help on the day, including some really skilled and knowledgable people, several of whom are newly retired / moving from full time work, and can be far more committed and reliable than I can.

Is there a case for a subscription forum? Discuss. 
Actually, the pro's and con's are obvious, and would take up a whole subsection with very heated debate no doubt.
As an occasional user, I find the debates fascinating and very informative.
It's run by volunteers on a shoestring.
Would funding help with an improved server and software and programming. 
Funding would then need a subscription secretary (voluntary or paid?)
I would be happy to contribute (funding-wise) just for the pleasure and information herein.
The purpose of a forum is to provide a debating platform which should be free for all.
Recently retired, I know nothing about programming (et al), sorry; not sure how I could be of any assistance.
Thank you to grahame, the mods and posters.

I personally do not have a problem paying a modest annual subscription BUT I'm not sure how well that would work generally (maybe we need a separate thread for this). My reason for saying this is that (a) it introduces a layer of administration as mentioned by GBM and (b) I think it may deter casual posters. I wouldn't be surprised , for example, if 20% of the membership are responsible for 80% of the posts.

However before we go down the subs/non subs route I think I would be good to have feedback from Graham and the mods for their view on how they think the forum environment could be improved.

Let's be honest, it's a lot more reliable that the subject matter it is set up to discuss :)

I also appreciate the excellent work that the team do - free of charge - to keep this going. Thanks team

I'm happy to lob in my two penn'orth as a PHP/MySQL/Wordpress developer...

The problem with charging a subscription is not only does it add to the administration tasks it also introduces new responsibilities.

Thankfully failures here are rare but if they were to occur with subscriptions in place we'd then be dealing with extending memberships or offering refunds.

A couple of forums I suscribe to ask for donations. That's a subtle difference - you give with no expectation of getting anything back.

However both methods do change what we have here and unless the running costs become prohibitive I am not sure they are roads we want to go down.

Subscriptions or donations is not a road I'd like to go down either.

I'm more amenable to advertising if its done properly and the forum doesn't offer dubious blue pills or devices to enlarge ones member.

A poll/vote to canvass opinion on advertising before such a step is taken. Any excess income donated to a good cause, but better managed than RailUK's recent attempt to dispose of excess income!

What is obvious to some, may not be obvious to others.  I would be very surprised to find a heated debate - if there was to be such a debate it would probably be between me and my wife (on one side) as we host it on the server that the company of which we're principals hires, at no charge and (on the other side) all of the members and guests who make use of the forum, and (unless we offered something spectacular) would probably prefer a free service.

It's a very interesting metric to look at our members and consider whether their contribution in terms of time here is net "give" or net "take" - are they getting more out than they're putting in, or vice versa?  And while we remain subscription, donation and advertising free (apart from my "hosted by" link, which you might just consider an advert), I think most people who come here will say they get more than they give ... some give a huge amount (and THANK YOU) to them, but then they get a lot of pleasure back out.

Where a subscription [or other funding] could be considered would if "we" went off in a different direction or someone else did - and we're not necessarily looking at a forum there, or even purely web based stuff. Some very interesting community rail discussions can come up from that, and the thought that elements of what community rail has done to "put new life into local lines" (the ACoRP strap line) could usefully extend to smaller stations and services on mainer lines, to commuter flows including London, and indeed to long distance and regional services. It's all a network, and even the "local" line feeds the network.  See my blog at http://www.wellho.net/mouth/4371_Why-are-people-using-the-TransWilts-.html for a scatter map of where TransWilts traffic ends were on our 11 to 13 October survey, and if you're a TransWilts CRP supporting member on this forum, you can see that data plotted dy by day and against all UK stations - at http://www.firstgreatwestern.info/coffeeshop/index.php?topic=15130.0 .   Actually, with my Community Rail hat on (at least until the end of this month), looking after some of the press and publicity aspects, there is some funding to cover advertising and very occasional ancilliary payments. Further, we're putting up departure screens in one of our towns for trains and buses, and they are funded partly by an area board grant and partly by the Melksham Railway Development group - which made a profit of ^40 on the Santa train last month.


It's run by volunteers for free.  Not sure where the shoestring came from ;-)

Funding - "would it improve xxxx".

Server - for what we're doing, server is more than adequate.  We would be unwise to replace the 150 we're currently travelling on, with seats for all and a comradely feeling, with a 180.  Maintenenace requirements would go up, it would feel very empty at times, and the extra speed really wouldn't be used because limits are line speed and the time people take to read a page, not the speed of delivery.

Software - probably not. We run with open source software which is free at the point of distribution and for which we are not required to pay a license fee.  Sounds curious - even more curious is that I earn my living from open source software - by training people on how to use it, for example.  This week I'm giving a Ruby Course (http://www.wellho.net/course/ruby.html) (and adding some Cucumber), next week it's Java (http://www.wellho.net/course/java.html) and I've then got public and private Python (http://www.wellho.net/course/python.html) courses prior to the end of the month.  I could envisage the need / desirability for something niche, and a consideration to pay in the future, but would be wary of doing so.

Programming - probably not.  These days, unless you're starting something pretty radical there's not much need to write programs - rather you use some else's open source (which they will update in time for free for you, perhaps!) and limit your own coding to interesting extra bells and whistles.

Ironically, there are things you didn's mention that funding can / could buy.

Backups. Admin. Placement. Logos. Graphics. - Although even here on many areas we have members and supporters who enjoy doing the work for the pleasure rather than the money.  For example, the graphics artwork for the Community Rail Partnership leaflet ( http://atrebatia.info/TWR_A4L-Winter.pdf ) was all done for us free of charge (my wife is rather good at this sort of thing), and the printing was sponsored for us by Wiltshire Council - but the distribution to 24,000 homes on this occasion via the local newspaper was paid at a rate advantageous to us (we have marvellous volunteers who deliver, but different methodologies help reach people in different ways, and we don't want to induce volunteering fatigue).  Backup spending - we protect against some things on our server, but could gain from extra protections against other circumstances, and there's an admin overhead on this too potentially. And by "placement" I mean marketing to reach people we don't currently reach.   When you look at it, 1600 members isn't relly very many when compared to the number of travellers on FGW trains each day, though it's so much more than our local rail user group with 100, (and more active too - they meet 4 times a year rather than all day, every day) - so perhaps some work on search engine placement (we can do that) and paid advertising would reach more people who would want to join.  Again taking the CRP as an example - we've spent money on Facebook promotion - well less than 0.1p per passenger journey BUT we know this has informed people and brought custom.


I'm glad you find it such - and indeed if members didn't, they wouldn't be here.

This whole "free" business is an interesting one / on one hand you say it should be free and on the other you would be prepared to pay a subscription (and you rightly point out the can of worms this would bring).  It would also bring an expectation of level of service (noting BobM's post following, but I dn't think that would be an issue - I believe we would be able to meet an uptime percentage that would make the rail industry jealous, and wellin excess of offered targets.

Free as in Freedom [to post] is another matter. No, you can't post pictures that are other's copyright without their permission, personal attacks on other members, adverts for products to enhance your manhood, pyramid schemes, details of how to defraud the railways or which chemicals are best to use to make vandalsim permanent.  Yes, we have removed all of those in the past, but such posts are a tiny, tiny minority.  We do want to encourage new users and having to pay to post your - one - question about why the train from Westbury stops so far our at the end of the platform at Swindon is going to be a real turn off.


Oh I know how you can help.   The way you are doing by asking questions, posting, keeping the forum alive. And perhaps extend that a little by adding in som enew posts to update us on what's happening around the railway in your area, and in connection with it.  I think you're in Cornwall - that' s laong way from Wiltshire but we can listen and learn from each other - indeed we have huge time to look at, admire, and learn from the CRP activities and how they're organised and run in Devon and Cornwall.


There's a degree of "it ain't bust, don't fix it".  However - note from earlier in this long reply - comments about increasing membership and that would bring more active individual line boards and the chance of a few more special topic ones too.  I would personally like to see a very much more active fares advise / ask questions section and indeed talking about extending and ne areas, ticket sales are interesting - based on the profit on commissions going to community support for rail.  We bought the Santa train tickets from our local agent in Trowbridge - they had been selling Santa tickets too and that helps them / the local community.  However, commissions on such sales have been slashed and I don't think this is a flyer any more.   But watch this space.

---

A long answer there (and going off topic from the original post) ... hope it's a useful fill in.  Many of the issues raised (probably "most" or even "all") have come up with the moderators / admins in the past , and on that basis I feel quite comfortable to comment here and know that I'll be pretty close to the general view.

A really good discussion to have ...

Noted - THANK YOU.   I am keeping tabs / a note for once I get past the too-busy-to-even-ask point which should come in a few weeks, based on workload on my day job and other changes at the month end

Another hand up here.

Most days do Perl and MySQL development, occasionally dabble in some next-gen search technology stuff.  Previously used PHP quite extensively.  On top of web design, the other main part of my job is shifting  large (~100 whole animal genomes) amounts of data from one format to another, and/or from one database to another, to create data with "added value".

And noted, thank you.  That's gathering a good team - please (others) put your hands up too.

Going back a few posts to Graham's (volumus) reply I'm a big fan of (1) if it's not broke don't fix it and (2) I'm not a fan of change for changes sake.

Whilst I know the forum is run for free (using borrowed capacity on a server and the time of the mods who have real day jobs after all) let's face it it is pretty reliable. I reckon if it is out of service for a few hours twice a year then that's as bad as it gets.

@graham on the rare occasions it does fail is there a common cause ? Is that something any members can help with? Sometimes there is a risk that having too many helpers - with the best of intentions- can make things worse :)

Too few failures for there to be a pattern in which I can see any common cause  :D

There's the hugest of help from admins and moderators - without whom this site couldn't run.   Adding direct technical support to it as it stands - I suspect 2 steps forward would mean 3 steps back as we all got used to the new ways and I communicated things through and perhaps there was an accident or two because of something I hadn't passed on.  However, new stuff in some areas / working in zones could benefit our user community, and 10 steps forward would only mean 6 steps back, giving us a net gain - problem is time and cost of making 10 steps!

Much more later and we may set up a "techie" area - and how are 22, 28, 29 March in your diary, technical folks?   Too busy until next Monday though to discuss things in any depth.

I am implacably against compulsory subscriptions. They don't work, and in my experience have a negative effect on membership. HOWEVER. On a busy and long-standing forum that I manage (underground rock music) we offer the opportunity to make voluntary PayPal donations, which can be either one-off or regular small payments. PayPal merch store gives you all the code to do this so it's the work of a moment. We have several people who regularly pay ^3 or ^5 a month, and one gentleman who every year sends us $100 from his income tax refund. Going down that route would get a thumbs up from me. But I'd rather it stayed completely anonymous.

Back to original topic ... improved and extra backups (to the main site on the host) results in our space being completely full in the early hours and the server offline until around 05:00 ... a bit bouncy as we came back up, should now be running.

Another hand up here.

Though my expertise is a lot more niche I'm afraid - back-office software mostly written using Visual FoxPro to handle the erratic analysis of big databases.

Goodness - there's quite a few of you / us out there.

I'm drowning in other work at present - away from forum during the day.   I have posted (frequent posters) a note on a big thing that's causing me issues and looking for help at the end of next week / early weekend to kick through a lot of refactoring of an old Perl application into an MVC structure - see http://www.firstgreatwestern.info/coffeeshop/index.php?topic=15194.0 and (please) let me know if you fancy (and practically can) help.  Thanks

It looks like a similar issue to 6th January - however, in the last 3 weeks I have significantly improved the backup facilities we have and I suspect very little or no data loss.   I can quickly and easily get back to where we were an hour before the failure, and I should e notified of failures every hour - so that means a maximum of 2 hours backtracking if I'm around for alerts.  I'm able - and over the past 45 minutes have been - cutting off the posts  since the last backup and adding them on to the end of the restored messages.    All of which should NOT be necessary because data should not get damaged, which is worrying.

I HAVE also added a backup to a remote system so that we have a copy away from the main server - that's not done quite so often, but much more than it was.  And I suspect that "my" problems relate to loading.

Anyway - cutting a long story short,  I may upgrade the database next weekend, and - please - for this week keep local copies of long posts you make - "just in case" you understand, as I'm getting to be much more experience than I like at database knitting!

I started getting new post email notifications of old posts.....it was weird until I realised what had probably happened again....it's settled down again now.

Is it just me, or is there a new character encoding issue since this database recovery? There always have been some posts with obviously garbled characters, but now there are a lot more. While it's not something I'm really familiar with, it looks like UTF-8 being interpreted as ISO-8859-1 (which is what I see as charset= in the page source). I can't see any examples in "Recent Posts", so it may only affect recovered material.

Yes I had noticed it too, ironically in a quote in one of your posts among others, (http://www.firstgreatwestern.info/coffeeshop/index.php?topic=14408.msg169335#msg169335 (http://www.firstgreatwestern.info/coffeeshop/index.php?topic=14408.msg169335#msg169335), but as you say recent posts appear to be ok.
[like]

The database we're running with is a far more recent one than the one that the forum software was written with, and there have been a handful of backups and restores which may have included an (accidental) character set change.   The best I can realistically do at this point is say "sorry, folks" about some of the stranger old encoding - too much on my plate to realistically look at sorting it out.   I WILL take a quick look to see if there's a quick fix, but please don't hold your breath.

A further follow up ... you will occasionally see the following:


Usually clears within a couple of minutes / usually caused by the databases being locked for backup.  Playing far safer these days - this is the 'cost' of having a more robust ability to restore in the event of a problem.  Yes, other solutions are available but at the cost of more setup and admin time and perhaps expense.

Techie / update issues that we were looking at / wondering about a "working weekend" or get-together.  I HAD hoped to suggest a late March date but my personal diary is crammed - and this is in addition to running a hotel and presenting 4 training courses of up to 5 days each during the month.  So we will need to plan for later - looking at June or even August.

Film 4th Westbury
MRDG 6th Melksham
TWSW 7th Taunton
((Seed Swap 7th Melksham))
(Dog Walk 14th Melksham)
ACoRP 16th and 17th Nottingham
Visit Wiltshire 18th Salisbury
FGW Community 19th Bristol
Mayors Reception 20th Melksham
TransWilts Link 21st Melksham
Customer Panel (1st Bus) 24th Bath
Transport Group 27th / 28th Melksham
CIC board 28th Melksham
RailFuture 11th April Melksham

Rule number 188356 of IT... Never do any big changes on Friday 13th... It will end in disaster :-X

Just a note that I have adjusted some of the scope of backups, and moved some evening work into the early hours. Call it fine tuning; I anticipate little effect will be seen - like a 3 minute improvement in journey time from Bath Spa to London would be useful in the long run, but not noticed by the passengers using the service ;D

For some reason - yet to be fully investigated - our database engine quit an hour or two back; I get flagged when there's something like this (it's part of the heartbeat monitor) so knew quite quickly, but the issue was unusual so I've take a while to research it ... doubly frustrating on a very slow and laggy connection, and with two sister just across from me having a flaming argument while I was trying to do it.   Anyway, they came across and apologised for the disturbance and as recompense I send them to the real "Coffee shop" to get me a cup.

I'll follow up further in just a moment - also using this as a test post to check we're properly back

OK ... next system checkpoint scheduled in about 20 mins and I'll be back

Looks ok ...

Spoken a little too soon - a further database unexpected stop about an hour ago; up and running again after what I think may be a fix applied.   However, I still have to be researching issues and this potentially could lead to a period with some frustration as the root cause is hard to find ... and if and when we find it or manage to put in a work around, we won't necessarily know that we have found it!

Sounds like the signals in the Thames Valley!!! :D

Topics that you have read in the last couple of hours may now be showing up as unread again.

I read them, but cannot remember whether I've read them before.  :P :-[ ::) :o ;D

Again - anyone who's read topics from 21:30 last night to 02:45 this morning.  (This message doubles up as a test post)

Noting some database issues this morning.   I will be looking to fix at around 10:30 when I have a good connection - please note posts, messages etc posted in next 2 hours may be lost.

Posting to test we're back up.

Looking good ... If anyone changed an ancient post after 8 a.m. this morning, your changes may have been lost.  Pretty improbable, me thinks.

Due to database issues, the server won't be available for a period of up to one hour later today; I'll try to be as quick as possible and choose a quiet time.

OK .. I think I managed that in a few minutes.   Any edits made to posts over a year old between 20:00 last night and 13:00 today will have been lost.   I don't expect there were any  ;)

Forum live again - but further backups happening over next half hour / may be a bit sluggish.

Thanks, as ever, for your magical work 'behind the scenes', grahame.  ;) :D ;D

Further issues / some tests done.   Currently watching how the server's performing.

Just a brief note (and not strictly databases) and an apology ... forum missing from 18:39 to 19:59.  Probably due to a relentless buildup of log files and other stuff (such as check pointed backs) which lead to the server being rather full. You folks will keep on using this forum  ;D

Sorry, we can't help it. Such a great place to be.  :)

Purely in the interests of continuity and clarity, I've now merged a couple of topics here - and renamed it accordingly.  ;)

Another of those occasions this morning - fixed.   Any posts from 10:18 to about 10:40 may need to be re-submitted - sorry about that.   A couple of stn age character have crept back in, but I can fix them next time and they won't overwhelm us again.

Sorry about the 10 minute outage - just back.    You may find special characters "breeding" again - don't worry; I'll take them out next time. I only realised once I'd taken the database down that the trick code was ... there in the database!

Something got seriously damaged on our database server ... I have rolled back forum to 16:18 and I am about to roll back the main set on this server - http://www.wellho.net - from which many images are used on the Coffee Shop.

Thanks, yet again, for your update and hard work 'behind the scenes', grahame.  ;)

I've just taken the opportunity to update the heading for this particular topic - purely in the interests of accuracy and completeness. ;D

A few problems this morning - sorry.  Just put a quick fix in (from a 153 between Chippenham and Swindon). Will check back later.

It's obviously occurred because Graham has passed 14,000 posts!  ;D

Seems to be OK now!  :)

Can we please move to a more stable platform?

The frequent down times are making the use of the forum less and less enjoyable.

I had some time pertinent info I wished to post last evening. Little point doing so now.

We have a thorough and frequent backup procedure in place, and as the forum has grown and the server got busier, there are becoming times that no connection is available.  At these times, it's not so much of a stability issue but rather like turning up at the station expeccting to find there's always a train waiting with spaces available, and finding that the only one there is already filled.  I will take a look at renicing the backups to help, but changing the platform to sort it ... (a) what do you suggest we switch to, (b) Who's resource will do it, espcially data conversion, and (c) it would be a real "bull at a gate" way if we can re-arrange the background stuff instaed.

There *are* some occasional database issues - I'm sure they're not what you're taking about as they're effecting us perhaps once a month, and you said "frequent", and there wasn't such an issue last night - there was, though, this morning and hence my comment on this thread.  I would dearly like to avoid these completely, but little information / evidence is left around to tell me what's happening;  I should probably take the server down, install a fresh database version and fresh PHP ... but I wouldn't like to predict if that would result in no change, an improvement, or fresh problems as I made the changes which forced me to roll back the work I did.  Complete guess.

I have looked at Google Ananlytics since 1st January, and I can only see one major disturbance visible in the pattern - therefore concude thet it'a mainly the first issues that's frustrating you.

Sorry, there was.

Yesterday evening at around 9pm I tried to read unread posts. That could not be done. 'Database error' message. I contacted an administrator as the message requested, to say the forum was down. There were no posts between 8.55pm last night and 10.08am this morning. Very quiet or inability to post? I certainly couldn't post, logging in from different locations (home and mobile 4G), just to check the problem wasn't my end.

Same if I clicked on individual boards. Couldn't read them. I could only read posts by going to 'Forum Stats' at the foot of the homepage and clicking on 'View the most recent posts on the forum'.

Unable to start a topic either as the 'Database Error' was shown.

Of all forums I contribute to, this one has more down time than others. Hence it is a more frequent occurrence compared to them. Comparing this forum's down times to trains at stations is a specious analogy and not at all relevant.

I spend a lot of time on this forum. Would you like me to record all instances when there are 'Database error' or 'Connection Problem' messages? They are regular and frequent. More so in recent months than the distant past. I not sure whatever analytics you are running are picking up these frequent problems.

I fully appreciate the forum is hosted by yourself grahame and you bear all the costs and have to fix the regular problems. There are ways of addressing that though.

For the purposes of understanding the issue and not wishing to put pressure on you Graham. I too as unable to access posts on the forum from about 8:30pm last night though strangely I could get the home page. 

I'm not privy to the inner workings for this forum but could the home page work because it does not have dynamic content which needs to be pulled from the database?

To confirm - there are two issues.  One is connections running out (in effect server overload) during backups and I have added further backups recently, and the database has grown - which tips the balance.  These connection errors can show up in odd ways as (I think) the software grabs several connections and / or takes them over from previous connections in the same thread.  This can be fixed be reducting the backups / non backing up / lowering the pririty of the backups, and I am inclined to do the latter as previously explained - but not in the middle of the day.

The second issue is the odd database faults.   I was on the forum soon after midlight last night, for about an hour.  I didn't attempt to post anything but had no issues in what I did.  I'm aware of problems just after breakfast time this morning, which I think I've fixed but as I was travelling it was a quick and finger-crossed fixed.

It seems from reports (and there are now several of them, and I believe yous!) the problem actulaly occurred yesterday evening, went away or was in areas I didn't explore, and came back this morning.   I'm not meaning to sound 'odd' about that - it reads improbable, but it probably did happen.

First thing for me to do is to hit the overloads. They may well be the root of the problem - the trigger. I have, however, another train to catch and there's a big gap to the next one so be back later.

Personally, I think it needs saying that I'm perfectly happy with Graham's offering to us....it's his time & more importantly, cost, that keeps this going.

As I don't contribute to this time or cost, I'm happy with what is offered by way of up-time.

If I was paying towards this, then comments upthread would apply, but as I'm not, they don't.

Thanks for keeping this going Graham!

Seconded. Couldn't agree more!

Many thanks for all those +ve messages ... overnight I'll take a look at the backup schedule and priority which I think may trigger the failures as well as running connections out.  As the problem isn't triggerable (and appears to right itself sometimes!) though I'm on educated guesses so cannot promise a fast solution - nor to know when it's fixed.  One of those with potential to be longwinded.

I do appreciate the apparent frustration that started this current discussion I would feel a bit the same ... I don't think you would be surprised, bignosemac, to know that I find this issue frustrating too and would really rather not have the issue(s) manifesting as it/they are.

One of the very unfortunate things about running a forum to which the public can post is that you have to be defensive - not only to protect your data from corruption by sotware bugs, but also from user (and admin!) errors, from automata that are erronously or maliciously loading the site, from users and potential users looking to use the sit efor purposes outside your stated remit (and perhaps outside the law) and indeed from the occasional 'inside' user who goes off the rails.  We have a number of flags and check processes running to help alert us to things goind wrong, a number of tools we can pull in when we suspect a problem to analyse and/or gather extra data and send up email alerts. At this point I can assure you I have zero evidence of any current / recent / ongoing forum / server activity arisng from any members, though one person who we had to ban here 7 years ago has been around and with some not-entirely-positive stuff on a page I look after hosted elsewhere.

Mechanisms are in place not only to alert us to issues should they occur, but also to ensure that (should they occur) we have a current "checkpoint" / data we can revert back to.  For the mostpart, these are "backups" - data copies taken from time to time so that we can roll back to a known point if the need arises.  There's a considerable degree of pragmatism needed here - judgement as to risks than can be taken and how far back data can slip in particular circumstances.  I'm sure there are those who will remind me that with a second server we could mirror all data, and that with a third server we could introduce a system which alleviated problems where one system went bad and damaged the second - but I would remind them of extra time resource and cost in doing this, which could be considerable.

As it stands, our main databases are backed up on the server three times a day, our most dynamic tables twice an hour, and our web sites and local OS application installations twice a week, wth a further full backup once a month. Certain backups are automatically copied to another system in a different location so that we can still get running again even if there's a catastrophic failure - but the data will almost certainly be more than just 30 minutes lost.

As time goes by, resources build up. You're looking at 190,000 posts and 12,000 personal messages in the dynamic tables. Plus 10,000 images (tend to be quite large) ... and at present the compressed backups are around 50 Mbytes for dynamic, 3.7 Gbytes for databases and 29 Gbytes for web sites on our shared server with local OS folders.  Not only do these backup operations take space, but they take io and cpu time, and compression takes noticable cpu resource. Each table is locked while being backed up - done to ensure the integrity of each backup, and we have to keep jobs waiting for the lock to release if they come in at that time.  We then have to decide whether to do a short, sharp compression that will continue to effect the server significantly, or to compress at a lower priority having less effect for longer.

If you take a look at http://www.wellho.net/mouth/3027_What-proportion-of-our-web-traffic-is-robots-.html , you'll see how the loading of our system was in 2010 from an image I caputured then, and how it is now (Feb 2016 as I write). Overnight, I have moved around a number of the backup times to clear the 3 x per day from happening at peak time, through our peak is not as sharp as a commuter peak so there is still going to be some of this activity at busy-ish times. I do bear in mind schemes like update logging and incremental backups, and I would welcome input from one or two trusted techically competent members to brainstorm pragmatically through the various issues; I have done some further research (not published here) on which peer review would be good, and I'm trying to make just one set of changes at a time in order to see / learn the effect of each.   We chose with this forum to carry on with existing databases and software where others have archived old forumns and dropped historic content from easy access ... so we're certainly not alone in having to consider these issues.  I'm glad of the opportunity afforded by the request yesterday to take a look at this, and note the follow ups from the requester.  I still await an answer as to what software to switch over to and proposals as to how to resource that, but I suspect that such routes will be paved with significant resource requrements and implications. But I'm getting no younger and as and when the time comes I would look to seeing a new generation or a new method take on the forum's role, and if a robust, supported way of doing that (including technical skill and resources) should come from the member(s) who would like to see an end to resource limits and backup queues, then that's a positive step.   The irony of the whole thing is, of course, that less that 1 in 100 backups is even looked at after it's completed!

Keep up the good work Graham!

My pleasure - I'm touched.  We've only once had a post with more likes!

Maybe Platform 14 at BRI?

Yes, good idea https://s-media-cache-ak0.pinimg.com/236x/7f/35/93/7f3593b1f72e0889c6825de3f3943053.jpg  ;) :P

Indeed it has been stable - in that it has been stably absent for many years. 

Neigh?

Another outage this evening. Unsure for how long.

Server been very busy in last week / will analyse in morning.

Message details (i.e. how it fails) and exact time would be appreciate as that would make the reports potentially traceable - thanks.

Traffic on the server has been 50% up over the last 3 days compared to the January and February average; actually surprised how well it's coping.  Educated guess the issue last night was during a deep database backup as at such times we lock the database to ensure that all the records no matter which table they're backed up from remain in step with each other in case we have to restore the system. Can't be more that a guess at this time ... if I had:
a) Note of error message / how it showed up
b) Note of time problem occurred
c) Duration (how long it lasted before it was [self]fixed)
I could either give you an easy or more certain answer or (if the issue was of a different sort) have the extra evidence I need to be able to know where to look.

Not very precise, but it was reported to me as down at 20:40 but when I looked three minutes later it was up and running.

A deep backup procedure was underway at 20:35 and completed at about 20:39.  I suspect that's not a co-incidence.