Server issues

[whole diary]

[grahame]

Just to let members know ... few funnies going on.  I am away with limited access / just note may be some service issues

[SandTEngineer]

Hi Grahame.  Just to let you know (not a criticism) that message still cropping up at odd times saying 'database gone away' or 'unable to connect to server'.  Usually solved by retrying a few times.

[grahame]

Hi Grahame.  Just to let you know (not a criticism) that message still cropping up at odd times saying 'database gone away' or 'unable to connect to server'.  Usually solved by retrying a few times.

Totally known ... but thanks for confirming it's not just me.

From where I am this week,  I dare not try too much; very limited access (no shell access to server, no scp, Lisa's vpn left her dead in the water.  And that's on what's called "the works".  A grumble to the provider here, and I'm told "funny you should ask - someone else was asking just this morning".

[grahame]

Just to let members know ... few funnies going on.  I am away with limited access / just note may be some service issues

Pinging this to the top ... normal service should resume from Friday, perhaps with some engineering overrun to Saturday.  Problems due to lack shortage of technical admins on one of the rare occasions we needed them!

[grahame]

Just to let members know ... few funnies going on.  I am away with limited access / just note may be some service issues

I am now back with regular access ... and have been looking around at flakiness during the week.  And I note from our logs that the Ahrefs Bot has been making up to 42,000 requests of our server per day - that's an average rate of one request for a page every 2 seconds!

So what is this robot?

AhrefsBot is a Web Crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. It constantly crawls web to fill our database with new links and check the status of the previously found ones to provide the most comprehensive and up-to-the-minute data to our users.

Link data collected by Ahrefs Bot from the web is used by thousands of digital marketers around the world to plan, execute, and monitor their online marketing campaigns.

This data has immense value for SEO community as it helps marketing professionals to better understand the fundamental algorithms of the world's largest search engines so that they could optimize websites accordingly. Every 24 hours our crawler visits over 6 billion web pages and updates Ahrefs’ index every 15-30 minutes.

Looking around, I see we're not the only ones to have this issue:

Not sure when Ahrefs went dark side, but with the amount it was hitting my forum I had to make it stop. It was killing everything.

And whilst it may be of value to the advertising community, I'm not sure what value if any it adds to our site.

They describe themselves as a "good bot":

According to a recent third-party study of “good bots”, AhrefsBot is the second most active crawler after Googlebot.

Now - Googlebot is certainly most welcome ... but in our site is far less active than AhrefsBot - averaging about 6,000 requests per day, with a peak of just over 10,000.   Me thinks that the AHRefsBots folks describe themselves as a "good bot" in the hope that none-technical people will believe them.   As far as we're concerned, they're aggressive users of bandwidth for purposes of their own business profit - harvesting data the harvesting of which is no use to us.  And indeed they've been so aggressive that they caused a great deal of worry while I was supposed to be forgetting everything for a week's holiday!

[grahame]

Unscheduled outage from about 04:30 for about 3 and a half hours - sorry folks.  My alarms went off but I was sleeping like a log.     Since the Ahrefs bot problems described a week ago and the banning of that naughty bot, the server has returned to a much more stable state.   I have been watching but refraining from making other adjustments until convinced with a good degree of certainty that it really was that loading that gave the truly massive problems I had the previous week.  Once I did wake, fix was minutes rather than hours and I would have wasted time (and left debris) had I shifted to our backup status page on another server.  So decision was few more minutes of no data for a quicker return.

Also been watching things like Google Analytics to ensure that there were no indirect consequences of banning the bot, and I'm delighted to confirm that telling it to go away has - thus far - had zero effect on visits.   Big sigh of relief as it confirm that Analytics data is a meaningful statistic.

[grahame]

I will be rebuilding some tables overnight after the backup system has started to play up in last hour.  Tiny chance of losing late evening posts.

[Chris from Nailsea]

As ever, grahame, may we offer you our very grateful thanks for your sterling efforts - usually behind the scenes - in maintaining the Coffee Shop forum. 

[grahame]

I will be rebuilding some tables overnight after the backup system has started to play up in last hour.  Tiny chance of losing late evening posts.

As ever, grahame, may we offer you our very grateful thanks for your sterling efforts - usually behind the scenes - in maintaining the Coffee Shop forum. 

I feel a bit of a fraud tonight ... no need to to do the rebuild in the end.

[Chris from Nailsea]

No worries - you can credit our collective thanks to your previous sleepless nights instead.

[Adelante_CCT]

I will be rebuilding some tables overnight....

[bobm]

My pet hate.  Always seem to end up with bits left over.

[grahame]

My pet hate.  Always seem to end up with bits left over.

Pocket screws ... the ones you put in your pocket after you've rebuilt the table?

[grahame]

Noting server down for a while this afternoon ... I have been offline and now changig trains - think I have kicked it back into order but need to run for train 2000.  Will check in later and see if it's AOK.

[bobm]

Think it is!