Train Graphic
Great Western Passengers' Forum [home] and [about]
from GWR - Travel with confidence
Forum in and beyond Coronavirus
DfT Covid Travel Advice
Read about the forum [here].
Register [here] - it's free.
What do I gain from registering? [here]
 today - Railfuture Severnside - CANX
29/09/20 - Tuesday Club - ONLINE
03/10/20 - RailFuture Annual - ONLINE
09/10/20 - Travelwach Southwest
Random Image
Train Running Polls Acronyms/Abbreviations Station Comparator Rail news GWR co. site Site Style 1 2 3 4
Next departures • Bristol Temple MeadsBath SpaChippenhamSwindonDidcot ParkwayReadingLondon PaddingtonMelksham
Exeter St DavidsTauntonWestburyTrowbridgeBristol ParkwayCardiff CentralOxfordCheltenham SpaBirmingham New Street
September 26, 2020, 06:11:40 pm *
Welcome, Guest. Please login or register.

Login with username, password and session length
Forgotten your username or password? - get a reminder
Most liked recent subjects
[132] AZTEC Tickets - sorting out the ticket barriers
[95] Journeys between places with similar names
[56] Newhaven Marine. Closure consultation, January 2020.
[55] lack of rail scenes in modern day films
[53] Weymouth tramway-the final curtain?
[50] Older types of fluorescent lamp now hard to find.
News: A forum for passengers ... with input from rail professionals welcomed too
 
   Home   Help Search Calendar Login Register  
Pages: 1 ... 5 6 [7]
  Print  
Author Topic: Server issues  (Read 17320 times)
grahame
Administrator
Hero Member
*****
Posts: 30660



View Profile WWW Email
« Reply #90 on: August 07, 2020, 06:38:26 am »

1. Some URLs on our server, such as complex searches, take a lot of cpu resources, and traffic on one in particular rose from around 400 to 600 a day to up to 1700 a day recent days compared to the same days last month.   These requests are very database intensive.

2. The noise of spurious server requests has rocketed in the last days - up 3 times from a pretty stable normal on Wednesday, and a further doubling (so now six times the norm) yesterday (Thursday).

3. One of the 'back' services we use appears to have stopped responding and that's resulted / resulting in queues of requests blocking available capacity with traffic waiting for timeouts.

Messages "Server too busy" yesterday were indeed accurate - sorry, folks who are "proper" users - that you have had significant problems over recent days.

This post is an analysis and not a promise of a solution. However, I have made some changes which may provide some relief while I look further.   

* I am hampered by the risk of closing / blocking doors which block genuine use as well as unwanted use.

* I am further hampered by looking for a commonality in unwanted use - looking for source and cutting it off there, when there probably is a coming together of multiple sources each bubbling over.   

* And once infected URLs are being requested, the requesting sources (often distributed so I can't just cut off a rogue remote address) that keeps on asking.

* Incoming rouge traffic is not uniformly predictable - whilst some tends to come in clumps, it also comes at just about any time of day and with some comes is underlying sporadic noise.  And because it's unpredictable, I'm hampered in testing any changes I make - often just a feeling that I'm getting somewhere.

I am a scientist. I would take my original situation as 'control', make one change, and see how it works before moving on to the next change.  But that takes time - more time that I/we have - so I am making multiple changes, quick-testing each and moving on to doing more; far from ideal and it precludes me from any definitive "that will sort it" statements.

In summary - I'm working on it.  Should be better today, fingers crossed - but we're not out of the woods yet.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 30660



View Profile WWW Email
« Reply #91 on: August 08, 2020, 11:59:48 am »

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

Edited to correct typo in URL
« Last Edit: August 08, 2020, 12:02:39 pm by bobm » Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 30660



View Profile WWW Email
« Reply #92 on: August 10, 2020, 08:40:01 am »

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

It has run well over the weekend ... and I suspect that's not entirely a co-incidence.  I have:
* Banned a couple of aggressive crawlers
* Retired some older heavy database URLs that crawlers loved (%)
* Ceased back bus service calls which were hanging processes (%)
* Widened SQL door to allow more parallel connections
* Cleaned out accumulated database logs including heavy context setters
Actions marked (%) could usefully have slimmer/better alternatives put in place - change is a bit of a sticking plaster. I have also enhanced a couple of logs - hoping to see if more can be learned about current running and help in future analysis.

Still a "watching brief'.  Our databases back up from time to time, and there are certain times bookkeeping is going on in the background, which may effect access. We rather crudely lock tables while we back them up and hold requests until the backup copy is completed and if the holding pen is full of excited requests ("Flying Scotsman spotted at Goonbarrow" or "First day of hourly Oxford to Fawley via Trowbridge and Eastleigh service - pictures") we may have an issue.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 30660



View Profile WWW Email
« Reply #93 on: September 03, 2020, 06:18:01 am »

Overnight issues - server down from around 03:00 to around 06:00 - now restored and "night sleeper" readers can now pass by, read and write.  Engineers just tidying up for an hour or so may results in some slow running.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
Do you have something you would like to add to this thread, or would you like to raise a new question at the Coffee Shop? Please [register] (it is free) if you have not done so before, or login (at the top of this page) if you already have an account - we would love to read what you have to say!

You can find out more about how this forum works [here] - that will link you to a copy of the forum agreement that you can read before you join, and tell you very much more about how we operate. We are an independent forum, provided and run by customers of Great Western Railway, for customers of Great Western Railway and we welcome railway professionals as members too, in either a personal or official capacity. Views expressed in posts are not necessarily the views of the operators of the forum.

As well as posting messages onto existing threads, and starting new subjects, members can communicate with each other through personal messages if they wish. And once members have made a certain number of posts, they will automatically be admitted to the "frequent posters club", where subjects not-for-public-domain are discussed; anything from the occasional rant to meetups we may be having ...

 
Pages: 1 ... 5 6 [7]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC Valid XHTML 1.0! Valid CSS!
This forum is provided by a customer of Great Western Railway (formerly First Great Western), and the views expressed are those of the individual posters concerned. Visit www.gwr.com for the official Great Western Railway website. Please contact the administrators of this site if you feel that the content provided by one of our posters contravenes our posting rules (email link). Forum hosted by Well House Consultants

Jump to top of pageJump to Forum Home Page