Train GraphicClick on the map to explore geographics
Great Western Passengers' Forum [home] and [about]
Current travel advice (as at 19.7.2021)
Forum in and beyond Coronavirus
Travel & transport from BBC stories as at 07:35 26 Sep 2021
- Amtrak train derails in Montana, killing three and injuring dozens
- Covid-19: Ireland ends mandatory hotel quarantine for travellers
- Taxi gardens and Egypt's pharaohs on a pencil - stories you may have missed
Read about the forum [here].
Register [here] - it's free.
What do I gain from registering? [here]
 28/09/21 - Ask a stupid question (1)
30/09/21 - Ask a stupid question (2)
16/10/21 - Coffee Shop AGM
17/10/21 - Wiltshire Climate Consultation
Random Image
Train RunningCancelled
09:30 Reading to Bath Spa
09:58 Exeter St Davids to Penzance
10:37 Swindon to Cheltenham Spa
11:13 Bath Spa to London Paddington
11:55 Cheltenham Spa to Swindon
12:34 London Paddington to Exeter St Davids
17:57 Exmouth to Paignton
18:10 Bristol Temple Meads to Taunton
20:15 Weston-Super-Mare to Bristol Temple Meads
Short Run
16:17 Paignton to Exmouth
19:45 Paignton to Exmouth
Delayed
06:57 Reading to Gatwick Airport
Abbreviation pageAcronymns and abbreviations
Stn ComparatorStation Comparator
Rail newsNews Now - live rail news feed
Site Style 1 2 3 4
Next departures • Bristol Temple MeadsBath SpaChippenhamSwindonDidcot ParkwayReadingLondon PaddingtonMelksham
Exeter St DavidsTauntonWestburyTrowbridgeBristol ParkwayCardiff CentralOxfordCheltenham SpaBirmingham New Street
September 26, 2021, 07:52:34 am *
Welcome, Guest. Please login or register.

Login with username, password and session length
Forgotten your username or password? - get a reminder
Most recently liked subjects
[129] SWR promoting issue of own senior railcards WHY?
[109] Closures - Ireland
[90] Storing petrol
[79] Railway bridges struck by road vehicles - merged topic, ongoin...
[52] Lorry driver crisis made worse by Covid - Shapps
[44] Macron announces big investment in French railways
News: A forum for passengers ... with input from rail professionals welcomed too
 
   Home   Help Search Calendar Login Register  
Pages: 1 ... 5 6 [7]
  Print  
Author Topic: Server issues  (Read 27192 times)
grahame
Administrator
Hero Member
*****
Posts: 34437



View Profile WWW Email
« Reply #90 on: August 07, 2020, 06:38:26 am »

1. Some URLs on our server, such as complex searches, take a lot of cpu resources, and traffic on one in particular rose from around 400 to 600 a day to up to 1700 a day recent days compared to the same days last month.   These requests are very database intensive.

2. The noise of spurious server requests has rocketed in the last days - up 3 times from a pretty stable normal on Wednesday, and a further doubling (so now six times the norm) yesterday (Thursday).

3. One of the 'back' services we use appears to have stopped responding and that's resulted / resulting in queues of requests blocking available capacity with traffic waiting for timeouts.

Messages "Server too busy" yesterday were indeed accurate - sorry, folks who are "proper" users - that you have had significant problems over recent days.

This post is an analysis and not a promise of a solution. However, I have made some changes which may provide some relief while I look further.   

* I am hampered by the risk of closing / blocking doors which block genuine use as well as unwanted use.

* I am further hampered by looking for a commonality in unwanted use - looking for source and cutting it off there, when there probably is a coming together of multiple sources each bubbling over.   

* And once infected URLs are being requested, the requesting sources (often distributed so I can't just cut off a rogue remote address) that keeps on asking.

* Incoming rouge traffic is not uniformly predictable - whilst some tends to come in clumps, it also comes at just about any time of day and with some comes is underlying sporadic noise.  And because it's unpredictable, I'm hampered in testing any changes I make - often just a feeling that I'm getting somewhere.

I am a scientist. I would take my original situation as 'control', make one change, and see how it works before moving on to the next change.  But that takes time - more time that I/we have - so I am making multiple changes, quick-testing each and moving on to doing more; far from ideal and it precludes me from any definitive "that will sort it" statements.

In summary - I'm working on it.  Should be better today, fingers crossed - but we're not out of the woods yet.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 34437



View Profile WWW Email
« Reply #91 on: August 08, 2020, 11:59:48 am »

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

Edited to correct typo in URL
« Last Edit: August 08, 2020, 12:02:39 pm by bobm » Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 34437



View Profile WWW Email
« Reply #92 on: August 10, 2020, 08:40:01 am »

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

It has run well over the weekend ... and I suspect that's not entirely a co-incidence.  I have:
* Banned a couple of aggressive crawlers
* Retired some older heavy database URLs that crawlers loved (%)
* Ceased back bus service calls which were hanging processes (%)
* Widened SQL door to allow more parallel connections
* Cleaned out accumulated database logs including heavy context setters
Actions marked (%) could usefully have slimmer/better alternatives put in place - change is a bit of a sticking plaster. I have also enhanced a couple of logs - hoping to see if more can be learned about current running and help in future analysis.

Still a "watching brief'.  Our databases back up from time to time, and there are certain times bookkeeping is going on in the background, which may effect access. We rather crudely lock tables while we back them up and hold requests until the backup copy is completed and if the holding pen is full of excited requests ("Flying Scotsman spotted at Goonbarrow" or "First day of hourly Oxford to Fawley via Trowbridge and Eastleigh service - pictures") we may have an issue.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 34437



View Profile WWW Email
« Reply #93 on: September 03, 2020, 06:18:01 am »

Overnight issues - server down from around 03:00 to around 06:00 - now restored and "night sleeper" readers can now pass by, read and write.  Engineers just tidying up for an hour or so may results in some slow running.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 34437



View Profile WWW Email
« Reply #94 on: January 01, 2021, 07:36:38 pm »

This issue has come up again ... love your pictures but PLEASE resize massive (4k x 3k) images before attaching them to posts, or sent them to me to do to  ... Thanks!

Members are allowed to attach files of certain types (such as .jpg images and .pdf documents) to their posts, subject to a maximum file size.  When displayed on the bottom of a post for the first time, certain of these file types generate a thumbnail so readers can quickly have an idea of what's in the file before they download it.  This thumbnailing process can have rather a large server memory requirement - especially when it's a large .jpg file involved.

The other day, a .jpg was uploaded which was within the maximum file size, but caused the forum software to require more than the maximum memory it's allowed when thumbnailling, causing the server request to crash "blank screen", without even an error log or message.  As the thumbnail hadn't been created, every time anyone called for the particular post to be displayed in the thread and with the attachment offered, the server tried again and crashed again.  It was, however, happy to show the post if users selected to print the thread or see it on recents, as neither of those viewing methods offers thumbnails.

The "user solution" is for me to ask members not to upload images larger than 2k x 1.5k pixels. The one the other day was over 4k x 3k.  It just contained a lot of flat grey, so the compression applied in the .jpg algorithm in the user's camera had resulted in quite a small file that had passed the server's file size limit test.

I could - but won't - limit the file upload size; the maximum is set where it is for other documents and all I would be doing is fixing one very rare problem and in the process creating a limit that would frustrate more often.  If it happens again (once in 12 years so far), please let me know - it's easy for me to resize the rogue image now that I know what I'm looking for.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
grahame
Administrator
Hero Member
*****
Posts: 34437



View Profile WWW Email
« Reply #95 on: June 24, 2021, 09:47:49 am »

A long time since I have posted here on this thread

Overnight - I'm investigating, but what looks like a denial of service attack of some sort - around 1500 requests from a single place in Texas, with over half of them taking (us) more than 20 seconds from request start to fulfilment completion.  Claiming to be a different model of browser on just about every request.  That IP now banned, server rebooted just to tidy up ... nothing lost ... steady as we go.

Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
GBM
Transport Scholar
Hero Member
******
Posts: 777


View Profile Email
« Reply #96 on: September 15, 2021, 05:37:18 am »

Ooh, exciting.
A new unannounced rail link opened up direct to the Scillies.
Well, looking at the map anyway  Wink
Logged

Personal opinion only.  Writings not representative of any union, collective, management or employer. (Think that absolves me...........)
grahame
Administrator
Hero Member
*****
Posts: 34437



View Profile WWW Email
« Reply #97 on: September 15, 2021, 06:57:33 am »

Ooh, exciting.
A new unannounced rail link opened up direct to the Scillies.
Well, looking at the map anyway  Wink

A relocation of Worcestershire Parkway High Level, which is abbreviated to Worcestershire Parkway Hl on certain data feeds and just as Worcestershire Parkway on others - only station with three different names in different places.
Logged

Coffee Shop Admin, Vice Chair of Melksham Rail User Group, and on the board of TravelWatch SouthWest.
Do you have something you would like to add to this thread, or would you like to raise a new question at the Coffee Shop? Please [register] (it is free) if you have not done so before, or login (at the top of this page) if you already have an account - we would love to read what you have to say!

You can find out more about how this forum works [here] - that will link you to a copy of the forum agreement that you can read before you join, and tell you very much more about how we operate. We are an independent forum, provided and run by customers of Great Western Railway, for customers of Great Western Railway and we welcome railway professionals as members too, in either a personal or official capacity. Views expressed in posts are not necessarily the views of the operators of the forum.

As well as posting messages onto existing threads, and starting new subjects, members can communicate with each other through personal messages if they wish. And once members have made a certain number of posts, they will automatically be admitted to the "frequent posters club", where subjects not-for-public-domain are discussed; anything from the occasional rant to meetups we may be having ...

 
Pages: 1 ... 5 6 [7]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC Valid XHTML 1.0! Valid CSS!
This forum is provided by a customer of Great Western Railway (formerly First Great Western), and the views expressed are those of the individual posters concerned. Visit www.gwr.com for the official Great Western Railway website. Please contact the administrators of this site if you feel that the content provided by one of our posters contravenes our posting rules (email link). Forum hosted by Well House Consultants

Jump to top of pageJump to Forum Home Page