Train GraphicClick on the map to explore geographics
 
I need help
FAQ
Emergency
About .
Travel & transport from BBC stories as at 14:55 28 Mar 2024
- Man held over stabbing in front of train passengers
- How do I renew my UK passport and what is the 10-year rule?
- Jet2 launches first flight from Liverpool airport
* Easter travel warning as millions set to hit roads
Read about the forum [here].
Register [here] - it's free.
What do I gain from registering? [here]
 02/06/24 - Summer Timetable starts
17/08/24 - Bus to Imber
27/09/25 - 200 years of passenger trains

On this day
28th Mar (1988)
Woman found murdered on Orpington to London train (*)

Train RunningCancelled
13:26 Weston-Super-Mare to London Paddington
13:28 Weymouth to Gloucester
13:30 London Paddington to Bristol Temple Meads
14:13 Par to Newquay
14:19 Westbury to Swindon
15:10 Newquay to Par
15:14 Swindon to Westbury
15:16 London Paddington to Cardiff Central
15:30 Bristol Temple Meads to London Paddington
16:04 Bristol Temple Meads to Filton Abbey Wood
16:51 Filton Abbey Wood to Bristol Temple Meads
17:04 Bristol Temple Meads to Filton Abbey Wood
17:51 Filton Abbey Wood to Bristol Temple Meads
17:54 Cardiff Central to London Paddington
Short Run
11:23 Portsmouth Harbour to Cardiff Central
11:29 Weymouth to Gloucester
11:30 Cardiff Central to Portsmouth Harbour
11:48 London Paddington to Carmarthen
12:03 London Paddington to Penzance
13:03 London Paddington to Plymouth
13:10 Gloucester to Weymouth
14:05 Salisbury to Bristol Temple Meads
15:10 Gloucester to Weymouth
15:23 Portsmouth Harbour to Cardiff Central
15:30 Cardiff Central to Portsmouth Harbour
16:19 Carmarthen to London Paddington
16:54 Cardiff Central to London Paddington
Delayed
10:04 London Paddington to Penzance
12:30 Cardiff Central to Portsmouth Harbour
13:57 Exmouth to Paignton
14:30 Cardiff Central to Portsmouth Harbour
PollsOpen and recent polls
Closed 2024-03-25 Easter Escape - to where?
Abbreviation pageAcronymns and abbreviations
Stn ComparatorStation Comparator
Rail newsNews Now - live rail news feed
Site Style 1 2 3 4
Next departures • Bristol Temple MeadsBath SpaChippenhamSwindonDidcot ParkwayReadingLondon PaddingtonMelksham
Exeter St DavidsTauntonWestburyTrowbridgeBristol ParkwayCardiff CentralOxfordCheltenham SpaBirmingham New Street
March 28, 2024, 15:02:30 *
Welcome, Guest. Please login or register.

Login with username, password and session length
Forgotten your username or password? - get a reminder
Most recently liked subjects
[142] West Wiltshire Bus Changes April 2024
[80] would you like your own LIVE train station departure board?
[56] Return of the BRUTE?
[46] If not HS2 to Manchester, how will traffic be carried?
[43] Infrastructure problems in Thames Valley causing disruption el...
[34] Reversing Beeching - bring heritage and freight lines into the...
 
News: the Great Western Coffee Shop ... keeping you up to date with travel around the South West
 
   Home   Help Search Calendar Login Register  
Pages: 1 ... 5 6 [7]
  Print  
Author Topic: Server issues  (Read 44517 times)
grahame
Administrator
Hero Member
*****
Posts: 40690



View Profile WWW Email
« Reply #90 on: August 07, 2020, 06:38:26 »

1. Some URLs on our server, such as complex searches, take a lot of cpu resources, and traffic on one in particular rose from around 400 to 600 a day to up to 1700 a day recent days compared to the same days last month.   These requests are very database intensive.

2. The noise of spurious server requests has rocketed in the last days - up 3 times from a pretty stable normal on Wednesday, and a further doubling (so now six times the norm) yesterday (Thursday).

3. One of the 'back' services we use appears to have stopped responding and that's resulted / resulting in queues of requests blocking available capacity with traffic waiting for timeouts.

Messages "Server too busy" yesterday were indeed accurate - sorry, folks who are "proper" users - that you have had significant problems over recent days.

This post is an analysis and not a promise of a solution. However, I have made some changes which may provide some relief while I look further.   

* I am hampered by the risk of closing / blocking doors which block genuine use as well as unwanted use.

* I am further hampered by looking for a commonality in unwanted use - looking for source and cutting it off there, when there probably is a coming together of multiple sources each bubbling over.   

* And once infected URLs are being requested, the requesting sources (often distributed so I can't just cut off a rogue remote address) that keeps on asking.

* Incoming rouge traffic is not uniformly predictable - whilst some tends to come in clumps, it also comes at just about any time of day and with some comes is underlying sporadic noise.  And because it's unpredictable, I'm hampered in testing any changes I make - often just a feeling that I'm getting somewhere.

I am a scientist. I would take my original situation as 'control', make one change, and see how it works before moving on to the next change.  But that takes time - more time that I/we have - so I am making multiple changes, quick-testing each and moving on to doing more; far from ideal and it precludes me from any definitive "that will sort it" statements.

In summary - I'm working on it.  Should be better today, fingers crossed - but we're not out of the woods yet.
Logged

Coffee Shop Admin, Acting Chair of Melksham Rail User Group, Option 24/7 Melksham Rep
grahame
Administrator
Hero Member
*****
Posts: 40690



View Profile WWW Email
« Reply #91 on: August 08, 2020, 11:59:48 »

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

Edited to correct typo in URL
« Last Edit: August 08, 2020, 12:02:39 by bobm » Logged

Coffee Shop Admin, Acting Chair of Melksham Rail User Group, Option 24/7 Melksham Rep
grahame
Administrator
Hero Member
*****
Posts: 40690



View Profile WWW Email
« Reply #92 on: August 10, 2020, 08:40:01 »

Substantially improved ... but not perfect.   The quiz I have put up at http://www.passenger.chat/23875 is especially testing of the web and database elements and to some extent I have put that up to "push" the system and provide evidence if it breaks further.

It has run well over the weekend ... and I suspect that's not entirely a co-incidence.  I have:
* Banned a couple of aggressive crawlers
* Retired some older heavy database URLs that crawlers loved (%)
* Ceased back bus service calls which were hanging processes (%)
* Widened SQL door to allow more parallel connections
* Cleaned out accumulated database logs including heavy context setters
Actions marked (%) could usefully have slimmer/better alternatives put in place - change is a bit of a sticking plaster. I have also enhanced a couple of logs - hoping to see if more can be learned about current running and help in future analysis.

Still a "watching brief'.  Our databases back up from time to time, and there are certain times bookkeeping is going on in the background, which may effect access. We rather crudely lock tables while we back them up and hold requests until the backup copy is completed and if the holding pen is full of excited requests ("Flying Scotsman spotted at Goonbarrow" or "First day of hourly Oxford to Fawley via Trowbridge and Eastleigh service - pictures") we may have an issue.
Logged

Coffee Shop Admin, Acting Chair of Melksham Rail User Group, Option 24/7 Melksham Rep
grahame
Administrator
Hero Member
*****
Posts: 40690



View Profile WWW Email
« Reply #93 on: September 03, 2020, 06:18:01 »

Overnight issues - server down from around 03:00 to around 06:00 - now restored and "night sleeper" readers can now pass by, read and write.  Engineers just tidying up for an hour or so may results in some slow running.
Logged

Coffee Shop Admin, Acting Chair of Melksham Rail User Group, Option 24/7 Melksham Rep
grahame
Administrator
Hero Member
*****
Posts: 40690



View Profile WWW Email
« Reply #94 on: January 01, 2021, 19:36:38 »

This issue has come up again ... love your pictures but PLEASE resize massive (4k x 3k) images before attaching them to posts, or sent them to me to do to  ... Thanks!

Members are allowed to attach files of certain types (such as .jpg images and .pdf documents) to their posts, subject to a maximum file size.  When displayed on the bottom of a post for the first time, certain of these file types generate a thumbnail so readers can quickly have an idea of what's in the file before they download it.  This thumbnailing process can have rather a large server memory requirement - especially when it's a large .jpg file involved.

The other day, a .jpg was uploaded which was within the maximum file size, but caused the forum software to require more than the maximum memory it's allowed when thumbnailling, causing the server request to crash "blank screen", without even an error log or message.  As the thumbnail hadn't been created, every time anyone called for the particular post to be displayed in the thread and with the attachment offered, the server tried again and crashed again.  It was, however, happy to show the post if users selected to print the thread or see it on recents, as neither of those viewing methods offers thumbnails.

The "user solution" is for me to ask members not to upload images larger than 2k x 1.5k pixels. The one the other day was over 4k x 3k.  It just contained a lot of flat grey, so the compression applied in the .jpg algorithm in the user's camera had resulted in quite a small file that had passed the server's file size limit test.

I could - but won't - limit the file upload size; the maximum is set where it is for other documents and all I would be doing is fixing one very rare problem and in the process creating a limit that would frustrate more often.  If it happens again (once in 12 years so far), please let me know - it's easy for me to resize the rogue image now that I know what I'm looking for.
Logged

Coffee Shop Admin, Acting Chair of Melksham Rail User Group, Option 24/7 Melksham Rep
grahame
Administrator
Hero Member
*****
Posts: 40690



View Profile WWW Email
« Reply #95 on: June 24, 2021, 09:47:49 »

A long time since I have posted here on this thread

Overnight - I'm investigating, but what looks like a denial of service attack of some sort - around 1500 requests from a single place in Texas, with over half of them taking (us) more than 20 seconds from request start to fulfilment completion.  Claiming to be a different model of browser on just about every request.  That IP now banned, server rebooted just to tidy up ... nothing lost ... steady as we go.

Logged

Coffee Shop Admin, Acting Chair of Melksham Rail User Group, Option 24/7 Melksham Rep
GBM
Transport Scholar
Hero Member
******
Posts: 1468


View Profile Email
« Reply #96 on: September 15, 2021, 05:37:18 »

Ooh, exciting.
A new unannounced rail link opened up direct to the Scillies.
Well, looking at the map anyway  Wink
Logged

Personal opinion only.  Writings not representative of any union, collective, management or employer. (Think that absolves me...........)
grahame
Administrator
Hero Member
*****
Posts: 40690



View Profile WWW Email
« Reply #97 on: September 15, 2021, 06:57:33 »

Ooh, exciting.
A new unannounced rail link opened up direct to the Scillies.
Well, looking at the map anyway  Wink

A relocation of Worcestershire Parkway High Level, which is abbreviated to Worcestershire Parkway Hl on certain data feeds and just as Worcestershire Parkway on others - only station with three different names in different places.
Logged

Coffee Shop Admin, Acting Chair of Melksham Rail User Group, Option 24/7 Melksham Rep
Do you have something you would like to add to this thread, or would you like to raise a new question at the Coffee Shop? Please [register] (it is free) if you have not done so before, or login (at the top of this page) if you already have an account - we would love to read what you have to say!

You can find out more about how this forum works [here] - that will link you to a copy of the forum agreement that you can read before you join, and tell you very much more about how we operate. We are an independent forum, provided and run by customers of Great Western Railway, for customers of Great Western Railway and we welcome railway professionals as members too, in either a personal or official capacity. Views expressed in posts are not necessarily the views of the operators of the forum.

As well as posting messages onto existing threads, and starting new subjects, members can communicate with each other through personal messages if they wish. And once members have made a certain number of posts, they will automatically be admitted to the "frequent posters club", where subjects not-for-public-domain are discussed; anything from the occasional rant to meetups we may be having ...

 
Pages: 1 ... 5 6 [7]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.2 | SMF © 2006-2007, Simple Machines LLC Valid XHTML 1.0! Valid CSS!
This forum is provided by customers of Great Western Railway (formerly First Great Western), and the views expressed are those of the individual posters concerned. Visit www.gwr.com for the official Great Western Railway website. Please contact the administrators of this site if you feel that the content provided by one of our posters contravenes our posting rules (email link to report). Forum hosted by Well House Consultants

Jump to top of pageJump to Forum Home Page