Great Western Coffee Shop

All across the Great Western territory => The Wider Picture Overseas => Topic started by: stuving on July 30, 2017, 12:26:09



Title: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on July 30, 2017, 12:26:09
Two years ago on this weekend I posted "SNCF have developed their own version of Sod's law, which is that something always goes wrong on a summer holiday weekend. The weekends in July and August are when the long-distance network is busiest. " Then it was fires.

This is the busiest of all weekends, and this time it's a total signal failure just outside Paris-Montparnasse. Their S&T have been working all night to try to find out what is going on - concetrating on the cables and comms links (sound familiar?)

No English report yet ... I'm sure one will be along shortly.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on July 30, 2017, 15:11:58
Here's a report, from RFi (http://en.rfi.fr/france/20170730-chaos-paris-railway-station-holiday-makers-trains-cancelled):
Quote
Chaos at Paris railway station as holiday-makers' trains cancelled
Issued on 30-07-2017 Modified 30-07-2017 to 14:19

There was chaos at one of Paris's main railway stations on Sunday as hundreds of holiday-makers found all trains heading for Brittany and the south-west had been cancelled. The SNCF rail company announced a gradual return to normal at midday.

Traffic jams clog the nation's motorways on the last weekend of July, when many French families hed off on holiday, so wise travellers take the train to avoid delays.

But, if their train was due to leave from Montparnasse railway station this Sunday, they were to be disappointed.

All trains, including high-speed TGV's heading for the other end of France, were cancelled because of an electrical failure.

The SNCF rail company sent text messages and emails to customers overnight but that did not stop hundreds turning up at the south Paris station only to find their train was not there waiting for them.

To make matters worse information screens were blank and, despite technicians working all night, the origin of the problem had not been found on Sunday morning.

At about midday the SNCF announced a "very gradual" return to normal but added that many trains would be leaving from other Paris stations for the rest of the day.

Another electrical fault caused disrupted services at Montparnasse station on 17 July.

That "very gradual" restart of services was so gradual that no-one noticed it for several hours ... during which the usual complaints rained down on SNCF and their Twitter account. Information, lack of and misleading nature of, was close to the top of that list. One advantage SNCF do have in Paris is that switching to another terminus does work to some extent, where in London even the track to allow this is in many cases missing. Of course that also adds further opportunities to be confusing, by announcing a train will leave from Austerlitz and then reversing that later on. And some trains were using Dreux as a Paris terminus - as far out as Didcot!


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: IndustryInsider on July 30, 2017, 15:15:39
Bloody useless Network Rail!  ;)


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on July 30, 2017, 19:19:46
SNCF are still saying they don't know what the underlying fault is. By shifting all TGVs to Poitiiers and south into Paris-Austerlitz, and presumably getting half the tracks into Montparnasse usable, 85% of passengers will have been got away today. Having that much spare terminus space isn't really surprising on a Sunday - that would be true in London - but being able to use it is another matter. 


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: Electric train on July 30, 2017, 19:48:59
Bloody useless Network Rail!  ;)

I know Network Rail often seems Omnipotent but even NR mystical powers cannot cross that much water  ;D


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: grahame on July 30, 2017, 20:00:45
SNCF are still saying they don't know what the underlying fault is. By shifting all TGVs to Poitiiers and south into Paris-Austerlitz, and presumably getting half the tracks into Montparnasse usable, 85% of passengers will have been got away today. Having that much spare terminus space isn't really surprising on a Sunday - that would be true in London - but being able to use it is another matter. 

Hmmm ... perhaps during the closure of Waterloo some trains from Basingstoke could run to Colchester (silly idea - I'm sure there's no route  ;D ) with people doubling back into Liverpool Street once the get out onto the GE main line.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on July 31, 2017, 13:52:49
SNCF now say their 50 technicians didn't find the problem again last night. Various statements and comments refer to damage to cables, missing insulation, a damaged battery - so it sounds like an intermittent short-circuit. However, if it was even a half-modern system you would expect locating that to be pretty straightforward. The effect, however, is that the whole Vanves sector has been operated manually, presumably mostly be locking it on fixed routes, hence only at one-third capacity.

TV has shown pictures of blokes in tabards prodding relays in big racks, but they are most likely to be library clips and not evidential. And then there's all the work done over the last three years to cope with the two new TGV lines served from Montparnasse (Rennes and Bordeaux). It's not clear whether that involved modernising the signalling in the Vanves sector itself, or just interfaced it to the brand-new control centre at Montparnasse. (Yes, the one that controlled the chaos yesterday.)


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: SandTEngineer on July 31, 2017, 15:05:41
I'll go and help them for a small fee..... ;) :D :P


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on July 31, 2017, 19:44:11
Still looking ... and tonight Montparnasse will close completely from 22:00 to 6:00 to allow more time for uninterrupted prodding, waggling, controlled application a micrometer hammer and other forms of electro-entomology. The signalling apparently dates from the 90s, whatever that implies.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: Worcester_Passenger on July 31, 2017, 21:30:10
Quite a lot of the signalling round here dates from the '90s.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on August 01, 2017, 09:40:15
SNCF are now saying they have found whatever was wrong, but not yet fixed it. So normal service will resume tomorrow - maybe. The transport minister has demanded a report on the ... subject ... by the end of the week; in the past SNCF have been pretty honest about doing these and making them public.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on August 01, 2017, 10:23:49
You may have seen, in some of the reports of this "grosse pagaille", mention of the previous one at Montparnasse. That was a power failure on 17th July, but while it affected the whole station (and one train just outside it) it was fixed within three hours. Which I guess is only a "petite pagaille".

However, there was an even more recent "local difficulty" on the same line, on 23rd July (Sunday). Paris-Rennes is now only two hours, but this TGV took nearly nine - arriving at 2 am. Power failed as it passed Le Mans, and it took over five hours to find a loco to go and drag it there. Apparently it was "at just the wrong place, a junction"... though at least it wasn't during a hot day. This, of course, is a brand-new high-speed line.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: IndustryInsider on August 01, 2017, 11:58:57
Silly me thinking railways in Europe always ran like clockwork...


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on August 01, 2017, 20:13:35
They did, in fact, fix the problem this morning. Services won't quite be back to normal until tomorrow for the usual reasons of trains and crew being out of place.

Various politicians, including the prime minister, have been opining about how unacceptable this all was and guessing why it happened. The transport minister referred to the new LGVs and the number of TSRs recently on the other lines (heat-related, no doubt). Apparently this tells her that they have a two-speed railway (er ... yes?). And of course for others it's due to inadequate spending on maintenance, the boss should resign, etc, etc.

SNCF have limited themselves to saying it was faulty insulation, and due to the work done recently rather than age and decrepitude. They promise us this report on Friday.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on August 03, 2017, 20:43:29
That report to the minister by SNCF has appeared today. That means they have produced a 28-page report, with graphics, in less time than the incident itself lasted. That kind of impressive, but maybe not in the way they'd like.

The story is still that it's some cabling fault, almost certainly an intermittent short to earth. However, describing it as "isolement fugitive" sounds wrong to me, and I'm baffled as to where it was - in a signal or power supply. The signal box dates from 1987, and is a big shed full of relays (i.e. a power box) but now controlled remotely.

The process of fault-finding was that they disconnected each bay of relay chassis where all cables were connected via multipole connectors. That didn't find it, so they moved on to the few bays with individual wire connections. At the same time they rechecked the earlier tests, and found the fault in a cable - which was disconnected during those tests and thus not tested. This cable had been put in during work in the last month as part of the "modernisation" for the new LGVs.

On Sunday, after the first fruitless night of searching, the planned restart at noon was delayed due to cable faults introduced by all the frantic plugging and unplugging done overnight. Hence they kept running trains until 3 a.m., and so had little time for testing the next night.

They did, in fact, fix the problem this morning. Services won't quite be back to normal until tomorrow for the usual reasons of trains and crew being out of place.

That's not quite true: having been running with a single track in and out, all points clipped, and all signals red - stop and proceed by sight - they were able to turn the block signalling on, but still with only the one clipped path. However, by still using Austerlitz that allowed more trains to run.

There's more about the plan - basically what not do do in future - and also how they got their information systems so screwed up, notably on Sunday. Basically, because the late start meant the plan had to be reworked too late, and so the implications of trains in the wrong place, and inaccessible depots, were missed and led to frantic replanning. As the various information systems are driven from separate data sources, and mostly captured by overstressed humans, they often made no sense. Plus for example the big departures board at Montparnasse can't show "from Austerlitz", blocking ticket sales for a train meant it showed as cancelled on voyages-sncf.
com, etc.

The report is here (https://www.ecologique-solidaire.gouv.fr/sites/default/files/2017_08_03_rapport_incident_montparnasse.pdf) - nowhere does it say the technology is relays, but there are photos at the end.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: SandTEngineer on August 04, 2017, 14:23:54
Hi Stuving, as its a long time since I did french at school have you found an English translation anywhere?


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on August 04, 2017, 19:51:40
Hi Stuving, as its a long time since I did french at school have you found an English translation anywhere?

No, and I'm sure there won't be in this case. For one thing it seems not to be on SNCF's own web site, only the ministry's one. (That's the ministry of the ecological and solidaire transition - and there's no English for that as we lack the French notion of solidarité.)

The media have been merrily putting their boots in on this, and it looks as if the useless communication to the public is what will be remembered most. Two specific examples:

One train, I'm not sure which day (but not the Sunday), was shown as cancelled on line, by phone it was leaving from Austerlitz, and on Twitter it would leave on time from Montparnasse. It left from Montparnasse, but 1:10 late and unsurprisingly with many empty seats.

A TGV through Brittany stopped short at Auray (not quite a village, but..) at 20:30, so its passengers had to await the next. SNCF staff were sure one would arrive, but had no idea when, and after four hours (thus at 00:30) it had still not turned up. So those waiting were sent off to a (paid-for) hotel, though the train did in fact arrive at 01:30. So internal communications were not much better.

As to the technical side, I still can't fathom how it can be impossible to localise a fault at all within the whole poste, before disconnecting everything a bit at a time. It was also said that the racks of equipment did not map onto the track, so there was no way to take half of it off-line and still use half of the tracks into the station. Maybe 7-day railway was not a known concept here in the 1980s either, but I'm sure the general idea had been thought of.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: SandTEngineer on August 04, 2017, 21:53:34
Thanks.  Shame its not though as I'm sure it would be a good (technical) read..... ::)  However, I think I can understand some of the basics of what went wrong.  When I have digested it a bit more I will try and explain it in not too technical terms ;)


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: Worcester_Passenger on August 05, 2017, 11:25:09
Go to Stuving's link and write down the title. Come back to your browser and you type the title into Google, then you'll get the link again, but this time with a 'Translate' option. Clicking on that gives you a start on the English. But ... be aware that it's not very good (understatement) when it comes to very technical terms.

Company staff out on the concourse dealing with the passengers are translated as "waistcoats". But you know what they mean.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: chuffed on August 05, 2017, 18:10:41
Go to Stuving's link and write down the title. Come back to your browser and you type the title into Google, then you'll get the link again, but this time with a 'Translate' option. Clicking on that gives you a start on the English. But ... be aware that it's not very good (understatement) when it comes to very technical terms.

Company staff out on the concourse dealing with the passengers are translated as "waistcoats". But you know what they mean.

Well I suppose that's better than 'Ribena' girls ...never did see any Ribena guys tho.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on August 06, 2017, 00:16:12
Thanks.  Shame its not though as I'm sure it would be a good (technical) read..... ::)  However, I think I can understand some of the basics of what went wrong.  When I have digested it a bit more I will try and explain it in not too technical terms ;)

I find it's not the jargon that's the problem - mostly that has only one meaning, and can be looked up (e.g. enclenchement=interlockiing or d’astreinte=on call). It's more the words with multiple ordinary meanings, used in a particular idiom. And often - like those gilets rouges - they are easier to cope with untranslated.

Having looked at it again, I can work out that in the system's label of PRCI-PVSI, PRCI means it had computer route-setting (remote, in this case) and relay interlocking. PVSI I have not found. Note that it is seen, in the continental fashion, as implementing 150 routes (itineraires) directly and interlocking them rather than point settings.

The fault is described as on the power supply (24V), and capable of changing a signal state - so presumably not to ground. I still do not see how its first appearance can be as "fleeting insulation/isolation" - I guess faute is missing there. But if that does mean a fleeting intermittent fault, would isolating a rack of relays to test it have a high chance of finding the fault? 

If you are struggling with any words, let me know and I'll try to help.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: Electric train on August 06, 2017, 08:00:19

The fault is described as on the power supply (24V), and capable of changing a signal state - so presumably not to ground. I still do not see how its first appearance can be as "fleeting insulation/isolation" - I guess faute is missing there. But if that does mean a fleeting intermittent fault, would isolating a rack of relays to test it have a high chance of finding the fault? 

Signalling supplies tend to be earth free, IT - Isolate Tera (sorry more French) and not TN Tera Neutral or TT Tera Tera systems.  The reason it allows a first earth fault to happen without blowing a fuse. IT systems do rely on good insulation monitoring.  The risks as high impeadance earth faults or fleeting earth faults when a second earth fault happens.

I am sure S&T Engineer will agree trying to locate a fault can take a long time, the charging down blind allies often happends especially when presure is being applied. 

Posibly the question that should be asked is not why the fault happened or what the fault was, as that is a matter of fact, but SNCF should be asking how they managed the faulting process and the management of the wider incident.


Title: Re: "They don't have major holiday weekend signal failures in the rest of Europe"..?
Post by: stuving on January 08, 2018, 20:48:59
As a follow-up to this string of ... difficulties, the bosses of SNCF and its network bit (Pepy and Jeantet) were summoned today to stand on the minister's naughty carpet (she's Élisabeth Borne, previously head of RATP but also both a préfet and a socialist politico). To recap, there was the big signals upgrade overrun in July, plus a couple of smaller ones before that, both at Montparnasse, and also a software upgrade that did much the same to St Lazare in December. The problem just before Christmas was more a matter of ticketing and PR, but all of these raised the well-known complaint of no or wrong information.

After this meeting, some concrete plans were announced for work on that information process, including weekly publication of data on train problems and the cause, and something like the red/amber/yellow weather alert system applied to railway "storms".

The main plan, however, makes no sense: it is to do an audit of the robustness of the system and make a shopping list of investments to improve it. Now that would no doubt help with some of the "routine"problems, like broken OLE and signals. But the biggies are due to doing upgrades, so surely the need is for a better plan for doing that, not a bigger list of things to do. Bearing in mind that Paris does have some spare terminal capacity at all times, and linking lines to use it, I would suggest something like this:

1. Rebuilding the systems bit by bit, with whole railway back working after each step, is bound to go wrong sometimes. To avoid the overruns, you'd need to close large chunks of the network for months, or allow several days each time with the "closures service" operating. Neither would keep the commuting bunnies happy.

2. But such issue only happen rarely. So go public on all of that, and say "we'll keep that recovery period as short as we can, provided you accept that once in a while it'll have to be extended". You publish a limited service plan that could be used for several days, and promise 3 hours notice that you are reverting to the normal timetable (as passenger need to go to different stations). In most cases this reversion happens overnight, so everyone knows by the night before. That gets round the issue of a control room full of headless chickens improvising a service and issue contradictory messages to the waiting public.

It would at least be interesting to see how it went down, and whether the meejah in France would help to get the message across or (as here) not.



This page is printed from the "Coffee Shop" forum at http://gwr.passenger.chat which is provided by a customer of Great Western Railway. Views expressed are those of the individual posters concerned. Visit www.gwr.com for the official Great Western Railway website. Please contact the administrators of this site if you feel that content provided contravenes our posting rules ( see http://railcustomer.info/1761 ). The forum is hosted by Well House Consultants - http://www.wellho.net