Great Western Coffee Shop

From the BBC (https://www.bbc.co.uk/news/uk-49300025)

The picture titles (below) give some idea of the problems cause yesterday - "Nearly a million people have been affected by a major power cut across large areas of England and Wales, affecting homes and transport networks."

* Passengers on a train near Kentish Town station got off and began walking along the tracks

* People walked in complete darkness at Clapham Junction station in London during a power cut

* Information was not showing on Waterloo departure boards

* All services in and out of King's Cross station were suspended

* A staff member guides train passengers with torch light at Clapham Junction station in London

* Passengers travelling on Thameslink were delayed

* London North Eastern Railway staff, pictured here at Peterborough station, was disrupted, with delayed passengers handed bottled water

Yesterday while all this was happening, i travelled on the Hammersmith & City from Whitechapel to Paddington and then on to Maidenhead with no problems at all.   

The only thing mentioned by the H&C driver was when we got to KX & St P that the was not Victoria line due to a power failure, it was not until I got a BBC new alert on my phone that I became aware of the scale.

I did not get any calls on my work phone, although I'm not on call this week (phew)

Often when there are these types of power outages it is the signalling systems that take the time to restore, this is can be due to in built safety features, ie they do not want interlocking compromised.  The stations  take their supplies for the local "street" supplies larger ones like Clapham will have more than one but in essence will be the same bulk supply point.

The last major power outage was back in 2004 (if I recall correctly) which affected South London

It was reported on the radio this morning that some trains needed "specialist engineers" to get them restarted. Surely this should not result from a temporary loss of power?

Also reported that one train delayed for 13 hours had no toilets after 5 hours delay.  Not good. 

Beaten to it but same question.

Heard on the BBC that stranded trains needed specialist engineers to restart them. Is this correct?

Or did they mean signal engineers to restore  the signalling system as mentioned by electric train.

The Thameslink class 700s which was the main cause of the chaos into and out of Kings Cross and St Pancras with EMT and LNER trains stuck behind them.

The 9.52 Aberdeen to Kings Cross arrived into Kings Cross 499 minutes late! Due to arrive 1648, finally pulled in at 0107.

Another internal review by the TOCs and promises of 'lessons learned' I expect!

Perhaps this might help a little bit (from the WNXX Forum).  Appologies for it being a 'bit technical':

As above, the national grid have a legal duty to maintain the frequency between 49.5 and 50.5 cycles, and can be fined for failing in this duty.

However it must be accepted that under rare emergency conditions that the grid frequency can vary by more than that.

I would expect that a properly designed electric train should be able to work over at least the range from 48 cycles to 52 cycles, and preferably be able to work down to 47.5 cycles.
I do not know how low the frequency dropped this time, but in a PREVIOUS similar event about 10 years ago the grid frequency very briefly dropped to 48-75 cycles and remained below 49-5 cycles for 9 minutes.
I would expect that an electric train should work correctly under such conditions.

The drop has been reported to be to 48.89Hz.

Another fact that made the response difficult was that there was a high amount of renewables (strong wind) and relatively little thermal plant generating at the time with little inertia in the mix, so the imbalance from the production side needed to come from starting up generation from scratch. Pumped storage was turned on almost immediately followed by the open cycle gas turbines.

This is a known issue with greater reliance on renewables, and there is a contract out at the moment for fast response battery provision.

Apparently two major generators - Little Barford gas-fired power station in Bedfordshire and Hornsea offshore wind farm - both went offline at about the same time. We should wait for the experts to establish how these events unravelled, but it is understood that the lower rotating mass of renewable sources causes grid managers to have sleepless nights.

Of course this is not an argument against renewables, but it may suggest that more investment in 'synthetic inertia' could be helpful.

When I was young we had good old fashioned power "cuts" (especially during the three day week) - now it seems we have new fangled power "outages".

It is the modern trend, never use a short word like "cuts" when a longer word such as "outages" will do.

One wonders what happens when a modern signalling system loses power - does it fail right side, everything red including AWS and TPWS, or not?

Restarting/booting must be a nightmare - all lines blocked initially but how does it know which sections to clear first, with most occupied with (possibly failed) but loaded passenger trains? Would it all be manual?

If this is to be the future then the phrases Bi-mode and UPS for ROC's spring to mind.

OTC

Agree, they really need UPS systems and other back up power supplies for signalling control centers.
And whilst I don't think much of the IETs for reasons given elsewhere, they DO have the merit of being able to proceed when the power fails.
IMO, all new electric trains should have either a small diesel engine or a battery so as to permit of low speed operation to the next station, OR to power the air conditioning if unable to proceed.

...only as far as the first failed signal..!


Wasn't there once talk of IETs being equipped with Transit-sized engines for 'limp-home' capability? Or did I dream that?


Doesn't 'cut' imply a deliberate, planned act, e.g. to save energy because (say) the miners were on strike (again)? I seem to remember we used the word 'failure' in other circumstances. 'Failure' has its origins in Old French, whilst 'Outage' is from an Old English word, for what it's worth...

AFAIK there was not any plan to fit IETs with "transit sized" engines to limp home.
What WAS planned was that IETs would have a standard type of engine across the various IET variants. The true bi mode units have multiple engines so as to give a performance under diesel power suitable for daily main line operation.
The nominally electric units have just a single engine of the same type, for limited performance when the electric power is not available.
All the IETs for GWR are of the bi mode type with multiple engines, 3 engines on the 5 car units.

Some of the IETs for use by other TOCs are the nominally electric variant, with but a single engine for emergency use. IIRC they can reach about 30 MPH on this single engine, most useful in case of emergency but obviously far short of the performance needed for regular use.

The term used in India is "load shedding". I understand this has a technical usage but there it's used in a far broader sense to describe the (pretty much daily) interruptions to the electricity supply whatever their reason.

Continuing the international theme, overhead supply was deliberately cut several times in Poland in the early 2000s because PKP were hugely in debt to the electricity generators. Signalling continued to function though.

There was .. top speed specified on the all-electric units (801) under diesel was 30 m.p.h; how that compare to a transit's engine I don't know.

WikiPedia tells us

More important than keeping moving, arguably, is the hotel power for many hours that a diesel engine will provide, so heating, lighting, air-con and toilets stay operational keeping passengers comfortable.  IETs also have bags of space for emergency water and food supplies.

All signalling centres since at least the mid-1960s have had some form of diesel generator backup.  More modern centres also have UPS backup.  However, in lots of cases, but not all, this does not extend to the lineside supplies that are usually area specific, so if one gets a total outage a relatively small area is affected (on average not more than 5-6 miles).  As it happens older mechanical installations can be more resilient that modern systems because everything is obviously mechanically operated, or if electrical, uses primary or secondary cells ;D

Track circuits will naturally 'reset' themselves after power restoration and only show clear if not occupied by any vehicles.  Axle Counters will also reset but the affected section will not show clear again until the first train through the section is correctly counted in and then counted out again.  Some axle counter systems are fitted with UPS to overcome some, but not all of those issues.

Signals will go out but will restore when power returns.  The aspect displayed will depend on what the track circuits/axle counters are telling the signalling system at the time.

AWS will fail to the warning status regardless of signal aspect.  The permanent magnet part of it ensures that.

TPWS will fail until power is restored.

Points sometimes have battery backup systems, but not in all cases.

And just for noting; the diesel generators are always tested on-load every thirteen weeks (well they used to be when I once had maintenance responsibility for them).

I'm left wondering what should be the driver's response to all the signal lamps going out - is a full brake application to be made with only the train's tail light protecting the rear, is the AWS warning immediate, can it be cancelled if there is no TPWS?

Nervous flyer!

OTC

No need to worry.  No light in a signal when there should be one is classed as a danger (stop) signal.

The AWS warning will sound a Horn in the cab if there is no power to it (its worked by a permanent magnet).  That indicates to the driver to obey the signal and if the driver doesn't acknowledge that the emergency brake is automatically applied.

"More power stations needing repairs than usual".........................??  ;)

...mean more stations without power!

The term 'power outage' has been around for well over a century. Yes, it may be from American English originally but I see nothing wrong with that. Language evolves, words and phrases live and die by usage. I used to decry 'train station' but now realise that it's perfectly acceptable. As long as the reader/listener comprehends then that's really all that matters.

Or should we have a prescriptive 'English Academy' (modelled on the Académie francaise) that tries to preserve the language in aspic and rails against words and phrases from other languages, and from international variants of English?

By the way. I'm just back from a late night walking of the dog around the recreation ground. I had to take a flashlight to see my way along the sidewalks and thru the gates. I was disappointed by all the trash around the soccer fields.

Oh dear. What is a night? Surely you meant to say nite!

One of the issues of switching from thermal combustion energy to renewable energy is the national Grid has lost much of its "inertia"  The heavy spinning mass of large steam driven alternators provided a lot of inertia to the Grid meaning voltage and frequency transients caused by faults were more easily absorbed.  Solar, wind nor the dc interconnectors to Europe give much if any inertia to the Grid.


The large signalling centres (IECC, ROC & PSB) do have diesel generator backup which typically take 10 to 15 seconds to come on load from a mains power failure, they have UPS systems to cover for this time, in the more modern ROC these have dual supplies either one DNO and one (some even have 2 traction and a DNO) where Traction is not used they will have 2 independent DNO supplies.

The problems arise away from the signalling centres at the remote interlocking / relay rooms etc, these will have 2 supplies either DNO with diesel gen as the second supply, DNO with traction as the second supply, in the case of the Southern they have 2 DNO and lost have a third DNO.

All these power supplies take time to detect the supply failure and operate the change over switchgear, this time can range from 2 to 15 seconds, however the supply may be unstable for a second or so before its detected.

UPS are increasingly used, they are sized however typically for 1 to 2 minuets this is only cover the momentary loss of supply while the mains supply changes over to its back up; the economics of providing a UPS that can cover for an hour becomes unviable (ROC UPS are sized slightly lager than a relay room but will still only typically give 15 to 30 mins)


If it was the Class 700 that caused so many problems then I am sure the TOC, NR, ORR and DfT will be placing pressure on Siemens to solve what the issue was,  at least when the Class 800 come into full service on the ECML they can move independent of the traction power

Soccer is a good example of words changing. Its origins are British, as an abbreviation of association to distinguish hooligan's football from the gentleman's union variety (or is it the other way round?).

National Grid's initial report has been published on line (https://www.ofgem.gov.uk/system/files/docs/2019/08/incident_report_lfdd_-_summary_-_final.pdf) by Ofgem. Quaintly, it is marked ESO - HIGHLY CONFIDENTIAL; which maybe it was before it got to Ofgem!

It's not a great surprise - it documents the grid itself doing pretty much what it should, given the loss of nearly 2 GW of generation while meeting 30 GW of demand. A few points I thought worth noting:

1. There was about 1 GW of what they call 'frequency response', meaning extra generation that is provided automatically when the grid frequency drops. That included 472MW of battery storage. I wonder to what extent that will replace spinning reserve ('inertia').

2. The two big generators that disconnected themselves after the lightning strike should not have done so - that is not the expected behaviour.

3. On the other hand, the 500 MW of 'embedded generation' that disconnected due to transients was meant to do that.

4. No supplies for traction power were disconnected, though a few bits of railway (DC, signalling, etc.) disconnected themselves.

5. All those class 700 and 717 trains that turned themselves off and wouldn't turn on without special help did so because of their own transient detection.

6. The frequency never got so low that self-disconnection was expected of grid generators or critical loads - like trains, hospitals etc.  Only some particularly sensitive things, as well as those little 'embedded' generators, are allowed to do that.

The report does IMHO make very damming reading for the railway industry.
It states that National Grid did not interrupt ANY traction current supplies. 25Kv AC traction current remained live throughout.
A couple of 750 volt DC traction current supplies failed, but only due to the RAILWAY OWNED transformer/rectifier units tripping out when the grid frequency was below normal.

The railway chaos as widely reported therefore seems to have two principle causes.
Firstly the failure of the new Thameslink EMUs to function with a lower than normal line frequency. It seems that this was NOT a "failure to re-start after the power cut" as was initially reported, there was no failure of the power supply. The new trains simply stopped working when the frequency dropped and could not be re-started by the driver when the frequency returned to normal.
This to me looks like a grossly defective design if it effectively closes a main line for hours after a brief and entirely foreseeable drop in grid frequency.

Secondly, the failure of power supplies to signalling controls and to stations. These really should have standby supplies that actually work when called upon and permit of continuation of normal train services.
The larger and more important facilities ought IMHO to have TWO standby supplies, one from the traction current and one from an "electrically distant" part of the national grid, in addition to the normal supply.
Smaller and lower priority should at least have one main supply and one standby supply.
In non electrified areas, a diesel generator is an alternative to the traction current.

You obviously didn't read the previous posts by Electric Train and myself........ ::)

From the ESO Report: https://www.nationalgrideso.com/sites/eso/files/styles/content_embedded_image/public/images/Power%20cut%20timeline%20infographic%20jpeg.jpg?itok=0j8Jyt23

(https://www.nationalgrideso.com/sites/eso/files/styles/content_embedded_image/public/images/Power%20cut%20timeline%20infographic%20jpeg.jpg?itok=0j8Jyt23)

....and the Rail specifics:

From the Daily Mail (https://www.dailymail.co.uk/news/article-7375255/German-built-trains-caused-carnage-railways-power-cuts.html)


Does the country of manufacture really have a big bearing on why the trains shut down?

The Germans built what was specified. The Daily Wail will need to hunt down the nationality of the designer if they want to find someone to vilify.

I think Broadgage overstates the case for the prosecution (with one obvious exception).

One thing which I think is clear in the interim report, though not stated as such (it's not a text book), is that interaction between connected machinery at all levels matters. In a sense, the network is only the sum of all its parts, large and small. Some parts NG can control or at least influence, but some have to do the right thing automatically, and must be designed for that.

So big generators must stay connected as the frequency falls, or even increase their output. That's what stabilises the system. And they mustn't disconnect when they see a transient from routine network operation, like that trip/reclose caused by lightning. But designing high-power systems to meet that is difficult and expensive, in part because if you get it wrong your machine will disassemble itself and disappear in all directions (or, if electronic, vaporise - which is true at much lower power levels too).

Outside the grid itself, most loads and sources won't be designed to such exacting standards, so will err on the safe side - hence the loss of so much 'embedded generation'. I think the supply industry's own rules even dictate it, since they can't trust such 'domestic' devices to be designed to their own higher standards. These sources normally feed local loads, which looks to the grid like a reduced load in the distribution network there. Presumably it would help if more local loads disconnected to balance that effect - that 500MW lost was as much as a big generator.

What's left in between is critical loads, where losing power has consequential effects that worry people, and aggregate effects. Those 700/717 trains are a bit of both. Somewhere in the design/requirements analysis/specification process, when looking at the section headed "response to supply transients", there is an important question to be asked: "what if a lot of trains throughout the network behave like this all at once? Is the expected response acceptable?". It's an obvious and necessary question for a railway, simply because it is a network - and a Chinese puzzle to be solved continuously. So that's a fail.

For railway power and signals, it depends on the effect and the time taken to restore operation. I don't see that demanding full uninterrupted operation through any transient event or supply loss is necessary. In part, that's because it's not going to happen - somewhere, something will trip - so concentrate in minimising the scope and duration of interruptions.

But for other systems, and even for Ipswich Hospital (for which I have little information), turning off (some loads, at least) and not reconnecting for a minute of two may be fine. It is, after all, a rare event. Just provided the back-up systems work (and as we know...), and reconnecting doesn't call for an hour-long communal manual-reading session.

And to answer my own question about battery storage... I was expecting some, but not as much as 472 MW. Apparently it was 314MW at the start of the year, and additions in planning or construction will/would take that to over 5 GW. So it's obviously really fashionable just now. A lot of it is buffering wind power and other renewables, but some is just being built to offer NG fast response to help in exactly this kind of event (i.e. 9/8/19).

And NG did a study during 2014/15 into trialling this for use in their "Enhanced Frequency Control Capability", with as objectives:


That 'emulation' they also label "virtual inertia". I gather that implementing the control of power and phase based on frequency (in a DC-to-3 phase converter) to do this safely and reliably is still work in progress.

Re post #31, yes I did read previous posts on this subject.

Looking at the scale of the chaos I still maintain that backup power supplies for signalling were insufficient. Lineside signals remote from control rooms also need back up power supplies.
Up to a few miles distant, they could use power from the control center. Distribution of signalling power at 650 or 690 volts used to be the norm.
When this is not viable then battery backup should be considered.

I also stand by my remarks that the new Thameslink trains are of a defective design if they can not tolerate a relatively rare but entirely foreseeable drop in line frequency.
Is this design feature to be included in the Crossrail trains ? The thought of thousands of passengers trapped below ground for hours after a brief drop in frequency is rather worrying.

Different manufacturer (Bombadier not Siemens), so likely to behave differently?  The Siemens units could receive tolerance modifications as well surely?

Emergency evacuation of Crossrail trains in tunnels should be a relatively simple matter compared with traditional underground trains.

As I said, I don't see disconnection of trains, or some other railway equipment, as unacceptable given the very fast frequency drop to below 49 Hz. That's provided these things are all back working within a minute or two, which as far as I know was the case for NR's signalling (at least on bigger lines) and DC traction.

Those 700/717 trains, however, didn't - and that is inexcusable. If it is "only software", that only makes it worse, and I'm sure it was a known fault, in the sense that more than one driver had been unable to reset and restart a train. And while trainmakers might source some bits of software from suppliers of their competitiors too, in general each will have its own herd of bugs. No doubt we'll find out what the 345s' ones are.

I find it a bit odd that even some well-qualified commentators seem to think that because this event was more extreme than 20 years ago due to the change in generation on the grid, that means NG don't realise that's happening and have no plans to cope with it. You could infer they are a bit late in their implmentation, perhaps. But even if that figure I found of 4.9 GW of battery capacity in the pipeline is largely wishful planning, and say 1 GW is added by the end of 2020, NG will be able to cope with an event like this one using batteries alone.

NG are in a way being criticised for not explaining what their evolution plans are, in a report that wasn't meant to be about that. But one thing I think ought to have been explained is why the recovery of supplies took half an hour after the grid was stabilised. Was that just the way the DNOs do things, in which case could it be done quicker? Or was the rate of demand rise dictated by NG  to match the rate of rise of available generation?

The most likely cause for signalling, CIS, even some rectifiers etc tripping out in this event was not frequency instability but the dip in voltage on the National 400kV system, it is an "infinite busbar" hence a short circuit almost anywhere in the country will be reflected Nationally.   Most systems will either not see the dip or will self reset.  At major stations only essential systems related to safe evacuation of the station are backed up by a UPS or generator; to install the generator capacity to power even just keep Paddington normally operational (ie without retail) would require a 1 MW generator.

Regards the class 700's allegedly there was a problem with Siemens own computer and communications systems at the depot suffering from the dip in the power supplies which I a lead to believe prevented the reset to be sent via GSM-R.

Like all these events there are work streams across the railway industry (NR and TOC's) looking at the lessons learnt and what can be done should something like this happen againg

Technical Report of the National Grid ESO has been released.

https://www.ofgem.gov.uk/system/files/docs/2019/09/eso_technical_report_-_final.pdf (https://www.ofgem.gov.uk/system/files/docs/2019/09/eso_technical_report_-_final.pdf)

The appendices referred to in the main report are also available from Ofgem here (http://www.ofgem.gov.uk/system/files/docs/2019/09/eso_technical_report_-_appendices_-_final.pdf).

Of the three major failures that contributed to this event, the root causes were:

Hornsea wind farm - instability in the system that controls power (real and reactive) flows, leading to oscillatory power flows and excessive currents that in turn led to a shutdown. That's software, which Orsted say has been upgraded.

Little Barford CCGT generator - the first shutdown, of the steam turbine, is as yet unexplained but could well be software (RWE describe it as "a discrepancy in the speed signals"). The shutdown of the two gas turbine was presumably inevitable in the long run, but was premature due to excessive steam pressure when the turbine was bypassed (and stem sent direct to the condenser). Unlikely to be software, unless it is very large, heatproof, and nesting in the steam pipes.

Govia's class 700 and 717 trains - for the 30 or so that needed a technician+laptop to get them going, undoubtedly software. Indeed, it was an "upgrade" that removed a feature necessary to meet the specification that made these ones unable to do what the other 17 did, being rebooted by the driver. So not just software, but an own goal. Siemens are going to patch that, but are unwilling to simply revert to the previous version, because that was needed to remove some other undesirable behaviours.

I do wonder whether it is really necessary for any external fault (not implying the input converter is itself faulty) to cause a lockout, or for its reset to take 10 minutes. But there you go, that's progress, isn't it?

So it could be 3/3 down to duff software. Welcome to the 21st century!

I'm also a bit puzzled that NG say the load-shedding worked as planned, when the disconnection of 931 MW of load was offset by so much local generation being lost (disconnected by the DNO or itself) that the nett load shed was only 350 MW! (Note: that's in addition to all the generation that disconnected itself as a direct result of the voltage transients caused by the grid and generation faults/switching.) NG do say they will be looking at this area, but not at all urgently.

Over reliance on complicated and unreliable software is known as progress, in the railway industry and elsewhere.

I fail to see WHY a train needs to shutdown at all due to a brief and entirely foreseeable drop in frequency, and as for taking ten minutes to re-start, that is very poor design. Not being able to re-start at all without the attendance of a specialist carrying a laptop is beyond poor design, it is IMO a significant design fault.

Mainly because these types of disruption are rare, the protection settings to react to these types of events is done is software modelling, as such its based on assumptions.

There were other disruptive weather related events happening elsewhere in the UK causing 132kV tripping's, this caused some traction rectifiers on the Mersey Rail system to lock out and signal power supply disruptions elsewhere which lead to a number of axel counters and SSI modules internal power supply crowbar to operate.

The frequency disruption was for a relatively long duration, some put this down to the lack of inertia on the National Grid due to the loss of so much heavy rotating generating plant

There is a figure for "inertia" in those reports - 200GVAs, which I suspect is about the usual level at the moment. A useful number to tuck away in your toolbox (ideally with a proper definition). NG do have research going on on the use of batteries to provide "synthetic inertia", but I could see no mention of that in the reports. They do talk about campaign to reprogram small (embedded) generators to trip on a RoCoF (rate of change of frequency) of 1 Hz/s rather than the current value of 0.125 Hz/z. That would have kept a lot more of that generation on line. They say that will take three years, but some of the other evolutionary changes do seem to be going rather slowly.

On more point that I don't think was addressed directly was the cliff-edge effect of generators tripping due to RoCoF. The 1st July event (described in the main report) saw 1,000 Belgian MW lost, and the system coped - just.  The 9th August event was initially not much larger, but was just big enough that the reserve could not cope and, for want of enough power, the frequency fell. That led to enough extra generation disconnecting itself that the power shortfall suddenly became very large. Changing the RoCoF limit does address this, of course - eventually. 

The Times has pointed out that the makers of the turbines farmed at Hornsea - who presumably supplied the control software - were Siemens Gamesa (59% owned by Siemens). They didn't labour the point, but ... Before anyone starts on about "German software", Gamesa's engineering centres are in Denmark, Spain, and India.

Incidentally, the Hornsea system is new to the grid - the three modules went live on 1/2/19, 30/4/19, and 15/7/19 - so a certain amount of optimisation based on observation is to be expected. In which case, why had they not picked up the tendency for oscillatory power flows (as in the example ten minutes before the outage) already?

You may have seen that Ofgem have issued their final report (https://www.ofgem.gov.uk/system/files/docs/2020/01/9_august_2019_power_outage_report.pdf) on this event, and announced some voluntary fines to be paid by the operators of the two generators that disconnected and one DNO (UKPN, for reconnecting before being asked to). They seem uncertain whether they have a clear right to impose such fines, but a bit of arm-twisting of the emabarrassed has had the same effect.

Their analysis of what happened changes NGET's original story very little. However, some more poor performance has shown up in the operation of the low-frequency response process (LFDD). Not only did less than 5% of the load get shed, but two DNOs reconnected some load before NG told them to. LFDD disconnected some of the generators contracted to supply reserve power too, which was unintended. And the total of distributed generation lost in the event is now estimated at 1300-1500 MW (and the exact figure isn't knowable).

ORR have also published their report on ... well Siemens, mainly (https://orr.gov.uk/__data/assets/pdf_file/0017/42164/railway-power-disruption-on-2019-08-09-report.pdf?). They do document all the minor loss of power incidents too, none of which was serious.

The background at Siemens was much as reported earlier; a concern that some genuine failures produce a trip that could be reset and cause more damage led to some trip conditions being moved across to the list causing permanent lock-out. Low frequency was one of those, and should not have been. ORR's explantion is that, such events being rare, the risks of doing this were not considered - potentially the whole fleet could shut down out on the network and need a technician to attend.

Siemens admit this. A couple of quotes: