Air traffic control problems

[whole diary]

[Clan Line]

A fast and direct train could take you 1000 miles in 10 hours, in greater comfort and with less stress than air travel.
Even better if it was a sleeper train.

Mmmmmm ! I flew to Denver and back in May this year, I went over 5000 miles in 10 hours. I went Business Class (out my own pocket !), I was in better comfort that ANY train I have ever travelled on; food, wine and service were top notch. My little "suite" on the aircraft converted into a very comfortable bed as well.  The only stress was worrying that GWR^▸ (Great Western Railway) would fail to get me to Heathrow on time - they did manage that OK, but I did claim delay repay on the return trip.

It may also be worth saying that my trip to Denver cost me 21p per mile, a recent trip to Temple Meads on a train that the RSPCA would have shown interest in, if cattle had been in it instead of mere humans, cost me 27p a mile.

(I used to travel 1st Class on the train, but the appalling "trains" that we now have are just not worth the extra money - I'll put the savings toward another very comfortable trip with BA^» (British Airways - about))

[grahame]

A fast and direct train could take you 1000 miles in 10 hours, in greater comfort and with less stress than air travel.
Even better if it was a sleeper train.

A London to Barcelona Sleeper then ... 9:30 p.m. in St Pancras, 8:30 a.m. in Barcelona.   Return 10:30 p.m. from Barcelona, 7:30 a.m. in St Pancras.  Just under 950 miles?  Daytime train with the same hours, swapping p.m. to a.m. with my timings?    You could do London to Munich in similar timings.

[TaplowGreen]

A fast and direct train could take you 1000 miles in 10 hours, in greater comfort and with less stress than air travel.
Even better if it was a sleeper train.

A London to Barcelona Sleeper then ... 9:30 p.m. in St Pancras, 8:30 a.m. in Barcelona.   Return 10:30 p.m. from Barcelona, 7:30 a.m. in St Pancras.  Just under 950 miles?  Daytime train with the same hours, swapping p.m. to a.m. with my timings?    You could do London to Munich in similar timings.

............or you could fly there in 2 hours, almost certainly a great deal more cheaply -  and the comments about comfort and "less stress" are entirely subjective, especially if you were relying on the railways to get you to St Pancras!!!

[TonyK]

I have seen a comment that the system is designed that if there is a significant data input problem then it will fail safe by reverting to manual.

In this case fail safe prevented a lot of flights.

Somewhere in the first chapter of every textbook on how to fly a plane, you will read the most indisputable piece of wisdom on the subject of whether or not to take off, repeated at regular intervals throughout the course:

"It is better to be down here wishing you were up there, than up there wishing you were down here."

I might add that it is better as a passenger to be inconvenienced at a full stop on the ground than at 500 mph at 35,000'. They did the right thing.

Grahame's piece on the unimagined consequences of a precise number of data cards was fascinating, and reminded me of my teenage years. My parents ran a guest house between Blackpool and Fleetwood, and our phone number was Cleveleys 2100. At that time, local calls didn't need you to dial the area code. From time to time, usually in the evenings, we would get a call from someone who was after the operator, and had somehow got the number 2 in front of the number. I had fun asking them to help with a frequency test by whistling the national anthem, but it remained a mystery until a friend left school to start an apprenticeship with the then GPO^‡(resolve) telephone people. He launched his own small investigation at the local exchange, which was then very much analogue, with row upon row of what were called "uniselctors", basically electrically operated switched with contacts matching each digit dialled. If one dialled a number, the first in the bank would click into life and connect the call. If that was in use, the next call passed to the second, and so on. He found a fault in number 596 (or whatever) in the queue that meant that the number 2 was always present. To get through to me at home, the caller had to the 596th (or whatever) making a call at the same time, and had to be trying to call the operator. Given the chances of both conditions being met, we had a surprisingly large number of calls. It is another example of how an unknown fault in a system can cause an unexpected outcome, no matter how tiny the chances of it happening.

I rather missed it when he fixed the fault, although there was a second string to my mischief bow. Fleetwood football club, then non-league was Fleetwood 2100, and we got quite a few calls on Saturdays from people who forgot to add the area code asking "What time is kick-off?" to which I would ask "Are you playing or watching?" We always listened for the full-time score though, so they wouldn't have to make a second call. Again, it surprised me how many people cared, but were a bit useless with a phone.

A fast and direct train could take you 1000 miles in 10 hours, in greater comfort and with less stress than air travel.
Even better if it was a sleeper train.

I'm thinking of heading back to Madeira for some of the cold bits of the winter at home. Nothing in the Trainline so far as I can see...

[eightonedee]

or you could fly there in 2 hours, almost certainly a great deal more cheaply -  and the comments about comfort and "less stress" are entirely subjective, especially if you were relying on the railways to get you to St Pancras!!!

Relishing a challenge, I tried to find out how quickly I could get from Reading to Sants Station, Barcelona, which is the closest to La Rambla, heart of tourist Barcelona, using the Rio2Rome website. OK - I assumed that I would be travelling by train to Heathrow, but starting out at 8 am tomorrow.

By air, the problem was that by the time I arrived at Heathrow, using the first available (08-20) train from Reading, the next available flight allowing for check-in was not until 12-55, arriving 16-00 local time, and at Sants (assuming I could recover any hold baggage in time to catch a metro train by 16-38) was 16-58. It would be 30 minutes later if not. The problem is the check-in time of 2 hours before departure, with the earliest arrival at Heathrow being 09-10 for a 10-35 previous departure. I guess it would work, but do I want the stress? I could also arrive a couple of minutes later using the Rail Air coach from Reading, but the same considerations apply, albeit if I did get the 10-35 flight it would be a 13-50 arrival at El Prat, and a likely 14-58 arrival at Sants.

For a train trip, I could delay departure from Reading until 09-30, and with changes across Paris, I can pick up an InOui TGV^▸ (Train a Grande Vitesse) direct, arriving at Sants at 21-25 locally.

So - yes, it's quicker, but I find airports much more stressful than railway stations. And while there's a choice of fiddling around to get to Heathrow for the air alternative and the nuisance of crossing Paris by train, at the other end it's straight into the heart of Barcelona if you go by rail. 7 hours 38 minutes versus 10 hours 55 minutes - and I would enjoy watching the French and Spanish countryside roll past, much nicer than hoping I get a window seat and straining my neck to see what's underneath - assuming clear skies!

Finally prices - the train alone (with transfers/Underground/Paris Metro) - £343. By air (ignoring Rail Air bus option) - if the 10-35 flight - £587-94; if the 12-55 flight, £320-94.

[TaplowGreen]

or you could fly there in 2 hours, almost certainly a great deal more cheaply -  and the comments about comfort and "less stress" are entirely subjective, especially if you were relying on the railways to get you to St Pancras!!!

Relishing a challenge, I tried to find out how quickly I could get from Reading to Sants Station, Barcelona, which is the closest to La Rambla, heart of tourist Barcelona, using the Rio2Rome website. OK - I assumed that I would be travelling by train to Heathrow, but starting out at 8 am tomorrow.

By air, the problem was that by the time I arrived at Heathrow, using the first available (08-20) train from Reading, the next available flight allowing for check-in was not until 12-55, arriving 16-00 local time, and at Sants (assuming I could recover any hold baggage in time to catch a metro train by 16-38) was 16-58. It would be 30 minutes later if not. The problem is the check-in time of 2 hours before departure, with the earliest arrival at Heathrow being 09-10 for a 10-35 previous departure. I guess it would work, but do I want the stress? I could also arrive a couple of minutes later using the Rail Air coach from Reading, but the same considerations apply, albeit if I did get the 10-35 flight it would be a 13-50 arrival at El Prat, and a likely 14-58 arrival at Sants.

For a train trip, I could delay departure from Reading until 09-30, and with changes across Paris, I can pick up an InOui TGV^▸ (Train a Grande Vitesse) direct, arriving at Sants at 21-25 locally.

So - yes, it's quicker, but I find airports much more stressful than railway stations. And while there's a choice of fiddling around to get to Heathrow for the air alternative and the nuisance of crossing Paris by train, at the other end it's straight into the heart of Barcelona if you go by rail. 7 hours 38 minutes versus 10 hours 55 minutes - and I would enjoy watching the French and Spanish countryside roll past, much nicer than hoping I get a window seat and straining my neck to see what's underneath - assuming clear skies!

Finally prices - the train alone (with transfers/Underground/Paris Metro) - £343. By air (ignoring Rail Air bus option) - if the 10-35 flight - £587-94; if the 12-55 flight, £320-94.

Or, given that you're starting from Reading, you could chuck your toothbrush and a pair of budgie smugglers in your bag, nip down to Gatwick, and be there later today with change from £100 for a pint of San Miguel and an ice cream!

[stuving]

I'd come to think the different reactions to this event were due to how "data" was understood. I thought (and still do) that flight path data are what this safety-critical always-on data processing system processes, and  nothing in that should compromise system safety. There should be a one-way barrier between the system and the data it processes, and if faulty data are put in they should be rejected. So I'd concluded that in this event "data" must be something on the system side of that barrier, not flight plan data at all.

But I was wrong: it was a flight plan wot dunnit. NATS gave the CAA^» (Civil Aviation Authority - about) an interim report on Monday, which was published yesterday. And in this it says:

Clearly a better way to handle this specific logic error would be for FPRSA-R to identify and remove the message and avoid a critical exception. However, since flight data is safety critical information that is passed to ATCOs the system must be sure it is correct and could not do so in this case. It therefore stopped operating, avoiding any opportunity for incorrect data being passed to a controller. The change to the software will now remove the need for a critical exception to be raised in these specific circumstances.

I don't follow that. It's true that this plan came from Eurocontrol, where it was accepted despite containing contradictory data (distinct waypoints with the same ID likely to be interpreted as an impossible path). But I can't see that a flight plan is safety critical in any sense that prevents it being rejected at least four hours before the aircraft in question reaches UK^▸ (United Kingdom) airspace. Isn't it safety critical in the sense that it must be rejected?

Incidentally, NATS have at least managed to name the supplier of this delinquent software (FPRSA-R) as the Austrian company Frequentis AG.

[ChrisB]

It should have validated it on receipt & kicked it back/rejected acceptance at that time, rather than simply shut everything down & gone to manual input.

[stuving]

The CAA^» (Civil Aviation Authority - about)'s independent Review Panel for this incident has issued an interim report. This concentrates on how NATS and other organisations coped with the loss of the National Airspace System service (NAS). Obviously that aspect will get most of the media interest, including the bit where the panel note that "some relationships between aviation sector stakeholders appear to be adversarial."

However, it does tell us a little more about what caused the system to fail. There was an earlier preliminary report from NATS on this (and there is also a final report, seen by the panel but not yet signed off by NATS and issued). That explained that a duplicate waypoint ID in the flight plan caused the processing failure, but his new report illuminates that further.

The relevant system is FPRSA-R, which takes flight plans from the AMS-UK^▸ (United Kingdom) (Aeronautical Messaging Switch) in the European standard format (ADEXP), identifies and marks the entry and exit points to UK airspace, and converts them into a domestic format and transfers them to NAS. Both AMS-UK and NAS are trusted to keep flight plans safely and not lose them; FPRSA-R is not - it's purely processing.

That means FPRSA-R is allowed to just shut down if it's unhappy, provided the processing is suitably interlocked. Thus when a flight plan is read from the output queue of AMS-UK, it is not removed - just copied. Only after it has been processed and handed over to NAS, and it has been confirmed as accepted, is the AMS-UK queue allowed to delete that entry and offer the next flight plan to FPRSA-R.

Despite all this talk about safety-critical errors, this behaviour of FPRSA-R looks to me like an unhandled exception. The initial recovery action was to restart it, but it tried to process the same flight plan and so failed again. The AMS-UK output queue was stuck with the invalid plan as its leading item. When the makers' expert from Frequentis AG was eventually called in, their advice was to transfer this flight plan into a new, unconnected, queue; in effect to quarantine it. It could then be given to a human operator to see if it could be entered manually, or if not, why not. Restarting FPRSA-R then succeeded, though it was another hour before the system was running again.

We are told NATS have a fix to stop this happening again, but not what this is. I can think to two obvious fixes. The shutdown can be allowed to happen as before, but the operators are trained to recognise its dying message and do this quarantining manually and restart. Alternatively some of this can be automatic: the quarantine action quite easily, avoiding the shutdown of both FPRSA-R systems with more difficulty.

This new report gives an explanation of how FPRSA-R works, and how the duplicate waypoint IDs arise and are handled (though no doubt this is simplified). From that, it seems clear it's just not clever enough. There is enough information available resolve the duplicates, but it isn't used. And behind that, I think there is work going on internationally to get rid of the short IDs that cause the duplications, but it's exactly the kind of non-urgent task for which progress in the international aeronautical community is at best very, very, slow.

[LiskeardRich]

I was caught up in this last august, delayed in Milan by 25 hours.

EasyJet paid me all my expenses within 5 working days, but no compensation eligible as it was outside of their control.

Do we know which airline made the faulty flight plan?

[stuving]

Do we know which airline made the faulty flight plan?

It wasn't "faulty" when written. The processing in Brussels adds extra interpolated waypoints to flight plans, and in this case added one at Deauville, which has the ID code DVL. It already had a waypoint at Devil's Lake (North Dakota), which also has ID code DVL. That was probably a mistake - there's loads of possible waypoints to pick. And, taken in sequence, they can be located 4000 miles apart and the ambiguity resolved. It was the rather simplistic processing in FPRSA-R that led to it being unusable. And that should not have been such a drama.

[TaplowGreen]

Interesting angle on it here too;

https://www.ft.com/content/8a90caa6-bf37-4de3-b0b2-e6d6e257e434

[GBM]

Interesting angle on it here too;

https://www.ft.com/content/8a90caa6-bf37-4de3-b0b2-e6d6e257e434

Unfortunately behind a paywall