Forty-Eight Point Eight

At 16:52:26 on Friday 9 August 2019, the control room that runs Britain's electricity was doing the ordinary thing. It was holding enough spare power to cover the loss of the biggest single generator on the system at that moment, which is what the rules oblige it to do and what it does every minute of every day. That evening the number was 1,000 megawatts.

Eighty-three seconds later it had lost 1,878, and a million homes were being switched off on purpose.

What the standard actually promises

The rule is in the Security and Quality of Supply Standard, and it is narrower than most people assume. For a normal infeed loss the frequency is not supposed to move more than half a hertz from fifty. For the rarer, larger loss it must not go outside 49.5 to 50.5 hertz for more than a minute. The largest infrequent loss the system was built around is 1,260 megawatts, which is Sizewell running flat out.

Nothing in it promises that two unrelated power stations will not fail in the same second.

Conditions were unremarkable. About 32 gigawatts of capacity was available, more than 30 per cent of it wind and half of it conventional plant, against a forecast demand of 29 gigawatts. The Met Office had issued yellow warnings, the lowest grade it has. The report is blunt: weather conditions on 9 August were not unusual.

Eighty-three seconds

At 16:52:33 three lightning strikes hit close together on the 400 kilovolt circuit between Eaton Socon and Wymondley. One of them caused a single phase fault to earth. The protection did exactly what protection is for. The Wymondley end opened 70 milliseconds after the fault and the Eaton Socon end 74 milliseconds after it, and the circuit was back in service twenty seconds later.

That part worked.

Half a second after the fault, the Hornsea offshore wind farm was generating 799 megawatts. Two tenths of a second later it began shedding load, and a tenth of a second after that it settled at 62. It had dropped 737 megawatts. At almost the same instant the steam turbine at Little Barford, a gas station in Bedfordshire eighty miles away with no connection to Hornsea beyond the grid itself, tripped and took 244 megawatts with it.

Neither should have happened. Generation is not expected to trip or de-load because of a lightning strike on a nearby circuit, and the two events were unconnected.

The generators nobody dispatched

Then came the part that is easy to miss and did most of the damage.

Scattered across the distribution networks, below the level the national control room dispatches or even sees in real time, sit hundreds of small generators. Solar farms, diesel sets, gas engines behind factories. Each carries protection designed to detect that it has been cut adrift from the wider grid, because a generator quietly energising a dead cable can kill an engineer working on it.

Two of those detection methods fired that evening. About 150 megawatts switched off on vector shift protection at the moment of the lightning strike. As the frequency then began to fall, roughly another 350 megawatts switched off on rate of change of frequency protection.

Add it up. Of the 1,481 megawatts gone within one second, about a third came from plant that nobody had ordered off, that the control room could not have held cover against, and that disconnected precisely because it was working as designed. Each of those machines protected itself and made the hole deeper for everyone else.

The frequency fall was arrested at 49.1 hertz at 16:52:58 and had recovered to 49.2 by 16:53:18. Britain nearly got away with it.

Then, at 16:53:31, one of Little Barford's gas turbines tripped from 210 megawatts, on a protection setting, following a bypass valve that had failed to operate. Cumulative loss, 1,691 megawatts. There were no reserves left.

48.8

At 16:53:49.398 the frequency crossed 48.8 hertz, and relays on the distribution networks began opening without asking anybody.

This is the Low Frequency Demand Disconnection scheme, the last defence before the grid comes apart. The trade is deliberate and old: cut a defined block of customers instantly so the remaining generation can hold up the rest. It disconnected 931 megawatts, about 3.2 per cent of national demand, and with it 1,152,878 customers. It had not been used in over a decade.

Nine seconds later Little Barford's second gas turbine went too, another 187 megawatts, bringing the total loss to 1,878. Nobody noticed, because the demand had already been cut to match.

Frequency was back at fifty by 16:57:15. Restoration was instructed from 16:58 and the last customers came back at 17:37. Most were off for between fifteen and forty five minutes. The scheme worked.

The trains

The Class 700 and 717 trains running north of Farringdon and King's Cross shut themselves down when the frequency dipped below 49 hertz. Their own technical specification says they will keep going down to 48.5 for short periods. They did not.

About sixty trains stopped. Half were recovered by their drivers, a reboot that takes around ten minutes. The other thirty were locked out and needed a technician to attend with a laptop. The fleet was midway through a software change, and the drivers could not recover the ones running the new version.

Twenty three trains were evacuated. Three hundred and seventy one services were cancelled, 220 part cancelled and 873 delayed. St Pancras and King's Cross closed for hours because of overcrowding, and Euston went exit only. Network Rail's own traction supply held up throughout.

Ipswich Hospital lost half its load in fourteen seconds to its own internal protection, and was never on the disconnection list at all. Newcastle Airport was on the list, and lost supply for eighteen minutes.

What it cost

Ofgem investigated. Hornsea One, co-owned by Orsted, and Little Barford, operated by RWE, each paid 4.5 million pounds into the regulator's redress fund. UK Power Networks paid in as well, for restoring customers before the system operator had confirmed it was safe.

The engineering lesson in the report is short: review the protection settings inside electric trains so they ride through normal disturbances, and check that no hospital or airport has been quietly placed on the list of things to sacrifice.

On 12 August, three days after the lights came back, Newcastle Airport wrote to its network operator asking to be categorised as a Protected Site. It is now registered as one.

Sources

• National Grid ESO, "Technical Report on the events of 9 August 2019", 6 September 2019 (the security standard limits of 0.5Hz for a normal infeed loss and 49.5 to 50.5Hz for an infrequent one; the 1,000MW being secured at 16:52:26 and the 1,260MW infrequent maximum loss risk with Sizewell at full load; system conditions of approximately 32GW capacity, over 30 per cent wind with an expected peak of 10GW, and 29GW forecast demand; the Met Office yellow warnings and the statement that conditions were not unusual; the three lightning strikes on the Eaton Socon to Wymondley 400kV circuit at 16:52:33 and the single phase to earth fault; the 70 and 74 millisecond protection operations and re-energisation at 16:52:53; Hornsea generating 799MW at 16:52:33.531, deloading from 16:52:33.728 and stabilising at 62MW at 16:52:33.835 for a loss of 737MW; the Little Barford steam turbine trip of 244MW; approximately 150MW of embedded generation lost on vector shift protection and approximately 350MW on rate of change of frequency protection; the cumulative loss figures of 1,131MW, 1,481MW, 1,691MW and 1,878MW; the frequency being arrested at 49.1Hz at 16:52:58 and recovering to 49.2Hz at 16:53:18; the GT1A trip of 210MW at 16:53:31 following the failure of a bypass valve to operate; the LFDD trigger at 48.8Hz at 16:53:49.398 disconnecting 931MW and 1,152,878 customers, being 3.2 per cent of national demand, and not having been used in over a decade; the GT1B trip of 187MW at 16:53:58; the return to 50Hz at 16:57:15, restoration instructed from 16:58 to 17:16 and completed at 17:37, with customers off for between 15 and 45 minutes; the Class 700 and 717 shutdown below 49Hz against a specification of 48.5Hz, approximately 60 trains with half recovered by drivers in around ten minutes and 30 requiring a technician with a laptop, and the fleet being midway through a software change; 23 evacuations, 371 cancellations, 220 part cancellations and 873 delays, with St Pancras and King's Cross closed and Euston exit only; Network Rail traction supply being maintained; Ipswich Hospital losing half its load in 14 seconds to its own protection while not being in the LFDD zone; Newcastle Airport losing supply for 18 minutes and requesting Protected Site status on 12 August; and the recommendations on train protection settings and on reviewing the list of facilities connected to the LFDD scheme).
• Ofgem, "Investigation into 9 August 2019 power outage" (the finding that the combined loss of two large generators together with smaller losses at local level triggered the disconnection; the voluntary redress payments of 4.5 million pounds each from Hornsea One and Little Barford; and the separate payment by UK Power Networks for beginning restoration before receiving confirmation from the system operator).
• Ofgem, "9 August 2019 power outage report" (the regulator's own account of the causes and of the compliance issues examined).
• Energy Emergencies Executive Committee, "GB power system disruption on 9 August 2019: Final report" (the cross industry review of the disruption and its consequences for critical services).