Five Hundred and Thirty One

The padlock in the address bar means a third party has checked that the site you are looking at is the site it claims to be. That third party is called a certificate authority, and the arrangement your browser makes with it is more generous than most people would guess.

Your browser trusts a list of them. Any one on that list can issue a valid certificate for any address on the internet. Not just its own customers. Any address. Your bank's, your government's, anyone's.

That means the security of the whole arrangement is the security of its weakest member, and in 2011 the weakest member turned out to be a company in the Netherlands that had been started by a notary.

What DigiNotar was

DigiNotar was set up in 1998 by Dick Batenburg, a Dutch notary, and it did what the name suggests. It issued digital certificates, largely for Dutch notaries and, increasingly, for the Dutch government. It was part of the state's own public key infrastructure, the thing that signs the pages where citizens file their taxes.

It was also, by the standards of the companies whose names sit in a browser's trust store, small. In January 2011 it was bought by VASCO Data Security International.

What happened in June

On 19 June 2011, DigiNotar detected an intrusion.

The response to that sentence is the whole story. A certificate authority that has been broken into has one obligation above every other, which is to tell the browser makers immediately, because the browser makers are the only people who can withdraw the trust. Every hour of silence is an hour in which certificates minted by the intruder are honoured by every computer on earth.

On 10 July, three weeks after the detection, a wildcard certificate for Google was issued from DigiNotar's systems. A wildcard covers every subdomain: mail, accounts, docs, everything.

In total, more than 531 fraudulent certificates were eventually identified. They were not only for Google. The list includes Yahoo, Mozilla, WordPress and The Tor Project, which is the software people use precisely when they believe somebody is watching them.

Three hundred thousand inboxes

A certificate on its own does nothing. It has to be presented to a browser by something sitting between the reader and the site, and that requires control of the network.

Through August, users on several Iranian internet service providers started seeing certificate problems. What was happening is what the certificate was for. Traffic to Google's mail service was being intercepted, decrypted, read and passed on, and the browsers were not complaining, because the certificate being presented was signed by an authority the browsers trusted.

The estimate is that around 300,000 Gmail users, almost all of them in Iran, were affected.

Consider what that means for the person on the other end. They had done everything correctly. They had used an encrypted connection. They had checked for the padlock, if they were the sort of person who checks. The padlock was there. It was simply lying, and there was no action available to them that would have revealed it.

On 28 August the rogue Google certificate was posted publicly. It had still not been revoked.

The month it took to die

After that, everything happened at once.

Google, Mozilla, Microsoft, Apple and Opera removed DigiNotar from their trust stores between late August and mid September. That is the only real sanction in this system and it is absolute: once the root is gone, every certificate the company ever issued stops working everywhere, including the legitimate ones.

On 3 September the Dutch government took over operational management of the company, because the state's own tax and public services were signed with its certificates and somebody had to keep them running while they were replaced.

On 14 September the Dutch telecoms regulator withdrew DigiNotar's accreditation. On 20 September it filed for bankruptcy. Ninety three days from the detected intrusion to the end of the company.

Fox-IT, the Dutch firm brought in to investigate, called its report Operation Black Tulip. Its findings on the state of the network were unkind, and the pattern is familiar from every breach report since: inadequate separation between systems, weak passwords, patching left undone, and logging too thin to reconstruct what had been done.

What was actually broken

It is tempting to file this as one badly run company, and DigiNotar was one. But the design problem it exposed had nothing to do with the Netherlands.

Browsers ship with a list of root certificates that runs to well over a hundred organisations, and the exact number moves as roots are added and removed. They sit in many different countries, under many different legal systems, with wildly different levels of competence and wildly different degrees of independence from whoever governs them. Every single one can issue a certificate for any name.

There is no technical reason a browser in Mumbai should accept a certificate for an Indian bank from an authority on another continent that the bank has never dealt with. It does so because the trust model has no notion of which authority is supposed to vouch for whom. It has a list, and the list is flat.

What came out of it

The fix, when it arrived, was not to make the authorities more trustworthy. It was to make them observable.

Certificate Transparency was published as RFC 6962 in June 2013, in the wake of DigiNotar and the cases that followed it. The idea is a public log, append only, into which every certificate is written as it is issued. Append only is enforced with a structure that makes it possible to prove that today's log contains everything yesterday's log contained, so entries cannot be quietly removed later.

That does not stop an authority issuing a certificate it should not. It makes it impossible to do so secretly. Google can watch a public log for certificates issued for Google's own names by authorities Google has never used, which is exactly the signal that was missing in July 2011.

Chrome began requiring it for extended validation certificates in January 2015, and for every publicly trusted certificate issued after 30 April 2018. A certificate that is not in a log is not valid, whoever signed it.

The part that did not change

The list is still flat. Your browser still trusts every authority on it to speak for every website in the world, and the reason that is tolerable now is not that the authorities got better. It is that somebody is finally writing down what they do, in public, where the people being impersonated can see it.

In July 2011 nobody was writing it down. A certificate for Google's mail was issued in the Netherlands and used in Iran for six weeks, and the first person to raise it was a user who noticed that his browser had started behaving strangely and posted about it.

Sources

• Wikipedia, "DigiNotar" (the founding of the company in 1998 by the Dutch notary Dick Batenburg and its role in Dutch government public key infrastructure; its acquisition by VASCO Data Security International in January 2011; the detection of an intrusion on 19 June 2011; the issue of a wildcard certificate for Google on 10 July 2011; the identification of more than 531 fraudulent certificates covering domains including Google, Yahoo, Mozilla, WordPress and The Tor Project; certificate problems observed on Iranian internet service providers on 28 and 29 August 2011 and the public posting of the rogue certificate; the man in the middle attack affecting approximately 300,000 Iranian Gmail users; the Dutch government taking over operational management on 3 September 2011; the bankruptcy filing on 20 September 2011; and the removal of DigiNotar roots by Mozilla, Microsoft, Google, Apple and Opera between late August and mid September 2011).
• Fox-IT, "Operation Black Tulip: report of the investigation into the DigiNotar certificate authority breach", via ENISA (the commissioning and scope of the investigation, the rogue certificate abused in a large scale man in the middle attack, and the findings on the state of DigiNotar's network including separation between systems, password strength, patching and the adequacy of logging).
• Computerworld, "DigiNotar dies from certificate hack caper" (the withdrawal of accreditation by the Dutch regulator and the bankruptcy that followed within days).
• The Register, "Audit says DigiNotar was a total shambles" (the Fox-IT findings on the state of DigiNotar's network, including separation between systems, password strength, patching and logging).
• Mozilla, "CA/Included Certificates" (the list of certificate authorities whose roots are included by default, which is the basis for the order of magnitude given in the text).
• Internet Engineering Task Force, "RFC 6962: Certificate Transparency", June 2013 (the protocol for publicly logging the existence of TLS certificates as they are issued or observed, and the append only log structure).
• Mozilla, "Certificate Transparency" (the public append only log of issued certificates, the use of Merkle trees to prove that one version of a log is a superset of an earlier one, and the origin of the scheme in the certificate authority compromises of 2011 and 2012).
• Google Chrome, "Certificate Transparency in Chrome" (the requirement that all publicly trusted certificates issued after 30 April 2018 be logged in order to be recognised as valid, and the earlier requirement for extended validation certificates from January 2015).