![](\"https:\/\/media.wired.com\/photos\/5e618de2dd088e00081102ab\/master\/pass\/Security_3milsites_934348-006.jpg\"\/)
<\/p>\n<\/p>\n
Credit to Author: Brian Barrett| Date: Mon, 09 Mar 2020 17:15:52 +0000<\/strong><\/p>\n Brian Barrett<\/span><\/a><\/span> <\/span><\/p>\n This is a story about something that could have gone wrong on the internet this week but instead turned out mostly OK. How often can you say that?<\/p>\n Around 9 o\u2019clock on the East Coast on Friday, February 28, bad news arrived on the doorstep of Let\u2019s Encrypt. An arm of the nonprofit Internet Security Research Group, Let\u2019s Encrypt<\/a> is a so-called certificate authority that lets websites implement encrypted connections at no cost. A CA parcels out digital certificates that essentially vouch that a website isn't an imposter. That cryptographic guarantee is the backbone of HTTPS<\/a>, the encrypted connections that keep anyone from intercepting or spying on your interactions with websites.<\/p>\n Those certificates expire after a set amount of time; Let's Encrypt caps its certificates at 90 days, at which point a site operator has to renew. It's a largely automated process, but if a site doesn't have an active certificate, your browser will notice and may not load the page you're trying to visit at all.<\/p>\n Think of it sort of like updating the registration on your car every year. If your tags expire, you'll get pulled over.<\/p>\n Let's Encrypt's work is technical and happens in the background. But in a few short years it has helped make the internet much more secure on a fundamental level. Plenty of companies offer security certificates; Let\u2019s Encrypt just took the audacious step of making them free. A week ago, it issued its billionth<\/a> certificate.<\/p>\n But that ubiquity also means that when a pebble drops in the middle of Let\u2019s Encrypt\u2019s pond, the ripples can travel a long way. On February 28, the pebble was a bug that threatened to effectively render 3 million sites nonfunctional in a matter of days.<\/p>\n "You can\u2019t be mostly correct."<\/p>\n Kenneth White, MongoDB<\/p>\n The flaw itself? Relatively minor in the grand scheme of the internet. Let's Encrypt uses software called Boulder to make sure that it's allowed to issue a certificate to a site. (Some high-value targets, like banks, specify that they'll only accept certificates from a particular CA. Let's Encrypt has solid security, but some paid certificate authorities offer warranties in the event anything goes wrong, as well as other upgrades. It's the difference between, say, having a strong deadbolt and adding renter's insurance.) Boulder confirms that Let's Encrypt is honoring those preferences when it first issues a certificate and again 30 days later. Or at least, it\u2019s supposed to; the bug meant it was skipping the second check. And that\u2019s a big no-no.<\/p>\n The actual security implications of that backend hiccup were minimal, says ISRG executive director Josh Aas. At the same time, Let\u2019s Encrypt couldn\u2019t let a bug that affected 2.6 percent of its active certificates\u20143,048,289 in all, when it confirmed the issue\u2014linger indefinitely. \u201cThe severity of the bug here is not very high,\u201d says Aas. \u201cBut these 3 million certificates were issued in a noncompliant way. We have an obligation to revoke them.\u201d<\/p>\n That obligation stems from the Certification Authority Browser Forum, or CA\/B, an industry group that sets strict standards about the use of certificates. In this case, those standards gave Let's Encrypt a five-day window to come back into compliance, which would entail revoking every certificate that was affected by the bug. The alternative for Let's Encrypt was ignoring the CA\/B and letting it slide, but that was really no option at all.<\/p>\n \u201cThey did the right thing. The CA\/B sets these rules and has fairly strict requirements, which you want. When a person or computer talks to another computer, you want to make sure they\u2019ve met some identity criterion,\u201d says Kenneth White, security principal at MongoDB, a massive database provider that uses Let\u2019s Encrypt. \u201cYou can\u2019t be mostly correct. You\u2019ve got to follow the guidelines for how to enforce these things.\u201d<\/p>\n The impact of pulling those certificates would be swift and severe. Once browsers like Chrome and Firefox found them missing, they would flash warnings to any visitors that the sites weren\u2019t safe. Some browsers would block access altogether. A not insignificant chunk of the internet would effectively be taken out of commission. All because of this one small flaw in one niche corner of the Let\u2019s Encrypt operation.<\/p>\n Within two minutes of confirming the bug, the Let\u2019s Encrypt team stopped issuing any new certificates in a bid to stanch the bleeding. A little over two hours after that, they fixed the bug itself. And then they let everyone know what was coming.<\/p>\n \u201cWe can\u2019t contact everybody, so we started contacting the largest subscribers, telling them about the situation, getting them as informed as possible,\u201d says Aas. \u201cAnd then we worked with them to get them to replace their certificates as quickly as possible.\u201d<\/p>\n Once a site operator renewed a certificate, Let\u2019s Encrypt could safely revoke the old one. No harm would befall the site. Which sounds like a simple enough solution\u2014but nothing\u2019s simple at this kind of scale.<\/p>\n Bigger organizations had an easier time fixing the problem, because they generally have the resources to monitor any signs of trouble that surface and the tools to automate the renewal process. \u201cIf you\u2019ve got a dozen or two dozen servers or something, that\u2019s some poor sleepy-eyed soul in the middle of the night at a keyboard,\u201d says MongoDB\u2019s White. \u201cWe reissued a little over 15,000 certificates [for clients], and we did it in a few hours. There was some work involved, but it wasn\u2019t catastrophic. We had measures in place to be able to rotate quickly.\u201d<\/p>\n Smaller sites got a big assist from the Electronic Frontier Foundation, which operates Certbot, a free software tool that automatically adds Let\u2019s Encrypt certificates to sites and renews them every 60 days. In the last two months alone, Certbot has generated certificates for 19.2 million unique sites. \u201cFortunately we had anticipated the need to check revoked certificates for renewal in 2015,\u201d says EFF engineering director Max Hunter. \u201cBecause Let's Encrypt communicated the issue early, and the code path for the query was already in place, our work was relatively straightforward.\u201d By Tuesday a team from EFF, along with volunteers in Paris and Finland, had updated Certbot to renew any revoked certificates.<\/p>\n Meanwhile, Let\u2019s Encrypt sent an email to every address it had on file. It created a searchable database of every affected domain so that hosting companies could see if they needed to act. \u201cWe marked those certificates as expired in our internal system, and then our normal automated processes kicked in to generate and deploy new certificates,\u201d says Justin Samuel, CEO of Less Bits, a startup that operates hosting company ServerPilot.<\/p>\n On Tuesday night, 30 minutes before the deadline, Let\u2019s Encrypt made another announcement. Of the 3 million potentially impacted sites, 1.7 million had managed to renew their certificates, an astonishing number given the short window of time. \u201cNo other CA comes close to making large-scale cert reissuing not only feasible but also fast,\u201d says Samuel.<\/p>\n That success also emboldened Aas to make a difficult call. Let\u2019s Encrypt would let the remaining certificates slide. \u201cWe made the decision that instead of breaking more than a million websites, potentially, we just aren\u2019t going to revoke them by the deadline,\u201d says Aas. \u201cWe think it\u2019s the right decision for the health of the internet.\u201d<\/p>\n It was the internet equivalent of a call from the governor minutes before midnight. Let\u2019s Encrypt will continue to revoke certificates if it can confirm that the sites have renewed them, but otherwise it is content to leave them be in their slightly broken form. The security risk is small, Aas says, and since Let\u2019s Encrypt certificates are only viable for 90 days to begin with, any stragglers will have washed out of the ecosystem by summertime at the latest.<\/p>\n \u201cIf anything, this just reinforces that they are one of the most transparent, modern certificate authorities in the world,\u201d says MongoDB\u2019s White, who points to previous certificate snafus that for-profit companies like Symantec<\/a> have badly mishandled. \u201cIt\u2019s easy to armchair quarterback. But I think if people are overly critical that\u2019s misplaced.\u201d<\/p>\n The intricacies of internet infrastructure are generally ignored until something goes terrible wrong. This time, though, it\u2019s useful to reflect on what went right. For once, the story is that nothing broke.<\/p>\n