KeyChest Blog

Let's Encrypt Certification Authority

Jul 2, 2020 11:21:48 PM / by Dan

Let's Encrypt has grown from zero to an organization that has the power to switch off half of the internet. It issues certificates needed for HTTPS on web servers. HTTPS is now required by all major web browsers.

 

Let's Encrypt is controlled by ISRG aka "A better internet". An organization founded by Mozilla, EFF (Electronic Frontier Foundation), the University of Michigan, Cisco, and Akamai. Its budget is today made up of donations from a large number of sponsors but 40% of all the funding comes from "platinum and gold level sponsors" - a much smaller group that consists of Mozilla, CISCO, OVHcloud, EFF, Chrome, Facebook, and Internet Society, IdenTrust, and The Ford Foundation. 

See "Is Let's Encrypt The Weakest Link of the Internet?" for more details

The initial idea behind Let's Encrypt was to make encrypted connections to World Wide Web servers ubiquitous by providing free SSL certificates. It was so successful that it now issues more than 80% of all internet HTTPS/SSL certificates and is deployed on 60% of all web servers. 

You may remember that the initial idea of HTTPS was to create one global certification authority for email encryption - RFC1422: Privacy Enhancement for Internet Electronic Mail: Part II: Certificate-Based Key Management.

The infrastructure specified in this document establishes a single root for all certification within the Internet, the Internet Policy Registration Authority (IPRA).

Let's Encrypt has got closer to this concept than anyone else.

See "Let's Encrypt - 1bln, Time To Beat Others On Uptime"

How is Let's Encrypt Different

Making certificates free simplifies the surrounding business infrastructure as it eliminates a payment system, web server configuration, validation email management and certificate renewal tasks. The process was eventually simplified to running a single command on Linux systems.

The popularity of Let's Encrypt has lead to its clients being included in official Debian and Ubuntu software repositories.

Let's Encrypt has developed its own protocol for certificate issuance as a technical foundation for full automation - Automatic Certificate Management Environment (ACME). While LetsEncrypt developed the initial version on its own, it also initiated an independent work on an internet standard. Once completed, Let's Encrypt named the final version ACMEv2 as it introduced some breaking changes to its own implementation.

The ACME protocol basically reverses the usual workflow. It firstly executes validation of the requester's control over the domain name for which a new certificate will be requested. This validation produces a token that is valid for 7 days and can be used for repeated requests of new certificates. The requester sends the "famous" CSR as the last step of the issuance process.

See: "How Let's Encrypt Works" for details

Let's Encrypt Disrupts the Internet

Current initiatives of major browser developers such as Mozilla and Google to deprecate unencrypted HTTP are counting on the availability of Let's Encrypt. The project is acknowledged to have the potential to accomplish encrypted connections as the default case for the entire Web (2015).

The effect of Let's Encrypt is however somewhat darker. While it provides encryption, the fact that web browsers (Apple, Microsoft, Google, Mozilla) have exploited this and started enforcing use of HTTPS. The result of this is a new ecosystem that controls our access to the internet. A control that has now independent supervision or governance. Unlike DNS system with a robust governance structure.

Let's Encrypt Technology

The service only issues domain-validated (DV) certificates, since they can be fully automated. Organization Validation and Extended Validation Certificates both require human validation of any registrants, and are therefore not offered by Let's Encrypt. The The domain validation was first introduced by GeoTrust in 2002. At the time, the fact that you obtain a certificate had an actual value and it separated legitimate companies from hackers and criminals.

While this subjective assumption still exists, it is not true any more. Phishing attacks, for example, exploit HTTPS to increase their trustworthiness. As a lifespan of any phishing attack is in hours, the attackers can even claim back the money they might have paid for the HTTPS certificate due to a common "money back guarantee".

While Let's Encrypt claims to aim for the highest level of transparency from the technical point of view, its effect on the security of the internet is not universally accepted with some claiming that it facilitated increase in online crime, including phishing attacks and various techniques of redirection attacks.

The domain ownership can be validated with two main methods: HTTP and DNS validation. The HTTP validation requires the requester to publish a certain file on the web server. The DNS validation checks existence of an TXT DNS record.

From 2018, Let's Encrypt offers wildcard certificates that require a DNS-based validation.

Downsides of Let's Encrypt

The more we thought about the practical aspects of Let's Encrypt, the more objective or possible downsides we found.

The most obvious are rate limits. These restrict the number of operations you can do per second, hour, week, depending on the type of requests. The limits are most severe for the number of certificates you can issue per “registered domain”, e.g. keychest.net. You can only issue 50 certificates per registered domain per week.

See "LetsEncrypt in numbers - all restrictions you should know" for details

But there is a number of other aspects. Let's Encrypt features significant operational differences and may require re-think of certificate management, i.e., how you manage such a certificate on your server.

Short life-span of  certificates - 90 days. As a result, you have to renew your certificates up to 6 times a year. This significantly increases business risk - despite all the automation.

Reliability - it is much worse than commercial CAs.

The recent almost immediate compromise of 3,000,000 certificates raised another question - incident response. How can you obtain quickly new certificates if Let's Encrypt fails and millions of users need to renew their certificates at the same time.

See "What are disadvantages of Let's Encrypt" for more details

Certificate Hierarchy / Certificate Chain

In June 2015, Let's Encrypt announced the generation of their first RSA root certificate, ISRG Root X1. The root certificate was used to sign two intermediate certificates, which are also cross-signed by the certificate authority IdenTrust.

The cross-signing was important to support a quick acceptance of Let's Encrypt on mobile devices and all operating systems where the Let's Encrypt own root certificate has not been accepted yet.

One of the intermediate certificates is used to sign issued certificates, while the other is kept offline as a backup in case of problems with the first intermediate certificate. Because the IdenTrust certificate is preinstalled in major Web browsers, Let's Encrypt certificates can normally be validated and are accepted upon installation even before browser vendors include the ISRG root certificate as a trust anchor.

The IdentTrust cross-signing expires in 2021.

Software Implementation

Let's Encrypt only implements the server side and depends on the open source community to develop end-point clients.

The certificate authority consists of Boulder, a software project written in Go, that implements the server side of the ACME protocol. It uses cfssl, a CloudFlare PKI/TLS toolkit, internally and adds a middleware layer to handle the load, which currently stands at more than 1,500,000 certificates issued daily.
Boulder is published as free software with source code under the terms of version 2 of the Mozilla Public License (MPL). It provides a RESTful API that can be accessed over a TLS-encrypted channel.
 
Let's Encrypt issues certificates valid for 90 days. The reason given is that these certificates "limit damage from key compromise and mis-issuance" and encourage automation. It is less controversial now than it was back in 2015 as browsers are cutting the maximum validity to 1 year in September 2020.
 

Clients

There is currently a large number of Let's Encrypt clients for a range of platforms and use cases. Certbot was the first  official client implementation (called letsencrypt then). Acme.sh is the most popular Linux "shell" client.

There are obvious, popular, clients that you can see mentioned everywhere. Here's a short list of projects - in no particular order - that tried to approach the integration differently. Maybe I also like them as they share something with our KeyChest.

  • tls_certificate_generation - a Docker project for renewing certificates for domain that are resolved by internal DNS. It launches a temporary AWS/DO VMs. ... the project looks dead (doesn't seem to support RFC8555 / ACMEv2).
  • haproxy-lua-acme - an HAproxy client. When you need to renew a certificate, you send a POST request to your HAproxy ... instead of to the LE API directly.
  • Greenlock - just because the author has built a whole set of tools written in JavaScript for various use cases - see below.
  • freshcerts - a Ruby project (also not supporting ACMEv2) that creates a centralized service allowing lightweight scripts on endpoints. I suspect that it fails with ACMEv2 as it doesn't allow changing the server address.
  • acme-client - the BSD version, well because it is part of the distro.
  • acmetool - the Go project - because it doesn't have any dependency and works as a server listening on port 402.

See "ACMEv2 client analysis" for a long list

Let's Encrypt Reliability / Uptime

Let's Encrypt publishes its planned changes as well as incidents at https://letsencrypt.status.io

We have analyzed the data for years 2017 and 2019. The results were substandard, compare to commercial CAs. The main question is whether Let's Encrypt can ever reach the reliability of commercial CAs and whether LE has such an aspiration.

See "Let's Encrypt uptime (2017) is 98.8% for some" for more details

See "Let's Encrypt Uptime - 2 years on" for more details.

LE's main feature is automation. While we generally don't mind if a certificate is renewed today or a week later, the relatively low uptime is a concern. Especially as there are no official service level guarantees.

This "exercise" made me also wonder whether it makes sense to implement an uptime monitoring system that would provide an independent uptime data. LE issues around 1.5 million certificates a day so every one minute downtime impacts over 900 certificate requests. Similarly, 1 minute downtime of its OCSP responders is significant as 50-60% of websites use Let's Encrypt certificates. Arguably, these websites will not include any of the large one but still.

Click the image below to open the KeyChest real-time monitoring dashboard.

 

First downtime detection - Let's Encrypt

History of Let's Encrypt

The Let's Encrypt project was started in 2012 by two Mozilla employees, Josh Aas and Eric Rescorla, together with Peter Eckersley at the Electronic Frontier Foundation and J. Alex Halderman at the University of Michigan. 

Let's Encrypt was announced publicly on November 18, 2014. The group has realized that more help is needed to develop a robust protocol. They submitted the first protocol design for the IETF standardization in January 2015.

The ISRG and the Linux Foundation declared their collaboration on the Let's Encrypt project later in 2015.

The first set of root and intermediate certificates was generated on June 16, 2015, the final launch schedule for the service was announced, with the first certificate expected to be issued sometime in the week of July 27, 2015, followed by a limited issuance period to test security and scalability.

General availability of the service was originally planned to begin sometime in the week of September 14, 2015 but the schedule has been changed to allow more time for testing and to ensure system security and stability.

On September 14, 2015, Let's Encrypt issued its first certificate, which was for the domain helloworld.letsencrypt.org. On the same day, ISRG submitted its root program applications to Mozilla, Microsoft, Google and Apple.

The intermediate certificates became cross-signed by IdenTrust in October 2015. As IdentTrust was accepted on all major platforms, Let's Encrypt has become trusted by all major browsers overnight.

The public beta of Let's Encrypt ran from December 3, 2015 to April 12, 2016.

Governance Structure

Board members (as of March 2020)

Technical Advisory Board

The Biggest Incident Ever

On March 3, 2020, Let's Encrypt announced that it would have to revoke over 3 million certificates on March 4, due to a flaw in its Certificate Authority software. Through working with software vendors and contacting site operators, Let's Encrypt was able to get 1.7 million of the affected certificates renewed before the deadline. They ultimately decided not to revoke the remaining affected certificates, as the security risk was low and the certificates were to expire within the next 90 days.

See: "Let's Encrypt Revokes 3,000,000 certificates" for more detail

In March 2020, Let's Encrypt was awarded the Free Software Foundation's annual Award for Projects of Social Benefit.[49]

 

Note: we used the Wikipedia Let's Encrypt entry as a basis and expanded the text with more details.

 

Dan

Written by Dan