The only things that continue to amaze me are the number of (mostly "enterprise") software products that simply won't get with the times (or get it wrong, like renewing the cert, but continuing to use the old one until something is manually restarted), and the countless IT departments that still don't support any kind of API for their internal domains...
You've got to remember that the reduction to a 45-day duration is industry-wide - driven by the browsers. Any CA not offering automated renewal (which in practice means ACME) is going to lose a lot of customers over the next few years.
Yeah, no one's rewriting a bunch of software to support automating a specific, internet-facing, sometimes-reliable CA.
Yes it's ACME, a standard you say. A standard protocol with nonstop changing profile requirements at LE's whim. Who's going to keep updating the software every 3 months to keep up? When the WebPKI sneeze in a different direction and change their minds yet again. Because 45 will become 30 will become 7 and they won't stop till the lifetime is 6 hours.
"Enterprise" products are more often than not using internal PKI so it's a waste.
I would like to see the metrics on how much time and resources are wasted babysitting all this automation vs. going in and updating a certificate manually once a year and not having to worry the automation will fail in a week.
Also, everything is using https now. Living in a low-income country, certificates were too expensive to use them where they weren't absolutely required, but not anymore. This is orthogonal to automation, I'm just pointing out that LE is not as demonic as you make it out to be.
I'm afraid enterprise users are on their own, probably approximately no-one else is interested in going back to the old ways of doing it. (Maybe embedded.)
Well, I could regale you with my anecdotes on how all my 'grab a LetsEncrypt cert on Docker deploy and whenever remaining lifetime goes below 75%' services have not had a single TLS-related outage in at least a year, and how we just had a multi-day meltdown caused by a load-bearing cert that everyone forgot about expiring, but I doubt you'll be interested.
I'm not here to take away your right to set up a few more 5-year+ internal CAs (or, hey, how about an entire self-signed hierarchy? can't be that hard...), and neither is LetsEncrypt. But on the Big Bad Internet (where Google is the leading cause of More Security, and LetsEncrypt is merely the leading vendor catering to that), things have moved on slightly.
I have multiple systems. Legacy that I've inherited, and modern that are able to automatically update their certs.
The automated updates require almost zero maintenance, there was a week a couple of years ago when I had to switch some configuration for a new root cert.
The legacy system requires manual installation and multiple weeks of my time every year because of the number of environments, and since generation of the certs requires me to file a request for someone to manually create it, they invariably typo something and it has to be redone everywhere.
So multiple engineers, over multiple weeks?
Manual process at a guess is £50k pa, while the automated is close to an annual £0?
After all, the logical approach is to schedule a daily cron job for your renewal process, which only contacts the server when the current cert has less than X days of validity remaining. Scheduling a one-off cron job every, say, 60 days would be extremely error-prone!
With regards to future reductions: the point is to force companies to automate renewal, as they have proven to be unable to renew in time during incidents due to dozens of layers of bureaucracy. 45 days means one renewal a month, which is probably sufficient for that. Anything below 7 days isn't likely to happen, because at that point the industry considers certification revocation unnecessary.
Internal PKI is not relevant here. Anyone with weird internal cert use shouldn't be using public certs anyways, so moving those people to certs backed by self-signed internal CAs is a win. They are free to use whatever validity they want with those.
I would also like to see those metrics, because I strongly suspect the cost is dramatically in favor of automation, especially when you consider the improved security posture you get from short lived certificates. My personal experience with Let's Encrypt has been that the only hassle is in systems that don't implement automation, so I get the worst of both worlds having to effectively manually renew short lived certificates every 90 days.
CRL based revocation is largely a failure, both because implementation is a problem and because knowing when to revoke a certificate is hard. So the only way to achieve the goal of security in WebPKI that's resilient to private key compromise is short-lived certificates. And the only way to implement those is with automated renewal.
It really doesn't change that often. And whether this is a "breaking" change is something that's debatable (you shouldn't hard-code the cert lifetime, but I suspect many programs do, so de-facto it may or may not be a breaking change).
If you look at the Go implementation for example (https://github.com/golang/crypto/commits/master/acme), then there haven't been any major changes since it was first written in 2018: just bugfixes and basic maintenance tasks. Maybe I missed a commit but it certainly doesn't require 3-month update cycles. You can probably take that 2018 code and it'll work fine today. And since it doesn't hard-code the 90 days it will continue to work in 2028 when this change is made. So it's more "you need to keep updating the software every decade to keep up".
> We expect DNS-PERSIST-01 to be available in 2026
Very exciting!
https://datatracker.ietf.org/doc/html/draft-sheurich-acme-dn...
I really would have felt better with a random token that was tied to the account, rather than the account number itself. The CA side can of course decide to implement it either way , but all examples are about the account ID.
There are pros and cons of various approaches.
Don't pin to certs you don't control.
The real modern way to do certificate pinning is to not do certificate pinning at all, but I'm sure that you've already heard this countless times before. An alternative option would be to run your own private CA, generate a new public/private keypair every 45 days, and generate certificates with that public key using both your private CA and Let's Encrypt, and then pin your private CA instead of the leaf certificates.
It's allowed, but the intent of short cert expiration is to also have short private key lifetimes, so that point in time key compromises have time limited forgery potential. If the same key is used for a long period, someone who got the key once can download your public certificate and use that with the key they have.
> The real modern way to do certificate pinning is to not do certificate pinning at all, but I'm sure that you've already heard this countless times before. An alternative option would be to run your own private CA, generate a new public/private keypair every 45 days, and generate certificates with that public key using both your private CA and Let's Encrypt, and then pin your private CA instead of the leaf certificates.
The tricky thing here is you need to serve your private CA signed cert to pinned clients and a PKI cert to browser clients. If you want cert pinning for browsers (which afaik, is very limited availability), you should pick at least two CAs that you are pretty sure will continue to issue you certs and won't be delisted; bonus if you're also confident they won't issue certs for your domains without your explicit consent.
I would also recommend two private CAs, if you're doing private CAs. Store them physically separate, so if one is compromised or becomes unavailable, you can use the other.
- Make sure your app checks that enough trusted embedded Signed Certificate Timestamps are present in the certificate (web browsers and the iOS and Android frameworks already do this by default).
- Disallow your app to trust certificates that are more recently requested than N hours. This might be hard to do.
- Set up monitoring to the certificate transparency logs to verify that no bad actor has obtained a certificate (and make sure you are always able to revoke them within N hours).
- Make sure you always have fresh keys with certificates in cold storage older than N hours, because you can't immediately use newly obtained certificates
On the other hand, if you're using an app specific server, there's no need for you to use public certificates. A self-generated one with a five or ten year validity will pin just as nicely. That breaks if you need web browsers or third parties to talk to the same API, of course.
But then they set some shorter lifetime, and I was forced to set up automation and now I've gotten a way of doing it and it's pretty easy to do. So now I either `cloudflared` or make sure certbot is doing its thing.
Perhaps if they'd made that more inconvenient I would have started using Caddy or Traefik instead of my good old trusty nginx knowledge.
It is already getting dangerously close to the duration of holiday freeze windows, compliance/audit enforced windows, etc.
Not to mention the undue bloat of CT logs.
How do those affect automated processes though? If the automation were to fail somehow during a freeze window, then surely that would be a case of fixing a system and thus not covered by the freeze window.
> Not to mention the undue bloat of CT logs.
I'm not sure what you mean by "CT logs", but I assume it's something to do with the certificate renewal automation. I can't see that you'd be creating GBs of logs that would be difficult to handle. Even a home-based selfhosted system would easily cope with certificate logs from running it hourly.
Reducing the lifetime of certificates increases the number of certificates that have to be issued, and therefore the number of certs that are logged to CT. This increases the cost to CT operators, which is unfortunate since the set of operators is currently very small.
However, a number of recent improvements (like static-ct-api and the upcoming Merkle Tree Certs) are making great strides in reducing the cost of operating a CT log, so we think that the ecosystem will be able to keep up with reductions in cert lifetime.
So you can start renewing with 30d of lifetime remaining. You probably want to retry once or twice before alerting. So lets say 28d between alert and expiry.
That seems somewhat reasonable. But is basically the lower margin of what I consider so. I feel like I should be able to walk away from a system for a month with no urgent maintenance needed. 28d is really cutting it close. I think the previous 60d was generous but that is probably a good thing.
I really hope they don't try to make it shorter than this. Because I really don't want to worry about certificate expiry during a vacation.
Alternatively they could make the acceptable behaviour much higher. For example make 32d certificates but it is acceptable to start renewing them after 24h. Because I don't really care how often my automation renews them. What matters is the time frame between being alerted due to renewal failure and expiry.
You might want to consider force-renewing all your certs a few days before your vacation. Then you can go away for over 40 days. (Unless something else breaks…)
Hopefully their software stack will be able to automate this by 2028.
- If you control the DNS record at the registrar, you can point it to whatever server you want and get a Let’s Encrypt certificate that way. Never mind the old model. So you need to trust the registrar, if they’re compromised that’s it.
- Back in the day, DNS is cached and plaintext and cannot be trusted
- Now, you can use DNSSEC. And you could have the registrar sign the DNS entry, so a cache/ISP/corporate can’t forge it…
- So: why not stick a key, with metadata (e.g. expiration) in a signed DNS record. Now you only trust one person (the registrar) rather than two (the registrar and the CA)
- And there’s no extra paperwork or process since you need to do all that stuff anyways to register the domain.
I don’t know much networking. What’s the flaw in this argument? Is there one?
I’m imagining a world where instead of a trust database of CAs, you instead trust the keys of the DNS root servers, and the CA system and DNS system merge into one.
This also gives you a neat story for custom certs, etc, as it becomes a domain problem. It’s now perfectly natural to say “I trust the X root of trust for domains Y” for e.g. your internal system because that’s what you’re using to resolve those URLs.
To be fair, part of this is my fascination with the idea of using cryptographic keys as addresses, like in many a cool protocol, making DNS literally into a CA of the form “assign this name to this identity (key)”.
https://sockpuppet.org/blog/2015/01/15/against-dnssec/
https://sockpuppet.org/blog/2016/10/27/14-dns-nerds-dont-con...
- For the bit.ly thing; as far as I understand Gaddafi could at least theoretically order the registrar to point somewhere else and then that somewhere else could get new keys from the automated CA, right? Or am I missing something? Maybe we do want some second layer process for extra security, but couldn’t you solve this by sticking extra certificates in the record? This can also replicate the corporate solution: “trust an extra MyCorp certificate as well as the actual one.”
- So… now that I know the details of DNSSEC beyond “signed encrypted DNS,” I’m not a fan. What stops us from phasing in (beyond, elas, politics and network effects) Actually Good Signed Encrypted DNS? Specifically;
- For backwards compat, can we separate the “signed” and “encrypted” part where you stick a signature on the record which signs the hash of the record minus the signature and any other signatures?
- And then, for encryption, can you just do DNS-over-TLS or something like that? Totally separate from the signing bit; the goal of signing is that you can trust the data, the goal of encryption is that people can’t see what you’re looking up.
- And let’s say we had this nice two-layer protocol. Why do we need proofs of nonexistence? Why not just treat it like https and, when adoption is good enough, default deny insecure HTTP? Seems like another mistake in DNSSEC to have this feature at all…
For a glimpse into the idealized mental model of an idealized future I’m grasping at here:
- An abstract identity is (waves at philosophy)
- A concrete identity is a set of cryptographic keys
- Two problems:
- How do I find out the identity associated with a name. Related subproblem: key rotation and friends.
- How do I reach the given identity, over a given network; here the Internet
- You can have other naming systems. For example, a social network with E2E chat can be viewed as mapping names to identities-as-keys
- You can have other comms channels: see here again the chat, which is at a higher level than the Internet. But also SMS, Tailscale, idk… also, see different kinds of DNS records so one naming system (DNS) can handle multiple reachability kinds!
So, in short, if anyone bothered to read this brain dump, I believe a form for trustable (if you trust the source but not the nodes) hierarchical name => identity systems is a Good Idea so far, and I guess DNSSEC is not the way… but it still seems like CA is also not the way and it should really be part of DNS.
How misguided am I?
/rant
Until web browsers started to believe that no, that was too much of a convenience, so now long expiration certs became rejected. What's the proposed solution from the "industry"? to run a whole automation pipeline just to update a file in each example folder every few months? bonkers. These should be static examples, no reason to having to update those any earlier than every few years, at most.
> Up to 50 certificates can be issued per registered domain (or IPv4 address, or IPv6 /64 range) every 7 days. This is a global limit, and all new order requests, regardless of which account submits them, count towards this limit.
This is hard to deal with when you have a large number of subdomains and you'd rather (as per their recommendations) not issue SAN certificates with multiple subdomains on them.
We are working on further improvements to our rate limits, including adding more automation to how we adjust them. We're not ready to announce that yet.
We wanted to get this post out as soon as we'd decided on a timeline so everyone's on the same page here.
- Have one node with its own ACME account. It controls key generation and certificate renewal, and then the new key+cert is copied to all nodes that need it. Some people don't like this solution because it means you're copying private keys around your infrastructure.
- Have one node with its own ACME account. The other nodes generate their own TLS keys, then provide a CSR to the central node and ask it to do the ACME renewal flow on their behalf. This means you're never copying keys around, but it means that central node needs to (essentially) be an ACME server of its own, which is a more complex process to run.
- Have one ACME account, but copy its account key to every node. Have each node be in charge of its own renewal, all using that shared account. This again requires copying private keys around (though this time its the ACME key and not the TLS key).
- Give every node its own ACME account, and have each node be in charge of its own renewal.
The last solution is arguably the easiest. None of the nodes have to care about any of the others. However, it might run into rate limits; for example, LE limits the number of new account registrations per IPv6 range, so if you spin up a bunch of nodes all at once, some of them might fail to register their new accounts. And if your organization is large enough, it might run into some of LE's other rate limits, like the raw certificates-per-domain limit. Any of the above solutions would run into that rate limit at the same time, but rate limit overrides are most easily granted on a per-account basis, so having all the nodes share one account is useful in that regard.
Another factor in the decision-making process is what challenge you're using. If you're using a DNS-based challenge, then any of these solutions work equally well (though you may prefer to use one of the centralized solutions so that your DNS API keys don't have to live on every individual node). If you're using an HTTP-based challenge, you might be required to use a centralized solution, if you can't control which of your frontends receives the HTTP request for the challenge token.
Anyway, all of that is a long-winded way to say "there's no particularly wrong or right answer". What you're doing right now makes sense for your scale, IMO.
> What you're doing right now makes sense for your scale, IMO
Absolutely. I use DNS validation, and I’m fine running it manually every quarter, but I’m sure I’ll be quite annoyed to have to do it every month.
[1] https://letsencrypt.org/docs/rate-limits/#new-certificates-p...
One of the biggest issues is handling renewals at scale, and I hate it. Another increasingly frusturation is challenges via DNS are not quick.
This is a living organism with moving parts and a time limit - you update nginx with a change that breaks .well-known by accident, or upgrade to a new version of Ubuntu and suddenly some dependency isn't loading correctly, or that UUID generator you depended on to generate the name for the challenge doesn't get loaded, or certbot becomes obsolete because of some API change and you can't upgrade to the latest because the OS is older and you installed it from the package manager.
You eventually see it in your exception monitoring or when an ssl monitor detects the cert is about to expire. Then you have to drop that other urgent thing you needed to get done, come in and debug it, fix it, and re-issue all the certs at the rate limit allowed. That's assuming you have that monitoring - most sites probably don't.
If you detect that issue with 1/3 of the cert left, you will now have 15 days to figure that out instead of 30. If you can't finish it in time, or you don't learn about it in time, the site(s) hard fail on every web browser that visits and you've effectively got a full site outage until you repair it.
So you discover it's because of certbot not working with a new API change, and you can't upgrade with the package manager. Now you need to figure out how to compile it from source, but it doesn't like the python that is currently installed and now you need to install that from source, but that version of python breaks your python web app so you have to figure out how to migrate your app to that version of python before you can do that, and the programmer that can do that is on a week long whitewater rafting trip in Idaho.
Aside from all that, what happens if a hacker manages to wreck the let's encrypt infra so badly they need 2 weeks to get it back online? The internet archive was offline for weeks after a ddos attack. The cloudflare outage took one site of mine down for less than 10 minutes, it's not hard to imagine a much worse outage for the web here.
Won't that introduce new security problems? Seems like a step back.
I predict that normal people will began to get comfortable with ignoring SSL errors, even more than they already are. Perhaps we will see the proliferation of https-stripping proxies too.
Is there any good argument for short lifetimes? The only argument I know of is that short lifetimes are supposedly better in case the key gets compromised, but I disagree. If the key can be compromised once it can be compromised again when it renews; the underlying cause of compromise doesn’t go away. NIST stopped recommending forced password rotation for this reason, it’s pseudosecurity.
45 days ? So long ? Who needs so long living certificates ? A couple of miliseconds shall be enough. /s
The goal is to move to short lived certs to make the fragile system of revocation lists and public certificate logs unnecessary.
I dont follow. Why? Why not an hour? A ssl failure is a very effective way to shut down a site.
"you should verify that your automation is compatible with certificates that have shorter validity periods.
To ensure your ACME client renews on time, we recommend using ACME Renewal Information (ARI). ARI is a feature we’ve introduced to help clients know when they need to renew their certificates. Consult your ACME client’s documentation on how to enable ARI, as it differs from client to client. If you are a client developer, check out this integration guide."
Oh that sounds wonderful. So every small site that took the LE bait needs expensive help to stay online.
Do they track and publish the sites they take down?
To your actual content, unless you did something weird and special snowflake like, everything will just keep working with this.
>So every small site that took the LE bait needs expensive help to stay online.
It's all automated. They don't need help to stay online.
I agree with the terminology "bait", because the defaults advocated by letsencrypt are horrible. Look at this guide [0].
They strongly push you towards the HTTP-01 challenge which is the one that requires the most amount of infrastructure (http webserver + certbot) and is the hardest to setup. The best challenge type in that list is TLS-ALPN-01 which they dissuade you from! "This challenge is not suitable for most people."
And yet when you look at the ACME Client for JVM frameworks like Micronaut [1], the default is TLS and its the simplest to set up (no DNS access or external webserver). Crazy.
[0] https://letsencrypt.org/docs/challenge-types/
[1] https://micronaut-projects.github.io/micronaut-acme/5.5.0/gu...
You’re completely misinterpreting the linked document. See what it says at the start:
> Most of the time, this validation is handled automatically by your ACME client, but if you need to make some more complex configuration decisions, it’s useful to know more about them. If you’re unsure, go with your client’s defaults or with HTTP-01.
This is absolutely the correct advice. For Micronaut, this will guide you to using TLS-ALPN-01, which is better than HTTP-01 if the software supports it. But for a user who doesn’t know what’s what, HTTP-01 is both the easiest and the most reliable, because, as they say, “It works with off-the-shelf web servers.” Typical web servers which don’t know about ACME themselves can be told “serve the contents of such-and-such a directory at /.well-known/acme-challenge/” which is enough to facilitate HTTP-01 through another client; but they don’t give you the TLS handshake control required to facilitate TLS-ALPN-01.