https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
How dysfunctional does a company have to be to let this happen?
Stuff like this happens when upper management has zero clue about the business they are in. They believe they are in the business of selling certificates, while in reality they are in the business of selling trust. They treat things like the CA/B Forum and the various Root Programs as more like an optional networking event than the combination of judge, jury, and executioner that it actually is - with a completely predictable outcome.
Which means their business continuity planning was bullshit.
The good news is this caused companies in the healthcare space (at least provider, facility, and insurer sides) to start asking more pointed BCP questions to their SaaS vendors.
Surprised no one pointed to the nature of the business as a source of this behavior.
In a non-innovative, compliance-based industry, you make money by cutting costs.
This affects the entire business, as you find managers who are effective at cutting costs and architects/engineers who will work for lower salary.
Multiply that over enough years, and we know where it leads...
>> I see three possible outcomes:
>> 1. The root programs continue to be lenient with Entrust indefinitely. Nothing changes.
>> 2. The root programs continue to be lenient with Entrust for a while, but eventually the mistakes pile up enough that one of the root programs pushes for distrust.
>> 3. The root programs immediately stop being lenient with Entrust. Entrust is forced to make internal changes to remain a CA.
It raises an interesting point about what constitutes a historical pattern of behavior, sufficient for infering future deficiencies reliably enough to take present action.
Here, the motivation seemed to be that (a) enough history had accumulated to estimate Entrust's rate of process improvement & (b) that rate was deemed insufficient. Which seems a decent metric: if perfection is not presently achieved, then remediation progress needs to be seen.
Here's the email their CEO/President sent to everyone that uses them:
Google Chrome announced yesterday that specific public roots used to issue public certificates by Entrust will no longer be trusted by default after October 31, 2024. This decision comes as a disappointment to us as a long-term member of the CA/Browser Forum community.
To address your concerns, there have been no security implications to the events that led to this distrust event, and you can be assured that your certificates are secure. I also want to assure you that Entrust can and will be able to serve your digital certificate needs now and in the future. And, our ability to do this extends beyond the public roots covered in Google’s decision.
Additionally, there is no impact on our private certificate offerings – including our PKI, PKI as a Service, and managed PKI – nor our code signing, digital signing, and email (S/MIME and VMC) offerings.
While the announcement is disappointing, Entrust has been in the public and private digital certificate business for over 25 years and we continue to bring that expertise and capability to your use cases every day. It is our hope that you will allow us to continue to serve your needs and we stand ready to answer any questions you have regarding your ongoing needs.
Sincerely,
Todd Wilkinson President & CEO
---
My personal take: I don't see why any of their customers (such as ey.com) would want to split their CA needs across multiple suppliers.
I was wondering how Chrome was able to revoke a certificate based on time without trusting the CA to not back date certificates and it looks like this is due to being able to trust certificate transparency logs instead. This is where they get the signed certificate timestamps (SCT) from.
[1] https://groups.google.com/a/ccadb.org/g/public/c/wRs-zec8w7k...
I’m also an author on https://webpki.substack.com. I will be writing my thoughts on the distrust soon.
I can try to answer any questions folks may have. I can also help folks find ways they can also be involved!
Root programs can only do so much and need surveillance of the CAs from the community.
Regardless of how Entrust is operated, there appears to be significant complexity in CA program that the browsers operate. On the flip side, Let’s Encrypt is basically effortless for me to use, as an end user of an LE secured site and as a developer. Why misallocate all this toil on root CA compliance on the one hand, when LE could redirect that labor towards something valuable instead? What is so challenging about giving LE full leadership on this issue? Where does the proverbial political strength of Entrust and similar entities come from, in an ecosystem where there are functionally 5 cooperating, more or less transparent entities that decide the trust of certificates for 99% of end users? Why does anyone care about any of the CAs?
Let's Encrypt started their operations with _automated_ certificate issuance only. They also do not do OV/EV certificates that are much, much harder to automate without providing any real benefits.
So, LE's mission is to issue certificates under the rules set by CAs and Browsers. (Yes, CAs do participate in setting up rules for CAs.
> Where does the proverbial political strength of Entrust and similar entities come from
Generally supporting non-automated certificate issuance. Effectively, technical debt. A lot of older enterprises have done manual certificate issuance, and they don't feel the pressure/reason to switch.
I don't get entrust here. It's not like they weren't told what to do.
"treasury.gov" and "uspto.gov" stand out (to me), let alone the lengthy list of banks and other gov places around the world.
https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
The older vendors are even less able to be properly open with their customers (let alone the general public) than Google. At Apple it's probably a firing offence to even confirm obvious decisions - it seemingly took months to get Apple's chosen representative to confirm that Apple's new 398 day rule was an issuance requirement, rather than just something where Apple wouldn't trust longer lived certs in Safari.
Any root program will refer to it for context on issues.
And for a CA, credibility is everything
If you believe https://bimiradar.com/glob
it looks like Entrust is selling on the order of a few dozen certs a week to maybe upwards of 100-200.
EDIT: I've asked Google if Gmail will be discontinuing support for Entrusts VMC certificate (and thus BIMI logos), I would guess not since BIMI has some actual requirements, but assumptions are not the best way to make decisions about risk (like our BIMI logo not working later this fall).
They've pivoted to payment cards.
This continues to annoy me. Chrome (and other browsers) have detailed trust constraints, e.g. SCTNotAfter, in their own root stores. Why can’t administrators do the same thing?
Directly from Entrust: "Yes, there has been ongoing internal discussion and reflection on the issues found in this and other incidents, which has led to the action items described previously and ongoing changes, including the decision to revoke the certificates affected by this bug. Exceptional circumstances would need to be provided and justified by the Subscribers. Given the nature of the feedback we have received to date, we doubt that the community has any real interest in anything that Entrust could suggest, except to use against Entrust in a destructive, not constructive, way. We therefore would like more explicit and clear guidelines or a definition of “exceptional circumstances” to be adopted and applied equally to all CAs, perhaps through updates in the CA/B Forum requirements."
We’ve been endlessly talking about our repeated screw-ups, which led us to revoke the affected certificates. If subscribers want an exception, they need to come up with an extraordinary excuse. We don't care, so we demand clear and strict rules about what counts as “exceptional circumstances” that apply to all CAs, and these should be updated in the CA/B Forum requirements. We are big, who are you?
... it's not promising
it looks like Entrust is selling on the order of a few dozen certs a week to maybe upwards of 100-200.
EDIT: I've asked Google if Gmail will be discontinuing support for Entrusts VMC certificate (and thus BIMI logos), I would guess not since BIMI has some actual requirements, but assumptions are not the best way to make decisions about risk (like our BIMI logo not working later this fall).
There's now also the problem of competing with a free alternative that increasingly almost everyone knows about.
If you read through some of the incidents in bugzilla, you get the strong impression that Entrust’s market is specifically the people for which something like LetsEncrypt isn’t currently a viable alternative (or at least a difficult one).
In trying to justify not revoking misused certificates, one example they gave for a customer they were granting extended deadlines to was some organization that was contractually obligated to their customers to provide at least 90 days notice of any certificate updates.
While the deliverable is essentially the same, I don’t think Entrust and LetsEncrypt have really been in competition.
The problem in this case is that Entrust displayed a complete disinterest into actually solving the underlying issues. Doing an oopsie is one thing. Doing an oopsie, lying about it, refusing to take precautions, and failing to take measures to prevent a repeat despite promising to do so? Completely different story.
If they can't be trusted to respond properly to minor administrative issues, why should they be trusted to respond adequately during a real security incident?
Entrust missed/ignored the updates to how certificates were supposed to be formed and when caught declined to revoke the incorrectly issued ones (because it's probably a more or less manual process for many admins working in a pre-Let's Encrypt style of fashion) and they didn't want to inconvenience their customers and assumed that they themselves were the important party in the equation (CA's was that historically compared to site-admins).
The certificate industry has always been quite ad-hoc with CA's being entitled middlemen, we have Let's Encrypt and almost ubiquitous encryption now because browser makers and other internet actors saw security as more important than protecting the CA's business and now that LE is established Google,etc aren't the slightest interested in pampering CAs if they aren't interested in cleaning up the system.
The problem is that to be a CA in the root stores you agree to (and entrust voted in support of) a pile of rules, and entrust demonstrated a complete disinterest in complying with those rules.
The reason for removing trust is not the severity of the original errors, it's the the severity of errors in the response.
1. The BRs requires revocation of invalid certificates with 5 days unless there is an exceptional reason not to. Entrust did not.
2. In the event a CA discovers that they are mis-issuing certificates they are required to stop issuing until they have resolved the error. In this case not only did entrust not do this, but they explicitly stated - after the issues were raised, and they were already told they were failing to revoke certs in the required time frame - that they were intentionally continuing to mis-issue
3. They repeatedly made errors in the past, promised to correct them, and then kept making the same errors
4. They made claims they were trying to get customers to prepare for revocation in the require 5 day window, but then it turned out they were telling customers that they had 30+ days (which was only discovered when one of the relevant customers forwarded info to someone else)
5. When a CA discovers miss issuance they are required to file an incident report, and provide detailed information about how it occurred, why it was not caught, what remediating steps are being taken, and what mechanisms are being introduced to ensure a similar failure cannot occur in future. None of Entrust's responses came close to this, until the chrome root store rep came in to say "this is unacceptable", and even then their "improved" reports were incomplete and lacked sufficient detail.
6. Once they were finally doing the basic steps they were meant to have done the moment they learned of the miss-issuance they repeatedly failed to produce an accurate set of the impacted certificates (as in the provided a list and people outside of Entrust were able to immediately turn around and say "but these certs are also broken, why aren't you listing those details")
and so on and so forth.
Google's post to CCADB provides more details than the blog post: https://groups.google.com/a/ccadb.org/g/public/c/29CRLOPM6OM...
This reminds me about discussion about Russian Goverment's NUC Root CA (not trusted by default in Chrome/Firefox, Trusted by Yandex Browser only with some additional verifications to prevent abuse by goverment). Discussion was not about why this cert was necessary in first place, it was about it's creation violating Russian laws and procedures AND violate a lot of technical rules. A lot of people just said - this cert is necessary and it's clear who made it so why we should look to "minor details"? (Links - in Russian https://habr.com/ru/articles/666520/ / https://habr.com/ru/articles/708970/ )
Wonder how secure that is? That has real potential for extracting value.
nCipher make HSMs - a lot are used in banks to encrypt transactions on behalf of devices through APIs like PKCS11.
To answer your question "how secure that is?"; The answer is yes, secure.
chase.com aa.com
washingtonpost.com, cdc.gov, dell.com, jpl.nasa.gov, mastercard.com
This is gonna cause me some headaches, along with everyone else who processes payments through Cybersource, and possibly others :(
We're already working on it. Keep an eye for merchant notifications if you use certificate pinning.
Now, back to rotating certificates....
It doesn't say much.