Fortunately one of our engineers figured out we could get our demo rigs working by setting the clock back a few days. This could have been a huge disaster for our company if we hadn't found that workaround though. Pretty annoyed with Oculus about this
I say this not to either criticize you or excuse the mistake by Oculus (they really needed to countersign their cert with a timestamp server), but to educate. These are non-obvious issues to people that don't follow the VR sector.
Monitors work without low-level drivers because their maturity (and lack of innovation) allows the hard stuff to be embedded in the operating system. VR is not at that state; it is emergent, and the capability stacks require additional integration into the OS. Vendors frequently add unique features, and will continue to do so for some time, making standardization difficult.
Even at its simplest level, a VR headset with 6 degrees of freedom is two monitors that must remain in absolute synchronization while also returning positional information to the CPU. This alone is enough to go beyond "standard monitor driver" functionality.
But there's much more. Here is a paste of a comment I made elsewhere:
Oculus (and Steam, via SteamVR) engineers a plethora of low-level code to reduce latency and add features. It's not just a monitor, but a whole set of SDKs, APIs, devices, and drivers.
For the Rift, the hand controllers are wireless input devices refreshing at 1,000Hz; the sensors (to know where you are in the room) are USB devices with tight 60 fps synchronization to LEDs on the headset; there is a custom audio stack with spacialized audio and ambisonic sound; video needs specialized warping to correct lens distortion, interpolate frames, and maintain a 90 fps image, etc.
Not to mention, the system creates a virtual monitor so you can see your 2D desktop while in VR. You can reach out and "grab" a window, rip it from the desktop and position it in your VR world. Pin it, and when you enter a game that window remains and is fully interactive with the Touch controllers emulating a mouse. Maybe you want to play Spotify in the background of a table tennis game, or be able to check a Discord screen while working through a puzzler, or watch YouTube videos while flying your ship in Elite:Dangerous. One guy set up a video feed from his baby monitor so he could watch his kid napping while in VR. This is obviously not a standard feature of the Windows compositor.
All this needs to work across AMD and Nvidia, in Unity, Unreal, or any custom game engine. It's not off-the-shelf driver stuff.
Not to mention, the premise that monitors don't have drivers is also mistaken. They may not be necessary, but they are available[1]. And, the decision to sign kernel drivers is not a poor choice by Oculus, but a mandate from Microsoft for Windows 10 build 1607 and above.[2] A cert is, indeed, necessary to function.
Hope that was informative.
[1] http://www.aocmonitorap.com/my/download_driver.php [2] "Starting with new installations of Windows 10, version 1607, Windows will not load any new kernel mode drivers which are not signed by the Dev Portal." - https://docs.microsoft.com/en-us/windows-hardware/drivers/in...
I would like to see more companies write a Thank You letter from the CEO, signed by his managers. Something that he could use during his performance evaluations at the company, or attach to his resume for any other jobs.
It's hard to get concrete evidence like that, which shows your value to the company. It would great to have documentation that could never be forgotten.
It seems ridiculous in this modern age, but there are a huge number of people who will never bother to look into their problems on their own before asking someone else. Then this other person does a simple Google search and becomes the hero expert.
This all too often results in further dependence, with no real reward for the guy who took this basic step except more requests in the future. If this one guy can get a day off in this instance, it'll be a victory for every person who has ever said "Oh, if you google that, you'll see one of the first results with instructions to do x, y, z." to a time-draining coworker.
I even seem to recall one that when I set the clock back much more than 30 days gave me as many more days beyond the 30 as as much as I had set it back.
Then there were a couple pieces of software that would detect such trickery and which would punish you by taking away the time you had left also if you set back the date before the 30 days were up.
Anyway, with this in mind, the first thing I thought when I read the headlines was, “I wonder if one can get around this by setting the clock back”, and I doubt I was alone in that, so to say that it “probably wasn’t his idea”... I dunno man.
How low has the SW development bar gone, if "it's okay" now means "at least it's not directly killing people"?
Something like this probably will happen with computer assisted surgery or medical procedures, or an aircraft in flight. Just a matter of time.
By the way, do you have any links to your surgical training startup? I'm doing some research into VR/AR for surgical telementoring and training and would be interested in seeing how it's in use.
We've had a fair amount of press coverage in specialist press recently so a search for Osso VR should turn up some recent articles too.
Wow. That guy sounds interesting...
It doesn't help that vendors are generally nervous about liability in medical equipment (this fear is often unfounded, but persistent). As a result, vendors of commercial and industrial equipment generally don't want to engage medical device OEMs with engineering and customization support. If there had been that sort of support in this case, Oculus might have made a custom build without the cert check, just as a de-risking measure.
This vendor reluctance is especially present at the FDA Class III (high risk device) level - most vendors outright prohibit use of their devices in these products. It's an open secret that this still happens anyway in a wink-wink nudge-nudge fashion, just without vendor support - which is arguably worse, but it keeps the lawyers happy.
The real MVP of this story. Sometimes a dirty hack is good enough.
This problem is deeper than forgetting to update it. It should never have caused a failure in the first place. Just the fact that the device apparently can't function at all without the internet is a problem too.
On the other hand, maybe this is really a lazy feature. It's probably a good idea for the system to disallow both incoming and outgoing network traffic to any program written in a non-memory-safe language that hasn't been signed in the past couple of years. The lazy version of this feature is just not to run any program not signed in the past couple of years.
Edit: Requiring a timestamped signature on the signature also makes it pretty easy to add auditing functionality to the timestamp server whereby the publisher can detect unauthorized signatures due to their private key being leaked/stolen by criminals or governments. If the timestamp server's logs show a signature by your key that you don't recognize, then something has gone wrong. On the attacker's side, they need to either steal the timestamp server's private key or publish their malicious signatures for scrutiny.
Isn't Oculus owned by Facebook? Of course it has internet-based mandatory data collection, er uh excuse me, license something something.
Except TLS certs. That's the whole point of those certs having an expiration date in the first place.
That’s exactly how they are supposed to work. In the public sector we rely heavily on certificates for inter sector communication for instance, if certificates kept working despite being invalid it would put security at risk.
You’re supposed to build your software with an enterprise certificate store in mind though, meaning you can auto renew and distribute certicates when needed.
I really don’t see the point of adding a certificate to your television though, even if it is a tv that you wear on your head.
What's weird is sometimes they still link or require these docs to be downloaded and completed.
A bit worse, the constant password change requirements - thankfully the password helpdesks in public sector are so used to doing password resets that you can usually get a reset very easily just by giving a username if you are directly accessing a system (ie, internally so have access to helpdesk). The passwords can be crazy long though with 90 days expiration (senior folks write them down or give them to assistants). Some actually expire even without a login (they email you and say unless you login and change it will expire).
Some might call this a feature.
https://msdn.microsoft.com/en-us/library/windows/desktop/bb9...
By default, this is currently enabled, as noted here: https://stackoverflow.com/questions/11712322/error-the-times...
You have to go out of your way when signing an app on macOS to disable timestamping in current OS versions.
I bet they use certificate pinning.
Process A launches process B and checks against a pinned certificate. This is even more secure than just using the windows code signing stuff.
Problem is, when their cert expired, they were supposed to renew the same cert. Instead, somebody got a new one and signed the build of process B.
The device automatically downloads process B, but then the certificate pin check fails when it tries to launch it.
All the security guides that tell you to do certificate pinning need flashing neon signs explaining this problem. You can't pin certs if you intend to ever change certs.
A lot of applications and environments seem to be built with the assumption that they can add arbitrary complexity to their interface, since they're only going to be used by "experts" who can be expected to know everything of relevance and work through a thick documentation to understand the system. In truth, the "experts" who use your programs are going to also be using a dozen other applications, each with their own piles of documentation (or equal amounts of lack-of-documentation,) and have little brain-space left for the intricacies of your framework. So, they're going to use your system while knowing the minimum possible amount about it; if that system contains traps that cause problems for this kind of user, that's bad design.
I think we're waaaay past the "wondering" part.
When I read the comment I was immediately flabbergasted: no, someone else fucked up. It's not my fault someone wrote software that sets up undocumented traps for me to fall into. Or provided three ways to do something and two of them are not recommended OOTB. Or is primarily documented by third parties.
Whether you are writing SQL or graphics code you are constantly told "just express what you want to express directly, and the system is smart enough to do things as efficiently as possible".
But that might not be very efficient at all. The people who write "the system" have to write software that does specific things in specific situations and there will be endless cases which cannot be dealt with efficiently. And the more the interface hides the implementation, the less likely it is that those cases will be obvious.
My pet theory is that because we typically understand our needs before we understand the code paths required to fulfill them, our V1 APIs are usually a declarative “this is what should be” interface. Then we spend days or months making it happen and by the time we understand the required code paths, we’ve totally baked our expectations about the “make it so” API into our architecture.
Getting to a good, explicit, imperative API requires a whole nother step, and often a major refactor. You have to step back and ask “what is really happening now and can I conduct it more directly?”
... but by that time your code works and it hardly seems worth the effort.
But it is worth the effort. The declarative API will just get uglier as you add to it, and provide no real guidance about where future additions should go. Just throw another key in the config. Add another conditional. An imperative one will constrain your future choices about how additions can work, and therefore helps you clarify your domain model as you add to it.
That's just an excuse to stop mediocre engineers who would fiddle endlessly with pointless micro optimizations.
You very much need to understand the performance of your code if the product has performance requirements that fail unless you do.
I presume most coding is just crudding small strings from UI to database, where performance issues don't kick in, and hence it would be wastefull to care about them.
I've watched production builds to crawl to a halt and become sluggish as molasses because someone in the dev team was indoctrinated in this creed. (Many reasons, including a cartresian explosion in complexity due to some innocuous LINQ calls).
Who is saying that? I've worked in video games for 18 years and never heard that. In fact, anyone who said that would get puzzled and suspicious looks. We generally use a very C-like variant of C++.
and efficiency of computation is the least of my worries. i would like things to just work.
The problem lays in the IP. It's considered to be a vital asset so that when a company goes belly up it will survive kept years or decades in a safe by law firms in the hope someone will buy it, or just to make profits through litigation against infringers. Unfortunately this has a deleterious effect on products derived from that IP, the people who bought them and the people living where the unusable products will be trashed.
Just recently picked up a Rift. I love the hardware and their exclusives are top notch, but this confirms my suspicions that their backend is super goofy.
They sell Rifts at Best Buy and want to pretend that it's a consumer-ready product, but here's why I am recommending people stay away for now:
- Non-existant repair or service out of Warranty.
- Basic things in the platform like changing your name or photo don't exist.
- Lots of non-response over other basic features requested by the community.
- Questionable future investment in the platform or hardware. It sounds like they are moving their efforts towards "lighter" experiences.
In short, it feels like being a legacy customer for a new product.
Would anyone with a straight face say the same about Nintendo's exclusives?
I don't see anyone else in the market funding great things like Medium, Quil, Lone Echo, Robo Recall. Feels it's just cutting off your nose to spite your face to complain about this.
An Oculus is not a standalone product; it is a peripheral that relies on an open PC platform for its processing. If software the Oculus uses can run on any PC and the Oculus does not have a unique hardware capacity to operate the software, then the exclusivity is an arbitrary constraint.
Oculus wants all the perks of being its own platform without the responsibility or technical merit.
Oculus is producing these at a loss given the current size of the market hoping it will pay out in the long term by growing a healthy ecosystem...it’s the only way
This also suggests that if the decide they stop supporting it, eventually the software will stop working due to these certificate errors for which will then there be no fix.
Still need the sensors, but those aren't that large.
https://www.theverge.com/2017/10/12/16463844/oculus-santa-cr...
It's not bricked, bricking means it's as useful as a brick as in the actual firmware is corrupted beyond repair.
But meanwhile my 10 year old nephew is having life shaping experiences with the Rift I set up for him.
I've been surprised how many investors / technologists view VR as a niche market for enterprise / porn / limited gaming appeal.
There are other education experiences like BBC Home which is one of my favourites. Another, Mission:ISS allows you to explore the ISS and control the Canadarm to dock a module. Highly recommend them.
https://www.oculus.com/experiences/rift/1246744618768922/
https://www.oculus.com/experiences/rift/1178419975552187/
edit: I almost forgot, Go For Launch:Mercury is also very worth it. You can choose manual mode which puts you in charge of setting the launch procedures. The graphics aren't as good as I'd like, but the experience is good.
https://www.oculus.com/experiences/rift/752373261529153/
---
As a platform I really think it's capable of more than it's really doing right now.
I wish there was a way to open it right up for the more adventurous instead of relying on sometimes-working-sometimes-not tools like VorpX.
- Owned by Facebook, who will use it to gather as much data about you as they can.
That.
My Rift is scratched since first use (I swear it came scratched) and I've never been able to have them acknowledge that the device can be scratched into a blurry mess. "Oh, that's only regular use wear and tear" ...
Lens cannot be replaced or repaired (and they could during rift dk1 era).
Can you explain what this means in practice? I'm not super familiar with Rift but I'm curious.
https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...
read the command for signing their code, and signed their code as instructed.
Today, the certificate they signed a driver with expired, and because the signature wasn't timestamped it means Windows can't know if the driver was signed with the certificate after it expired, so the signature is now treated as expired as well, so Windows doesn't trust the driver.
Why wasn't it timestamped? Probably because instructions like the link above treat that as a separate subject to signing your code, and when you sign your code it looks and works like it's fully correctly signed.
or, as wtallis puts it (https://news.ycombinator.com/item?id=16542204), someone left a foot-gun lying around that didn't have much value except to cause incidents like this.
...and if your own company makes Windows apps, go check they are timestamped ;)
With a timestamp, as long as the signing date was within the signing cert's validity period, the signed driver continues to be trusted beyond the signing certificate expiration.
I politely ask authors or publishers to release non-DRM ebooks when I can. Apart from best-sellers (or time sensitive books), I believe most books are stuck in the long tail of obscurity and releasing them as non-DRM won't affect their sales.
This also means I don't visit pirate websites that claim to have ebooks. It takes two to tango. :-)
That's not the future. That's the present! Which is even more worrying.
Generally this attitude doesn't backfire, because individual users loosing access to their data, their accounts or their software can be simply dismissed. But in this case it happened to everyone at once, so it's suddenly a big deal.
If one doesn't, well, she isn't much of a "security expert" after all, is she? Firewalling off TCP port 80/443 at your perimeter firewalls isn't a very good solution if you're an e-commerce company selling your product on your web site -- and the "security experts" know this.
Because Mark Zuckerberg said so, that’s why.
More relevant by the day: http://locusmag.com/2018/03/cory-doctorow-lets-get-better-at...
* data is not tampered
* signing certificate was time valid at signing time: signing time is within signing cert's validity
* Neither certificate was revoked before signature generation
* both, signing and timestamp certificates chain up to trusted root CAs (regardless of their time validity, just must be in trust store).
That doesn't make sense to me: If the certificates are compromised, an attacker could backdate the timestamp to whatever he wanted and sign anything.
What's the thinking here?
Nate Mitchell of Oculus posted on Reddit saying "We're working on resolving this issue right now. We'll keep everyone posted on progress here." https://www.reddit.com/r/oculus/comments/82nuzi/cant_reach_o... . Top-level of that thread has a workaround involving setting the clock back or using a utility called RunAsDate to fake the clock for a single application.
https://docs.microsoft.com/en-us/windows-hardware/drivers/da...
EDIT: Looks like the standard TTL for these code signing certificates is none, 1, 2, or 3 years.
https://www.entrustdatacard.com/products/categories/digital-...
https://www.globalsign.com/en/code-signing-certificate/
https://www.instantssl.com/ssl-certificate-products/code-sig...
Fortunately it was a problem with the tests, rather than the code itself, but these things do happen.
At my old job, we had a bunch of tests fail when daylight saving ticked over. For some reason, some things were using local time, rather than UTC. We also had a test that would fail if the minute was the same as the hour.
Time is hard
We furry 'self-reproducing' (YMMV) mammals are simply not ready for all of this.
This was all years ago, so my recollection may be fuzzy, but I spent entirely too much time futzing with SIP traces and certs. Weird, weird things can result from time inconsistencies is my takeaway, however.
Presumably this is what Oculus thought and so how they got in this mess.
BTW, most commercial installer progtams will apply a valid timestamp if codesigning is enabled. So to save ~$1000 someone decided the tools built into Visual Studio were good enough. Anyone that ships commercial software that does more than a basic install into C:\program files will know to spend the money, it’s worth it.
> If your antivirus software restricts the file from opening, temporarily disable your AV and continue.
Good Patch Procedure, 2018.
Please, please don't say: "Our teams apologize for any inconvenience this may be causing you"
instead opt for "Our teams apologize for any inconvenience this caused you"
They soft bricked every single headset worldwide.
"Oopsie we may have caused you inconvenience"
vs
"We're sorry for the damage"
Get rid of the may entirely. I don't know what I was thinking.
Including the word 'may' shifts the sentence from apologising for causing actual inconvenience, to apologising for causing a minor risk of possible inconvenience, which is not what you are trying to convey after selling someone something for a lot of money and then remotely breaking it at no notice.
In many customers, it is likely to elicit a response something along the lines of:
"Any inconvenience this may be causing? I'll give them may be causing. The fucking thing won't boot. May be fucking causing. I wasn't using the damn thing as a doorstop."
EDIT: presumably you need your client apps/libraries in the field write back when they use a cert that is <X months away from expiry.
I use simple Nagios checks for keeping an eye on certificate expiration. It's simple to set up checks for new hosts/services and I have them set to trigger an e-mail alert 30 days before expiration (20 days for certificates from Let's Encrypt). It does the job; I have yet to wake up one day to an expired certificate.
Apparently, this ("send me an e-mail before my certificate expires") is also the sole reason some companies even exist (i.e., it is their only product/service). It amazes me that this is something folks will pay for.
Obviously it should be part of a handoff process when you leave, but companies aren't always good at smoothly handling transitions like this.
What happens if I leave, though? I don't know and, TBH, I won't really care; it won't be my problem at that point. How -- if -- things are handed off or transferred off to someone else when I leave will be up to my boss, I suppose.
(disclaimer: work for Azure, but not on Key Vault)
If it’s a much longer time scale, people start to forget that it’s even possible for stuff to expire.
If my fridge filter can display a little reminder light on a timer every few months, cryptography-dependent devices might need something similar. That way, your customers could know in advance and be asking you for an update.
So many comments agree that (a) security is hard, (b) countersigning with a timestamp server is easy to miss, (c) countersigning makes build processes difficult, and (d) they've done or seen similar things in other apps/companies.
This sounds like a classic UI/UX issue for developers around a literally mandated and mission-critical requirement of the OS.
At the least, MS should provide a validation tool to surface errors or risks before production. Better, signtool.exe should make omissions (like a timeserver) very difficult and make them an override, not a default. Best, they would do both.
I don't agree that the OS should reject non-timestamped signatures as faulty per se (and throw an error), as that puts the burden on the user to understand a developer's mistake. Sometimes running without a timestamp may be desirable - ultimately that's the dev's choice.
It should just be a choice made explicitly.
[0] https://msdn.microsoft.com/en-us/library/windows/desktop/bb9...
However, this affected only one single customer of ours and we had a fix within a couple of hours. -- I certainly learn from this mistake.
I thinks IT is used to managing HTTPS certificates, domain name auto-renewals but app level certs are more of a new thing.
Edit: I don't follow Windows, I'm really curious what the consequences for stuff like this can be generally.
I don't see credit on my Oculus account? Am I supposed to have received it already? Or is this maybe because I don't have payment method added to my account?
It looks like their auto-updater used the same cert though, so they can't distribute it as a normal update. They're probably figuring out the least sketchy/most automated way to distribute it right now.
When this is all said and done, there will be a handful of people who will never, ever forget to use the /t flag in signtool.
[1] http://www.brucebnews.com/2017/11/look-at-the-red-flowers-wi...
It's not just people shrugging it off, many are defending this as being a perfectly fine state of affairs.
https://en.wikipedia.org/wiki/Transport_Layer_Security#Digit...
This includes phones, cars, self-driving cars, watches, farm equipment, computing devices and anything marketed as an IoT appliance.
One glitch, as minor as an improper system time, and you’re dead in the water.
The big question is why on earth can drivers that have been verified and are already installed in your system can suddenly stop working? If this mechanism is intended to protect against malware disguised as drivers then it's already too late. The malware had several years to exploit your system.
Expiration after installation simply doesn't make sense for code signing. The signed executable won't change unlike a website. The driver is always going to have the same file hash, forever.
It makes absolutely no sense to the end user, acting as possessor or potentially a reseller of an object, since the very premise implies that an owner should not be provided total control over their device, that it's never really "theirs", and that a vendor should retain the capacity to take a "sold good" away from the owner, under the guise of expected behavior, built as designed, effectively converting a sale into a rental, in time, perhaps after statutes expire.
It's effectively a back door for manufacturers, so that they can count on well-made products not lasting forever, not in museums, not for resale, not for nostalgia.
