Have the ECU only do the engine thing. Have the AC control just do AC control. Decouple dependencies and make it as simple as possible. Old cars already do it. Blinker switch send signal directly to light controller, not to some central box deciding what it should do with it.
If something needs config in addition to control signals, have it keep it own config and only be updated from the "config manager" (inforatinment box). If infotainment box dies, everything else still works.
Cars already are basically "microservices on a message bus". Let's just use what works with that - minimal coupling and maximum independence of "services"
> Had I been in a less complex car, a local garage could most likely have fixed the problem and sent us on our way. The sophistication and gadgets in modern cars are great until something goes wrong then they fail hard. Small local garages that used to be a life saver are next to useless now as they don't have the tools and knowledge to fix a mobile data centre.
Out of curiosity, what was the issue ?
Sure, old carbie with distributor might've just ran with 3 cylinders , but that also might damage something.
Also auto makers don't really want to give user sensible error messages or even just metrics because without experience they might just misinterpret it as different problem.
For example if car have oil pressure gauge it is either nearly fake or heavily filtered. Oil pressure changes according to load but gauge going up and down might cause user to think something is wrong with car...
https://thenewstack.io/how-the-u-s-air-force-deployed-kubern...
If this setup can run my 95% uptime enterprise apps, I am sure as hell it can run my car too.
Fwiw - I am a kubernetes fan. Just not in cars.
A lot of software is created on powerful developer machines. But fill up a normal consumer machine with this software, and you start to notice that it maybe isn't so fine.
This is how things like Electron come to exist. I'm sure Electron works fine on developer machines, but once it trickles down to someone's cheap Celeron netbook, it runs worse than retro computers with 384KB of RAM.
Does it really have to be this way? Is more software "fine", if the same could be accomplished with much less code bloat?
P.S. As far as I've heard, one of the best ways for developers to combat this is to target your software for cheap netbooks proactively; test compiled artifacts there rather than on your powerful development machine. If you can make it fast in that situation, it'll be fast pretty much anywhere.
I once met someone who had optimized their DOOM clone using this method, and they claim to get millions of FPS on any vaguely modern machine, just through optimizing it for cheap netbooks.
Based on the story in the parent, it sounds like this was precisely a problem with software on the critical path, otherwise local mecs/breakdown service would have been able to fix it rather than give up.
I'd go further than cars and say, "in most things". Smart-anything, washing machines, printers, sewing machines, thermostats, appliances in general...
My mother in-law has two sewing machines. One of them is one of the first electronic sewing machines (from the 70s) and one is much older. Guess which one still works like a charm?
I'm not arguing against electronics, here-- many of these things are no doubt improved by electronics to such a degree that the tradeoff is worth it, but it's good to at least acknowledge that there is a tradeoff. It's also good to try to minimize the impact of electronic failure. Smart things would ideally just revert back to being functional dumb things (rather than bricks) if their electronics fail.
But hey, feeding everything from single microcontroller is $2 cheaper...
Anyway. A big part of that class was learning to clean, repair, and manage printers. Again, it was the 90's, and we were high school kids. We came out quite capable with many computer skills but the printer stuff really stuck with me. I've done technical support throughout the years and have setup hundreds of printers.
The printers of today are awful landfill fodder compared to the Okidata's of the 90's. Pure simplicity and speed vs FULL COMPUTERS, with scanning, faxing, and every other imaginable feature crammed in with zero hope of doing anything other than replacing the toner.
Never serviced, always cooked perfectly. She had to get rid of it around 2006 or so, when she got a pacemaker.. They didn't shield them as well back then...
There's a bit of survivorship bias and N=1 here.
I'm old enough to remember machines full of relays and discrete components that failed pretty often and required a lot of troubleshooting with schematics on hand. Modern appliances – if built out of decent components – have a much better shot at surviving long term. Less discrete components that can fail, more debugging capabilities, logic that's implemented in a rock solid processor rather than an unreliable mess of digital gates (or worse, analog logic).
There's obviously a point where there are diminishing returns, and probably another one where more complexity actually decreases reliability.
> Smart things would ideally just revert back to being functional dumb things (rather than bricks) if their electronics fail.
If possible, yes. That's only really an option for simple devices.
Ford apparently ended the core charge program for lighting in 2020, but other manufactures continue, and that is just one thing that was common for users to service themselves in the past. It's not going to get better.
On the other hand right to repair is very important. Walled gardens suck. Still hundreds of millions of people live in rural areas in the so called developed world, etc. And I don't want to subsidize the industry, I'm willing to pay more up-front, if it means I can just to replace the fucking light bulb.
Trust me, it is a difference you can feel.
If you’re a kid and you also have IT skills you’re going to be interested in IT unless there are extenuating circumstances, like wanting to stay rural, or a family business, or friends and family with union influence. Easier on your body and pays at least as well. So a car mechanic with heavy IT or electrical skills is going to be in short supply. Which is a problem when all cars are electrified.
At least from my experience.
It's supposed to have a backup but, like most backups, I hadn't tested that it works and for some reason the RFID reader part wouldn't connect with the car.
Hoping to get back to it once this version of podman is released. Thank you RedHat team; we'll send you one of our juicers as a thank you gift.
If we have a right to repair here, we also need to see how to handle liability here. If you flash your own software on the motor controller and subsequently mow through a group of people because you forgot to do a plausibility check on the accelerator pedal value who takes responsibility then?
Even if you just get the original software, how do we ensure you flash it correctly?
If you get the schematics, how do we know used the right parts that are rated for 125°C temperatures.
Who is liable if you tweak the software on your 2023 Mercedes? The same person who is liable if you tweak the hardware on your 1987 Chevy. There's plenty of precedent on how to deal with this.
I would, obviously, for making the unsafe modification. In what multiverse would the manufacturer be liable for something entirely outside the manufacturer's control?
What can a mechanic not do to a modern car with the standard scan tools and training? Most old school mechanics still lack the training to work on computers, but those that have that training have no problem fixing cars.
Well first of all that happens even under the current draconian anti repair setups already and secondly thats a felony. Just because you can do sth illegal doesn't mean we should child proof our whole society so you can't do anything anymore just because someone MAY do something illegal.
We still allow you to buy knifes, in some places even guns.
Nah it would figure out what is the best time and place to break, order you an uber, and Uber would psy you manufacturer for the order flow.
It would also show you ads while you wait
'cos the car has blown a gasket
in the middle o' a place called Rannoch Moor
So, what was the problem?
I think I see the problem already. Why does anyone think its a good idea to put everything in an embedded system into a container? Particularly as everything comes from a single vendor and so the usual argument about "but libraries are too hard!" doesn't apply.
Imagine calling up breakdown assistance because your car won't start, mechanic comes out, cracks the hood and is like "ah there's your problem right there, ignition service has only 1/2 pods healthy because the node went into NotReady due to DiskPressure. I can clear up some log files so it goes underneath 80% disk usage again but sucks teeth it's gonna cost ya. I'd recommend throwing the whole car out and getting a new one. You shouldn't get an attachment to these things, they're cattle not pets."
Truly breathtaking.
And what happens now is, that you need a huge effort to integrate all those components. End to end system tests get much more important, but are still harder to do than simple unit/integration tests. traditional testing strategies start to get pointless, because most bugs now only appear when combining services in a production setup.
Yes, development gets easier, because every team can just develop, without aligning too much with other teams. But the deployment/ops/acceptance step often gets impossibly complex.
Obviously going too big here is problematic as it can slow it down when tens or hundreds of people are involved in every update, but going too small have similar problems, on top of generally more smaller services eating more resources. We don't need "front reflector LED setting app" being called from "lighting setting app" called from "car setting app", it can probably just be one service.
Smaller services also mean more services to update if some commonly used lib gets a security bug. Updating SSL lib in big monolith is just update, run tests, but in microservices that's multiplied by amount of teams and services.
> In general if your teams are not aligned, it doesn't really matter if they are developing microservices, libraries, or one spaghetti ball mess, its going to be problem anyways
Moot point. We pick the tools to make the job easier. Good team with bad tools will still be slower and less efficient than good team with good tools.
Because technology almost always carries cultural values with it.
What people should understand is that container in Linux is just a separate process running in powerful chroot (which isolates not just file tree, but also process tree and other things).
So the same reasoning which applies to running some code in a separate process also applies to running some code in a separate container.
I'd even argue that in an ideal world, almost every process should run in a separate container. The tooling is not here, but concept is just fine.
I happen to know a small amount of the Tesla internals and they are using cgroups, namespaces, app armour and ebpf based syscall filtering to secure various processes on the car.
You almost certainly should not use docker or podman to manage processes on a car but that doesn't mean you shouldn't embrace the subsystems they are built on in order to increase security resilience and defense in depth.
I'm not saying we should be running k8 on an 8051 or even a cortex m33, but on an arm7? Maybe.
Cult of Ferris time, static linking in rust means your binary is your container, particularly if you statically link in musl.
If they have to work together the communication protocol is clearly defined well in advanced and limited to exactly what they need to say. Thus we are reasonably sure if any one combination works all possible combinations will work. Even then there is typically higher level control to only release combinations that are tested to work together.
Comparable would be to having an inode de-duplicating file systems, and deterministic binary generation. But it's hard to prove "the correctness" of inode de-duplicating file systems in extreme environments like auto is required to, and deterministic binary generation is hard to control 3ven if it is possible with the specific build tools (it usually isn't).
I think people underestimate the amount of software that is ALREADY running in their cars/airplanes/helicopters and even elevators.
This forced application of new technologies into every possible domain can have real security and reliability consequences - and all because some VP somewhere decided they needed to use the latest shiny thing in cars or fridges or ACs or whatever.
In my experience updates to smaller services are often trivial, or for updates that are actually impactful it would be way easier to coordinate in a monolithic codebase.
trivial updates to individual services could be iterated extremely quickly
systematic changes to behavior across services were so hard they became incredibly uncommon
There is also security, I don't care if someone hacks my radio system nearly as much if someone hacks the brake system. Containers is one part of the total package to isolate parts so if there is a hack the whole system isn't taken out.
Computers are a large system. Someone in "the other group" making a mistake can bring your part of the system down. Much of the code is written in C or C++, and so one "old school" programmer can make a mistake and write past their memory into a data structure you are using, and the bug report goes to you not them.
If you have the above system, when splitting the monolith apart you will discover libfoo.so that both depend on, and the two groups will want to upgrade separately: containers allow this to happen, without modifying your build system to give them different versions or otherwise allow two different libraries with the same name to sit on your system.
The above is what is obvious enough that I can talk about it. (I work on an embedded system at John Deere so I cannot comment more than the above even if you ask.)
I've done this analysis for lots of software before, Windows has a really nice tool called Process Monitor that I've used to find huge slow downs before. Point it at a process and it'll tell you the every bit of IO that the application does, and at that point you can just start digging and opening bugs.
IMHO almost every piece of software of any significant size horribly misbehaves. Opening the same file again and again, writing out lots of tiny temp files, reading the same key from the registry (or some config file) over and over again , and worst of all, using fixed timeouts waiting for a task to complete instead of doing the work to make things properly event based.
On that last note, timeouts instead of event callbacks (or some sort of signaling system) is the cause of so much slowness across our entire industry. If something you are doing on a computer takes a human noticable amount of time, and you aren't throwing around tens of GBs of data or waiting on a slow network, then you are flat out doing something wrong.
* it makes it easy to separately update different apps that are shown on the same screen
* it unifies the update process
* you can download the update in background while the app is running, easily roll back to an old version
* it's familiar technology to many engineers (you could call that lazy, but it also reduces risk)
* it's easier to use an existing networking implementation than having separate chips for each task, and then having to connect them through busses
* allows for pretty good resource sharing (RAM/CPU)
* pretty good isolation out of the box
I wouldn't want that for my engine controls, but navigation/radio/camera/climate control, why not?
I wouldn't consider this a night & day change, but an incremental one. But a good incremental one overall; I wouldn't drop everything to implement this but I'd definitely see it as a good thing even in the absence of functionality improvements. There's other benefits too like being able to update just one container in case of some problem, and having the blast radius more thoroughly contained than it would be with everything installed into one big base system.
This bring costs and supply chain issues, and we had plenty of supply chain issues earlier this year.
When you ask, why containers?, reask, why sandboxes? if security is important at all, then you have your answer.
Containers are so convenient people forget we used to use chroot jails.
podman works fine until it doesn't. My hypothesis is that it has some fundamental design philosophy that makes it brittle. Properly cleaning up doesn't exist in their vocabulary.
For example, a cancelled download or image extraction can bring the whole thing down at the worst time, you have to hunt down the corrupted folder and remove so that anything works again.
A failed compose startup can leave the network in a undefined state hard to diagnose and impossible to recover without wiping some folders within /run/user and killing some rogue processes manually.
This is further cemented by the fact that a lot of minor issues are answered with: podman system reset, which reeks of rm -fr node_modules.
docker was always a pleasure to work with, I still don't understand why I suffered with podman so long.
Docker isn't perfect. I wish they would put more development into rootless mode. But it has never given me the kind of issues podman has. It just does what I ask it to and gets out of the way.
In 2018 I opened a github issue around container startup time[0] with Docker. A couple of things have changed since that issue but generally speaking we are talking about ~5s (containers) vs 150ms (no containers) to start a realistic process that depends on networking and other things you'd expect in a typical web app.
Folding a piece of paper (just like binary numbering) 6 times will provide you with a stack of 64 sheets.
They did not have a performance increase of 64 times.
This is identical to the idea of stating "magnitude" as being the number of times based on 10.
How wrong am I?
If apps were like on Mac - self-contained directories that can be installed at any path - then Docker would probably be a footnote.
There is DT_RUNPATH probably since before I was born. The problem is it's not always utilised, distributions prefer to share libraries over isolating applications, and loading shared libraries isn't the only host-dependent thing done by application code.
Also you realise that docker provides more functionality than a tarball, right?
[0]: https://www.redhat.com/en/about/press-releases/red-hat-and-g...
Constraints (car env) drive creativity !
