Some time ago (around ~10 years) this guy (the presenter) was internet famous for being a Rubik cube speed solver and making tutorials and videos about that: https://www.youtube.com/watch?v=609nhVzg-5Q
What a blast from the past! I learned how to solve the Rubik's Cube blindfolded by watching him, back in the day. His tutorials are perfect and I've probably recommended his channel to ~50 people myself. Crazy he only has 36.6k subs.
Glad to see he's doing well.
It's as if everybody wants to be the one to exclusively own the tech. Imagine every car manufacturer having a completely different take on what a car should be like from a safety perspective. We have standards bodies for a reason and given the fact that there are plenty of lives at stake here maybe for once the monetary angle should get a back-seat (pun intended) to safety and a joint effort is called for. That would also stop people dying because operators of unsafe software are trying to make up for their late entry by 'moving fast and breaking things' where in this case the things are pedestrians, cyclists and other traffic participants who have no share in the monetary gain.
It would probably slow down. 9 women can't have a baby in 1 month. Besides that, the disagreements about approach, politics, or eventual competitive interests would probably bring things to a halt for a long time.
I don't think the solutions to this problem are resource-constrained. Many companies would happily find more resources in order to be first to market with this technology.
https://www.amazon.com/gp/product/1119482089/ref=ppx_yo_dt_b...
That would be a bad idea because like in evolutionary processes you need this diversity of ideas to locate better local optima even if it will take longer.
Of course, it's not a binary choice. Things like data should probably be pooled but the use of data in tech should compete.
So actually, I think it's one of these situations where having a lot of independent efforts might be worth it.
In the self-driving world, the duplication is necessary - different companies are taking different directions, and nobody really knows which will work out.
In the ML hardware world, the duplication is mostly unnecessary. People are developing their own inference hardware ASIC's because they're relatively simple (compared to designing a CPU from scratch, designing a TPU is pretty simple because there are so few operations, and no complex out-of-order execution), and you can't buy one off the shelf yet.
As soon as ML hardware becomes available to buy off the shelf without a massive price premium, everyone will switch to that.
So which one design you suggest we all use for all our ML needs?
What I've seen personally is what can be loosely termed "emergent consensus". Historical competitors (and often it gets whittled down to two giants, such as Boeing and Airbus) will work in secret on research. But after years of experimentation arrive at very similar outcomes. An optimal answer that could only be arrived at through constant trial and error, product evolution and iteration.
Regarding Karpathy's PyTorch presentation I don't thing anything that wasn't already public was revealed. The FSD board with custom NPUs is a Work of Art. I like that there are dual redundant streams. And the scale of the dataset is already well know: 4096 HD-images per step!
If I had to speculate, the "Dojo Cluster" may be envisioned as an effort to share data and compute with industry partners as a cloud SaaS product and ancillary revenue stream. But that is pure speculation ;)
Inside Tesla’s Neural Processor In The FSD Chip
https://fuse.wikichip.org/news/2707/inside-teslas-neural-pro...
Of course, papers and data aren't code. But I think a lot more is being shared than people realize.
[0] https://eng.uber.com/author/raquel-urtasun/ [1] https://eng.uber.com/research/?_sft_category=research-self-d... [2] https://waymo.com/open [3] https://arxiv.org/pdf/1812.03079.pdf
Right now, the competition in the self-driving area is metaphorically as close to do-or-die as you can have in peace time. GM as a regular car manufacturer is toast, Cruze is pretty much their only hope. Uber is bleeding, if they pull self-driving off, they are kings. The German manufacturers are watching in disbelief as Tesla is starting to eat their pie. Conversely, Tesla knows that in what they are doing (electric cars) they don't enjoy any fundamental moat. If the Germans get their act together, they'll be able to make equally performant electric cars, but true luxury. The only one that's not really about survival is Waymo.
Maybe the companies actually building the nuclear plants and trains and rails were actually in competition ?
Once that kind of secrecy was gone our whole technical progress was accelerated, because people could build on the discoveries of other people.
Right now we are going back to the alchemist model in some ways (the highest profile people work for the big companies and don’t share their discoveries). This makes progress slower.
I have to strongly disagree with this for the specific case of AI/ML. The big company labs are publishing open access papers non-stop, often with code and sometimes even datasets. They're more open than some areas of academia, in fact.
Easier said than done, but I think it would strike the right balance between reducing duplicate work, and incentivising progress.
Roads are also governed by public bodies. Road signs are standardized and public.
I think the government should take a much larger role in defining self driving cars. For example, rather than using computer vision to recognize signs, signs could be active standardized beacons; instead of having to recognize lanes, they could be repainted with rfid chips that are trivial for cars to recognize and follow.
Avoiding driving into people is also something that was somewhat regulated by crosswalks with pedestrian lights. Would it be absurd for the crosswalk to know roughly how many people are at it, then broadcast this to the car, rather than having the car have to recognize them?
There are many things the government could do with transportation infrastructure that would benefit everyone, many of which are literally impossible for companies to do separately. Can you imagine if we had to wait until IBM (or Siemens or Google or Apple) got into the business of launching satellites before we got GPS? There is a good chance that to this day cell phones wouldn't know their location or give anyone any mapping applications.
To me, self driving cars are similar. Many parts of transportation are a public good.
Yes, completely absurd. For one, many places are too sparse, poor, or unstandardized for this to be remotely economical. Two, people often don't cross at crosswalks (jaywalking). Level 5 AV is a thing where 99% coverage isn't good enough. You need a lot of nines.
That's why many think lidar systems are a crutch. If lidar can't work in snow or heavy rain (at least 1-2% of days in the north), then you need fallback which must still be >99% as effective to avoid incidents. But then why not just use the fallback?
Generating data? Sure. But for actual use in the data pipeline? Makes you rely on an ultimately untenable solution.
I can see the case for doing their own edge hardware for the cars (barely), but I really don't think doing training hardware will pay off for them. If they're serious about it, they should spin it out as a separate business to spread the development cost over a larger customer base.
Also, I'm really curious whether the custom hardware in the cars is benefiting them at all yet. Every feature they've released so far works fine on the previous generation hardware with 1/10 the compute power. At some point won't they need to start training radically larger networks to take advantage of all that untapped compute power?
It's not surprising that they also build the hardware for training. Correct me if I'm wrong, but Google use the same TPUs for training and inference, because the underlying operations are the same : multiply then add numbers. Once Tesla built the hardware for inferring, the design of the hardware for training is probably similar.
Unlike Google's TPUs, Tesla have a specific use case for the hardware (computer vision for automotive), and maybe than means they can further optimize the computation pipeline with their own specialized hardware.
- It's under 100W so they can retrofit into old cars
- lower part cost, so they can do full redundancy with doubling the parts
- they estimated that 50 TOPS is needed for self-driving
- lower latency with batch size of 1 compared to TPU's 256
+ GPU for post-processing
- security: only code signed by Tesla can run on the chip
- at the time (2016) there was no neural net accelerator chips
- some part's are built from bought IP (so not reinventing them) Probably things like the 12 ARM CPUs, LP DDR4 memory, video encoder, maybe the separate post-processing GPU too...
- physical size of the board is small
- performance example: on CPU 1.5 FPS, on GPU (600 gflop) 17 FPS, on Tesla's NN accelerator 2100 FPS
- Besides the convolution even the ReLU and Pooling is implemented in hardware
- Paying attention to the energy efficiency down to the arithmetic and data type usage.
- The silicon cost is less than their previous hardware (HW 2.5)
- old hardware 110 FPS new one 2300 FPS
- 144 TOPS compared to NVidia's Drive Xavier 21 TOPS
I would expect their training hardware to be something specifically aimed at optimizing memory bandwidth to support distributing training of their “shared” hydra feature. It’s interesting that the shared hydra feature extractor is able to converge as they keep adding more and more output predictions under a training regime of interleaving asynchronous updates to the model from different predictor networks ...
Seems to me the formula they are pursuing with custom hardware might be to support a strategy of 1. keep adding more predictions based on same feature 2. Increase the span of time represented by batches used to train the recurrent networks
Both pursuits seem very data efficient in terms of the amount of training data they could conceivably collect per unit time of observation ...
Custom hardware with a problem specific memory architecture aimed at efficiently supporting training with very large rnn time slices could be developed that’s more about “make it possible to train this proposed model at all” rather than “make it faster/cheaper to train existing common model architectures”. When custom hardware is required to make it possible to train the model they want, the validity of the hardware development cost bet might end up being more about the effectiveness of the model they think they want than it is about maintaining general purpose performance parity vs any off the shelf hardware options ...
Not agreeing or disagreeing with their decisions, but if you have the resources, you can certainly design a custom chip that performs a specific type of task very well that beats other competitors. Nvidia's GPUs are have to be reasonably good at training across different NNs. You could have a chip that's exceptional good at training one/two specific types of tasks.
For most companies, this would be a bad idea. However, Tesla knows how to produce hardware.
Not sure if it’s still the case today, but previously Tesla’s training was done on-prem and with their own in-house Tensorflow clone.
And yes, if you get TPUs from GCloud, you are likely to be working with their engineers to get things working. Those engineers tend not to have much business conflict of interest, though. They want to help you because your problems are likely more interesting than what they’d otherwise be assigned.
By hardware-hours, Tesla is hardly one of the top companies training deep networks. Planet Labs (satellite imaging), Netflix, Pornhub, to name a few.
What's the info they'd leak to the Google consultants? How much data or TPUs they're using? This is practically public information.
Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would be a good idea to upload Tesla's proprietary data into Google Cloud even if it did not. Cloud, after all, is just somebody else's computer.
I don't think this is true. If you're talking about https://news.ycombinator.com/item?id=19855099, it doesn't apply to TPU hardware, as is explained in the comments there.
> TPU-like stuff is ~10x the energy efficiency of GPUs
10x is probably overstating it when talking about newer GPUs because they have ML hardware in them now. Also, that still doesn't make it a good idea to build your own chips because there will soon be many third party options to choose from. Doing your own chips is a bet that you will out execute dozens of companies ranging from startups to industry giants. Simply taking your pick of the best commercially available options is likely to be a better choice in the near future.
GPU advantage: more refined ecosystem and you can buy them for $<1000 or get laptops with them built in, and if NVDA has sweat more software engineering blood and tears than GOOG into your model's functions, it will run better on them
TPU advantage: Colab has a free tier that lets you play with them at no charge and if GOOG has sweat more software engineering blood and tears into your model's functions, it will run better on them.
All IMO of course. And deep down it can get more complicated than that, but I salute GOOG for being the first company to ship competitive AI HW, doubly so at scale.
TPU? Seems like it has a lot of potential, but not for people directly competing with them.
Waymo's Laserbear lidar? Seems like it has a lot of potential, but not for AV companies directly competing with them.
Google's playing this game pretty fiercely... which given their size is pretty bad/daunting.
could you share the stats on this? Google told me to use a K-80 for training.
Can it be true? Then again, Apple's app store behaviour seems to suggest such demands are tolerated. Antitrust is really asleep in the US, isn't it.
Also, vendor lock-in is a huge challenge in the cloud space. I don’t think Tesla would be comfortable with the fact that all their training data sits on a potential competitor’s datacenter.
The latest OTA finally brings a hardware v3 only feature, traffic cone visualization, and traffic cone automatic lane change.
Looks like they are really nicely orchestrating workloads and training on numerous nets asynchronously.
As a person in the AV industry I think Tesla's ability to control the entire stack is great for Tesla... maybe not for everyone who can't afford/doesn't have a Tesla.
Affordability is not as much of an issue as some make it out to be. Cost-wise it's like owning a Camry or an Accord, if you go for the lower end models. If you mean not everyone can afford a new car, then sure I agree with you.
Edit: if you think I'm wrong about this, please explain or ask me to clarify anything?
Similarly, the infrastructure around electric charging stations I believe hasn't fully matured yet and as a result many people who've already owned a car, I believe will stick with gas cars since there's no huge incentive to change, unless it becomes easier to charge (faster, more convenient).
Do note that I don't have a drivers license. I never intend on getting one (I believe in what I do in the AV industry). I'm just guessing on the habits of people, not that I have any real experience in buying a gas/electric car.
Also note I didn't think you were wrong, not sure why the downvotes.
The smart solution would be to consider a map a probabilistic thing, which neural networks are really good at handling.
LIDAR's are cheap, but not cheap enough yet to not seriously affect the bottom line if you put them into every car. It also will kill the resale price of cars without it, which in turn hurts the companies image and stock price.
>“Lidar is a fool’s errand,” Elon Musk said. “Anyone relying on lidar is doomed. Doomed! [They are] expensive sensors that are unnecessary. It’s like having a whole bunch of expensive appendices. Like, one appendix is bad, well now you have a whole bunch of them, it’s ridiculous, you’ll see.” https://techcrunch.com/2019/04/22/anyone-relying-on-lidar-is...
https://cleantechnica.com/2016/07/29/tesla-google-disagree-l...
To me it seems we're still in really early days.
https://youtu.be/Ucp0TTmvqOE?t=8137
And later in the video they show 3D reconstruction from cameras and saying they use it in the car.
Watching the full talk is recommended if you have the time (talk starts around 1:10:00 in the video)
He describes the sub-graph training in the context that they they have all the predictors in one big model, and with control of the network can feedforward and train sub-graph (read sub-parts) of the model.
They talk about the concept of "modular network". The article itself links to the Wikipedia page : https://en.wikipedia.org/wiki/Modular_neural_network
Not sure it's exactly the same idea, but it looks similar.
- TensorFlow is great at deployment, but not the easiest to code. PyTorch isn't frequently used in production until recently.
- If you have the resources for great AI engineers and researchers, your team will be good enough to build and deploy both frameworks.
- Preference toward the easier framework your tech leads prefer.
- Lots of new academic research is coming in PyTorch
- TensorFlow is undergoing a massive change from 1.1x to 2.0; if you choose TensorFlow, write on 1.1x just to then refactor to TF 2.0? Or write on TF 2.0 now and deal with all new edge cases? Or write in PyTorch (easier) but handle the more difficult deployment process.
- ML code quickly rots. Bad PyTorch code is just bad Python code. Bad TensorFlow code can be a nightmare to debug.
- PyTorch's eager execution makes coding NNs much easier to prototype and build.
He is an excellent presenter who really has a passion for teaching.
Im not really involved with the industry, so I cant really speak to how he holds up to other experts. However he is by far the most digestable resource I have found for learning about NN and science behind them.
If you are just discovering him now, google his name and just start reading. His work is truly binge worthy in the most meaningful way.
https://kitsunesoftware.wordpress.com/2017/07/27/why-do-peop...
Lidar helps you spot obstructions, but won’t tell you what they are and won’t help you figure out what to do to avoid them.
Want an example? Cruise’s first real world demo got stuck behind a simple taco truck in downtown SF.
Tesla is already using a model to rebuild a 3D space from Camera data only, the parent suggests to improve the quality of the transformation with high quality 3D representations from Lidar.
It's deep learning all the way down.
Your comment just reeks of anger and hostility.
It seems like you'd rather Tesla didn't try at all, and instead we all just give up and go back to the status quo.
How can we rely on the output of eight cameras? This is not a kid's science project.
It's all fancy neural networks until someone dies. Pretty callous and Silicon valley-mindset for such an important and critical function of the car.
Will never buy a Tesla after having seen this.
It's also mind boggling to think we currently trust organic tissue to do this crap, some of which is bathed in psychoactive chemicals.
And yet we do, and as a result, horrendous catastrophes occur every minute of every day.
> It's all fancy neural networks until someone dies
No, that can't be the standard, not when people are dying right now in the current regime.
Unless the new regime kills and maims at a higher rate than the current regime, there is no reason for fear.
Current averages for the US are around 10 per billion miles and decreasing; best countries in Europe are already below 5 per billion miles.
There's some really interesting progress both on vehicles monitoring driver attention, and on monitoring for alcohol in the air, that would yield substantial improvements even beyond 2019 tech. I have no doubt we'll see fatality rates below 1 per billion miles in Western Europe within a decade.
The corollary of the statistics quoted above is that you need to observe your self-driving vehicle system over tens of billions of miles before you even know if you're safer or not.
The appropriate standard for self-driving cars is a sober professional driver, not the average idiot on Saturday night. It isn't unusual for a driver to go a million miles without an accident, when that's their job.
However the reason are not eight cameras. You should be able to drive fine with just one camera (thought experiment: could you drive a car 1000 miles from you, just by seeing what the driver of that car would see, no extra cameras, sensors or lidars?).
And that took many decades not even years and in a very constrained problem space.
Releasing a "self driving" metal torpedo even if it's a limited Summons should at least be regulated if not outright banned.
I feel like even a single accident let alone death should put these idiots behind bars.
Please explain, in detail, what your specific objections are, and how you are more qualified on this subject matter than the presenter.
It's often reactionary - people wait for someone to die and then suddenly you have nervous Musk in front of Congress and such crap. Why wait? Why can't these be regulated before allowing on roads?
And no, failover control is not acceptable given the past incidents and deaths.
Tesla makes this very clear.
It's also all over the Internet.
But there will always be people who do not get this.
Those people should not drive Teslas, or pretty much any modern car for that matter.
If you are under the impression that you would be relying on this system to drive the car, then I agree you should not get a Tesla.
Of course full self driving is coming at some point, but that's a conversation for another day. Meantime Tesla is making steps toward it very incrementally with things like the "stop mode" rolling out right now.
Watching the AI visualization of summons in-action was horrifying, and made clear why many have reported summons mode as resembling a drunk person navigating a parking lot.
