Could we construct a neutral net from nodes with more complex behaviour? Probably, but in computing we’ve generally found that it’s best to build up a system from simple building blocks. So what if it takes many ML nodes to simulate a neuron? That’s probably an efficient way to do it. Especially in the early phase where we’re not quite sure which architecture is the best. It’s easier to experiment with various neural net architectures when the building blocks are simple.
This is probably what you're remembering: https://www.sciencedirect.com/science/article/pii/S089662732...
Well there's spiking neural networks (SNN)[1], which are modeled more closely to how neurons actually work.
Main obstacle is still, as far as I know, that there's no way to train a SSN as efficiently as a "regular" neural network, which lends itself very nicely to gradient descent and similar[2].
An infinitely-fast computer wouldn't meaningfully change the "expensive training vs fast, static inference" workflow that neural networks have always been developed around (except in the most brute force-y "retrain on the entire world, every single nanosecond" sense).
Neural networks fundamentally aren't designed to be otherwise. The workflow that has guided their entire development for over a decade is based around expensive training and static inference.
To me that just means nobody has figured out how to do that effectively. The majority will simply make use of what's been done and proven, so we got a plateau at object recognition, and again at generative AI (with applications in several domains). One problem with continuous adaptation and learning is providing an "entity" and "environment" for it to "live" in which doing the adaptive learning. There are some researchers doing that either with robots, or simulations. That's much harder to set up than a lot of cloud compute resources. I do agree with you that these aspects are missing and things will be much more interesting when they get addressed.
Please don't claim things the author didn't. What I read was "ergo (artificial) neural networks may be missing a trick"
I mean, sure, but the topology is exactly what makes both work, so we only really care about the topology.
Agreed, a widespread fallacy of category.
But computers still do some pretty cool things. Powerful tools.
Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong."
Neurologists seem to have a really hard time comprehending that researchers working on neural networks aren't as clueless about computers as neurology is about the brain. They also vastly overestimate how much engineers working on neural networks even care about how biological brains work.
Virtually every attempt at making neural networks mimic biological neurons has been a miserable failure. Neural networks, despite their name, don't work anything like biological neurons and their development is guided by a combination of
A) practical experimentation and refinement, and
B) real, actual understanding about how they work.
The concept of resnets didn't come from biology. It came from observations about the flow of gradients between nodes in the computational graph. The concept of CNNs didn't come from biology, it came from old knowledge of convolutional filters. The current form and function of neural networks is grounded in repeated practical experimentation, not an attempt to mimic the slabs of meat that we place on pedestals. Neural networks are deep because it turns out hierarchical feature detectors work really well, and it doesn't really matter if the brain doesn't do things that way.
And then you have the nitwits searching the brain for transformer networks. Might as well look for mercury delay line memory while you're at it. Quantum entanglement too.
There are insights that can come from studying the brain, that do indeed apply. Some researchers may not glean anything from such studies, and some may. I have no doubt that as neural networks get more an more powerful, we will continue to find more ways they are similar to the brain, and apply things we've learned about the brain to them.
I certainly prefer to see people making comparisons of neural networks to the brain, that the old "it's just a glorified autocomplete" and the like.
Relax.
1. https://braininitiative.nih.gov/sites/default/files/document...
> The concept of CNNs didn't come from biology
I just opened a survey paper on CNNs and literally the first sentence of the paper reads:
> “Convolutional Neural Network (CNN) is a well-known deep learning architecture inspired by the natural visual perception mechanism of the living creatures. In 1959, Hubel & Wiesel [1] found that cells in animal visual cortex are responsible for detecting light in receptive fields. Inspired by this discovery…”
Source: https://arxiv.org/pdf/1512.07108.pdf%C3%A3%E2%82%AC%E2%80%9A
The C in CNN isn't "Convolution" for no reason. It came from work with convolutional filters (yay Sobel kernels!) which at it's height became filter banks and gabor filters and so on before neural networks pretty much killed off handcrafted feature development. Every explanation of how CNNs work still falls back to the original convolutional kernel intuition.
Before that: comparing brain with hydraulic machines. There has been tendency to compare brain with most complex machine known to us at that particular time.
"Descartes was impressed by the hydraulic figures in the royal gardens, and developed a hydraulic theory of the action of the brain. We have since had telephone theories, electrical field theories, and now theories based on computing machines… . We are more likely to find out how the brain works by studying the brain itself, and the phenomenon of behavior, than by indulging in far-fetched physical analogies." -- Karl Lashley 1951
There's little sense in ignoring the whole basic mode of operation, physics, chemistry and biology of the brain in order to analogise it to another system without any of those properties.
This, at best, provides a set of inspirations for engineers -- it does nothing for science.
[1] - https://en.wikipedia.org/wiki/Convolutional_neural_network
It's not surprising that we found out later the brain also uses such a fundamental element of signal theory.
[1] https://www.amazon.com/Biophysics-Computation-Information-Co...
Please don't claim things the author didn't. What I read was "ergo (artificial) neural networks may be missing a trick"
But in reality, we’re equipped exactly to exist, and we still wonder why in a backwards way, even with education (guilty!)
AI is the task of playing God like toddlers at recess, and LLMs the tower of babel. I still wanna play, it’s fun
Second, there is no need to compare brains to neural networks because brains are neural networks. Neurons form vertices and axons edges connecting the aforementioned. What you are perhaps thinking of are artificial neural networks - most of which are very dissimilar to brains. But even then you are wrong. Artificial Izhikevich and Hodgkin-Huxley neural networks attempts to closely mimic the behavior of real neurons.
While deep, hierarchical artificial neural networks have been more successful than biologically plausible ones, that may be because the technology isn't ready yet. After all, the perceptron was invented in the 1950's but didn't become prominent until the 2010's (or so). Perhaps we need new memories that better map to (real) neural network topologies, or perhaps 3d chips that can pack transistors in the same way brains pack neurons.
Changes in mechanical pressure, electric field, other molecules attachment, photon absorption, can control the conductivity.
Organic semiconductors designed to fit like lego bricks to naturally build the desired structure are IMHO the way to go to produce 3d circuits, rather than layered silicone litography.
As a developmental neuroscientist, I found the article insightful and thought provoking. Further, it is quite consistent with major hypotheses in psychology, how the hippocampus works (a subcortical structure) and combines information into memories: See fuzzy trace theory [1], for example.
Your dismissive tone is unappreciated, ill-informed, and crass.
Value of this comment aside, it kind of makes me chuckle how casually it (and other comments in this thread) just drops the word "artificial" from neural networks here, specifically when comparing with neurology. The irony is funny. Like, somehow we've forgotten why we call them that in the first place, exactly when talking about the thing that inspired the approach.
There are things the brain does we have not yet been able to reproduce with a neural network, or to the extent we have seemingly with excessive resources of training and network size. Therefore there is some salient feature of neurology which has been overlooked. I don't think it is necessary to mimic biology down to the exact function of real neurons, but there must in fact be something we are neglecting to mimic.
"Book smart, not street smart" (to use a catchphrase) would apply perfectly to GPT models: brain the size of a rodent's, with 50,000 year's experience of reading Reddit, Wikipedia, and StackOverflow, but no "real life" experiences of its own.
It is more useful to use AI to develop more ecologically valid measurement methods for biology.
This is why today, if you need a low-latency NN, which means a shallow one, often your best bet is to train a deep one first and then distill or prune it down into a shallow one. Because the deep one is so much easier, while training a shallow one from scratch without relying on depth may be an open research question and effectively impossible.
It doesn't. You can speak perfectly fine with children. And in fact some teenagers think they know everything.
As I understand it the thalamus is basically a giant switchboard though. I see no reason to believe that it never connects the output of one cortical area to the input of another, thus doubling the effective depth of the neural network. (I haven’t read this paper though, as it was behind a paywall.)
Your comment would be very valuable to me if it included pointers to better sources. I have sufficient background to see gaps in Jeff's book, and would be interested in exploring these, perhaps through the references you seem to be aware of.
