Speculative proposal to create artificial neural nets with human-like performance by high-learning-rate/regularization training of overparameterized NNs to trigger catapulting/grokking. Over-parameterization as a route to true generalization would resolve many outstanding mysteries of artificial versus natural intelligence.
There are many mysteries about deep learning and human intelligence, but we could describe the biggest anomaly as this: “why are artificial neural nets smart in stupid ways, and biological brains stupid but in smart ways?”
I propose a major change in deep learning scaling paradigms: the architectural differences between human brains and NNs (particularly LLMs) may be due to a bias-variance tradeoff, where LLMs minimize variance and human brains minimize bias. Human brains do this by deep double descent-style overparameterization, and adopting a scaling strategy of extremely high-learning-rate training of extremely overparameterized models on small diverse highly-filtered datasets. This approach would lead to sample-efficiently and compute-efficiently traveling (or catapulting) to a highly-generalizing human-like basin in the model loss landscape, while performing poorly up until the end and failing to memorize much data.
If true, this would explain a number of odd stylized facts about how humans/NNs perform well/poorly.
Such a ‘catapulted LLM’ would generalize much better than existing NNs, be immune to adversarial attacks, have better economics and be more resistant to cloning, could potentially enable extremely efficient MLP architectures, and by giving true generalization, provide a sturdy foundation for AI safety in the form of useful NNs which are aligned & safe for the right reasons.
This could be feasibly tested by training multi-trillion-parameter models for relatively few steps at high cyclical learning rate schedules, and benchmarking adversarial and hard examples on tasks like arithmetic and small-image classification.
Because deep learning has continued to scale up and smash through benchmarks and begun to look like it really will be the final AI paradigm, and thus in some sense the same thing as human ‘intelligence’, to a considerable degree, we can regard ‘intelligence’ as solved: intelligence is sufficient compute applied to search over programs (like Turing machines or circuits) to predict or optimize where the optimal solution is a relatively long program.
(This is a companion piece to “Guardian Angels: LLM Personalization for Productivity and Security”.)
Intelligence, Broadly
A scaling-centric view might be summed up like this:
Anomalies
But this paradigm, as broadly correct as it now seems to be, doesn’t explain everything. We still have many specific problems that this paradigm is too general to explain.
While current NNs, and LLMs in particular, are by far the most human-like AI software ever created, in having human-like strengths and weaknesses, there are a number of anomalies in machine & biological intelligence that have no good answers.
We have many puzzles here, but they all feel connected, somehow.
Artificial
Sample Inefficiency
Why do NNs require Chinchilla-style scaling of data and compute, when humans appear to learn from multiple orders of magnitude less data, and it is increasingly plausible (given various estimates of human-brain equivalents) that they learn from less total compute? Why, as so many connectionist pioneers like Alan Turing expected, do we not train AI like children, with a curriculum and clear developmental stages?
There are many answers offered, none satisfactory. (And what should we make of theoretical results like Rosenfeld 2021’s “Nyquist learners”?)
Multi-modality: while useful, multi-modality has failed to yield any major change of scaling law exponents; unimodal models work shockingly well, and language models turn out to already encode a large amount of visual knowledge and can easily be plugged into vision models (eg. Flamingo, Tsimpoukelli et al 2021).
Human sensory input is actually large: Another common explanation is to deny that humans learn from less data, and argue from raw sensory bandwidth: if vision+sound+touch is such-and-such bits per second and you accumulate over an adult’s lifetime, it can look much more comparable to the trillions of tokens we train an LLM on.
This is unconvincing because the raw sensory bitrate is meaningless: the input is extremely redundant & predictable for the most part. (Imagine sitting in a room staring at a computer screen.) Attempts at quantifying the information content of images, video, or sound, usually indicate that they boil down to the equivalent of a few hundred or thousand tokens and those modalities are easily learned by small models (eg. iGPT/DALL·E 1). The asymmetry is particularly striking in text-to-image generative models, where the text encoder (usually an afterthought) is often far bigger than the image generator itself.
And on the human side, disabled people are not much less intelligent than normal humans: deaf/blind people are much worse at language tasks, but their fluid intelligence often remains normal. If the sensory bandwidth were so critical, this would be impossible.
Active Learning: human children, unlike models confined to offline imitation learning, can choose what to learn about by exploring their environment or asking questions. In theory, active learning & optimal exploration can be far more sample-efficient than indiscriminate training (exponential rather than power law, at a minimum), and this could account for the entire gap.
However, if we look at the things children actually choose, the data in question doesn’t appear all that amazing. Further, in stark violation of any notion of optimal Bayesian exploration, children often choose to learn on the same data point—eg. watching the same YouTube video hundreds of times.1 Or if we watch them ‘explore’ a game or computer, it looks like it is by acting largely at random, and an adult would learn far faster by more carefully thought-out exploration.2
Embodiment: a closely-related topic is the idea of “embodied cognition”, which used to be quite popular as an explanation for the weaknesses of AI—AI models simply lacked commonsense & generalization for lack of a body and an appropriate environment.
But thus far, ‘embodiment’ like training on robotics data (eg. Gato) has exhibited zero transfer to other tasks, never mind massive scaling law gains, and ironically, it is, in fact, embodied tasks like robotics models which have been greatly benefiting from non-embodied pretrained models (including LLMs!).
Architecture Magic: Perhaps in some way, Homo sapiens-style biological neurons are just some near-perfect architecture, and this explains most of the gap; someday we will understand how all artificial neurons are severely hobbled by mistakes that will seem as tragically obvious in hindsight as earlier mistakes like not using backpropagation or using sigmoid activation functions now seem to us, but they remain a mystery for now.
This view was highly plausible until recently, but has been running into many problems.
For starters, we simply have not found any architecture magic. The most obvious place to find magic would be the learning rule for biological NNs, whatever they use in place of backpropagation… But while people have proposed many biologically-plausible learning rules since Hebb proposed the first learning rule in 194977ya, which respect the requirements like locality, in every case, those learning rules perform worse than, or at best similar to, backprop! To quote Geoff Hinton:
So maybe it’s [GPT-4] actually got a much better learning algorithm than us.
And if biological NNs are not so good but there is something special about humans which does make them much better, then why do Homo sapiens not appear to have any major neuroscientific breakthroughs compared to our primate relatives? Why are we so genetically similar, and we have failed in the search for major novel mutations that create humans, and human brains seem increasingly like nothing but “a scaled-up primate brain”? If human (or primate) brains are so uniquely efficiently tuned by evolution, why are bird brains so much more efficient in size & thermodynamics than primate brains, with clear genetic changes, and better scaling to the point where small bird brains like ravens or parrots or vultures exhibit eerie levels of intelligence & behavioral complexity almost on par with dolphins, chimpanzees—or humans? Why does human intelligence exhibit so many bizarre drawbacks or anomalies if it is so optimized, like childhood amnesia or lurching through developmental phases over decades?
If the story of NNs is one of us gradually recapitulating evolution’s perfect neural networks, why does neuroscience provide so little useful inspiration (as is emphasized constantly by neuroscientists & AI researchers, even the “neuroscientific” inspiration for things like self-attention is a very loose inspiration)? Why don’t all major improvements come from success in reverse-engineering the human brain to ever greater biologically-realistic detail? Why is the rapid progress in neuroscience, like scanning entire connectomes, completely irrelevant to cutting-edge NN models? Why do all the scaling laws for CNNs & Transformers look so similar in the exponent, and durable improvement so difficult, to the point where 7 years after Vaswani et al 2017, it is still a relevant baseline? And why, if we have such fundamentally inferior architectures, are the scaling laws so smooth & reliable, instead of breaking frequently or predicted to asymptote at levels far below human?3
Conversely, why do NNs provide little insight into biological brains? In the other direction, neuroscience and individual psychology has hardly benefited from DL; DL has provided tools, and has provided good predictive models of brains, but that is about it. One could open an issue of a psychometrics or neuroscience journal, and note that, over a decade into the deep learning revolution, if it had never happened, that issue would look about the same and would be completely intelligible to a researcher from 201016ya.
It is impressive that we can now turn fMRI scans into crude visualizations of what a person is looking at, or we can use LLMs to generate possible questions for surveys, but DL has provided essentially no major insights into such fundamental questions as “what is fluid intelligence? why is the g factor so general? how do neuroanatomic traits like neuron count or network properties cause intelligence or other cognitive abilities?”
Isn’t this astounding? We can now create models of enormous generality like Gato or GPT-4o from scratch to match or exceed humans without the hand-engineering of GOFAI, which seem so eerily human-like in many ways, and which recapitulate so many aspects of human cognition down to heuristics & biases, and our ability to create these artificial intelligences tells us… nothing important about the human brain? Really?
So what, all the DL is just a dead-end and coincidentally capable of all that, and sometime in the future we’ll discover the real route to brain-like intelligence…?
Sample Efficiency
Why do NNs require so much data to pretrain, when NNs are as sample-efficient in narrow comparisons?
Despite the huge amount of indiscriminate data used in NN pretraining, we are puzzled because if we examine how well models do in apples-to-apples comparisons, NNs appear to be as roughly as good as humans or biological neural networks at learning from small data.
The simplest answer to the exorbitant data-scaling would be that NNs do a bad job of learning each datapoint—but that doesn’t seem to be true when we compare things like in-context learning (even GPT-3), transfer learning smoothly scaling, child-sized datasets (eg. SAYCam, BabyLM, TinyStories), compare total AlphaZero Go/chess games to total games played by human pros over history, or disable human priors to compare learning speed (cf. human vs Transformer meta-learning), and remarkable experiments like learning an unknown language from a book or raising chickens in virtual reality & comparing their visual capabilities with a Transformer (or vice-versa, have profoundly mistaken beliefs about the world when raised passively in a bubble).
How can we reconcile this apparent contradiction?
Smallness
Why are NNs so intelligent, and yet so small?
A NN like GPT-3 is ~0.1t parameters (which are absurdly simplified caricatures of biological neurons), while the human brain is extremely loosely estimated at 100t ‘parameters’, and many neuroscience results are taken as implying that each of those ‘parameters’ translates to thousands of equivalent parameters, at a minimum; an LLM is further disadvantaged by lack of recurrency & online learning, and is generally unable to engage in adaptive computation or ‘ponder’, the way that a brain can continually learn and ruminate. And yet, despite all these severe disadvantages, LLMs seem too good. It does not feel like a GPT-3 knows or does less than 1⧸100,000th of what the human brain does; but if we argue that a more plausible number like 1⧸1,000th is true, that appears to commit us to the position that human brains are highly wasteful of parameter-equivalents (despite the touted superiority of biological brain architecture & evolutionary pressure for efficiency). And if a GPT-3 really does know that little, then what are those >99,999 things missing from a GPT-3—which each require an entire GPT-3 to handle?
This observation becomes even more extreme as a NN like GPT-3 is well-known to be highly overparameterized and can be shrunk down by orders of magnitude, and it is also known that small NNs can be trained for a long time to achieve performance of large models (albeit compute-inefficiently). We appear to be between Scylla & Charybdis.
Alternate version: why are human brains so overparameterized? Many humans get by with much smaller brains than others; there is a real and causal correlation with brain volume, but even allowing for how crude a proxy that is for the underlying neural nets’ capacity & speed, the effect size seems surprisingly small. Biological brains can survive shocking amounts of physical damage or brain loss, as long as it occurs early in development (eg. hemispherectomy; see also hydrocephalus), and it’s surprisingly hard to find brain lesions which damage IQ instead of specific cognitive abilities. Cross-species, there is clear allometric scaling and the rankings make sense, but the slope is shallow: chimpanzees with less than a third of the brain often seem competitive with some humans in terms of intelligence and ingenuity. And remarkable instances like Portia spiders or the ability of the dragonfly to home in on prey using barely 16 neurons further raise the question of why so many neurons are necessary (as does, indeed, power law scaling in general).
Superhuman Prediction
In particular, why do even tiny LLMs like GPT-2 appear to already be superhuman at next-token prediction in terms of perplexity/bits-per-character, and yet still greatly underperform humans on benchmarks & real-world and benefit in every way from scaling up next-token prediction pretraining?
Next-token prediction pretraining clearly works (and continues to work), and yet, also something is clearly missing from the simple story we tell about it. Humans appear to predict better the more important tokens; why and how?
Persistent Adversarial Examples
Why are NN adversarial examples still unsolved over a decade later, while the best efforts at human adversarial examples are debatable?
Scarcely any problems from 2013–201412ya DL remain, and yet, in 2024, adversarial examples are not merely still around, but as powerful as ever on standard NN models. Countless solutions have been proposed—and failed. The best defenses are weak & compromise performance, like adversarial training damaging generalization (but oddly, seeming more human-like), or unable to scale, like robustness certification. SOTA NNs rarely even attempt to reduce adversarial examples, LLMs are highly susceptible (and have rough decision boundaries), and DL practitioners usually accept them as a fact of life & work around them.
Meanwhile, attempts to transfer to humans, or create new ones… work to some degree, but not much, and pushing them to the point of meaningful human error requires such large distortions that they no longer seem meaningful.
Biological
Human Amnesia
Why are humans so forgetful compared to NNs?
If forgetting is critical to flexible decision-making in biological brains like rodent studies, as has been argued, in part by analogy to successful machine learning regularization techniques like weight decay or dropout, why do the most successful NNs still remember so much compared to humans? For example, an LLM like GPT-3 can casually memorize entire pages of text after training on it once. This is not an extraordinary capability, but happens routinely with LLMs. And the bigger and better they get, the more they are able to memorize.
Whereas with a human, even ones who pride themselves on their memory for text (like myself), memorizing any page of text is effectively impossible without extensive practice in mnemonic techniques, and we tell awed stories of those few human beings who appear to have photographic memories (and demonstrate it is possible for human brains to do such things, they just don’t usually). The usual human’s lack of a photographic memory, and the shallowness of our understanding, can be demonstrated with hilariously simple examples, like “how is the letter ‘g’ written?” or “which way does Lincoln face on the American penny?” Our memories (and competence in general) are so bad, in fact, that in addition to struggling to remember what we ate for breakfast yesterday or learning basic general knowledge like “the capital of Russia”, we forget almost our entire childhood, and treat this childhood amnesia as completely normal and may even deny that fact.
This is particularly puzzling given that: we never have more neural connections than we do in childhood, as they continually die or are pruned away; it is not universal cross-species; the amnesia doesn’t seem to damage more core knowledge like motor ability or language; important traumatic memories can be retained; and that those lost episodic memories are, to a considerable degree, still there—detectable by implicit measures like lower reaction times or faster recognition of blurred images or needing fewer hints/cues.
Nor is this because memorizing facts is useless to learning; memorization is critical to learning. But then, if memorizing is so useful and more knowledge is better, why do the most successful human memorizers seem to underperform, and it is a mixed blessing at best? Why are examples of photographic memory anecdotally associated with odd limitations in creativity or generalization, like John von Neumann or Luria’s Solomon Shereshevsky, when experimentally in spaced-repetition memory research, better memorization appears to help generalization & understanding without such drawbacks? Why are normal humans capable of memorizing some things, like the Quran or Paradise Lost or the Homeric epics, given enough time & motivation & tricks like memory palaces, but not routinely?
The most extreme cases are idiot savants, whose overall performance profile and pre-DL descriptions bear an uncanny description to LLMs. The feats of savants are too well-documented to doubt, but are impossible to fit comfortably into standard frameworks: if any humans could defeat LLMs in their strong points, it is savants, but then why does savantism seem to only come with such severe costs?
Human Ignorance
And why are humans so ignorant compared to NNs, overall?
Not only do we not recall much easily, we don’t know much either: the breadth of knowledge of even a low-quality LLM in 2024 vastly surpasses that of any individual human. Individual expert humans still usually beat the best LLMs in their narrow niche of expertise, but if we quizzed humans with the same breadth of questions that we benchmark LLMs, I don’t think there is a human alive who would come anywhere near them.
Human Intelligence
So why are humans so smart in general, ie. why do humans generalize more robustly (eschewing the use of non-robust features/‘dimpled’ manifolds like GAN steganography4), are more creative, and largely immune to adversarial examples?
One of the most counterintuitive aspects of NNs is that for all the incredible fluency & genuine capabilities, they nevertheless still occasionally make simple blatant mistakes. LLMs in particular—even excluding technical issues caused by BPEs, RLHF & other safety tuning, greedy sampling, and so on, it remains true that they will make baffling errors which are fatal and cause them to go in circles and fail to generalize for things where there seems to be ample training data.
Another version: given that NNs seem to so easily memorize & overfit (and famously can memorize all training data), and operating by “direct fitting” to ‘interpolate’ between the “underspecified” training data, why do they generalize at all?
Need to Sleep
Why do humans need to sleep—so much so that we usually die if sleep-deprived harshly enough?
In contrast, there is no known equivalent of sleep in current DL, nor any apparent need for it.
Even for continual learning or reinforcement learning replay, ordinary minibatch SGD suffices. There is no need for extended periods of offline inactivity or non-interactive computation simply to train on some new data.
Stage-Wise Development
Why does human development spurt through stages to a much greater degree than NNs do?
Children seem to often develop in phases, with sudden spurts (Van Der Maas et al 2006; van Geert 1991; cf. phase transition neural dynamics). Further, children say, believe, and do all sorts of crazy or stupid or evil things—while people make fun of the bizarre things LLMs sometimes say, LLMs have nothing on the things little kids routinely say. (It makes one wonder how human brains reach the adult stage when they apparently must all go through the little kid stages, where if an adult said such things, they would be committed to a mental hospital or feared as a future serial killer.)
That is strikingly different from NNs. NNs demonstrate various kinds of phases within-training and emergence across-scaling, but many of the identified phase transitions, like ‘induction heads’, barely even change the training loss or more than a few benchmarks. In general, NN training appears much smoother than humans do, aside from changes in learning rates, particularly cyclical learning rates.
Slow Development
Why does intelligent development take so long?
Since larger models are more sample-efficient, and many intelligent animals (and definitely humans) are massively larger, why does it take so long?
One of the most universal observations in NN scaling is that, whatever the task, “larger parameter-count models are more sample-efficient”: more precisely, they reduce the loss more per data point than a smaller model does, and so, when the training is plotted comparing loss with data processed, larger models show a distinctly ‘steeper’ plunge than small models do.5 (This raises important questions that I’m not sure we know the answers to: “How sample-efficient are extremely, extremely large models—multiple orders of magnitude larger than we now train? How large do they have to become before they stop becoming more sample-efficient? If they do stop at some point, can better regularization fix them? Why do we not train these if we are data-limited in some regimes?”)
Since many animal brains appear to be so much vastly more parameterized than any DL model ever, and to have a large compute-budget and be bottlenecked by processing samples sequentially, we would naively expect an even steeper plunge. The expense of intelligence would be running the brain, not wallclock time.
But everywhere we look, like primates or cetaceans or elephants or birds, we see that intelligence tends to be associated with prolonged childhoods & longevity & low reproductive rates. These long developmental periods don’t seem explicable in purely metabolic terms—you cannot produce a college-level human intelligence by force-feeding toddlers, and while we do see changes like average of puberty dropping like a stone from contemporary caloric abundance, it doesn’t seem to accelerate brain development much.
And why does the development stop? Why don’t we go on being neuroplastic our whole lives but seem to coast after young adulthood, appearing fully functional even if we’ve almost stopped learning like many old people?
Human Slowness
Why can human development take so long for many things that other animals learn quickly?
For example, some prey animals are able to walk and even run within minutes or hours of birth (thanks in part to fetal training of motor control via twitching), while a human child is both physically & mentally incapable of crawling for >5,000 hours.
And where are the informative priors from evolution? Evolutionary psychology in particular suggested that many complicated social behaviors could be traced to genes and the human brain would be found to have “massive modularity”; this research program has failed completely.
Qualitative Differences in Training Dynamics
Where are the missing equivalents for humans of major NN dynamics like deep double descent (true even with 1,000-degree polynomial splines) & “epoch-wise double descent”, super-convergence, and grokking/catapult or patient-teachers?
And if there are none, why not?
Sleep
Tononi’s SHY is one of the most interesting theories attempting to explain sleep in general (rather than dreaming). In SHY, ‘sleep’ is forced by neural net local learning, where the neural net ‘weights’ increase in size over time as they learn; this is bad because it directly increases the energetic demands of firing a synapse, and because (just like an artificial NN), it makes it easier to memorize rather than generalize. The ‘weights’ need to all be multiplied by a fraction, to shrink them back down to normal energy consumption, and helping regularize the brain; but it can’t be done one by one, because they are all in use.
Sleep, then, is when the entire brain is taken offline, to shrink them all simultaneously. So, you can skip sleep for a while, and your brain keeps chugging along, weights growing slowly and becoming more and more inefficient, but it causes increasing problems, and eventually something diverges.
(Dreams, then are not essential; they are probably more of an enhancement for additional sample-efficiency, by doing a more extreme form of experience replay than can be done during waking.)
Savantism
Savants are often capable of remarkable feats of memory or perception, like making near-photorealistic drawings from a single glance at a photograph or memorizing books.
How is it possible for the (non-autistic) Kim Peek to “at age 16–20 months…memorize every book that was read to him. His parents moved Kim’s finger along each sentence being read.”? He could hardly have developed mnemonic palace techniques as an infant or other adult-like memory tricks like hours of repetition; instead, he seems to simply memorize books in a single epoch, so to speak, to parrot them back like an LLM.
Further, why do Peek & other savants seem to avoid childhood amnesia entirely (accounts emphasize that their memory tends to be lifelong, and Peek seems to remember books read as a young child), while also appearing to not go through childhood development stages?
Why do they usually suffer from severe neurological (associated with left hemisphere trauma) & physical problems and demonstrating remarkable deficits in abstract understanding and general intelligence? For example, “he [Peek] cannot, for example, explain many commonplace proverbs”—and even this level of abstraction understanding some proverbs is considered unusual for a savant! As far back as Down 1887, observers were struck by the contrast between the lifelong “verbal adhesion” of many savants and their understanding, or general intelligence. (Treffert & Wallace 2002 note that of the ~100 well-described cases, the highest IQ was 114.) Strikingly for LLM parallels, savants can also be capable of confabulation: Down 1887139ya describes an illiterate child who “take up a book…and improvise stories of all kinds with a great deal of skill, and in any variety, to suit the supposed tastes of his auditors”.6
In cases of mathematical calculation prodigies like calendar dates or taking roots, the underlying algorithm can be inferred to some degree, and appear to mix huge amounts of memorization & distilled intuition with explicit algorithms. These all appear to benefit from disabling higher levels of the brain.
Savants are often intellectually-disabled or otherwise abnormal. An important example in this context is Luria’s famous mnemonist, Solomon Shereshevsky (The Mind of a Mnemonist), who simply remembered everything he saw or heard: in part by using extensive visualization & mnemonic approaches like the “memory palace”, but his level of synesthesia also meant that everything was inextricably associated for him.7
He struggled with non-literal interpretations, reading, or even recognizing faces—he could remember exactly how a person’s face looked previously, of course, but then he could not generalize to how their face looked now. Why did he not benefit from the ability to recall every version of a person’s face, and gain uncanny super-recognizer powers? (See also hyperthymesia: people with HSAM do not have an intrinsically superior memory for learning, but for not forgetting—they seem to struggle to focus on daily life and cope with anxiety or memories of bad days.) He has what sounds like maladaptive daydreaming, and ultimately died of alcoholism.
On a similar note, one of John von Neumann’s famous party tricks was memorizing pages of books, and being able to recite books on command—just one of his many remarkable feats of cognition which have entered legend. But von Neumann was criticized for pedanticism, explicitness, inelegance, brute-force calculation, comfort with following complicated lines of reasoning by reliance on his photographic memory. Eugene Wigner, who knew Von Neumann so well, while describing his feats, struck a cautionary note: none of them were truly original, and Von Neumann himself reportedly said that “[I] will be forgotten while Kurt Gödel is remembered with Pythagoras”. Was von Neumann ever original & creative enough to be truly great like Pythagoras or Gödel? Was he held back by his great knowledge & calculating ability (if not exactly “library work”), and unable to drill to the essence of matters like a Grothendieck?
If we combine the ideas of lack of abstraction or later stages of childhood cognitive development, memorization, synesthesia (failing to separate modalities), injuries to the left hemisphere (particularly the temporal lobe), ability to confabulate, and general extreme imbalances in cognitive capabilities compared to normal humans, I am left with the impression that savants are the “LLM version” of humans. Savants are what can sometimes happen when a key node in the brain is knocked out and higher level processes become less effective, exposing more of the brain’s low-level raw intelligence, which is satisfied by simpler forms of learning like massive memorization.
Grokking
A good example is the much-discussed phenomenon of “grokking”. In the original grokking paper, a small NN is trained on a simple arithmetic problem; as expected, it quickly memorizes the training data, achieving 0% error while failing to generalize with ~100% error on the held-out data. What is more surprising is that after training for long enough, sometimes, apparently at random it gradually begins to improve on the held-out data despite the ~0% error. The odds of grokking are improved the more the NN is regularized—such as by oversampling implied data which cannot be easily memorized, and particularly by weight decay, which tries to shrink the size of the NN weights. (The larger the weights, the easier it is to encode information into them, and memorize the training data.) What is going on?
Mechanistically, what seems to happen when it groks: is that the initial hyperparameters are ‘poorly’ chosen and the NN quickly finds a nearby local optimum in the model-space (loss landscape), representing the ‘memorization’ solution, but that after enough training, the weight decay or other regularization makes the local optimum too ‘narrow’ to contain the NN forever, and then it randomly walks out, or is “catapulted” out of the original solution—and eventually reaches a new region of model-space.
This new region still has ~0% training error… but it corresponds to an algorithm completely different from memorization, which is simpler in an intrinsic sense than “memorizing all the training data” and so unsurprisingly, generalizes better, even in the presence of data error. (It could in fact be many algorithms—there are a surprising number of valid algorithms for that particular arithmetic problem.) And since it is simpler, there are many more possible models which all correspond to the same thing, so this new region of model-space is ‘wider’ than the memorization region, and so the model generally does not randomly walk out of that one; or if it does, it will probably find a third, bigger & better region, and so on.
This is a quite intriguing result, but it’s not obvious how to apply it to anything else, like a full-size LLM. It would doubtless be possible to train an LLM like GPT-3 to zero training error if it was trained for enough epochs—but it would be impossibly expensive, and then you would still have to train long enough past that to trigger “grokking”. Given that it can take hours to trigger “grokking” in even the tiniest possible toy dataset, it is impossible to do true “LLM grokking”. But the phenomen
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.