Thousand Faces of a Dry Leaf: Why the AI of 2026 Won’t Be a Bigger LLM
Next gen AI needs to start building world models instead of just writing about them
I try to walk at least 5 km each day. Usually in the forest behind our house with my wife. Sometimes we talk, sometimes we walk in silence, absorbing the changes in nature around us. A few weeks ago I was trying to explain to her how a simple thing like a dry leaf beneath our feet can be perceived in many different ways—and what that means if we think of future AI not only learning “about” the world from texts and other sources, but actually understanding the world as a complex network of “data points” like that leaf and the various ways they can relate to other data points and interact with them. That conversation made me put all those ideas together and turn it into a prediction for the year 2026. So, building on the conversation with my wife: what can a leaf be?

It is plant matter. It is “brown.” It is fuel, fertilizer, or waste. But it’s also a sign of decay, a symbol of impermanence, a memory cue, or a tiny emotional trigger you didn’t ask for. A leaf is not one thing; it’s a crossroads of meanings, measurements, and relationships happening at once.
As I said, today’s AI mostly meets that leaf the way a very gifted librarian meets the ocean: through descriptions, text, and second-hand reports. It can talk about the leaf brilliantly… but it doesn’t live in the same reality as the leaf.
So here’s my provocation for the near future:
I don’t think the best-performing AI model in 2026 will be “a bigger LLM.” I think it will be something that grows out of LLMs, but outgrows them by becoming part of a more complex system: a world model that stops merely reading about reality and starts predicting what reality will do next. Crucially, that means predicting not just what will happen in general, but what will happen if the AI takes a specific action – what Yann LeCun calls action-conditioned prediction. It’s the pivot from passive narration to active planning, turning our AI from a know-it-all storyteller into a decision-maker that can mentally test its moves before acting.
This isn’t a vague prophecy about “superintelligence soon.” It’s a concrete, falsifiable bet about what architecture will lead. At the same time, we should temper the optimism: even AI insiders urge caution on timelines. Andrej Karpathy, for instance, calls this coming period the “decade of agents” rather than the “year of the agent,” because today’s AI agents still “just don’t work” for real-world tasks. They lack key capabilities – they’re not “multimodal enough,” can’t reliably use tools or remember new information (no “continual learning”) – and “it will take about a decade to work through all those issues,” Karpathy predicts. Likewise, Ilya Sutskever suggests truly general AI systems could be 5 to 20 years away, not just a couple. In other words, this isn’t about magic fixes arriving overnight. It’s about a shift in approach that could set us on the right path.
Let’s build the idea from the ground up.
The efficiency twist: getting more intelligence per GPU
The popular story of AI progress has been simple: scale compute, scale data, scale parameters. And yes, that recipe worked for a while. But lately, something else has been happening in parallel: “more intelligence per GPU,” not just “more GPUs.”
DeepSeek-V3 is a good example of this new direction. It’s a Mixture-of-Experts model with an enormous total parameter count (around 671 billion), but it activates only a small fraction (~37 billion) of those parameters per token. In practice, this means DeepSeek-V3 achieved performance comparable to GPT-4 while using a fraction of the training compute. This provides concrete evidence that capability doesn’t have to scale linearly with cost – you can get more bang for each FLOP. The cautious takeaway is not “efficiency = destiny.” Greater efficiency alone doesn’t guarantee a new paradigm. But it does something extremely important: it buys runway. Every improvement in efficiency (recent algorithms have slashed compute requirements by roughly 4× per year) extends the “budget” available for trying new ideas. If we can reduce the compute spent just on compressing internet text, developers can redirect more of their budgets toward something that has barely been exploited at scale: non-text reality.
It’s worth noting that these breakthroughs aren’t all coming from the usual Silicon Valley suspects. Yann LeCun often points out that nobody has a monopoly on good ideas in AI – and recent events back him up. Teams around the world, many working openly, have been pushing the envelope. DeepSeek (even if used by the Chinese autocratic government) itself is a product of this broader, democratized wave of AI research. This matters because the next paradigm shift most probably won’t arrive as a single Big Tech product launch; it might emerge as a swarm of innovations from many contributors across the globe.
Which brings us to the next bottleneck.
The scaling trap: diminishing returns on text-only training
We’ve mostly used up the cheap part of the text-scaling curve. Squeezing ever more intelligence from internet text is hitting diminishing returns. Even Ilya Sutskever, reflecting on recent progress, describes the past few years as an “age of scaling,” and says the next era will be an “age of research” — because “just do 100× more” is no longer the easy or cost-effective strategy it once was. In fact, he has put his money (and reputation) where his mouth is on this point: in 2024 he left OpenAI to co-found a new venture called Safe Superintelligence Inc. (SSI), explicitly aiming to develop novel AI architectures beyond the “just make it bigger” approach. The existence of SSI — with massive funding to experiment at full throttle — is a sign of the times. We now have the computing power (and the investor backing) to treat these new ideas not as academic side projects, but as serious contenders at scale. Based on Sutskaver’s remarks on the Dwarkesh Podcast I expect to see some SSI product launched in 2026.
And here’s the dirty secret hiding in plain sight: the internet is not a curated library; it’s a chaotic bazaar of information. Andrej Karpathy puts it bluntly: if you actually look at random documents in the frontier training datasets, a lot of it is “slop and garbage.” In other words, enormous model capacity gets burned memorizing noise and redundancies rather than learning deep structure. So if you’ve felt like the jump from the previous model to the newest model sometimes seems more cosmetic than groundbreaking, you’re not imagining it. (Even if headline benchmarks improve, much of the gain comes from brute-force scale on mediocre data.)
Indeed, these systems often excel on benchmarks while struggling in real life. Sutskever himself has expressed confusion about why LLMs “look so good on benchmarks but fail so badly in practical applications,” noting that their generalization to the real world is still “inadequate.” Text alone is a strange diet for an intellect. Humans don’t learn what a “leaf” is by reading ten thousand leaf-related paragraphs. We learn what a leaf is by seeing leaves, touching them, stepping on them, smelling them, watching them rot, seeing what happens to soil underneath, noticing how leaves change across seasons. In short, we learn through multimodal experience.
So if we want AI systems that genuinely model the world, at some point we have to feed them the world – not just more text about the world. This shift is already underway in the industry: OpenAI’s GPT-4 was introduced as a multimodal model (it can accept images as well as text), and Google’s upcoming Gemini is reportedly multimodal from the ground up. Expanding beyond text isn’t a speculative idea for “later”; it’s happening now because it needs to happen.
The core idea: LLMs don’t disappear — they get demoted (and promoted)
Here’s the move I’m betting on: keep LLMs, but stop treating them as the central “physics engine” of intelligence. Instead, split the cognitive workload into two distinct jobs that form a pipeline:
Job A: LLMs as meaning expanders. An LLM’s job in this pipeline is not to be the arbiter of truth, but to serve as a kind of structured imagination engine. Its guiding question is, “In what all ways can this thing be?” When you present it with a concept, it generates hypotheses, interpretations, possibilities. A leaf can be plant matter, fuel, fertilizer, waste… but also a symbol, a trigger, a poetic image, a warning sign, a measurement unit, a piece of evidence, an ingredient in a story your brain tells itself. That sounds fluffy, but it’s brutally practical: the LLM expands the hypothesis space. It enumerates the candidate meanings or potential interpretations of a thing or situation.
Job B: Multimodal world models as reality fillers. This part takes the hypotheses or prompts from the LLM and fills them with empirical content. Its guiding question is, “What can we find out about this thing — from reality itself?” That means grounding those ideas in data from the world: not only text, but also video, audio, sensor readings, scientific data, the works. If you want a name for the container holding all this knowledge, think of it as a multimodal ontology of observable reality. Not “reality as it is in itself” (none of us get that), but reality as it presents itself to observers – human and machine – through various phenomena. In that ontology, every “thing” is like a node with many layers and many possible relations.
For example, take the concept of a dog. In a pure text model today, the concept “dog” is tightly bound to word associations (barks, fetch, leash, “good boy,” etc.). In a world model, “dog” becomes a richly connected node inside a web of relations:
-
Physical relations: how a dog moves on grass, how it swims in water, how it reacts to heat or cold.
-
Biological relations: the dog as a mammal, its typical vital signs, its genetic relationship to wolves, health indicators for dogs.
-
Emotional relations: a dog might be a stress reducer for one person or a source of fear for another.
-
Cultural relations: dogs as taboo in one culture but family members in another; what a dog symbolizes in art or myth.
-
Policy relations: dogs as factors in interventions (e.g. therapy dogs in hospitals, or the legal status of dogs in various jurisdictions).
Once the system can represent and update these kinds of relations through experience, the interesting question shifts. It’s no longer “Can it write a plausible paragraph about dogs?” It becomes “Can it predict what will actually happen if we make a change in the world involving a dog (or any other entity)?” For instance: “If a new virus starts spreading among dogs, how will it affect humans, ecosystems, or policies?” At that point, intelligence starts to look less like fancy autocomplete and more like navigation or simulation. We’re asking the AI to chart paths through reality, not just through text.
One important practical point: a system like this needs to handle both measurement and meaning. Notice in the leaf example that some attributes were literal (e.g. a leaf’s moisture level as a “measurement unit”) and some were figurative or contextual (a leaf as a “symbol of decay”). A robust world model would need to encode both types of information and know the difference. It must link concrete, quantitative observations (what you can measure or detect about the leaf) with qualitative, contextual knowledge (what the leaf means in various narratives or systems) without mixing them up. In other terms, that might mean having structures that separately store empirical sensor data versus symbolic or relational data, but allow them to interact. The ontology has to accommodate hard measurements and squishy meanings.
Crucially, in this two-job architecture the LLM doesn’t vanish – it gets demoted and reassigned. It’s no longer the all-powerful brain, but it’s still an indispensable component: an interface and idea generator. The LLM becomes the explainer, the hypothesis drafter, the translator between the user’s intent and the system’s knowledge. In other words, LLMs become the front-end, and the world-model (with whatever planning modules it contains) becomes the back-end “mind.” LLMs are extremely good at one thing – producing fluent, contextually relevant text – and we’ll continue to use that strength. They’ll be the conversational interface and the imaginer of possibilities. But the heavy lifting of deciding what is true, what is relevant, and what will actually happen next in the world will increasingly migrate to the multimodal predictive systems that are grounded in data.
That raises a question: if not an LLM, what is at the core of such an AI system? What’s doing the “thinking” after the LLM has done its imagining? The answer is what you might call a new cognitive core: essentially, the world model itself (or a collection of models) that can reason over the knowledge it has, simulate outcomes, and plan actions. This core might still be a large neural network model, perhaps on the order of billions of parameters, but its architecture and training would be very different from today’s chatbots. Instead of predicting the next token in text, it would learn to predict state trajectories of the world – the kind of model that, given the current state of everything it has seen, can suggest what’s likely to happen next, or what would happen if you intervened in a certain way. You can think of it as an abstract simulator or strategist.
Interestingly, one way to make this cognitive core more intelligent is to actually give it less baked-in knowledge. Andrej Karpathy argues that today’s models are too much like overstuffed encyclopedias – they memorized so many facts that they can just regurgitate answers instead of truly reasoning. His counterintuitive suggestion is to strip out most of that static knowledge and only keep the reusable “algorithms for thought” inside the core model. Any time the AI needs a specific fact, it should look it up externally (in a database or via a tool); meanwhile, the core focuses on strategy, abstraction, and problem-solving. In practice, this means building a smaller core model that isn’t a walking Wikipedia, but rather a clever thinker that knows how to learn and query for information as needed. A smart human isn’t the one who can recite every trivia fact – it’s the one who knows how to ask the right questions, notice what they don’t know, find answers, and then reason effectively from that evidence. That’s the kind of intelligence we want to cultivate in the AI’s core.
Yann LeCun and others have sketched out blueprints for such systems: imagine something like a game engine for reality, where the AI can mentally test hypotheses (with the LLM acting as the narrator and interpreter). In a sense, we’re inserting an entire extra layer of cognition: System-2 style reasoning and modeling, sitting on top of the fast System-1-style pattern matching that LLMs excel at.
From plausibility to prediction: why JEPA-style thinking matters
All of this points to a shift in priorities. It’s a move from generating plausible sequences to making accurate predictions about the world. Yann LeCun has been hammering on a simple complaint for years: LLMs are powerful, but they lack a true internal world model for planning and reasoning. You can bolt on tools, memory, retrieval, etc., but if the core mechanism remains “predict the next likely word,” you’re fighting the wrong battle. The system will still tend to say whatever sounds plausible rather than maintain a coherent model of how things actually work.
This is where LeCun’s ideas like JEPA (Joint Embedding Predictive Architecture) come into play. JEPA is not presented as “the one true answer,” but it’s an example of a different center of gravity for AI. In a JEPA (and related approaches), the AI doesn’t try to generate full data (like an image or paragraph) from scratch; instead it learns by predicting missing information in an abstract representation of the data. For example, Meta AI’s recent V-JEPA research project is framed as a step toward learning video by predicting missing parts of a video – but crucially, it does this in an abstract latent space rather than at the pixel level. The logic, as the authors put it, is that the world is full of unpredictable noise; intelligence is about learning what can be predicted, and doing so at the right level of abstraction.
LeCun argues that trying to model every little detail of the world is a fool’s errand – “mathematically intractable,” as he’s said. We don’t need an AI that simulates every photon and air molecule; we need one that captures the regularities of the world and ignores the irrelevancies. In practice, that means the AI’s internal model should be abstract. It might represent the world in terms of objects, relations, and dynamics rather than raw pixels or raw text. By focusing on predicting the meaningful aspects of a situation (and by not sweating the small random stuff), a JEPA-style system inherently sidesteps a lot of the nonsense that purely generative models can produce. It’s not freeform hallucinating; it’s constrained to fill in blanks that make sense given what it has seen.
To make this concrete: if you spill a glass of water on the floor, a purely generative model (like an LLM or a diffusion model) might describe in vivid detail how the water splashes and the exact ripple patterns – it’s trying to imagine the scene in full. A world model, on the other hand, will zero in on what actually matters next: the floor will get wet and slippery, which changes what actions are safe to take. It learns the consequences and correlations that affect planning (wet floor -> risk of slipping). The generative approach produces a plausible description; the predictive world-model approach draws out the implications for action. This illustrates why I keep saying “LLMs will be outgrown,” not simply “LLMs will be replaced.” We will still use LLMs, but their role will change. They’ll move seats in the architecture’s “org chart,” so to speak. They’ll be the interface, the explainer, the brainstormer – the meaning expander. The thinking will happen in these new predictive, multimodal models that are optimized for what actually happens next in the world, not just for what sounds right.
There’s also a fundamental shift in how such a system learns. Today, many advanced models rely on reinforcement learning fine-tuning: the AI generates a long output and then gets a single grade at the end (“good” or “bad”) to update its behavior. Karpathy memorably calls this “sucking supervision through a straw” – the model does an enormous amount of work, and only then receives one tiny sip of feedback. By contrast, a world model can be trained with richer, continuous feedback via an internal value function. Think of it like having a built-in coach: the AI doesn’t have to wait until the end of a trial to learn if it’s on the right track; it can get hints along the way. Ilya Sutskever notes that using value functions lets you “short-circuit the wait until the very end” – if a plan is clearly going awry, the system can figure that out early and adjust, rather than driving off a cliff just because the reward hasn’t arrived yet. This approach is far more compute-efficient (you learn from many intermediate signals, not just one final outcome) and far more robust (you avoid those lucky, misleading successes where the end result looks okay even though the reasoning was flawed). In plain terms, it’s the difference between a student who only finds out their final exam score versus one who gets corrections on each homework assignment. The latter is going to learn more reliably and waste a lot less effort.
And yes, this shift in emphasis is already visible in research buzzwords: you hear terms like “world models,” “predictive representations,” “planning,” “causal modeling,” “hierarchical abstraction” gaining traction in papers and conference talks. The field’s language is beginning to move away from just “generative everything” toward these ideas of modeling the world at some deeper level.
The bottleneck nobody can avoid: continual learning
Even a gorgeous world model, however, would be like a savant kid with amnesia if it can’t learn continuously on the job. This is a hard truth lurking at the edge of all these discussions. Today’s most advanced models are still largely frozen after training; they don’t autonomously accumulate new understanding from each task or interaction. They don’t grow in the way a human employee would, by seeing new problems and slightly improving each time. They don’t sleep and consolidate experiences into long-term knowledge. As AI commentator and podcaster Dwarkesh Patel has repeatedly pointed out, continual learning remains a major unsolved bottleneck: our models can be trained on heaps of data, but they don’t live and learn in the world the way humans (or animals) do. Andrej Karpathy underscores the same point – he notes that you can’t simply tell current AI agents something and expect them to remember it tomorrow; they have no persistent memory of new events. This lack of lifelong learning is one reason he believes human-level AI is not just a year or two away but likely a multi-year (or multi-decade) project.
Why does continual learning matter so much? Because if you can deploy a capable system into the world and let it learn from its own experience – not just via user prompts or manual updates, but by actually updating its weights or internal knowledge in real-time – you get compounding returns. The system’s performance can keep improving without a complete retraining run. Imagine not one monolithic model being trained in isolation, but many instances of an AI agent spreading out, each learning from its local environment or users, and then merging those learnings back into a shared core model. If that ever becomes robust and safe, you don’t need to keep scaling up the training compute in the same brute-force way; the model gets better by doing its job, by living. You’d have a different growth curve for AI capabilities: one driven by experience and deployment, rather than just by offline dataset size.
That, of course, is also where things get nerve-wracking for many observers. A self-improving, self-learning AI in the wild is the classic recipe for “AI takeoff” speculation – and even in more mundane scenarios, it raises big engineering and safety challenges. So when I describe how transformative continual learning could be, I’m describing feasibility and potential, not necessarily endorsing that we unleash it without safeguards. It’s the feasibility that’s exciting: this is how we outgrow the current paradigm, if we can figure out how to do it responsibly.
The blunt reality is that we don’t yet know how to do continual learning at scale without the system collapsing (e.g. catastrophically forgetting old knowledge or spiraling out of control). Many smart people are actively researching this. It might turn out that robust continual learning won’t be achieved in the near term. If it stays stuck, we’ll continue to get impressive one-off demos that never turn into the kind of compounding competence I described. In fact, this is one of those make-or-break issues that will determine how quickly the “world model” vision can progress. If continuous learning proves intractable for a long time, then the era of just “bigger LLMs + good prompting” might last longer than I’m betting.
An “epistemic ladder” to avoid a giant delusion machine
One critique often raised (rightly) against these grand visions of AI world models is: how do we ensure this doesn’t become one giant hallucination or garbage-in-garbage-out mess? Emily M. Bender and other skeptics have argued that today’s large models often sound authoritative without any grounded understanding, and a multimodal mega-model could just as easily spew nonsense with great confidence. If you naively let an AI ingest “all of the internet and more” and connect every modality, you might just amplify the misinformation and bias along with the insight. Decades of AI research on knowledge representation show that encoding “everything” is virtually impossible and fraught with peril. We have to be very careful to avoid building a magnificent delusion engine. There’s even a term for a related failure mode: “model collapse.” This is what happens when an AI trains too much on its own synthesized outputs. Over time, it’s like making photocopies of photocopies – the signal degrades and the system starts to lose touch with reality. If future models learn primarily from data generated by older models, they risk spiraling into a closed loop of self-reinforcement. Karpathy’s warning here is that these AIs need plenty of fresh, unpredictable input (entropy!) from the real world to stay sane. Without new noise and novelty – like a person who never leaves an echo chamber – the AI’s “mental world” can become coherent but delusional, internally consistent but detached from reality. In other words, the system needs to live in a dynamic environment, not just read its own fiction. It should become less like a book on a shelf and more like a creature in a habitat, constantly corrected and surprised by the real world.
One reason I favor the world-model approach is that, in principle, it forces the system to continually check its beliefs against reality. But that only happens if we design the training and deployment with rigorous feedback loops. We need what you might call an epistemic ladder – layers of verification that the AI must climb before it can trust its conclusions (and before we trust its conclusions). If I were building this system, I’d enforce something like the following ladder of checks:
Step 1: Known knowns. Test the AI’s predictions against established scientific knowledge and well-replicated results. Before trusting it on new discoveries, make sure it can at least match what’s already in textbooks and databases. If the model’s world knowledge says one thing but decades of experiments say another, the model is wrong – and it needs to adjust. This step roots the AI in the solid ground of facts we are confident about.
Step 2: Unknown knowns. These are phenomena we can observe but don’t fully understand yet (we see what happens, but not why). Here the AI can generate hypotheses, but we treat them as provisional. The system proposes explanations or causal models for some puzzling observation, and then we design proper experiments or gather new data to test those hypotheses. The AI is effectively using its world model to bridge gaps in science or knowledge – but under human guidance and scientific protocol, so we can verify which of its ideas hold water.
Step 3: Unknown unknowns. Occasionally, the system might flag something entirely unexpected – a wild prediction or a solution that seems to come out of left field. These are the most tenuous and risky, so they should be explored only under strict conditions: small-scale trials, controlled environments, reversible experiments, heavy supervision. The point is to give the AI room to be creative and possibly point us to new discoveries, but not to let it conflate speculation with established fact. This is the difference between “a model that’s confident” and “a model that earns confidence.” The AI has to climb the ladder: from basics to frontier ideas to truly novel propositions, with testing at each stage.
In this framing, truth becomes a pragmatic concept: truth is what reliably predicts and survives contact with reality. An AI world model shouldn’t be considered trustworthy just because it’s big or trained on lots of data; it should earn trust by making predictions that we can validate in the real world. This is how we avoid fooling ourselves. If we skip these steps, we might end up with an AI that generates elaborate, interconnected falsehoods – an artificial but convincing alternate reality. By imposing an epistemic ladder, we force the system to tether itself to the real world at every level. It’s how a multimodal world model can be kept honest, essentially incorporating the scientific method into its development process.
How we’ll know if this bet fails (by 2026)
A prediction is only useful if it can be proven wrong. So, to keep myself honest, here are a few concrete ways my bet could fail – and I’ll happily eat my words if I see these:
-
If the best AI systems of 2026 are still primarily LLM-centric – and they somehow achieve major leaps in planning, reasoning, and reliability without leaning on any kind of new world-model architecture. In that case, I’d have underestimated how far the “LLM + plugins” paradigm can go. (It would surprise me, because the limitations of text-trained models are well documented, but surprise is always possible in this field.)
-
If the whole multimodal ontology approach turns out to collapse under engineering reality. Maybe it’s too complex to build or too brittle in practice. Maybe maintaining and updating this giant knowledge web becomes a nightmare. Maybe we can’t effectively verify or validate such a system’s outputs, and it just proves impractical compared to simpler alternatives. In short, it could be that even though richer world models are conceptually appealing, the industry finds it faster and more reliable to just keep scaling up language models plus heuristics. I consider this a real possibility – many promising AI ideas have hit a wall in the past due to “messy” implementation details.
-
If continual learning remains unsolved – so we keep getting impressive static models (a great demo every year) but never the systems that improve through usage. This would mean the compounding advantage I hoped for from deployed learning never materializes, and progress continues to rely on big offline training runs. That could slow down the trajectory I’m imagining, perhaps keeping LLM-centric approaches on top for longer.
If any of those outcomes dominate, then my thesis is wrong (or at least premature). In that scenario, “bigger LLMs + clever toolchains” might remain the winning recipe for AI longer than I’m predicting. Essentially, the current paradigm would keep scaling (with diminishing returns but no viable alternative that beats it in the short term). Many experts, as noted, actually lean toward this cautious view: they suspect we’re stuck with what we have for a while, and that true “agentic” AI will require a lot more research breakthroughs. They might turn out to be right.
Final image to carry with you: the librarian steps outside
Right now, an LLM is like a profoundly well-read librarian who has never left the archive. They can quote every travel book ever written, but they don’t know how a heavy wooden door feels when you try to push it open, because they’ve never been outside. The next era of AI is that librarian finally stepping into the sunlight. Not to become whimsical or poetic, but to become grounded and predictive.
My bet is simple: by 2026, the most interesting and capable AI systems won’t feel like “chatbots with more parameters.” They’ll feel like early prototypes of something else entirely – systems where an LLM is just one component inside a larger architecture that models the world, tests hypotheses, updates its beliefs, and maybe even learns continuously. They will navigate multi-modal data and real-world tasks with a competence that a text-bound AI could never achieve. We’ll start asking them not just to write essays or code, but to solve problems in the world – and they’ll actually do it (albeit clumsily at first, like a child learning).
If I’m wrong, at least I’m wrong in a way that can be checked fairly soon. And if I’m right… then we as a community should probably stop arguing only about prompt engineering tricks and instead start arguing about what kind of world models we’re building – and what we’re willing to let them do. Because the moment the librarian steps outside, the game changes.
In any case, hold me to my bet. The proof will be in the systems we actually build over the next few years. Let’s see what the future brings.
Sources:
-
Dwarkesh Podcast:
- interviews with Andrej Karpathy and Illya Sutskever, article about continual learning.
-
Big Technology Podcast:
- interview with Yann LeCun
-
LeCun, Yann. A Path Towards Autonomous Machine Intelligence. OpenReview (2022)
-
Assran, M. et al. (Meta AI). Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA). arXiv (2023)
-
Meta AI / LeCun-FAIR line of work. V-JEPA 2 (video JEPA / world-model direction referenced in the text). arXiv (2025)
-
Bender, Emily M., Gebru, Timnit, McMillan-Major, Angelina, & Shmitchell, Shmargaret. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? FAccT (2021)
-
Shumailov, Ilia et al. The Curse of Recursion: Training on Generated Data Makes Models Forget (commonly cited in “model collapse” discussions; this backs the “photocopies of photocopies” idea my text references). arXiv (2023)
-
DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv (2024)
-
OpenAI. GPT-4 Technical Report. arXiv (2023)
-
Google / Google DeepMind. Gemini: A Family of Highly Capable Multimodal Models. arXiv (2023)