Part 2 of 3. Previously: Your Brain Is a Rendering Engine. So Is Every LLM. Next: Two Render Layers Deep Before You've Even Formed an Opinion (coming soon).

I got into an argument recently about whether large language models will eventually become conscious. Not a productive argument. The kind where both people are sure the other person is missing something obvious.

His position: more compute, larger models, better cognition, eventual mind. Scale it up enough and something wakes up in there.

I kept saying no, and I kept not being able to explain why in a way that landed. Went home, sat with it for a while.

What a sunset has to do with it

Part 1 talked about how the same input produces different outputs depending on the renderer. The sunset version: two people, same light, same vibration frequency. One feels calm. The other feels sick because the last sunset like this, someone was leaving.

Nobody would find that mysterious if it were two LLMs producing different outputs from the same prompt. You'd just say: different weights. Somehow when it's brains, we forget this.

But there's a part I skipped in Part 1. It's not just that the rendering is different. The rendering is stacked. Multiple layers, each one adding its own distortion.

The stack between you and the world

Start at the bottom. Your eyes only pick up a sliver of the electromagnetic spectrum. Ears: 20Hz to 20kHz. Everything outside that window, your brain has literally zero training data for. It's not that you see it wrong. You don't see it at all. The rendering engine reports 404.

Within the frequencies you can detect, your brain has spent decades building associations. Tree plus wind plus childhood memory of a backyard. It doesn't show you "a pattern of vibrations." It shows you "a tree," fully rendered, emotions included. That's your pre-training.

Then there's the emotional layer. Same tree, different day, different mood, different render. Your brain doesn't separate the signal from how you felt the last time you saw something similar.

And on top of all that, years of people telling you what things mean. What's appropriate to feel. History classes that teach narratives, not data. Moral frameworks. Social norms. This layer doesn't help you perceive reality more accurately. It teaches you how to render "correctly" per your culture.

That last one is basically RLHF. Someone tells the model "this answer is good, that answer is bad," and the model learns to match human preferences. Non-science education does the same thing to brains. It doesn't teach you what the world is. It teaches you how you're supposed to render it.

Science education at least tries to go the other direction. Math describes vibrations directly. Physics gives you equations for things your senses can't reach. Route around the renderer.

Bigger screen, same problem

Improve a screen from 480p to 4K to 8K. Better picture every time. Sharper edges, more detail. At no point does the screen become a person watching the screen.

You can make the rendering layer arbitrarily good. You can make it produce outputs that are indistinguishable from what a conscious being would produce. It's still a renderer. Whether there's an audience watching the screen, that's a different question, and making the screen bigger doesn't answer it.

I genuinely don't know whether compute can ever produce consciousness. I'm not making the strong claim. What I'm saying is that the argument "better rendering therefore eventual mind" has a gap in it, and a lot of people in tech seem to be driving through it without noticing.

Quantum mechanics got close

The math can describe what's happening outside our sensor window. Superposition, entanglement, wave functions that don't map to anything you can visualize. The equations work.

Nobody intuitively gets them, though. Not because the math is wrong. Because your brain has no training data for those frequencies. You can describe ultraviolet mathematically. You can't experience it. Your rendering engine has no weights for that input.

Quantum mechanics proved there's more to the world than what our sensors pick up. Then it ran into the fact that the tool doing the proving is itself a constrained renderer. The math can describe the territory. The brain can't render it.

Philosophy of mind is not my day job, for what that's worth.

If you build software with these things

If you buy any of this, even partially, it shifts how you think about systems that use language models. You stop expecting the model to "understand" things. You start treating it as a rendering layer that produces useful local outputs from high-dimensional input. That's not less valuable. It's more honest.

The fluency trap is real. A model can produce a perfectly articulate explanation of a concept it has no experiential basis for. So can a human who read about it on Wikipedia. We know fluency isn't understanding. We keep forgetting.

The architectural risk is worse. If your language model is generating natural language that then controls a critical pipeline, and there's no hard boundary between "the model's interpretation" and "the system's actual state," you've wired the renderer directly into the physics engine. That breaks in ways that are hard to debug because the output looks reasonable.

The "will it wake up" question honestly matters less than people think. The risk isn't consciousness. The risk is that your team treats a polished renderer as a source of truth because the rendering is good enough to fool the people watching.

The consciousness question is genuinely open. I'm not going to pretend I've settled it. "More compute" might get you a better renderer. Whether it gets you a mind is a different question entirely, and I'm not confident most people making that argument have noticed they're asking two different things.

Part 3 gets into what happens when you try to build systems where the rendering layer can actively hurt you. Like, say, anything that touches real money.

Find me on StratCraft | GitHub