There's a line from a recent David Lin conversation with Clem Chambers that stuck with me: the real money in AI isn't the chatbots. It's the physical layer. Chips, servers, drives, cables, cooling, power, buildings, and the banks writing the cheques for all of it.

You can't buy OpenAI or Anthropic directly. You can buy the people selling them electricity and silicon.

That framing is easy to nod along to and hard to act on, because most people arguing about AI hardware are actually arguing about one company's stock price. So I want to go a level down. Not "buy the picks and shovels" as a slogan, but: what is the actual bottleneck inside the box, who is attacking it from which angle, and what does a founder do with that information on a Tuesday morning.

I'll use Groq as the worked example, because it's the clearest case of a company that ignored the obvious problem and went after a different one.

Part 1: Why the physical layer argument holds

The podcast made four points about infrastructure that I think are correct, and one that I'd push back on.

Demand for compute has no natural ceiling. This is underrated. A social network saturates when everyone has an account. Compute doesn't saturate, because every improvement in model quality creates new things worth computing. Reasoning models burn 10-50x the tokens of a single-shot answer for the same question. Agents burn more again, because one user action becomes fifteen chained model calls. The unit of demand keeps getting bigger even when the user count stays flat.

There's no prize for second place. Correct at the lab level, and it's why capex runs hot for years rather than months. If you believe you might have the best model in eighteen months, underspending now is how you guarantee you won't.

Electricity is the binding constraint. This one I'd put first, not fourth. You can order more chips. You cannot order more grid. A gigawatt of new generation plus interconnect is a multi-year permitting problem in the US, and that timeline doesn't compress with money. Which is why "performance per watt" stopped being a spec-sheet vanity metric and became a business constraint. If your data centre is power-capped rather than budget-capped, the chip that does more inference per watt literally lets you serve more customers from the same building.

Where I'd push back: "spread across the AI supply chain, hold ten years, ignore the news" treats the supply chain as one thing. It isn't. Power generation, transmission, and cooling are genuinely long-cycle businesses with physical moats. Silicon is a knife fight where architectures get obsoleted in 24-month cycles. Holding both for ten years with the same conviction is treating two different risk profiles as one bet.

Which brings me to the interesting part.

Part 2: The bottleneck outside infra

Running a language model is two completely different jobs wearing one name.

Job one is prefill: reading your prompt. The model processes all your input tokens at once, in parallel. This is compute-heavy. Lots of maths, all of it independent, all of it able to happen simultaneously. GPUs are exceptional at this. It's essentially what they were built for.

Job two is decode: writing the answer. The model produces one token, looks at everything it has written so far, then produces the next one. It cannot parallelise. Token 50 needs token 49 to exist first.

And here's the catch that makes decode a hardware problem rather than a maths problem: to generate each single token, the chip has to read the entire model's weights out of memory. All of them. For a 70-billion-parameter model, that's roughly 140GB moved from memory to compute, per token.

So the compute units sit mostly idle, waiting. The pipe from memory is the ceiling. People in the field call this the memory wall.

An H100 has around 3.35 TB/s of memory bandwidth. Do the arithmetic on 140GB per token and you can see why GPU decode speeds land where they land. The chip isn't thinking slowly. It's queuing.

Every serious inference chip company is an answer to this one question: how do you stop the weights from having to travel so far?

Part 3: How Groq works...

Groq was founded in 2016 by Jonathan Ross, who led the team that built Google's first TPU. He left with a specific prediction: memory bandwidth, not compute, would become the thing that decides AI performance. He was early by about six years.

Their answer has three ideas. All three are simple once you see them.

Idea one: don't fetch the weights from anywhere

A GPU keeps model weights in HBM, which is memory that sits next to the processor and gets read across a bus. Fast, but it's still a journey.

Groq's LPU puts hundreds of megabytes of SRAM directly on the die and uses it as primary weight storage rather than as a cache. The weights don't live somewhere else and get fetched. They live on the chip, already there, permanently.

The Groq 3 generation carries around 500MB of on-chip SRAM with roughly 150 TB/s of bandwidth, against the H100's 3.35 TB/s. That's not a 10% improvement. It's a different order of magnitude, and it's the whole ballgame for decode.

The analogy I use with non-technical people: a GPU is a chef with a huge, well-stocked walk-in freezer down the hall. Every dish, they walk to the freezer and back. Fast walker, big freezer, but the hallway is the constraint. The LPU is a chef whose ingredients are all pre-laid out on the counter within arm's reach. Tiny counter. Nothing in the freezer. But no walking.

Idea two: schedule everything before you start

A GPU decides at runtime what happens next. Threads get scheduled, caches hit or miss, memory access patterns vary. This is flexible and it's why GPUs run anything. It also means the exact same request takes 40ms sometimes and 95ms other times.

The LPU throws that out. There are no caches, no dynamic scheduling, no out-of-order execution. A compiler works out the entire schedule ahead of time, down to which data arrives at which unit on which clock cycle. The chip then just executes the score.

Groq describes this as a programmable assembly line, and the metaphor is exact. On an assembly line nobody decides at 2pm what station three does next. It was decided when the line was designed, and the part arrives at station three at the moment station three is ready for it.

Notably, Groq wrote the compiler before they designed the chip. The hardware was shaped to fit the software rather than the other way round.

Idea three: make many chips behave like one

Because 500MB is small, a real model doesn't fit on a single LPU. It gets split across many chips, which then have to coordinate.

On GPU clusters this coordination is a well-known source of unpredictable delay. Switches, routing, packets arriving in odd orders. Groq extends the static schedule across the whole rack. The compiler knows when data will arrive at each chip, so chips connect directly with no switches deciding anything at runtime. Hundreds of chips act as one large processor.

What all this actually buys you

Speed, obviously. Roughly 5-10x GPU decode throughput depending on model and generation.

Tail latency, which matters more and gets discussed less. Deterministic execution means the same work takes the same time every time. On a GPU your median response might be 200ms and your 99th percentile 900ms. When one user action fans out into fifteen sequential model calls, you're not exposed to the median. You're exposed to the worst one in the chain, fifteen times over. This is why agent products feel unpredictably slow in production while feeling fine in demos.

Energy per token. No HBM stacks, no complex cooling. Groq's racks are air-cooled by design. Circle back to the electricity bottleneck from part one and you can see why this is a commercial argument, not an environmental one.

And what it costs you

I'd rather you hear this from me than find out during a migration.

It can't train. Training needs backpropagation, gradient accumulation, optimiser states. Wrong shape entirely for a statically-scheduled inference pipeline. You still need GPUs upstream.

Compile time is real. Solving that scheduling puzzle for a new model takes serious time. Great for a model you'll serve for six months. Painful for research where the architecture changes weekly.

Small memory per chip means you need lots of chips, which is why Groq serves a curated catalogue of open models rather than letting you upload anything.

Prefill isn't its strength. Long-context, document-heavy work leans on the phase GPUs win. The emerging pattern is hybrid: GPU does prefill, hands the KV cache to the LPU, LPU does decode. That works but it adds real infrastructure complexity.

Part 4: The competitive picture

Five different bets on the same problem. I've kept this to what changes a decision.

Nvidia GPU (H200/B200) Groq LPU Cerebras WSE SambaNova RDU Google TPU / AWS Trainium
Core bet General-purpose parallel compute wins everything Kill the memory wall with on-chip SRAM + static scheduling One wafer-sized chip removes chip-to-chip overhead Reconfigurable dataflow, sits between GPU and fixed ASIC Vertical integration; own the silicon under your own cloud
Train or infer Both Inference only Both Both Both
Speed vs GPU Baseline ~5-10x on decode ~15x claimed ~5x claimed (SN50) Competitive, varies
Where it wins Anything and everything; unmatched ecosystem Real-time decode, agent loops, voice Very large models, long context, HPC On-prem enterprise, very large models Cost at hyperscale inside one cloud
Where it loses Cost per token; latency variance No training, weak prefill, curated models only Cost, footprint, niche software Smaller ecosystem Locked to GCP / AWS
How you access it Buy, or any cloud API (GroqCloud) or on-prem racks API or on-prem Mostly on-prem Only that cloud
Data residency Fully yours if on-prem Vendor infra unless you buy racks Either On-prem strength Vendor cloud
Switching cost — Near zero, OpenAI-compatible API Low, API-based High, on-prem commitment Medium, cloud lock-in
Company risk Investment grade Independent post-Nvidia deal, $1.75B raised Public (NASDAQ: CBRS) Private, $1.5B raised Backed by trillion-dollar parents

Two patterns worth pulling out of that table.

Nobody is beating Nvidia at being Nvidia. Every challenger wins a specific dimension and loses the general case. Nvidia's roughly 80% share isn't inertia. It's that one architecture credibly does training and inference and vision and research, with a software ecosystem fifteen years deep. The challengers aren't trying to replace that. They're trying to peel off the workload where specialisation pays.

Inference is where challengers can actually win, because inference rewards specialisation and grows with usage rather than with lab budgets. Training demand is concentrated in maybe a dozen buyers. Inference demand is every company shipping a feature.

Part 5: The open-source angle, which is the part founders should care about

This is the connection most coverage misses.

Groq doesn't serve Claude or GPT. It can't. Those weights aren't available. The catalogue is open-weight models: the GPT-OSS family, Qwen, Llama, Whisper for speech.

That's usually described as a limitation. I'd argue it's the entire strategic point.

The economics of open models only work if inference is cheap and fast. Open weights are free to download and expensive to serve. That's why "just self-host Llama" is advice that sounds better than it works. The moment someone specialises the hardware for exactly those models, the calculus shifts. Prices land around $0.05 per million input tokens on smaller models, against a couple of dollars for frontier proprietary ones. Meta partnered with Groq for the official Llama API. That's an ecosystem forming, not a vendor selling boxes.

Where this leaves a founder: the honest answer is not "switch to Groq." It's tiering.

Most products have a boring tier and a hard tier, and most teams route everything to the hard tier because it's one API key. Classification, routing, extraction, summarisation, reformatting — that's the boring tier, and an 8B open model on fast cheap hardware handles it. Genuine multi-step reasoning, ambiguous judgement, anything customer-facing where quality is the product — that's the hard tier, and a frontier model earns its price.

Two providers, a router in between. Because both speak an OpenAI-compatible API, this is a config change, not a rewrite. That's the whole migration.

And there's a category where the speed isn't an optimisation but the product itself. Voice agents live or die on a sub-second turn budget split between speech-to-text, model, and text-to-speech. At GPU decode speeds the loop doesn't close and the conversation feels stilted. At LPU speeds it does. Same for live translation, coding autocomplete, and anything where a human is watching tokens appear.

Part 6: The uncomfortable footnote

I'd be doing you a disservice if I stopped at the technology.

In late 2025 Nvidia paid roughly $20B for access to Groq's inference architecture. Jonathan Ross and president Sunny Madra went to Nvidia. Simon Edwards is now Groq's CEO. Nvidia showed a Groq 3 LPX inference accelerator at GTC 2026. Reporting still differs on whether that was a non-exclusive licence or an acquisition of the inference unit.

Groq raised $650M in June 2026 and GroqCloud continues to operate independently. And in June 2026, Groq deprecated most of the model IDs that every LangChain tutorial written before that date depends on. I found that out the way everyone does, with a 400 error that says the model has been decommissioned.

Which is the real lesson, and it isn't about Groq. Speed is a feature you rent. Portability is a feature you own. Write your code against an OpenAI-compatible interface. Keep model IDs in config, never in source. Query the provider's model list programmatically instead of hardcoding what a tutorial told you eighteen months ago. Then the question "which chip is fastest this quarter" becomes a pricing decision rather than an engineering project.

The podcast's advice was to hold the physical layer for ten years and ignore the daily noise. For investors, maybe. For builders, the opposite applies. The silicon underneath you will get replaced two or three times in that window. Build so you don't care which one is winning.

David Lin Podcast: https://www.youtube.com/watch?v=12GGEIB4Tco

Sources: Groq's LPU architecture documentation and engineering blog, Groq's TSP papers from ISCA 2020 and 2022, arXiv work on prefill/decode-aware accelerator evaluation, and current vendor benchmarks. Performance figures are largely vendor-supplied and should be treated accordingly. Model availability on GroqCloud changes frequently — check the live catalogue before building against any specific model ID.