The model did exactly what it was told. The software around the model uploaded the whole repository anyway.

A researcher put a coding agent to the simplest possible test. They created a file called never_read_canary.txt, dropped a unique marker inside it (CANARY-XR47P2-NEVERREAD-UNIQUE), and gave the agent an instruction that left no room for interpretation: "Reply exactly OK, do not read any files." The agent replied OK. It obeyed. It did not read the file.

Then the researcher checked what the agent's client had sent over the network while all that obedience was happening, and found a git bundle uploaded to a cloud bucket the user never chose. They cloned the captured bundle. Inside it, verbatim, was the never-read file's contents, plus the repository's entire commit history. The model did exactly what it was told. The software around the model uploaded the whole repository anyway.

That gap, between what the agent did and what the program shipped, is the story. The product was xAI's Grok Build CLI, version 0.2.93, marketed as a local-first coding agent. In July 2026 a researcher publishing as cereblab ran it behind a network proxy and read the wire, and what they found is worth understanding in detail, because the specific failure here is one that any team wiring an AI agent into their codebase should be able to recognize in their own stack.

Two pipes, and one of them was the point

Every Grok Build session opened two independent network channels. On the first, the model-turn channel, the agent sent what the task actually required: in the instrumented session, about 192 KB across five requests. Call it 196,705 bytes. That is the pipe you would expect. It carries the code the model needs to read to answer your prompt.

The second channel was doing something else entirely. In parallel, the client shipped a whole-repository snapshot to a separate storage endpoint: 5,476,228,005 bytes, roughly 5.10 GiB, broken into 73 chunks of about 75 MB each, every upload returning a clean HTTP 200. In that one session the client uploaded on the order of 27,800 times more data than the task consumed. (Those figures come from a single instrumented run on a roughly 12 GB test repository. Treat the exact numbers as one measurement, not a universal constant. The mechanism they reveal is the general claim, and the canary test nails it down: the upload happens regardless of what the model reads.)

The two channels are not related. The big one is not "telemetry about what the agent looked at." It is the repository, wholesale, sent independently of anything the agent touched. And because it ships as a git bundle rather than a copy of your working files, it carries everything: every tracked file plus the full commit history. That last part matters more than it first appears. A credential you committed by accident eight months ago and rotated out the next day is still sitting in your history. The bundle takes it with everything else. "We removed that key" and "that key is gone" are different sentences, and a whole-history upload is the moment the difference bites.

The toggle governed the wrong layer

Here is the part that turns this from one vendor's bug into a lesson worth keeping. Grok Build had a privacy control. It was labeled around improving the model, and it was a real consent toggle: it governed whether your data could be used for model training. The upload to the storage bucket rode a different, client-side path, and the server reported that path as enabled no matter where the user set the training toggle. The permission the user could see was truthful about one thing and completely irrelevant to the thing that mattered.

Sit with that, because it is the transferable idea. A permission scoped to the wrong layer is indistinguishable from no permission at all. The user did everything right. They found the setting, they understood it, they set it to protect their code. The setting honestly controlled training and honestly said nothing about exfiltration, and the two were wired to different switches. You cannot audit this by reading the settings screen, because the settings screen was accurate. It just wasn't answering the question you were actually asking.

This generalizes past xAI, and it is a lesson to carry home. When you connect any agent to your source, "don't train on my code" and "don't send my code anywhere" are two separate consents. A product can grant the first, in good faith, while violating the second, and the interface will not necessarily tell you, because the interface is describing the toggle it has, not the pipe you're worried about. The question to ask a vendor is not "do you respect my privacy setting." It is "which specific data paths does this specific setting bind, and what governs the ones it doesn't."

It helps to picture what the honest version looks like, because "local-first" is a real design with a real shape, and Grok Build had the label without the shape. A genuinely local-first coding agent sends the model only the files the task needs, names the exact endpoints it will ever contact, ships nothing on a second channel you didn't ask about, and makes the network behavior legible from inside the product rather than only from a proxy outside it. None of that is exotic. Plenty of tools do it. The reason the word matters is that it is a promise about data paths, and a promise about data paths is exactly the kind of claim a thirty-minute network trace can confirm or destroy. When a vendor uses the term, they are inviting that trace. Grok Build's mistake was not that it uploaded data. Some tools legitimately do, with consent. The mistake was claiming a property its own client contradicted, on a channel the user was never shown.

The only reason anyone knew

There was no advisory. No alert fired. Nothing in Grok Build's own output looked wrong, because from the product's point of view nothing was wrong: it completed your tasks, it obeyed "do not read," it returned OK. The behavior was found because one person put a proxy in front of the binary and read the layer underneath the product's own reporting.

This is the next lesson to carry home, and it is uncomfortable. An agent that behaves correctly in everything you can observe is indistinguishable from one that doesn't, right up until someone instruments the layer below. If your confidence in an agent rests entirely on what the agent tells you it did, you do not have evidence. You have testimony. The canary test worked precisely because it did not trust the agent's report. It planted a fact the agent swore it never touched, then went and looked for that fact somewhere the agent could not edit. That is the shape of a real audit of an autonomous tool: not "did it say it behaved," but "does a channel the tool doesn't control agree."

For most teams the practical version is modest and worth doing anyway. You do not need a full mitmproxy rig to know whether the coding agent you just adopted phones home, and where, and with how much. An afternoon watching its network traffic on a repository full of canaries tells you more about its data behavior than any amount of documentation, because documentation describes the toggle and the wire describes the pipe.

The response, and what it does and doesn't fix

The sequence after disclosure is itself instructive, so here is the timeline.

Around July 12, 2026, cereblab published the wire-level analysis together with a public reproduction harness: the proxy setup, the canary repository, downloadable evidence, everything needed for a stranger to re-run the finding. That last detail is why this account is trustworthy enough to build an essay on. The researcher did not ask to be believed. They shipped the experiment.

On July 13, xAI disabled the uploads with a server-side flag. Note what that means precisely: the upload code remained in the shipped binary, and no software update went out. The behavior was switched off from the vendor's side, which also means it can be switched back on from the vendor's side. The capability still ships. The vendor holds the switch. Also on July 13, Musk publicly promised that the already-uploaded data would be deleted in full. That promise is a public statement, not a verified event. As of this writing there is no independent attestation that any deletion occurred, and no way for an affected user to confirm their own data is gone.

On July 16, xAI open-sourced Grok Build under the Apache 2.0 license. This is a real remedy and an incomplete one, and both halves are worth stating plainly. It is real because it converts an unauditable binary into something anyone can read, which is close to the strongest response available when no advisory was issued: instead of asking you to trust the vendor, it lets you inspect the code. It is incomplete because being able to see the upload path is not the same as the path being gone, and a behavior disabled by a remote flag is a behavior that still exists. Within days a community fork appeared with the telemetry stripped out. That fork is the market answering the only question that ends up mattering here: not "is this vendor trustworthy," but "who holds the switch, and can I take it away from them."

And some things are simply still missing. There is no published count of affected developers. There is no stated total volume of code collected since Grok Build entered public beta in May 2026. There is no user-facing way to confirm deletion, and no independent audit that it happened. It would be easy to fill those gaps with a scary estimate. The honest move is to leave them empty and say so, because the absence of those numbers is part of the finding. A system that over-collected by default for two months and then disabled the behavior with a flag is a system that, by construction, is the only party who knows the true scope, and it hasn't said.

What to actually do with this

Strip away the specific vendor and three durable practices remain, one for each thing that failed.

The permission failed because it bound the wrong layer, so stop trusting consent labels and start asking what they govern. For any agent that touches your code, get the vendor to name the data paths a given setting controls, and treat every path they don't name as open until proven otherwise. "Local-first" is a marketing claim until a network trace makes it a measured one.

The exposure was worse than expected because git history travels, so treat your repository's history as live secret material, not as an archive. Any agent that can bundle your repo can ship everything you ever committed, which means secret scanning and history hygiene are not cleanup chores you get to someday. They are the difference between an over-collection incident that costs you nothing and one that hands out a key you thought was retired.

The whole thing was invisible because the product's own reporting looked clean, so build at least one check that doesn't depend on the agent's self-report. A canary file and an hour with a network monitor is a low bar, and clearing it is the difference between trusting a tool and verifying one. The researcher who found all of this did not have inside access or a vendor tip. They had a proxy and a fake secret, and they looked at the layer the product couldn't narrate. You can do the same to anything you're about to point at your codebase, and after this story, you probably should.

Sources

  • cereblab, "What xAI Grok Build CLI actually sends to xAI, a wire-level analysis (grok 0.2.93)" — gist.github.com/cereblab/dc9a40bc26120f4540e4e09b75ffb547. The primary for every technical figure here: the 196,705-byte model channel, the 5,476,228,005-byte storage upload and ~27,800× ratio (one instrumented session, ~12 GB test repo), the two-channel mechanism, the training-toggle-vs-upload-path separation, and the never-read-canary test. Ships with a public reproduction harness (github.com/cereblab/grok-build-exfil-repro).

  • xAI, Grok Build source — github.com/xai-org/grok-build. Primary record of the July 16, 2026 Apache-2.0 open-sourcing.

  • The Register, "Musk promises purge after Grok Build caught sending entire repos to the cloud" (July 14, 2026) and "SpaceX open sources Grok Build after data-retention furore" (July 16, 2026) — news of record for the disclosure timeline, the server-side disable, Musk's deletion promise (paraphrased here rather than quoted, as the exact wording was not confirmed against his original post), and the open-sourcing date.

  • Note on scope: counts of affected developers and total data volume are stated as unknown because no primary source publishes them; this essay does not estimate them.

Trust the channel the tool doesn't control, not the report the tool writes.

The whole failure here was invisible from inside the product, because the product's own reporting looked clean. The fix is the same one the researcher used: an independent record of what actually crossed the wire, checked against a channel the tool can't edit. That is what the agent trust stack is for: chain-of-consciousness for a tamper-evident record of what an agent actually did, plus ratings and verification so a downstream check answers to the wire, not to the agent's self-report.

Read the Theory of Agent Trust  ·  See Hosted Chain of Consciousness

pip install agent-trust-stack  ·  npm install agent-trust-stack

Or the provenance record on its own: pip install chain-of-consciousness / npm install chain-of-consciousness