CarSegNet v2: One Lot Photo to a Showroom Composite

A buyer does not open a listing and ask what the mask IoU was. They look at the mirror stalk, the antenna, the gaps between wheel spokes, the glass, and the tire line where black rubber meets whatever floor you claim the car is sitting on. If asphalt from the old lot still glows through the spokes, or the car floats a finger-width above a showroom floor that never saw those tires, they feel it before they can name it. The click goes cold. The unit sits.

That is the sales problem in one frame. CarSegNet v2 is the still-image system I built so a dealership can take one usable lot photo and walk away with showroom-grade listing media: a composite, a cutout, and an alpha matte written as first-class artifacts. The buyer stakes are real. So are the ops stakes that never show up in a segmentation paper.

The ops case: same-day media without inventing ROI

A car that arrives on a Tuesday and waits until Thursday for a photo package is inventory that cannot earn an inquiry online. It is on the lot. It is paid for. It is not sellable as a listing until someone produces media. Same-day listing media turns delivery into something a shopper can find the same afternoon. That is money saved by shortening unsellable days, and money earned by putting the unit in front of demand sooner. I am not going to invent a dollar figure or an ROI percentage for that. CarSegNet and AutoLens measure matting, topology, and pipeline plumbing. They do not publish a controlled study of dollars per unit. The category of loss is still obvious to anyone who has watched a delivered car sit dark on the website while the lot fills with snow.

Production crews and staging are the other spend. Not every unit needs a full crew day, a swept bay, and a lighting package to go live. When the ask is routine inventory media, paying that setup for every VIN is a recurring cost you feel in the calendar as much as in the invoice. A path that starts from one lot photo and lands on a cached showroom plate cuts how often that crew has to turn up for ordinary listings. Special cars can still get special shoots. The rest of the row should not wait on them.

I am writing this from Buffalo winters, not from a coasting climate. Outdoor lot shoots fail when snow is falling sideways and the windshields are white. Not every dealer can stage indoor photoshoots for every unit all season. When weather kills traditional production, inventory media still has to move. One photo to a showroom composite is the ops path for those weeks: keep listings alive when the lot is unusable as a set.

What one photo produces

One still goes in. The pipeline builds a vehicle matte, places the car on a cached or prompted showroom plate, and writes three artifacts:

• *_composite.png for the listing
• *_cutout.png for reuse
• *_alpha.png as evidence, not a hidden intermediate

The alpha file is part of the product. If a merchandising manager rejects the frame, you can see what the matte actually decided. The stack behind that path is three named pieces, each with a clear honesty boundary:

1. CarSegNet still path: semantic prior, NeuralSegJet (NSJ) alpha refine, optional interior openings, plate, composite.
2. WindowSweep: paired glass supervision so cabin stays while reflected lot scenery can leave. Capture contract and audit tooling are real; a production glass model is not shipped.
3. CCR (Counterfactual Correctability Routing): product name for propose-then-verify repair routing. Openings safety and the metrics review gate are the live machinery today; a standalone CCR module is not.

Cached plates are the reliable production path: same floor, same lighting language, no provider drift between units. Prompted plates are optional when the plate library needs a new look, not something you should spend on every VIN.

Topology is the hard part, and the buyer feels it first

Most car cutout failures are edge failures: halo on the roofline, jagged mirror stalks, weak tire contact. Cars add a second class of error that region IoU barely notices: enclosed background. Wheel openings, spokes, grilles, roof rails, and glass are holes inside the object. A model can score well on the outer silhouette while painting those openings as solid vehicle. The lot then shines through a "solid" wheel, or the glass turns into painted metal.

Think of it like cutting a paper stencil of a car for a spray booth. Cutting the outer outline is one job. Cutting the windows and the gaps between spokes so paint can pass through is another. A perfect outline with the windows left filled is still a bad stencil. Boundary refinement can sharpen the contour. It cannot reopen every interior gap once the prior marked it foreground. That is why CarSegNet keeps those jobs in separate stages: each error has one place to inspect.

flowchart LR
  photo(["Dealer Lot Photo"]) --> prior["Semantic Vehicle Prior"] --> alpha["NSJ Alpha Refiner"]
  alpha --> holes{{"Opening Repair Optional"}} --> plate[("Cached Or Generated Plate")]
  plate --> comp["Composite"] --> booth(["TypeScript Booth Client"])

  classDef input fill:#1c2e28,stroke:#4a7a68,color:#d5ebe3
  classDef model fill:#1a2740,stroke:#4a6d9a,color:#d0dced
  classDef gate fill:#2a2436,stroke:#6e5f8c,color:#e2d8f0
  classDef plateNode fill:#2a261a,stroke:#8a7848,color:#efe6c8
  classDef output fill:#1a2c36,stroke:#4a7a8e,color:#cde4ee
  classDef client fill:#1e2a24,stroke:#5a8a6e,color:#d4ebe0

  class photo input
  class prior,alpha model
  class holes gate
  class plate plateNode
  class comp output
  class booth client

Enter fullscreen mode Exit fullscreen mode

The openings stage is real code in carsegnet/openings.py, but configs/default.yaml ships with openings.enabled: false. Keep it off until the locked 608-frame topology acceptance gate passes. The topology thesis still drives the design when that stage is disabled, because region IoU alone will green-light a filled wheel, and a filled wheel is exactly what a shopper rejects.

Carvana topology: strong outer, almost no hole recovery

The loudest evidence for that split is the Carvana topology audit on a 608-frame, 38-vehicle validation split with the corrected still path (semantic text prompt, no detector box, NSJ on the training band).

Outer metrics for the trained NSJ checkpoint look strong: IoU 0.9908, boundary IoU 0.4762, SAD 5.4460, zero frames below 0.90 IoU. Topology tells a different story. Against 1,193 ground-truth enclosed holes, trained NSJ matched 5, left 1,188 missing, and introduced 111 extras. Raw semantic SAM matched 1. The fallback refiner matched 28 and exploded extras.

Preserve the trained checkpoint as a boundary ablation, not a production model that "solved openings." Those numbers are checkpoint validation on studio backgrounds with binary glass masks. They are not an untouched holdout and not a dealer-lot benchmark. Real lot photos and WindowSweep need separate evaluation. That contrast is why openings is a separate stage, not a footnote under IoU.

Glass, correctability, and what the booth actually does

Glass is where cheap background tools lie. They treat a windshield as keep-or-remove, and either erase the cabin or leave the lot reflection welded into the glass. What you want on a listing is different: keep the seats, the headrests, the cabin volume; replace the scene that is only sitting in the reflection.

WindowSweep is the supervision contract for that job. Same vehicle, same camera pose, same cabin rays, once with the target side glass raised and once lowered. Similar framing is not enough. If the camera moves, the pair is not teaching glass. It is teaching camera shake. The F-150 pilot under data/windowsweep/f150_pilot_001/ holds five declared pairs. Audit accepts both close pairs at the coarse geometry gate and rejects all three wide pairs because the camera moved. Those accepted pairs are enough to test registration and a first glass-loss prototype. They are not a training corpus. AutoLens product copy describes WindowSweep as in training for the same reason: the capture contract and audit tooling are real; a production glass model is not being claimed as shipped.

CCR names the dry-fit habit before you glue a repair: try a local detail fix in a controlled way, keep it only when the result earns its keep. There is no ccr.py in the CarSegNet tree today. What ships is the same discipline on interior openings and batch QA. openings.py proposes a hole cut, then refuses it if the proposal adds foreground, leaves the interior trust region, or changes the outer silhouette when verify_outer=True. metrics.py reports region accuracy, boundary accuracy, matting error, enclosed-hole topology, and a pass/fail gate so a batch job can route only questionable frames to a human. AutoLens currently lists CCR as coming soon. The openings safety contract and the metrics review gate are already how CarSegNet refuses to "fix" a unit into a worse listing.

After the matte lands on a plate, compose does four deterministic jobs in one pass: shift car colour toward the plate (global LAB, gain clamped), blur a plate that is sharper than the subject, drop a contact shadow under the tire band, and wrap a little plate colour into the vehicle edge. That stack can make a good matte believable. It cannot hide a broken one, and global LAB will not fix mixed lot lighting or reflective chrome that still mirrors the original lot.

The booth itself is thin on purpose. A TypeScript client talks to a FastAPI service over a single CUDA worker so concurrent jobs do not fight one 24 GB card. Progress rides a WebSocket with HTTP polling as fallback. Voice is a small control layer for overrides, not a second vision stack. CLI and booth share the same CarSegNet class: booth for interactive unit work, CLI for batch folders and matte-only runs.

Failure modes I keep in the open

A few limits are load-bearing, not footnotes:

• If the segmenter fills a wheel and supplies no interior contour, NSJ cannot reopen it. That sentence is in PAPER.md and matches the Carvana topology table.
• Openings stay disabled until acceptance evidence locks decoder thresholds. Enabling them without a readable weights checkpoint is a hard PipelineError.
• Openings are still-image only; enabling them on video raises PipelineError.
• WindowSweep's F-150 pilot is a geometry and glass-loss test bed, not a multi-vehicle training corpus.
• CCR as a full product routing layer is named in the AutoLens stack and not yet a standalone shipped module; openings safety and the metrics review gate are the live correctability machinery today.
• Harmonisation is global LAB. Mixed lot lighting will not suddenly look like a single softbox.
• Reflective bodywork that mirrors the original lot is out of scope; the pipeline does not invent occluded geometry or rewrite chrome reflections.
• Plate parallax on video is capped (about 3.5 percent of frame width in config). It cannot invent occluded geometry.
• Isolated 1080p wall-clock timing is still an open benchmark in the README. This post does not invent an end-to-end minute count.
• AGPL YOLO26 weights matter for commercial packaging; the detector is optional, depth is harder to replace.
• Two external integrations still need live confirmation per the README: SAM 3.1 handle_stream_request on a real clip, and one live gpt-image-2 call. carsegnet doctor is the first stop before spending GPU hours or image credits.

The product ask sounds simple: one lot photo, one showroom composite, list the car today. The engineering ask is different: keep outer silhouette quality, interior hole honesty, glass supervision, plate style stability, and booth UX from collapsing into a single opaque model call that you cannot debug when a merchandising manager rejects the frame. A showroom composite is only as strong as the mask topology, the glass contract, and the refusal to publish a bad repair. That is also the sales case: same-day listing media without waiting on crews, staging bays, or a clear Buffalo sky, with named components you can point to when a buyer, a GM, or a skeptic asks what happens when the wheel spokes and the windshield disagree with the lot behind them.

---

For implementers

Repo evidence for the claims above lives in named files: carsegnet/pipeline.py, carsegnet/openings.py, carsegnet/metrics.py, carsegnet/compose.py, carsegnet/background.py, scripts/audit_windowsweep.py, and the config flags in configs/default.yaml. The rest of this section is the CLI, cascade, and booth detail a sales reader does not need in the first third.

CLI and booth entry

CLI still path with a cached plate:

carsegnet run photos/unit.jpg --bg-file plates/showroom.png

Enter fullscreen mode Exit fullscreen mode

Prompted plate generation (optional path; needs OPENAI_API_KEY):

carsegnet run photos/unit.jpg --bg-prompt "wet city street at night, neon reflections, no cars"

Enter fullscreen mode Exit fullscreen mode

Booth / service entry:

python server/app.py --host 0.0.0.0 --port 8080 --config configs/default.yaml

Enter fullscreen mode Exit fullscreen mode

-c is the short form of --config in server/app.py. --warm exists and starts a background thread that imports carsegnet.pipeline; it does not yet call the worker's full model warm path, so treat it as a light import warm, not a guarantee that every weight is resident.

The repo targets a single 24 GB GPU worker (the README calls out an RTX 3090 Ti class card). Video uses the same service shape and runs longer. Still images are the center of the sellable path.

Still path cascade

carsegnet/pipeline.py owns CarSegNet.process_image and CarSegNet.matte_image. The still flow is linear: read the image, build the matte, load or create the plate, composite, write artifacts. Models load lazily on first use so a cold CLI call pays import cost once per process, not at module import time.

Default segmentation uses the text prompt car from configs/default.yaml. The YOLO26 box path in carsegnet/backends.py is opt-in via segmenter.use_box_prompt. Depth from yolo26x-depth.pt is on by default and feeds hero selection, contact shadow attenuation, and video plate parallax. If depth fails, the pipeline logs a warning and continues without it; shadows and parallax degrade, the matte path does not hard-stop. On a crowded lot that matters: you want the near car in the frame, not the largest blob two rows back.

Alpha refinement lives in carsegnet/nsj.py and carsegnet/refine.py. NeuralSegJet (NSJ) replaced dense Conditional Random Field (CRF) cleanup because pydensecrf does not build cleanly on the Python 3.12 stack the rest of the repo requires. NSJ works inside a trimap: confident foreground, confident background, and a narrow unknown band. A colour guided filter uses image colour so local smoothing follows visible edges. That band is good for hairline silhouette geometry: antenna wire, mirror stem, tire contact. Enclosed holes are left to carsegnet/openings.py when enabled, which is the right separation of jobs for a product you intend to sell and support.

Seed, extent, and the interior trust region

carsegnet/openings.py treats each candidate hole as a connected-component problem with two maps and one safety region.

• Seed map: strict. High-confidence enclosed background only (wheel wells, floor through spokes, grille voids, glass). Seeds authorize a cut; they do not invent support alone.
• Extent map: looser. Grows from a seed through nearby pixels that look like the same opening, capped by growth distance and area limits.
• Interior trust region: prior-derived vehicle interior after excluding an outer silhouette guard band. Accepted pixels must stay inside it so the stage cannot eat fenders, bumpers, mirrors, or tire edges.

Decode rejects components that lack a seed, fall outside size limits, leave the trust region, or would change the exterior silhouette after subtraction (apply_opening_mask). Application is subtractive only inside the trust region. It raises OpeningSafetyError if the proposal adds foreground alpha, edits pixels outside trust, or changes the outer silhouette when verify_outer=True.

The craft analogy is a gasket, not a paint roller. The outer seal has to stay true. The bolt holes inside the gasket are separate cuts. If you enlarge a bolt hole until it breaches the outer seal, the part is scrap. Openings is built to prefer a missed hole over a breached fender, because a merchandising manager can send a filled wheel back for review, and cannot unsell a chewed quarter panel that already went live.

When openings stay disabled (the current default), the topology gate in carsegnet/metrics.py still reports the problem. Enclosed background components are matched one-to-one only when both directional coverages clear 50 percent. QA can flag every missing or extra hole even if region IoU looks excellent.

WindowSweep audit path

python scripts/audit_windowsweep.py --self-test
python scripts/audit_windowsweep.py \
  data/windowsweep/f150_pilot_001/windowsweep_manifest.json \
  --report runs/windowsweep/f150_pilot_001.audit.json

Enter fullscreen mode Exit fullscreen mode

scripts/audit_windowsweep.py verifies the manifest, source SHA-256 hashes, and full-frame camera geometry before a pair is allowed near training. All eleven F-150 pilot images come from one vehicle and one session, and the manifest does not yet seal local glass regions, capture controls, split assignments, or training-rights evidence. The README says that plainly.

For new captures, mount the camera, then lock focus, exposure, and white balance before moving the window. Keep every frame from a vehicle and capture session in one split.

Plates and compositing detail

carsegnet/background.py owns plates. The prompt path calls OpenAI Images via client.images.generate, reads data[0].b64_json, and defaults to the pinned snapshot gpt-image-2-2026-04-21 in configs/default.yaml and that module. Pinning costs migration work when the provider changes. It also stops a cached matte from pairing with a drifting plate style. Retired DALL-E aliases raise BackgroundError rather than silently falling through. Prompt results land in ./.bgcache keyed by SHA-256 of prompt, model, size, and quality.

Default background size is 2048x1152 at quality: high, with retries and a 180 second timeout. That is why a live plate call is called out separately from matte work: it is money and latency outside the GPU path.

carsegnet/compose.py applies four deterministic adjustments after fitting the plate to the frame:

Step	Job	Cost
Harmonisation	Shift the car LAB statistics toward the plate (Lightness, A green-red, B blue-yellow), gain clamped.	Unusual paint and tinted glass can move too far. Global LAB will not fix mixed illuminants.
Defocus matching	Blur the plate when it is sharper than the subject (Laplacian variance, capped sigma).	Missing subject sharpness cannot be recovered.
Contact shadow	Ground the tires on the new floor from the lowest band of the silhouette, depth-attenuated when depth exists.	A bad footprint gives a weak shadow, and the car starts to float.
Light wrap	Mix a little plate colour into the vehicle edge.	Excess wrap reads as haze on the body line.

Server, WebSocket, and voice

There are two booth surfaces in the repo. server/app.py serves the FastAPI API and the static web/index.html client at /. web-next/ is a separate Next.js booth client that talks to the same API. Neither hosts the models in the browser.

GPU work runs behind FastAPI on CUDA. One worker thread serializes jobs so concurrent model loads do not fight a single 24 GB card. Job states move queued → running → done | error. Progress publishes on WebSocket /api/jobs/{job_id}/events. web-next/lib/useJob.ts follows that socket and falls back to HTTP polling every second (up to about 15 minutes) when sockets are blocked. Uploads are POST /api/jobs with multipart file plus JSON overrides; outputs are path-confined under runtime/outputs/{job_id}/.

Voice is a small control layer, not a second vision stack. web-next/lib/useVoice.ts uses browser speech-to-text; audio stays on the machine. The server parses the transcript in carsegnet/voice.py into a typed action (with a regex fallback when no API key is present) and clamps overrides before they hit the worker.

flowchart LR
  browser(["TypeScript Booth Client"]) --> api["FastAPI Service"] --> worker["Single GPU Worker"]
  worker --> models[["Prior NSJ Openings"]]
  worker --> outputs[("Composite Cutout Alpha")]

  classDef client fill:#1e2a24,stroke:#5a8a6e,color:#d4ebe0
  classDef service fill:#1a2740,stroke:#4a6d9a,color:#d0dced
  classDef compute fill:#241e30,stroke:#6e5f8c,color:#e2d8f0
  classDef model fill:#1a2740,stroke:#4a6d9a,color:#d0dced
  classDef artifact fill:#1a2c36,stroke:#4a7a8e,color:#cde4ee

  class browser client
  class api service
  class worker compute
  class models model
  class outputs artifact

Enter fullscreen mode Exit fullscreen mode

CLI supports batch folders, config overrides (--set refine.band_px=18), and matte-only runs (carsegnet matte). Pick CLI or booth based on who is watching the screen and how many VINs are in the queue, not based on which path sounds more advanced.

Measurements that prove plumbing vs topology

python tests/selftest.py exercises orchestration, encoding, compositing, openings safety contracts, and quality-assurance reporting on CPU with stubbed model backends. Forty checks, no network, no GPU, no model downloads. That proves plumbing, not dealer-lot matte quality.

The synthetic selftest hard case (thin antenna, dilated and blurred prior) shows the fallback refiner can recover boundary IoU on that scene; it does not measure wheel openings or glass transmission. Upstream SAM 3.1 speed claims and config comments about video.compile are not local dealer-lot timings either. If you are selling this system, sell the measured parts as measured and the gated parts as gated.

Separating SAM prior, NSJ band refine, optional openings repair, WindowSweep glass supervision, CCR-style propose-then-verify routing, pinned or cached plates, and deterministic compose means each failure has a file. carsegnet/openings.py for seed or trust violations. carsegnet/compose.py for a shadow that does not sit under the tires. carsegnet/background.py for a plate model that drifted. scripts/audit_windowsweep.py for a glass pair whose camera moved. web-next/lib/useJob.ts for a socket that died and should have fallen back to polling.

---

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series