Your gRPC stream is "healthy" and serving nothing: synthetic monitoring for server-side streams

By Daniil Romashov — SRE/DevOps engineer. The tool described here is open source:
github.com/youngpabl0/grpc-streams-checker (Apache-2.0).

Uptime checks are a solved problem for request/response APIs: hit the endpoint, read the status code, done.
Server-side streaming RPCs break that model completely. A gRPC stream isn't a request and a response — it's a channel the server holds open and pushes messages into over time. The
connection can be up, the handshake can succeed, the health endpoint can be green — and yet the
only thing that matters, frames arriving with valid data, can be silently broken.
Your probe says "healthy" while consumers get nothing.

I hit this running real-time market-data streams behind a gRPC-web/Envoy stack. In a realistic
production topology the stream crosses several hops before anyone consumes it:

client ──▶ LB ──▶ Envoy (grpc-web / HTTP/2) ──▶ backend ──▶ upstream feed

Enter fullscreen mode Exit fullscreen mode

Every hop is a place where the stream dies while everything still looks healthy:

• an idle timeout on the LB or proxy quietly kills long-lived connections;
• Envoy / grpc-web translation buffers or drops frames while the TCP session stays up;
• the backend keeps the stream open but its upstream feed went quiet — nothing to push;
• frames arrive, but they're garbage — empty payloads, missing fields — after a bad deploy.

None of this is visible to an HTTP 200 probe or a standard gRPC health check.
"The stream stopped
producing" was simultaneously our worst failure mode and the one nothing caught — customers reported it before monitoring did. So I built a synthetic checker for it, ran it in production for
a year across six streaming methods, and have now open-sourced a clean rewrite.

The core idea: consume the stream like a real client

The only probe that answers "is this stream actually serving valid data?" is a real client.
For each configured stream, on a infinite cycle, grpc-streams-checker doing:

1. loads the .proto at runtime (via @grpc/proto-loader — no codegen, no stale stubs; adding a stream to monitoring = pointing at a file and naming a method);
2. opens the server-side streaming RPC — TLS or plaintext, auth metadata if needed;
3. collects N frames (not just the first!) within a time budget, measuring time-to-first-frame and the largest gap between consecutive frames;
4. validates every frame's payload — required fields present, values matching exact strings or regexes — because "frames are flowing" and "frames are correct" are different failures;
5. cancels the stream and exports the outcome as Prometheus metrics.

Point 3 deserves emphasis. A stream that emits one frame and hangs is a different (and sneakier) failure than one that never starts. Demanding, say, minFrames: 3 per check verifies sustained flow.
And the largest inter-frame gap turns out to be a great early-warning signal: a stuttering
producer degrades there long before it goes fully dark.

Point 4 catches the failure nobody instruments for: the deploy that keeps the stream up but breaks the payload.
An empty symbol field in a price tick is an outage for the consumer even though every transport-level signal is green.

Metrics designed for alerting, not just dashboards

Every failure mode gets its own result label, and — the key design decision — every per-stream metric carries a method label, so per-method dashboards and ownership-routed alerts fall out of the label set for couple of minutes:

grpc_stream_up{stream,method,addr,dc}                      # 1 = frames arrived AND payloads valid
grpc_stream_check_total{...,result}                        # ok|timeout|insufficient_frames|validation_failed|error
grpc_stream_error_total{...,code}                          # UNAVAILABLE, DEADLINE_EXCEEDED, UNAUTHENTICATED…
grpc_stream_first_frame_ms / _duration_seconds (histogram) # degrades before it breaks
grpc_stream_max_inter_frame_ms                             # sustained-flow health
grpc_stream_frames_last_check / _total                     # did we get the 3-5 we demanded?
grpc_stream_validation_fail_total{...,rule}                # missing_field | mismatch | empty_frame
grpc_stream_last_success_timestamp_seconds                 # staleness alerting
grpc_stream_last_run_timestamp_seconds                     # a dead checker must never look like health

Enter fullscreen mode Exit fullscreen mode

Success rate by method — one query:

sum by (method) (rate(grpc_stream_check_total{result="ok"}[5m]))
/
sum by (method) (rate(grpc_stream_check_total[5m]))

Enter fullscreen mode Exit fullscreen mode

The repo ships ready-made Prometheus alert rules
— stream down, slow first frame, stuttering producer, invalid payloads, repeated gRPC errors, and
"the checker itself died" (silence must never masquerade as health) — plus
Kubernetes manifests
with probes, resource limits, and a ServiceMonitor.

Try it in 32 seconds

The repo includes a demo gRPC server with intentional failure modes:

git clone https://github.com/youngpabl0/grpc-streams-checker.git
cd grpc-streams-checker && npm ci

node examples/demo-server.js &        # a PriceStream on :50051
cp streams.example.json streams.json
node src/index.js --once
# → {"stream":"price_stream","result":"ok","frames":3,"firstFrameMs":223,...}

node examples/demo-server.js --silent   # opens, never sends → result: timeout
node examples/demo-server.js --broken   # empty fields       → result: validation_failed
node examples/demo-server.js --slow=2000 # stuttering        → max_inter_frame_ms grows

Enter fullscreen mode Exit fullscreen mode

--once exits non-zero on any failure, so the same binary doubles as a CI gate or cron probe.

What a use in production taught me

• Per-method labels pay for themselves. With team-routed alerting on top, each squad gets paged only for its own streams — one checker serves the whole org.
• Time-to-first-frame is the canary. Every full outage we had was preceded by visible first-frame degradation. Alert on the trend, not just the corpse.
• Validate payloads. Two of our worst incidents were "stream up, data wrong." Transport metrics can't see that class of failure at all.
• Monitor the monitor. last_run_timestamp with a "checker is dead" alert is non-negotiable; a crashed checker otherwise reads as a fleet of healthy streams.

The pattern generalizes to any long-lived push transport — WebSocket, SSE, queue consumers. The
value is turning "is data actually flowing, and is it valid?" into labelled metrics you page
yourself about, instead of something a customer reports.

---

Daniil Romashov is a Site Reliability / DevOps engineer specializing in reliability and
observability for high-load systems.
Code and manifests: github.com/youngpabl0/grpc-streams-checker.