Building a GitHub App That Reviews Its Own Code: Lessons in Security Hardening

How I turned 27 silent failures into logged ones, closed three real attack surfaces, and pushed test coverage from 62% to 76% while building an automated code-review bot.

A few months ago I set out to build something simple on paper: a GitHub App that reviews pull requests, scans for leaked secrets, applies safe autofixes, and responds to slash commands like a teammate would. I called it GitHub Autopilot. It runs on Flask, uses Redis for job queuing, and talks to GitHub through webhooks.
The first working version took a couple of weeks. Making it safe enough to trust with someone else's repository took much longer - and taught me more about security engineering than any tutorial could.
This article walks through three real vulnerabilities I found and fixed during an internal audit, why each one mattered, and what the process looked like end to end.
Why a Code-Review Bot Is a Security Target
A GitHub App that can read code, comment on PRs, and push autofixes sits in a privileged position. It has write access, it processes untrusted input (every PR, every webhook payload), and it often talks to other services - in my case, an MCP (Model Context Protocol) server for AI-assisted review.
That combination means three things need to be airtight: authentication, request validation, and resource limits. I had working code for all three. What I didn't have, until I audited it properly, was proof that each one failed safely under attack.
Problem 1: Authentication That Failed Open
The MCP integration handled requests from an external service. My original authentication check worked like this: if a token was present, validate it; if validation itself threw an error - say, the auth service was slow or unreachable - the code caught the exception and let the request through.
This is called failing open, and it's one of the more common mistakes in systems that bolt security onto an existing code path. The intention was reasonable: don't let a flaky dependency take down the whole app. The result was dangerous: an attacker who could trigger an auth-service timeout could skip authentication entirely.
The fix was to flip the default. Any exception during authentication now results in an automatic denial, not a pass-through. If the auth service is unreachable, the request is rejected, logged, and retried - never silently trusted. This is the standard fail-closed pattern, and it should be the default for any security check, full stop.
Problem 2: A Content-Length Bypass
Webhook payloads come in with a Content-Length header, and the app used it to enforce a size limit before processing - a reasonable defense against oversized or malicious payloads. The gap: the check trusted the header value itself rather than the actual bytes received.
A request could declare a small Content-Length while streaming a much larger body, slipping past the size check entirely. This is a known class of bug in HTTP handling, and it's easy to miss because the code "looks" correct - it reads a header and compares a number.
The fix was to validate against the actual size of the data read from the stream, not the client-supplied header. It's a small code change, but it closes a real gap between what a client claims and what a server receives - a distinction that matters anywhere you're parsing untrusted input.
Problem 3: A Rate Limiter That Leaked Memory
The app rate-limits requests per IP address to prevent abuse. The original implementation stored a counter per IP in memory, incrementing on each request. What it didn't do was clean up entries for IPs that stopped sending requests.
Under normal traffic this is invisible. Under sustained traffic from many different IPs - which is trivial to generate - the counter dictionary grows without bound. Eventually the process runs out of memory and crashes. This turns a defensive feature into an attack vector: the very thing meant to stop abuse becomes the tool for causing an outage.
The fix added a time-based eviction policy, clearing stale IP entries on a rolling window instead of letting them accumulate forever. Rate limiters need to bound their own memory usage, not just the request rate - a detail that's easy to skip when the feature works correctly in every manual test.
The Quieter Problem: 27 Silent Failures
None of the three bugs above would have been easy to catch through code review alone, and that pointed to a deeper issue: 27 places in the codebase caught exceptions and did nothing with them. A try/except: pass pattern, repeated across error-handling paths, meant that when something went wrong, the app kept running - silently, with no log entry, no alert, no trace.
This is a comfortable pattern to write and a dangerous one to ship. It hides exactly the kind of failure that matters most: the one that happens in production, once, under conditions you didn't test for.
Every one of those 27 handlers was rewritten to log the failure with enough context to debug it later - what operation failed, what input triggered it, and the original exception. None of them changed what the app does when something breaks. All of them changed whether you'd ever find out.
Proving It: Testing and Coverage
Fixes without tests are opinions. Each of the security changes above shipped with tests that reproduce the original failure mode - a forged Content-Length, an auth-service timeout, a rate-limit counter under sustained load - and assert the new, safe behavior.
Across the full audit cycle, the project went from 62% to 76% test coverage, with 654 out of 654 tests passing and 834 total tests in the suite. Coverage numbers alone don't prove correctness, but combined with targeted tests for each vulnerability, they gave me confidence to say the fixes actually work - not just that the code compiles.
What This Project Taught Me About Documentation
Fixing the bugs was half the work. The other half was writing it down in a way a future contributor - or an auditor, or a client - could actually use: an architecture diagram showing how webhooks flow through auth, queueing, and processing; a threat-model table mapping each attack surface to its mitigation; and a changelog that explains why each fix happened, not just what changed.
That last part turned out to matter most. Code shows what a system does. Documentation is the only place that shows what it's defending against, and why the defense looks the way it does. For any system that handles untrusted input - which is most systems - that record is worth as much as the fix itself.
GitHub Autopilot is open source. The full architecture docs, threat-model table, and audit history are available on GitHub: Shweta-Mishra-ai/github-autopilot.
I write about building and securing developer tools at TechNova World. If you're looking for someone who can both build the system and document it clearly for your team, let's talk.