This is a submission for *DEV's Summer Bug Smash: Clear the Lineup** powered by Sentry.*

Eliminating Non-Deterministic Exit Codes in StayPresent

Project Overview

StayPresent is an open-source Python library for supervising long-running bots and background services. It automatically restarts crashed bots, monitors their health, and exits with an appropriate status code when recovery is no longer possible.

While reviewing the crash recovery logic, I discovered that identical failures could sometimes produce different process exit codes.

Bug Fix or Performance Improvement

When more than one supervised bot ultimately failed, StayPresent needed to choose a single exit code for its own process.

The implementation selected the first exit code stored in an internal dictionary of failures.

At first glance this looked perfectly reasonable.

The subtle problem was how that dictionary was populated.

Each bot is monitored by its own thread, and whichever monitor thread happened to finish first inserted its entry first.

That meant the framework's own exit code depended entirely on thread scheduling.

Two runs with the same bots, the same crashes, and the same individual exit codes could still report different overall exit codes simply because the operating system scheduled monitor threads differently.

Nothing about the application had changed.

Only timing.

The Fix

The solution was to remove timing from the decision entirely.

Instead of depending on dictionary insertion order, StayPresent now deterministically chooses the exit code belonging to the lowest-indexed failed bot.

Interestingly, the failure summary printed immediately before shutdown was already sorted by bot index.

The process exit code now follows the same rule, ensuring logging and behavior remain perfectly consistent.

Why It Matters

One of the hardest classes of bugs isn't crashes—it's non-deterministic behavior.

When identical failures produce different observable results, debugging becomes significantly more difficult.

Automation, CI systems, deployment tooling, and shell scripts all rely on predictable exit codes.

By making exit-code selection deterministic, StayPresent now behaves consistently regardless of thread scheduling.

Conclusion

Concurrency bugs aren't always race conditions that crash a program.

Sometimes they quietly introduce randomness into behavior that should be completely predictable.

Removing that source of randomness makes the framework easier to debug, easier to automate, and more reliable in production.

Code

GitHub Repository

https://github.com/StayElite/StayPresent

Changelog (Fixed in v1.5.15)

https://github.com/StayElite/StayPresent/blob/main/CHANGELOG.md

Commit

https://github.com/StayElite/StayPresent/commit/def77e96a722de3a6705ed6dd4b654e82142adc6