There is a specific kind of outage that does not show up in any log you are reading.

A backup job runs nightly at 3am. One night the machine reboots for a kernel patch and comes back with crond masked. No error is raised, because nothing ran. No alert fires, because nothing failed. The job's log file keeps its last entry from the night before, and that entry says SUCCESS.

Eleven days later somebody needs the backup.

This is not a story about backups. It is about the shape of the failure: the absence of an event is not an event, and almost every monitoring setup only watches for events.

Why exit codes are not enough

The standard advice is to check exit codes and alert on non-zero. That is correct and insufficient. An exit code is produced by a process that started. Here is an incomplete list of ways a scheduled job produces no exit code at all:

  • crond is not running, or was never enabled after a reboot
  • the crontab line has an unescaped %, so cron silently truncates the command
  • the script is on a filesystem that failed to mount
  • the user's shell was changed to /sbin/nologin
  • the disk is full, so the redirect target cannot be opened and the shell bails
  • PATH under cron is not your login PATH, so python is not found
  • the container the job lived in was rescheduled and the sidecar never came back
  • DST shifted the wall clock and the 2:30am job was skipped for the year

And the classic: cron mails output to the local user, MAILTO is unset, nobody has read /var/mail/root since 2019.

Every one of those is silent. A monitoring rule of the form "alert when the job reports a failure" cannot catch any of them, because the job never reported anything.

Invert the check

The fix is old and has a grim name: a dead man's switch. Instead of waiting for bad news, you require good news on a schedule. The job tells you it ran. If the report does not arrive inside the window you expect, the absence itself is the alert.

Concretely, that means every scheduled job gets three lines around it:

#!/usr/bin/env bash
set -euo pipefail

PING="https://example-monitor.invalid/ping/db-backup"

curl -fsS "$PING?status=run" >/dev/null || true
./do-the-backup.sh
curl -fsS "$PING?status=complete" >/dev/null || true

Enter fullscreen mode Exit fullscreen mode

Two details in there matter more than they look.

The || true is deliberate. Your monitoring must never be able to fail your job. If the monitoring endpoint is down, the backup still has to run. Any wrapper that can turn a monitoring outage into a production outage is worse than no wrapper.

Note that -S makes curl print connection errors to stderr, and under cron stderr becomes mail. If a monitoring outage would fill your inbox, add 2>/dev/null to the ping, not to the job.

status=run is not optional. Reporting only completion tells you whether jobs finish. Reporting the start as well tells you how long they take, and it lets you catch the run that started at 3am and is still going at 7am. A job that hangs forever never reports failure either.

Report the failure too, and mean it

set -e will exit before your failure ping if you are not careful. Use a trap:

#!/usr/bin/env bash
set -euo pipefail

PING="https://example-monitor.invalid/ping/db-backup"
SERIES="$(date +%s)-$$"        # correlates this run's start and end

# curl -G puts every --data on the query string, and --data-urlencode escapes
# free text for us. No jq, no manual %20 arithmetic.
ping() { curl -fsS -G "$PING" --data "series=$SERIES" "$@" >/dev/null || true; }

fail() {
  local code=$?
  ping --data "status=fail" --data "exit_code=$code" \
       --data-urlencode "msg=failed at line ${BASH_LINENO[0]}"
  exit "$code"
}
trap fail ERR

ping --data "status=run"
START=$(date +%s)

./do-the-backup.sh

ping --data "status=complete" --data "duration=$(( $(date +%s) - START ))" \
     --data "exit_code=0"

Enter fullscreen mode Exit fullscreen mode

The series value is what turns two independent pings into one run. Without it, a job that runs every five minutes and takes six minutes gives you interleaved starts and ends that no amount of timestamp guessing will pair up correctly.

${BASH_LINENO[0]} is the line in the caller, not the line inside fail, which is the one you actually want in the alert. Getting this backwards produces a message that always points at the same line of your error handler.

The part everyone gets wrong: the grace period

Once absence is an alert, you have to define absence, and this is where these setups earn their reputation for crying wolf.

A job scheduled every five minutes does not run exactly every five minutes. It runs when the scheduler gets to it, after the previous run of something else, on a machine whose clock drifts. If your rule is "alert if no report for five minutes and one second", you will be paged at 3am by jitter, you will mute the alert, and three weeks later the real outage will arrive muted.

Two rules that have held up for me:

  1. Grace period covers the worst normal run, not the average one. Look at your actual durations for a month. If a nightly job usually takes 4 minutes but takes 40 on the first of the month when the table is bigger, your window is built from the 40.
  2. Alert on the second miss for anything that runs more than once an hour. A single missed five-minute run is noise. Two in a row is a signal. For daily jobs, invert this: one miss is already a full day of nothing, so alert immediately.

The number you are choosing is not a technical parameter. It is the answer to "how long am I willing to not know", and that answer is different for a backup than for a cache warmer.

Three things worth watching beyond "did it run"

Once the pings are in place, they carry more than presence:

Duration drift. A backup that grew from 4 to 26 minutes over six months has not failed yet, and it is going to. Compare each run against a rolling median of the last 30. This is the cheapest early warning you will get anywhere in your infrastructure.

Overlap. If the next run starts while the previous one is still going, you have either a slow job or a data race, and both are worth knowing about. flock prevents it:

* * * * * /usr/bin/flock -n /tmp/job.lock /opt/scripts/job.sh

Enter fullscreen mode Exit fullscreen mode

Note that flock -n makes the second run exit immediately, which your monitoring will see as a missed run. That is correct behaviour and you want the alert, not silence.

Partial success. A job that processes 900 of 1000 rows and exits 0 is a lie your exit code is telling you. If a job can partially succeed, ping the count, not just the status, and alert on the count.

Where the state field trips people up

One last practical note, because it costs me time in every integration. Whatever service you use, be exact about which field carries the state. Most APIs of this kind take a status or state query parameter with a small set of values, typically along the lines of run, complete, fail, skip. Sending a state the service does not recognise is usually a 400 you will not notice, because you wrote || true earlier and quite rightly do not want a broken ping to break the job. Test the failure path once, on purpose, before you trust it:

# does a real fail ping actually produce an alert? find out on a Tuesday afternoon,
# not during an incident
curl -fsS "$PING?status=fail&msg=deliberate+test"

Enter fullscreen mode Exit fullscreen mode

An untested alert path is an assumption, and this whole article is about not assuming that silence means everything is fine.

Disclosure: I build Cronitorex, a monitoring service for exactly this: cron jobs, scheduled tasks, and HTTP/SSL endpoints, with the dead man's switch as the default rather than an add-on. The free plan covers 15 monitors without a card, which is more than enough to put the pattern above on every job you actually care about. The examples in this post are deliberately vendor-neutral, because the pattern is the useful part and it works against anything that accepts an HTTP request.

If you just want to check whether a cron expression means what you think it means, there is a cron expression parser that shows the next five runs in your own timezone. It runs entirely in your browser, so nothing you paste goes anywhere, and it needs no account.