Docs / Checks / Maintenance

Updated July 31, 2026 · Maintenance · Medium severity

Production is crash-looping and no one gets notified

Seen in 16 of 450 scanned projects (4%). Based on Deep Scan runs across 450 self-hosted production apps checked for process-manager health and external uptime coverage.

Common stacks: pm2 node cron uptime-monitoring logging

How to tell you have it

Why it matters

A process manager like pm2 restarts a crashing app automatically, which is exactly what hides the problem. Users hit a half-working app, get a 502 every few seconds during the restart window, and nobody on the team sees it because the process is technically running again by the time anyone looks. Restart count is the single most informative number in `pm2 jlist` and almost no one reads it.

Debug logging left on in production is not a monitoring problem by itself, but it becomes one. When every request logs three lines of noise, the one line that says the database connection pool is exhausted gets buried between thousands of routine entries, and log rotation deletes it before a human scrolls that far back. Verbose logs also make grep-based alerting worthless because the same string matches constantly.

A scheduled job that silently fails is worse than one that crashes loudly, because the calling code often wraps the job in a try/catch that logs and moves on, so exit code 0 gets reported even though the actual task never ran. Nightly backups, report emails, and cache warmers are the classic victims - they fail for weeks and the failure is discovered only when someone needs the output that was never produced.

Without an external uptime check hitting the public URL from outside the network, an app can be fully down and still report healthy internally, because DNS, the reverse proxy, TLS, and the firewall are all outside the process manager's view. The gap between deploy time and detection time is the entire cost of an incident; a five-minute uptime check turns a two-day outage into a five-minute one.

How Heygents detects it

How to fix it

1. Read the actual restart count before assuming the app is healthy Run this on the server. A restart count in the hundreds, or an uptime under a few minutes, means the app is crash-looping right now, not just once in the past.
2. Add an external uptime check on the public URL Use a free tier of an external monitor (UptimeRobot, Better Stack, or a cron job on a different host) that hits your live URL every few minutes and alerts on non-200 or timeout. It must run outside your own infrastructure, otherwise a full network outage never triggers it.
3. Alert on restart count and disk space, not on every log line A cron that checks two numbers catches almost every real incident: pm2 restart count climbing and disk usage crossing a threshold. Keep the check dumb on purpose so it never becomes another thing to babysit.
4. Turn off debug logging in production and set rotation Gate verbose logs behind an environment variable so production only logs warnings and errors by default, then cap log file size so a flood cannot fill the disk before the alert above fires.
5. Make scheduled jobs report failure honestly Wrap cron jobs so a non-zero exit or an uncaught exception actually notifies someone, instead of being swallowed by a try/catch that logs and continues. Ping a dead-man's-switch URL on success so a job that stops running entirely (not just erroring) also gets caught.
6. Deduplicate repeated errors before they bury real ones If the same stack trace appears more than a handful of times in the same log window, collapse it to one alert with a count instead of one log line per occurrence, so a new, different error is still visible in the tail of the file.

Read the actual restart count before assuming the app is healthy

pm2 jlist | node -e "const d=JSON.parse(require('fs').readFileSync(0)); d.forEach(p => console.log(p.name, 'restarts:', p.pm2_env.restart_time, 'uptime_ms:', Date.now()-p.pm2_env.pm_uptime))"

Add an external uptime check on the public URL

*/5 * * * * curl -sf -o /dev/null -w '%{http_code}\n' https://example.com/healthz || curl -s -X POST https://hooks.example.com/alert -d 'text=example.com is down'

Alert on restart count and disk space, not on every log line

#!/bin/bash
DISK=$(df -h / | awk 'NR==2{print $5}' | tr -d '%')
RESTARTS=$(pm2 jlist | node -e "console.log(JSON.parse(require('fs').readFileSync(0)).reduce((m,p)=>Math.max(m,p.pm2_env.restart_time),0))")
if [ "$DISK" -gt 85 ] || [ "$RESTARTS" -gt 20 ]; then
  curl -s -X POST https://hooks.example.com/alert -d "text=disk ${DISK}% restarts ${RESTARTS}"
fi

Turn off debug logging in production and set rotation

{
  "log_level": "warn",
  "max_size": "20M",
  "retain": 5,
  "compress": true
}

Frequently asked questions

Do I need a full monitoring platform like Datadog to fix this?

No. For a solo developer or small team, an external uptime check plus a cron that reads `pm2 jlist` for restart count and `df -h` for disk usage covers the incidents that actually matter. A full observability platform is worth adding later, once you have enough traffic that per-request tracing pays for itself, not before.

Why does pm2 restarting the app automatically hide the problem?

Because the app looks up when you check it manually. The crash and restart cycle can happen every few seconds for days, serving errors intermittently, while a quick look at the dashboard shows a green running process. Only the restart count in `pm2 jlist` reveals the loop; process status alone does not.

How do I catch a cron job that fails silently?

Add a dead-man's-switch ping: the job calls an external URL only on successful completion, and that external service alerts you if the ping does not arrive on schedule. This catches both a job that errors out and a job that stops running entirely, which a simple exit-code check inside the job cannot do.

Why does debug logging left on in production matter for alerting?

Because alerting usually works by scanning recent log lines for error patterns, and a log flooded with routine debug output either buries the real error between thousands of noise lines or rotates it out of the file before anyone reads it. Cutting log volume is a prerequisite for any alert being trustworthy, not an optional cleanup step.

What is the minimum viable alerting setup for one person?

One external uptime check on the public URL, one cron reading pm2 restart count and disk usage every few minutes, and one webhook or email that fires when either crosses a threshold. That is three moving parts, all read-only against existing tools, and it catches the outages that actually cost money.