Making automation safe to run unattended

Automation that reads data is easy. Automation that changes things, such as updating records, adjusting budgets, posting content, or charging cards, is where systems get dangerous. Run it once and it works. Run it ten thousand times a week against a third-party API, and every rare failure becomes a daily event.

I build and run services that make tens of thousands of changes a week in external systems with no human in the loop. Three habits do most of the work.

1. Idempotency: running twice must equal running once

Networks fail in the worst possible place: after the other system applied your change, but before you heard back. Your code sees an error, retries, and now the change has happened twice. Two posts. Two charges. A budget raised by 40% instead of 20%.

An operation is idempotent if doing it twice has the same effect as doing it once. Ways to get there:

Set state, don’t increment it. “Set the budget to $120” is safe to repeat. “Add $20 to the budget” is not.
Idempotency keys. Attach a unique key to each intended change. Many APIs honor these directly. If yours does not, record the key yourself before calling and check it on every retry.
Check before acting. Read the current state, compare it with the desired state, and only act on the difference.

The goal is simple: any job can crash at any point and be safely re-run from the start. Once that is true, recovering from failure becomes boring, which is exactly what you want.

2. Retries that help instead of hurt

Retries are necessary, and naive retries make outages worse. If a downstream API is struggling and every client retries instantly, the extra traffic keeps it down.

What works:

Exponential backoff with jitter. Wait longer after each failure, with some randomness so clients don’t retry in sync.
Retry only what is retryable. Timeouts and rate-limit responses, yes. “Invalid request” errors, no. Retrying a bad request a hundred times just produces a hundred failures.
Respect rate limits proactively. Track your own request rate and stay under the published limits rather than discovering them through errors.
A dead-letter queue. After a set number of attempts, move the job aside where a person can inspect it. One bad record should never block everything behind it.

Queues make all of this much easier. When work is a message on a queue, a failed attempt simply becomes visible again later, and a crashed worker loses nothing.

3. Reconciliation: trust, but verify

Even with idempotency and careful retries, your view of an external system will drift from reality. Someone edits a record by hand. An API accepts a change and then quietly drops it. A webhook never arrives.

Reconciliation is a regular job that compares what should be true with what is true in the system of record, and then fixes or flags the difference.

Treat the external system as the source of truth for its own data.
Run reconciliation on a schedule, not only when something looks wrong.
Make corrections through the same idempotent path as normal changes.
Alert on unusual amounts of drift. A sudden spike usually means something upstream changed.

This is what turns “we think it worked” into “we know it worked.”

Observability: know before your customer does

An unattended system needs to tell you when it is unhappy. At minimum, I track:

Success and failure counts per operation and per external API.
Queue depth and the age of the oldest message. A growing backlog is an early warning.
Dead-letter queue size.
Drift found by reconciliation.

Alerts go on the few signals that mean a person needs to act. Everything else goes on a dashboard.

A quick checklist

Before letting automation make changes on its own, ask:

If this job crashes halfway and restarts, is the result still correct?
Which errors are retried, how many times, and with what backoff?
Where do permanently failing jobs go, and who looks at them?
How do we find out if our records drift from the external system?
What alert fires if this stops working at 3 a.m.?

If any answer is “not sure,” the system is not ready to run unattended yet. The good news is that each one is a solvable engineering problem, and solving them up front is far cheaper than cleaning up afterward.

If you have an integration or automation that keeps breaking in quiet, expensive ways, this is the kind of problem I enjoy fixing.