Integrations
When the integration stops working
How to find out before the user does, and what to check first.
A broken integration almost never announces itself. Everything looks fine until someone asks why a supplier is missing from the ERP, and by then it has been failing for two weeks.
The four causes, by frequency
| Cause | How to spot it | Fix |
|---|---|---|
| Key revoked or expired | Everything fails at once with an auth error | Rotate the key and update it |
| Your endpoint returns an error | Notifications retry and pile up | Check your logs, not ours |
| A field changed shape | Only one kind of operation fails | Compare what is sent against what is expected |
| Nobody watches the failures | Everything looks fine and half is missing | Alert on the failed queue |
Important
The fourth does the most damage and is the only one that raises no error: if nobody watches failed notifications, the integration looks healthy while losing half the events. Alert on that queue before anything else.
The minimum to find out in time
Watch out
Do not retry in an unbounded loop. An endpoint erroring on everything plus a sender retrying indefinitely produces days of noise and hides the real fault.
›Can lost events be resent?
Retries have a window. Past it, reconcile by querying the API.
›How do I test without touching production?
Against the test environment, with its own key.
›Should I alert on every failure?
No: alert on the pattern, not the single event, or you will stop reading alerts.
A real case
The situation
An ERP has received no notifications for two weeks and nobody noticed.
What you do
- Finds its endpoint has been erroring since a deployment
- Adds an alert on the failed queue
What you get
The next failure is caught in hours instead of weeks.
The situation
The integration has been down five days and it is discovered because a file is missing.
What you do
- Sets an alert if activity stops arriving
- Reviews the call log weekly
What you get
The break is caught in hours rather than when somebody misses something.
The situation
It fails and nobody knows whether the problem is on one side or the other.
What you do
- Checks the call log with its error code
What you get
Diagnosis starts where it should look.
The situation
It is retried by hand and files get duplicated.
What you do
- Checks what arrived before reprocessing
What you get
The retry does not create what already existed.
The situation
The key expired and nobody knew.
What you do
- Records key expiries with a warning
What you get
Rotation happens before it cuts the service.
The situation
It is fixed and no record remains of what happened.
What you do
- Records the cause and what was done
What you get
Next time it is resolved in minutes.
This article answers
- the integration stopped working
- webhooks are not arriving
- api errors in production
- monitoring an integration