One service talks to every provider. Our products get one kind of answer.

A customer wants 500 naira of airtime on their phone. That is the entire request.

{ "product": "airtime", "network": "mtn",
  "amount": 50000, "phone": "0801 000 0000" }

To fulfil it, our product had to know a great deal more than that. We have two products that sell airtime, so it had to know it twice.

What a product had to know

We buy airtime, data, electricity and TV from more than one provider. The providers agree on almost nothing.

Coverage differs. One rail carries nine of the twelve electricity distribution companies. The other three are on the incumbent only. "Buy electricity" is therefore not one question; it is "buy electricity for this disco, from whichever provider has it this month". Each product carried that map, and the map moved whenever a provider signed a new disco.

Vocabulary differs. The same one-gigabyte weekly bundle is a different code, a different label and a different price on every rail. The codes are not interchangeable.

incumbent:  { code: "mtn-1gb-7d",     label: "1GB Weekly",   price: 30000 }
newer rail: { code: "NR-MTN-1024-7D", label: "1GB (7 Days)", price: 29500 }

Send one provider the other's code and you are not asking for the wrong price. You are speaking the wrong language. It gets rejected, or it sells the customer something else.

Authentication differs. Basic auth on one. API key plus a signed payload on another. Each rotates on its own schedule.

Timing differs. One provider says in the HTTP response whether the purchase worked. Another answers pending and posts a webhook some minutes later. The product has to know which one it is talking to just to know whether to wait.

Failure vocabulary differs, and this is where the money is. Each provider has its own status codes. Some mean "safe to retry". Some mean "never arrived". Some mean "we may have delivered this and lost track of it", and retrying that one charges the customer twice. Classifying a code wrongly in either direction costs real money, and every product had its own copy of the classification.

Availability differs. Each provider goes down on its own schedule, so each product needed its own health checks, its own failover order, its own opinion about who was working this afternoon.

Two products, two copies of all of it. When a provider renamed a plan, both had to learn. When one started timing out on a Tuesday, both discovered it separately, at different times, with different alerts.

The product should say what it wants

The product should say what it wants, in its own vocabulary, and something else should know how to get it.

We call the dispatcher Heimdall. A product sends it a request in the product's own terms. Heimdall picks a provider that can serve it today, translates the request into that provider's vocabulary, handles that provider's authentication, waits or does not wait depending on how that provider answers, and maps the outcome back into one shared status set. If the first provider cannot serve it, Heimdall tries the next, and the product never finds out.

The product's job shrinks to: ask, then act on a normalised answer.

type Outcome =
  | { status: 'success';           providerRef: string; data?: unknown }
  | { status: 'pending';           providerRef: string }  // outcome arrives later
  | { status: 'failed_refundable'; reason: string }       // certainly not delivered
  | { status: 'failed_unknown';    reason: string };      // may have delivered. HOLD.

Four statuses, and the product only ever has to reason about these four. Which provider produced them, and what that provider's own code was, is Heimdall's business.

The decision that carried everything else

When a purchase needs following up, the system goes back to whoever it asked and says: what became of this?

That follow-up has to go to the right party. If the product asked Heimdall, ask Heimdall. If it asked a provider directly, ask that provider. Obvious enough that nobody writes it down, which is how it nearly cost us a double charge.

Our transaction records had always stored which provider served a purchase. They had never stored who we asked, because until the dispatcher existed those were the same fact.

before Heimdall         after Heimdall, first attempt
 
served_by: incumbent    served_by: incumbent   <- true: served via the incumbent
                        asked:     (nothing)   <- so a follow-up goes... where?

Here is the failure, step by step. A product sends a purchase through Heimdall. Heimdall serves it via the incumbent and mints its own reference. The product records served_by: incumbent and the reference. Someone later switches that product back to calling the incumbent directly. A retry job wakes up, sees served_by: incumbent, and asks the incumbent about the reference. The incumbent has never seen it, because Heimdall created it. It answers with its "no such transaction" code. The product maps that code to failed_refundable, which means "certainly not delivered", which is exactly the signal that authorises a re-send.

One debit. Two deliveries. Every step is correct. The record simply cannot express that the reference in its hand belongs to a conversation the product did not have.

So the record carries both facts now:

served_by: incumbent
asked:     dispatcher     <- follow-ups go here, whatever routing says now

And the rule:

A follow-up goes to whoever you asked. The routing setting only decides where new purchases go.

That rule is what made the migration boring. We moved a live payments product onto Heimdall one category at a time, airtime then data then electricity then TV, and at any point we could have moved a category straight back with purchases still in flight. Every in-flight purchase would still be followed up with whoever was holding it. Without the second field, a rollback is a double charge waiting for a retry to fire.

Two things we got wrong before we got that right:

Write the marker before the request leaves, not when the answer returns. A marker written on the answer is missing in exactly the case it exists for: the process dying mid-purchase. And if writing it fails, refuse the purchase. Sending money you cannot record is the one state that produces a double delivery.

A missing marker means "asked the provider directly", never "nothing was sent". Every reference we had ever minted went to a provider, so a row with no marker is a row the provider holds. Read it the other way and you manufacture a failure and re-send a live purchase.

What went wrong on the way

Six false alarms from a probe that hangs

The incumbent publishes no health endpoint. There is no URL you can call to ask whether it is up, let alone whether it can still sell a particular disco's electricity this afternoon. The only way to find out is to ask it to do some real work and see what comes back, and the one piece of that work that moves no money is verification. So that is the probe: every few minutes, for every electricity and TV service on every rail, ask about an account that does not exist. A structured "no such meter" reply proves the rail answered and the biller behind it is reachable, and nobody's real account is involved. Airtime and data have nothing to verify, so for those the record of real purchases is the signal, which on a busy service is the better one anyway.

Monitoring the provider is one more job the provider leaves to you, and it used to be done twice, badly, with two different answers. Now it is done once, and because Heimdall is the only place that knows which rail serves which service, its answer is the useful one: not "this disco is unhappy" but "this rail cannot verify this disco, and no other rail is configured for it". It shows on a status page and raises an alert when a cell flips, before a customer's purchase finds out the hard way.

For a long time that was all it did. It reported, and a human decided. The rule was written into the code: nothing reads this to route a purchase, because a prober that could reshape routing would let one flaky biller take a provider out of service with nobody deciding it. That rule was right about the risk and wrong about the remedy, and we changed it once we could answer the risk instead of avoiding it. A provider that is failing one service is now tried last for that service, and only for that one. It is a re-order, never a switch: nothing is disabled, nothing is removed, and a service with only one provider still goes to that provider first. Being wrong costs a purchase taking the slower road. Being slow costs a customer.

The probe has a trap of its own. A valid account resolves from an index. An invalid one makes some billers do a full search, which sometimes hangs. Our classifier called a hang down.

// before
catch (e) { status = 'down' }
 
// after
catch (e) {
  status = isTimeout(e) ? 'unknown' : 'down';
  // unknown: absence of evidence. It is not evidence of failure,
  // and it must not page anyone or flip a routing decision.
}

We fixed it, then found the same mistake one layer up, in the deadline wrapped around the prober itself. The confusion between "did not answer" and "answered no" lives at every layer that can give up waiting. Fixing it in one place is not fixing it.

Two correct halves that lied at the seam

Heimdall's health record preserves staleness on purpose. When a probe cannot determine anything, it writes nothing, so the record keeps its last known status and its last known checked_at.

The product that reads it writes whatever status it is handed into its own table, and the database layer stamps updated_at on every write.

Heimdall's record (correct)          product's copy (also correct)
 
status:     healthy                  status:     healthy
checked_at: 09:14   <- frozen        updated_at: 14:31   <- bumped every tick
 
                    outage began 09:15. it is now 14:31.
                    dashboard reads: "healthy, checked 30s ago"

One side preserved the freshness signal by not writing. The other destroyed it by writing. Neither was wrong. The property that broke lived in the join between them, which is exactly why no test on either side could have caught it. It rendered a five-hour outage as healthy.

Test the seam, not the halves.

A provider said no, and we wrote down "maybe"

Of the four statuses, two carry the whole weight: one means the purchase certainly did not happen, the other means it might have. The first authorises a refund and a second attempt elsewhere. The second holds the money and waits for a person.

A rail sent us an explicit failure for six purchases in one batch, naming itself as the cause. We filed all six as "might have". The money sat, nothing retried, and a person had to release it by hand hours later.

The reason we had filed it that way was caution, and it was reasonable when it was written: that rail documents no signal for whether it returned our float, so every failure it reported was treated as ambiguous. But "we cannot tell whether you were charged" is a different sentence from "we never heard back", and we had collapsed them into one. A rail that says no is not a rail that said nothing.

There was a second half to it, and it was the more expensive one. Even once the failure was classified correctly, nothing could act on it. Moving a purchase to another provider only ever happened at the moment of asking, and that rail answers late, by callback, minutes after the request returns. So the failure mode that provider actually has was the one case we had no path for. Every outage found the gap; the classification just decided how long the money sat in it.

The answer was written for us, and a customer read it

One of our two products talks to people in a chat app. When a purchase is still in flight it sends a short message saying so, and that message was built by quoting whatever the provider had told us.

That worked for years, because what a provider tells you is a sentence. After the change it was no longer a sentence. The dispatcher answers in one vocabulary, and that vocabulary is compact and machine-shaped, closer to pending_unconfirmed than to "we are still waiting on the disco".

Chat platforms read certain characters as formatting. The underscore in the middle of that status opened a styling span that never closed, so the platform rejected the whole message as malformed. The rejection threw, and the error handler did what error handlers do: it cleared the conversation and started again.

So a customer who had just spent a large amount received nothing at all. No confirmation, no pending notice, no error. They went back to the beginning, walked the entire purchase again, and re-entered their meter number. They gave up rather than paying twice. The next person might not.

The function that built that string carries a comment saying it is for operators only and is never shown to a customer. The comment says it was checked. It was already untrue when it was written.

An assertion written as a comment is not an invariant, because nothing can falsify it. The test we added does not look for today's format; it drives the purchase through every outcome and asserts that no customer-facing message contains the provider's status, the rail's name or its code, whatever those happen to be.

A test that could not fail

Late on we un-pinned meter and decoder verification from a single provider. That meant a field recording "which provider's vocabulary is this code in" could now hold a second value. We wrote the fix, wrote the test, then mutated the code back to check the test would catch a regression.

The test stayed green with the bug in place. At three separate sites.

The list of providers whose verify response we could parse had exactly one entry. So the correct code and the broken code evaluated to the same string on every reachable path. No test written against current behaviour could distinguish them.

The fix was to make the test push a second entry into that list, constructing the future the code was written for. Where a field records which of several things did something and there is currently one candidate, the test is decorative until it widens the set.

The rules we kept

Heimdall never moves money. It reports a typed outcome. The product decides about refunds. Two systems that can each decide a refund will eventually both decide it on the same transaction.

A provider call has three outcomes, not two. Accepted, rejected, unknown. Collapsing unknown into rejected is how you refund a purchase that went through. Unknown holds the money and asks a human. And the mistake runs both ways: collapsing rejected into unknown is how you strand money that was never at risk. Caution in the wrong direction is still a wrong answer, it just costs the customer time instead of costing you cash, so it goes unnoticed for longer.

Fail over only when the first provider took nothing or certainly delivered nothing. Never when it might be holding the customer's purchase.

Whoever will make the purchase must be the one that showed the menu. A plan code belongs to the provider that published it. If the bundle list came from one provider and the purchase goes to another, you will eventually sell someone the wrong thing.

Who you asked and who served are two facts. Store both. This is the one that lets you change your mind mid-migration.

Where it ended up

One service owns every conversation with a provider, for both products: buying, following up, listing plans, verifying a meter or decoder. Nothing else in the estate talks to a rail.

One improvement we did not plan. Decoder verification used to be pinned to one provider, because the other cannot check a decoder without knowing which package you are buying, and you have not chosen yet. Heimdall already models "this provider cannot answer this question", so it asks the next one. A verification outage used to block every electricity and TV purchase. Now it falls through.

And "which provider is better", the question that started this, is a page instead of a database dump. The numbers on it deserve one caveat.

rail         vends    failed   rate
incumbent    14,354   862      6.0%
newer rail   37       7        19.0%    <- 37 is not a sample

That was the comparison that got us moving. It is a real measurement of a sample too small to mean anything, and the page says so: a rate over a handful of purchases is flagged rather than ranked, and a provider with nothing completed says "no vends" in words rather than rendering zero percent. Absent is not zero. A dashboard that forgets that will rank a provider it has never tried.

The rule we took from it

Most of what went wrong here was a correct component making a correct claim about something that had changed underneath it. A record that said who served a purchase, read as who to ask. A status fresh by one clock and five hours old by another. A test that stayed green because the set it was written for still had one member.

If your products each know how to talk to a provider, count the places that have to change when that provider renames one plan. If the answer is more than one, you have this problem. You have just not been charged for it yet.