Enterprise Billing Integration

Executive Summary

A C# / .NET 8 Azure Function that pulls tens of thousands of records from a third-party billing API and lands them in Dataverse. The API is paginated and rate-limited; the target has to end up correct whether the sync runs cleanly, dies halfway through, or gets run twice by mistake.

The whole design turns on one decision: writes are upserts keyed on a composite alternate key derived from the source system's own identifiers. That makes every write idempotent, which in turn makes the entire batch safe to replay from any point, and removes the need for the fragile "where did we get to" bookkeeping that this class of integration usually accumulates.

The Problem

The billing data lived in a third-party platform and was needed against records in Dataverse. Bulk synchronisation across a boundary like that has three problems that all show up at scale and none of which show up in a five-record test.

The source paginates, so you are never holding the full set. The source rate-limits, so a naive loop will be throttled partway through a run and get slower the harder you push it. And the run will fail partway through at some point, a transient network fault, a deployment, a timeout, which means the interesting question is not how the sync behaves when it succeeds, but what state it leaves behind when it does not.

Answer that badly and you get duplicates, which are far more expensive than missing rows: a missing row gets picked up on the next run, while a duplicate has to be found and reconciled by a human.

The Solution

A scheduled .NET 8 isolated Azure Function runs the sync as a four-stage loop, with the correctness properties pushed down into the write rather than managed in the orchestration.

• Cursor pagination. The function pages through the source feed, holding one page at a time rather than trying to materialise the whole set in memory.
• Rate-limit back-off. Throttling responses are treated as normal operating conditions, not errors. The function backs off and retries rather than failing the run or hammering the endpoint.
• Mapping. Each source record is mapped to the target shape and validated before anything is written, so bad input fails at the boundary instead of halfway into the database.
• Composite alternate key upsert. Records are written against an alternate key composed from the source system's identifying fields. Dataverse resolves it to an insert or an update, so the function never has to ask "does this exist yet".

Architecture

One deployable, one direction, one write pattern. The function is triggered on a schedule, pages through the billing API, and upserts into Dataverse against the alternate key, which is what decides insert versus update, rather than any state the function keeps.

Pushing identity resolution into the alternate key is the whole trick. The alternative, querying for an existing record before every write, doubles the request count against the target, introduces a race between the check and the write, and still gets it wrong under retry. Letting the platform resolve identity from a key it already indexes is both faster and more correct.

Engineering Highlights

• Idempotent by construction. Every write is an upsert on a composite alternate key, so re-running the sync converges on the same state instead of duplicating it.
• Replay-safe from any point. Because no write depends on what came before it, a run that dies at page forty can simply be run again, no reconciliation pass, no manual cleanup.
• Throttling is a normal condition. Rate-limit responses trigger back-off and retry rather than a failed run, so the sync degrades in throughput rather than in correctness.
• Bounded memory. Page-at-a-time processing means the memory profile is flat regardless of how far the source set grows.
• Validated at the boundary. Malformed source records are rejected before the write, so a bad record cannot make it into the target and then have to be explained later.

Technologies

Lessons Learned

The instinct on a large sync is to make the orchestration smart: checkpoint the position, track what has been processed, resume from the last known good page. All of that is state, and all of that state can be wrong, particularly after the kind of ungraceful failure it exists to protect against.

Making the write idempotent instead makes most of that machinery unnecessary. If running the same record twice is harmless, then "where did we get to" stops being a correctness question and becomes a performance one, and performance questions are far easier to be wrong about safely.

The second lesson is that a rate limit is a specification, not a failure. Integrations that treat throttling as an error end up either fragile or aggressive; treating it as expected back-pressure and building the retry path first produces something that runs unattended.

Results & Impact

Billing data lands in Dataverse on a schedule with no manual intervention, and a failed run is recovered by running it again rather than by anyone reconciling records by hand.

The pattern has held up on every integration I have built since. Pagination, back-off and an idempotent write are not three separate features; they are the three things that decide whether a bulk sync is something you can leave running or something somebody has to watch.