Donation infrastructure has the unusual property that downtime literally costs nonprofits money in lost contributions. A five-minute outage during a peak year-end hour can vaporize five-figure revenue that will not come back. That reality sets the bar for availability very high — much higher than a typical B2B SaaS surface.

This is a walkthrough of the engineering principles that shape a high-availability donation workflow, drawn from running match-tracking pipelines that process millions of transactions a year.

Idempotent Everything

Every write path — donations, matches, disbursements, webhook callbacks — must be idempotent. Assume every request will be retried at least once, sometimes many times, and design the system so that duplicate delivery cannot double-charge a donor or double-count a match.

Practically, that means every inbound event carries a stable external ID, and the ingestion layer refuses to write the same external ID twice.

Separate Read and Write Paths

Donation workflows have wildly different SLAs for reads and writes. A donor completing checkout must succeed within milliseconds. An admin generating a reconciliation report can tolerate seconds. Building both against the same primary database is a recipe for peak-hour incidents.

Split them. Writes go to the primary. Reads go to a replica or a purpose-built read model. When the read side is slow, checkout still works.

Replay Queues Instead of Direct Calls

When a donation triggers a match request to a third-party CSR portal, do not call the portal synchronously. Enqueue the request. A worker drains the queue, calls the portal, records the result, and retries on failure with exponential backoff.

This does three things at once:

  • Isolates portal outages from your checkout success rate
  • Gives you a natural replay mechanism when a portal recovers
  • Provides an audit trail of every attempt and outcome

Three-Stage Validation

Every employer match record in our pipeline passes through three validation stages before it's considered trustworthy:

  1. Schema. Does the payload have the fields we expect, in the shapes we expect?
  2. Business rules. Does the match amount fall within the employer's stated policy? Is the donor eligible?
  3. Cross-source reconciliation. Does the match record from the portal agree with the donation record from the donation form?

Failures at any stage route to a human review queue with the raw payload attached. Nothing gets quietly dropped.

Observability Is a Feature

You cannot operate a donation workflow you cannot see. Minimum instrumentation:

  • Checkout success rate, broken down by payment processor and geography
  • Match submission latency and success rate, per employer portal
  • Queue depth and drain rate for every async worker
  • Reconciliation lag between donation, match request, and disbursement

Every dashboard should have a matching alert. Every alert should have a runbook.

Design for the Recovery, Not the Happy Path

Every donation platform has good days. The teams that win are the ones whose systems degrade gracefully on bad ones — where a portal outage doesn't take down checkout, a bad deploy can be rolled back in minutes, and a mis-classified match can be fixed with a replay instead of a database surgery. Build for the recovery, and the happy path takes care of itself.