Skip to content
← All posts
· Delta1 Labs LicensingKeyright

Avoiding a license revalidation stampede with jitter

When thousands of clients share a cached-validation TTL or all reconnect after an outage, their refresh calls synchronise into a thundering herd. Here is how to jitter the SDK's background revalidation so you never DDoS your own issuing service.

An online license check is cheap until everyone does it at once. A single Keyright client refreshing its cached lease against your self-hosted issuing service is a sub-second request that happens every few minutes and costs nothing. Ten thousand clients refreshing in the same second is a different event entirely — and the two situations that produce it are both ordinary, not exotic.

The first is a fixed refresh interval. If your app refreshes every fifteen minutes and a few thousand copies were rolled out in the same deploy window, their timers start phase-locked and stay that way. They do not drift apart on their own; a plain PeriodicTimer is deterministic. So every fifteen minutes, on the minute, the whole fleet calls the service together. The second is a recovery edge: your issuing service has a brief blip — a rolling upgrade, a cert rotation — and every client fails over to its cached lease. The moment the service is reachable again, all of them retry simultaneously, and the service that was just coming back up gets a thundering herd for a welcome-back present.

Both are the same failure: synchronised revalidation, a stampede you inflicted on yourself. The fix is not more capacity. It is to stop the clients from agreeing on when to call.

Key takeaway: Add randomised jitter to the SDK’s background revalidation so a fleet spreads its refreshes across the interval instead of spiking at the edges. Use full jitter for the steady-state periodic refresh and decorrelated jitter plus Retry-After for post-outage retries. Lean on the grace window so that Validate() never waits on a refresh — which is what makes jitter free to add.

Why a fixed interval never desynchronises

The intuition that “they’ll spread out eventually” is wrong for a fixed timer. Consider the naive background refresh almost everyone writes first:

// ANTI-PATTERN: a fixed interval keeps the fleet phase-locked forever.
using var timer = new PeriodicTimer(TimeSpan.FromMinutes(15));
while (await timer.WaitForNextTickAsync(ct))
    await License.RefreshAsync(ct);

PeriodicTimer fires at a steady cadence relative to when it started. If two processes started within a second of each other — which every process in a single deploy did — they stay within a second of each other indefinitely. There is no mechanism pulling them apart. The only thing that ever perturbs the phase is a refresh that takes longer than usual, and under a stampede every refresh takes longer, which tends to re-synchronise them rather than scatter them.

So the fleet’s request rate against the issuing service is not a smooth trickle of one request per client per fifteen minutes. It is a spike of ten thousand requests every fifteen minutes and near silence in between. You have built a fixed-frequency DDoS of your own service.

Synchronised refresh spikes versus jittered refresh spread across the intervalSynchronised: a fixed interval fires the whole fleet at oncetime10k req10k req10k reqt=0+15m+30mJittered: full jitter spreads the same load across each intervaltime~same total, no spike

Full jitter on the periodic refresh

The remedy for the steady-state case is to schedule each next refresh at a uniformly random point inside the interval rather than exactly one interval later. This is full jitter: the delay until the next refresh is a uniform draw over [0, interval] (or, if you want a floor so a client never hammers, over [interval/2, interval]). Within a cycle or two, a fleet that started perfectly phase-locked is smeared evenly across the window.

Here is a BackgroundService that warms the license on startup and then revalidates on a fully jittered schedule. It leans on the SDK the right way: Validate() is the pure, offline, never-throws check that serves gate decisions from the cached lease; RefreshAsync is the only call that touches the network, and it is what we jitter.

public sealed class LicenseRefreshService(
    IKeyrightClient license, ILogger<LicenseRefreshService> log) : BackgroundService
{
    static readonly TimeSpan Interval = TimeSpan.FromMinutes(15);

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        // Warm the cache once at startup so the first gate decision is fresh.
        await SafeRefreshAsync(ct);

        while (!ct.IsCancellationRequested)
        {
            TimeSpan delay = NextDelay();
            log.LogDebug("Next license refresh in {Delay}.", delay);
            try { await Task.Delay(delay, ct); }
            catch (OperationCanceledException) { break; }

            await SafeRefreshAsync(ct);
        }
    }

    // Full jitter over the second half of the interval: never sooner than interval/2,
    // never later than interval, uniformly random in between.
    static TimeSpan NextDelay()
    {
        double half   = Interval.TotalMilliseconds / 2;
        double jitter = Random.Shared.NextDouble() * half;   // [0, interval/2)
        return TimeSpan.FromMilliseconds(half + jitter);     // [interval/2, interval)
    }
}

Two things make this safe. Random.Shared is a per-process, independently seeded source, so ten thousand processes draw ten thousand uncorrelated delays — there is no shared seed to re-synchronise them. And the jitter is recomputed every cycle, not once at startup, so even if two processes happen to draw the same first delay they diverge on the next tick. The fleet stays spread.

Refreshing without ever letting a failure throw

The refresh itself must never take the app down, and it must distinguish a definitive answer from an unreachable service, because only the latter should trigger a retry. This is the same transient-vs-definitive split the SDK already draws for grace windows.

    async Task SafeRefreshAsync(CancellationToken ct)
    {
        try
        {
            RefreshResult r = await license.RefreshAsync(ct);
            switch (r.Status)
            {
                case RefreshStatus.Confirmed:
                    // Lease renewed; grace window resets. Nothing to do.
                    break;

                case RefreshStatus.Revoked:
                case RefreshStatus.Expired:
                case RefreshStatus.SeatWithdrawn:
                    // Definitive. Validate() will now report Community Edition;
                    // our existing gating disables paid features. No retry.
                    log.LogWarning("License no longer valid: {Reason}.", r.Reason);
                    break;

                case RefreshStatus.ServiceUnreachable:
                    // Transient. Cached lease still stands within its grace window.
                    // Do NOT retry in a tight loop — honour Retry-After and back off.
                    await BackoffRetryAsync(r.RetryAfter, ct);
                    break;
            }
        }
        catch (Exception ex) when (ex is not OperationCanceledException)
        {
            // A background refresh must never crash the host.
            log.LogWarning(ex, "License refresh failed; continuing on cached lease.");
        }
    }

Notice what does not happen here: a ServiceUnreachable result does not gate the app. Validate() keeps answering from the signed cached lease for the whole grace window, which is far longer than the refresh interval, so a missed refresh — or several — costs the customer nothing. That is precisely why jitter is free to add: moving the refresh a few minutes either way inside a grace window measured in days can never expire a license.

Decorrelated jitter and Retry-After for the recovery edge

The periodic jitter solves the steady state. The recovery edge needs more, because when the service comes back every client wants to retry at the same instant. Retrying at all on a fixed short delay just rebuilds the stampede. Two tools defeat it.

Honour Retry-After. When the issuing service is shedding load or coming back up, it can tell clients how long to wait. If the SDK surfaces a RetryAfter on the unreachable result, respect it as a floor — never retry sooner.

Add decorrelated jitter on top. Decorrelated jitter both backs off and randomises: each sleep is drawn from a range whose upper bound grows with the previous sleep, so repeated failures spread further and further apart instead of clustering. The classic formulation is sleep = min(cap, random_between(base, previous * 3)).

    async Task BackoffRetryAsync(TimeSpan? retryAfter, CancellationToken ct)
    {
        TimeSpan baseDelay = TimeSpan.FromSeconds(2);
        TimeSpan cap       = TimeSpan.FromMinutes(5);
        TimeSpan sleep     = baseDelay;

        for (int attempt = 0; attempt < 6 && !ct.IsCancellationRequested; attempt++)
        {
            // Decorrelated jitter: next sleep is random in [base, previous * 3], capped.
            double next = Random.Shared.NextDouble()
                        * (sleep.TotalMilliseconds * 3 - baseDelay.TotalMilliseconds)
                        + baseDelay.TotalMilliseconds;
            sleep = TimeSpan.FromMilliseconds(Math.Min(cap.TotalMilliseconds, next));

            // Server's Retry-After is a floor; never retry sooner than it asks.
            if (retryAfter is { } ra && ra > sleep) sleep = ra;

            try { await Task.Delay(sleep, ct); }
            catch (OperationCanceledException) { return; }

            RefreshResult r = await license.RefreshAsync(ct);
            if (r.Status != RefreshStatus.ServiceUnreachable) return;   // recovered or definitive
            retryAfter = r.RetryAfter;                                  // may change between tries
        }
        // Still unreachable: fall back to the periodic loop. Grace window keeps us licensed.
    }

Because each client draws its own decorrelated sequence, a fleet of ten thousand that all lost contact at the same second will not all retry at the same second. They fan out — a few probe early, most arrive spread across the backoff range — and the recovering service sees a ramp, not a wall. The ones that probe early and succeed stop retrying; the rest find the service healthy on a later, naturally staggered attempt.

Why fail-open-within-grace is what makes this safe

The whole design only works because a late refresh is harmless. If your gate decision called the network synchronously — if Validate() blocked on the issuing service — then jitter would be a luxury you could not afford, because a delayed refresh would be a delayed user. By keeping Validate() a pure offline check over the signed cached lease, and honouring that lease through a grace window far longer than the refresh interval, you decouple when you revalidate from whether the user can work right now. Revalidation becomes a background housekeeping task whose exact timing is yours to scatter however you like.

That decoupling is also what keeps a stampede from becoming an outage. If a synchronised herd does briefly overwhelm the service, the clients that time out simply keep running on their cached leases and retry later with backoff. The failure mode of “too many refreshes at once” is a few refreshes landing a little late — not a fleet of locked-out customers. You get to tune for load without ever trading away availability.

Choosing your numbers

A few rules of thumb close this out. Set the refresh interval well below the grace window — a 15-minute refresh under a 72-hour grace gives you hundreds of chances to reconnect before anything fails closed, so no single missed refresh matters. Use full jitter over the second half of the interval for the periodic loop, so refreshes are spread but no client goes much longer than one interval between checks. Cap the decorrelated backoff at a few minutes so a long outage settles into a steady, staggered re-probe rather than drifting to hours. And always treat Retry-After as a floor, because the one component that knows how much load the issuing service can take right now is the issuing service.

Do those four things and the two stampedes — the fixed-interval spike and the recovery-edge herd — both flatten into a load your self-hosted instance barely notices. The licensing check stays invisible to your customers, which is exactly where a licensing check belongs.

Try Nebula.NET

Harden your .NET code in minutes — start with the free edition.