B2B API Extraction Runbook: Pagination and Rate Limits

Pagination and request controls solve different problems. Pagination chooses the next part of a lead result set; concurrency and retry rules control how much work your integration starts and how it responds to failure.

For Dievio, build around numeric pages and a separate API credit balance. Do not invent a universal requests-per-minute limit, daily reset or rate-limit response header when the current contract does not document one.

A credit error is not evidence of temporary traffic throttling. Dievio's current public-search route can return 402 when the request cannot proceed with the available API credits. Repeating the same call in a tight loop will not restore that balance.

Public search accepts _page, _per_page and max_results. Its response includes preview_data, has_more and next_page. The pagination implementation guide explains the checkpoint loop.

Keep filters and page size fixed within one job. Store the successful response before recording the page as complete. An opaque cursor and a numeric page are different contracts; neither should be assumed to guarantee a stable snapshot unless that guarantee is documented.

Start with a deliberately limited job and enough visibility to explain every source attempt. Increase parallel work only after observing latency, errors, account balance and downstream queue growth for the actual audience.

• Set a maximum number of concurrent source requests.
• Keep source retrieval separate from CRM write workers.
• Bound request time, retry attempts and the total job duration.
• Stop automatically on repeated unexplained responses or a budget boundary.
• Give paused jobs an owner and a clear recovery condition.

A faster request loop can increase cost without improving accepted output. Measure the whole handoff, including duplicates, missing fields and review effort, rather than celebrating raw rows per second.

Some providers return 429 or a Retry-After value. Follow the response and its documentation when present; do not assume Dievio always emits those fields. The HTTP specification defines the retry field as a delay or an HTTP date.

When the same keyed page is still running, Search returns HTTP 409 with Retry-After. Wait for that delay, then retry with the original key and parameters. HTTP 409 without that header signals that the key was used with different parameters. Use a fresh key for a different logical page. Read the Search API retry rules.

For recoverable failures, increase the delay between attempts, add jitter where useful and cap both attempts and elapsed time. An endless retry loop is a stalled job, not resilience.

A circuit breaker can pause new attempts after a sustained failure pattern. Keep its recovery probe small and observable. Do not restart every pending page at once when the pause ends.

For implementation details, read the error handling and retry guide. Record provider-specific behavior separately from general HTTP design.

A successful search followed by a failed CRM import leaves useful source data in your storage. Retry the failed write using that stored response when its use remains appropriate. A second search can return overlapping records and spend credits again.

Store source attempt references, returned row counts, checkpoints and destination outcomes. Redact credentials and unnecessary contact details from logs. The credit consumption guide should be read alongside the operation's actual billing rules.

1. Reconcile requested scope, stored pages and reported completion.
2. Count unique accepted records and unresolved identity conflicts.
3. Check repeated source requests and why each happened.
4. Compare available balance and observed paid usage.
5. Inspect destination backlog and failed actions.

Scale the bottleneck you can demonstrate. If the destination is already behind, increasing source concurrency only adds pressure and makes recovery harder to reason about.