Skip to content

B2B Leads API Batch Processing: Chunks and Recovery

Design application-owned lead batches with separate source pages, validation states and destination deliveries.

Dievio Team · Growth Systems · August 22, 2026 · 4 min read

Contact parcels enter a storage shelf before a separate delivery conveyor, while one parcel waits aside beneath Store Each Batch Before Delivery.

Key points

  • Keep source pages, processing batches and delivery batches distinct.
  • Persist successful retrieval before transformation and destination writes.
  • Report accepted output and unresolved work as separate outcomes.

Batch processing turns a large lead-retrieval task into bounded units that can be stored, reviewed and recovered independently. The goal is a complete, explainable handoff, not the largest possible request.

Dievio's public API provides request-and-response lead search. Your application supplies the job queue, checkpoints, transformation stages and destination delivery. A client-owned batch job does not imply a native asynchronous export endpoint or webhook subscription.

Define the job before splitting it into batches

Write down the audience, desired result cap, operating budget, destination and acceptance rules. Use a stable job reference so records and failures can be traced to the same definition.

Choose the input from supported operations. Search can retrieve available contacts using documented filters; a known LinkedIn profile URL may suit profile lookup. Do not send arbitrary signup emails or company domains to an assumed generic enrichment endpoint.

For agency work, keep each client's configuration, credentials, output and suppression rules isolated. The agency API workflow covers the handoff between source retrieval and client delivery.

Keep the source page and the processing batch distinct

Three units that should not share an ambiguous counter
UnitPurposeCheckpoint
Source pageOne documented search responseStored response and reported pagination state
Processing batchA bounded set of stored records for validation or transformationAccepted, rejected and unresolved records
Delivery batchA bounded destination import or write operationDestination acknowledgements and final outcomes

One source page can become several delivery batches if a CRM accepts fewer records per write. Several stored source pages can also feed one processing task. Keep those choices separate from the source's pagination contract.

Retrieve and store before doing downstream work

  1. Load the approved job definition. Confirm filters, page size, result cap and budget.
  2. Request one bounded page. Dievio uses numeric _page and _per_page inputs, with max_results for scope.
  3. Validate the response envelope. Inspect success, preview_data and pagination state.
  4. Persist the response. Store its job reference and metadata before advancing the source checkpoint.
  5. Queue validation. Transform stored records independently of the next source request.
  6. Stop or continue deliberately. Respect completion, empty results, inconsistent allowance and operating limits.

Use the pagination guide for the actual request shape. Numeric pages are not an incremental change feed or a guaranteed immutable snapshot.

Keep the source page stable when a batch ends

Put the domains you want to search together in the company_websites array. Keep the same domains, role filters, _per_page and max_results while following next_page. These are contact-search pages across the selected domains. Keep a separate record of which domains had results if the delivery needs account-level coverage.

For a run with _per_page set to 4 and max_results set to 6, a full first page contains four rows and the final page can contain two. Leave _per_page at 4 for both calls and use count to read how many records the response contains.

If the API returns HTTP 402 because the balance cannot cover a page, that page has not been charged. Top up and retry the same page with the same filters and limits.

Before sending a page, save its request body and a unique Idempotency-Key with the job. After a timeout or dropped connection, retry the same body with the same key within one hour of completion. A saved response includes Idempotency-Replayed: true and uses no additional credits. If the first request had no key, check the job log and credit activity before retrying. See the Search API retry rules.

Track destination writes separately. If the page is available in the job log, resume from that saved response. If a write may already have succeeded, verify the destination before appending rows again. For a visual workflow, see the n8n guide for sending search results to Google Sheets.

If you are using Apify for the source, find leads by company domain and job title and keep the same filters when continuing the result set.

Dievio

Your workflow. Powered by Dievio.

Bring B2B contact search and LinkedIn enrichment into the tools you already use.

Explore Dievio API

Search contacts · Enrich profiles · Export results

Dievio APIExample
POST /api/public/search{ "job_titles": ["Founder"], "max_results": 10 }
Contacts ready for your workflow

Choose the next action after a failed page

How to recover a keyed Search API request
What happenedWhat to do
Network timeout or lost responseRetry the saved body and key within the replay window. If the API returns the stored result, it marks the response Idempotency-Replayed: true.
HTTP 409 with Retry-AfterWait for the indicated delay and retry the same key and body. The matching request is still in progress.
HTTP 409 without Retry-AfterCompare the saved and current request bodies. Reuse the original body for recovery. Assign a new key when intentionally requesting a different page or filters.
HTTP 402 for insufficient creditsTop up before retrying the same body and key. The rejected page has not been charged.
HTTP 413 for an oversized replay responseThe keyed attempt is unpaid. If you need a smaller page size, start a new pagination run with fresh keys and reconcile any rows already stored. Changing the size partway through also changes the offset.
CRM write failed after Search succeededRetry or reconcile the destination write using the saved response. Check whether the CRM already applied the previous attempt.

The replay window is one hour after a successful page completes. After it expires, another request can spend credits. Use has_more and next_page to continue and keep a separate key per page. The reported total_count is a capped paging value. For an audience-size check, use free Preview. Search API replay rules.

Validate records with explicit outcomes

Check required fields, field types and missing values against the destination's needs. Keep a missing field separate from a malformed value, and keep uncertain identity separate from a confirmed duplicate.

  • Accepted: meets the written handoff criteria.
  • Needs research: identity, responsibility or a required field is unresolved.
  • Excluded: fails the approved audience or suppression rules.
  • Technical failure: cannot be processed because of a schema or system problem.

A record with an email is not automatically approved for outreach. Availability does not prove current mailbox reachability, role relevance or permission for every use. Define the review steps needed for the intended campaign.

Make destination retries safe

Use a destination matching strategy that your identifiers support. Document whether an incoming value may create a record, update a field or leave a reviewed value unchanged. Route ambiguous matches for review instead of silently merging people.

Keep the delivery reference stable across retries of the same logical action. Where the destination supports idempotency, use its documented contract. Otherwise, define duplicate detection and reconciliation in your application; a local job ID alone does not make a remote operation idempotent.

The CRM mapping workflow explains field ownership, missing values and activation eligibility.

Bound concurrency around the actual bottleneck

Source retrieval, validation and destination delivery can have different capacity limits. Give each stage a bounded queue and an observable backlog. Stop producing more work when the next stage cannot safely absorb it.

Do not publish an arbitrary universal batch size, requests-per-minute figure or expected throughput. Start with a controlled run and adjust from observed response size, latency, error patterns and destination constraints. See request controls for failure categories and retry boundaries.

Report completed work and unresolved work separately

MeasureQuestion it answers
Returned source rowsWhat did paid retrieval supply?
Unique accepted recordsWhat met the stated delivery criteria?
Pending reviewWhat requires a decision before handoff?
Applied destination actionsWhat actually reached the intended final state?
Repeated requests or writesWhere did recovery create extra work or cost?

Do not label the job complete merely because all source pages were fetched. Finish when the accepted records are accounted for and exceptions have an owner. Test interruption and recovery with local fixtures before expanding a live retrieval.

Keep reading

Build your first list.

100 free credits. No credit card required.

Start free