Key points
- Keep source pages, processing batches and delivery batches distinct.
- Persist successful retrieval before transformation and destination writes.
- Report accepted output and unresolved work as separate outcomes.
Batch processing turns a large lead-retrieval task into bounded units that can be stored, reviewed and recovered independently. The goal is a complete, explainable handoff, not the largest possible request.
Dievio's public API provides request-and-response lead search. Your application supplies the job queue, checkpoints, transformation stages and destination delivery. A client-owned batch job does not imply a native asynchronous export endpoint or webhook subscription.
Define the job before splitting it into batches
Write down the audience, desired result cap, operating budget, destination and acceptance rules. Use a stable job reference so records and failures can be traced to the same definition.
Choose the input from supported operations. Search can retrieve available contacts using documented filters; a known LinkedIn profile URL may suit profile lookup. Do not send arbitrary signup emails or company domains to an assumed generic enrichment endpoint.
For agency work, keep each client's configuration, credentials, output and suppression rules isolated. The agency API workflow covers the handoff between source retrieval and client delivery.
Keep the source page and the processing batch distinct
| Unit | Purpose | Checkpoint |
|---|---|---|
| Source page | One documented search response | Stored response and reported pagination state |
| Processing batch | A bounded set of stored records for validation or transformation | Accepted, rejected and unresolved records |
| Delivery batch | A bounded destination import or write operation | Destination acknowledgements and final outcomes |
One source page can become several delivery batches if a CRM accepts fewer records per write. Several stored source pages can also feed one processing task. Keep those choices separate from the source's pagination contract.
Retrieve and store before doing downstream work
- Load the approved job definition. Confirm filters, page size, result cap and budget.
- Request one bounded page. Dievio uses numeric
_pageand_per_pageinputs, withmax_resultsfor scope. - Validate the response envelope. Inspect success,
preview_dataand pagination state. - Persist the response. Store its job reference and metadata before advancing the source checkpoint.
- Queue validation. Transform stored records independently of the next source request.
- Stop or continue deliberately. Respect completion, empty results, inconsistent allowance and operating limits.
Use the pagination guide for the actual request shape. Numeric pages are not an incremental change feed or a guaranteed immutable snapshot.
Keep the source page stable when a batch ends
Put the domains you want to search together in the company_websites array. Keep the same domains, role filters, _per_page and max_results while following next_page. These are contact-search pages across the selected domains. Keep a separate record of which domains had results if the delivery needs account-level coverage.
For a run with _per_page set to 4 and max_results set to 6, a full first page contains four rows and the final page can contain two. Leave _per_page at 4 for both calls and use count to read how many records the response contains.
If the API returns HTTP 402 because the balance cannot cover a page, that page has not been charged. Top up and retry the same page with the same filters and limits.
Before sending a page, save its request body and a unique Idempotency-Key with the job. After a timeout or dropped connection, retry the same body with the same key within one hour of completion. A saved response includes Idempotency-Replayed: true and uses no additional credits. If the first request had no key, check the job log and credit activity before retrying. See the Search API retry rules.
Track destination writes separately. If the page is available in the job log, resume from that saved response. If a write may already have succeeded, verify the destination before appending rows again. For a visual workflow, see the n8n guide for sending search results to Google Sheets.
If you are using Apify for the source, find leads by company domain and job title and keep the same filters when continuing the result set.
Choose the next action after a failed page
| What happened | What to do |
|---|---|
| Network timeout or lost response | Retry the saved body and key within the replay window. If the API returns the stored result, it marks the response Idempotency-Replayed: true. |
HTTP 409 with Retry-After | Wait for the indicated delay and retry the same key and body. The matching request is still in progress. |
HTTP 409 without Retry-After | Compare the saved and current request bodies. Reuse the original body for recovery. Assign a new key when intentionally requesting a different page or filters. |
| HTTP 402 for insufficient credits | Top up before retrying the same body and key. The rejected page has not been charged. |
| HTTP 413 for an oversized replay response | The keyed attempt is unpaid. If you need a smaller page size, start a new pagination run with fresh keys and reconcile any rows already stored. Changing the size partway through also changes the offset. |
| CRM write failed after Search succeeded | Retry or reconcile the destination write using the saved response. Check whether the CRM already applied the previous attempt. |
The replay window is one hour after a successful page completes. After it expires, another request can spend credits. Use has_more and next_page to continue and keep a separate key per page. The reported total_count is a capped paging value. For an audience-size check, use free Preview. Search API replay rules.
Validate records with explicit outcomes
Check required fields, field types and missing values against the destination's needs. Keep a missing field separate from a malformed value, and keep uncertain identity separate from a confirmed duplicate.
- Accepted: meets the written handoff criteria.
- Needs research: identity, responsibility or a required field is unresolved.
- Excluded: fails the approved audience or suppression rules.
- Technical failure: cannot be processed because of a schema or system problem.
A record with an email is not automatically approved for outreach. Availability does not prove current mailbox reachability, role relevance or permission for every use. Define the review steps needed for the intended campaign.
Make destination retries safe
Use a destination matching strategy that your identifiers support. Document whether an incoming value may create a record, update a field or leave a reviewed value unchanged. Route ambiguous matches for review instead of silently merging people.
Keep the delivery reference stable across retries of the same logical action. Where the destination supports idempotency, use its documented contract. Otherwise, define duplicate detection and reconciliation in your application; a local job ID alone does not make a remote operation idempotent.
The CRM mapping workflow explains field ownership, missing values and activation eligibility.
Bound concurrency around the actual bottleneck
Source retrieval, validation and destination delivery can have different capacity limits. Give each stage a bounded queue and an observable backlog. Stop producing more work when the next stage cannot safely absorb it.
Do not publish an arbitrary universal batch size, requests-per-minute figure or expected throughput. Start with a controlled run and adjust from observed response size, latency, error patterns and destination constraints. See request controls for failure categories and retry boundaries.
Report completed work and unresolved work separately
| Measure | Question it answers |
|---|---|
| Returned source rows | What did paid retrieval supply? |
| Unique accepted records | What met the stated delivery criteria? |
| Pending review | What requires a decision before handoff? |
| Applied destination actions | What actually reached the intended final state? |
| Repeated requests or writes | Where did recovery create extra work or cost? |
Do not label the job complete merely because all source pages were fetched. Finish when the accepted records are accounted for and exceptions have an owner. Test interruption and recovery with local fixtures before expanding a live retrieval.



