B2B Leads API Batch Processing: Handling Large Volume Lead Extraction Without Timeout Errors

Every team that has built a B2B lead extraction pipeline hits the same wall. The first 1,000 records come back clean. The first 5,000 work fine with a simple loop. Then you push to 50,000 records and the pipeline silently fails at 2:00 AM, or worse, it throws a timeout error that kills the entire job and leaves you with zero records.

The root cause is almost never the API itself. It is the assumption that a pattern which works for small datasets will scale linearly. B2B lead APIs, like the Dievio B2B leads API, are designed to return large result sets, but they enforce constraints: rate limits, pagination depth, request timeouts, and payload size caps. When you ignore these constraints, your pipeline breaks.

HubSpot's sales prospecting research consistently shows that teams who scale their outbound efforts without corresponding infrastructure changes see a 40% drop in data quality and a 60% increase in pipeline failures. The same principle applies to batch extraction. You need a deliberate architecture, not a bigger loop.

This guide covers the engineering patterns that keep large-volume lead extraction running reliably: chunking strategies, pagination at scale, rate limit management, retry logic, queue-based architectures, memory management, and monitoring. By the end, you will have a blueprint for pipelines that handle 10x the volume you need today without timeout errors.

Chunking is the single most important pattern for avoiding timeout errors. Instead of requesting 50,000 records in one call, you break the work into smaller, independently executable units. There are two primary approaches: fixed-size chunks and time-window chunks.

Fixed-size chunks split your total record set into batches of a predetermined size. For most B2B lead APIs, a chunk size of 500 to 1,000 records per request is a safe starting point. This keeps each request within typical timeout windows (usually 30 to 60 seconds) and prevents memory buildup on the client side.

Use fixed-size chunks when:

• Your API supports offset or page-based pagination
• You have a known total record count before extraction
• Your downstream processing (CRM import, enrichment, deduplication) expects uniform batch sizes

Time-window chunks distribute requests across a defined time period. For example, instead of pulling 50,000 records in one minute, you spread the work over one hour, processing 10,000 records every 12 minutes. This approach is essential when the API enforces a per-minute rate limit.

Use time-window chunks when:

• Your API has a strict rate limit (e.g., 100 requests per minute)
• You are sharing API credentials across multiple pipelines
• Your downstream system cannot handle high-frequency writes

In practice, most production pipelines combine both strategies. You set a fixed chunk size of 500 records and then pace those chunks across a time window to stay within rate limits. This dual approach gives you predictable memory usage and compliant request rates.

Pagination is how you navigate through a large result set across multiple API calls. Two patterns dominate B2B lead APIs: offset-based pagination and cursor-based pagination. The choice between them directly impacts your ability to extract large datasets without timeout errors.

Offset-based pagination uses a page number or record offset to move through results. For example, ?page=1&limit=500, then ?page=2&limit=500. This pattern is simple to implement and works well for small datasets.

The problem is that offset-based pagination degrades as the dataset grows. At offsets beyond 10,000 records, the database must scan and skip that many rows before returning results. This increases response time, often pushing past API timeout limits. Additionally, if records are added or deleted between requests, you can miss or duplicate entries.

Cursor-based pagination uses a token or cursor that points to a specific record in the dataset. The API returns a cursor with each response, and you pass that cursor to the next request. This pattern scales linearly because the database starts scanning from the cursor position rather than from the beginning.

Most modern B2B lead APIs, including the Dievio API, support cursor-based pagination for large exports. The implementation is straightforward:

<code># Pseudocode for cursor-based pagination
cursor = None
has_more = True
batch_size = 500

while has_more:
    params = {
        "limit": batch_size,
        "cursor": cursor
    }
    response = api.get("/leads", params=params)
    data = response.json()
    
    # Process this batch
    process_batch(data["records"])
    
    # Advance cursor
    cursor = data.get("next_cursor")
    has_more = data.get("has_more", False)
</code>

For a deep dive into pagination patterns, including edge cases and error handling, see the B2B leads API pagination guide. It covers the full tradeoff analysis and provides production-ready code examples.

Rate limits are the most common cause of batch pipeline failures. When you exceed the allowed request rate, the API returns a 429 status code. How you handle that response determines whether your pipeline recovers gracefully or fails entirely.

For additional context, see HubSpot on sales prospecting.

Most well-designed APIs include rate limit information in response headers. The standard headers are:

• X-RateLimit-Limit: Total requests allowed in the current window
• X-RateLimit-Remaining: Requests remaining in the current window
• X-RateLimit-Reset: Unix timestamp when the window resets

Your pipeline should read these headers on every response and adjust its request rate dynamically. If X-RateLimit-Remaining drops below a threshold (e.g., 10% of the limit), pause and wait until the reset time.

Adaptive throttling means your pipeline self-regulates based on real-time rate limit feedback. The implementation involves three components:

1. Request pacing: Calculate the ideal interval between requests based on the rate limit. If the limit is 100 requests per minute, space requests at 600 milliseconds apart.
2. Backoff on 429: When you receive a 429 response, immediately stop all requests and wait for the duration specified in the Retry-After header or the X-RateLimit-Reset timestamp.
3. Concurrent request caps: Limit the number of concurrent API calls. Even with pacing, too many concurrent requests can overwhelm the rate limit. A cap of 5 to 10 concurrent requests is a safe starting point.

• Read rate limit headers on every response
• Pause requests when remaining limit drops below 10%
• Respect Retry-After headers on 429 responses
• Cap concurrent requests to a safe maximum
• Log rate limit violations for post-mortem analysis

Not all errors are equal. A 429 rate limit error is recoverable with a backoff. A 400 bad request error is not recoverable because the request itself is malformed. Your retry logic must classify errors and apply the appropriate strategy.

Exponential backoff increases the wait time between retries exponentially. The standard formula is:

<code>wait_time = base_delay * (2 ^ attempt_number) + random_jitter
</code>

Where base_delay is typically 1 second, and random_jitter is a random value between 0 and 1 second to prevent thundering herd problems. A maximum retry count of 3 to 5 attempts is standard before failing the batch.

For a comprehensive treatment of error classification, retry strategies, and idempotency patterns, see the companion article on B2B leads API error handling and retry architecture. It includes production retry pseudocode and edge case handling for partial batch failures.

The most resilient batch pipelines use a queue-based architecture that decouples extraction from processing. Instead of extracting records and immediately processing them in the same thread, you push extracted data into a queue and have a separate consumer process it.

The pattern has three stages:

1. Extract: The extraction worker paginates through the API, chunks records into batches, and pushes each batch onto a message queue.
2. Queue: The queue holds batches until they are consumed. This provides buffering against downstream outages and enables reprocessing of failed batches.
3. Process: The consumer pulls batches from the queue, performs enrichment, deduplication, and writes to the destination (CRM, database, or file).

This architecture prevents memory buildup because the extraction worker only holds one batch in memory at a time. It also handles downstream outages gracefully: if the CRM is down, batches remain in the queue and are processed when the CRM recovers.

For most B2B lead extraction pipelines, Redis-backed queues (like Bull or Sidekiq) or cloud message brokers (like AWS SQS or Google Pub/Sub) work well. The key requirements are persistence (messages survive restarts) and at-least-once delivery (messages are not lost on consumer failure).

This queue-based pattern is directly applicable to white-label workflows where you extract leads on behalf of multiple clients. For a deeper discussion of that use case, see the guide on lead generation API for agencies building recurring client lists.

For very large exports exceeding 100,000 records, memory management becomes critical. Loading the entire result set into memory will cause out-of-memory errors in most environments. Three approaches exist, each with different memory profiles.

The streaming approach writes each record to disk or database as it arrives, keeping memory usage constant regardless of dataset size. This is the preferred pattern for exports exceeding 100,000 records. If your API supports streaming responses (chunked transfer encoding), use it. If not, paginate with small page sizes (100-200 records) and write each page immediately.

Without monitoring, batch pipelines are black boxes. You do not know if a job is running, stuck, or failed until someone checks manually. Building observability into your pipeline from the start saves hours of debugging later.

• Records processed: Total records extracted per batch job
• Error rate: Percentage of requests that returned errors
• Duration per batch: Time to complete each batch chunk
• Queue depth: Number of batches waiting in the queue
• Rate limit hits: Count of 429 responses received

• Error rate exceeds 5% over a 5-minute window
• Queue depth exceeds 100 batches (indicates downstream bottleneck)
• Batch duration exceeds 2x the average (indicates performance degradation)
• Rate limit hits exceed 10 per minute (indicates pacing issues)

Assign a unique batch ID to each extraction job and include it in every log line. This enables you to trace a specific batch through the entire pipeline, from extraction to CRM insertion. When a batch fails, you can replay it by its batch ID without reprocessing the entire dataset.

Even with the right architecture, teams make predictable mistakes. Here is a checklist of the most common pitfalls and how to avoid them.

For additional context, see Salesforce Lead Management implementation guide.

When a batch of 500 records fails after 300 have been processed, most pipelines either discard the entire batch or silently skip the failure. The correct approach is to track which records succeeded and which failed, then retry only the failed subset.

Avoidance: Use idempotency keys on each record. If a record has already been inserted, the API or database should skip it on retry. This allows you to safely retry entire batches without duplication.

Many teams only handle 429 responses and ignore the rate limit headers on successful responses. By the time you get a 429, you have already exceeded the limit. Reading the headers proactively lets you slow down before hitting the limit.

Avoidance: Parse rate limit headers on every response and adjust request pacing dynamically.

Processing records one at a time in a single thread is simple but slow. For large datasets, single-threaded loops create bottlenecks and increase the risk of timeout errors.

Avoidance: Use concurrent requests with a capped thread pool. Start with 5 concurrent workers and adjust based on rate limit feedback.

A page size that works for one API endpoint may cause timeouts on another. Hardcoding also prevents you from adjusting to changing API behavior.

Avoidance: Make page size a configurable parameter. Start with a conservative value (e.g., 200) and increase it based on observed response times.

Without idempotency keys, retrying a failed batch can insert duplicate records. This corrupts your CRM data and erodes trust in your pipeline.

Avoidance: Generate a unique idempotency key for each record or batch. Include it in the API request so the server can detect and skip duplicates.

Here is an end-to-end architecture that incorporates all the patterns discussed in this guide. This is the architecture used by teams extracting millions of B2B leads per month through the Dievio API.

1. Trigger: A scheduled job or webhook initiates the extraction. Parameters include the search query, filters, and total record limit.
2. Paginate: The extraction worker uses cursor-based pagination to navigate through the result set. Each page contains 500 records.
3. Chunk: Pages are grouped into batches of 5 pages (2,500 records). Each batch is pushed onto a Redis queue.
4. Queue: The queue buffers batches and provides persistence. If the consumer crashes, batches remain in the queue.
5. Consume: A pool of 5 consumer workers pulls batches from the queue. Each worker processes one batch at a time.
6. Enrich: Each record is enriched with additional data (company size, industry, tech stack) using the Dievio contact enrichment API.
7. Store: Enriched records are written to the destination CRM or database. The Salesforce lead management implementation guide provides patterns for batch ingestion into Salesforce.

This architecture scales horizontally. To handle 10x the volume, you add more consumer workers and increase the queue capacity. No changes to the extraction logic are required.

For teams running recurring batch workflows for multiple clients, this architecture is the foundation. The lead generation API for agencies article covers how to extend this pattern for multi-tenant environments with isolated client data.

Batch processing for B2B lead extraction is not complicated, but it requires deliberate design. The patterns covered here chunking, cursor pagination, rate limit management, retry logic, queue-based architecture, and memory management form the foundation of pipelines that handle millions of records without timeout errors.

Start by implementing cursor-based pagination and fixed-size chunks. Add rate limit header parsing and adaptive throttling. Then layer in the queue-based architecture for decoupling and resilience. Finally, build monitoring and alerting so you know when something goes wrong.

The Dievio B2B leads API supports all the patterns discussed in this guide: cursor-based pagination, rate limit headers, configurable page sizes, and idempotency keys. It is designed for teams that need to extract large lead datasets reliably, whether for a single CRM import or a multi-tenant agency workflow.

Build Your First Outbound List to validate the segment before you commit to full outreach.