Definition
- Chunked processing -- break the dataset into fixed-size pages using keyset pagination or a scrollable
Stream, so you never hold the entire result set in memory at once. - Flush + clear -- after each chunk call
entityManager.flush()thenentityManager.clear()so processed entities detach and become eligible for garbage collection. - Per-chunk transactions -- each chunk commits independently, so completed chunks survive a crash and do not roll back with a later failure.
- Checkpoint cursor -- record the last processed id after each chunk so the job can resume from where it stopped.
- Throttling -- use bounded reads,
jdbc.batch_size, a small pause between chunks, and schedule off-peak to avoid starving live traffic. - Skip and log -- catch exceptions on individual records, log them for review, and continue processing the rest of the chunk.
National Bank of India must recalculate interest on ten million accounts every night. Pulling all ten million files onto one desk at once collapses the desk (out of memory). The fix is to work through them a tray of five hundred at a time, finishing and filing each tray before fetching the next (chunking), stamping a bookmark after each tray so a crash can resume (checkpoint), not overwhelming the archive clerk with requests (throttling), and setting aside any unreadable file to review later instead of abandoning the whole night's work (skip and log).
Understanding the Concepts
What is the problem with a naive @Scheduled job on a large dataset?
A naive scheduled job calls repository.findAll(), which loads every row into a List and every entity into the persistence context with a dirty-checking snapshot. For millions of rows this exhausts the heap and causes OutOfMemoryError. Even if memory holds, a single-threaded loop over millions of rows can run for hours, overlap the next trigger, and starve the database.
package com.bank.batch;
import com.bank.model.Account;
import com.bank.repository.AccountRepository;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.util.List;
@Component
public class NaiveInterestJob {
private final AccountRepository accountRepo;
public NaiveInterestJob(AccountRepository accountRepo) {
this.accountRepo = accountRepo;
}
@Scheduled(cron = "0 0 2 * * *")
public void accrueInterest() {
// WRONG: loads every row and snapshot into the heap
List<Account> all = accountRepo.findAll();
for (Account a : all) {
a.accrueInterest(); // dirty checking queues UPDATE
}
// flush at commit -- but memory already exploded
}
}
Expected output for 10 million accounts:
java.lang.OutOfMemoryError: Java heap space
How do you fix the memory problem with chunked processing?
The fix is keyset pagination: fetch a bounded page, process it, flush and clear, then fetch the next page using the last id seen. After each chunk, flush() pushes SQL to the database and clear() detaches all entities so the garbage collector reclaims them.
package com.bank.batch;
import com.bank.model.Account;
import com.bank.repository.AccountRepository;
import jakarta.persistence.EntityManager;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import java.math.BigDecimal;
@Component
public class ChunkedInterestJob {
private static final int CHUNK = 500;
private final AccountRepository accountRepo;
private final EntityManager em;
public ChunkedInterestJob(AccountRepository accountRepo, EntityManager em) {
this.accountRepo = accountRepo;
this.em = em;
}
@Scheduled(cron = "0 0 2 * * *")
public void accrueInterest() {
String cursor = "";
Page<Account> page;
do {
page = accountRepo.findByIdGreaterThan(cursor, PageRequest.of(0, CHUNK));
for (Account a : page.getContent()) {
a.accrueInterest();
}
em.flush(); // push this chunk's UPDATEs to the database
em.clear(); // detach all entities -- free memory
if (!page.isEmpty()) {
cursor = page.getContent().get(page.getContent().size() - 1).getId();
}
} while (!page.isEmpty());
}
}
Expected behavior:
- Memory stays flat in a sawtooth pattern (rises during chunk, drops after clear)
- 10 million accounts processed in 20,000 chunks of 500
How do you handle long execution and prevent overlapping runs?
A single run over millions of rows can last hours and overlap the next trigger. By default Spring's single-threaded scheduler will not start the next run until the current finishes, but if you use a pool you must guard with a lock. Bound the work per run and commit per chunk so progress is durable.
package com.bank.batch;
import net.javacrumbs.shedlock.spring.annotation.SchedulerLock;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
@Component
public class GuardedInterestJob {
@Scheduled(cron = "0 0 2 * * *")
@SchedulerLock(name = "nightlyInterest",
lockAtMostFor = "PT2H",
lockAtLeastFor = "PT5M")
public void accrueInterest() {
// Only one instance runs; lock auto-releases after 2 hours if crashed
interestService.processAllChunks();
}
}
Expected behavior:
- Instance A acquires the lock and runs the job
- Instance B finds the lock held and skips
- If Instance A crashes, lock auto-releases after 2 hours
How do you throttle database load?
Unbounded queries and tight loops saturate the database and starve live banking traffic. Use bounded chunk reads, a sensible fetchSize, batched writes with jdbc.batch_size, and a small pause between chunks.
package com.bank.batch;
import jakarta.persistence.EntityManager;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.util.List;
@Service
public class ThrottledBatchService {
private static final int CHUNK = 500;
private static final long THROTTLE_MS = 100;
private final EntityManager em;
public ThrottledBatchService(EntityManager em) {
this.em = em;
}
public void processAllChunks() {
String cursor = "";
List<Object[]> ids;
do {
ids = processOneChunk(cursor);
if (!ids.isEmpty()) {
cursor = (String) ids.get(ids.size() - 1)[0];
}
try { Thread.sleep(THROTTLE_MS); } catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
} while (!ids.isEmpty());
}
@Transactional
public List<Object[]> processOneChunk(String cursor) {
// Each chunk is its own transaction
// Use bounded reads, batched writes, and a fetch size
return List.of(); // placeholder
}
}
Expected behavior:
- 100ms pause between chunks gives the database breathing room
- Live banking traffic is not starved during the batch window
How do you handle errors without losing progress?
If the job runs as one giant transaction and fails at row nine million, everything rolls back and the night is wasted. Use per-chunk transactions so completed chunks stay committed, and skip-and-log poison records.
package com.bank.batch;
import com.bank.model.Account;
import jakarta.persistence.EntityManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import java.util.List;
@Service
public class ResilientChunkProcessor {
private static final Logger log = LoggerFactory.getLogger(ResilientChunkProcessor.class);
private final EntityManager em;
public ResilientChunkProcessor(EntityManager em) {
this.em = em;
}
@Transactional
public void processChunk(List<Account> chunk) {
for (Account a : chunk) {
try {
a.accrueInterest();
} catch (Exception e) {
log.warn("Skipping account {}: {}", a.getId(), e.getMessage());
// Record the failure for later review
}
}
em.flush();
em.clear();
}
}
Expected output:
WARN Skipping account SAV8821: Division by zero in rate calculation
-- Job continues with remaining accounts --
How do you make the job resumable with a checkpoint?
Record the last processed cursor after each chunk commit. On restart, read the checkpoint and resume from that point. This makes the job idempotent and resumable.
package com.bank.batch;
import com.bank.model.JobCheckpoint;
import com.bank.repository.JobCheckpointRepository;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
@Service
public class CheckpointService {
private final JobCheckpointRepository checkpointRepo;
public CheckpointService(JobCheckpointRepository checkpointRepo) {
this.checkpointRepo = checkpointRepo;
}
@Transactional
public void saveCheckpoint(String jobName, String lastCursor) {
JobCheckpoint cp = checkpointRepo.findByJobName(jobName)
.orElse(new JobCheckpoint(jobName));
cp.setLastCursor(lastCursor);
checkpointRepo.save(cp);
}
public String loadCheckpoint(String jobName) {
return checkpointRepo.findByJobName(jobName)
.map(JobCheckpoint::getLastCursor)
.orElse("");
}
}
Expected behavior:
- Job crashes after processing 500,000 accounts
- Checkpoint table has lastCursor = 'SAV500000'
- Next run resumes from SAV500000, not from the beginning
When should you use Spring Batch instead of a hand-written loop?
For genuinely large, restartable, auditable batch work, Spring Batch is the production-grade answer. It provides chunk-oriented processing, a JobRepository that persists progress, restartability, skip/retry policies, and metrics out of the box.
package com.bank.batch;
import com.bank.model.Account;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.item.database.JpaPagingItemReader;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.transaction.PlatformTransactionManager;
@Configuration
public class InterestBatchConfig {
@Bean
public Job interestJob(JobRepository jobRepository, Step interestStep) {
return new JobBuilder("nightlyInterest", jobRepository)
.start(interestStep)
.build();
}
@Bean
public Step interestStep(JobRepository jobRepository,
PlatformTransactionManager txManager) {
return new StepBuilder("interestStep", jobRepository)
.<Account, Account>chunk(500, txManager)
.reader(accountReader())
.processor(account -> { account.accrueInterest(); return account; })
.writer(chunk -> { /* dirty checking handles UPDATEs */ })
.faultTolerant()
.skip(Exception.class).skipLimit(100)
.build();
}
@Bean
public JpaPagingItemReader<Account> accountReader() {
JpaPagingItemReader<Account> reader = new JpaPagingItemReader<>();
reader.setQueryString("SELECT a FROM Account a ORDER BY a.id");
reader.setPageSize(500);
return reader;
}
}
Expected behavior:
- Spring Batch processes 500 accounts per chunk
- Progress persisted in BATCH_STEP_EXECUTION table
- On crash, restart resumes from the last committed chunk
- Up to 100 bad records skipped without aborting
What are common mistakes when processing large datasets in scheduled jobs?
The most common mistake is calling findAll() inside a scheduled job. The second is running the entire job in one transaction so a late failure rolls back everything.
// WRONG: one giant transaction, no chunking, no error handling
@Scheduled(cron = "0 0 2 * * *")
@Transactional
public void naiveJob() {
List<Account> all = accountRepo.findAll(); // OOM
for (Account a : all) {
a.accrueInterest(); // one failure rolls back everything
}
}
// CORRECT: chunked, per-chunk transactions, skip-and-log, checkpoint
@Scheduled(cron = "0 0 2 * * *")
public void resilientJob() {
String cursor = checkpointService.loadCheckpoint("interest");
Page<Account> page;
do {
page = chunkProcessor.processChunk(cursor); // own TX
if (!page.isEmpty()) {
cursor = page.getContent().get(page.getContent().size() - 1).getId();
checkpointService.saveCheckpoint("interest", cursor);
}
} while (!page.isEmpty());
}
Naive approach risks:
1. OutOfMemoryError on large datasets
2. Hours-long transaction holding locks
3. Single failure rolls back all progress
4. No resumability after crash
Correct approach:
1. Flat memory via flush+clear
2. Per-chunk commits bound lock duration
3. Skip-and-log keeps the job running
4. Checkpoint enables crash recovery
What should a senior engineer know about large scheduled jobs?
A senior engineer knows that the @Scheduled method itself should delegate to a service quickly and not hold the scheduler thread doing heavy work inline. They configure idempotent updates so re-running a chunk never double-counts. They monitor the job with metrics and alerts on both missed runs and duplicate runs. They run heavy jobs against a read replica where possible and schedule off-peak. They use Spring Batch when the dataset is large enough to warrant restartability, skip/retry, and audit.
package com.bank.batch;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.stereotype.Service;
@Service
public class MonitoredInterestService {
private final Timer chunkTimer;
public MonitoredInterestService(MeterRegistry registry) {
this.chunkTimer = Timer.builder("interest.chunk.duration")
.description("Time per interest chunk")
.register(registry);
}
public void processChunk(String cursor) {
chunkTimer.record(() -> {
// process one chunk
});
}
}
Metrics to monitor:
- interest.chunk.duration: time per chunk (detect slowdowns)
- interest.job.total.rows: total rows processed
- interest.job.skipped: poison records skipped
- interest.job.status: success/failure per run
Visual Overview
@Scheduled trigger
│
▼
┌─────────────────────────────────────────┐
│ Load checkpoint cursor │
└───────────────┬─────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Fetch next chunk (500 rows) │◄──┐
│ WHERE id > cursor ORDER BY id │ │
└───────────────┬─────────────────────────┘ │
│ │
▼ │
┌─────────────────────────────────────────┐ │
│ Process chunk │ │
│ - accrueInterest() per account │ │
│ - skip-and-log poison records │ │
└───────────────┬─────────────────────────┘ │
│ │
▼ │
┌─────────────────────────────────────────┐ │
│ flush() -- push SQL to DB │ │
│ clear() -- detach entities, free memory │ │
│ COMMIT -- per-chunk transaction │ │
└───────────────┬─────────────────────────┘ │
│ │
▼ │
┌─────────────────────────────────────────┐ │
│ Save checkpoint (last cursor) │ │
│ Throttle (sleep 100ms) │───┘
└───────────────┬─────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ All chunks done -- job complete │
└─────────────────────────────────────────┘
Memory profile:
Heap ▲
│ ┌─┐ ┌─┐ ┌─┐ ┌─┐ ┌─┐
│ ─┘ └─┬─┘ └─┬─┘ └─┬─┘ └─┬─┘ └─
│ │ │ │ │
└───────────────────────────────► Time
chunk1 chunk2 chunk3 chunk4
(flush+clear keeps memory flat)
Key Takeaway
Processing a very large dataset in a scheduled job has four risks, and each requires a deliberate fix. The first is memory: loading everything with findAll() materializes every row and entity snapshot in the heap, so you process in bounded chunks using keyset pagination and flush+clear the persistence context after each chunk so entities detach and memory stays flat. The second is long execution and overlap: you bound the work, commit per chunk so progress is durable, and guard against overlapping runs with ShedLock or a single-threaded scheduler. The third is database load: you use bounded chunk reads, batched writes, a throttle pause between chunks, and schedule off-peak.
The fourth, and most important for correctness, is error handling. If the job runs as one giant transaction and fails near the end, everything rolls back and the night is wasted. Per-chunk transactions mean completed chunks stay committed. A checkpoint cursor lets a crash resume from the last committed chunk. Idempotent updates (set rather than add) make re-running a chunk safe. And skip-and-log on individual poison records keeps the job running instead of aborting.
At National Bank of India the nightly interest accrual job originally called findAll() and crashed with an OutOfMemoryError once we grew past a few hundred thousand accounts. Rajesh rebuilt it to process five hundred accounts per keyset-paged chunk, each chunk in its own transaction with flush+clear, advancing a checkpoint cursor, throttling briefly between chunks, and skipping corrupt accounts. For genuinely large, restartable, auditable batch work the production-grade answer is Spring Batch, which gives chunk-oriented processing, a job repository, restartability, and skip/retry policies out of the box.
Remember one sentence: process large scheduled jobs in bounded chunks with per-chunk transactions, flush+clear for memory, a checkpoint for resumability, and skip-and-log for resilience.
---