Definition
Scanner– convenient but slow for big files due to regex delimiter parsing and internal synchronization.BufferedReader.readLine()– the memory-friendly baseline; reads one line at a time through a reusable internal buffer.Files.lines(path)– returns a lazyStreamthat reads on demand; must be closed because it holds an open file handle.FileChannel+ByteBuffer– reads raw bytes into a reusable buffer for maximum throughput; requires manual charset decoding.MappedByteBuffer– memory-maps file pages for zero-copy access; fastest but lowest-level.- The one rule – process incrementally and never accumulate the whole file into a
ListorString.
Suppose National Bank of India must process a transaction file with a hundred million lines. Loading the whole file is like trying to carry every ledger book at once – you collapse with an OutOfMemoryError. BufferedReader is a steady clerk handing you one line at a time. Files.lines is a conveyor belt delivering lines on demand, but you must switch it off when done. FileChannel is a forklift moving raw pallets – fastest for huge loads, but you must unpack them yourself.
Understanding the Concepts
What is the danger of reading large files the wrong way?
If you call Files.readAllLines(path) or Files.readString(path), the entire file is loaded into memory. For a multi-gigabyte transaction log, this guarantees an OutOfMemoryError. Every good approach keeps only a bounded amount of data in memory at any instant.
package com.bank.io;
import java.nio.file.Files;
import java.nio.file.Path;
public class WrongWay {
public static void main(String[] args) throws Exception {
// WRONG: loads entire file into memory
// For a 10 GB file, this throws OutOfMemoryError
var lines = Files.readAllLines(Path.of("transactions.csv"));
System.out.println("Lines: " + lines.size());
}
}
How does BufferedReader.readLine() work?
BufferedReader wraps a Reader with an internal character buffer, typically 8192 characters. It reads a block from the underlying stream into the buffer and then hands you one line at a time via readLine(). Only the current line and the internal buffer are in memory, so the footprint is constant regardless of file size.
package com.bank.io;
import java.io.BufferedReader;
import java.io.FileReader;
import java.math.BigDecimal;
import java.nio.charset.StandardCharsets;
public class BufferedReaderExample {
public static void main(String[] args) throws Exception {
BigDecimal total = BigDecimal.ZERO;
long count = 0;
try (BufferedReader br = new BufferedReader(
new FileReader("transactions.csv", StandardCharsets.UTF_8))) {
String line;
while ((line = br.readLine()) != null) {
// Each line: accountId,amount
String[] parts = line.split(",");
BigDecimal amount = new BigDecimal(parts[1]);
total = total.add(amount);
count++;
}
}
System.out.println("Processed " + count + " transactions.");
System.out.println("Total: " + total);
}
}
Processed 50000000 transactions.
Total: 1250000000000.00
How does Files.lines() work and why must it be closed?
Files.lines(path) returns a lazy Stream that reads lines on demand as the stream is consumed. It combines the bounded memory of readLine() with the expressiveness of the Streams API.
The critical caveat is that this stream holds an open file handle. It must be closed, which means using it inside a try-with-resources block. Forgetting to close it leaks handles, and under load you exhaust the OS handle limit.
package com.bank.io;
import java.math.BigDecimal;
import java.nio.file.Files;
import java.nio.file.Path;
public class FilesLinesExample {
public static void main(String[] args) throws Exception {
// CORRECT: try-with-resources closes the stream
try (var lines = Files.lines(Path.of("transactions.csv"))) {
BigDecimal total = lines
.map(line -> line.split(",")[1])
.map(BigDecimal::new)
.reduce(BigDecimal.ZERO, BigDecimal::add);
System.out.println("Total: " + total);
}
}
}
// WRONG: stream never closed, leaks file handle
var total = Files.lines(Path.of("transactions.csv"))
.map(line -> line.split(",")[1])
.map(BigDecimal::new)
.reduce(BigDecimal.ZERO, BigDecimal::add);
Why is Scanner slow for large files?
Scanner synchronizes internally, compiles and applies regular expressions for delimiters, and does extra parsing per token. For small inputs this overhead is negligible, but for millions of lines you feel it sharply.
package com.bank.io;
import java.io.File;
import java.math.BigDecimal;
import java.util.Scanner;
public class ScannerExample {
public static void main(String[] args) throws Exception {
BigDecimal total = BigDecimal.ZERO;
long count = 0;
// Scanner works but is measurably slower than BufferedReader
try (Scanner sc = new Scanner(new File("transactions.csv"))) {
while (sc.hasNextLine()) {
String line = sc.nextLine();
String[] parts = line.split(",");
total = total.add(new BigDecimal(parts[1]));
count++;
}
}
System.out.println("Scanner processed: " + count);
}
}
How does FileChannel with ByteBuffer give maximum throughput?
FileChannel.read(ByteBuffer) reads raw bytes into a reusable buffer. You decode those bytes with a CharsetDecoder and split on line boundaries yourself. This avoids the overhead of line-by-line string creation when you only need to extract numeric fields.
package com.bank.io;
import java.io.RandomAccessFile;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.charset.StandardCharsets;
public class ChannelReader {
public static void main(String[] args) throws Exception {
try (RandomAccessFile raf = new RandomAccessFile("transactions.csv", "r");
FileChannel channel = raf.getChannel()) {
ByteBuffer buffer = ByteBuffer.allocateDirect(64 * 1024);
long byteCount = 0;
while (channel.read(buffer) > 0) {
buffer.flip();
byteCount += buffer.remaining();
// Process raw bytes: decode, find newlines, extract fields
buffer.clear();
}
System.out.println("Read " + byteCount + " bytes via channel.");
}
}
}
Read 2147483648 bytes via channel.
How do the approaches compare?
| Approach | Memory | Speed | Ease of use | Streaming? |
|----------|--------|-------|-------------|------------|
| Files.readAllLines() | Entire file in heap | Fast for small files | Very easy | No |
| Scanner | Bounded | Slow (regex overhead) | Easy | Yes |
| BufferedReader.readLine() | Bounded (buffer + line) | Fast | Easy | Yes |
| Files.lines() | Bounded (lazy stream) | Fast | Moderate (must close) | Yes |
| FileChannel + ByteBuffer | Bounded (buffer) | Fastest | Hard (manual decoding) | Yes |
| MappedByteBuffer | OS-managed pages | Fastest (zero-copy) | Hard (manual decoding) | Yes |
What are common mistakes?
The most common mistake is collecting the entire stream into a list, which defeats the purpose of streaming.
// WRONG: collect(toList()) loads everything into memory
List<String> all = Files.lines(path).collect(Collectors.toList());
// CORRECT: reduce to a bounded result
BigDecimal total = Files.lines(path)
.map(line -> new BigDecimal(line.split(",")[1]))
.reduce(BigDecimal.ZERO, BigDecimal::add);
Another mistake is not specifying the charset, which causes data corruption across platforms.
// WRONG: uses platform default charset
new BufferedReader(new FileReader("transactions.csv"));
// CORRECT: explicit charset
new BufferedReader(new FileReader("transactions.csv", StandardCharsets.UTF_8));
What about parallel processing of large files?
For CPU-heavy per-line work, you can parallelize, but Files.lines().parallel() splits poorly on unindexed text. A better pattern is to read sequentially and dispatch line batches to a thread pool.
package com.bank.io;
import java.io.BufferedReader;
import java.io.FileReader;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class BatchParallel {
private static final int BATCH_SIZE = 10_000;
public static void main(String[] args) throws Exception {
ExecutorService pool = Executors.newFixedThreadPool(4);
List<Future<?>> futures = new ArrayList<>();
try (BufferedReader br = new BufferedReader(
new FileReader("transactions.csv", StandardCharsets.UTF_8))) {
List<String> batch = new ArrayList<>(BATCH_SIZE);
String line;
while ((line = br.readLine()) != null) {
batch.add(line);
if (batch.size() == BATCH_SIZE) {
List<String> toProcess = new ArrayList<>(batch);
futures.add(pool.submit(() -> processBatch(toProcess)));
batch.clear();
}
}
if (!batch.isEmpty()) {
futures.add(pool.submit(() -> processBatch(batch)));
}
}
for (Future<?> f : futures) f.get();
pool.shutdown();
}
static void processBatch(List<String> lines) {
// Heavy per-line processing here
System.out.println("Processed batch of " + lines.size());
}
}
What should a senior engineer know?
Always specify the charset explicitly as StandardCharsets.UTF_8. For fixed-width or binary records, channels and mapping shine because you can address records by offset. For append-and-tail scenarios like watching a growing log, use a FileChannel with positional reads. And remember that the API choice is a performance decision, but streaming line-by-line is a correctness requirement.
Visual Overview
Large File (10 GB)
│
├──▶ Files.readAllLines() ──▶ List<String> in heap ──▶ OOM!
│
├──▶ Scanner ──▶ line-by-line (slow, regex)
│
├──▶ BufferedReader ──▶ line-by-line (fast, simple)
│ │
│ └── buffer (8 KB) + current line in memory
│
├──▶ Files.lines() ──▶ lazy Stream<String>
│ │
│ └── MUST close (holds file handle)
│
└──▶ FileChannel/Mapped ──▶ raw bytes (fastest)
│
└── manual decode + line split
The One Rule: ┌─────────────────────────────────────────┐ │ Process line-by-line or in chunks. │ │ Reduce to a bounded result as you go. │ │ NEVER load the whole file into memory. │ └─────────────────────────────────────────┘
Key Takeaway
There are several ways to read a very large text file in Java, and the choice is about speed versus simplicity. Scanner is convenient but slow for big files because of regex parsing. BufferedReader.readLine() is the simple, memory-friendly baseline that keeps only the current line and a small buffer in memory. Files.lines() gives a lazy stream that reads on demand and lets you filter and reduce expressively, but it holds an open file handle so you must always close it with try-with-resources. FileChannel and MappedByteBuffer are fastest because the OS pages data with no system call overhead, but they force you to decode bytes and split lines manually.
The one rule that dominates all API choices is to process incrementally. Stream line-by-line or in bounded chunks and reduce to a bounded result as you go. Never load the whole file into a List or a String, because that makes memory grow with file size and guarantees an OutOfMemoryError on a large enough input.
In banking at National Bank of India, Rajesh's first version of a daily reconciliation script used Files.readAllLines() and crashed in production with an OutOfMemoryError on a fifty-million-line file. Pankaj rewrote it to stream with BufferedReader, keeping only a running BigDecimal total per account, and it ran comfortably in a tiny heap. For the reporting dashboard, Files.lines() inside try-with-resources reduced to a single total rather than collecting.
Remember one sentence: the API is a performance choice, but streaming line-by-line is a correctness requirement that prevents OutOfMemoryError on any file size.
---