Definition
- LinkedHashSet(list): iterates the list, adds each element; duplicates are rejected, first occurrence is kept, insertion order is preserved
- new ArrayList<>(linkedHashSet): converts back to a List if needed
- Stream:
list.stream().distinct().toList()also preserves encounter order (for ordered streams like ArrayList) - HashSet(list): removes duplicates but loses order
- TreeSet(list): removes duplicates and sorts (not insertion order)
You have a stack of transaction receipts where some transactions appear multiple times. Pour them through a LinkedHashSet funnel: duplicates fall out, but the remaining receipts come through in the exact order you fed them in. The first copy of each transaction is kept, and all later copies are discarded.
Understanding the Concepts
How do you remove duplicates while keeping insertion order?
The idiomatic one-liner is:
List<String> deduped = new ArrayList<>(new LinkedHashSet<>(original));
This works because LinkedHashSet rejects duplicates (keeping the first occurrence) and maintains insertion order.
package com.bank.service;
import java.util.*;
import java.util.stream.Collectors;
public class DeduplicationDemo {
public static void main(String[] args) {
List<String> transactionIds = List.of(
"TXN-001", "TXN-002", "TXN-003", "TXN-001", "TXN-004", "TXN-002", "TXN-005"
);
System.out.println("Original: " + transactionIds);
System.out.println("Size: " + transactionIds.size());
// Method 1: LinkedHashSet one-liner
List<String> deduped1 = new ArrayList<>(new LinkedHashSet<>(transactionIds));
System.out.println("\nLinkedHashSet: " + deduped1);
// Method 2: Stream distinct()
List<String> deduped2 = transactionIds.stream().distinct().toList();
System.out.println("Stream: " + deduped2);
// Method 3: HashSet – loses order!
List<String> unordered = new ArrayList<>(new HashSet<>(transactionIds));
System.out.println("HashSet: " + unordered);
// Method 4: TreeSet – sorts instead of preserving order
List<String> sorted = new ArrayList<>(new TreeSet<>(transactionIds));
System.out.println("TreeSet: " + sorted);
}
}
Original: [TXN-001, TXN-002, TXN-003, TXN-001, TXN-004, TXN-002, TXN-005]
Size: 7
LinkedHashSet: [TXN-001, TXN-002, TXN-003, TXN-004, TXN-005]
Stream: [TXN-001, TXN-002, TXN-003, TXN-004, TXN-005]
HashSet: [TXN-003, TXN-004, TXN-005, TXN-001, TXN-002]
TreeSet: [TXN-001, TXN-002, TXN-003, TXN-004, TXN-005]
Which method should you use?
| Method | Preserves order | Time | Space | Returns |
|---|---|---|---|---|
| new LinkedHashSet<>(list) | Yes (insertion) | O(n) | O(n) | Set |
| list.stream().distinct().toList() | Yes (encounter) | O(n) | O(n) | List |
| new HashSet<>(list) | No | O(n) | O(n) | Set |
| new TreeSet<>(list) | Sorted | O(n log n) | O(n) | SortedSet |
For most cases, the LinkedHashSet one-liner or stream().distinct() are the best choices.
How do you deduplicate by a specific field?
When deduplicating complex objects, you may want to keep the first occurrence based on a specific field rather than the entire object.
package com.bank.service;
import java.math.BigDecimal;
import java.util.*;
import java.util.stream.Collectors;
public class FieldDeduplicationDemo {
record Transaction(String id, String account, BigDecimal amount) {}
public static void main(String[] args) {
List<Transaction> transactions = List.of(
new Transaction("TXN-001", "SAV2001", new BigDecimal("50000")),
new Transaction("TXN-002", "SAV2001", new BigDecimal("25000")), // same account
new Transaction("TXN-003", "CUR3001", new BigDecimal("75000")),
new Transaction("TXN-004", "SAV2001", new BigDecimal("10000")) // same account
);
// Keep first transaction per account
LinkedHashMap<String, Transaction> byAccount = new LinkedHashMap<>();
for (Transaction txn : transactions) {
byAccount.putIfAbsent(txn.account(), txn);
}
List<Transaction> uniqueByAccount = new ArrayList<>(byAccount.values());
System.out.println("Unique by account:");
uniqueByAccount.forEach(t ->
System.out.println(" " + t.id() + " " + t.account() + " " + t.amount()));
// Stream equivalent using Collectors.toMap with merge function
Map<String, Transaction> streamResult = transactions.stream()
.collect(Collectors.toMap(
Transaction::account,
t -> t,
(first, second) -> first, // keep first occurrence
LinkedHashMap::new // preserve order
));
}
}
Unique by account:
TXN-001 SAV2001 50000
TXN-003 CUR3001 75000
What are common mistakes?
Mistake 1: Using HashSet when order matters
// WRONG: loses order
List<String> deduped = new ArrayList<>(new HashSet<>(list));
// Output order is unpredictable!
// CORRECT: preserves order
List<String> deduped = new ArrayList<>(new LinkedHashSet<>(list));
Mistake 2: Forgetting that distinct() on parallel streams may not preserve order
// distinct() on ordered sequential stream: preserves encounter order
list.stream().distinct().toList(); // order preserved
// distinct() on parallel stream: may not preserve order
list.parallelStream().distinct().toList(); // order NOT guaranteed
// Use forEachOrdered or .sequential() if order matters
What should a senior engineer know?
- distinct() uses a LinkedHashSet internally. The stream distinct() operation maintains a LinkedHashSet of seen elements under the hood for ordered sequential streams.
- Large-scale dedup. For very large datasets that do not fit in memory, consider external sorting + sequential dedup, or Bloom filters for approximate dedup.
- Collectors.toCollection(LinkedHashSet::new). In streams, you can collect directly to a LinkedHashSet: stream.collect(Collectors.toCollection(LinkedHashSet::new)).
What are common interview follow-up questions?
- "How do you remove duplicates from a List?" – new ArrayList<>(new LinkedHashSet<>(list)).
- "Does stream().distinct() preserve order?" – Yes, for ordered sequential streams.
- "How do you deduplicate by a field?" – Use LinkedHashMap with putIfAbsent or Collectors.toMap.
Visual Overview
Deduplication flow:
Input: [TXN-001, TXN-002, TXN-003, TXN-001, TXN-004, TXN-002]
↓
LinkedHashSet:
add TXN-001 → [TXN-001] (new)
add TXN-002 → [TXN-001, TXN-002] (new)
add TXN-003 → [TXN-001, TXN-002, TXN-003] (new)
add TXN-001 → rejected (duplicate!) (skip)
add TXN-004 → [TXN-001, TXN-002, TXN-003, TXN-004] (new)
add TXN-002 → rejected (duplicate!) (skip)
↓
Output: [TXN-001, TXN-002, TXN-003, TXN-004]
(insertion order preserved, duplicates removed)
Key Takeaway
Removing duplicates while preserving insertion order is a common task, and the idiomatic solution is new ArrayList<>(new LinkedHashSet<>(list)). LinkedHashSet keeps the first occurrence of each element and rejects subsequent duplicates, maintaining the original insertion order throughout.
In the National Bank of India's transaction processing pipeline, Rajesh receives transaction feeds from multiple sources where the same transaction ID may appear more than once. He deduplicates using new LinkedHashSet<>(feedTransactions) to ensure each transaction is processed exactly once, in the order it first arrived. For field-based dedup (unique per account), he uses a LinkedHashMap with putIfAbsent.
The stream equivalent list.stream().distinct().toList() achieves the same result for ordered sequential streams, but be aware that parallel streams may not preserve encounter order.
Remember one sentence: LinkedHashSet is the one-liner for dedup with order preservation – first occurrence wins, duplicates are silently dropped.
---