Definition
Two different jobs look the same but need different tools.
- **`distinct()`** – an intermediate, stateful operation that keeps a set of elements it has already emitted and drops any element that `equals` one already seen. - **`equals`/`hashCode` dependency** – `distinct()` has no idea what a duplicate means; it delegates that entirely to the element's own `equals` and `hashCode`. - **Key-based dedup with a stateful filter** – `Set<Long> seen = new HashSet<>(); stream.filter(e -> seen.add(e.getId()))` keeps only the first element per key because `Set.add` returns `true` once per key. - **Key-based dedup with `toMap`** – `collect(Collectors.toMap(Employee::getId, e -> e, (a, b) -> a))` keeps one value per key and is safe in parallel.
Priya has a pile of account-opening forms and some are exact photocopies of the same form. distinct() throws away the exact photocopies. But sometimes two forms carry the same account number while the name or address differs slightly. To keep just one form per account number she keeps a checklist of numbers she has already filed and skips any form whose number is already ticked.
Understanding the Concepts
What does distinct() do?
distinct() is an intermediate, stateful operation on a stream. It returns a new stream with duplicate elements removed. Internally it maintains a set of elements it has already emitted, and for each incoming element it checks whether an equal element is already in that set. If yes, the element is dropped; if no, the element passes downstream and is recorded.
The key fact for an interview is that distinct() has no built-in notion of what makes two objects duplicates. It asks the element itself through equals and hashCode.
package com.bank.dedup;
import java.math.BigDecimal;
import java.util.List;
import java.util.stream.Collectors;
public class DistinctDemo {
public static void main(String[] args) {
List<String> accountTypes = List.of("SAVINGS", "CURRENT", "SAVINGS", "LOAN", "CURRENT");
List<String> unique = accountTypes.stream()
.distinct()
.collect(Collectors.toList());
System.out.println(unique);
}
}
[SAVINGS, CURRENT, LOAN]
How does distinct() rely on equals and hashCode?
distinct() decides equality using the same contract a HashSet uses: it computes hashCode() to find the bucket and calls equals() to confirm. If your class does not override these methods, Java uses identity equality from Object, meaning two different instances are never considered equal even if every field matches.
So for a plain class without equals/hashCode, distinct() removes nothing unless you pass the very same reference twice.
package com.bank.model;
import java.math.BigDecimal;
// WRONG for distinct-by-value: no equals/hashCode override
public class AccountNoEquals {
private final long id;
private final BigDecimal balance;
public AccountNoEquals(long id, BigDecimal balance) {
this.id = id;
this.balance = balance;
}
// distinct() will keep BOTH of two accounts with id 101
// because the default equals compares object identity.
}
The correct version overrides both methods so value equality works:
package com.bank.model;
import java.math.BigDecimal;
import java.util.Objects;
public class Account {
private final long id;
private final BigDecimal balance;
public Account(long id, BigDecimal balance) {
this.id = id;
this.balance = balance;
}
public long getId() { return id; }
public BigDecimal getBalance() { return balance; }
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof Account)) return false;
Account other = (Account) o;
return id == other.id && balance.compareTo(other.balance) == 0;
}
@Override
public int hashCode() {
return Objects.hash(id, balance);
}
}
How do records make distinct() work automatically?
A Java record (Java 16+) generates equals and hashCode from every component. That means distinct() treats two records as duplicates only when they are identical in every field. This is the cleanest way to get value-based distinct().
package com.bank.model;
import java.math.BigDecimal;
public record Employee(long id, String name, String department, BigDecimal salary) { }
package com.bank.dedup;
import com.bank.model.Employee;
import java.math.BigDecimal;
import java.util.List;
import java.util.stream.Collectors;
public class RecordDistinctDemo {
public static void main(String[] args) {
Employee a = new Employee(1, "Rajesh", "Trading", new BigDecimal("900000"));
Employee b = new Employee(1, "Rajesh", "Trading", new BigDecimal("900000"));
Employee c = new Employee(2, "Priya", "Risk", new BigDecimal("850000"));
List<Employee> unique = List.of(a, b, c).stream()
.distinct()
.collect(Collectors.toList());
System.out.println(unique.size());
}
}
2
Because a and b are equal in every component, the record's generated equals reports them identical and distinct() keeps only one.
How do you remove duplicates by a key such as id?
Real data frequently has duplicates that share a business key while other fields differ. A feed may send the same employee id twice with a stale name on one copy. Here distinct() cannot help, because the two objects are not equal. You deduplicate by the key instead.
The stateful-filter technique uses a Set of already-seen keys. Set.add returns true only the first time a key appears, so the filter keeps the first occurrence per key.
package com.bank.dedup;
import com.bank.model.Employee;
import java.math.BigDecimal;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
public class DedupByKeyDemo {
public static void main(String[] args) {
List<Employee> feed = List.of(
new Employee(101, "Rajesh Kumar", "Trading", new BigDecimal("900000")),
new Employee(101, "Rajesh K.", "Trading", new BigDecimal("900000")),
new Employee(102, "Priya Sharma", "Risk", new BigDecimal("850000"))
);
Set<Long> seen = new HashSet<>();
List<Employee> deduped = feed.stream()
.filter(e -> seen.add(e.id()))
.collect(Collectors.toList());
deduped.forEach(e -> System.out.println(e.id() + " -> " + e.name()));
}
}
101 -> Rajesh Kumar
102 -> Priya Sharma
How do you deduplicate by a key using toMap?
The collector approach builds a Map keyed by the business key. A three-argument toMap takes a merge function that decides what to do on a key collision. Keeping the first value gives the same result as the stateful filter, and it is safe in parallel.
package com.bank.dedup;
import com.bank.model.Employee;
import java.math.BigDecimal;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
public class DedupByKeyMapDemo {
public static void main(String[] args) {
List<Employee> feed = List.of(
new Employee(101, "Rajesh Kumar", "Trading", new BigDecimal("900000")),
new Employee(101, "Rajesh K.", "Trading", new BigDecimal("900000")),
new Employee(102, "Priya Sharma", "Risk", new BigDecimal("850000"))
);
Map<Long, Employee> byId = feed.stream()
.collect(Collectors.toMap(Employee::id, e -> e, (first, second) -> first));
Collection<Employee> deduped = byId.values();
System.out.println(deduped.size());
}
}
2
Without the merge function, toMap throws IllegalStateException on a duplicate key, so the merge function is required for deduplication.
How do distinct() and key-based dedup compare?
| Aspect | distinct() | Stateful filter by key | toMap by key |
| --- | --- | --- | --- |
| Removes duplicates by | Full equals/hashCode | A chosen key | A chosen key |
| Keeps which copy | First by encounter order | First by encounter order | Decided by merge function |
| Needs equals/hashCode | Yes | No (only on the key) | No (only on the key) |
| Parallel safe | Yes | No | Yes |
| Result type | Stream of elements | Stream of elements | Map, then values() |
// Side by side: same feed, two strategies
List<Employee> exact = feed.stream().distinct().collect(Collectors.toList());
Set<Long> seen = new HashSet<>();
List<Employee> byKey = feed.stream()
.filter(e -> seen.add(e.id()))
.collect(Collectors.toList());
What are common mistakes?
The first mistake is expecting distinct() to remove key-duplicates. It will not, because the objects are not equal.
// WRONG: two records with the same id but different name are NOT equal,
// so distinct keeps both.
List<Employee> result = feed.stream().distinct().collect(Collectors.toList());
// result still contains both id=101 rows
// CORRECT: dedupe on the id key explicitly.
Set<Long> seen = new HashSet<>();
List<Employee> result = feed.stream()
.filter(e -> seen.add(e.id()))
.collect(Collectors.toList());
The second mistake is reusing a stateful filter in a parallel stream, which produces non-deterministic output because threads race on the shared Set.
// WRONG: shared mutable Set in a parallel stream.
Set<Long> seen = new HashSet<>();
List<Employee> result = feed.parallelStream()
.filter(e -> seen.add(e.id())) // data race, unpredictable result
.collect(Collectors.toList());
// CORRECT: use toMap, which is parallel safe.
List<Employee> result = new ArrayList<>(feed.parallelStream()
.collect(Collectors.toMap(Employee::id, e -> e, (a, b) -> a))
.values());
What are the edge cases and senior nuances?
On an ordered stream, distinct() preserves the encounter order of the first occurrence. The stateful filter also keeps the first occurrence in a sequential stream. The toMap approach loses encounter order because a HashMap has no order; to preserve it, supply LinkedHashMap::new as the map factory.
import java.util.LinkedHashMap;
// Preserve first-seen order while deduping by key
Map<Long, Employee> byId = feed.stream()
.collect(Collectors.toMap(
Employee::id,
e -> e,
(first, second) -> first,
LinkedHashMap::new));
Another edge case is that BigDecimal equality by equals is scale-sensitive: new BigDecimal("1.0").equals(new BigDecimal("1.00")) is false. If money fields drive your equals, use compareTo in the body so distinct() treats equal values as equal regardless of scale.
When should you use each approach?
Use distinct() when a duplicate means identical in every field, which is the common case for records and well-defined value objects. Use the stateful filter in a sequential stream when you want the first row per business key and prefer a one-liner. Use toMap when you run in parallel, when you want control over which duplicate to keep through the merge function, or when you already need the result as a map keyed by id.
// Keep the HIGHER salary on an id collision instead of the first
Map<Long, Employee> byId = feed.stream()
.collect(Collectors.toMap(
Employee::id,
e -> e,
(a, b) -> a.salary().compareTo(b.salary()) >= 0 ? a : b));
Visual Overview
distinct() -- removes EXACT duplicates by equals/hashCode
[A id=1] [B id=1 same fields] [C id=2]
│ │ │
▼ ▼ ▼
┌───────────────────────────────────────┐
│ seen set: {} │
│ A -> not in set -> emit, add A │
│ B -> equals A -> drop │
│ C -> not in set -> emit, add C │
└───────────────────────────────────────┘
│
▼
[A id=1] [C id=2]
filter(e -> seen.add(e.getId())) -- dedup by KEY
[id=101 Rajesh Kumar] [id=101 Rajesh K.] [id=102 Priya]
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────────────────┐
│ seen keys: {} │
│ 101 -> add returns true -> keep │
│ 101 -> add returns false -> drop (stale name copy) │
│ 102 -> add returns true -> keep │
└─────────────────────────────────────────────────────┘
│
▼
[id=101 Rajesh Kumar] [id=102 Priya]
Choose the tool
┌──────────────────────────────┐
│ Duplicate = identical fields? │
└──────────────┬───────────────┘
yes │ no
▼ │ ▼
distinct() │ dedup by key
│ ┌────────────┬──────────────┐
│ │ sequential │ parallel │
│ ▼ ▼
│ stateful toMap(..., merge)
│ filter
Key Takeaway
To remove exact duplicate objects from a stream I use distinct(), which keeps a set of elements it has already emitted and drops any element that equals one already seen. The crucial point is that distinct() relies entirely on the element's equals and hashCode, so for a record, where both methods cover every component automatically, distinct() removes only records that are identical in every field.
In my banking work at the National Bank of India one feed occasionally sent the exact same employee record twice, and a plain distinct() cleaned those up perfectly. But a different feed sent two records that shared an employee id where one name was stale, and distinct() could not help because those two objects are not equal. For that I deduplicated by the business key: I kept a set of seen ids and filtered the stream with seen.add(e.getId()), which returns true only the first time an id appears, so only the first record per id survives.
The senior nuance I always mention is that this stateful filter is not safe in a parallel stream, because the shared set makes the outcome depend on thread timing. When I need parallel-safe deduplication by a key I collect into a map keyed by id with a merge function that keeps the first value, and then take the map's values, which is deterministic regardless of parallelism. That same merge function also lets me decide to keep the higher salary or the latest record instead of the first.
Think of Priya filing account forms: she bins the exact photocopies, and for forms that merely share an account number she keeps a ticked checklist of numbers so each account is filed only once.
Remember one sentence: distinct() dedupes by equals/hashCode, but to dedupe by a key you track seen keys in a Set or collect into a Map and keep one value per key.
---