Remove duplicates from a list: by value and by object field using streams

• distinct() removes duplicates based on the object's equals() and hashCode() methods, using an internal HashSet to track seen elements.
• distinctByKey pattern: a reusable Predicate backed by ConcurrentHashMap.newKeySet() where seen.add(keyExtractor.apply(t)) returns true for new keys and false for duplicates.
• toMap alternative: toMap(keyExtractor, identity, (first, second) -> first).values() collects into a Map (inherently unique keys) with a merge function keeping the first occurrence.
• distinct() alone only works if equals/hashCode are defined on the desired dedup field. For dedup by a specific field without modifying equals, use the distinctByKey pattern.
• Both approaches preserve first-occurrence order in ordered streams.

What is the difference between dedup by value and dedup by field?

Dedup by value means two objects are duplicates if they are equals to each other. Dedup by field means two objects are duplicates if they share the same value for a specific field, even if other fields differ.

How do you remove duplicates by value?

distinct() uses equals() and hashCode() to identify duplicates.

package com.bank.service;

import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Function;
import java.util.function.Predicate;
import java.util.stream.Collectors;

public class DedupDemo {
    public static void main(String[] args) {
        // BY VALUE: distinct() uses equals + hashCode
        List<String> symbols = List.of(
            "RELIANCE", "TCS", "RELIANCE", "HDFC", "TCS");

        List<String> unique = symbols.stream()
            .distinct()
            .toList();

        System.out.println("Unique symbols: " + unique);
    }
}

Unique symbols: [RELIANCE, TCS, HDFC]

How do you remove duplicates by a specific field?

Pattern 1: Use the distinctByKey utility predicate.

// Reusable distinctByKey utility
public static <T> Predicate<T> distinctByKey(
        Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

record Employee(String id, String name, String department) {}

List<Employee> employees = List.of(
    new Employee("E001", "Rajesh",  "TRADING"),
    new Employee("E002", "Priya",   "RISK"),
    new Employee("E001", "Rajesh",  "OPERATIONS"),  // duplicate ID
    new Employee("E003", "Amit",    "TRADING"),
    new Employee("E002", "Priya",   "COMPLIANCE")   // duplicate ID
);

// Deduplicate by employee ID
List<Employee> uniqueById = employees.stream()
    .filter(distinctByKey(Employee::id))
    .toList();

uniqueById.forEach(e ->
    System.out.println(e.id() + ": " + e.name() + " " + e.department()));

E001: Rajesh TRADING
E002: Priya RISK
E003: Amit TRADING

The first occurrence of each ID is kept. The duplicates with different departments are discarded.

How does the toMap alternative work?

// Pattern 2: toMap with merge function
List<Employee> uniqueViaMap = employees.stream()
    .collect(Collectors.toMap(
        Employee::id,
        e -> e,
        (first, second) -> first))  // keep first occurrence
    .values().stream().toList();

The Map's unique-key property naturally deduplicates. The merge function (first, second) -> first determines which duplicate to keep.

How do you deduplicate by multiple fields?

Concatenate the fields into a composite key.

// Deduplicate by name + department combination
List<Employee> uniqueByComposite = employees.stream()
    .filter(distinctByKey(e -> e.name() + "|" + e.department()))
    .toList();

How is distinct() different from distinctByKey?

| Feature | distinct() | distinctByKey(keyExtractor) |
|---------|-------------|------------------------------|
| Dedup criterion | equals() + hashCode() | Extracted key field |
| Requires | Correct equals/hashCode on the object | Only equals/hashCode on the KEY type |
| Modifies class? | Depends on existing equals | No modification needed |
| Thread safety | Internal HashSet | ConcurrentHashMap (parallel-safe) |
| Reusable? | Built-in | Utility method you write once |

What are common mistakes?

WRONG: Assuming distinct() deduplicates by a specific field without proper equals/hashCode.

// WRONG: distinct() uses object identity unless equals/hashCode are overridden
// Two Employee objects with the same ID but different departments are NOT equal
// unless equals is based on ID
List<Employee> wrong = employees.stream()
    .distinct()  // does nothing if equals compares all fields
    .toList();

CORRECT: Use distinctByKey for field-based dedup.

// CORRECT: explicit field-based dedup
List<Employee> correct = employees.stream()
    .filter(distinctByKey(Employee::id))
    .toList();

What about parallel stream safety?

The distinctByKey utility uses ConcurrentHashMap.newKeySet(), which is thread-safe. Multiple threads can call seen.add() concurrently without data races. This makes it safe for parallel streams.

// Safe for parallel streams
List<Employee> parallelDedup = employees.parallelStream()
    .filter(distinctByKey(Employee::id))
    .toList();

What should a senior engineer know?

• distinct() uses an internal LinkedHashSet in the stream implementation, giving O(n) memory usage.
• The distinctByKey pattern is stateful: it captures a mutable Set in the predicate closure. This is acceptable because ConcurrentHashMap is thread-safe, but it means the predicate cannot be reused across multiple stream pipelines.
• For very large datasets, consider whether dedup should happen at the database level (SQL DISTINCT or GROUP BY) rather than in memory.
• The toMap approach does not preserve the original order unless you use LinkedHashMap::new as the fourth argument.