Definition
flatMapwithsentence.split("\\s+")turns each sentence into a stream of words and merges them.distinct()removes duplicates so each unique word appears once.- Normalisation:
map(String::toLowerCase)before distinct ensures case-insensitive uniqueness. - Regex
\\s+matches one or more whitespace characters, preventing empty tokens from double spaces. - Pattern: flatMap(split) → map(lower) → distinct → collect.
The documentation officer at the National Bank of India has several policy sentences and wants the vocabulary list: every distinct word used, each listed once. She breaks every sentence into words, throws them all into one pile, then removes repeats.
Understanding the Concepts
How do you get unique words from sentences?
Splitting a sentence gives an array of words. Each array becomes a stream with Arrays.stream(...), and flatMap concatenates all those per-sentence streams into one flat stream of words.
package com.bank.docs;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
public class UniqueWords {
public static void main(String[] args) {
List<String> sentences = List.of(
"The bank processes loan applications daily",
"The loan department reviews the applications",
"Daily reports are sent to the bank");
List<String> uniqueWords = sentences.stream()
.flatMap(s -> Arrays.stream(s.split("\\s+")))
.map(String::toLowerCase)
.distinct()
.sorted()
.collect(Collectors.toList());
System.out.println(uniqueWords);
}
}
[applications, are, bank, daily, department, loan, processes, reports, reviews, sent, the, to]
Why lower-case before distinct?
Without normalisation, distinct() treats differently-cased forms as separate words.
| Without toLowerCase | With toLowerCase |
|---|---|
| "The" and "the" are two words | "the" appears once |
| "Bank" and "bank" are two words | "bank" appears once |
// WRONG: "The" and "the" survive as separate
sentences.stream()
.flatMap(s -> Arrays.stream(s.split("\\s+")))
.distinct() // "The" != "the"
.collect(Collectors.toList());
// CORRECT: normalise first
sentences.stream()
.flatMap(s -> Arrays.stream(s.split("\\s+")))
.map(String::toLowerCase) // "The" → "the"
.distinct()
.collect(Collectors.toList());
What about punctuation?
Whitespace splitting does not strip punctuation, so "applications" and "applications," (with a comma) would be treated as different words.
// Strip non-letter characters after splitting
List<String> clean = sentences.stream()
.flatMap(s -> Arrays.stream(s.split("\\s+")))
.map(w -> w.replaceAll("[^a-zA-Z]", ""))
.map(String::toLowerCase)
.filter(w -> !w.isEmpty())
.distinct()
.collect(Collectors.toList());
What are common mistakes?
Mistake: using split(" ") instead of split("\\s+"), producing empty tokens from double spaces.
// WRONG: split on single space, double spaces produce empty strings
"loan application".split(" "); // ["", "loan", "", "application"]
// CORRECT: split on one or more whitespace
"loan application".split("\\s+"); // ["loan", "application"]
What about collecting into a Set instead of using distinct?
import java.util.Set;
Set<String> wordSet = sentences.stream()
.flatMap(s -> Arrays.stream(s.split("\\s+")))
.map(String::toLowerCase)
.collect(Collectors.toSet());
Using toSet() gives uniqueness for free, but does not preserve the encounter order. Use distinct() + toList() when order matters.
What about counting word frequency instead of just unique words?
import java.util.Map;
import java.util.function.Function;
Map<String, Long> frequency = sentences.stream()
.flatMap(s -> Arrays.stream(s.split("\\s+")))
.map(String::toLowerCase)
.collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));
frequency.forEach((word, count) ->
System.out.println(word + ": " + count));
What should a senior engineer know?
- Splitting on whitespace is language-naive. For real text processing use a proper tokenizer.
- For very large texts, consider reading line by line with
Files.lines()to avoid loading everything into memory. distinct()is stateful and holds a Set internally, so memory usage grows with the number of unique words.- The flatMap + distinct pattern is the standard interview answer for "unique words from sentences".
What are common interview follow-up questions?
- "How do you count word frequency?" Use
groupingBy(identity(), counting()). - "How do you handle punctuation?" Strip non-letter characters after splitting.
- "What if sentences come from a file?" Use
Files.lines(path)as the stream source.
Visual Overview
List<String> sentences
["The bank processes loan", "The loan department"]
│
│ flatMap(s -> split("\\s+"))
▼
Stream<String>
["The", "bank", "processes", "loan", "The", "loan", "department"]
│
│ map(String::toLowerCase)
▼
["the", "bank", "processes", "loan", "the", "loan", "department"]
│
│ distinct()
▼
["the", "bank", "processes", "loan", "department"]
│
│ collect(toList())
▼
List<String> [the, bank, processes, loan, department]
Key Takeaway
To get the unique words across many sentences I combine flatMap with distinct. I split each sentence into an array of words, turn that array into a stream, and flatMap concatenates all those per-sentence word streams into one flat stream. Then distinct removes the repeats so each word survives exactly once.
Before distinct I lower-case every word, otherwise a capitalised "The" at the start of a sentence and a lowercase "the" in the middle would be treated as two different words. I split on the regex \\s+ to handle multiple spaces without producing empty tokens.
In my banking project at the National Bank of India this built a vocabulary index across policy documents for terminology consistency. The same pattern extends to word frequency counting with groupingBy(identity(), counting()). For real natural-language processing I would use a proper tokenizer, but for a reporting vocabulary index, split plus lower-case plus distinct is the pragmatic answer.
Remember one sentence: flatMap(split) → toLowerCase → distinct extracts unique words from multiple sentences into one flat, deduplicated list.
---