Back to CI/CD Pipeline
Q20Question 20 of 56

What Elasticsearch is, and how it searches internally: inverted index, analyzers, sharding, near-real-time

Key Concepts
  • Elasticsearch is a distributed, document-oriented search and analytics engine built on Apache Lucene.
  • The inverted index maps each term to the list of documents containing it, which is what makes full-text search fast.
  • Analyzers transform text into normalized terms during indexing and querying so searches match regardless of case or word form.
  • Sharding splits an index across nodes for horizontal scaling; replica shards provide fault tolerance and read throughput.
  • Near-real-time means a newly indexed document becomes searchable after a refresh interval, typically one second.
  • Elasticsearch is optimized for read-heavy search and analytics, not for strong transactional consistency.

Definition

Elasticsearch flips the problem around: instead of scanning documents for a word, it keeps a word-to-documents map, splits it across machines, and refreshes it about once a second.
  • Elasticsearch – a distributed, document-oriented search and analytics engine built on Apache Lucene, storing data as JSON documents in indices.
  • Inverted index – the core data structure that maps each unique term to the sorted list of documents containing it, making full-text search fast.
  • Analyzer – a pipeline of character filters, tokenizer, and token filters that transforms text into normalized terms during both indexing and querying.
  • Sharding – splitting an index into pieces (primary shards) distributed across nodes, with replica shards for fault tolerance and read scaling.
  • Near-real-time – a newly indexed document becomes searchable after a refresh interval (default one second), not instantly.

Imagine the back of a thick banking regulations manual at National Bank of India. Instead of reading every page to find the word "overdraft," you flip to the index at the back, which lists "overdraft" and the exact pages it appears on. That back-of-book index is an inverted index. The analyzer is the librarian who lowercases everything and ignores commas before building the index. Sharding is splitting the manual into ten volumes kept in ten rooms so ten clerks can search at once. Near-real-time is the rule that a newly added page shows up in the index a second after filing.

Understanding the Concepts

What is Elasticsearch?

Elasticsearch stores data as JSON documents grouped into an index, which is roughly analogous to a table in a relational database but optimized for search. Under the hood every index is powered by Apache Lucene; Elasticsearch adds the distributed layer, the REST API, and cluster management.

BASH
# Index a transaction document
curl -X POST "http://localhost:9200/transactions/_doc/1" \
  -H "Content-Type: application/json" \
  -d '{
    "accountId": "ACC001",
    "type": "TRANSFER",
    "amount": 50000.00,
    "description": "International wire transfer to HDFC savings",
    "timestamp": "2025-01-15T10:23:45Z",
    "status": "COMPLETED"
  }'
TEXT
{"_index":"transactions","_id":"1","result":"created"}

How does the inverted index work?

A forward index maps each document to the words it contains. An inverted index does the reverse: it maps each unique term to the sorted list of documents that contain it, called a postings list. Finding all documents containing "overdraft" is a single lookup, not a scan.

DIAGRAM
Forward index (slow for search):
  Doc1 → [international, wire, transfer, hdfc, savings]
  Doc2 → [overdraft, protection, hdfc, current]
  Doc3 → [wire, transfer, icici, savings]

Inverted index (fast for search):
  hdfc         → [Doc1, Doc2]
  international → [Doc1]
  overdraft    → [Doc2]
  savings      → [Doc1, Doc3]
  transfer     → [Doc1, Doc3]
  wire         → [Doc1, Doc3]
BASH
# Search for "wire transfer" -- Elasticsearch looks up both terms
curl -X GET "http://localhost:9200/transactions/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "description": "wire transfer"
      }
    }
  }'

How do analyzers work?

Analyzers decide what the terms in the inverted index actually are. When text is indexed, it passes through three stages: character filters, a tokenizer, and token filters.

TEXT
Input:  "International Wire Transfer to HDFC Savings!"

Step 1 - Character filter:  (strip punctuation)
         "International Wire Transfer to HDFC Savings"

Step 2 - Tokenizer:  (split on whitespace)
         ["International", "Wire", "Transfer", "to", "HDFC", "Savings"]

Step 3 - Token filters:  (lowercase, remove stop words)
         ["international", "wire", "transfer", "hdfc", "savings"]
BASH
# Test how the standard analyzer processes text
curl -X POST "http://localhost:9200/_analyze" \
  -H "Content-Type: application/json" \
  -d '{
    "analyzer": "standard",
    "text": "International Wire Transfer to HDFC Savings!"
  }'
TEXT
{"tokens":[{"token":"international"},{"token":"wire"},{"token":"transfer"},{"token":"to"},{"token":"hdfc"},{"token":"savings"}]}

How does sharding enable horizontal scaling?

When you create an index you choose a number of primary shards; each shard is a complete, independent Lucene index holding a subset of documents. A search is scattered to every shard in parallel, each returns its best matches, and the coordinating node merges them.

BASH
# Create an index with 3 primary shards and 1 replica each
curl -X PUT "http://localhost:9200/transactions" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  }'
TEXT
Shard distribution across 3 nodes:
  Node 1: [Primary-0] [Replica-2]
  Node 2: [Primary-1] [Replica-0]
  Node 3: [Primary-2] [Replica-1]

  If Node 2 fails:
    Replica-1 on Node 3 is promoted to Primary-1
    Cluster stays available

When you index a document it is written to an in-memory buffer and a durable transaction log (translog) for crash safety, but it is not yet searchable. A refresh operation, by default every second, writes buffered documents into a new searchable Lucene segment.

BASH
# Force an immediate refresh (normally not needed)
curl -X POST "http://localhost:9200/transactions/_refresh"

# Change the refresh interval for a high-throughput index
curl -X PUT "http://localhost:9200/transactions/_settings" \
  -H "Content-Type: application/json" \
  -d '{"index":{"refresh_interval":"5s"}}'

How is Elasticsearch different from a relational database?

| Aspect | Elasticsearch | Relational DB (PostgreSQL) |
|---|---|---|
| Data model | JSON documents | Rows in tables |
| Search | Full-text via inverted index | LIKE or full-text extensions |
| Transactions | No multi-document ACID | Full ACID |
| Consistency | Near-real-time | Immediate |
| Scaling | Horizontal (sharding) | Vertical (primarily) |
| Best for | Search, analytics, logs | System of record, money |

JAVA
package com.bank.service;

import org.springframework.data.elasticsearch.core.ElasticsearchOperations;
import org.springframework.data.elasticsearch.core.SearchHits;
import org.springframework.data.elasticsearch.core.query.Query;
import org.springframework.data.elasticsearch.core.query.StringQuery;
import org.springframework.stereotype.Service;

@Service
public class TransactionSearchService {

    private final ElasticsearchOperations esOps;

    public TransactionSearchService(ElasticsearchOperations esOps) {
        this.esOps = esOps;
    }

    // Elasticsearch for search, PostgreSQL for the ledger
    public SearchHits<TransactionDocument> searchTransactions(
            String queryText) {
        Query query = new StringQuery(
            "{\"match\":{\"description\":\"" + queryText + "\"}}");
        return esOps.search(query, TransactionDocument.class);
    }
}

What are common mistakes?

The most common mistake is using Elasticsearch as the system of record for money. It has no multi-document transactions and only near-real-time visibility.

JAVA
// WRONG: using Elasticsearch as the ledger
esOps.save(new TransactionDocument(transfer)); // no ACID!

// CORRECT: PostgreSQL is the ledger, ES is the search index
transactionRepository.save(transfer);           // ACID in PostgreSQL
esOps.save(TransactionDocument.from(transfer)); // searchable copy in ES

What should a senior engineer know?

A senior engineer knows that Elasticsearch shines for read-heavy search, logs, metrics, and analytics, while the system of record for money stays in a transactional database. They size shards deliberately, keep analyzers consistent between index time and query time, and use index lifecycle management to control cost.

Visual Overview

DIAGRAM
  Document indexed
       │
       ▼
  ┌──────────────────────────┐
  │     In-Memory Buffer      │
  │  + Translog (crash safe)  │
  └────────────┬─────────────┘
               │  refresh (every 1s)
               ▼
  ┌──────────────────────────┐
  │   New Lucene Segment      │ ← now searchable
  └──────────────┬───────────┘
                 │  background merge
                 ▼
  ┌──────────────────────────┐
  │   Merged Segment          │ ← larger, fewer segments
  └──────────────────────────┘

  Search query
       │
       ▼
  Coordinating Node
       │  scatter
       ├──→ Shard 0 (search local inverted index)
       ├──→ Shard 1 (search local inverted index)
       └──→ Shard 2 (search local inverted index)
              │  gather
              ▼
       Merged results → client

Key Takeaway

Elasticsearch is a distributed search engine that stores JSON documents and makes them searchable in near-real-time through four mechanisms. The inverted index maps each term to the documents containing it, so search is a lookup, not a scan. Analyzers normalize text into tokens so searches match regardless of case or word form. Sharding splits an index across nodes for horizontal scaling, with replicas for fault tolerance. Near-real-time means a one-second refresh delay between indexing and searchability.

The architecture is optimized for read-heavy search and analytics, not for transactional consistency. At National Bank of India, the system of record for money always stayed in PostgreSQL, while Elasticsearch powered the fraud and support teams' fast searching over millions of transaction audit records.

The senior-level judgement is knowing where Elasticsearch fits and where it does not, sizing shards deliberately, keeping analyzers consistent, and using index lifecycle management to control storage cost.

Remember one sentence: Elasticsearch flips the search problem by mapping words to documents instead of scanning documents for words, and that inverted index is why it searches billions of records in milliseconds.

---

Share
Use ← → arrow keys to navigate