Back to Event-Driven Architecture & CQRS
Q6Question 6 of 8

The pinball architecture / chained-choreography anti-pattern

Key Concepts
  • Pinball architecture is a complex workflow bouncing through services as chained events with no coordinator.
  • Observability collapses because no component holds the workflow state, making 'where is my order?' unanswerable.
  • Compensation breaks down because rollback logic is scattered across services with implicit ordering.
  • The hidden event dependency web makes changing or testing the flow risky and error-prone.
  • The remedy is an orchestrated Saga with a central state machine that owns steps and compensations.
  • Workflow engines like Temporal, Camunda, or Spring State Machine provide durable state, retries, and visibility.
  • Keep choreography for simple independent fan-outs and switch to orchestration when the flow is complex and stateful.

Definition

Pinball architecture is a complex workflow bouncing through services with no coordinator, which destroys observability and compensation; the fix is an orchestrated Saga with a central state machine.
  • Pinball architecture – the anti-pattern where a complex, multi-step workflow is modeled purely as reactive events chained across many services with no central coordinator, so a request ricochets like a ball in a pinball machine.
  • Loss of observability – no component holds the workflow state, so answering "what state is this order in?" requires stitching together logs from many services.
  • Broken compensation – rollback logic is scattered across failure-event listeners with implicit ordering, making it fragile and incomplete.
  • Remedy – switch to an orchestrated Saga with a central state machine (Spring State Machine, Temporal, or Camunda) that owns the steps and compensating transactions.
  • Decision rule – keep choreography for simple fan-outs; switch to orchestration when the flow is complex, ordered, and stateful.

Imagine a loan application at National Bank of India where each department just reacts to the previous one and flings the file onward with no manager. Intake flings it to verification, verification to credit scoring, scoring to approval, approval to disbursement, like a pinball bouncing off bumpers. When Priya asks "where is my loan application?" nobody can answer. When disbursement fails, there is no manager to say "undo the approval, unfreeze the reserved funds." Replacing the pinball machine with a case manager who holds the file and knows every step is the orchestrated Saga.

Understanding the Concepts

What is the pinball architecture anti-pattern?

The pinball anti-pattern is what you get when you push choreography far past its comfort zone. A complex, ordered, stateful workflow is modeled as a long chain of events across many services, with each service only knowing its own slice. The flow is implicit, scattered across subscriptions, and owned by no one.

JAVA
// Pinball: 7-step loan disbursement as chained choreography
// IntakeService → publishes LoanRequested
// VerificationService → hears LoanRequested → publishes VerificationCompleted
// CreditScoringService → hears VerificationCompleted → publishes ScoreReady
// ApprovalService → hears ScoreReady → publishes LoanApproved
// FundReservationService → hears LoanApproved → publishes FundsReserved
// DisbursementService → hears FundsReserved → publishes LoanDisbursed
// NotificationService → hears LoanDisbursed → sends SMS

// The flow is a chain: A → B → C → D → E → F → G
// Nobody owns the chain. Nobody knows the full picture.

Why does observability collapse?

Because no component holds the workflow state, there is no single place to ask "what state is this loan in?" or "how far did Priya's application get?"

JAVA
package com.bank.support;

public class CustomerSupportProblem {

    // Priya calls: "Where is my loan application LOAN-5001?"
    // Support agent must:
    // 1. Check Intake Service logs for LOAN-5001
    // 2. Check Verification Service logs
    // 3. Check Credit Scoring Service logs
    // 4. Check Approval Service logs
    // 5. Check Fund Reservation Service logs
    // 6. Check Disbursement Service logs

    // This takes 15 minutes of log searching across 6 services
    // And if any service dropped the event silently? Nobody knows.

    // With orchestration:
    // sagaRepo.findBySagaId("LOAN-5001") → state: "FUNDS_RESERVED"
    // Answer in 1 second from one query
}
DIAGRAM
// Choreography: "Where is LOAN-5001?" → search 6 services' logs → 15 min
// Orchestration: "Where is LOAN-5001?" → query state machine → 1 second

Why does compensation break down?

In a complex flow, a late failure must undo earlier steps in the right order. With no coordinator, each service handles failure events independently, and the compensation order is implicit and fragile.

JAVA
// Disbursement fails after funds are reserved and loan is approved
// Who compensates? In what order?

// Choreography attempt:
// DisbursementService → publishes DisbursementFailed
// FundReservationService → hears DisbursementFailed → releases funds
// ApprovalService → hears DisbursementFailed → revokes approval
// But what if ApprovalService processes BEFORE FundReservation?
// What if FundReservation's listener has a bug and misses the event?
// What if VerificationService also needs to be compensated?

// Compensation logic is smeared across every service
// The ordering is implicit and depends on event delivery timing
// Missed compensations leave money in a bad state
JAVA
// Orchestration: compensations driven from ONE place
package com.bank.saga;

public class LoanSagaOrchestrator {

    public void compensateOnDisbursementFailure(String loanId) {
        // Compensations in explicit reverse order:
        fundReservationClient.releaseFunds(loanId);  // C5
        approvalClient.revokeApproval(loanId);         // C4
        // Verification and scoring are read-only, no compensation needed
        intakeClient.cancelApplication(loanId);        // C1
        sagaRepo.updateState(loanId, "COMPENSATED");
    }
}
TEXT
// Choreography: compensation scattered, implicit, fragile
// Orchestration: compensation centralized, explicit, reliable

When should you switch to orchestration?

Switch when the workflow becomes complex, ordered, and stateful, or when it needs real compensations and observable status.

JAVA
// Warning signs that you need orchestration:
// 1. You cannot answer "what state is this order/loan in?" from any single place
// 2. Rollback logic is scattered across failure event listeners
// 3. Changing the flow order (inserting a step) risks breaking the chain
// 4. Testing the end-to-end flow requires starting all services
// 5. Customer support spends 15 minutes searching logs for one request

// Keep choreography when:
// 1. Reactions are independent (notification, analytics, fraud scoring)
// 2. No coordination or ordering needed between reactions
// 3. Adding a new reactor should be zero-change

How do workflow engines help?

Workflow engines like Temporal, Camunda, and Spring State Machine provide durable state, retries, timeouts, versioning, and visibility out of the box.

JAVA
package com.bank.saga;

import org.springframework.statemachine.StateMachine;
import org.springframework.statemachine.config.StateMachineBuilder;

public class LoanSagaStateMachine {

    // Spring State Machine definition:
    // States: STARTED, VERIFIED, SCORED, APPROVED, FUNDS_RESERVED,
    //         DISBURSED, COMPENSATING, COMPENSATED, FAILED
    // Transitions:
    //   STARTED → VERIFIED (on VERIFICATION_COMPLETE)
    //   VERIFIED → SCORED (on SCORE_READY)
    //   SCORED → APPROVED (on LOAN_APPROVED)
    //   APPROVED → FUNDS_RESERVED (on FUNDS_RESERVED)
    //   FUNDS_RESERVED → DISBURSED (on DISBURSEMENT_COMPLETE)
    //   any → COMPENSATING (on STEP_FAILED)
    //   COMPENSATING → COMPENSATED (on COMPENSATION_COMPLETE)

    // Benefits:
    // - State is queryable at any time
    // - Transitions are explicit and testable
    // - Compensation path is a first-class transition
    // - State is persisted for crash recovery
}
TEXT
// Temporal workflow (pseudocode):
// @WorkflowMethod
// public void processLoan(LoanRequest request) {
//     verify(request);       // retries automatically on failure
//     score(request);        // durable state survives restarts
//     approve(request);      // timeout triggers compensation
//     reserveFunds(request); // all state queryable via Temporal UI
//     disburse(request);     // version-safe for long-running workflows
// }

What are common mistakes? (wrong vs correct)

The most common mistake is starting with choreography for a complex flow and not switching to orchestration when the pain becomes obvious.

JAVA
// WRONG: 7-step loan flow as chained choreography
// "We chose choreography for loose coupling"
// Result: nobody can track loan state, compensation is broken,
// changing the flow breaks event chains, testing requires all 7 services

// CORRECT: orchestrate the core flow
// LoanSagaOrchestrator owns the state machine
// Query state: sagaRepo.findByLoanId("LOAN-5001") → APPROVED
// Compensate: orchestrator drives in reverse from one place
// Test: mock the service clients, test the state machine in isolation

// ALSO CORRECT: choreograph the independent reactions
// After disbursement: publish LoanDisbursed event
// Notification, Statement, Analytics react independently

What should a senior engineer know?

A senior engineer matches coordination to workflow complexity. They watch for the warning signs: a business process whose state cannot be answered from any single place and whose rollback logic is scattered across failure listeners. That is the signal to put a state machine in charge.

JAVA
// Senior decision matrix:
// Simple, independent reactions     → Choreography
// Complex, ordered, compensatable   → Orchestration
// Long-running (hours/days)         → Temporal or Camunda
// Short, in-process state machine   → Spring State Machine
// Mixed flow                        → Hybrid (orchestrate core, choreograph fan-out)

// Never: god orchestrator (domain logic in the coordinator)
// Never: pinball (complex flow as chained events)

How is choreography different from orchestration? (comparison table)

| Aspect | Choreography | Orchestration |
|---|---|---|
| Coordination | Decentralized, event-driven | Centralized state machine |
| Flow visibility | Implicit (scattered across consumers) | Explicit (one component owns the flow) |
| Compensation | Scattered across failure listeners | Centralized, driven in reverse order |
| Coupling | Services coupled to events, not each other | Services coupled to orchestrator |
| Testing | Requires all services running | Mock service clients, test state machine |
| Adding a step | Zero-change for independent reactions | Change one orchestrator |
| Best for | Simple, independent fan-outs | Complex, ordered, stateful workflows |

JAVA
package com.bank.architecture;

public class ChoreographyVsOrchestration {
    public static void main(String[] args) {
        System.out.println("Choreography (independent reactions):");
        System.out.println("  After disbursement: publish LoanDisbursed");
        System.out.println("  Notification listens -> sends SMS");
        System.out.println("  Analytics listens -> updates dashboard");
        System.out.println("  Adding a new listener = zero change to existing services");
        System.out.println();
        System.out.println("Orchestration (ordered, stateful workflow):");
        System.out.println("  Orchestrator drives: verify -> score -> approve -> disburse");
        System.out.println("  State queryable: sagaRepo.findByLoanId('LOAN-5001')");
        System.out.println("  Failure at step 4: compensate steps 3, 2, 1 in order");
    }
}
TEXT
Expected output:
Choreography (independent reactions):
  After disbursement: publish LoanDisbursed
  Notification listens -> sends SMS
  Analytics listens -> updates dashboard
  Adding a new listener = zero change to existing services

Orchestration (ordered, stateful workflow):
  Orchestrator drives: verify -> score -> approve -> disburse
  State queryable: sagaRepo.findByLoanId('LOAN-5001')
  Failure at step 4: compensate steps 3, 2, 1 in order

Visual Overview

DIAGRAM
  Pinball Architecture (Anti-Pattern)
  ────────────────────────────────────
  Request ──▶ Svc A ──event──▶ Svc B ──event──▶ Svc C ──event──▶ Svc D
                                                                   │
               ◀──failure──  ◀──failure──  ◀──failure── Svc D FAILS│
              Who compensates?  In what order?  Nobody knows!

  Orchestrated Saga (Remedy)
  ──────────────────────────
  ┌──────────────────────────────────────┐
  │         Saga Orchestrator             │
  │  State: [STARTED → VERIFIED → SCORED │
  │          → APPROVED → DISBURSED]     │
  │                                       │
  │  On failure at any step:             │
  │  → Drive compensations in reverse    │
  │  → Record state: COMPENSATED         │
  │  → Answer: "Loan is at APPROVED"     │
  └──┬─────┬─────┬──────┬──────┬────────┘
     │     │     │      │      │
     ▼     ▼     ▼      ▼      ▼
  Intake Verify Score Approve Disburse

  Each service does its local work.
  Orchestrator owns the flow, state, and compensation.

Key Takeaway

The pinball architecture is what happens when a genuinely complex, multi-step, stateful workflow is modeled purely as reactive events bouncing from service to service with no coordinator. Each service knows only its own slice, and the overall flow is implicit, scattered across subscriptions, and owned by no one. Choreography is perfectly good for simple, independent reactions, but pushing it onto a seven-step loan process with money and rollbacks turns it into a nightmare.

At National Bank of India, Pankaj's team first built loan disbursement as chained choreography: intake flung the file to verification, verification to scoring, scoring to approval, and so on. When Priya called asking where her loan was, customer support spent fifteen minutes searching logs across six services. When disbursement failed, the compensation was scattered across failure-event listeners, and twice the fund reservation was not released because the listener had a bug. The team switched to an orchestrated Saga with a central state machine, and overnight the loan's state became queryable from one place, compensations were driven in explicit reverse order from one component, and the flow could be changed, tested, and observed as a unit.

The decision rule is to keep choreography for simple, independent fan-outs where adding new reactors should be zero-change, and switch to orchestration the moment the workflow becomes complex, ordered, and stateful, or needs real compensations and observable status. A workflow engine like Temporal or Camunda gives durable state, retries, timeouts, and visibility out of the box.

Remember one sentence: when a business process's state cannot be answered from any single place and its rollback logic is scattered across failure listeners, replace the pinball machine with an orchestrated Saga and a central state machine.

---

Share
Use ← → arrow keys to navigate