Back to CI/CD Pipeline
Q42Question 42 of 56

Concurrent Pushes and How Parallel and Matrix Builds Work

Key Concepts
  • Git serialises concurrent pushes: each commit gets a unique SHA and position in history.
  • The CI system triggers one independent pipeline run per commit, each on its own fresh agent.
  • Shared resources like a QA deploy slot are protected with concurrency controls so only one run uses them at a time.
  • Parallel builds run independent stages of a single pipeline at the same time to cut wall-clock time.
  • A matrix build auto-expands one job across parameter combinations like Java versions or services, running each in parallel.
  • Concurrent deploys to main must be ordered so the newest commit deploys last.

Definition

Concurrent pushes get separate pipeline runs in parallel, shared resources are guarded by concurrency locks, and a matrix build auto-expands one job across parameter combinations.
  • Git serialisation – Git accepts pushes one at a time; each commit lands at a unique position with a unique SHA.
  • Independent runs – the CI system triggers a separate pipeline per commit, each on its own fresh agent, building in parallel.
  • Concurrency control – shared stateful resources (deploy slots, licenses) are protected so only one run uses them at a time.
  • Parallel stages – independent stages within one pipeline run at the same time in a fan-out/fan-in pattern.
  • Matrix build – a strategy that auto-generates one job per combination of declared parameters (e.g., Java 17 x 21 cross transfer x fraud = 4 jobs).

Imagine two tellers at the National Bank of India each submitting a batch of cheques at the same moment. The clearing system gives each batch its own serial number and processes them on separate machines at once. But there is only one armoured van to the vault, so they take turns. A matrix build is testing a new cheque scanner against every combination of paper type and ink colour at once: you list the combinations and the system runs them all in parallel.

Understanding the Concepts

How does Git handle concurrent pushes?

Git on the server accepts pushes one at a time. If Rajesh and Priya push to the same branch simultaneously, one wins and the other is rejected with a non-fast-forward error. If they push to different branches, both succeed independently.

BASH
# Rajesh pushes to main
git push origin main
# Success: commit abc1234

# Priya pushes to main at the same time
git push origin main
# Error: non-fast-forward. Priya must pull and rebase.
git pull --rebase origin main
git push origin main
# Success: commit def5678

Every commit ends up at a unique position with a unique SHA, so there is never ambiguity about what each pipeline builds.

How does the CI system handle concurrent runs?

The CI system triggers one independent run per triggering event. Two concurrent pushes produce two independent runs, each on its own fresh agent.

YAML
# GitHub Actions: each push triggers its own run
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest   # Each run gets its own fresh runner
    steps:
      - uses: actions/checkout@v4
      - run: mvn clean verify

On Jenkins-on-Kubernetes, each run gets its own pod. The two runs build in parallel with no shared workspace and cannot corrupt each other.

How do you protect shared resources?

The build and test phases are naturally isolated, but some resources are singular: a shared QA deploy slot, a limited license count, or an external system that cannot handle two simultaneous writers.

YAML
# GitHub Actions: concurrency group ensures only one deploy at a time
jobs:
  deploy-qa:
    concurrency:
      group: deploy-qa          # Only one run in this group at a time
      cancel-in-progress: true  # Cancel older run if a newer one starts
    steps:
      - run: kubectl apply -f qa/
GROOVY
// Jenkins: lock step serialises access to a shared resource
pipeline {
    agent any
    stages {
        stage('Build') { steps { sh 'mvn package' } }   // Runs in parallel
        stage('Deploy QA') {
            steps {
                lock('qa-deploy-slot') {   // Only one job at a time
                    sh 'kubectl apply -f qa/'
                }
            }
        }
    }
}

How do parallel stages work inside one pipeline?

Independent stages like unit tests, integration tests, and a security scan can run at the same time because they do not depend on each other's output.

GROOVY
// Jenkins: parallel stages cut wall-clock time
pipeline {
    agent any
    stages {
        stage('Build') { steps { sh 'mvn clean compile' } }
        stage('Parallel Checks') {
            parallel {
                stage('Unit Tests')        { steps { sh 'mvn test' } }
                stage('Integration Tests') { steps { sh 'mvn verify -Pintegration' } }
                stage('Security Scan')     { steps { sh 'mvn sonar:sonar' } }
            }
        }
        stage('Deploy') { steps { sh 'kubectl apply -f deploy/' } }
    }
}
YAML
# GitHub Actions: parallel jobs with no needs dependency
jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps: [{run: 'mvn test'}]
  integration-tests:
    runs-on: ubuntu-latest
    steps: [{run: 'mvn verify -Pintegration'}]
  security-scan:
    runs-on: ubuntu-latest
    steps: [{run: 'mvn sonar:sonar'}]
  deploy:
    needs: [unit-tests, integration-tests, security-scan]  # Fan-in
    runs-on: ubuntu-latest
    steps: [{run: 'kubectl apply -f deploy/'}]

How does a matrix build work?

A matrix declares dimensions and the CI system generates one job per combination automatically.

YAML
# GitHub Actions: matrix across Java versions and services
jobs:
  build:
    strategy:
      matrix:
        java: [17, 21]
        service: [transfer-service, fraud-service]
      fail-fast: false      # Do not cancel others on one failure
      max-parallel: 4       # Cap concurrent jobs
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-java@v4
        with: { java-version: '${{ matrix.java }}', distribution: temurin }
      - run: mvn clean verify -pl ${{ matrix.service }}

This generates four jobs: transfer-service on Java 17, transfer-service on Java 21, fraud-service on Java 17, fraud-service on Java 21. All four run in parallel.

What are common mistakes with concurrency?

YAML
# WRONG: no concurrency group on deploy.
# Two merges to main both try to deploy at the same time.
# The environment may end up on the older commit if it finishes last.
jobs:
  deploy:
    steps:
      - run: kubectl apply -f prod/

# CORRECT: concurrency group ensures the newer commit deploys last.
jobs:
  deploy:
    concurrency:
      group: deploy-prod
      cancel-in-progress: false   # Let the newer one finish
    steps:
      - run: kubectl apply -f prod/
YAML
# WRONG: unbounded matrix swamps the runner fleet.
strategy:
  matrix:
    service: [s1, s2, s3, s4, s5, s6, s7, s8, s9, s10]  # 10 services
    java: [17, 21]                                        # x 2 = 20 jobs!

# CORRECT: cap with max-parallel.
strategy:
  matrix:
    service: [s1, s2, s3, s4, s5, s6, s7, s8, s9, s10]
    java: [17, 21]
  max-parallel: 6   # Only 6 run at once

How do parallel tests expose shared state bugs?

Running tests in parallel can surface tests that share state, like a static field, a fixed port, or the same database row.

JAVA
// WRONG: tests share a static counter.
public class TransferServiceTest {
    private static int transferCount = 0;

    @Test
    void testTransferA() {
        transferCount++;
        assertEquals(1, transferCount); // Fails if another test ran first!
    }
}

// CORRECT: each test has its own state.
public class TransferServiceTest {
    @Test
    void testTransferA() {
        int localCount = 0;
        localCount++;
        assertEquals(1, localCount); // Always passes.
    }
}

What should a senior engineer know?

  • Concurrent deploys to the same environment must be ordered so the newest commit wins and the environment ends on the latest code.
  • Cap matrix expansion with max-parallel and sensible dimension sizes to avoid swamping a shared CI fleet.
  • Parallel tests expose any test that secretly shares state, which must be made independent.
  • The cancel-in-progress option on concurrency groups can cancel older runs, which is useful for CI but dangerous for CD where you might cancel a production deploy.

Visual Overview

DIAGRAM
  Concurrent pushes:

  Rajesh push ──▶ commit abc1234 ──▶ Pipeline Run 1 (Pod A)
  Priya  push ──▶ commit def5678 ──▶ Pipeline Run 2 (Pod B)
                                          │
                   Both run in parallel   │
                   on separate agents     │
                                          ▼
                   Shared QA slot ──▶ Lock: one at a time

  Parallel stages (fan-out / fan-in):

              ┌─── Unit Tests ────────┐
  Build ────▶ ├─── Integration Tests ─┤ ────▶ Deploy
              └─── Security Scan ─────┘

  Matrix build:

  java: [17, 21]  x  service: [transfer, fraud]
       ▼
  ┌────────────────┐  ┌────────────────┐
  │ transfer + J17 │  │ transfer + J21 │
  └────────────────┘  └────────────────┘
  ┌────────────────┐  ┌────────────────┐
  │ fraud    + J17 │  │ fraud    + J21 │
  └────────────────┘  └────────────────┘
       All 4 run in parallel

Key Takeaway

When two developers push at the same time, nothing gets mixed up because Git serialises the writes and the CI system triggers one independent run per commit, each on its own fresh agent. The two runs build in parallel with no shared workspace. The only thing that needs care is shared stateful resources like a QA deploy slot, which I protect with a concurrency group or lock so only one run uses it at a time.

Parallelism inside a single run speeds up the critical path. Independent stages like unit tests, integration tests, and a security scan run simultaneously in a fan-out, and the pipeline continues only when all finish (fan-in). A matrix build is parameterised parallelism: I declare dimensions like Java versions crossed with services, and the CI system generates one parallel job per combination automatically.

The senior concerns are ordering and cost. Concurrent deploys to main must be ordered so the newest commit deploys last and the environment ends on the latest code. I cap matrix expansion with max-parallel so a huge matrix does not swamp the CI fleet, and I make tests independent so parallel execution does not surface shared-state bugs.

Remember one sentence: concurrent pushes get independent parallel runs, shared resources get locks, and a matrix auto-expands one job across every parameter combination.

---

Share
Use ← → arrow keys to navigate