Definition
- UPSERT – A single atomic statement that inserts a row if the key does not exist and updates it if the key already exists.
INSERT ... ON CONFLICT– PostgreSQL syntax where you name the conflicting constraint and provide aDO UPDATE SETclause; the proposed row is available via theEXCLUDEDalias.INSERT ... ON DUPLICATE KEY UPDATE– MySQL syntax that performs the same insert-or-update against a unique or primary key.MERGE– The SQL-standard statement withWHEN MATCHEDandWHEN NOT MATCHEDbranches, supporting insert, update, and delete in one pass.- Race condition – The naive SELECT-then-INSERT/UPDATE pattern allows another transaction to slip in between the read and write, causing duplicate-key errors or lost updates.
- Atomicity – The UPSERT performs the check-and-write as one indivisible operation protected by the unique constraint, so concurrency cannot interfere.
At National Bank of India, Rajesh maintains a daily-balance row per account. The naive way is to check if today's row exists, insert if absent, or update if present. Under load, two requests can both check at the same time, both see no row, and both try to insert – one crashes with a duplicate key. An UPSERT is like telling the clerk "create today's row, or if it already exists just add to it" in one breath, so two simultaneous requests cannot both think they are the first.
Understanding the Concepts
What is an UPSERT?
An UPSERT is a single SQL statement that inserts a row when the key does not exist and updates the row when the key already exists. The name combines "update" and "insert" into one word. It is an atomic operation, meaning the database treats the check-and-write as one indivisible step that cannot be interrupted by another transaction.
In our National Bank of India system, Rajesh needs to maintain a daily running balance per account. Each time a transfer arrives, the system must either create a new daily-balance row or add to an existing one.
-- PostgreSQL UPSERT: insert or update daily balance
INSERT INTO daily_balance (account_id, balance_date, total_amount)
VALUES ('SAV2001', '2024-01-15', 50000.00)
ON CONFLICT (account_id, balance_date)
DO UPDATE SET total_amount = daily_balance.total_amount + EXCLUDED.total_amount;
Expected output:
INSERT 0 1
-- If the row already existed, total_amount is incremented by 50000.00
-- If the row did not exist, a new row is created with 50000.00
Why does UPSERT exist?
The UPSERT exists because the naive alternative – a separate SELECT followed by an INSERT or UPDATE – has a race condition. Under concurrency, two transactions can both read the same state at the same time, both decide the row is missing, and both attempt to insert. One succeeds and the other crashes with a duplicate-key violation, or worse, one overwrites the other's data silently.
Consider the dangerous pattern in a banking scenario:
-- DANGEROUS: race condition in concurrent transfers
-- Transaction A and Transaction B both run this at the same time
SELECT total_amount FROM daily_balance
WHERE account_id = 'SAV2001' AND balance_date = '2024-01-15';
-- Both see: no row exists
-- Both execute:
INSERT INTO daily_balance (account_id, balance_date, total_amount)
VALUES ('SAV2001', '2024-01-15', 50000.00);
-- Transaction A succeeds. Transaction B gets: ERROR duplicate key.
How does PostgreSQL implement UPSERT?
PostgreSQL uses the INSERT ... ON CONFLICT syntax. You name the columns or constraint that might conflict, and provide a DO UPDATE SET clause. Inside the update clause, the EXCLUDED keyword refers to the row that was proposed for insertion but could not be inserted due to the conflict.
-- PostgreSQL: accumulate daily transaction totals per account
INSERT INTO daily_balance (account_id, balance_date, total_amount, txn_count)
VALUES ('SAV2001', '2024-01-15', 75000.00, 1)
ON CONFLICT (account_id, balance_date)
DO UPDATE SET
total_amount = daily_balance.total_amount + EXCLUDED.total_amount,
txn_count = daily_balance.txn_count + EXCLUDED.txn_count;
You can also use ON CONFLICT DO NOTHING to silently skip the insert if the key already exists, which is useful for idempotent imports.
-- Skip duplicate customer imports silently
INSERT INTO customers (pan, full_name, branch)
VALUES ('ABCDE1234F', 'Priya Sharma', 'Mumbai-Central')
ON CONFLICT (pan) DO NOTHING;
How does MySQL implement UPSERT?
MySQL uses INSERT ... ON DUPLICATE KEY UPDATE. The concept is identical but the syntax differs. The new values are referenced using the VALUES() function in older versions or an alias in MySQL 8.0.19 and later.
-- MySQL: upsert daily balance
INSERT INTO daily_balance (account_id, balance_date, total_amount)
VALUES ('SAV2001', '2024-01-15', 50000.00)
ON DUPLICATE KEY UPDATE
total_amount = total_amount + VALUES(total_amount);
How does the SQL-standard MERGE work?
The MERGE statement is the SQL-standard approach. It compares a target table against a source and runs different actions depending on whether rows match. It is more powerful than a simple upsert because it supports insert, update, and delete branches.
-- SQL-standard MERGE: reconcile partner bank feed
MERGE INTO accounts AS target
USING partner_feed AS source
ON target.account_id = source.account_id
WHEN MATCHED AND source.status = 'CLOSED' THEN
DELETE
WHEN MATCHED THEN
UPDATE SET target.balance = source.balance,
target.updated_at = CURRENT_TIMESTAMP
WHEN NOT MATCHED THEN
INSERT (account_id, balance, status, created_at)
VALUES (source.account_id, source.balance, 'ACTIVE', CURRENT_TIMESTAMP);
How is UPSERT different from SELECT-then-INSERT/UPDATE?
| Aspect | UPSERT | SELECT-then-INSERT/UPDATE |
|---|---|---|
| Atomicity | One indivisible statement | Two separate statements with a gap |
| Concurrency safety | Protected by the unique constraint lock | Race condition between read and write |
| Performance | Single round trip to the database | Two or more round trips |
| Error handling | No duplicate-key errors possible | Must catch and retry on conflicts |
| Code complexity | One SQL statement | Branching logic in application code |
package com.bank.service;
import java.math.BigDecimal;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
public class DailyBalanceService {
// WRONG: race condition under concurrency
public void updateBalanceUnsafe(Connection conn, String accountId,
BigDecimal amount) throws SQLException {
var rs = conn.prepareStatement(
"SELECT total_amount FROM daily_balance WHERE account_id = ? AND balance_date = CURRENT_DATE"
).executeQuery();
if (rs.next()) {
// Another thread may have inserted between SELECT and UPDATE
conn.prepareStatement(
"UPDATE daily_balance SET total_amount = total_amount + ? WHERE account_id = ? AND balance_date = CURRENT_DATE"
).executeUpdate();
} else {
// Another thread may also see 'no row' and try INSERT
conn.prepareStatement(
"INSERT INTO daily_balance (account_id, balance_date, total_amount) VALUES (?, CURRENT_DATE, ?)"
).executeUpdate();
}
}
// CORRECT: atomic upsert, no race condition
public void updateBalanceSafe(Connection conn, String accountId,
BigDecimal amount) throws SQLException {
String sql = """
INSERT INTO daily_balance (account_id, balance_date, total_amount)
VALUES (?, CURRENT_DATE, ?)
ON CONFLICT (account_id, balance_date)
DO UPDATE SET total_amount = daily_balance.total_amount + EXCLUDED.total_amount
""";
try (PreparedStatement ps = conn.prepareStatement(sql)) {
ps.setString(1, accountId);
ps.setBigDecimal(2, amount);
ps.executeUpdate();
}
}
}
What are the rules and constraints of UPSERT?
The ON CONFLICT clause requires an actual unique or exclusion constraint to conflict against. You cannot resolve on an arbitrary WHERE condition. The constraint must be declared on the table, and the conflict columns must exactly match those in the constraint.
- The conflicting constraint must be a
UNIQUEindex orPRIMARY KEY. - The
EXCLUDEDalias is only available inside theDO UPDATE SETclause. - You can add a
WHEREclause to theDO UPDATEto conditionally skip the update even on conflict. ON CONFLICT DO NOTHINGsilently skips without any update.
-- Conditional upsert: only update if the new amount is larger
INSERT INTO daily_balance (account_id, balance_date, total_amount)
VALUES ('SAV2001', '2024-01-15', 50000.00)
ON CONFLICT (account_id, balance_date)
DO UPDATE SET total_amount = EXCLUDED.total_amount
WHERE EXCLUDED.total_amount > daily_balance.total_amount;
What about auto-increment and sequence gaps?
A conflicting insert may still consume a sequence value before the conflict is detected. Sequences are non-transactional by design – they never roll back. The drawn value is simply lost, producing a gap in the generated identifiers. This is normal and intentional.
-- Sequence gap demonstration
CREATE TABLE loan_applications (
id SERIAL PRIMARY KEY,
pan VARCHAR(10) UNIQUE,
amount NUMERIC(15,2)
);
-- First insert: id = 1
INSERT INTO loan_applications (pan, amount) VALUES ('ABCDE1234F', 500000.00)
ON CONFLICT (pan) DO UPDATE SET amount = EXCLUDED.amount;
-- Second insert with same PAN: id 2 is consumed but conflict routes to update
INSERT INTO loan_applications (pan, amount) VALUES ('ABCDE1234F', 750000.00)
ON CONFLICT (pan) DO UPDATE SET amount = EXCLUDED.amount;
-- Next new PAN gets id = 3 (id 2 is a gap)
INSERT INTO loan_applications (pan, amount) VALUES ('FGHIJ5678K', 300000.00)
ON CONFLICT (pan) DO UPDATE SET amount = EXCLUDED.amount;
Expected result:
id | pan | amount
----+------------+-----------
1 | ABCDE1234F | 750000.00
3 | FGHIJ5678K | 300000.00
-- id 2 is a gap, which is normal
What about concurrency and deadlocks?
Under high contention, UPSERT can still deadlock or wait on row locks. Two transactions upserting different keys in opposite order can each hold one lock and wait for the other. The mitigation is to keep transactions short and insert keys in a consistent order.
-- Deadlock scenario:
-- Transaction A: UPSERT account SAV2001, then SAV2002
-- Transaction B: UPSERT account SAV2002, then SAV2001
-- Each holds one row lock and waits for the other.
-- Prevention: always upsert in sorted account_id order
-- Application code sorts the batch before executing
What are common mistakes with UPSERT?
The most common mistake is forgetting the unique constraint. Without a declared unique index, ON CONFLICT has nothing to conflict against and the statement fails.
-- WRONG: no unique constraint on (account_id, balance_date)
CREATE TABLE daily_balance_bad (
account_id VARCHAR(20),
balance_date DATE,
total_amount NUMERIC(15,2)
);
INSERT INTO daily_balance_bad (account_id, balance_date, total_amount)
VALUES ('SAV2001', '2024-01-15', 50000.00)
ON CONFLICT (account_id, balance_date) DO UPDATE SET total_amount = EXCLUDED.total_amount;
-- ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification
-- CORRECT: add the unique constraint first
CREATE TABLE daily_balance_good (
account_id VARCHAR(20),
balance_date DATE,
total_amount NUMERIC(15,2),
UNIQUE (account_id, balance_date)
);
When should you use UPSERT?
Use UPSERT when your application must idempotently create-or-update a row and concurrency is possible. Common banking scenarios include maintaining daily balances, importing customer feeds, tracking running totals, and processing idempotent payment notifications.
What should a senior engineer know?
A senior engineer knows that ON CONFLICT requires a real unique constraint, that sequences produce harmless gaps, that high contention demands short transactions and consistent key ordering, and that for gapless numbers like legal invoice series you must use a dedicated counter row updated inside the transaction rather than a bare sequence.
Visual Overview
┌─────────────────────────────────────────────────┐
│ UPSERT Flow (PostgreSQL) │
├─────────────────────────────────────────────────┤
│ │
│ INSERT INTO daily_balance (account_id, ...) │
│ VALUES ('SAV2001', '2024-01-15', 50000) │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ Check unique constraint │ │
│ │ (account_id, balance_date)│ │
│ └──────────┬───────────────┘ │
│ ┌────┴─────┐ │
│ │ Conflict?│ │
│ └────┬─────┘ │
│ No │ Yes │
│ ┌─────┴──────┐ │
│ ▼ ▼ │
│ ┌─────────┐ ┌──────────────────┐ │
│ │ INSERT │ │ DO UPDATE SET │ │
│ │ new row │ │ total = old + │ │
│ │ │ │ EXCLUDED.total │ │
│ └─────────┘ └──────────────────┘ │
│ │ │ │
│ └─────┬──────┘ │
│ ▼ │
│ ┌──────────────┐ │
│ │ COMMIT │ │
│ └──────────────┘ │
│ │
├─────────────────────────────────────────────────┤
│ vs. Naive SELECT-then-INSERT (RACE CONDITION) │
├─────────────────────────────────────────────────┤
│ │
│ Txn A: SELECT → no row │
│ Txn B: SELECT → no row ← both see empty │
│ Txn A: INSERT → success │
│ Txn B: INSERT → DUPLICATE KEY ERROR │
│ │
└─────────────────────────────────────────────────┘
Key Takeaway
An UPSERT is a single atomic statement that inserts a row if it does not exist and updates it if it does. PostgreSQL writes it as INSERT ... ON CONFLICT (key) DO UPDATE SET ..., where the proposed new row is available under the EXCLUDED alias. MySQL writes it as INSERT ... ON DUPLICATE KEY UPDATE. The SQL standard offers MERGE with matched and not-matched branches for more complex scenarios. The whole point is that the check and the write happen as one indivisible operation protected by a unique or primary-key constraint.
The UPSERT is safer than the naive SELECT-then-INSERT-or-UPDATE pattern because that pattern contains a check-then-act race condition. Under concurrency, two transactions can both run the SELECT, both see the key missing, and both attempt the INSERT. One succeeds and the other either fails with a duplicate-key violation or, in a worse design, overwrites and loses the first write. The UPSERT removes that window entirely because the engine takes the right row-level lock so exactly one competing statement wins the insert and the others are routed into the update branch.
In our National Bank of India system, Rajesh's team maintains a daily running balance per account. Under load, many transfers for the same account arrive at once. With the naive check-then-act path they saw duplicate-key errors and incorrect totals because concurrent inserts collided instead of accumulating. Switching to INSERT ... ON CONFLICT DO UPDATE SET total_amount = daily_balance.total_amount + EXCLUDED.total_amount made every concurrent contribution serialize correctly on the primary key, so four simultaneous fifty-thousand-rupee transfers produced exactly two lakh with no errors.
The caveats to keep in mind are that ON CONFLICT needs a real unique constraint to conflict against, that heavy contention can still cause waits so you keep the transaction short, and that a conflicting insert can consume a sequence value and leave a harmless gap in generated IDs. For gapless numbers like legal invoice series, use a dedicated counter row inside the transaction rather than a bare sequence.
Remember one sentence: An UPSERT inserts-or-updates in one atomic statement, and it beats SELECT-then-write because the separate read and write race under concurrency.
---