Definition
Unity Catalog provides centralized governance for all data and AI assets.
- Three-level namespace – Tables are addressed as
catalog.schema.table(e.g.,banking_prod.trades.settlements), replacing the old two-level model. - Centralized access control – SQL GRANT and REVOKE on users and groups synced from the company identity provider.
- Fine-grained security – Row filters and column masks so sensitive fields like account numbers can be partially hidden.
- Automatic auditing – Central, queryable log of who accessed which asset and when.
- Column-level lineage – Automatically captured as queries run, tracing data from source to consumer.
- Data discovery – Searchable catalog with descriptions, tags, and ownership.
It operates across clouds and workspaces from one place.
Imagine the National Bank of India has many branches (workspaces), and before Unity Catalog, each branch kept its own separate list of who could open which safe, with no shared log and no map of where money moved. Unity Catalog is like installing one central security office for the entire bank that controls every safe's keys, keeps one tamper-proof log of every door opened, draws a live map of how documents flow from one department to another, and provides a single searchable directory so staff can find the right record.
Understanding the Concepts
What is Unity Catalog?
Unity Catalog is Databricks' centralized governance layer that manages access control, auditing, data lineage, and data discovery for all data and AI assets across multiple workspaces and clouds from one place. Before Unity Catalog, governance was per-workspace, so permissions had to be duplicated and reconciled everywhere.
-- Three-level namespace: catalog.schema.table
SELECT * FROM banking_prod.trades.settlements
WHERE trade_date = '2025-06-15';
-- Separate environments as different catalogs
-- banking_prod.trades.settlements (production)
-- banking_dev.trades.settlements (development)
-- banking_staging.trades.settlements (staging)
How does the three-level namespace work?
Unity Catalog organizes everything as catalog, schema (database), and object (table, view, function). This replaces the old two-level database.table and lets you separate environments under one governance umbrella.
-- Create a catalog for banking production
CREATE CATALOG banking_prod;
-- Create a schema within the catalog
CREATE SCHEMA banking_prod.trades;
-- Create a table within the schema
CREATE TABLE banking_prod.trades.settlements (
trade_id STRING,
account_id STRING,
amount DECIMAL(18,2),
settlement_date DATE,
status STRING
);
-- Full path: banking_prod.trades.settlements
How does centralized access control work?
Access control uses familiar SQL GRANT and REVOKE statements. Permissions are defined on users and groups synced from the company identity provider, so removing someone from the provider revokes their access everywhere.
-- Grant SELECT on a specific table to the risk analysts group
GRANT SELECT ON TABLE banking_prod.trades.settlements TO risk_analysts;
-- Grant USE on schema so the group can see the schema's objects
GRANT USE SCHEMA ON SCHEMA banking_prod.trades TO risk_analysts;
-- Revoke access when a team no longer needs it
REVOKE SELECT ON TABLE banking_prod.trades.settlements FROM contractors;
-- Grant ALL PRIVILEGES on a catalog to the data engineering team
GRANT ALL PRIVILEGES ON CATALOG banking_prod TO data_engineering;
How do row filters and column masks work?
Unity Catalog supports fine-grained controls so sensitive data like account numbers can be masked for certain groups.
-- Column mask: show only last 4 digits of account_id to risk analysts
CREATE FUNCTION banking_prod.masks.mask_account_id(account_id STRING)
RETURNS STRING
RETURN CASE
WHEN is_member('data_engineering') THEN account_id
ELSE CONCAT('XXXX-', RIGHT(account_id, 4))
END;
ALTER TABLE banking_prod.trades.settlements
ALTER COLUMN account_id SET MASK banking_prod.masks.mask_account_id;
-- Row filter: risk analysts can only see their assigned branches
CREATE FUNCTION banking_prod.filters.branch_filter(branch_code STRING)
RETURNS BOOLEAN
RETURN CASE
WHEN is_member('data_engineering') THEN TRUE
ELSE branch_code IN (SELECT branch FROM user_branch_assignments WHERE user = current_user())
END;
ALTER TABLE banking_prod.trades.settlements
SET ROW FILTER banking_prod.filters.branch_filter ON (branch_code);
How does automatic auditing work?
Unity Catalog automatically records access events in a central, queryable log. Every read and write is captured.
-- Query the audit log to see who accessed sensitive data
SELECT
event_time,
user_identity.email AS user_email,
request_params.full_name_arg AS table_accessed,
action_name
FROM system.access.audit
WHERE request_params.full_name_arg = 'banking_prod.trades.settlements'
AND event_time >= '2025-06-01'
ORDER BY event_time DESC;
event_time user_email table_accessed action_name
2025-06-15 14:30:00 rajesh@bank.com banking_prod.trades.settlements commandSubmit
2025-06-15 10:15:00 priya@bank.com banking_prod.trades.settlements commandSubmit
2025-06-14 22:00:00 etl_service@bank.com banking_prod.trades.settlements commandSubmit
How does column-level lineage work?
Lineage is captured automatically at the column level as queries run. You can trace a gold table back to its bronze sources or forward to every dashboard that consumes it.
Lineage trace for gold.fraud_report.risk_score:
bronze.kafka_events.raw_payload
│
▼ (parsed in silver ETL)
silver.transactions.amount
│
▼ (aggregated in gold ETL)
gold.fraud_report.risk_score
│
▼ (consumed by)
Dashboard: "Daily Fraud Summary"
ML Model: "Credit Risk Scorer v3"
-- View lineage programmatically via the Unity Catalog REST API
-- or visually in the Databricks Catalog Explorer UI
-- Lineage shows which columns flow into which downstream objects
How does Unity Catalog work across clouds?
Unity Catalog operates across AWS, Azure, and GCP workspaces under one governance model. It also integrates with Delta Sharing to share data securely with external parties without copying it.
Unity Catalog (account-level metastore)
│
├── AWS Workspace (us-east-1)
│ └── banking_prod catalog
│
├── Azure Workspace (westeurope)
│ └── banking_eu catalog
│
└── GCP Workspace (us-central1)
└── banking_analytics catalog
One set of grants, one audit log, one lineage graph
What should a senior engineer know?
- The combination of centralized access control, automatic audit logs, column-level lineage, and masking is precisely what regulators demand. This is a major reason regulated industries adopt Unity Catalog.
- Define sensitive data permissions and masks once in Unity Catalog rather than scattering them across workspaces where they drift out of sync.
- Unity Catalog governs not just tables but also views, external locations, functions, ML models, and feature tables.
- Delta Sharing integration lets you share data with external partners using an open protocol without copying data.
-- Best practice: create a dedicated catalog for sensitive data
-- with strict access controls and column masks
CREATE CATALOG banking_restricted;
GRANT USE CATALOG ON CATALOG banking_restricted TO compliance_team;
GRANT SELECT ON CATALOG banking_restricted TO compliance_team;
-- No other groups get access unless explicitly granted
Visual Overview
Before Unity Catalog: ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Workspace A │ │ Workspace B │ │ Workspace C │ │ Own metastore │ │ Own metastore │ │ Own metastore │ │ Own ACLs │ │ Own ACLs │ │ Own ACLs │ │ No shared │ │ No shared │ │ No shared │ │ audit/lineage │ │ audit/lineage │ │ audit/lineage │ └──────────────┘ └──────────────┘ └──────────────┘ After Unity Catalog: ┌────────────────────────────────────────────────┐ │ Unity Catalog (account-level metastore) │ │ ┌─────────────────────────────────────┐ │ │ │ Centralized ACLs + Column Masks │ │ │ │ Centralized Audit Log │ │ │ │ Column-Level Lineage │ │ │ │ Searchable Data Discovery │ │ │ └─────────────────────────────────────┘ │ │ │ │ │ │ │ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ │ │ │ WS - A │ │ WS - B │ │ WS - C │ │ │ │ (AWS) │ │ (Azure) │ │ (GCP) │ │ │ └─────────┘ └─────────┘ └─────────┘ │ └────────────────────────────────────────────────┘
Key Takeaway
Unity Catalog is Databricks' centralized governance layer, providing one place to manage access control, auditing, data lineage, and discovery for all data and AI assets across multiple workspaces and clouds. Before it, governance was per-workspace, so permissions had to be duplicated everywhere, there was no unified audit, and lineage across workspaces was impossible.
Unity Catalog fixes this with an account-level metastore and a three-level namespace of catalog.schema.table. Access control uses SQL GRANT and REVOKE on users and groups synced from the identity provider, so removing someone revokes their access globally. Column masks and row filters handle sensitive data, and automatic column-level lineage traces data from source to consumer.
At the National Bank of India this solved two painful problems: a contractor's access lingered in a forgotten workspace after his engagement ended, and auditors wanted a single log of who had read customer account numbers. Unity Catalog centralized all of that: removing the contractor from the identity provider revoked his access globally, every read was captured in one audit log, and column masks let risk analysts see trades but only the last four digits of the account number.
The combination of centralized access control, audit, lineage, and masking is precisely what regulators demand, which is why regulated banks adopt Unity Catalog.
Remember one sentence: Unity Catalog provides one centralized place for access control, auditing, lineage, and discovery across all workspaces and clouds, replacing fragmented per-workspace governance.
---