Definition
- Encryption in transit (TLS) -- protects data on the wire between clients and brokers and between brokers, preventing eavesdropping and tampering.
- Authentication (mTLS or SASL) -- proves who each client is using mutual TLS certificates, SCRAM passwords, Kerberos, or OAuth tokens.
- Authorization (ACLs) -- controls which authenticated principals may produce, consume, or administer which topics and groups, enforcing least privilege.
- Encryption at rest -- protects data on disk using encrypted volumes or cloud-provider disk encryption, since Kafka itself does not encrypt log files.
- Network segmentation -- isolates the cluster in private subnets with tight firewall rules so even internal attackers cannot reach brokers without passing controls.
Think of National Bank of India's vault. TLS is the armored, sealed pipe that moves cash so no one can see or tamper with it in transit. Authentication is checking each person's verified ID badge at the door. ACLs are the rule that a teller may open drawer five but not the main vault. Encryption at rest is keeping the cash in locked, tamper-proof boxes even inside the vault. Network segmentation is building the vault in a restricted wing that only authorized corridors reach. And every entry is written in a logbook for auditors.
Understanding the Concepts
What are the security layers for a Kafka cluster?
Securing Kafka in a regulated environment means layering five controls that together enforce confidentiality, integrity, and least privilege. Each layer is necessary because it covers a different threat.
How do you configure encryption in transit with TLS?
You configure brokers to use TLS on their listeners so every client-to-broker and broker-to-broker connection is encrypted. Without TLS, transaction data crosses the network in plaintext, which no regulated bank can accept.
package com.bank.kafka;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class TlsProducerConfig {
public static Properties createSecureProps() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9093");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// TLS encryption in transit
props.put("security.protocol", "SSL");
props.put("ssl.truststore.location", "/etc/kafka/certs/truststore.jks");
props.put("ssl.truststore.password", "changeit");
System.out.println("Producer configured with TLS encryption");
return props;
}
public static void main(String[] args) {
Properties props = createSecureProps();
System.out.println("Security protocol: " + props.get("security.protocol"));
}
}
Expected output:
Producer configured with TLS encryption
Security protocol: SSL
How do you set up authentication with mTLS or SASL?
Mutual TLS means both sides present certificates, so the broker verifies the client's certificate and the client verifies the broker's. The alternative is SASL, which supports SCRAM for username-password, GSSAPI for Kerberos, and OAUTHBEARER for OAuth tokens.
package com.bank.kafka;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
public class MtlsProducerConfig {
public static Properties createMtlsProps() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9093");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
// Mutual TLS -- both sides present certificates
props.put("security.protocol", "SSL");
props.put("ssl.keystore.location", "/etc/kafka/certs/settlement-service.jks");
props.put("ssl.keystore.password", "changeit");
props.put("ssl.key.password", "changeit");
props.put("ssl.truststore.location", "/etc/kafka/certs/truststore.jks");
props.put("ssl.truststore.password", "changeit");
System.out.println("Producer configured with mutual TLS (mTLS)");
return props;
}
// SASL/SCRAM alternative
public static Properties createSaslProps() {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9093");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put("security.protocol", "SASL_SSL");
props.put("sasl.mechanism", "SCRAM-SHA-256");
props.put("sasl.jaas.config",
"org.apache.kafka.common.security.scram.ScramLoginModule required "
+ "username=\"settlement-service\" password=\"secret\";");
System.out.println("Producer configured with SASL/SCRAM over TLS");
return props;
}
public static void main(String[] args) {
createMtlsProps();
createSaslProps();
}
}
Expected output:
Producer configured with mutual TLS (mTLS)
Producer configured with SASL/SCRAM over TLS
How do you configure ACLs for least privilege?
Kafka's authorizer checks, for each operation, whether the authenticated principal is allowed. You grant specific rights and deny everything by default.
# Grant settlement service read access to transfers topic
kafka-acls.sh --add \
--allow-principal User:settlement-service \
--operation Read \
--topic transfers \
--bootstrap-server kafka-1:9093 --command-config admin.properties
# Grant settlement service write access to results topic
kafka-acls.sh --add \
--allow-principal User:settlement-service \
--operation Write \
--topic settlement-results \
--bootstrap-server kafka-1:9093 --command-config admin.properties
# Verify ACLs
kafka-acls.sh --list \
--principal User:settlement-service \
--bootstrap-server kafka-1:9093 --command-config admin.properties
Current ACLs for resource `Topic:transfers`:
(principal=User:settlement-service, host=*, operation=Read, permissionType=Allow)
Current ACLs for resource `Topic:settlement-results`:
(principal=User:settlement-service, host=*, operation=Write, permissionType=Allow)
How do you handle encryption at rest?
Kafka itself does not encrypt its log files, so you rely on the storage layer: encrypted volumes such as LUKS or cloud-provider disk encryption, and for tiered storage the object store's server-side encryption with managed keys.
# Linux: create an encrypted volume for Kafka data
cryptsetup luksFormat /dev/sdb
cryptsetup open /dev/sdb kafka-data
mkfs.ext4 /dev/mapper/kafka-data
mount /dev/mapper/kafka-data /var/kafka/data
# AWS: use encrypted EBS volumes
aws ec2 create-volume --encrypted --kms-key-id alias/kafka-key \
--availability-zone us-east-1a --size 500 --volume-type gp3
How do you implement network segmentation?
You place brokers in private subnets with no public addresses, allow only the required ports from only the required client subnets, and separate the control plane from general access.
# Security group rules for Kafka brokers
Inbound:
Port 9093 (TLS) from 10.0.1.0/24 (application subnet)
Port 9093 (TLS) from 10.0.2.0/24 (broker-to-broker)
Port 9094 (admin) from 10.0.3.0/24 (admin subnet only)
Outbound:
Port 9093 to 10.0.2.0/24 (broker-to-broker)
Port 443 to s3.amazonaws.com (tiered storage)
What are common mistakes in Kafka security?
package com.bank.kafka;
public class SecurityMistakes {
public static void main(String[] args) {
// WRONG: TLS without ACLs
System.out.println("WRONG: TLS without ACLs");
System.out.println(" Any authenticated client can read ANY topic");
System.out.println(" A compromised analytics service can read settlements");
// WRONG: ACLs without TLS
System.out.println("WRONG: ACLs without TLS");
System.out.println(" Credentials and data travel in plaintext");
System.out.println(" Anyone on the network can sniff account data");
// CORRECT: All layers together
System.out.println("CORRECT: TLS + mTLS + ACLs + encryption at rest");
System.out.println(" Confidentiality, integrity, and least privilege");
}
}
Expected output:
WRONG: TLS without ACLs
Any authenticated client can read ANY topic
A compromised analytics service can read settlements
WRONG: ACLs without TLS
Credentials and data travel in plaintext
Anyone on the network can sniff account data
CORRECT: TLS + mTLS + ACLs + encryption at rest
Confidentiality, integrity, and least privilege
What about audit logging?
You enable authorizer logging so every allowed and denied operation is recorded. You monitor for anomalies and keep certificates and credentials in a secrets manager with rotation.
# Broker configuration for audit logging
# server.properties
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
super.users=User:admin
log4j.logger.kafka.authorizer.logger=INFO, authorizerAppender
# Every ACL check is logged:
# [2024-01-15] ALLOWED: User:settlement-service Read Topic:transfers
# [2024-01-15] DENIED: User:analytics-service Write Topic:transfers
What should a senior engineer know?
These layers compose and you need all of them, because TLS without ACLs lets any authenticated client read everything, and ACLs without TLS send credentials and data in clear. Certificate expiry is a common outage cause, so automate certificate issuance and renewal. For the strongest security, use deny-by-default ACLs, keep unclean.leader.election.enable=false, and rotate encryption keys on a schedule.
Visual Overview
Kafka Security Layers ===================== ┌──────────────────────────────────────────────────┐ │ Network Segmentation │ │ (Private subnets, security groups, firewalls) │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ Encryption in Transit (TLS) │ │ │ │ (All client-broker and broker-broker) │ │ │ │ │ │ │ │ ┌────────────────────────────────────┐ │ │ │ │ │ Authentication (mTLS/SASL) │ │ │ │ │ │ (Prove identity of every client) │ │ │ │ │ │ │ │ │ │ │ │ ┌────────────────────────────┐ │ │ │ │ │ │ │ Authorization (ACLs) │ │ │ │ │ │ │ │ (Least privilege per │ │ │ │ │ │ │ │ principal per topic) │ │ │ │ │ │ │ └────────────────────────────┘ │ │ │ │ │ └────────────────────────────────────┘ │ │ │ └────────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ Encryption at Rest │ │ │ │ (Encrypted volumes + S3 SSE) │ │ │ └────────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ Audit Logging │ │ │ │ (Every ACL check recorded for compliance) │ │ │ └────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────┘
Key Takeaway
Securing Kafka in a regulated environment means layering five controls that together enforce confidentiality, integrity, and least privilege. TLS encrypts data in transit, mTLS or SASL authenticates every client, ACLs enforce least privilege per principal per topic, encryption at rest protects data on disk, and network segmentation isolates the cluster.
The critical insight is that these layers only work together. TLS without ACLs lets any authenticated client read everything. ACLs without TLS send credentials and data in the clear. A regulated bank needs all of them, plus audit logging for compliance.
In my National Bank of India project, we ran mutual TLS between services, deny-by-default ACLs scoped tightly per service, encrypted disks with a managed key service, brokers in a segmented private subnet, and a full audit trail. Certificate expiry automation prevented the outages that manual renewal causes.
Remember one sentence: secure Kafka with all five layers together -- TLS, authentication, ACLs, encryption at rest, and network segmentation -- because each layer covers a different threat and none is sufficient alone.
---