Back to Advanced Data Engineering
Q16Question 16 of 24

What is Amazon MSK vs Kinesis Data Streams, and when do you choose each

Key Concepts
  • MSK runs genuine open-source Apache Kafka with full protocol, API, and ecosystem compatibility.
  • Kinesis Data Streams is AWS-native, shard-based, with its own API and deep integration with Lambda, Firehose, and managed Flink.
  • Choose MSK for Kafka compatibility, portability across clouds, and the Kafka ecosystem (Connect, Streams).
  • Choose Kinesis for simplest AWS-native operations, on-demand auto-scaling, and tightest AWS integration.
  • Both are durable, scalable, and support multiple consumers and ordering within a partition/shard.
  • Both integrate with managed Flink, so the processing layer can be the same regardless of backbone choice.

Definition

MSK is managed open-source Kafka with full ecosystem portability, while Kinesis is simpler AWS-native streaming with deep integration; choose based on compatibility needs vs operational simplicity.
  • Amazon MSK – runs genuine Apache Kafka with topics, partitions, consumer-group model, and full protocol/API compatibility, giving portability across Kafka deployments.
  • Kinesis Data Streams – AWS-native streaming organized around shards, with its own API, on-demand scaling, and first-class integration with Lambda, Firehose, and managed Flink.
  • Decision criteria – compatibility and portability (favors MSK) vs operational simplicity and AWS integration (favors Kinesis).
  • Common ground – both are durable, scalable, multi-consumer, ordered within partition/shard, and both integrate with managed Flink.

The National Bank of India needs a high-speed conveyor to move transaction events between systems. MSK is the industry-standard Kafka conveyor that every engineer knows and that can be moved to any data center unchanged, with AWS handling maintenance. Kinesis is AWS's own built-in conveyor, simpler to switch on and wired directly into every other AWS machine in the building, but proprietary to this building. If the bank values the universal standard and portability, it picks MSK; if it values least setup and seamless AWS wiring, it picks Kinesis.

Understanding the Concepts

What is Amazon MSK?

MSK runs real Apache Kafka. You get Kafka topics, partitions, the consumer-group model, the full Kafka protocol, and the entire ecosystem.

PYTHON
# MSK: standard Kafka producer/consumer code works unchanged
from kafka import KafkaProducer, KafkaConsumer
import json

# Producer: send trade events to MSK
producer = KafkaProducer(
    bootstrap_servers="b-1.msk-cluster.kafka.ap-south-1.amazonaws.com:9092",
    value_serializer=lambda v: json.dumps(v).encode("utf-8")
)
producer.send("trades", {
    "trade_id": 12345,
    "account_id": 1001,
    "amount": 50000,
    "branch_code": "NBI-MUM-001"
})
producer.flush()

# Consumer: read trade events
consumer = KafkaConsumer(
    "trades",
    bootstrap_servers="b-1.msk-cluster.kafka.ap-south-1.amazonaws.com:9092",
    group_id="settlement-processor",
    value_deserializer=lambda v: json.loads(v.decode("utf-8"))
)
for message in consumer:
    process_trade(message.value)

This same code works on MSK, on-premises Kafka, or Confluent Cloud. That is the portability benefit.

What is Kinesis Data Streams?

Kinesis is AWS-native, organized around shards rather than partitions, with its own API.

PYTHON
# Kinesis: AWS SDK, shard-based
import boto3
import json

kinesis = boto3.client("kinesis", region_name="ap-south-1")

# Produce: put record into the stream
kinesis.put_record(
    StreamName="trades-stream",
    Data=json.dumps({
        "trade_id": 12345,
        "account_id": 1001,
        "amount": 50000,
        "branch_code": "NBI-MUM-001"
    }),
    PartitionKey="1001"  # determines which shard
)

# Consume: use Lambda event source mapping, KCL, or direct API
# Lambda automatically processes batches from each shard

Kinesis integrates natively with Lambda, Firehose, managed Flink, IAM, and CloudWatch.

How do they compare architecturally?

| Feature | MSK (Kafka) | Kinesis Data Streams |
|---|---|---|
| Protocol | Apache Kafka (open) | AWS API (proprietary) |
| Unit of parallelism | Partition | Shard |
| Consumer model | Consumer groups | Shared or enhanced fan-out |
| Retention | Configurable (days to infinite) | 1-365 days |
| Scaling | Manual partition count or Serverless | On-demand auto-scaling |
| Ecosystem | Kafka Connect, Kafka Streams, full | Lambda, Firehose, Analytics |
| Portability | Any Kafka deployment | AWS only |
| Operational complexity | More concepts (partitions, ISR, rebalancing) | Simpler |

When do you choose MSK?

Choose MSK when Kafka compatibility and portability matter.

PYTHON
# Reasons to choose MSK:
# 1. Existing Kafka investment (producers, consumers, connectors)
# 2. Portability across clouds or on-premises
# 3. Kafka Connect ecosystem (hundreds of connectors)
# 4. Kafka Streams for stream processing
# 5. Organization standardizing on Kafka

# Example: existing Kafka Connect connectors work unchanged
# JDBC source connector → MSK → Flink → Delta Lake

When do you choose Kinesis?

Choose Kinesis when AWS-native simplicity and integration are the priority.

PYTHON
# Reasons to choose Kinesis:
# 1. Greenfield AWS project with no Kafka investment
# 2. On-demand capacity (no partition management)
# 3. Deepest AWS integration (Lambda, Firehose, CloudWatch)
# 4. Simplest operations (fully serverless)
# 5. Team is AWS-centric

# Example: stream → Lambda → DynamoDB, all AWS-native
# Zero Kafka knowledge required

How do cost models differ?

Model your throughput carefully because billing differs significantly.

PYTHON
# MSK cost model:
# - Broker instances (or serverless usage units)
# - Storage per broker
# - Data transfer
# Better for: high, steady throughput

# Kinesis cost model:
# - Per shard-hour (provisioned) or per-GB throughput (on-demand)
# - Per PUT payload unit
# - Enhanced fan-out per consumer-shard-hour
# Better for: variable, bursty throughput with on-demand mode

# Always model with real numbers before choosing

Can both use the same processing layer?

Yes. Both integrate with managed Flink, so your processing logic stays the same.

PYTHON
# Flink reads from MSK:
# env.addSource(FlinkKafkaConsumer("trades", ...))

# Flink reads from Kinesis:
# env.addSource(FlinkKinesisConsumer("trades-stream", ...))

# The processing logic is identical regardless of source
# This is a useful decoupling point

What should a senior engineer know?

  • The decision is rarely about raw capability; both are durable, scalable, and ordered within a partition/shard.
  • MSK trades more operational concepts for Kafka's ecosystem and portability.
  • Kinesis trades API-level lock-in for simpler operations and tighter AWS wiring.
  • Both integrate with managed Flink, so standardize on Flink and decouple from the backbone.
  • For organizations standardizing on Kafka across clouds, MSK preserves the standard.
  • For all-in AWS shops, Kinesis is often the simpler choice.
PYTHON
# Senior pattern: abstract the streaming source behind a configuration
if config["streaming_backend"] == "msk":
    source = FlinkKafkaConsumer("trades", kafka_props)
else:
    source = FlinkKinesisConsumer("trades-stream", kinesis_props)

# Processing logic is backend-agnostic

What are common interview follow-up questions?

  • Can you migrate from Kinesis to MSK? Yes, but it requires rewriting producers and consumers to use the Kafka protocol.
  • Does MSK support serverless? Yes, MSK Serverless removes capacity planning.
  • Can Kinesis use Kafka Connect? No, Kafka Connect requires the Kafka protocol. You would use Firehose or Lambda instead.

Visual Overview

DIAGRAM
  MSK vs Kinesis Decision Matrix
  ───────────────────────────────

  ┌─────────────────────────┐    ┌─────────────────────────┐
  │     Amazon MSK          │    │  Kinesis Data Streams    │
  │  (Managed Kafka)        │    │  (AWS-native)            │
  ├─────────────────────────┤    ├─────────────────────────┤
  │ + Kafka protocol        │    │ + Simplest operations    │
  │ + Full ecosystem        │    │ + On-demand scaling      │
  │ + Portable across clouds│    │ + Lambda integration     │
  │ + Consumer groups       │    │ + Firehose delivery      │
  │ - More operational      │    │ - AWS API lock-in        │
  │   concepts              │    │ - Narrower ecosystem     │
  └────────────┬────────────┘    └────────────┬────────────┘
               │                              │
               └──────────┬───────────────────┘
                          │
               ┌──────────▼──────────┐
               │  Managed Flink      │
               │  (same processing   │
               │   logic either way) │
               └─────────────────────┘

Key Takeaway

Amazon MSK and Kinesis Data Streams are both managed streaming backbones on AWS, but they differ in lineage and philosophy. MSK runs genuine open-source Kafka with full protocol compatibility, the consumer-group model, and the entire Kafka ecosystem including Connect and Streams, with portability to move workloads across clouds. Kinesis is AWS-native, built around shards, with its own API, on-demand auto-scaling, and the deepest integration with Lambda, Firehose, and managed Flink.

The choice is about compatibility, portability, and operational preference, not raw capability. Both are durable, scalable, and ordered within a partition or shard. Choose MSK when the organization already uses Kafka, needs portability, or relies on the Kafka ecosystem. Choose Kinesis when AWS-native simplicity, on-demand scaling, and tight AWS integration are the priorities. The cost models differ, so always model with real throughput numbers.

At the National Bank of India, this was a real split decision. The core transaction backbone already ran on-premises Kafka with many Connect connectors, so MSK preserved full compatibility without rewriting producers or consumers. Separately, a greenfield analytics project valued minimal operations and tight Lambda integration, so Kinesis with on-demand capacity was the right fit. Both fed the same managed Flink processing layer, so standardizing on Flink decoupled the processing from the backbone choice.

Remember one sentence: MSK is portable open-source Kafka with the full ecosystem, Kinesis is simpler AWS-native streaming with deep integration, and the decision is about compatibility and portability versus operational simplicity.

---

Share
Use ← → arrow keys to navigate