Back to Advanced Data Engineering
Q9Question 9 of 24

How does Kinesis Data Analytics integrate with AWS Lambda

Key Concepts
  • Lambda integrates at input (pre-processing/enrichment) and at output (serverless per-result actions) of the analytics pipeline.
  • On output, the Flink application sinks to a Kinesis stream and a Lambda event source mapping invokes functions in batches.
  • Batched invocation with tunable batch size and batching window keeps Lambda cost-efficient and able to keep up.
  • Delivery is at-least-once, so the function must be idempotent and a dead-letter destination must capture persistent failures.
  • Keep heavy stateful stream processing in Flink and use Lambda only for stateless per-result actions and integrations.
  • The combination creates a fully serverless, event-driven pipeline with no servers to manage.

Definition

Kinesis Data Analytics integrates with Lambda at input for pre-processing and at output for serverless custom actions, creating a fully serverless event-driven pipeline.
  • Input integration – a Lambda function pre-processes or enriches raw records before the analytics application sees them (e.g., format conversion, field derivation).
  • Output integration – analytics results trigger Lambda via an event source mapping from the output Kinesis stream, processing flags in batches for enrichment, routing, or alerting.
  • At-least-once delivery – batches may be retried, so the function must be idempotent with a dead-letter destination for persistent failures.
  • Design discipline – heavy stateful processing belongs in Flink; Lambda handles stateless per-result actions.

The National Bank of India's analytics engine produces a stream of flagged suspicious transactions. A Lambda function is like an on-call specialist who springs into action only when a flag arrives: for each flagged transaction, the specialist looks up customer details, decides whether to freeze the card, sends an SMS, and files a case, then goes back to sleep. The bank pays only for the seconds worked, and no room is staffed waiting.

Understanding the Concepts

What are the two integration points?

Lambda participates in the Kinesis Data Analytics pipeline at both ends: input pre-processing and output actions.

PYTHON
# Conceptual architecture (pseudo-code, AWS-native)

# INPUT: Lambda pre-processes raw records
# Raw stream → Lambda (decode, normalize, enrich) → Analytics app

# OUTPUT: Analytics results trigger Lambda for actions
# Analytics app → Output stream → Lambda (enrich, decide, act)

How does input pre-processing work?

Before the analytics application processes records, a Lambda function transforms, decodes, or enriches them.

PYTHON
# AWS Lambda function for input pre-processing (Python)
import json
import base64

def lambda_handler(event, context):
    output = []
    for record in event["records"]:
        # Decode the raw record
        payload = base64.b64decode(record["data"]).decode("utf-8")
        data = json.loads(payload)

        # Normalize and enrich
        data["amount_inr"] = data["amount"] * data.get("exchange_rate", 1.0)
        data["branch_region"] = lookup_region(data["branch_code"])

        # Return the enriched record
        output.append({
            "recordId": record["recordId"],
            "result": "Ok",
            "data": base64.b64encode(
                json.dumps(data).encode("utf-8")
            ).decode("utf-8")
        })
    return {"records": output}

The analytics engine then operates on the Lambda's cleaned output.

How does output integration work?

The Flink application sinks results to a Kinesis Data Stream, and a Lambda event source mapping processes them in batches.

PYTHON
# AWS CDK / CloudFormation pseudo-code
# Create event source mapping from output stream to Lambda

# Lambda function for output actions
import json
import boto3

dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("FraudFlags")
sns = boto3.client("sns")

def lambda_handler(event, context):
    for record in event["Records"]:
        payload = json.loads(
            base64.b64decode(record["kinesis"]["data"]).decode("utf-8")
        )

        flag_id = payload["flag_id"]
        account_id = payload["account_id"]
        amount = payload["amount"]

        # Idempotency check: skip if already processed
        existing = table.get_item(Key={"flag_id": flag_id})
        if "Item" in existing:
            continue  # already handled

        # Record the flag
        table.put_item(Item={
            "flag_id": flag_id,
            "account_id": account_id,
            "amount": amount,
            "status": "FLAGGED"
        })

        # Send alert
        if amount > 500000:
            sns.publish(
                TopicArn="arn:aws:sns:ap-south-1:123456:FraudAlerts",
                Message=f"High-value fraud flag: {account_id}, INR {amount}"
            )

    return {"statusCode": 200}

Why must the function be idempotent?

Kinesis delivery is at-least-once. A batch may be retried due to timeout, throttling, or transient errors, so the same record can arrive more than once.

PYTHON
# Idempotency pattern: use the flag_id as a deduplication key
existing = table.get_item(Key={"flag_id": flag_id})
if "Item" in existing:
    return  # skip duplicate

# Without this check:
# - A card might be frozen twice
# - An SMS might be sent twice
# - A case might be filed twice

How do you handle persistent failures?

Configure retries and a dead-letter destination so records that repeatedly fail are captured instead of lost.

PYTHON
# Event source mapping configuration (AWS CDK pseudo-code)
# MaximumRetryAttempts: 3
# BisectBatchOnFunctionError: true  (split batch to isolate bad record)
# DestinationConfig:
#   OnFailure:
#     Destination: arn:aws:sqs:...:FraudFlagsDLQ

# The DLQ captures records that failed all retries
# Operations team monitors the DLQ and investigates

How do you tune batch processing?

Tune the batch size and batching window so Lambda is cost-efficient and keeps up with the stream.

PYTHON
# Event source mapping settings:
# BatchSize: 100              (records per invocation)
# MaximumBatchingWindowInSeconds: 5  (wait up to 5s to fill a batch)
# ParallelizationFactor: 2    (concurrent Lambda invocations per shard)

# Monitor the IteratorAge metric in CloudWatch
# If IteratorAge is growing, Lambda is falling behind
# Fix: increase ParallelizationFactor or optimize the function

What is the design discipline?

Keep heavy processing in Flink and stateless actions in Lambda.

PYTHON
# RIGHT: Flink does stateful windowing, pattern detection, joins
# Lambda does per-result enrichment, alerting, card freeze

# WRONG: pushing complex stream processing into Lambda
# Lambda has no native state management, windowing, or join support
# It would require external state (DynamoDB) for every operation

| Responsibility | Flink | Lambda |
|---|---|---|
| Windowed aggregation | Yes | No |
| Pattern detection | Yes | No |
| Stateful joins | Yes | No |
| Per-result enrichment | No | Yes |
| Alerting/notification | No | Yes |
| External API calls | No | Yes |

What should a senior engineer know?

  • Mind Lambda's timeout (15 min max), memory, and concurrency limits.
  • Make external calls fast or asynchronous so the batch completes within the timeout.
  • Grant the function least-privilege IAM access to only the services it touches.
  • Watch the IteratorAge metric to ensure Lambda keeps pace with the stream.
  • The fully serverless combination (Kinesis + managed Flink + Lambda) eliminates all server management.
PYTHON
# IAM policy: least privilege
# Lambda can only:
# - Read from the Kinesis output stream
# - Write to DynamoDB FraudFlags table
# - Publish to SNS FraudAlerts topic
# - Write to SQS DLQ

What are common interview follow-up questions?

  • Can Lambda be the input AND output? Yes, Lambda can pre-process input records and act on output results.
  • What if Lambda takes too long? The batch times out and is retried. Optimize the function or increase the timeout.
  • Does this work with managed Flink? Yes, the output stream from managed Flink triggers Lambda via event source mapping.

Visual Overview

DIAGRAM
  Kinesis Data Analytics + Lambda Pipeline
  ──────────────────────────────────────────

  Input Stream                 Flink Application           Output Stream
  ┌──────────────┐            ┌──────────────┐            ┌──────────────┐
  │ Raw events   │──Lambda──▶│ Stateful     │──────────▶│ Fraud flags  │
  │ (pre-process)│  enrich   │ windowing,   │            │              │
  └──────────────┘            │ pattern      │            └──────┬───────┘
                              │ detection    │                   │
                              └──────────────┘          Event Source Mapping
                                                                │
                              ┌──────────────────────────────────▼──┐
                              │  Lambda (per-result actions)         │
                              │  1. Idempotency check (DynamoDB)    │
                              │  2. Enrich with customer details    │
                              │  3. Decide: freeze card?            │
                              │  4. Send SMS alert (SNS)            │
                              │  5. File case                       │
                              └───────────┬─────────────────────────┘
                                          │ failures
                              ┌───────────▼─────────────────────────┐
                              │  Dead-Letter Queue (SQS)            │
                              │  (captures persistent failures)     │
                              └─────────────────────────────────────┘

Key Takeaway

Kinesis Data Analytics integrates with Lambda at two points. On input, Lambda pre-processes or enriches raw records before the Flink application sees them, normalizing formats or adding derived fields. On output, the Flink application sinks results to a Kinesis stream and a Lambda event source mapping invokes functions in batches for per-result actions: enriching against external systems, deciding on actions, sending alerts, freezing cards, or routing to other services. The result is a fully serverless, event-driven pipeline.

The operational details matter. Output invocation is batched, so you tune the batch size and batching window to keep Lambda cost-efficient. Delivery is at-least-once, meaning batches can be retried and contain duplicates, so the function must be idempotent. Configure a maximum retry count and a dead-letter queue so persistently failing records are captured rather than lost. Keep heavy stateful processing in Flink and use Lambda only for stateless per-result integrations.

At the National Bank of India, the managed Flink fraud application emitted flagged transactions, and each flag needed custom actions: enriching with customer details, deciding whether to freeze the card, and sending alerts. Lambda handled these stateless per-result actions in batches. The function was idempotent by recording each flag id in DynamoDB, and a dead-letter SQS queue captured persistent failures for the operations team. All stateful windowing and pattern detection stayed in Flink.

Remember one sentence: Kinesis Data Analytics triggers Lambda at input for enrichment and at output for serverless per-result actions, with batched at-least-once delivery requiring idempotency and dead-letter handling.

---

Share
Use ← → arrow keys to navigate