What are the advantages of Kinesis Data Analytics (managed Flink) over self-managed Flink, and the trade-offs

Self-managed Flink responsibilities:
  - Provision JobManagers and TaskManagers
  - Configure memory, slots, state backends
  - Set up high availability (ZooKeeper/K8s)
  - Handle version upgrades and security patches
  - Wire up monitoring (Prometheus, Grafana)
  - Respond to failures at 3 a.m.

Managed Flink responsibilities:
  - Upload application jar
  - Configure KPU count and auto-scaling
  - Set checkpoint interval
  - AWS handles everything else

package com.bank.fraud;

import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

// Same Flink code runs on managed or self-managed
public class FraudPipeline {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();

        env.addSource(new KinesisSource<>("nbi-transactions"))
            .keyBy(Transaction::getAccountId)
            .process(new FraudRuleEngine())
            .addSink(new KinesisSink<>("nbi-alerts"));

        env.execute("NBI Fraud Pipeline");
        // Runs identically on managed or self-managed Flink
    }
}

Advantage 1: No infrastructure management
  - No servers to provision, patch, or monitor
  - Automatic high availability and failure recovery
  - No ZooKeeper or Kubernetes to maintain

Advantage 2: Auto-scaling
  - Kinesis Processing Units (KPUs) scale with load
  - Handles salary-day and festival traffic spikes
  - No manual cluster resizing

Advantage 3: AWS integration
  - S3: checkpoint and savepoint storage
  - Glue: schema registry and data catalog
  - Lambda: serverless actions on output
  - Kinesis/MSK: first-class sources and sinks
  - IAM: fine-grained access control
  - CloudWatch: metrics and logs, no extra wiring

Advantage 4: Pay-per-use
  - Billed per KPU-hour consumed
  - Cost-effective for variable/spiky workloads
  - No paying for an always-on peak-sized cluster

package com.bank.config;

public class ManagedFlinkConfig {
    // AWS CloudFormation for managed Flink application
    // AWSTemplateFormatVersion: '2010-09-09'
    // Resources:
    //   NbiFraudApp:
    //     Type: AWS::KinesisAnalyticsV2::Application
    //     Properties:
    //       RuntimeEnvironment: FLINK-1_18
    //       ApplicationConfiguration:
    //         FlinkApplicationConfiguration:
    //           CheckpointConfiguration:
    //             ConfigurationType: CUSTOM
    //             CheckpointingEnabled: true
    //             CheckpointInterval: 30000
    //           ParallelismConfiguration:
    //             ConfigurationType: CUSTOM
    //             Parallelism: 4
    //             AutoScalingEnabled: true
}

Trade-off 1: Reduced customization
  - Curated set of configuration options
  - Cannot run arbitrary Flink versions immediately
  - Cannot install custom native libraries or sidecars
  - May not support niche connectors

Trade-off 2: Operational vendor lock-in
  - Auto-scaling, checkpoint management, deployment are AWS-specific
  - Moving off managed service means rebuilding operational layer
  - Application code (Flink) stays portable
  - Lock-in is at operational layer, not business logic layer

Trade-off 3: Cost cross-over
  - For very large, steady, always-on workloads:
    self-managed on reserved instances may be cheaper
  - Model your specific usage pattern
  - Variable/spiky workloads favor managed auto-scaling

// Code is identical on both:
env.addSource(kafkaOrKinesisSource)
    .keyBy(Transaction::getAccountId)
    .process(new FraudRuleEngine())
    .addSink(alertSink);
// Only the infrastructure wrapper changes

Decision guide:

  Small team, no platform engineers?    --> Managed Flink
  Tight AWS integration needed?         --> Managed Flink
  Variable/spiky workload?              --> Managed Flink (auto-scale)
  Need bleeding-edge Flink version?     --> Self-managed
  Need custom native plugins?           --> Self-managed
  Large, steady, always-on workload?    --> Model cost: may be self-managed
  Have a capable platform team?         --> Self-managed is an option
  Multi-cloud or cloud-agnostic goal?   --> Self-managed on K8s

package com.bank.streaming;

// Portable Flink application code
public class PortableFraudPipeline {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env =
            StreamExecutionEnvironment.getExecutionEnvironment();

        // Source and sink are configured externally
        // On AWS: Kinesis source/sink
        // On self-managed: Kafka source/sink
        // Business logic is IDENTICAL:
        env.addSource(createSource(args))
            .keyBy(Transaction::getAccountId)
            .process(new FraudRuleEngine()) // portable logic
            .addSink(createSink(args));

        env.execute("NBI Fraud Pipeline");
    }

    // Factory methods isolate AWS-specific connectors
    private static SourceFunction<Transaction> createSource(
            String[] args) {
        if (isAws(args)) return new KinesisSource<>("nbi-txn");
        return new KafkaSource<>("nbi-txn");
    }

    private static SinkFunction<Alert> createSink(String[] args) {
        if (isAws(args)) return new KinesisSink<>("nbi-alerts");
        return new KafkaSink<>("nbi-alerts");
    }
}

# WRONG: self-managed without platform team
# Result: team spends 60% of time on cluster ops
# Business logic delayed for months

# WRONG: managed Flink for huge steady workload
# 100 KPUs running 24/7 at fixed load
# Result: pay-per-use cost exceeds reserved instances

# CORRECT: managed for spiky workload, small team
# Auto-scales from 4 KPUs (normal) to 20 KPUs (peak)
# Team focuses on fraud logic, not cluster management

# CORRECT: self-managed for large steady workload + platform team
# Reserved instances, full control, lower cost at scale

package com.bank.config;

public class SeniorManagedFlinkChecklist {
    // Senior engineer's managed Flink decision checklist:
    //
    // 1. Default: managed Flink (unless proven otherwise)
    // 2. You still own: watermarks, state TTL, parallelism,
    //    checkpoint interval, exactly-once semantics
    // 3. AWS handles: cluster provisioning, HA, patching,
    //    auto-scaling, monitoring infrastructure
    // 4. Mitigate lock-in: keep business logic as clean Flink,
    //    isolate AWS connectors in factory methods
    // 5. Model cost: compare KPU-hours vs reserved instances
    //    for your specific load profile
    // 6. Choose self-managed only when:
    //    - Need specific Flink version or custom plugin
    //    - Large steady workload where self-managed is cheaper
    //    - Have a capable platform team
    //    - Multi-cloud requirement
}