How does gRPC load balancing work, why does an L4 load balancer fail, and what are the fixes?

• L4 load balancer trap – A layer-four load balancer routes at the TCP connection level, so once one HTTP/2 connection lands on a backend, every multiplexed gRPC request rides that same connection to that one backend.
• L7 fix – A layer-seven load balancer like Envoy, Istio, or AWS ALB inspects individual HTTP/2 streams and distributes them across backends for true per-request balancing.
• Client-side fix – The gRPC client discovers all backends through DNS, a service registry, or xDS and distributes calls itself using a policy such as round_robin or pick_first.
• Headless service – A Kubernetes service with clusterIP: None that returns all pod IPs via DNS, enabling client-side round-robin.

Why does an L4 load balancer fail for gRPC?

The root cause is a mismatch between what a layer-four load balancer balances and how HTTP/2 works. A layer-four balancer makes its decision when a TCP connection is established and then forwards all bytes on that connection to the chosen backend. HTTP/1.1 opened a new connection per request or used a small pool, so a layer-four balancer spread load naturally. HTTP/2 deliberately uses one long-lived connection and multiplexes thousands of request streams over it, so the balancer picks a backend once and every request from that client goes to that single backend, leaving the rest idle.

package com.bank.config;

import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

public class L4TrapDemo {

    // This channel opens ONE connection.
    // An L4 LB routes that connection to ONE backend.
    // Every transfer call goes to that same backend.
    public ManagedChannel createChannelBehindL4() {
        return ManagedChannelBuilder
                .forTarget("l4-lb.bank.internal:9090")
                .usePlaintext()
                .build();
    }
}

What is the L7 load balancer fix?

A layer-seven load balancer understands HTTP/2 and inspects individual request streams. Envoy, Istio, Linkerd, and AWS ALB with gRPC support terminate the HTTP/2 connection, see the individual streams, and distribute each stream to a backend. This gives true per-request balancing and a central place for retries, outlier detection, and observability, at the cost of an extra network hop.

# Envoy L7 config snippet for gRPC
listeners:
  - name: grpc_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 9090
    filter_chains:
      - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              codec_type: AUTO
              stat_prefix: grpc
              route_config:
                virtual_hosts:
                  - name: transfer_service
                    domains: ["*"]
                    routes:
                      - match: { prefix: "/" }
                        route:
                          cluster: transfer_cluster

What is client-side load balancing?

The gRPC client itself discovers all backends and distributes calls. The client uses a name resolver to discover backends (DNS, Consul, Eureka, or xDS) and applies a load-balancing policy on the channel.

package com.bank.config;

import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

public class ClientSideBalancing {

    // round_robin opens a subchannel to every backend
    // and rotates requests across them
    public ManagedChannel createBalancedChannel() {
        return ManagedChannelBuilder
                .forTarget("dns:///transfer-service:9090")
                .defaultLoadBalancingPolicy("round_robin")
                .usePlaintext()
                .build();
    }
}

How is L7 different from client-side balancing? (Comparison)

| Feature | L7 Load Balancer | Client-Side Balancing |
|---|---|---|
| Where | Proxy (Envoy, Istio, ALB) | Inside the gRPC client |
| Balancing unit | Per HTTP/2 stream | Per RPC call |
| Extra hop | Yes (through proxy) | No |
| Observability | Centralized at proxy | Distributed per client |
| Retries | Proxy handles retries | Client must handle |
| Setup | Deploy and operate proxy | Configure channel policy |
| Best for | Service mesh, centralized control | Direct pod-to-pod calls |

Why does Kubernetes need a headless service for client-side balancing?

A normal ClusterIP service returns one virtual IP, which re-creates the single-connection pinning problem. A headless service (clusterIP: None) returns all pod IPs via DNS, so the round_robin policy sees every backend.

# Headless service: returns all pod IPs
apiVersion: v1
kind: Service
metadata:
  name: transfer-service
spec:
  clusterIP: None
  selector:
    app: transfer-service
  ports:
    - port: 9090
      targetPort: 9090

package com.bank.config;

import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

public class HeadlessServiceChannel {

    // dns:/// with headless service returns all pod IPs
    // round_robin spreads calls across them
    public ManagedChannel create() {
        return ManagedChannelBuilder
                .forTarget("dns:///transfer-service.bank-ns:9090")
                .defaultLoadBalancingPolicy("round_robin")
                .usePlaintext()
                .build();
    }
}

What are round_robin and pick_first policies?

The pick_first policy tries endpoints in order and sticks to the first that works. The round_robin policy opens a subchannel to every discovered backend and rotates requests across them.

package com.bank.config;

import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

public class PolicyExamples {

    // pick_first: good for single-backend or failover
    public ManagedChannel pickFirst() {
        return ManagedChannelBuilder
                .forTarget("dns:///account-service:9090")
                .defaultLoadBalancingPolicy("pick_first")
                .usePlaintext()
                .build();
    }

    // round_robin: good for spreading load evenly
    public ManagedChannel roundRobin() {
        return ManagedChannelBuilder
                .forTarget("dns:///account-service:9090")
                .defaultLoadBalancingPolicy("round_robin")
                .usePlaintext()
                .build();
    }
}

What about xDS and service meshes?

For large dynamic fleets, xDS-based balancing gives weighted and locality-aware routing that plain DNS cannot. Service meshes like Istio combine both: a sidecar proxy does L7 balancing transparently so the application code stays simple.

What are common mistakes?

The most common mistake is using a ClusterIP service with client-side round_robin, which returns one virtual IP and re-creates the pinning. Another is forgetting to re-resolve DNS periodically so new pods are not picked up after scaling.

// WRONG: ClusterIP returns one IP, defeats round_robin
// forTarget("dns:///transfer-service-clusterip:9090")
// .defaultLoadBalancingPolicy("round_robin")
// round_robin sees only one address, all calls go there

// CORRECT: headless service returns all pod IPs
// forTarget("dns:///transfer-service-headless:9090")
// .defaultLoadBalancingPolicy("round_robin")
// round_robin opens subchannel to each pod

What should a senior engineer know?

Round_robin only works if the resolver actually returns multiple addresses, which is why a normal ClusterIP service fails and a headless service is required. Re-resolve periodically so new pods are picked up after scaling. For large dynamic fleets, xDS-based balancing provides weighted and locality-aware routing. In practice, service meshes let a sidecar proxy do L7 balancing transparently so the application code stays simple.