Definition
Kubernetes Service discovery gives Pods a stable virtual identity in front of ephemeral Pod IPs.
- ClusterIP – exposes Pods on an internal virtual IP reachable only inside the cluster.
- NodePort – builds on ClusterIP by also opening a port (30000-32767) on every node for external access.
- LoadBalancer – builds on NodePort by provisioning an external cloud load balancer with a public IP.
- CoreDNS – resolves Service names like
transfer-service.prod.svc.cluster.localto the Service's ClusterIP. - Ingress – routes external HTTP traffic by hostname and path to internal Services, which forward to healthy Pods.
At the National Bank of India, individual tellers (Pods) come and go and each gets a new desk number (Pod IP) every morning. A Service is the permanent phone extension for the Transfers department. You dial the extension, and the switchboard (kube-proxy) connects you to whichever teller is on duty. CoreDNS is the phone directory that turns the department name into that extension number. The Ingress is the reception desk that reads the visitor's form and walks them to the right department.
Understanding the Concepts
What is a Kubernetes Service?
A Service is an abstraction that gives a stable virtual IP and DNS name to a set of Pods. Pod IPs are ephemeral and change on every restart, so nothing should ever talk to a raw Pod IP. A Service selects Pods by labels, and the EndpointSlice controller maintains a live list of matching ready Pod IPs.
# service.yaml - ClusterIP Service for the transfer service
apiVersion: v1
kind: Service
metadata:
name: transfer-service
namespace: prod
spec:
type: ClusterIP
selector:
app: transfer-service
ports:
- port: 80
targetPort: 8080
protocol: TCP
kubectl get svc transfer-service -n prod
# NAME TYPE CLUSTER-IP PORT(S)
# transfer-service ClusterIP 10.96.45.12 80/TCP
kubectl get endpointslices -l kubernetes.io/service-name=transfer-service -n prod
# NAME ENDPOINTS
# transfer-service-abc12 10.0.1.5:8080,10.0.1.6:8080,10.0.1.7:8080
How do ClusterIP, NodePort, and LoadBalancer differ?
These three Service types build on each other in layers.
| Type | Accessibility | How it works |
|---|---|---|
| ClusterIP | Internal only | Virtual IP from the Service CIDR, reachable inside the cluster |
| NodePort | Internal + any node IP | ClusterIP + a port (30000-32767) on every node |
| LoadBalancer | Internal + external | NodePort + a cloud load balancer with a public IP |
# NodePort Service - accessible from outside on node IP:30080
apiVersion: v1
kind: Service
metadata:
name: transfer-service-nodeport
spec:
type: NodePort
selector:
app: transfer-service
ports:
- port: 80
targetPort: 8080
nodePort: 30080
# LoadBalancer Service - provisions an AWS ELB
apiVersion: v1
kind: Service
metadata:
name: transfer-service-lb
spec:
type: LoadBalancer
selector:
app: transfer-service
ports:
- port: 80
targetPort: 8080
How does CoreDNS resolve Service names?
CoreDNS runs as a Deployment in kube-system and is itself exposed by a Service, usually named kube-dns. Every Pod's /etc/resolv.conf points at the CoreDNS Service IP and includes search domains.
When code calls http://transfer-service:80 inside namespace prod, the resolver appends the search domain to form transfer-service.prod.svc.cluster.local, which CoreDNS answers with the Service ClusterIP.
# Inside a Pod, check the resolver configuration
cat /etc/resolv.conf
# nameserver 10.96.0.10
# search prod.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5
# Resolve the service name
nslookup transfer-service
# Server: 10.96.0.10
# Name: transfer-service.prod.svc.cluster.local
# Address: 10.96.45.12
How does the Ingress route external traffic?
An Ingress resource defines host and path rules. The Ingress controller (NGINX, ALB, or Traefik) watches those rules and configures itself to route external HTTP to the right Service.
# ingress.yaml - route external traffic to internal Services
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: banking-ingress
namespace: prod
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
rules:
- host: api.nbi.com
http:
paths:
- path: /transfers
pathType: Prefix
backend:
service:
name: transfer-service
port:
number: 80
- path: /loans
pathType: Prefix
backend:
service:
name: loan-service
port:
number: 80
kubectl get ingress -n prod
# NAME CLASS HOSTS ADDRESS PORTS
# banking-ingress nginx api.nbi.com 203.0.113.50 80
How does kube-proxy load-balance traffic?
When a Pod sends traffic to a Service ClusterIP, kube-proxy, via iptables, IPVS, or eBPF with Cilium, rewrites the destination to one of the backing Pod IPs from the EndpointSlice. This is why a Service survives Pod churn. The ClusterIP never changes even though Pod IPs do.
# Check kube-proxy mode
kubectl get configmap kube-proxy -n kube-system -o jsonpath='{.data.config\.conf}' | grep mode
# mode: "ipvs"
# View the IPVS rules for a Service
kubectl exec -n kube-system kube-proxy-abc12 -- ipvsadm -ln | grep 10.96.45.12
# TCP 10.96.45.12:80 rr
# -> 10.0.1.5:8080 Masq 1 0
# -> 10.0.1.6:8080 Masq 1 0
How do you debug DNS resolution failures?
When a Pod cannot resolve a Service name, work outward layer by layer.
# Step 1: Confirm the name and namespace
# A Service in another namespace needs its fully qualified name
kubectl get svc -A | grep transfer
# Step 2: Check resolv.conf inside the Pod
kubectl exec -it transfer-client-pod -- cat /etc/resolv.conf
# nameserver 10.96.0.10
# search prod.svc.cluster.local ...
# Step 3: Test DNS resolution
kubectl exec -it transfer-client-pod -- nslookup transfer-service
# If this fails, test a known-good name
kubectl exec -it transfer-client-pod -- nslookup kubernetes.default
# Step 4: Check CoreDNS health
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
# Step 5: Check Service endpoints
kubectl get endpointslices -l kubernetes.io/service-name=transfer-service -n prod
# Empty endpoints = selector does not match any ready Pod
# Step 6: Check for NetworkPolicy blocking DNS (port 53)
kubectl get networkpolicy -n prod
# A default-deny egress policy without a DNS allow rule blocks resolution
What are common mistakes?
The most common DNS debugging mistake is not checking the namespace. A Service in namespace payments is not reachable by short name from namespace prod. You must use the fully qualified name.
# WRONG - calling a Service in another namespace by short name
curl http://transfer-service:80
# Result: nslookup failed, name not found
# CORRECT - use the fully qualified name
curl http://transfer-service.payments.svc.cluster.local:80
Another common mistake is a Service selector that does not match the Pod labels, which produces an empty endpoint list.
# WRONG - Service selector does not match Pod labels
# Service selector: app: transfer
# Pod labels: app: transfer-service
# Result: zero endpoints, no traffic reaches the Pods
# CORRECT - labels must match exactly
# Service selector: app: transfer-service
# Pod labels: app: transfer-service
What should a senior engineer know?
In a locked-down banking cluster, a default-deny NetworkPolicy can block egress to CoreDNS on port 53, which breaks all DNS resolution. You must explicitly allow egress to DNS.
# Allow egress to CoreDNS for DNS resolution
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: prod
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Verify DNS works after applying the policy
kubectl exec -it transfer-client-pod -- nslookup transfer-service
# Should now resolve successfully
Visual Overview
External Request Path
======================
Client (Priya's mobile app)
│
│ HTTPS: api.nbi.com/transfers
▼
┌──────────────────────────┐
│ Cloud Load Balancer │
│ (external IP) │
└────────────┬─────────────┘
│
▼
┌──────────────────────────┐
│ Ingress Controller │
│ (NGINX / ALB) │
│ host: api.nbi.com │
│ path: /transfers → │
└────────────┬─────────────┘
│
▼
┌──────────────────────────┐
│ Service: transfer-svc │
│ ClusterIP: 10.96.45.12 │
│ kube-proxy rewrites to: │
└────────────┬─────────────┘
│
┌───────┼───────┐
▼ ▼ ▼
[Pod-A] [Pod-B] [Pod-C]
10.0.1.5 10.0.1.6 10.0.1.7
Internal DNS Resolution
========================
Pod calls: http://transfer-service:80
│
│ resolv.conf → nameserver 10.96.0.10
▼
┌──────────────────────────┐
│ CoreDNS (kube-system) │
│ transfer-service.prod │
│ .svc.cluster.local │
│ → 10.96.45.12 │
└──────────────────────────┘
Key Takeaway
Kubernetes Service discovery exists because Pod IPs are ephemeral. A Service gives a stable virtual identity in front of those changing Pods. ClusterIP is internal-only, NodePort adds a port on every node, and LoadBalancer provisions an external cloud load balancer. Behind every Service, an EndpointSlice tracks the live list of ready Pod IPs, and kube-proxy rewrites traffic to one of those Pods.
CoreDNS handles name resolution. It runs in the cluster and every Pod's resolver points at it. A simple call to a name like transfer-service gets expanded to its fully qualified cluster name and answered with the Service ClusterIP. For external HTTP, an Ingress routes by host and path to the right Service, giving the full path of Ingress to Service to Pod.
When a Pod cannot resolve a Service name, debug outward in layers: confirm the name and namespace, check resolv.conf, test DNS lookup, verify CoreDNS health, check endpoints, and look for a NetworkPolicy blocking port 53. That last cause is exactly what bit the team at the National Bank of India when a new NetworkPolicy blocked egress to CoreDNS and every transfer failed with an unknown-host error.
Remember one sentence: Services give Pods a stable IP, CoreDNS maps names to that IP, and when DNS fails debug from the inside out: resolv.conf, CoreDNS, endpoints, then NetworkPolicy.
---