Troubleshooting Common Issues with Containers on Amazon EKS

I. Introduction to Troubleshooting EKS

Amazon Elastic Kubernetes Service (EKS) has become a cornerstone for deploying and managing containerized applications at scale. However, the inherent complexity of distributed systems, combined with the layered abstractions of Kubernetes and AWS, means that issues are inevitable. Effective troubleshooting is not merely a reactive skill but a critical competency for ensuring application reliability and performance. The journey from encountering an error to implementing a resolution requires a systematic approach, blending deep technical knowledge with practical experience. This guide aims to equip you with that methodology, focusing on common pain points and proven solutions.

Containerized applications on EKS introduce a unique set of challenges. Unlike monolithic applications, issues can stem from the application code, the container runtime, the Kubernetes orchestration layer, the underlying AWS infrastructure, or the intricate interactions between all these components. A pod failing to start could be due to a misconfigured liveness probe, an exhausted resource quota, a missing IAM role, or a network security group blocking traffic. The first step in troubleshooting is recognizing this multi-faceted nature and avoiding assumptions about the root cause. Professionals who have undergone rigorous training, such as a Microsoft Azure AI course, often appreciate the value of a structured, data-driven diagnostic approach, which is equally applicable in the AWS ecosystem.

Fortunately, AWS and the Kubernetes community provide a rich toolkit for investigation. Core utilities like kubectl are your first line of defense for inspecting cluster state. AWS-specific tools such as the EKS console, CloudTrail for API auditing, and CloudWatch for metrics and logs offer deep visibility into the managed control plane and worker nodes. Furthermore, open-source observability stacks like Prometheus and Grafana can be integrated for custom monitoring. Understanding when and how to use each tool—whether it's kubectl describe for pod events, kubectl logs for container output, or CloudWatch Insights for log analytics—forms the foundation of efficient troubleshooting. For teams seeking formal accreditation in cloud management practices, engaging with legal CPD providers for certified professional development can ensure these skills are both current and recognized.

II. Common EKS Deployment Issues

The initial deployment phase is where many EKS challenges first manifest. These issues often prevent applications from running at all and require a methodical check of configuration and permissions.

A. Networking Problems (DNS resolution, Service Discovery)

Networking is the circulatory system of a Kubernetes cluster, and problems here can cause widespread failures. A frequent issue is DNS resolution failure within pods. Amazon EKS uses CoreDNS as the cluster DNS provider. If a pod cannot resolve the internal Kubernetes service name (e.g., my-service.default.svc.cluster.local), inter-service communication breaks. Common causes include CoreDNS pod failures, incorrect network policy rules blocking DNS traffic (port 53), or misconfigured VPC DNS settings. To diagnose, exec into a pod and try to resolve a service name: nslookup kubernetes.default. Check the CoreDNS pod logs and ensure the kube-dns service is active. Another prevalent problem is the inability of a eks container to reach resources outside the cluster, such as an RDS database. This is often due to the worker node's security group lacking the necessary egress rules or the subnet's route tables being incorrectly configured.

B. IAM Permissions and Access Denied Errors

AWS Identity and Access Management (IAM) integration is a powerful feature of EKS but a common source of "AccessDenied" errors. EKS uses IAM roles for service accounts (IRSA) to grant pods fine-grained AWS permissions, and IAM instance profiles for worker node permissions. A pod needing to write to an S3 bucket will fail if its associated service account IAM role lacks the s3:PutObject permission. Similarly, the worker node role needs permissions for ECR image pulls. Troubleshooting involves verifying the IAM role trust policy (ensuring it trusts the correct OIDC provider for your cluster), checking the role policies attached, and confirming the annotation in the Kubernetes service account YAML manifest. Using AWS CloudTrail is invaluable here, as it logs the specific API call that was denied and the assumed role that attempted it, pinpointing the exact policy gap.

C. Configuration Errors in Kubernetes Manifests

YAML manifests define the desired state of your applications, and even a minor indentation error or a wrong API version can cause deployment failures. Common errors include specifying a container image that doesn't exist, using a non-existent ConfigMap or Secret key as an environment variable, or defining incorrect resource requests/limits. The kubectl apply command may succeed (as the manifest is syntactically valid), but the resource will fail to create properly. Always use kubectl get events --all-namespaces --sort-by='.lastTimestamp' to see recent cluster-wide warnings and errors. Linting tools like kubeval or IDE plugins can catch many errors before deployment. For teams managing complex deployments, knowledge from a Microsoft Azure AI course on MLOps can translate well, emphasizing the importance of Infrastructure as Code (IaC) validation and CI/CD pipeline checks for Kubernetes configurations.

III. Container Runtime Issues

Once a pod is scheduled onto a node, the container runtime (typically Docker or containerd) takes over. Issues at this layer affect the lifecycle and performance of the application containers themselves.

A. Image Pull Errors

This is one of the most frequent startup failures. The error message ErrImagePull or ImagePullBackOff in kubectl describe pod indicates the kubelet cannot retrieve the container image. Causes are varied:

• Authentication Failures: Pulling from a private registry like Amazon ECR requires correct IAM permissions on the worker node or pod role. The error may be "no basic auth credentials".
• Network Issues: The node may not have outbound internet access to the registry, or corporate proxies may be blocking traffic.
• Image Tag Issues: The specified tag may not exist, or you may have attempted to use :latest which can lead to unpredictable behavior.
• Registry Quotas: Especially relevant in regions with high developer density, such as Hong Kong's AWS region (ap-east-1), where teams might hit storage limits if images are not cleaned up regularly.

B. Container Crashes and Restarts

A container that starts but then repeatedly crashes (CrashLoopBackOff status) points to an application-level problem. The primary source of truth is the container logs: kubectl logs <pod-name> --previous (to see logs from the last crashed instance). Common root causes include:

• Application bugs or unhandled exceptions.
• Incorrect startup command or arguments in the Dockerfile or pod spec.
• Missing runtime dependencies or environment variables inside the eks container.
• Failing liveness or readiness probes due to slow startup or incorrect probe configuration.

C. Resource Limits and OOMKilled Errors

Kubernetes allows you to set requests (guaranteed resources) and limits (maximum resources) for CPU and memory. When a container exceeds its memory limit, the Linux kernel's Out-Of-Memory (OOM) killer terminates the process, resulting in the OOMKilled status. This is a critical failure mode that can affect cluster stability. Troubleshooting involves:

1. Checking the pod's defined limits versus its actual usage using kubectl top pod or metrics from CloudWatch Container Insights.
2. Understanding that memory limits include the total RSS (Resident Set Size) of all processes in the container, including child processes and cached memory in some runtimes.
3. Recognizing that Java applications running inside an eks container require careful tuning of the JVM heap size (-Xmx) to stay well below the container memory limit, leaving room for off-heap memory.

IV. Monitoring and Logging

Proactive monitoring and centralized logging transform troubleshooting from a frantic search into a structured investigation. Without visibility, you are flying blind in a complex distributed system.

A. Using kubectl to Inspect Pods and Services

The kubectl command is the Swiss Army knife for cluster inspection. Beyond basic get and logs commands, mastering descriptive commands is essential:

• kubectl describe pod/<pod-name>: Provides a comprehensive overview of the pod's lifecycle, including events (scheduling, pulling images, starting containers), configuration details, and current status. The "Events" section at the bottom is often the fastest path to the root cause.
• kubectl describe service/<svc-name>: Shows the service selector, endpoints, and port mappings, helping verify if pods are correctly registered as endpoints.
• kubectl get pods -o wide: Shows which node a pod is running on, crucial for correlating pod issues with node-level problems.
• kubectl api-resources and kubectl explain: Help understand the object schema and available fields, reducing configuration errors.

B. Analyzing Container Logs with CloudWatch Logs

While kubectl logs is great for ad-hoc checks, production systems require centralized, durable, and searchable logging. The Amazon CloudWatch Logs agent (Fluent Bit) can be deployed as a DaemonSet on EKS to stream logs from all containers and worker nodes to CloudWatch. This enables powerful analysis:

• Log Groups and Streams: Each cluster/namespace/pod/container creates a logical log stream, organized into log groups.
• CloudWatch Logs Insights: Allows you to run SQL-like queries across your log data. For example, to find all errors from a specific application in the last hour:
  fields @timestamp, @message | filter @logStream like /my-app/ and @message like /ERROR/ | sort @timestamp desc | limit 50
• Metric Filters: You can create CloudWatch alarms based on specific log patterns (e.g., a sudden spike in "Connection refused" errors), enabling proactive alerting.

C. Monitoring Cluster Health with Prometheus and Grafana

For granular, custom metrics beyond what CloudWatch provides, the Prometheus-Grafana stack is the de facto standard. Prometheus scrapes metrics from various exporters (node-exporter for node metrics, kube-state-metrics for Kubernetes object state, cAdvisor for container metrics) and stores them as time-series data. Grafana provides rich dashboards for visualization. Key metrics to monitor include:

V. Advanced Troubleshooting Techniques

When standard methods fall short, advanced techniques provide deeper introspection into the system's state. These methods require more caution but can uncover issues invisible to surface-level checks.

A. Debugging with `kubectl debug`

Introduced in Kubernetes v1.18, the kubectl debug command is a game-changer for troubleshooting. It allows you to create an ephemeral debugging container that runs alongside the target pod in its namespaces (PID, network, IPC). This is perfect for situations where you need tools not present in the application's minimal container image. For example, to debug a pod named web-app that lacks network debugging tools: kubectl debug web-app -it --image=busybox --target=web-app This creates a temporary busybox container sharing the pod's network namespace. From here, you can run nslookup, telnet, tcpdump, or inspect network connections with netstat. The --share-processes flag allows you to see the processes running in the main container. This technique minimizes the need to modify the original pod spec or build custom debug images, adhering to security best practices while providing powerful introspection.

B. Analyzing Core Dumps

When an application crashes catastrophically (e.g., a segmentation fault in a native binary), a core dump containing the memory state at the time of the crash is invaluable. Capturing and analyzing core dumps in a containerized environment is challenging but possible. The process involves:

1. Enabling Core Dumps: Set the container's ulimit for core file size (e.g., ulimit -c unlimited in the Dockerfile or pod security context) and mount a volume where the core dump will be written.
2. Configuring the Kernel: On the worker node, set the kernel pattern (e.g., sysctl -w kernel.core_pattern=/var/coredumps/core.%e.%p) to direct dumps to a known location, which should be the mounted volume.
3. Analysis: Copy the core file to a development machine with debugging symbols (the exact binary and libraries used in the eks container). Use gdb or a similar debugger to load the core dump and analyze the stack trace.

C. Consulting AWS Documentation and Community Forums

No engineer is an island. The collective knowledge of the AWS and Kubernetes communities is an immense resource. When stuck, a structured approach to seeking help is crucial:

• AWS Official Documentation: Always the first stop. The EKS User Guide and Troubleshooting Guide are continuously updated with new issues and solutions.
• AWS Knowledge Center & Premium Support: For specific error codes or scenarios, the Knowledge Center articles provide concise solutions. For production-critical issues, AWS Premium Support can provide direct engineering assistance.
• Community Forums: Platforms like the AWS re:Post for EKS, Stack Overflow, and the Kubernetes Slack #aws-eks channel are where practitioners share real-world experiences. When posting, always provide anonymized but detailed information: EKS version, kubectl and aws-cli versions, relevant YAML snippets, and error logs from kubectl describe.