Docker & Kubernetes Security (Part 2): Taking the Scanner to AWS

In this series, I’m learning Docker and Kubernetes security by building a phishing URL scanner and applying security practices along the way. This is Part 2, where I move the Kubernetes setup from my local Minikube to the cloud where I will deploy it on AWS EKS.

The source code of the project is available on GitHub.

Architecture Diagram

AWS Infrastructure with Terraform

Since I will be constantly setting up and tearing down the cloud infrastructure for my learning process, I decided to use Terraform to provision my infrastructure as it allows me to easily create and destroy resources on AWS. The first thing that I provisioned is a VPC with two private subnets and two public subnets, with the public subnets used for the ALB and NAT gateway and the private subnets used for everything else. This ensures that my EKS nodes and database are never directly reachable from the internet.

For the EKS cluster itself, I provisioned a managed node group, which lets AWS handle the underlying EC2 instances while I focus on the workloads running on top. I also created an IAM OIDC identity provider for my EKS cluster for it to access my other AWS resources securely using IRSA. For my database, I created a RDS Postgres database in my private subnet, configuring the security group to only allow ingress traffic originating from the EKS nodes through port 5432. By deploying the RDS instance into a private subnet and limiting the traffic that can enter, it effectively reduces the attack surface of my RDS instance and ensures that only my application can access the database.

Storing secrets using AWS Secrets Manager

In the first part of my project, the secrets are stored directly in Kubernetes Secrets, which is not very secure as they are just base64 encoded but not encrypted. Now that I am deploying the application onto AWS, I have the option to use AWS Secrets Manager to securely store my secrets. To access the secrets stored on AWS, I made use of Kubernetes External Secret Operator (ESO) to pull the secret from AWS and turn them into native Kubernetes Secrets. I chose to use IRSA to give ESO the IAM permissions required to read from Secrets Manager as it grants the permissions without needing static credentials. Using IRSA also helps to ensure that only the ESO pod have access to the secrets on AWS, limiting the blast radius should another pod be compromised.

ALB Ingress Controller

To route traffic into the EKS cluster, I chose to use the AWS Load Balancer Controller, which natively integrates with AWS networking and automatically provisions and manages ALBs directly from Kubernetes Ingress resources. Similar to ESO, I also used IRSA to provide the necessary permissions required for the ALB Controller to function. I was initially confused by what permissions I need to grant to the IAM role of the ALB Controller, but luckily I found the official policy document that I can download from the ALB Controller GitHub repository, helping me ensure that the role is granted only the minimum permissions required.

Kustomize: Running on Both Minikube & EKS

When migrating my Kubernetes setup to EKS, I wanted to retain compatibility with my original Minikube configuration. This allows me to continue using Minikube as a local sandbox for future Kubernetes experimentation, while using EKS to practice deploying Kubernetes on the cloud. I decided to use Kustomize to achieve this, as it allows me to maintain a shared base configuration and apply environment specific overlays for Minikube and EKS without modifying the original manifests. I considered using Helm for this purpose but decided against it as the added complexity of chart templating and release management would take away from my main focus of learning Kubernetes.

The base layer contains everything that is common across the two environments, such as namespace, deployment, and ingress. The security hardening that I have done in Part 1 of the project, like non-root user and network policy, are kept in this layer as this ensures that these controls are present regardless of which environment I am deploying to. The Postgres StatefulSet that I had created in Part 1 is placed under the Minikube overlay as I will not be requiring it on my EKS environment.

Bridging the gap to EKS

The resources required for running the cluster on EKS are placed in the AWS overlay. Rather than creating new resources to replace existing resources in the base layer, I made use of patches to modify the base manifest. For example, the ingress patch swaps the nginx ingress class for ALB annotations so the AWS Load Balancer Controller picks it up, while the network policy patch changes egress rules to account for the RDS database that is outside of the cluster. On top of the patches, the overlay introduces two new resources, which are the ClusterSecretStore and ExternalSecret. These two resources allow the cluster to fetch and use the secrets that are stored in the AWS Secrets Manager that I have previously provisioned.

Conclusion

Overall, deploying the application onto EKS gave me a much clearer picture of how Kubernetes is actually used and managed in the cloud, beyond what I knew from using Minikube. It allowed me to better understand the infrastructure required for maintaining a cluster on AWS, and gave me a better sense of what running workloads in the industry actually looks like. Using Kustomize also showed me how a single source of truth can manage configurations across different environments, eliminating the drift that comes with maintaining separate manifests.

This marks the end of the two-part project of learning Docker and Kubernetes security. Along the way, I picked up the kind of practical knowledge that's hard to get from documentation alone: how images, manifests, networking, and IAM all fit together, and how each layer introduces its own security considerations. More importantly, it gave me a better understanding of the attack surface that containers and Kubernetes introduce in production, and the measures available to defend against it.