Guest post by Pete Lesko, DevOps Engineer and Dan Richardson, Director of DevOps at Aledade

At Aledade, we perform ETL on the healthcare data of millions of patients from thousands of different sources, and the primary tool we leverage is the workflow management tool Airflow.

Because the amount of data we process is growing exponentially, we have quickly outgrown the ability to scale our dockerized Airflow deploy horizontally. We decided to move Airflow into Kubernetes to take advantage of their native support for scaling pods up and down, as needed, to handle tasks. With zero experience running a Kubernetes cluster, EKS allowed us to get up and running rapidly.

Here is how we did it.

There are a few tools that allow you to get up and running quickly on EKS. Cloudformation, Terraform, and eksctl are all good options with eksctl probably being the quickest way to get started. We picked Terraform because we were already using it to manage our AWS infrastructure. Terraform provides a nice tutorial and sample code repository to help you create all the necessary AWS services to run EKS. Their sample code is a good starting place and you can easily modify it to better suit your AWS environment.

NOTE: This tutorial will create a cluster in us-west-2 using the 10.0.0.0/16 subnet.

What You’ll Need

Before you get started, you’ll need a few tools installed. Terraform is tool to create, change, and improve infrastructure. Helm is a package management tool for Kubernetes. You’ll need to install them both:

terraform – https://www.terraform.io
helm – https://helm.sh

Terraform

Let’s start by cloning terraform’s EKS git repository from their AWS EKS Introduction. You’ll need to have installed the git client, a version control tool, for your operating system for the next command. On Ubuntu systems, you can accomplish this with apt-get install git, and RedHat based systems with yum install git. Now you can clone the terraform aws repository:

git clone https://github.com/terraform-providers/terraform-provider-aws.git

Terraform tracks the state in which it makes changes to your infrastructure in a state file. You’ll want to go into the examples directory, and initialize terraform with init. This will initialize terraform, creating the state file to track our work:

cd terraform-provider-aws/examples/eks-getting-started<br />terraform init

Now, to see a detailed outline of the changes terraform would make, run plan. This should include the EKS cluster, VPC, and other AWS resources that will be facilitated in this project:

terraform plan

Make sure to review the changes. The plan command will additionally warn you if there are any errors in your terraform code. Assuming everything looks alright, since this is a fresh checkout, you should be able to apply the default configuration using the apply:

terraform apply

Terraform will prompt you to make sure that you want to apply the changes, since this will create resources that will incur charges on our AWS account. You’ll want to go ahead and apply the changes since you already reviewed them with the plan command previously:

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes

By default, the resources are targeted to be created in us-west-2, so bear that in mind if you go looking for the resources created in your console. This apply step will create many of the resources you need to get up and running initially, including:

  • VPC
  • IAM roles
  • Security groups
  • An internet gateway
  • Subnets
  • Autoscaling group
  • Route table
  • EKS cluster
  • Your kubectl configuration

Setting Up kubectl

You will need the configuration output from terraform in order to use kubectl to interact with your new cluster. Create your kube configuration directory, and output the configuration from terraform into the config file using the terraform output command:

mkdir ~/.kube/<br />terraform output kubeconfig &gt; ~/.kube/config

You’ll need kubectl, a command line tool to run commands against Kubernetes clusters, for the next step. Installation instructions can be found here. Once you’ve got this installed, you’ll want to check to make sure that you’re connected to your cluster by running kubectl version. Your output may vary slightly here:

$ kubectl version<br />Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}<br />Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Now let’s add the ConfigMap to the cluster from terraform as well. The ConfigMap is a kubernetes configuration, in this case for granting access to our EKS cluster. This ConfigMap allows our ec2 instances in the cluster to communicate with the EKS master, as well as allowing our user account access to run commands against the cluster. Youu’ll run the terraform output command to a file, and the kubectl apply command to apply that file:

terraform output config_map_aws_auth &gt; configmap.yml<br />kubectl apply -f configmap.yml

Once this is complete, you should see your nodes from your autoscaling group either starting to join or joined to the cluster. Once the second column reads Ready the node can have deployments pushed to it. Again, your output may vary here:

$ kubectl get nodes -o wide<br />NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME<br />ip-10-0-0-193.us-west-2.compute.internal Ready 83s v1.12.7 10.0.0.193 18.236.129.186 Amazon Linux 2 4.14.106-97.85.amzn2.x86_64 docker://18.6.1<br />ip-10-0-1-237.us-west-2.compute.internal Ready 83s v1.12.7 10.0.1.237 34.222.17.198 Amazon Linux 2 4.14.106-97.85.amzn2.x86_64 docker://18.6.1

At this point, your EKS cluster is up, the nodes have joined, and they are ready for a deployment!

Helm

Next, you’ll install helm. First you need to create a Kubernetes ServiceAccount for tiller, which allows helm to talk to the cluster:

cat &amp;gt;tiller-user.yaml &amp;lt;&amp;lt;EOF&lt;br /&gt;apiVersion: v1&lt;br /&gt;kind: ServiceAccount&lt;br /&gt;metadata:&lt;br /&gt;name: tiller&lt;br /&gt;namespace: kube-system&lt;br /&gt;---&lt;br /&gt;apiVersion: rbac.authorization.k8s.io/v1&lt;br /&gt;kind: ClusterRoleBinding&lt;br /&gt;metadata:&lt;br /&gt;name: tiller&lt;br /&gt;roleRef:&lt;br /&gt;apiGroup: rbac.authorization.k8s.io&lt;br /&gt;kind: ClusterRole&lt;br /&gt;name: cluster-admin&lt;br /&gt;subjects:&lt;br /&gt;- kind: ServiceAccount&lt;br /&gt;name: tiller&lt;br /&gt;namespace: kube-system&lt;br /&gt;EOF


Now, you apply the ServiceAccount with kubectl, and install helm with the init command:

$ kubectl apply -f tiller-user.yaml<br />$ helm init --service-account tiller

You will need a way for our airflow deployment to communicate with the outside world. For this, you will install nginx-ingress, an ingress controller that uses ConfigMap to store nginx configurations. Nginx is an industry standard software for web and proxy servers. We will use the proxy feature to serve up our airflow web interface. Install nginx-ingress via the helm chart:

helm install \<br />stable/nginx-ingress \<br />--name my-nginx \<br />--set rbac.create=true

Airflow

You need to override some values in the airflow chart to tell it to use the nginx ingress controller. You’ll want to replace airflow-k8s.aledade.com with a hostname of your own:

cat &gt;values.yaml &lt;&lt;EOF<br />ingress:<br />enabled: true<br />web:<br />path: "/"<br />host: "airflow-k8s.aledade.com"<br />tls:<br />enabled: true<br />annotations:<br />kubernetes.io/ingress.class: "nginx"<br />EOF

Finally, you install airflow via the helm chart and the values file you just created using the helm install command:

helm install \<br />--namespace "airflow" \<br />stable/airflow \<br />--name airflow \<br />-f values.yaml

This may take a few moments before all of the pods are ready, and you can monitor the progress with:

<p> </p><p>$ watch "kubectl get pods -n airflow"</p>

Even after the pods are running, I’ve found it takes at least 5 minutes for everything to completely spin up.

You can find out the internet accessible endpoint by querying the services and looking for the LoadBalancer Ingress

$ kubectl describe services |grep ^LoadBalancer<br />LoadBalancer Ingress: 8a69022e5f102be1072e5fb1087f5fbe-e907efv7e8.us-west-2.elb.amazonaws.com

If you visit this URL, you will find the flower interface, a web tool for monitoring and administering celery clusters.

To reach the airflow administrative interface, you will need to add an entry to /etc/hosts, but first you need to get the IP address of that LoadBalancer Ingress, and add it to your /etc/hosts:

$ host 8a69022e5f102be1072e5fb1087f5fbe-e907efv7e8.us-west-2.elb.amazonaws.com&amp;lt;br /&amp;gt;8a69022e5f102be1072e5fb1087f5fbe-e907efv7e8.us-west-2.elb.amazonaws.com has address 35.161.144.248&amp;lt;br /&amp;gt;8a69022e5f102be1072e5fb1087f5fbe-e907efv7e8.us-west-2.elb.amazonaws.com has address 34.211.133.201&amp;lt;br /&amp;gt;$ su&amp;lt;br /&amp;gt;Password:&amp;lt;br /&amp;gt;# echo "34.211.133.201 airflow-k8s.aledade.com" &amp;amp;gt;&amp;amp;gt; /etc/hosts

Afterwards, you can reach the the airflow administative interface from the URL http://airflow-k8s.aledade.com in your browser. Under a production environment, you would replace airflow-k8s.aledade.com with a FQDN that you can add as an alias in route53 to point to the ELB created by the LoadBalancer Ingress.

Cleaning Up

To destroy these resources, delete the helm deployments, and issue a destroy with terraform

$ helm del --purge airflow;<br />$ helm del --purge my-nginx;<br />$ terraform destroy

Conclusion + Future

At Aledade, we help transform primary care by delivering more efficient technology-enabled workflows to primary care providers. We analyze data with Python, Docker, and Terraform and have a CI/CD pipeline into EKS. If that sounds good to you, consider joining our team!

from AWS Startups Blog