Guest post by Pete Lesko, DevOps Engineer and Dan Richardson, Director of DevOps at Aledade
At Aledade, we perform ETL on the healthcare data of millions of patients from thousands of different sources, and the primary tool we leverage is the workflow management tool Airflow.
Because the amount of data we process is growing exponentially, we have quickly outgrown the ability to scale our dockerized Airflow deploy horizontally. We decided to move Airflow into Kubernetes to take advantage of their native support for scaling pods up and down, as needed, to handle tasks. With zero experience running a Kubernetes cluster, EKS allowed us to get up and running rapidly.
Here is how we did it.
There are a few tools that allow you to get up and running quickly on EKS. Cloudformation, Terraform, and eksctl are all good options with eksctl probably being the quickest way to get started. We picked Terraform because we were already using it to manage our AWS infrastructure. Terraform provides a nice tutorial and sample code repository to help you create all the necessary AWS services to run EKS. Their sample code is a good starting place and you can easily modify it to better suit your AWS environment.
NOTE: This tutorial will create a cluster in
us-west-2 using the
What You’ll Need
Before you get started, you’ll need a few tools installed. Terraform is tool to create, change, and improve infrastructure. Helm is a package management tool for Kubernetes. You’ll need to install them both:
Let’s start by cloning terraform’s EKS
git repository from their AWS EKS Introduction. You’ll need to have installed the git client, a version control tool, for your operating system for the next command. On Ubuntu systems, you can accomplish this with
apt-get install git, and RedHat based systems with
yum install git. Now you can clone the terraform aws repository:
Make sure to review the changes. The plan command will additionally warn you if there are any errors in your terraform code. Assuming everything looks alright, since this is a fresh checkout, you should be able to apply the default configuration using the
Terraform will prompt you to make sure that you want to apply the changes, since this will create resources that will incur charges on our AWS account. You’ll want to go ahead and apply the changes since you already reviewed them with the plan command previously:
By default, the resources are targeted to be created in
us-west-2, so bear that in mind if you go looking for the resources created in your console. This apply step will create many of the resources you need to get up and running initially, including:
- IAM roles
- Security groups
- An internet gateway
- Autoscaling group
- Route table
- EKS cluster
- Your kubectl configuration
Setting Up kubectl
You will need the configuration output from terraform in order to use kubectl to interact with your new cluster. Create your kube configuration directory, and output the configuration from terraform into the
config file using the terraform
kubectl, a command line tool to run commands against Kubernetes clusters, for the next step. Installation instructions can be found here. Once you’ve got this installed, you’ll want to check to make sure that you’re connected to your cluster by running kubectl
version. Your output may vary slightly here:
Now let’s add the
ConfigMap to the cluster from terraform as well. The ConfigMap is a kubernetes configuration, in this case for granting access to our EKS cluster. This ConfigMap allows our ec2 instances in the cluster to communicate with the EKS master, as well as allowing our user account access to run commands against the cluster. Youu’ll run the terraform output command to a file, and the kubectl apply command to apply that file:
Once this is complete, you should see your nodes from your autoscaling group either starting to join or joined to the cluster. Once the second column reads
Ready the node can have deployments pushed to it. Again, your output may vary here:
At this point, your EKS cluster is up, the nodes have joined, and they are ready for a deployment!
Next, you’ll install helm. First you need to create a Kubernetes
ServiceAccount for tiller, which allows helm to talk to the cluster:
Now, you apply the ServiceAccount with kubectl, and install helm with the init command:
You will need a way for our airflow deployment to communicate with the outside world. For this, you will install nginx-ingress, an ingress controller that uses ConfigMap to store nginx configurations. Nginx is an industry standard software for web and proxy servers. We will use the proxy feature to serve up our airflow web interface. Install nginx-ingress via the helm chart:
You need to override some values in the airflow chart to tell it to use the nginx ingress controller. You’ll want to replace
airflow-k8s.aledade.com with a hostname of your own:
Finally, you install airflow via the helm chart and the values file you just created using the helm
This may take a few moments before all of the pods are ready, and you can monitor the progress with:
<p> </p><p>$ watch "kubectl get pods -n airflow"</p>
Even after the pods are running, I’ve found it takes at least 5 minutes for everything to completely spin up.
You can find out the internet accessible endpoint by querying the services and looking for the
$ kubectl describe services |grep ^LoadBalancer<br />LoadBalancer Ingress: 8a69022e5f102be1072e5fb1087f5fbe-e907efv7e8.us-west-2.elb.amazonaws.com
If you visit this URL, you will find the
flower interface, a web tool for monitoring and administering celery clusters.
To reach the airflow administrative interface, you will need to add an entry to /etc/hosts, but first you need to get the IP address of that
LoadBalancer Ingress, and add it to your
To destroy these resources, delete the helm deployments, and issue a destroy with terraform
Conclusion + Future
At Aledade, we help transform primary care by delivering more efficient technology-enabled workflows to primary care providers. We analyze data with Python, Docker, and Terraform and have a CI/CD pipeline into EKS. If that sounds good to you, consider joining our team!
from AWS Startups Blog