Tag: AWS CLI

Building and testing polyglot applications using AWS CodeBuild

Building and testing polyglot applications using AWS CodeBuild

Prakash Palanisamy, Solutions Architect

Microservices are becoming the new normal, and it’s natural to use multiple different programming languages for different microservices in the same application. This blog post explains how easy it is to build polyglot applications, test them, and package them for deployment using a single AWS CodeBuild project.

CodeBuild adds support for Polyglot builds using runtime-versions. With CodeBuild, it is possible to specify multiple runtimes in the buildspec file as part of the install phase. Runtimes can be composed of different major versions of the same programming language, or different programming languages altogether. For a complete list of supported runtime versions and how they must be specified in the buildspec file, see the build specification reference for CodeBuild.

This post provides an example of a microservices application comprised of three different microservices written using different programming languages. The complete code is available in the GitHub repository aws-codebuild-polyglot-application.

  • A microservice providing a greeting message (microservice-greeting) is written in Python.
  • A microservice obtaining users’ details (microservice-name) from an Amazon DynamoDB table is written using Node.js.
  • Another microservice making API calls to the above-mentioned name and greeting services, and provide a personalized greeting (microservice-webapp), is written in Java.

All of these microservices are deployed as serverless functions in AWS Lambda and front-ended by an API interface using Amazon API Gateway. To enable automated packaging and deployment, use AWS Serverless Application Model (AWS SAM) and the AWS SAM CLI. The complete application is deployed locally using DynamoDB Local and the sam local command. Automated UI testing uses the built-in headless browsers in the standard CodeBuild containers. To run DynamoDB Local and SAM Local docker runtime is being used hence the privileged mode should be enabled in the CodeBuild project.

The example buildspec file contains the following code:

version: 0.2

env:
  variables:
    ARTIFACT_BUCKET: "polyglot-codebuild-artifact-eu-west-1"

phases:
  install:
    runtime-versions:	
      nodejs: 10
      java: corretto8
      python: 3.7
      docker: 18
    commands:
      - pip3 install --upgrade aws-sam-cli selenium pylint
  pre_build:
    commands:
      - python --version
      - pylint $CODEBUILD_SRC_DIR/microservices-greeting/greeting_handler.py
      - own_ip=`hostname -i`
      - sed -ie "s/CODEBUILD_IP/$own_ip/g" $CODEBUILD_SRC_DIR/env-webapp.json
  build:
    commands:
      - node --version
      - cd $CODEBUILD_SRC_DIR/microservices-name
      - npm install
      - npm run build
      - java -version
      - cd $CODEBUILD_SRC_DIR/microservices-webapp
      - mvn clean package -Plambda
      - cd $CODEBUILD_SRC_DIR
      - |
        sam package --template-file greeting-sam.yaml \
        --output-template-file greeting-sam-packaged.yaml \
        --s3-bucket $ARTIFACT_BUCKET
      - |
        sam package --template-file name-sam.yaml \
        --output-template-file name-sam-packaged.yaml \
        --s3-bucket $ARTIFACT_BUCKET
      - |
        sam package --template-file webapp-sam.yaml \
        --output-template-file webapp-sam-packaged.yaml \
        --s3-bucket $ARTIFACT_BUCKET
  post_build:
    commands:
      - cd $CODEBUILD_SRC_DIR
      - echo "Starting local DynamoDB instance"
      - docker network create codebuild-local
      - |
        docker run -d -p 8000:8000 --network codebuild-local \
        --name dynamodb amazon/dynamodb-local
      - |
        aws dynamodb create-table --endpoint-url http://127.0.0.1:8000 \
        --table-name NamesTable \
        --attribute-definitions AttributeName=Id,AttributeType=S \
        --key-schema AttributeName=Id,KeyType=HASH \
        --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5
      - |
        aws dynamodb --endpoint-url http://127.0.0.1:8000 batch-write-item \
        --cli-input-json file://ddb_names.json
      - echo "Starting APIs locally..."
      - |
        nohup sam local start-api --template greeting-sam.yaml \
        --port 3000 --host 0.0.0.0 &> greeting.log &
      - |
        nohup sam local start-api --template name-sam.yaml \
        --port 3001 --host 0.0.0.0 --env-vars env.json \
        --docker-network codebuild-local &> name.log &
      - |
        nohup sam local start-api --template webapp-sam.yaml \
        --port 3002 --host 0.0.0.0 \
        --env-vars env-webapp.json &> webapp.log &
      - cd $CODEBUILD_SRC_DIR/website
      - nohup python3 -m http.server 8080 &>webserver.log &
      - sleep 20
      - echo "Starting headless UI testing..."
      - python $CODEBUILD_SRC_DIR/tests/testsuite.py
artifacts:
  files:
    - greeting-sam-packaged.yaml
    - name-sam-packaged.yaml
    - webapp-sam-packaged.yaml

In the env section, an environment variable named ARTIFACT_BUCKET is uploaded and initialized. It contains the name of the S3 bucket in which the sam package command generated the AWS SAM template. This variable can be overridden either in the build project or while running the build. For more information about the order of precedence, see Run a Build (Console).

In the install phase, there are two sequences: runtime-versions and commands. As part of the runtime versions, four runtimes are specified: Three programming languages’ runtime (Node.js, Java, and Python), which are required to build the microservices, and Docker runtime to deploy the application locally using the sam local command. In the commands sequence, the required packages are installed and updated.

In the pre_build phase, use pylint to lint the Python-based microservice. Get the IP address of the CodeBuild container, to be used later while running the application locally.

In the build phase, use the command sequence to build individual microservices based on their programming language. Use the sam package command to create the Lambda deployment package and generate an updated AWS SAM template.

In the post_build phase, start a DynamoDB Local container as a daemon, create a table in that database, and insert the user details in to that table. After that, start a local instance of greeting, name, and web app microservices using the sam local command. The repository also contains a simple static website that can make API calls to these microservices and provide a user-friendly response. This website is hosted locally using Python’s built-in HTTP server. After the application starts, execute the automated UI testing by running the testsuite.py script. This test script uses Selenium WebDriver. It runs UI tests on headless Firefox and Chrome browsers to validate whether the local deployment of the application works as expected.

If all the phases complete successfully, the updated AWS SAM template files are stored as artifacts based on the specification in artifacts section.

The repository also contains an AWS CloudFormation template that creates a pipeline on AWS CodePipeline with AWS CodeCommit as sources. It builds the application using the previously mentioned buildspec in CodeBuild, and deploys the application using AWS CloudFormation.

Conclusion

In this post, you learned how to use the runtime-versions capability in CodeBuild to build a polyglot application. You also learned about automated UI testing using built-in headless browsers in CodeBuild. Using polygot build capabilities of CodeBuild you shall efficiently handle building applications developed on multiple programming languages using a single build project, instead of creating and maintaining multiple build projects. Similarly, since all the dependent components are built as part of the same project, it also improves the ease of testing including the integration between components.

from AWS DevOps Blog

Introducing AWS Chatbot: ChatOps for AWS

Introducing AWS Chatbot: ChatOps for AWS

DevOps teams widely use chat rooms as communications hubs where team members interact—both with one another and with the systems that they operate. Bots help facilitate these interactions, delivering important notifications and relaying commands from users back to systems. Many teams even prefer that operational events and notifications come through chat rooms where the entire team can see the notifications and discuss next steps.

Today, AWS introduces a public beta of AWS Chatbot, a new service that enables DevOps teams to receive AWS notifications and execute commands in Slack channels and Amazon Chime chat rooms with only minimal effort. AWS fully manages the integration, and the service takes only a few minutes to configure.

AWS Chatbot is in beta with support for receiving notifications from the following services:

  • Amazon CloudWatch
  • AWS Health
  • AWS Budgets
  • AWS Security Hub
  • Amazon GuardDuty
  • AWS CloudFormation

For the up-to-date list of supported services, see the AWS Chatbot documentation.

What our customers say

Revcontent is a content discovery platform that helps advertisers drive highly engaged audiences through technology and partnerships with some of the world’s largest media brands. By using AWS Chatbot, Revcontent has avoided potential downtime.

Our engineering teams have leveraged AWS Chatbot to enhance our system monitoring capabilities through integration with Amazon SNS and Amazon CloudWatch alarms. The initial setup was simple, and the return has been substantial! Slack functionality has enabled more efficient real-time notifications. For example, we avoided potential outages when AWS Chatbot alerted us of elevated error rates on a load balancer. We identified and resolved Amazon Redshift load aborts within minutes when AWS Chatbot notified our engineering teams of reduced network throughput on our cluster. — Christopher Ekeren, DevOps engineer, Revcontent

Walkthrough

In this post, I walk you through the configuration steps to set up a CloudWatch alarm with a Slack channel or Amazon Chime webhook using AWS Chatbot.

AWS Chatbot uses Amazon SNS to integrate with other AWS services, as shown in the diagram. This process sets up a CloudWatch alarm to notify an SNS topic, which in turn activates AWS Chatbot to notify a chat room.

Setting up AWS Chatbot for this example follows these steps:

  1. Create an SNS topic (optional).
  2. Create a CloudWatch alarm.
  3. Create an AWS Chatbot configuration.
  4. Complete the setup.
  5. Test the alarm.

Prerequisites

To follow along with this example, you need an AWS account, as well as a Slack channel or Amazon Chime webhook to configure with AWS Chatbot.

Step 1: Create an SNS topic

First, create an SNS topic to connect CloudWatch with AWS Chatbot. If you already have an existing SNS topic, you can skip this step.

In the SNS console, choose Topics, Create topic. Give your topic a descriptive name and leave all other parameters at their default.

Step 2: Create a CloudWatch alarm

For this post, create an alarm for an existing Lambda function. You want to receive a notification every time the function invocation fails so that you can diagnose and fix problems as they occur.

In the CloudWatch console, choose Alarms, Create alarm.

Select the metric to monitor, such as the Errors metric for a Lambda function. Configure the following fields:

  • For Period, enter 1 minute.
  • For Statistic, enter Sum.
  • For Threshold, enter Greater/Equal than 1.

These settings make it easier to trigger a test alarm.

For Send a notification to…, choose the SNS topic that you created in Step 1. To receive notifications when the alarm enters the OK state, choose Add notification, OK, and repeat the process.

Complete the creation process with the default settings.

Step 3: Create an AWS Chatbot configuration

To start configuring AWS Chatbot with the chat client, in the AWS Chatbot console, choose Configure new client, and choose either Amazon Chime or Slack.

Using AWS Chatbot with Slack

In the Configure new client pop-up, choose Slack.

The setup wizard redirects you to the Slack OAuth 2.0 page. In the top-right corner, select the Slack workspace to configure and choose Agree. Your Slack workspace installs the AWS Slack App, and the AWS account that you logged in with can now send notifications.

Slack redirects you from here to the Configure Slack Channel page. Select the channel in which to receive notifications. You can either select a public channel from the dropdown list or paste the URL or ID of a private channel.

Find the URL of your private Slack channel by opening the context (right-click) menu on the channel name in the left sidebar in Slack, and choosing Copy link. AWS Chatbot can only work in a private channel if you invite the AWS bot to the channel by typing /invite @aws in Slack.

Using AWS Chatbot with Amazon Chime

On the Configure new client pop-up, choose Amazon Chime.

For Webhook URL, copy the webhook URL from Amazon Chime and paste it into the text box. Give your webhook a description that reflects its location. I used the chat room name and the webhook name, separated by a slash.

Step 4: Complete the setup

After you choose the Slack channel or an Amazon Chime webhook, under IAM Permissions, create a role or select an existing role from the template. CloudWatch alarms can only display metric trends if AWS Chatbot has permission to call the CloudWatch API and retrieve metric details. To do this, choose Notifications permissions.

Finally, under SNS topics, select the SNS topic that you created in Step 1. You can select multiple SNS topics from more than one public Region, granting them all the ability to notify the same Slack channel.

After you choose Configure, the configuration completes.

Step 5: Test the alarm

You can test whether you properly configured AWS Chatbot by manually forcing your test Lambda function to fail. This should trigger an alarm and a notification in either Slack or Amazon Chime.

Test the alarm in Slack

Test the alarm in Amazon Chime

Conclusion

AWS Chatbot expands the communication tools that your team already uses every day to coordinate and bond. In this post, I walked you through the configuration steps to set up a CloudWatch alarm using AWS Chatbot in a Slack channel or Amazon Chime chat xroom.

from AWS DevOps Blog