Tag: Management Tools

Introducing Service Quotas: View and manage your quotas for AWS services from one central location

Introducing Service Quotas: View and manage your quotas for AWS services from one central location

By Lijo George

Today we are introducing Service Quotas, a new AWS feature which enables you to view and manage your quotas, also known as limits, from a central location via the AWS console, API or the CLI.

Service Quotas is a central way to find and manage service quotas, an easier way to request and track quota increases, and a simplified way to request quota increases for new accounts created in AWS Organizations. For supported services, you can proactively manage your quotas by configuring Amazon CloudWatch alarms that monitor usage and alert you to approaching quotas. You can access Service Quotas by visiting directly or by searching for it in the AWS management console.

Going forward, we will be referring to service quotas (instead of service limits) to better represent our philosophy of providing you with better control of your AWS resources. Please note that you may encounter both terms being used interchangeably.


  1. Central management of AWS service quotas: Using Service Quotas, you can manage your AWS service quotas in one central location, eliminating the need to go to multiple sources or maintain your own list. You can access Service Quotas either through the console or programmatically via the API or CLI.
  2. Improved visibility of service quotas: You can view default and account-specific (applied) quotas for multiple AWS services. Applied quotas are overrides that are specific to your account that have been granted to you in the past. At launch, you can view default quotas for 90 AWS services, with more coming soon.
  3. Easier quota increase requests: You can request a quota increase through a single page on the console or through an API call. You simply search for a quota and put in your desired value to submit a quota increase request. You can also view and track the status of your requests.
  4. Paving the way for proactive quota management: Service Quotas integrates with CloudWatch to alert you when you reach a threshold, enabling you to proactively manage your quotas.
  5. Simplify quota requests for new accounts in AWS Organizations: Customers often request quota increases for new accounts that they create in their organization. Service Quotas automates this process so that you spend less time requesting increases for new accounts in your organization, and ensures that all your accounts are consistently configured in accordance with the needs of your workloads.

Using Service Quotas

Emma’s serverless world

Emma is a lead developer in a startup that does image processing for customers. Her company uses AWS Lambda extensively. They love the powerful real time file and stream processing capabilities that Lambda provides. Emma diligently ensures that her application is highly available and flexible to match user workloads. However, she’s concerned about unexpectedly hitting quotas. She previously spent time gathering data about Lambda to make sure her workflows did not stop growing.

To learn about Lambda quotas, Emma visits Service Quotas, navigates to the AWS Lambda page where she can view information about all the quotas for the service.

One quota that catches Emma’s attention is ‘Concurrent executions’. She clicks on the quota to see additional details, including her actual usage against the quota in the CloudWatch graph on the same page.

Emma expects her concurrent executions to increase and decides to puts in a request to increase it so as to have excess concurrency for her new workloads. Requesting a quota increase is a common task that is now made easier through Service Quotas. Emma can now click on ‘Request quota increase’ and fill out the quota increase form.

Once Emma finishes submitting her quota increase request, she can track it through the ‘Quota increase history’ page. Her quota request needed additional review, so it was sent to customer support. Emma can see this and find the support center case number on the quota increase detail page. Selecting the case redirects Emma to the customer support console, where Emma can communicate with the agent who is handling this case.

After requesting a quota increase, Emma also sets up a CloudWatch alarm to alert her when her application’s usage is 80% of the default quota so that she can monitor usage and request for increases, if needed.

Alan’s expanding AWS workloads

Alan is a Cloud Admin in a large financial services company. His task is to provision resources for his growing AWS fleet. Alan uses AWS Organizations to manage his 500 AWS accounts. Once Alan creates an account, he routinely requests quota increases to make sure the account is configured in a consistent manner.

With Service Quotas, Alan can accomplish the same goal with a pre-defined Service Quota template in his organizations’ master account.

Once this template is active, for each new account he creates in his organization, the same quota increase requests will be created. Alan can track the status of these requests in the ‘Quota increase history’ page of the newly created accounts.

Available Now

Currently, Service Quotas is available in US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon), Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Asia Pacific (Osaka), Canada (Central), EU (Frankfurt), EU (Ireland), EU (London), EU (Paris) and South America (São Paulo). Service Quotas is available free of cost – you will only be charged for the CloudWatch alarms you set up.


Service Quotas allows you to view and manage quotas for AWS services from one central location. Service Quotas enables you to easily raise and track quota increase requests, and integrates with AWS Organizations to save you time and effort in setting up quotas for new accounts in a consistent manner.

If you have questions about or suggestions for this solution, start a new thread on the Service Quotas forum.

from AWS Management Tools Blog

Managing multiple stacks and overriding parameters in AWS CloudFormation

Managing multiple stacks and overriding parameters in AWS CloudFormation

In Q4 2018, the AWS CloudFormation team released a refreshed version of the management console as an opt-in experience. This redesign re-explored many of the common ways that you can interact with this service.

In this post, I’d like to introduce a suite of updates that makes the redesign into the default console experience. The updates bring a more robust approach to accessibility, performance enhancements, and better metrics, which helps AWS iterate on feedback quicker. AWS thanks all of you who used the opt-in experience and submitted feedback; we are listening.

I’d like to take you through some of the flows that have improved since the 2018 release. For more information, see Working with Nested Stacks and Override Parameters on Stack Instances.

Managing multiple stacks

While the initial implementation worked well for drilling into individual stacks, customers asked for a better way to navigate multiple stacks. We’re pleased to announce a revamped split-pane view for the stacks page. This new view makes navigating between different stack events, resources, or outputs much easier.

In this example screenshot, I created a nested stack for a fictitious company named AnyCompany. This nested stack deploys a full reproduction of their production environment for testing purposes.

Nested stack parent creation

As stacks are being created, you can quickly cycle through the events, resources, and outputs of any of the stacks. After all the items have been created, you can limit your view to the parent stacks, to simplify the view.

Nested stack parent-only view

This view allows you to quickly traverse through large groups of stacks and browse through events easily. You can also use the powerful filtering tools to nimbly navigate through any numbers of stacks and keep relevant pieces of information together.

Overriding parameters on your StackSets

Back in Q4 2017, CloudFormation StackSets released a feature that allowed you to override parameter values by account and Region in stack instances. Before this release, you could only have one value for any parameter in your template shared across all Regions.

The new console allows you to use this feature to have fine-grained control of exactly which parameter is getting overridden and which parameter is getting reset back to the StackSet default.

In this example, AnyCompany must create a config rule to verify that all accounts and Regions in the StackSet have the mandatory two-factor authentication (2FA), to be verified every twelve hours. AnyCompany decides that they want three specific accounts to have a limit of 24-hours instead of 12 hours in the us-east-1 Region.

In the details view of the StackSet, choose Actions, Override StackSet parameters.

StackSet details

On the Set Deployment options page, enter the accounts and Regions.

Find the specific parameters to override and choose Edit override value, Override StackSet value. The dialog box allows you to specify the values to which the parameters can be set, as defined by the StackSet template.

Confirm the new correct value on the page, confirm one more time on the confirmation page, and then choose Submit.

The transaction is complete and a deployment begins to roll out all the overridden parameters to these selected stack instances. When the deployment completes, all selected stack instances now have the overridden parameter values.


I’ve shown you a couple of new user flows that the AWS CloudFormation console experience has improved in the latest update. I walked through an example of how to browse events and resources between multiple stacks, as well as how to override select parameters in a StackSet. I’m looking forward to hearing your feedback!

About the Author

Dan Blanco is a Developer Advocate for AWS CloudFormation based in Seattle. Dan writes blogs, templates, code, and tips to consistently improve the developer experience for CloudFormation users. When Dan’s not bolted to his laptop, you’ll find him playing a new board game or cooking up something in the kitchen. You can find more of Dan on Twitter (@TheDanBlanco) or on the AWS Developers #cloudformation Slack Channel

from AWS Management Tools Blog

Understanding AWS Lambda behavior using Amazon CloudWatch Logs Insights

Understanding AWS Lambda behavior using Amazon CloudWatch Logs Insights

In this guest post, Alice Xiao, Data Analyst at financial services company State Street Corporation discusses how State Street built a framework using CloudWatch Logs Insights to support their innovation process of rapidly converting business ideas into client-centered applications.


Customers want to do rapid prototyping and ensure critical defects are quickly resolved, so understanding application behavior is critical. Amazon CloudWatch Logs Insights enables you to create ad hoc queries to understand AWS Lambda behavior based on Lambda logs stored in Amazon CloudWatch Logs. You can then create quick visualizations from that data.


State Street uses serverless applications to support their innovation process, which focuses heavily on rapid prototyping. For serverless applications—and more specifically, Lambda functions—the application teams must ensure that they are configuring Lambda runtime parameters appropriately.

To do this, State Street uses CloudWatch Logs Insights to perform heuristic analysis of Lambda logs and gain insight into the sizing and billing of each of their Lambda functions. Additionally, they do further analysis by correlating runtime and memory usage.

“The goal was to have a solution that is concise, uses AWS native functionality, empowers the developer, and supports rapid prototyping,” said Nauman Noor, Managing Director of Public Cloud Platform at State Street Corporation.

Lambda automatically monitors Lambda functions on your behalf, reporting metrics through Amazon CloudWatch. To help you troubleshoot failures in a function, Lambda logs all requests handled by your function and also automatically stores logs generated by your code in CloudWatch Logs.

Each invocation of a Lambda function publishes a “REPORT” log entry at the end of the invocation. The entry provides details about the actual duration, billed duration, and amount of memory used, as shown in the following example:

REPORT RequestId: b68fd6c0-f38e-42d0-82f2-48b7484e5fe5 Duration: 172.63 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 35 MB
REPORT RequestId: f610598b-9adc-430e-96f4-58340016e6b9 Duration: 179.20 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 35 MB
REPORT RequestId: 0809c288-8b19-4685-9d1a-6b98852299ff Duration: 179.29 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 35 MB
REPORT RequestId: 8b4a32a4-7dc0-4342-a998-48b051d683e0 Duration: 182.29 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 35 MB

By using CloudWatch Logs Insights, you can extract this information in the Lambda REPORT logs and better understand and ensure proper Lambda configuration. At a deeper level, this means determining runtime duration characteristics of invocation and analyzing memory usage to mitigate memory occurrences. You can also tell if there is a correlation between memory usage and runtime duration for any specific Lambda function.


Before CloudWatch Logs Insights, a common approach was to subscribe a Lambda function to the CloudWatch logs generated by Lambda. The function would then parse the log information (Duration, Billed Duration, and Memory Size) and store it in a data store, such as Amazon DynamoDB. Although this approach worked, it required additional effort from the developer to manage code and orchestrate the deployment, configuration, and integration into such a solution. This did not meet State Street’s goal of empowering the developer.

Our approach was to leverage CloudWatch Logs Insights to provide the heuristic analytic capabilities while empowering the developer and removing operational overhead.

The following screenshot shows the user interface for CloudWatch Log Insights, including the query, distribution of log events over time, and the log data.

With no setup needed, CloudWatch Logs Insights provides the developer with the ability to create ad hoc queries. They can use the queries to understand Lambda function behavior based on logs stored in CloudWatch Logs. They can also create quick visualizations.

Scenario Illustration

To illustrate some of these concepts, I will walk you through a scenario focused on understanding the runtime characteristics of a Lambda function.

1.     In CloudWatch Logs Insights, select a Lambda log group.

2.     Execute the following queries:

filter @type="REPORT"
| stats avg(@billedDuration) as mean_billed_duration,
min(@billedDuration) as min_billed_duration,
max(@billedDuration) as max_billed_duration,
percentile(@billedDuration, 95) as Percentile95

filter @type="REPORT"
| stats avg(@maxMemoryUsed/1024/1024) as mean_MemoryUsed,
min(@maxMemoryUsed/1024/1024) as min_MemoryUsed,
max(@maxMemoryUsed/1024/1024) as max_MemoryUsed,
percentile(@maxMemoryUsed/1024/1024, 95) as Percentile95

The executed queries perform the following:

·       Filter out the “REPORT” logs.
·       Select a key field (@duration, @billedDuration, or @maxMemoryUsed).
·       Get the statistics, such as average, minimum, maximum, and percentile.

This results in the following output:

# mean_billed_duration min_billed_duration max_billed_duration Percentile95
1 30 38 46 54
# mean_MemoryUsed min_MemoryUsed max_MemoryUsed Percentile95
1 33.7255 30.5176 69.6182 33.3786





3. To gather more insight and understand the distribution of the fields, generate a frequency table for @maxMemoryUsed with the bin(period) function and determine the range of each bucket, as follows:

filter @type="REPORT"
| fields floor(@maxMemoryUsed/1024/1024) as floor_var,
  ceil(@maxMemoryUsed/1024/1024) as ceil_var
| stats min(floor_var) as min_Memory_Used,
  (min_Memory_Used + (max_Memory_Used - min_Memory_Used)/5) as bucket1,
  (min_Memory_Used + 2 * (max_Memory_Used - min_Memory_Used)/5) as bucket2,
  (min_Memory_Used + 3 * (max_Memory_Used - min_Memory_Used)/5) as bucket3,
  (min_Memory_Used + 4 * (max_Memory_Used - min_Memory_Used)/5) as bucket4,
  max(ceil_var) as max_Memory_Used

# min_Memory_Used bucket1 bucket2 bucket3 bucket4 max_Memory_Used
1 30 38 46 54 62 70





4.     Count the number of invocations that fall within the @maxMemoryUsed range bucket:

filter @type="REPORT"
| fields (@maxMemoryUsed/1024/1024) as MemoryMB,
  (MemoryMB>=30 and MemoryMB<38) as R30_38,
  (MemoryMB>=38 and MemoryMB<46) as R38_46,
  (MemoryMB>=46 and MemoryMB<54) as R46_54,
  (MemoryMB>=54 and MemoryMB<62) as R54_62,
  (MemoryMB>=62 and MemoryMB<=70) as R62_70
| stats sum(R30_38) as MemoryUsed30MB_38MB,
  sum(R38_46) as MemoryUsed38MB_46MB,
  sum(R46_54) as MemoryUsed46MB_54MB,
  sum(R54_62) as MemoryUsed54MB_62MB,
  sum(R62_70) as MemoryUsed62MB_70MB

# MemoryUsed30MB_38MB MemoryUsed38MB_46MB MemoryUsed46MB_54MB MemoryUsed54MB_62MB MemoryUsed62MB_70MB
1 1242 0 0 0 49

So far, most of your Lambda invocations have consumed between 30–38 MB of memory, while 49 invocations consumed between 62–70 MB. These insights make me question if these high-memory usage situations could be correlated with longer invocation runtime.

5.     Repeat this process for @billedDuration to get the following results:

filter @type="REPORT"
| stats min(@billedDuration) as min_billed_duration,
  (min_billed_duration + (max_billed_duration - min_billed_duration)/5) as bucket1,
  (min_billed_duration + 2 * (max_billed_duration - min_billed_duration)/5) as bucket2,
  (min_billed_duration + 3 * (max_billed_duration - min_billed_duration)/5) as bucket3,
  (min_billed_duration + 4 * (max_billed_duration - min_billed_duration)/5) as bucket4,
  max(@billedDuration) as max_billed_duration

# min_billed_duration bucket1 bucket2 bucket3 bucket4 max_billed_duration
1 200 360 520 680 840 100




filter @type="REPORT"
| fields (@billedDuration>=200 and @billedDuration<360) as R200_360,
  (@billedDuration>=360 and @billedDuration<520) as R360_520,
  (@billedDuration>=520 and @billedDuration<680) as R520_680,
  (@billedDuration>=680 and @billedDuration<840) as R680_840,
  (@billedDuration>=840 and @billedDuration<=1000) as R840_1000
| stats sum(R200_360) as billedDuration200_360,
  sum(R360_520) as billedDuration360_520,
  sum(R520_680) as billedDuration520_600,
  sum(R680_840) as billedDuration680_840,
  sum(R840_1000) as billedDuration840_1000

# billedDuration200_360 billedDuration360_520 billedDuration520_600 billedDuration680_840 billedDuration840_1000
1 1121 161 4 3 2

As you can see from the results, most of the invocations get billed between 200–360 ms of runtime, with a few taking more than 520 ms.

At this point, I wondered if the 49 invocations that consumed the most memory corresponded to the top @billedDuration ranges? In other words, is runtime duration correlated to memory usage, so that high-memory usage requires a longer runtime?

The following query shows a count of the invocations that used 62–70 MB of memory along with the billed duration and memory used.

fields (@maxMemoryUsed/1024/1024) as MemoryMB
| filter @type="REPORT" and (MemoryMB>=62 and MemoryMB<=70)
| stats count(@billedDuration) by MemoryMB, @billedDuration

# MemoryMB @billedDuration count(@billedDuration)
1 69.6182 300 7
2 69.6182 400 21
3 69.6182 500 4
4 69.6182 200 16
5 68.6646 1000 1








Analyzing the 49 invocations that used 62–70 MB memory and their corresponding @billedDuration shows that only one corresponded to the 1,000-ms runtime. The majority of the others ran 400 ms or less. At this point, the relationship is not obvious and the correlation appears to be weak.

6.     Expand the scope of study to a larger period (and greater sample size) of 10 hours. Attempt to visualize a five-minute moving average of @billedDuration (ms) vs. @memoryUsed (in 0.1 MB), as shown in the following code:

filter @type="REPORT"
| stats avg(@billedDuration) as BilledDuration,
avg(@maxMemoryUsed/1024/102.4) as MemoryUsedMB_scaled by bin(5m)

The following screenshot shows a line chart visualization of the billed duration and memory used (MB) on the y-axis against time on the x-axis.

After you expand the scope of study to a larger period, in some situations, a spike in memory usage may correspond to a longer runtime duration. However, it appears that the correlation between @maxMemoryUsed and @billedDuration is not clear.

This insight suggests that the application team should review the execution flow. They can see if the runtime is a function of exception handling that is normally not the case in other scenarios. Or, the memory consumption might be a function of the data being processed.

Ultimately, the information that CloudWatch Logs Insights provides helps the team narrow in on the specific invocations of interest from a runtime and performance-optimization perspective.


In this post, I demonstrated how CloudWatch Logs Insights provides a quick understanding of Lambda runtime behavior, such as duration and memory usage and the potential correlation between the two dimensions.

With CloudWatch, you can parse, filter, sort, and infer statistical outputs in a simple and repeatable manner without overhead from the application team having to instrument their applications to enable the analysis.

I really enjoy the ease of use of CloudWatch Logs Insights. And I look forward to future enhancements, such as additional robust query functions and the ability to set custom attributes within queries that aid more complex analysis while simplifying some of the proposed techniques.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

from AWS Management Tools Blog

Using AWS Cost Explorer to analyze data transfer costs

Using AWS Cost Explorer to analyze data transfer costs

AWS Cost Explorer helps you manage your AWS costs by giving you detailed insights into the line items in your bill. In Cost Explorer, you can visualize daily, monthly, and forecasted spend by combining an array of available filters. Filters allow you to narrow down costs according to AWS service type, linked accounts, and tags. You can uniquely tag your AWS services. For example, you can tag Amazon EC2 instances in Amazon Elastic Compute Cloud (EC2), or you can tag Application Load Balancers in the Elastic Load Balancing service.

In this blog post, we’ll discuss how to analyze data transfer costs using Cost Explorer. Data transfer charges are incurred when data is transferred out from AWS services to the Internet, or between AWS regions or Availability Zones. When EC2 instances are behind a load balancer, it becomes difficult to narrow down the cost of the data transfer or to understand the cost associated with Internet outbound, inter-Availability Zone or inter-Region data transfers, as well as which environment these costs occurred in.

Setting up cost allocation tags

Before you set up cost allocation tags, categorize your instances. In the example in this blog post, we’ll use either production or development, but you can tag them in whatever way makes sense for your business. To do this, go to your Amazon EC2 console, and, for each instance, create a tag (if it wasn’t  already created at the time of instance creation) with Key=Environment and  Value=Production or Development.

EC2 Instances that are launched dynamically as part of an Auto Scaling group can be tagged as described here.

Choose Load Balancers in the left navigation pane, and create an Environment tag for each load balancer in the same way.

After you tag your EC2 Instance and your load balancer, the next step is to activate the cost allocation tags as follows:

1. Sign in to your Billing and Cost Management console at https://console.aws.amazon.com/billing/home?#/.

2. In the navigation pane, choose Cost Allocation Tags.

3. Select the Environment tag and choose Activate.

Note: It might take up to 24 hours for the tags to gather costs associated with the AWS service.

Using filters to analyze data transfer costs

After your cost allocation tags have been activated, and your workloads have run for at least a day, you can use filters in Cost Explorer to analyze your costs over that period.

1. Sign in to AWS Cost Explorer at https://console.aws.amazon.com/cost-reports/home?#/

2. Choose Explore in the navigation pane, and then choose Cost and Usage.

3. Choose the date range for the period for which you want to see the costs, and choose the Apply button.













4. Choose Filters – Service, then EC2-instances, and then EC2-ELB.Next choose Apply filters.












5. To see the total EC2 data transfer cost:

Choose Filters, Usage Type Group, EC2: Data Transfer – inter-Availability Zone, Internet (Out), and Region to Region (Out), then choose Apply filters.

You can also choose each individual data transfer type by checking only the box for that type.













6. Next, choose Filters – Tag, then choose the tags that you want to see, and choose Apply filters. In this case we’ll choose Development and Production.











7. Choose Apply filters to see the total data transfer costs for all EC2 instances and load balancers.


  1. To view the costs broken down by environment, choose Tag in the Group By menu, and then choose Environment. This shows both a graph and a line-item summary of the data transfer costs for each of your environments.

2. Choose Download CSV to save and share the results.

Reducing Data Transfer Costs

If data transfer from EC2 to the public internet shows up as a significant cost, consider using Amazon CloudFront. Any image, video, or static web content can be cached at AWS edge locations worldwide, using the Amazon CloudFront Content Delivery Network (CDN).

Here are some other ways to reduce the data transfer cost based on specific scenarios.

  1. You can architect your solution to avoid inter-Region data transfer costs. For example, accessing the data from Amazon S3 via Amazon EC2 within the same region is free of charge, whereas accessing Amazon S3 data from a different region incurs a cost.
  2. Also, be aware of inter-Availability Zones data transfer charges between Amazon EC2 instances, even within the same region. If possible, the instances in a development or test environment that need to communicate with each other should be co-located within the same Availability Zone to avoid data transfer charges. (This doesn’t apply to production workloads which will most likely need to span multiple Availability Zones for high availability.)
  3. You can avoid using public IP addresses for internal data transfers within the same Availability Zone. Inter-Availability Zone data transfers are free, provided you use private IP addresses
  4. If you have resources in different regions or multiple accounts then using VPC Peering or VPC Sharing can further help you to optimize the data transfer cost.

In this blog post, we have shown you how to visualize and understand costs associated with different data transfer types within your AWS environment. We explained how to apply and activate your tags for your AWS resources. Then we showed you how to use tags as filters within Cost Explorer. We also demonstrated how to combine different filters and create a narrowed down cost report which can provide further insights into the charges. You can modify this example and use filters that are suitable for your individual use cases.

About the authors



Ashish Mehra is a Solutions Architect at AWS. He is a Serverless, IoT and Containers enthusiast and loves helping customers design secure, reliable and cost-effective solutions on AWS.




Rana Dutt is a Senior Solutions Architect at AWS. He is a containers enthusiast and enjoys helping customers implement cost-effective microservices architectures that scale.




Belle Guttman manages a team of Solutions Architects. She works with startups and small businesses to help them adopt the cloud in the most scalable, secure and cost-effective ways.

from AWS Management Tools Blog

Tracking software licenses with AWS Service Catalog and AWS Step Functions

Tracking software licenses with AWS Service Catalog and AWS Step Functions

Enterprises have many business requirements for tracking how software product licenses are used in their organization for financial, governance, and compliance reasons. By tracking license usage, organizations can stay within budget, track expenditures, and avoid unplanned true-up bills from their vendors’ true-up processes. The goal is to track the usage licenses as resources are deployed.

In this post, you learn how to use AWS Service Catalog to deploy services and applications while tracking the licenses being consumed by end users, and how to prevent license overruns on AWS.

This solution uses the following AWS services. Most of the resources are set up for you with an AWS CloudFormation stack:


Here are some of AWS Service Catalog concepts referenced in this post. For more information, see Overview of AWS Service Catalog.

  • A product is a blueprint for building the AWS resources to make available for deployment on AWS, along with the configuration information. Create a product by importing an AWS CloudFormation template, or, in case of AWS Marketplace-based products, by copying the product to AWS Service Catalog. A product can belong to multiple portfolios.
  • portfolio is a collection of products, together with the configuration information. Use portfolios to manage user access to specific products. You can grant portfolio access for an AWS Identity and Access Management (IAM) user, IAM group, or IAM role level.
  • provisioned product is an AWS CloudFormation stack; that is, the AWS resources that are created. When an end-user launches a product, AWS Service Catalog provisions the product from an AWS CloudFormation stack.
  • Constraints control the way that users can deploy a product. With launch constraints, you can specify a role that AWS

Solution overview­

The following diagram maps out the solution architecture.


Here’s the process for the license administrator:

  1. The cloud administrator uses the AWS Service Catalog product called License Mgmt Setup, created automatically using the setup scripts provided as part of this post.
  2. As part of the deployment process, AWS CloudFormation attaches the license to the AWS Service Catalog product as a tag option.
  3. Amazon CloudWatch detects that the License Mgmt Setup product is being deployed and triggers a Lambda function.
  4. The Lambda function stores the information about the license and the product to which it is attached in a DynamoDB table.

Here’s the process when the end-user launches a product:

  1. The end-user selects and launches an AWS Service Catalog product to which a license was attached by the license administrator.
  2. Behind the scene, invisible to the end-user CloudWatch detects that the product being deployed has a license attached and launches a Step Functions process to track and monitor the license.
  3. The Step Functions process checks the database to determine how many licenses have been used. It then compares the number of used licenses against the setting for notifying the administrator when licenses are low or if too many licenses have been used.
  4. When a particular product has reached its license limit, the product is deactivated.
  5. Notification is sent to the admin using Amazon SES that the product has been deactivated.

Step 1: Configuring an environment

I used the following steps to set up the components for this walkthrough. Use an Amazon S3 bucket to upload your Lambda functions.

To get the setup material:

  1. Download the sc_lic_mgr.zip file with the configuration content.
  2. Unzip the contents and save them to a folder. Note the folder’s location.

To create your S3 bucket:

  1. Log in to your AWS account as an administrator. Ensure that you have an AdministratorAccess IAM policy attached to your login because you’re going to create AWS resources, including IAM roles and users.
  2. In the Amazon S3 console, create a bucket. Leave the default values except as noted.
    • For Bucket name, enter sclicmgr-<accountNumber>.

To upload content to the new bucket:

  1. Select your bucket, and choose Upload, Add files.
  2. Navigate to the folder that contains the configuration content. Select all the files and choose Open. Leave the default values except as noted.
  3. After the Review page, from the list of files, select the sc_lic_managment_setup.json file.
  4. Right-click the link under Object URL and choose Copy link address.

To launch the configuration stack:

  1. In the AWS CloudFormation console, choose Create Stack, Amazon S3 URL, paste the URL you just copied, and then choose Next.
  2. On the Specify stack details page, specify the following:
    • Stack name: sclicmgrSetup-<accountNumber>
    • S3Bucket: sclicmgrSetup-<accountNumber>
    •  SCEndUser: The current user name
  3. Leave the default values except as noted.
  4. On the Review page, check the box next to I acknowledge that AWS CloudFormation might create IAM resources with custom names, and choose Create.
  5. After the status of the stack changes to CREATE COMPLETE, select the stack and choose Outputs to see the output.
  6. Find the SCPrivBucket entry and copy the value. It has the format prod-xxxxx.

The SCPrivBucket is a sample AWS Service Catalog product that you use for your first test.

Congratulations! You have completed the setup. Now, test it by following the license setup process.

Step 2: Setting up a new license

In an organization, typically an administrator determines that a license must be tracked and gathers the following information:

  • The email address of the person to manage the license.
  • The license name.
  • The notification percentage (if usage goes above this percentage, a notification is sent to the license manager).
  • The deactivate percentage (if usage goes above this percentage, the product is deactivated).
  • The number of licenses to start with.
  • The AWS Service Catalog product ID of the product to consume the license.

Step 3: Assigning a new license

As an administrator, use the following steps to assign a new license.

  1. Log into the AWS Service Catalog console as an admin user.
  2. On the left navigation pane, choose Product list, LicenseMgtSetup, and Launch product.
  3. On the Launch page, for By name, enter a name and choose Next.
  4. On the Parameters page, enter the following values, and choose Next:
    • ProductId—The value that you copied in step 2.
    • OwnerEmail—The email address of the license administrator.
    • LicenseName—The name of the license (for example, serverless2).
    • PercentNotify—30 (A notification is sent if usage goes above 30%.)
    • PercentDisable —90 (The product is removed if usage goes above 90%.)
    • TotalLic—3 (The total number of licenses to track.)
  5. Leave the remaining default values.
  6. On the Review page, choose Launch.

After the product is launched, a CloudWatch process attaches the license name to the AWS Service Catalog product as a tag option, and then adds the license information in a DynamoDB table.

Step 4: Provisioning an end-user resource (license consumption)

To launch an AWS Service Catalog product to consume your first license:

  1. In the AWS Service Catalog console, in the left navigation pane, choose Products list.
  2. On the Products list page, point to the stack of dots to the left of your product name and choose Launch product.
  3. On the Product Version page, for Name, enter testdeployment, and choose Next.
  4. On the Parameters page, choose Next.
  5. On the TagOptions page, make sure that the LIC_Name tag is assigned, and choose Next.
  6. On the Review page, choose Launch.
  7. The product has now been successfully deployed, and one license has been used. You should see a screen like the one in the following screenshot.

Congratulations, you have deployed and used the first license.

The deployment process is initiated. A CloudWatch rule detects that the product being launched has an attached license to be tracked and increments the number of licenses being used. The process calculates the total license percentage being used, then compares the percentage being used to the percent to be notified and percent for the product to be deactivated:

  • If the percent used is greater than the percent to be notified, the administrator is notified.
  • If the percent used is greater than the percent to deactivate the product, the AWS Service Catalog product is deactivated.  If the AWS Service Catalog product is terminated, the process recalculates the total license percentage being used.
  • If the percent used is less than the percent to deactivate the product, the AWS Service Catalog product is reactivated.

Creating new licenses for other products

Follow these steps to create new licenses for other products:

  1. To get other product IDs, AWS Service Catalog console as an admin user.
  2. In the left navigation pane, choose Portfolios list.
  3. On the Local Portfolios page, in the Name column, select a portfolio. You see the product names and IDs.
  4. Copy the ID of the product to which to attach the license. Now that you have the product ID, repeat step 2 to add the license to other AWS Service Catalog products.

Cleanup process.

To avoid incurring cost, please delete resources that are not needed.  You can terminate the Service Catalog product deployed the by selecting Action then Terminate.


In this post, you learned an easy way to track your license use on AWS with AWS Service Catalog. You also saw how there’s an extra layer of governance and control when you use AWS Service Catalog to deploy resources to support business objectives.

About the Author

Kenneth Walsh is a New York-based Solutions Architect focusing on AWS Marketplace. Kenneth is passionate about cloud computing and loves being a trusted advisor for his customers.  When he’s not working with customers on their journey to the cloud, he enjoys cooking, audio books, movies, and spending time with his family.


from AWS Management Tools Blog

How DocuTAP automates cloud operations using AWS Management Tools

How DocuTAP automates cloud operations using AWS Management Tools

Now that large organizations have the flexibility to quickly launch infrastructure and leverage new services, they must find the means to maintain consistent controls without restricting development velocity.

In this guest post, Brian Olson, Cloud Architect at health care company DocuTAP, discusses how a combination of AWS Management and Governance services and AWS developer tools allows a small, agile team of engineers to programmatically provision, monitor, and automate all the components of their cloud environment.


DocuTAP is an on-demand focused health care company. Our main two applications, Practice Management (PM) and Electronic Medical Records (EMR), run on Windows. DocuTAP has over a thousand clinics running our software, powered by several hundred Amazon EC2 instances.

Our engineering team has fewer than 10 members, including DBAs, DevOps engineers, and system administrator.  It’s impractical for this small team to manage hundreds of EC2 instances by hand. In fact, no one on the engineering team has even logged into all of those servers.

In this blog post, I’m going to walk you through our solution for scaling configuration management in a Windows environment. Our solution consists of source control, a pipeline for delivering changes, several AWS services (AWS CodePipeline, AWS CodeBuild, AWS CodeDeploy, and AWS CloudFormation), some Windows PowerShell tools and techniques (Modules, Desired State Configuration, and Pester for unit tests), and the idea of paranoid defensive programming.

The Pipeline

We use a continuous integration (CI), continuous delivery (CD), and Git workflow pipeline. You can use source control to track changes to your automation, and every time you commit a change, the pipeline kicks off automated tests and starts moving that change toward production.


At DocuTAP we built out this pipeline with a combination of GitHub and several AWS services, as shown in the following diagram:

Infrastructure Deployment: CloudFormation

We’re invested in CloudFormation at DocuTAP. We use the YAML configuration language because it’s intuitive enough for developers and infrastructure engineers to collaborate on. The CloudFormation service usually does a good job of managing updates to resources, and CloudFormation Change Sets allow you to preview the changes CloudFormation is going to make before you turn it loose.

At DocuTAP we decompose CloudFormation templates into a several distinct types.

Pipeline stack
The pipeline stack holds our CodePipeline setup, CodeBuild jobs, and a reference to the GitHub repository. This is the only stack we deploy manually, so we keep it as small as possible.

Dependency (DEP) stack
The DEP stack holds things that apply to an entire environment of the application like IAM roles, CodeDeploy groups, load balancers, and general AWS Systems Manager State Manager associations that run patch scans and server inventory. This stack groups things that are painful and time consuming to delete and recreate.

Instances (INST) stack
Our instances stacks have the EC2 instances, security groups that are not referenced by other stacks or services, and Auto Scaling policies. These stacks group things that are meant to be ephemeral. These are things that, when they break, you’re likely to say, “Delete it and try again.”

Systems Manager stacks
These hold State Manager associations that run instance-specific configuration. We keep these broken out so it’s easy to connect Run Command documents to the stacks they’re created by.

PowerShell Modules
Here’s a diagram of a Systems Manager State Manager run:

We keep our Systems Manager Run Command documents as thin as possible. These documents have to contain some PowerShell to kick off a config run, but it’s difficult to unit test or debug PowerShell wrapped in YAML. The less they do the better.

We break apart our PowerShell scripts into modules that expose common functions and give us some organization to group common features together.

As you start to break apart your scripts into modules, managing module dependencies becomes very important. Here are some general guidelines:

  1. Keep your module dependencies going in one direction (two modules should never import each other).
  2. Group code into layers (e.g., Windows, application type, server type).
  3. Document, document, document what’s in each module and when and how to use it.

For example, at DocuTAP we have a Windows module, Electronic Medical Records (EMR) and Practice Management (PM) module, application server, and database server modules that interact like this:

Configuration Management and Idempotency: Defensive, paranoid programming

The PowerShell code that we maintain in these repositories runs every half hour to check configuration and apply changes if configuration has drifted.

Some of these changes could interrupt production workloads, like renaming the server and triggering a reboot, or updating directory permissions.

That means a couple of things: every script must be idempotent (meaning you can run the script multiple times and always get the same result), every change must be tested and retested, and good logging is very important.

Other than some simple PowerShell DSCs, most idempotency is the responsibility of the developer building the script. Developers need to make sure they have reasonable “happy state” checks to see if they need to make changes, and good safety checks before and after they apply changes.

In our example, we have a test to make sure the server name needs to be updated:

$tags = Get-EC2Tag -Filter @{Name = "resource-id"; Value = Get-MyInstanceId}
$serverName = ($tags | Where-object {$_.Key -eq "Name"}).value
if ($serverName -ne $env:computername) {
    write-log -message "Computer name does not match host name! Renaming to match, this will trigger a reboot!" -level "WARNING" -category "Windows" -item "hostname";
    rename-computer -newname "$serverName" -force -restart

And on any real server you would want an extra test before triggering the reboot to make sure your server can be safely rebooted. In our case, the most relevant check is for logged-on users.

The ideal format for a configuration management script or function is to do the following:

  1. Test if we need to make a change, and log if we do.
  2. Test if it’s safe to make a change, and log if it is.
  3. Make the change.
  4. Test that the change worked, and log the result.

Having three tests or “safety checks” for every actual change you might make may seem excessive, but these scripts are going to be running unattended. If something goes wrong the more logging you have to look into the problem the better off you’ll be.

Continuous Integration with CodeBuild: Unit tests

We run unit tests on our PowerShell code with a PowerShell module called Pester.

Lots of languages have unit testing frameworks, but here are some quick tips for writing PowerShell unit tests:

  1. Keep your PowerShell functions as simple as possible:
    1. Limit the number of inputs.
    2. Limit the number of outputs, or Cmdlets your functions call.
    3. Limit the amount of branching inside of a function.
  2. Use functions to abstract over other management SDKs:
    1. We use the Citrix PowerShell SDK a lot. Instead of installing it everywhere we need unit tests, we have a wrapper module we can mock in our unit tests.
    2. Some built-in PowerShell functions are difficult to mock (for example, Rename-Computer).
  3. Document, document, document:
    1. Document your code so it’s clear what you’re testing for, and why the results are valid.

Let’s walk through a quick example from the GitHub repo.

Inside of CommonFunctions.psm1 we have a function for getting the instance ID for the server your code is running on. This function is pretty small, and looks like this:

function Get-MyInstanceId {
    if($global:instanceid -eq $null) {
        try {
            $global:instanceID = Invoke-RestMethod ""
        } catch {
            invoke-bomb "Could not get instanceid from ec2 meta data" 99;
    write-host $global:instanceid;
    return $global:instanceid;

In this example invoke-bomb is a function that performs graceful error handling.

This function depends on EC2 metadata, which wouldn’t be available on my laptop. However, I’d still like to make sure it will return the right value when EC2 metadata is available. To do this I add a Pester test that mocks Invoke-RestMethod that looks like this in another .ps1 file that sits in the “tests” directory:

It "Returns the meta data result for the instance name" {
    Mock Invoke-RestMethod {return "instance_name"} -modulename CommonFunctions -Verifiable;
    Get-MyInstanceId | Should Be "instance_name"

This is a trivial example, but you get the idea.

One last point on unit tests is to be pragmatic about them. The more unit test code you have, the more unit test code you have to maintain. Only add unit tests when they clearly add value, because testing the same result repeatedly doesn’t buy you anything but more maintenance effort.

Continuous Integration with CodeBuild: PowerShell DSCs

The last step we have for CodeBuild is to output our MOF files for our PowerShell DSCs. If you’re not familiar with PowerShell DSC, there’s a good overview here.

DSC is a good fit when you can use a built-in DSC resource for creating files or adding Windows features, or you have a very small configuration, like installing chocolatey. If you find yourself writing complex TestScripts (the script DSC runs to decide if it needs to update configuration), you’re probably better off writing a function in a PowerShell module that will be easier to unit test.

Here’s an example of our custom script to install chocolatey:

Script InstallChoco {
  GetScript = {
      $result = Test-Path C:\ProgramData\chocolatey\choco.exe;
      return @{Result = $result}
  SetScript= {
      Set-ExecutionPolicy Bypass -Scope Process -Force; Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
  TestScript = {
      Test-Path C:\ProgramData\chocolatey\choco.exe

CodeBuild builds the MOF by importing the PowerShell module and calling the name of the DSC resource:

Import-Module .\DscHsWindowsManage.psm1;
Remove-Module DscHsWindowsManage;

This gets pushed by CodeDeploy later on, but we’ll cover that in more detail later.

Configuration Run

All of that works gets us to running instances, using IAM profiles. We use AWS CodeDeploy to push our PowerShell management modules out to these instances. In our example, we’re just renaming the server to match the EC2 Name tag, but we do more complex things like setting up Citrix applications, configuring directory permissions, installing applications, etc.

Once the modules are on the server we use a simple Run Command document to import those modules and call the functions inside of them. Here’s a high level overview of the config run:

Common configurations like scanning for patches and configuring the CloudWatch Logs agent run once a day.


We built out a logging function that prepends a time stamp, formats the log messages as JSON (so they’re easier to search with CloudWatch Logs), and outputs them to a file that rolls over regularly.

function write-log([string]$message, [string]$level = "INFO", [string]$category = "general", [string]$item = "NA") {
    $logLine = "$((get-date).ToUniversalTime().toString("yyyy-MM-dd HH:mm:ss")) {`"computername`": `"$($env:computername)`", `"level`": `"$($level)`", `"category`": `"$($category)`", `"item`": `"$($item)`", `"message`": `"$($message)`"}"

    # Dump the log message into std out
    write-host $logLine

    $logFile = "$($global:logDir)\$((get-date).toString("yyyyMM")).log"

    $logLine | out-file -encoding 'UTF8' -append -filepath $logFile

These logs get routed to CloudWatch Logs by the CloudWatch Logs agent so we can view them there.

Here’s a snippet of our CloudWatch Logs agent config that gets applied by a Systems Manager State Manager association:

    "Id": "SSMConfigLogs",
    "FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
    "Parameters": {
        "LogDirectoryPath": "C:\\Docutap\\SSMLogs\\",
        "TimestampFormat": "yyyy-MM-dd HH:mm:ss",
        "Encoding": "UTF-8",
        "Filter": "",
        "CultureName": "en-US",
        "TimeZoneKind": "UTC",
        "LineCount": "1"

Because we format our logs as JSON we can use CloudWatch Logs Insights to filter and search our logs using the Insights query language.

For example, here’s a query looking for the most recent application versions that have been installed on one of our worker servers.


AWS is how!

About the Authors

Brian Olson is a Cloud Architect at DocuTAP Inc. He has a B.S. and an M.S. in Computer Science, and became passionate about DevOps in late 2013 when he took a job as a Sr. Security Analyst and started to work on automating simple steps of security research.

If you’re interested in more content on using PowerShell to interact with AWS and manage Windows servers, he runs a blog at https://rollingwebsphere.blogspot.com/.

Reach out to him on LinkedIn if you’re interested in talking about DevOps or cloud technology!


Eric Westfall is an Enterprise Solutions Architect at AWS. He helps customers in the Education vertical deploy workloads to AWS. In his spare time, he helps customers manage their cloud and hybrid environments using AWS Management Tools.




The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

from AWS Management Tools Blog

Standardizing infrastructure delivery in distributed environments using AWS Service Catalog

Standardizing infrastructure delivery in distributed environments using AWS Service Catalog

A common security design pattern and best practice among many of our enterprise customers is to provide application isolation through the adoption of a multi-account strategy. Many customers choose to create individual AWS accounts based on software development lifecycle (SDLC) phases such as Development (Dev), Quality Assurance (QA), and Production (Prod), to provide complete separation across environments. However, if application requirements aren’t fully understood at time of account creation, it can be difficult to provision the necessary infrastructure components. Additionally, as more accounts are created, customers are looking for a way to enforce infrastructure compliance and consistency across these different accounts.

AWS Service Catalog can help address these challenges and enable developers to quickly, securely, and easily deploy infrastructure components in any environment. The following diagram illustrates this workflow. Here, AWS Service Catalog is used to share both production and non-production infrastructure components to application Dev/QA/Production accounts.

While many customers are familiar with the benefits of using AWS Service Catalog as a “single pane of glass” to provision infrastructure, the deployment of products can also be automated. In the diagram above, products shared with application accounts can be deployed directly from their respective continuous integration/continuous delivery (CI/CD) pipelines. This creates an environment where developers can tightly couple code and infrastructure dependencies, with ownership of individual components distributed across separate teams.

This model provides two key benefits:

  1. It allows a centralized team to enforce compliance and standardization by defining and sharing approved versions of infrastructure.
  2. It creates a self-service environment, where application owners can decide for themselves which infrastructure components to use.

The following diagram describes this process in more detail. Here, the definition of infrastructure is handled by the Shared Services team, which creates a catalog of network and compute-based resources that are shared with application accounts. Multiple versions can be shared, with the application owner taking responsibility for determining which component best meets their requirements. These products can then be deployed as part of application CI/CD processes using an AWS service like AWS CodePipeline. Deploying infrastructure in this manner also has a security advantage because application pipeline permissions can then be scoped to the AWS Service Catalog portfolio rather than the underlying AWS services, ensuring least privilege.

Automation for building out a CI/CD pipeline leveraging AWS Service Catalog account portfolio sharing is available in the following GitHub repository under Amazon software license:

GitHub repository –  https://github.com/aws-samples/aws-service-catalog-reference-architectures/tree/master/labs/xacct-pipeline

The repository contains a read-me guide that includes step-by-step instructions and AWS CloudFormation templates required to create a CI/CD pipeline using the AWS CodePipeline service.

About the AWS Service Catalog CI/CD pipeline solution

In this blog post, I’ll show you how to build a CI/CD pipeline that uses infrastructure delivery through AWS Service Catalog. In this scenario, a shared services team has defined a virtual private cloud (VPC) landing zone and they want to make this infrastructure available to application teams for use in their CI/CD pipelines.

This demonstration will cover the following areas:

  1. Provisioning an AWS Service Catalog portfolio that hosts an approved VPC template in a shared services account.
  2. Sharing the VPC product with our application account.
  3. Creating a CI/CD pipeline with AWS CodePipeline.
  4. Deploying the VPC product and an Amazon EC2 instance into our newly built VPC.

The process is described in the following architecture diagram:

How do I deploy the AWS Service Catalog CI/CD pipeline solution?

The Git repository contains all the necessary AWS CloudFormation templates and a read-me guide. Separate CloudFormation templates are provided to configure the required AWS Service Catalog infrastructure in the shared services and application accounts, along with the sample CI/CD pipeline. Detailed instructions on how and where to execute these CloudFormation templates are available in the read-me guide.

The guide will take the user through the following process of building out our reference architecture:

In the shared service account:

  1. Upload the VPC CloudFormation template to a local Amazon S3 bucket.
  2. Create a master AWS Service Catalog portfolio to host our production infrastructure.
  3. Create an AWS Service Catalog product for our VPC infrastructure.
    • Provide product, support, and version details.
    • Reference the VPC template stored in our Amazon S3 bucket.
  4. Share the master AWS Service Catalog portfolio with the application account.

In the application account:

  1. Accept the AWS Service Catalog portfolio share from the shared service account.
  2. Create the following AWS Identity and Access Management (IAM) roles:
    • An IAM role (e.g., PipelineRole) providing the CodePipeline service with the ability to launch our various pipeline steps.
    • An IAM role (e.g., CFNLaunchRole) providing the CloudFormation service with the ability to launch our AWS Service Catalog products.
    • An IAM role (e.g., ProductLaunchRole) providing the AWS Service Catalog service with the ability to deploy products into our local AWS account.
  3. Create a local AWS Service Catalog portfolio for our VPC infrastructure.
    • Associate the IAM CFNLaunchRole to the portfolio, allowing CloudFormation to launch products.
    • Add the VPC product from the shared service AWS Service Catalog to our local catalog.
    • Create a launch constraint and assign the IAM ProductLaunchRole to the portfolio, allowing our AWS Service Catalog account to deploy products to the local AWS application account.
  4. Create a local Amazon S3 bucket that will act as our source code repository for CodePipeline artifacts.
  5. Create a CodePipeline workflow with the following stages:
    • An Amazon S3 source stage that triggers when a file is uploaded to the S3 artifact created earlier.
    • A VPC deployment stage that deploys the AWS Service Catalog VPC product.
      • Reference the VPC template from within the Amazon S3 artifact.
      • Use the IAM CFNLaunchRole to launch the VPC CloudFormation template.
    • An Amazon EC2 deployment stage that deploys an EC2 instance into our newly built VPC:
      • Reference the Amazon EC2 template from within the S3 artifact above.
      • Use the IAM CFNLaunchRole to launch the Amazon EC2 CloudFormation template.
    • Test the AWS Service Catalog automated pipeline.
      • Upload our source code artifact to the S3 bucket created in step 4.
        • A VPC is deployed through AWS Service Catalog.
        • An EC2 instance is launched within this VPC.
      • Wait for the pipeline to finish deploying our network and compute resources.


The architecture used in this blog post describes the process of standardizing infrastructure delivery by invoking shared AWS Service Catalog products from within a simple CI/CD pipeline. Customers will likely choose to enhance this solution to support more complex scenarios, such as populating the master catalog with additional infrastructure components or compiling and deploying custom application code using services such as AWS CodeBuild and AWS CodeDeploy. As you have seen in this blog post, AWS Service Catalog provides customers with a secure and flexible delivery framework.

About the Author

Kristopher Lippe is a Boston-based Enterprise Solutions Architect for AWS. He is a technology enthusiast who enjoys helping customers find innovative solutions to complex business challenges. His core areas of focus are Storage, Networking, and Security. When he’s not working with customers on their journey to the cloud, he enjoys reading, golf, and home renovation projects.



from AWS Management Tools Blog

Packaging to Distribution – Using AWS Systems Manager Distributor to deploy Datadog

Packaging to Distribution – Using AWS Systems Manager Distributor to deploy Datadog

AWS Systems Manager Distributor automates the process of packaging and publishing software to managed Windows and Linux instances across the cloud landscape, as well as to on-premises servers, through a single simplified interface. Customers can now leverage AWS Systems Manager Distributor to package custom software like monitoring agents and security agents, and then distribute them to a large group of instances on a schedule or on-demand.

In this blog post, I’ll describe how to use Systems Manager Distributor to package and publish a monitoring agent, Datadog, across multiple AWS instances. As an example, I’ll describe the distribution of the Datadog agent version 6 to an AWS landscape that consists of Windows and Amazon Linux instances. Datadog is a SaaS-based monitoring and analytics platform for large-scale applications and infrastructure that delivers actionable alerts and powerful visualizations to provide full-stack observability.

The typical process for distributing the software packages to your AWS instances using Systems Manager Distributor involves the following steps.

Step 1: Create a package in Distributor

These are the steps for creating a package in Distributor:

  1. Prepare the compressed zip folders of the Datadog agent
  2. Upload the zip files to an Amazon S3 bucket
  3. Prepare the manifest file
  4. Create the package in the AWS Systems Manager Distributor console

Prepare the compressed zip folder of the Datadog agent

The Distributor package for any software includes two files apart from the actual software binaries. These files are install.ps1 and the uninstall.ps1 for Windows and install.sh and uninstall.sh for Linux. The installation commands for the software are included in the install.ps1/sh files, and the uninstallation commands are included in the uninstall.ps1/sh files.

The Datadog Agent webpage offers instructions on how to install their agent on many popular operating systems and distributions. I’ll describe how to create the Datadog Distributor packages for both Windows and Amazon Linux instances.

Windows instances

To target Windows instances, download the Datadog msi installation file from the URL provided on the Datadog Account settings page. Create a zip file with the installation file and two additional files: install.ps1 and uninstall.ps1. Let’s call this datadog-windows_6.zip.

The install.ps1 file includes the steps you must use deploy the Datadog agent in a silent fashion on your Windows instances using the command line.

You can reference the content of install.ps1 to deploy the Datadog agent as shown in the following code.

In this example, I’ll include the Datadog installation API key directly in the install script. Whenever you’re distributing potentially sensitive data like a key, make sure that you’re evaluating the impact and considering the use of other tools, like AWS Systems Manager Parameter Store and AWS Secrets Manager to secure access where necessary.

msiexec /qn /i datadog-agent-6-latest.amd64.msi APIKEY="<DATADOG APIKEY>" TAGS="Windows"

Based on your customization of the Datadog agent, you might need to include additional parameters in the install.ps1 file.

The uninstall.ps1 file includes the following reference instructions to uninstall the agent.

(Get-WmiObject -Class Win32_Product -Filter "Name='Datadog Agent'").Uninstall()

As an additional security measure to protect the integrity of the files, the checksum value of the zip file is verified during the package installation process. The checksum value is captured in the package manifest file during the package creation. After you create the zip file with the Datadog binaries, install.ps1, and uninstall.ps1 files, you can calculate the SHA256 checksum of the file Datadog-windows_6.zip. On a Windows system, you can use the following PowerShell command.

Get-FileHash -Algorithm SHA256 datadog-windows_6.zip

Linux instances

To target Amazon Linux instances, create a zip file with the Datadog installation file and two additional files: install.sh and uninstall.sh. Download the latest stable Datadog binaries from the yum repository URL provided at the Datadog Agent Upgrade page. Create a zip file with the binaries and the install.sh and uninstall.sh files. Let’s call this datadog-linux_6.zip.

Note: Although the binaries can be pulled during runtime from the Datadog yum repository without downloading the binaries beforehand, we recommend that you create the Distributor installation package zip file with the binaries and all required dependencies. This ensures that the package installation, including the dependent files, is consistent and has the same version across all instances. This makes it easier to maintain and troubleshoot.

A reference install.sh to deploy the Datadog agent is as below

yum install datadog-agent-6.9.0-1.x86_64.rpm --assumeyes
sed 's/^[:space:]*api_key:.*/api_key: <APIKEY>/' /etc/datadog-agent/datadog.yaml.example > /etc/datadog-agent/datadog.yaml
sudo systemctl start datadog-agent

The uninstall.sh file includes the instructions to uninstall the agent

yum remove datadog-agent --assumeyes

On Linux, you can use the below command to obtain the checksum

sha256sum datadog-linux_6.zip

Upload the zip files to an Amazon S3 bucket

Upload the datadog-windows.zip and datadog-linux.zip files to the S3 bucket created for the Distributor package. Configure the permissions on the bucket to have List and Get objects access the user/role that is creating the package in the Distributor.

Prepare the manifest file

A reference manifest file for the Datadog agent package looks like the following:

    "schemaVersion": "2.0",
    "version": "6.0.0",
    "packages": {
        "windows": {
            "_any": {
                "x86_64": {
                    "file": "datadog-windows_6.zip"
        "amazon": {
            "_any": {
                "x86_64": {
                    "file": "datadog-linux_6.zip"
    "files": {
        "datadog-windows_6.zip": {
            "checksums": {
                "sha256": "<SHA256 value of the file datadog-windows_6.zip>"
        "datadog-linux_6.zip": {
            "checksums": {
                "sha256": "<SHA256 value of the file datadog-linux_6.zip>"

Create the package in AWS Systems Manager console

Navigate to the Distributor page in AWS Systems Manager console and create the package. Provide the name of the package and a version associated with the package.

Enter the S3 bucket url where you uploaded the package earlier.

Choose New Manifest in the Manifest section and paste the contents of the manifest file that you prepared earlier.

Choose Create package button to create the Datadog package after completing all the above mentioned steps. Upon successful creation of the package, the package gets listed on the Distributor main page.

Step 2: Set up appropriate permissions on the package

By default, all packages are set to Private, meaning only those with access to the package creator’s AWS account can view package information, update or delete the package and distribute it.

Customers can distribute the package to all the systems managed by the Systems Manager using the Distributor. However, to distribute the package to systems in other AWS accounts managed by the systems manager of that account in the same Region, you can share the Datadog agent package by providing the account number of the target account in the Edit package permission box.

After the package is shared, you can access the package in the target accounts. In the target account, select the Region where the package was originally created at source, navigate to the Systems Manager Distributor page, and select the search filter as Account owner : Equal : All.

Step 3: Distribute the package

The following preparation steps are required before you can distribute the package to the target instances.

  • Ensure that the version of the Systems Manager agent running on the instances is or later. To ensure the latest version of the Systems Manager agent is installed, run the AWS-UpdateSSMAgent document on all the targeted instances.

After your readiness is complete, you are ready to distribute the Datadog package. In the Systems Manager console, navigate to the datadog-agent package in the Distributor and select either Install one time or Install on schedule.

Install one time

When you choose Install one time, this takes you through the Run Command page of the Systems Manager console. In the command parameters, the Install action is selected by default for the ‘datadog-agent’. In the Version field, by default, the latest published version of the package is picked up for installation when the version field is left blank. Alternately, you can choose a specific version of the package to be installed as the following screenshot shows.

For Targets, select the instances using a tag. You can also manually select the instances. Then choose Run.

This will start distributing the package to the target instances.

Install on schedule

When you choose Install on schedule, the State Manager Create Association page opens. Provide an appropriate name for the association.

The Document section includes the ‘AWS-ConfigureAWSPackage’ document by default.

In Parameters, Action is set to the default, Install. Name defaults to the package name. In the Version field, by default, the latest published version of the package is picked up for installation when the version field is left blank. Alternately, you can choose a specific version of the package to be installed.

In the Targets section, choose the instances either through a tag or by manually selecting the instances. In the Specify schedule section, choose the schedule for running the association.

Optionally, in Advanced options, choose the Compliance severity, specify the Concurrency and Error Threshold in the Rate Control pane and Write to S3 in the Output Options window. Choose Create Association to kick-start the package distribution on schedule.

Step 4: Compliance Status

You can review the distribution status of the Datadog agent through the Run Command command history if the package is distributed using Install one time. You can review status in the State Manager association status if the package is distributed using the Install on schedule option.

If you used State Manager, you can also see the status of your package distribution through the Compliance console. The following screenshot shows the compliance summary.

Managing Package Versions

Managing changes and updates is part of the lifecycle of any software package. AWS Systems Manager Distributor provides the ability to manage this process.

If you want to make any changes to a package, like updating the version of the Datadog agent, adding support for a new operating system, or modifying the install or uninstall scripts, you can add a new version of the package. Once you’ve done that, both the old and new versions are available to install or uninstall on your instances.

To create a new version of the Datadog Distributor package, navigate to the datadog-agent package in the Distributor, and on the Versions tab, choose Add version.

This opens up the package version page with the Version name, S3 Bucket URL, and the Manifest file fields to capture the new package version specifics.

After you provide the details, choose Add version to add a new version to the package. The new version will appear in the Versions tab of the datadog-agent package.

Although all versions are available for you to distribute, you can mark one version as the default so that you can easily keep all of your instances up to date. To make a specific version the default, select the version of the package and choose the Set default version option. If you need to do a rollback, you’ll follow the same process of selecting a specific older version and distributing it, which will override the newer version on the managed instances.


In this blog post I covered the process of creating software packages and distributing them using AWS Systems Manager Distributor with Datadog agent as an example. I started by showing the overall process and steps involved. First, I created the Datadog software package through the Distributor. Then, I showed you how to configure the permissions on the package, distribute the package to the managed instances, and check the status of the distribution. Finally, I showed you how to create multiple versions of the package in line with software updates or changes to the installation commands.

About the author

Chandra Allaka is a Senior Consultant with AWS Professional Services. He is passionate about helping customers with their cloud production readiness and building effective solutions to manage cloud environments at scale.

from AWS Management Tools Blog

Using AWS License Manager to track your Microsoft SQL Server licenses

Using AWS License Manager to track your Microsoft SQL Server licenses

AWS License Manager can manage any software licenses that are based on vCPU, physical cores, physical sockets, or number of instances. In this blog post, we’ll walk through a common use case of using AWS License Manager to help you manage your SQL Server licenses.

Step 1: Launch License Manager
You can access License Manager through the AWS Management Console under Management & Governance. Go to Services and search for License Manager. In addition to the console, you could use the AWS SDKs and the AWS CLI to work with License Manager.


Step 2: Enter your SQL Server licensing rules into AWS License Manager
In this step, you’ll enter into License Manager the quantity of Microsoft SQL Server licenses that you own. Open AWS License Manager and choose Start using AWS License Manager.


  • Grant AWS Identity and Access Management (IAM) permissions to AWS License Manager.


  • Choose License configurations in the left navigation pane, and then choose Create license configuration.


  • You will need the following details for your Microsoft SQL Server licensing.
    • License configuration name: Identifies your license configuration and its resource associations. Enter a descriptive name (e.g., Microsoft SQL Server Enterprise).
    • Description (optional): Brief information about the license configuration. Enter a descriptive name (e.g., Microsoft SQL Server Enterprise).
    • License type: Vendors license their software by vCPUs, cores, sockets, or instances. For example, if you have licensed Microsoft SQL Server by vCPU then select vCPUs.
    • Number of <<License Type>> (optional): Indicate the number of licenses that you own. License Manager uses this information to help you manage the number of licenses used by your organization.
    • Enforce license Limit: Check this check box to limit licensing overage based on the number of license type. License Manager blocks instance launches when you enforce a limit. An example of this enforcement capability is if a user attempts to create an Amazon EC2 instance that uses Microsoft SQL Server and this new instance would put you over your license count, then the EC2 instance will be blocked from being created.


  • Rules (optional): In the Rules area you can specify additional licensing constraints. For example, a software vendor might have a special rule that defines the minimum number of vCPUs to be 4. This section allows you to keep track of these rules. The following chart outlines some of the available rules.

  • Tags (optional): You can tag these license rules with AWS tags.
  • After you choose Create license configuration, the system creates the license configuration rule.


Step 3: Enable automatic tracking of any new Microsoft SQL Servers
You can configure License Manager to automatically track the creation of new servers that consume SQL Server licenses. This process is transparent to your users. When your users create a new EC2 instance running Microsoft SQL Server, License Manager tracks the SQL Server license usage. There are multiple ways to set up this configuration. We are going to walk through the process of using Amazon Machine Images (AMI) to perform this automatic tracking. An AMI is a template of an EC2 instance. Read more details in the AMI documentation.


Associating the SQL Server license rule with an AMI

In this scenario, License Manager will track the SQL Server licenses used when you create EC2 instances running Microsoft SQL Server from an AMI. To enable License Manager to do this tracking, you associate the Microsoft SQL Server AMI with the SQL Server license rule. Then, every time that a new EC2 instance is created from that AMI, the SQL Server licenses are tracked.

License Manager also provides an Enforce option. With this option, the EC2 launch operation would check the number of licenses that you have and if the new instance that is being created would exceed your available licenses, then the EC2 Launch Wizard would not create the instance.

To set this configuration, perform the following steps:

  • Select the licensing rule that you created earlier (in this example it’s Microsoft SQL Server Enterprise). Choose the Actions menu and select Associate AMI.


  • Select the AMI that has Microsoft SQL Server and then choose the Associate button.


  • Now, any time that an instance is created using that AMI, License Manager will keep track of the number of Microsoft SQL server licenses used by that instance. If the Enforce option on the Licensing Rule was selected, then the EC2 instance creation process will check if you have enough licenses before proceeding with the instance creation. If there aren’t enough licenses, then the instance creation process will stop.


Step 4: Capturing the usage of your current SQL Server licenses
The next step is capture the number of Microsoft SQL Server licenses that you are currently using. For instances running in AWS, License Manager communicates with AWS Systems Manager to determine the software running on the EC2 instances. If you want to inventory the servers running on-premises, then you need to install the Systems Manager Agent on those servers. Read more details in the Systems Manager documentation.

To collect your current SQL Server license usage, perform the following steps:

  • Select Search Inventory in the left side menu. In the search box, select Application name. Then, select the Begins with qualifier. This step will find all instances running Microsoft SQL Server.


  • To associate the instance with the SQL Server license rule, choose the check box next to the instance and then choose Associate License Configuration.


  • In the next window, select the license configuration rule (in our example it’s Microsoft SQL Server Enterprise), and then choose Associate.


Licensing Manager reporting and alerting
License Manager provides you a dashboard of your SQL Server license usage.

  • Go to the AWS License Manager console and choose Dashboard on the left navigation pane. The Dashboard indicates the number of license configurations tracked and any usage limit alerts. In the following screenshot, License Manager highlighted one usage limit alert.


  • By choosing View exceeded license configurations, we discover that in this example, we are over-subscribed on our SQL Server license usage.


  • If you want to receive an alert when you are over-subscribed, you can set up alerting. On the left navigation pane, choose Settings.


  • On the Settings page, specify an Amazon Simple Notification Service (SNS) topic that License Manager will use to send messages to your team. You will need to create this SNS topic before choosing it here.


Thanks for joining us as we walked through how to use AWS License Manager to manage your SQL Server licenses. The steps described in this blog focus on managing Microsoft SQL Server licenses, but the same steps can be used to manage any software licenses.

In future posts, we will be covering:

  • Using License Manager to track software usage across multiple AWS accounts
  • Using License Manager to track software usage across hybrid (on-premises and cloud) scenarios


About the author:

Dean Suzuki is a Senior Architect focused on helping customers run Microsoft Workloads in AWS. Dean has 20+ years of experience working with Microsoft technologies and enjoys helping customers gain the benefits of running their workloads in the cloud.

from AWS Management Tools Blog

How Twitch monitors its services with Amazon CloudWatch

How Twitch monitors its services with Amazon CloudWatch

by Alex Cioc and Steve McCurry

Twitch is the leading service and community for multiplayer entertainment and is owned by Amazon. Twitch also provides social and features and micro-transaction features that drive content engagement for its audiences. These services operate at a high transaction volume.

Twitch uses Amazon CloudWatch to monitor its business-critical services. It emits custom metrics then visualizes and alerts based on predefined thresholds for these key metrics. The high volume of transactions handled by the Twitch services makes it difficult to design a metric ingestion strategy that provides sufficient throughput of data while balancing the cost of data ingestion.

Amazon CloudWatch client-side aggregations is a new feature of the PutMetricData API service that helps customers to aggregate data on the client-side, which increases throughput and efficiency. In this blog post we’ll show you how Twitch uses client-side data aggregations to build a more effective metric ingestion architecture while achieving substantial cost reductions.

CloudWatch custom metrics overview

Amazon CloudWatch enables customers to publish their own system or application metrics as custom metrics. The recommended approach to publishing custom metrics is to deploy the CloudWatch agent on your hosts. The CloudWatch agent automatically aggregates data on the client-side and sends data to the CloudWatch PutMetricData API service.

Previously, PutMetricData only supported publishing a single data point per metric per API call. Now, you can use the client-side aggregations feature to publish arrays of values and counts (or histograms) to PutMetricData. With client-side aggregations you can send more data at lower cost, reducing the number of PutMetricData API calls. This results in lower resource utilization on your hosts.

DevOps at Twitch

Twitch has a centralized tools team that develops and manages the Twitch Telemetry platform and drives operational excellence across the organization. Twitch Telemetry provides a collection of libraries for collecting, aggregating, sending, and utilizing metrics that can be leveraged by service teams at Twitch to monitor their services.

The Twitch Rewards and Twitch Emotes services are two of the DevOps teams that use Twitch Telemetry to monitor their APIs and emit their metrics to Amazon CloudWatch.

  • Twitch Rewards enables viewers of broadcasts like Overwatch League to cheer for players using Bits. The Rewards service can experience up to 1500 transactions per second (TPS).
  • Twitch Emotes displays emoticons on social channel pages. The Emotes service experiences daily peaks around 10,000 TPS.

How Twitch approaches monitoring

Before Twitch Telemetry, individual services used custom tools for sending metrics to Amazon CloudWatch. One of biggest monitoring challenges Twitch services faced was efficiently scaling their metric publishing tools. As a workaround, services only published a small percentage of available data observations. This sampling approach reduced calls to the monitoring system but resulted in data loss that created less accurate metrics.

Twitch Telemetry provides agent libraries that service teams can easily integrate with their API based services to send custom metrics to Amazon CloudWatch. Services like Twitch Emote and Twitch Rewards have adopted Twitch Telemetry. Twitch Telemetry batches metric data before sending to Amazon CloudWatch.

The key metrics that the DevOps teams at Twitch monitor are latency, errors, and throttling. Twitch DevOps engineers monitor the percentile statistic of the metrics which is calculated automatically by CloudWatch. The key benefit of percentile statistics is that the percentile removes the noise of outliers in the data, providing a clearer signal. This is particularly useful when the metric sample count over a period of time is high (for example, data observations for a high-volume API service).

In CloudWatch, the sample count and percentile statistics can be selected under the statistic option of the Graphed metrics tab:

Implementing Amazon CloudWatch client-side aggregations

The Twitch engineering team immediately seized the opportunity to use Amazon CloudWatch client-side aggregations to help scale their metric publishing tooling. Using client-side aggregations, Twitch reduced the number of calls to the CloudWatch PutMetricData API service, allowing them to simultaneously increase the volume of data published whilst reducing resource utilization and costs. By publishing a higher number of data observations per minute, Twitch was able to create more accurate metrics that are better suited to monitoring percentiles.

Twitch Telemetry deploys CloudWatch client-side aggregations in a modified version of the Amazon CloudWatch agent. The agent automatically aggregates data client-side, reducing the number of API calls to CloudWatch PutMetricData.

As a result of switching to client-side aggregations, the Twitch Emotes service went from 200 million PutMetricData calls per day to 2 million calls per day. This represented a 99% cost saving. Previously, the Emotes service was being sampled at a rate of 10 percent. However, after client-side aggregations was deployed the sampling was removed and all data was published to CloudWatch. Another benefit was far lower CPU utilization on the service hosts, which led to greater fleet scalability.

The graphs below show the impact that the migration to client-side aggregations on 11/30 had on the Twitch fleet.

Likewise, the Twitch Rewards service went from around 450 million calls per day to PutMetricData to between 4 million and 5 million calls. This 99% reduction in call volume created proportional drop in API costs. By switching to client-side aggregations sampling was eliminated.

For just these two services, the direct savings to the business over the course of a year was substantial. Also, sampling was eliminated, allowing for more accurate metrics and fleet utilization reductions, allowing greater fleet scalability.


Client-side metric aggregations is a new feature of the CloudWatch PutMetricData API service that allows you to ingest metric data more efficiently. This efficiency can be used to lower the number of API calls to Amazon CloudWatch, reducing resources and costs. Client-side aggregations can be used to monitor high volume APIs by publishing metrics like latency, errors, and throttling. You can use the percentiles statistics calculated by CloudWatch to reduce noise that can lead to false alarms. The CloudWatch agent has client-side aggregations enabled by default, and is the recommended best practice for publishing custom metrics to CloudWatch.

This blog post demonstrated how Twitch monitors its services efficiently by leveraging Amazon CloudWatch client-side aggregations. You can use this feature by deploying the Amazon CloudWatch agent to publish custom metrics for your applications. The CloudWatch agent has client-side aggregations enabled by default and is the most efficient way to publish custom metrics at scale.

For more information, see the Amazon CloudWatch agent documentation, or visit our homepage to learn more about how Amazon CloudWatch can help you achieve monitoring visibility for your cloud resources and applications. Also, there is a demonstration application that publishes data to Amazon CloudWatch using client-side aggregations through the AWS SDK.

It would be great to hear your feedback.

About the Authors

Alex Cioc is a Software Development Engineer for the Builder Platform team at Twitch. He is passionate about metrics and data visualization and leads company-wide telemetry standardization initiatives. Alex has worked on high throughput services at Twitch and Amazon Video and continues focusing on scalability and deployment concerns.

Stephen McCurry is a Senior Product Manager for Amazon CloudWatch based in Dublin, Ireland. He spends his time talking with customers and using feedback to prioritize and develop new products and features. He is a builder and likes to write code to solve customer problems. He’s also an ultra runner and spends much of his spare time in the mountains.

from AWS Management Tools Blog