Tag: AWS News Blog

New – VPC Traffic Mirroring – Capture & Inspect Network Traffic

New – VPC Traffic Mirroring – Capture & Inspect Network Traffic

Running a complex network is not an easy job. In addition to simply keeping it up and running, you need to keep an ever-watchful eye out for unusual traffic patterns or content that could signify a network intrusion, a compromised instance, or some other anomaly.

VPC Traffic Mirroring
Today we are launching VPC Traffic Mirroring. This is a new feature that you can use with your existing Virtual Private Clouds (VPCs) to capture and inspect network traffic at scale. This will allow you to:

Detect Network & Security Anomalies – You can extract traffic of interest from any workload in a VPC and route it to the detection tools of your choice. You can detect and respond to attacks more quickly than is possible with traditional log-based tools.

Gain Operational Insights – You can use VPC Traffic Mirroring to get the network visibility and control that will let you make security decisions that are better informed.

Implement Compliance & Security Controls – You can meet regulatory & compliance requirements that mandate monitoring, logging, and so forth.

Troubleshoot Issues – You can mirror application traffic internally for testing and troubleshooting. You can analyze traffic patterns and proactively locate choke points that will impair the performance of your applications.

You can think of VPC Traffic Mirroring as a “virtual fiber tap” that gives you direct access to the network packets flowing through your VPC. As you will soon see, you can choose to capture all traffic or you can use filters to capture the packets that are of particular interest to you, with an option to limit the number of bytes captured per packet. You can use VPC Traffic Mirroring in a multi-account AWS environment, capturing traffic from VPCs spread across many AWS accounts and then routing it to a central VPC for inspection.

You can mirror traffic from any EC2 instance that is powered by the AWS Nitro system (A1, C5, C5d, M5, M5a, M5d, R5, R5a, R5d, T3, and z1d as I write this).

Getting Started with VPC Traffic Mirroring
Let’s review the key elements of VPC Traffic Mirroring and then set it up:

Mirror Source – An AWS network resource that exists within a particular VPC, and that can be used as the source of traffic. VPC Traffic Mirroring supports the use of Elastic Network Interfaces (ENIs) as mirror sources.

Mirror Target – An ENI or Network Load Balancer that serves as a destination for the mirrored traffic. The target can be in the same AWS account as the Mirror Source, or in a different account for implementation of the central-VPC model that I mentioned above.

Mirror Filter – A specification of the inbound or outbound (with respect to the source) traffic that is to be captured (accepted) or skipped (rejected). The filter can specify a protocol, ranges for the source and destination ports, and CIDR blocks for the source and destination. Rules are numbered, and processed in order within the scope of a particular Mirror Session.

Traffic Mirror Session – A connection between a mirror source and target that makes use of a filter. Sessions are numbered, evaluated in order, and the first match (accept or reject) is used to determine the fate of the packet. A given packet is sent to at most one target.

You can set this up using the VPC Console, EC2 CLI, or the EC2 API, with CloudFormation support in the works. I’ll use the Console.

I already have ENI that I will use as my mirror source and destination (in a real-world use case I would probably use an NLB destination):

The MirrorTestENI_Source and MirrorTestENI_Destination ENIs are already attached to suitable EC2 instances. I open the VPC Console and scroll down to the Traffic Mirroring items, then click Mirror Targets:

I click Create traffic mirror target:

I enter a name and description, choose the Network Interface target type, and select my ENI from the menu. I add a Blog tag to my target, as is my practice, and click Create:

My target is created and ready to use:

Now I click Mirror Filters and Create traffic mirror filter. I create a simple filter that captures inbound traffic on three ports (22, 80, and 443), and click Create:

Again, it is created and ready to use in seconds:

Next, I click Mirror Sessions and Create traffic mirror session. I create a session that uses MirrorTestENI_Source, MainTarget, and MyFilter, allow AWS to choose the VXLAN network identifier, and indicate that I want the entire packet mirrored:

And I am all set. Traffic from my mirror source that matches my filter is encapsulated as specified in RFC 7348 and delivered to my mirror target. I can then use tools like Suricata to capture, analyze, and visualize it.

Things to Know
Here are a couple of things to keep in mind:

Sessions Per ENI – You can have up to three active sessions on each ENI.

Cross-VPC – The source and target ENIs can be in distinct VPCs as long as they are peered to each other or connected through Transit Gateway.

Scaling & HA – In most cases you should plan to mirror traffic to a Network Load Balancer and then run your capture & analysis tools on an Auto Scaled fleet of EC2 instances behind it.

Bandwidth – The replicated traffic generated by each instance will count against the overall bandwidth available to the instance. If traffic congestion occurs, mirrored traffic will be dropped first.

Now Available
VPC Traffic Mirroring is available now and you can start using it today in all commercial AWS Regions except Asia Pacific (Sydney), China (Beijing), and China (Ningxia). Support for those regions will be added soon. You pay an hourly fee (starting at $0.015 per hour) for each mirror source; see the VPC Pricing page for more info.

Jeff;

 

from AWS News Blog https://aws.amazon.com/blogs/aws/new-vpc-traffic-mirroring/

AWS Security Hub Now Generally Available

AWS Security Hub Now Generally Available

I’m a developer, or at least that’s what I tell myself while coming to terms with being a manager. I’m definitely not an infosec expert. I’ve been paged more than once in my career because something I wrote or configured caused a security concern. When systems enable frequent deploys and remove gatekeepers for experimentation, sometimes a non-compliant resource is going to sneak by. That’s why I love tools like AWS Security Hub, a service that enables automated compliance checks and aggregated insights from a variety of services. With guardrails like these in place to make sure things stay on track, I can experiment more confidently. And with a single place to view compliance findings from multiple systems, infosec feels better about letting me self-serve.

With cloud computing, we have a shared responsibility model when it comes to compliance and security. AWS handles the security of the cloud: everything from the security of our data centers up to the virtualization layer and host operating system. Customers handle security in the cloud: the guest operating system, configuration of systems, and secure software development practices.

Today, AWS Security Hub is out of preview and available for general use to help you understand the state of your security in the cloud. It works across AWS accounts and integrates with many AWS services and third-party products. You can also use the Security Hub API to create your own integrations.

Getting Started

When you enable AWS Security Hub, permissions are automatically created via IAM service-linked roles. Automated, continuous compliance checks begin right away. Compliance standards determine these compliance checks and rules. The first compliance standard available is the Center for Internet Security (CIS) AWS Foundations Benchmark. We’ll add more standards this year.

The results of these compliance checks are called findings. Each finding tells you severity of the issue, which system reported it, which resources it affects, and a lot of other useful metadata. For example, you might see a finding that lets you know that multi-factor authentication should be enabled for a root account, or that there are credentials that haven’t been used for 90 days that should be revoked.

Findings can be grouped into insights using aggregation statements and filters.

Integrations

In addition to the Compliance standards findings, AWS Security Hub also aggregates and normalizes data from a variety of services. It is a central resource for findings from AWS Guard Duty, Amazon Inspector, Amazon Macie, and from 30 AWS partner security solutions.

AWS Security Hub also supports importing findings from custom or proprietary systems. Findings must be formatted as AWS Security Finding Format JSON objects. Here’s an example of an object I created that meets the minimum requirements for the format. To make it work for your account, switch out the AwsAccountId and the ProductArn. To get your ProductArn for custom findings, replace REGION and ACCOUNT_ID in the following string: arn:aws:securityhub:REGION:ACCOUNT_ID:product/ACCOUNT_ID/default.

{
    "Findings": [{
        "AwsAccountId": "12345678912",
        "CreatedAt": "2019-06-13T22:22:58Z",
        "Description": "This is a custom finding from the API",
        "GeneratorId": "api-test",
        "Id": "us-east-1/12345678912/98aebb2207407c87f51e89943f12b1ef",
        "ProductArn": "arn:aws:securityhub:us-east-1:12345678912:product/12345678912/default",
        "Resources": [{
            "Type": "Other",
            "Id": "i-decafbad"
        }],
        "SchemaVersion": "2018-10-08",
        "Severity": {
            "Product": 2.5,
            "Normalized": 11
        },
        "Title": "Security Finding from Custom Software",
        "Types": [
            "Software and Configuration Checks/Vulnerabilities/CVE"
        ],
        "UpdatedAt": "2019-06-13T22:22:58Z"
    }]
}

Then I wrote a quick node.js script that I named importFindings.js to read this JSON file and send it off to AWS Security Hub via the AWS JavaScript SDK.

const fs    = require('fs');        // For file system interactions
const util  = require('util');      // To wrap fs API with promises
const AWS   = require('aws-sdk');   // Load the AWS SDK

AWS.config.update({region: 'us-east-1'});

// Create our Security Hub client
const sh = new AWS.SecurityHub();

// Wrap readFile so it returns a promise and can be awaited 
const readFile = util.promisify(fs.readFile);

async function getFindings(path) {
    try {
        // wait for the file to be read...
        let fileData = await readFile(path);

        // ...then parse it as JSON and return it
        return JSON.parse(fileData);
    }
    catch (error) {
        console.error(error);
    }
}

async function importFindings() {
    // load the findings from our file
    const findings = await getFindings('./findings.json');

    try {
        // call the AWS Security Hub BatchImportFindings endpoint
        response = await sh.batchImportFindings(findings).promise();
        console.log(response);
    }
    catch (error) {
        console.error(error);
    }
}

// Engage!
importFindings();

A quick run of node importFindings.js results in { FailedCount: 0, SuccessCount: 1, FailedFindings: [] }. And now I can see my custom finding in the Security Hub console:

Custom Actions

AWS Security Hub can integrate with response and remediation workflows through the use of custom actions. With custom actions, a batch of selected findings is used to generate CloudWatch events. With CloudWatch Rules, these events can trigger other actions such as sending notifications via a chat system or paging tool, or sending events to a visualization service.

First, we open Settings from the AWS Security Console, and select Custom Actions. Add a custom action and note the ARN.

Then we create a CloudWatch Rule using the custom action we created as a resource in the event pattern, like this:

{
  "source": [
    "aws.securityhub"
  ],
  "detail-type": [
    "Security Hub Findings - Custom Action"
  ],
  "resources": [
    "arn:aws:securityhub:us-west-2:123456789012:action/custom/DoThing"
  ]
}

Our CloudWatch Rule can have many different kinds of targets, such as Amazon Simple Notification Service (SNS) Topics, Amazon Simple Queue Service (SQS) Queues, and AWS Lambda functions. Once our action and rule are in place, we can select findings, and then choose our action from the Actions dropdown list. This will send the selected findings to Amazon CloudWatch Events. Those events will match our rule, and the event targets will be invoked.

Important Notes

  • AWS Config must be enabled for Security Hub compliance checks to run.
  • AWS Security Hub is available in 15 regions: US East (N. Virginia), US East (Ohio), US West (Oregon), US West (N. California), Canada (Central), South America (São Paulo), Europe (Ireland), Europe (London), Europe (Paris), Europe (Frankfurt), Asia Pacific (Singapore), Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Seoul), and Asia Pacific (Mumbai).
  • AWS Security Hub does not transfer data outside of the regions where it was generated. Data is not consolidated across multiple regions.

AWS Security Hub is already the type of service that I’ll enable on the majority of the AWS accounts I operate. As more compliance standards become available this year, I expect it will become a standard tool in many toolboxes. A 30-day free trial is available so you can try it out and get an estimate of what your costs would be. As always, we want to hear your feedback and understand how you’re using AWS Security Hub. Stay in touch, and happy building!

— Brandon

from AWS News Blog https://aws.amazon.com/blogs/aws/aws-security-hub-now-generally-available/

AWS Control Tower – Set up & Govern a Multi-Account AWS Environment

AWS Control Tower – Set up & Govern a Multi-Account AWS Environment

Earlier this month I met with an enterprise-scale AWS customer. They told me that they are planning to go all-in on AWS, and want to benefit from all that we have learned about setting up and running AWS at scale. In addition to setting up a Cloud Center of Excellence, they want to set up a secure environment for teams to provision development and production accounts in alignment with our recommendations and best practices.

AWS Control Tower
Today we are announcing general availability of AWS Control Tower. This service automates the process of setting up a new baseline multi-account AWS environment that is secure, well-architected, and ready to use. Control Tower incorporates the knowledge that AWS Professional Service has gained over the course of thousands of successful customer engagements, and also draws from the recommendations found in our whitepapers, documentation, the Well-Architected Framework, and training. The guidance offered by Control Tower is opinionated and prescriptive, and is designed to accelerate your cloud journey!

AWS Control Tower builds on multiple AWS services including AWS Organizations, AWS Identity and Access Management (IAM) (including Service Control Policies), AWS Config, AWS CloudTrail, and AWS Service Catalog. You get a unified experience built around a collection of workflows, dashboards, and setup steps. AWS Control Tower automates a landing zone to set up a baseline environment that includes:

  • A multi-account environment using AWS Organizations.
  • Identity management using AWS Single Sign-On (SSO).
  • Federated access to accounts using AWS SSO.
  • Centralize logging from AWS CloudTrail, and AWS Config stored in Amazon S3.
  • Cross-account security audits using AWS IAM and AWS SSO.

Before diving in, let’s review a couple of key Control Tower terms:

Landing Zone – The overall multi-account environment that Control Tower sets up for you, starting from a fresh AWS account.

Guardrails – Automated implementations of policy controls, with a focus on security, compliance, and cost management. Guardrails can be preventive (blocking actions that are deemed as risky), or detective (raising an alert on non-conformant actions).

Blueprints – Well-architected design patterns that are used to set up the Landing Zone.

Environment – An AWS account and the resources within it, configured to run an application. Users make requests (via Service Catalog) for new environments and Control Tower uses automated workflows to provision them.

Using Control Tower
Starting from a brand new AWS account that is both Master Payer and Organization Master, I open the Control Tower Console and click Set up landing zone to get started:

AWS Control Tower will create AWS accounts for log arching and for auditing, and requires email addresses that are not already associated with an AWS account. I enter two addresses, review the information within Service permissions, give Control Tower permission to administer AWS resources and services, and click Set up landing zone:

The setup process runs for about an hour, and provides status updates along the way:

Early in the process, Control Tower sends a handful of email requests to verify ownership of the account, invite the account to participate in AWS SSO, and to subscribe to some SNS topics. The requests contain links that I must click in order for the setup process to proceed. The second email also requests that I create an AWS SSO password for the account. After the setup is complete, AWS Control Tower displays a status report:

The console offers some recommended actions:

At this point, the mandatory guardrails have been applied and the optional guardrails can be enabled:

I can see the Organizational Units (OUs) and accounts, and the compliance status of each one (with respect to the guardrails):

 

Using the Account Factory
The navigation on the left lets me access all of the AWS resources created and managed by Control Tower. Now that my baseline environment is set up, I can click Account factory to provision AWS accounts for my teams, applications, and so forth.

The Account factory displays my network configuration (I’ll show you how to edit it later), and gives me the option to Edit the account factory network configuration or to Provision new account:

I can control the VPC configuration that is used for new accounts, including the regions where VPCs are created when an account is provisioned:

The account factory is published to AWS Service Catalog automatically. I can provision managed accounts as needed, as can the developers in my organization. I click AWS Control Tower Account Factory to proceed:

I review the details and click LAUNCH PRODUCT to provision a new account:

Working with Guardrails
As I mentioned earlier, Control Tower’s guardrails provide guidance that is either Mandatory or Strongly Recommended:

Guardrails are implemented via an IAM Service Control Policy (SCP) or an AWS Config rule, and can be enabled on an OU-by-OU basis:

Now Available
AWS Control Tower is available now and you can start using it today in the US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Ireland) Regions, with more to follow. There is no charge for the Control Tower service; you pay only for the AWS resources that it creates on your behalf.

In addition to adding support for more AWS regions, we are working to allow you to set up a parallel landing zone next to an existing AWS account, and to give you the ability to build and use custom guardrails.

Jeff;

 

from AWS News Blog https://aws.amazon.com/blogs/aws/aws-control-tower-set-up-govern-a-multi-account-aws-environment/

New – UDP Load Balancing for Network Load Balancer

New – UDP Load Balancing for Network Load Balancer

The Network Load Balancer is designed to handle tens of millions of requests per second while maintaining high throughput at ultra low latency, with no effort on your part (read my post, New Network Load Balancer – Effortless Scaling to Millions of Requests per Second to learn more).

In response to customer requests, we have added several new features since the late-2017 launch, including cross-zone load balancing, support for resource-based and tag-based permissions, support for use across an AWS managed VPN tunnel, the ability to create a Network Load Balancer using the AWS Elastic Beanstalk Console, support for Inter-Region VPC Peering, and TLS Termination.

UDP Load Balancing
Today we are adding support for another frequent customer request, the ability to load balance UDP traffic. You can now use Network Load Balancers to deploy connectionless services for online gaming, IoT, streaming, media transfer, and native UDP applications. If you are hosting DNS, SIP, SNMP, Syslog, RADIUS, and other UDP services in your own data center, you can now move the services to AWS. You can also deploy services to handle Authentication, Authorization, and Accounting, often known as AAA.

You no longer need to maintain a fleet of proxy servers to ingest UDP traffic, and you can now use the same load balancer for both TCP and UDP traffic. You can simplify your architecture, reduce your costs, and increase your scalability.

Creating a UDP Network Load Balancer
I can create a Network Load Balancer with UDP support using the Console, CLI (create-load-balancer), API (CreateLoadBalancer), or a CloudFormation template (AWS::ElasticLoadBalancing::LoadBalancer), as usual. The console lets me choose the desired load balancer; I click the Create button underneath Network Load Balancer:

I name my load balancer, choose UDP from the protocol menu, and select a port (514 is for Syslog):

I already have suitable EC2 instances in us-east-1b and us-east-1c so I’ll use those AZs:

Then I set up a target group for the UDP protocol on port 514:

I choose my instances and click Add to registered:

I review my settings on the next page, and my new UDP Load Balancer is ready to accept traffic within a minute or so (the state starts out as provisioning and transitions to active when it is ready):

I’ll test this out by configuring my EC2 instances as centralized Syslogd servers. I simply edit the configuration file (/etc/rsyslog.conf) on the instances to make them listen on port 514, and restart the service:

Then I launch another EC2 instance and configure it to use my NLB endpoint:

And I can see log entries in my servers (ip-172-31-29-40 is my test instance):

I did have to do make one small configuration change in order to get this to work! Using UDP to check on the health of a service does not really make sense, so I clicked override and specified a health check on port 80 instead:

In a real-world scenario you would want to build a TCP-style health check into your service, of course. And, needless to say, I would run a custom implementation of Syslog that stores the log messages centrally and in a highly durable form.

Things to Know
Here are a couple of things to know about this important new NLB feature:

Supported Targets – UDP on Network Load Balancers is supported for Instance target types (IP target types and PrivateLink are not currently supported).

Health Checks – As I mentioned above, health checks must be done using TCP, HTTP, or HTTPS.

Multiple Protocols – A single Network Load Balancer can handle both TCP and UDP traffic. You can add another listener to an existing load balancer to gain UDP support, as long as you use distinct ports. In situations such as DNS where you need support for both TCP and UDP on the same port, you can set up a multi-protocol target group and a multi-protocol listener (use TCP_UDP for the listener type and the TargetGroup).

New CloudWatch Metrics – The existing CloudWatch metrics (ProcessedBytes, ActiveFlowCount, and NewFlowCount) now represent the aggregate traffic processed by the TCP, UDP, and TLS listeners on a given Network Load Balancer.

Available Now
This feature is available now and you can start using it today in all commercial AWS Regions. For pricing, see the Elastic Load Balancing Pricing page.

Jeff;

 

from AWS News Blog https://aws.amazon.com/blogs/aws/new-udp-load-balancing-for-network-load-balancer/

Now Available: New C5 instance sizes and bare metal instances

Now Available: New C5 instance sizes and bare metal instances

Amazon EC2 C5 instances are very popular for running compute-heavy workloads like batch processing, distributed analytics, high-performance computing, machine/deep learning inference, ad serving, highly scalable multiplayer gaming, and video encoding.

Today, we are happy to expand the Amazon EC2 C5 family with:

  • New larger virtualized instance sizes: 12xlarge and 24xlarge,
  • A bare metal option.

The new C5 instance sizes run on Intel’s Second Generation Xeon Scalable processors (code-named Cascade Lake) with sustained all-core turbo frequency of 3.6GHz and maximum single core turbo frequency of 3.9GHz.

The new processors also enable a new feature called Intel Deep Learning Boost, a capability based on the AVX-512 instruction set. Thanks to the new Vector Neural Network Instructions (AVX-512 VNNI), deep learning frameworks will speed up typical machine learning operations like convolution, and automatically improve inference performance over a wide range of workloads.

These instances are also based on the AWS Nitro System, with dedicated hardware accelerators for EBS processing (including crypto operations), the software-defined network inside of each Virtual Private Cloud (VPC), and ENA networking.

New C5 instance sizes: 12xlarge and 24xlarge

Previously, the largest C5 instance available was C5.18xlarge, with 72 logical processors and 144 GiB of memory. As you can see, the new 24xlarge size increases available resources by 33%, in order to scale up and reduce the time required to compute intensive tasks.

Instance Name Logical Processors Memory EBS-Optimized Bandwidth Network Bandwidth
c5.12xlarge 48 96 GiB 7 Gbps 12 Gbps
c5.24xlarge 96 192 GiB 14 Gbps 25 Gbps

Bare metal C5

Just like for existing bare metal instances (M5, M5d, R5, R5d, z1d, and so forth), your operating system runs directly on the underlying hardware with direct access to the processor.

As described in a previous blog post, you can leverage bare metal instances for applications that:

  • do not want to take the performance hit of nested virtualization,
  • need access to physical resources and low-level hardware features, such as performance counters and Intel VT that are not always available or fully supported in virtualized environments,
  • are intended to run directly on the hardware, or licensed and supported for use in non-virtualized environments.

Bare metal instances can also take advantage of Elastic Load Balancing, Auto Scaling, Amazon CloudWatch, and other AWS services.

Instance Name Logical Processors Memory EBS-Optimized Bandwidth Network Bandwidth
c5.metal 96 192 GiB 14 Gbps 25 Gbps

Now Available!

You can start using these new instances today in the following regions: US East (N. Virginia), US West (Oregon), Europe (Ireland), Europe (London), Europe (Frankfurt), Europe (Stockholm), Europe (Paris), Asia Pacific (Singapore), Asia Pacific (Sydney), and AWS GovCloud (US-West).

Please send us feedback and help us build the next generation of compute-optimized instances.

Julien;

from AWS News Blog https://aws.amazon.com/blogs/aws/now-available-new-c5-instance-sizes-and-bare-metal-instances/

Amazon S3 Update – SigV2 Deprecation Period Extended & Modified

Amazon S3 Update – SigV2 Deprecation Period Extended & Modified

Every request that you make to the Amazon S3 API must be signed to ensure that it is authentic. In the early days of AWS we used a signing model that is known as Signature Version 2, or SigV2 for short. Back in 2012, we announced SigV4, a more flexible signing method, and made it the sole signing method for all regions launched after 2013. At that time, we recommended that you use it for all new S3 applications.

Last year we announced that we would be ending support for SigV2 later this month. While many customers have updated their applications (often with nothing more than a simple SDK update), to use SigV4, we have also received many requests for us to extend support.

New Date, New Plan
In response to the feedback on our original plan, we are making an important change. Here’s the summary:

Original Plan – Support for SigV2 ends on June 24, 2019.

Revised Plan – Any new buckets created after June 24, 2020 will not support SigV2 signed requests, although existing buckets will continue to support SigV2 while we work with customers to move off this older request signing method.

Even though you can continue to use SigV2 on existing buckets, and in the subset of AWS regions that support SigV2, I encourage you to migrate to SigV4, gaining some important security and efficiency benefits in the process. The newer signing method uses a separate, specialized signing key that is derived from the long-term AWS access key. The key is specific to the service, region, and date. This provides additional isolation between services and regions, and provides better protection against key reuse. Internally, our SigV4 implementation is able to securely cache the results of authentication checks; this reduces latency and adds to the overall resiliency of your application. To learn more, read Changes in Signature Version 4.

Identifying Use of SigV2
S3 has been around since 2006 and some of the code that you or your predecessors wrote way back then might still be around, dutifully making requests that are signed with SigV2. You can use CloudTrail Data Events or S3 Server Access Logs to find the old-school requests and target the applications for updates:

CloudTrail Data Events – Look for the SignatureVersion element within the additionalDataElement of each CloudTrail event entry (read Using AWS CloudTrail to Identify Amazon S3 Signature Version 2 Requests to learn more).

S3 Server Access Logs – Look for the SignatureVersion element in the logs (read Using Amazon S3 Access Logs to Identify Signature Version 2 Requests to learn more).

Updating to SigV4



“Do we need to change our code?”

The Europe (Frankfurt), US East (Ohio), Canada (Central), Europe (London), Asia Pacific (Seoul), Asia Pacific (Mumbai), Europe (Paris), China (Ningxia), Europe (Stockholm), Asia Pacific (Osaka Local), AWS GovCloud (US-East), and Asia Pacific (Hong Kong) Regions were launched after 2013, and support SigV4 but not SigV2. If you have code that accesses S3 buckets in that region, it is already making exclusive use of SigV4.

If you are using the latest version of the AWS SDKs, you are either ready or just about ready for the SigV4 requirement on new buckets beginning June 24, 2020. If you are using an older SDK, please check out the detailed version list at Moving from Signature Version 2 to Signature Version 4 for more information.

There are a few situations where you will need to make some changes to your code. For example, if you are using pre-signed URLs with the AWS Java, JavaScript (node.js), or Python SDK, you need to set the correct region and signature version in the client configuration. Also, be aware that SigV4 pre-signed URLs are valid for a maximum of 7 days, while SigV2 pre-signed URLs can be created with a maximum expiry time that could be many weeks or years in the future (in almost all cases, using time-limited URLs is a much better practice). Using SigV4 will improve your security profile, but might also mandate a change in the way that you create, store, and use the pre-signed URLs. While using long-lived pre-signed URLs was easy and convenient for developers, using SigV4 with URLs that have a finite expiration is a much better security practice.

If you are using Amazon EMR, you should upgrade your clusters to version 5.22.0 or later so that all requests to S3 are made using SigV4 (see Amazon EMR 5.x Release Versions for more info).

If your S3 objects are fronted by Amazon CloudFront and you are signing your own requests, be sure to update your code to use SigV4. If you are using Origin Access Identities to restrict access to S3, be sure to include the x-amz-content-sha256 header and the proper regional S3 domain endpoint.

We’re Here to Help
The AWS team wants to help make your transition to SigV4 as smooth and painless as possible. If you run in to problems, I strongly encourage you to make use of AWS Support, as described in Getting Started with AWS Support.

You can also Discuss this Post on Reddit!

Jeff;

 

from AWS News Blog https://aws.amazon.com/blogs/aws/amazon-s3-update-sigv2-deprecation-period-extended-modified/

Amazon Personalize is Now Generally Available

Amazon Personalize is Now Generally Available

Today, we’re happy to announce that Amazon Personalize is available to all AWS customers. Announced in preview at AWS re:Invent 2018, Amazon Personalize is a fully-managed service that allows you to create private, customized personalization recommendations for your applications, with little to no machine learning experience required.

Whether it is a timely video recommendation inside an application or a personalized notification email delivered just at the right moment, personalized experiences, based on your data, deliver more relevant experiences for customers often with much higher business returns.

The task of developing an efficient recommender system is quite challenging: building, optimizing, and deploying real-time personalization requires specialized expertise in analytics, applied machine learning, software engineering, and systems operations. Few organizations have the knowledge, skills, and experience to overcome these challenges, and simple rule-based systems become brittle and costly to maintain as new products and promotions are introduced, or customer behavior changes.

For over 20 years, Amazon.com has perfected machine learning models that provide personalized buying experiences from product discovery to checkout. With Amazon Personalize, we are bringing developers that same capability to build custom models without having to deal with the complexity of infrastructure and machine learning that typically accompanies these types of solutions.

With Amazon Personalize, you provide the unique signals in your activity data (page views, signups, purchases, and so forth) along with optional customer demographic information (age, location, etc.). You then provide the inventory of the items you want to recommend, such as articles, products, videos, or music as an example. Then, entirely under the covers, Amazon Personalize will process and examine the data, identify what is meaningful, select the right algorithms, and train and optimize a personalization model that is customized for your data, and accessible via an API. All data analyzed by Amazon Personalize is kept private and secure and only used for your customized recommendations. The resulting models are yours and yours alone.

With a single API call, you can make recommendations for your users and personalize the customer experience, driving more engagement, higher conversion, and increased performance on marketing campaigns. Domino’s Pizza, for instance, is using Amazon Personalize to deliver customized communications such as promotional deals through their digital properties. Sony Interactive Entertainment uses Personalize with Amazon SageMaker to automate and accelerate their machine learning development and drive more effective personalization at scale.

Personalize is like having your own Amazon.com machine learning personalization team at your beck and call, 24 hours a day.

Introducing Amazon Personalize

Amazon Personalize can make recommendations based on your historical data stored in Amazon S3, or on streaming data sent in real-time by your applications, or on both.

This gives customers a lot of flexibility to build recommendation solutions. For instance, you could build an initial recommender based on historical data, and retrain it periodically when you’ve accumulated enough live events. Alternatively, if you have no historical data to start from, you could ingest events for a while, and then build your recommender.

Having covered historical data in my previous blog post, I will focus on ingesting live events this time.

The high-level process looks like this:

  1. Create a dataset group in order to store events sent by your application.
  2. Create an interaction dataset and define its schema (no data is needed at this point).
  3. Create an event tracker in order to send events to Amazon Personalize.
  4. Start sending events to Amazon Personalize.
  5. Select a recommendation recipe, or let Amazon Personalize pick one for you thanks to AutoML.
  6. Create a solution, i.e. train the recipe on your dataset.
  7. Create a campaign and start recommending items.

Creating a dataset group

Let’s say we’d like to capture a click stream of movie recommendations. Using the the first time setup wizard, we create a dataset group to store these events. Here, let’s assume we don’t have any historical data to start with: all events are generated by the click stream, and are ingested using the event ingestion SDK.

Creating a dataset group just requires a name.

Then, we have to create the interaction dataset, which shows how users are interacting with items (liking, clicking, etc.). Of course, we need to define a schema describing the data: here, we’ll simply use the default schema provided by Amazon Personalize.

Optionally, we could now define an import job, in order to add historical data to the data set: as mentioned above, we’ll skip this step as all data will come from the stream.

Configuring the event tracker

The next step is to create the event tracker that will allow us to send streaming events to the dataset group.

After a minute or so, our tracker is ready. Please take note of the tracking id: we’ll need it to send events.

Creating the dataset

When Amazon Personalize creates an event tracker, it automatically creates a new dataset in the dataset group associated with the event tracker. This dataset has a well-defined schema storing the following information:

  • user_id and session_id: these values are defined by your application.
  • tracking_id: the event tracker id.
  • timestamp, item_id, event_type, event_value: these values describe the event itself and must be passed by your application.

Real-time events can be sent to this dataset in two different ways:

  • Server-side, via the AWS SDK: please note ingestion can happen from any source, whether your code is hosted inside of AWS (e.g. in Amazon EC2 or AWS Lambda) or outside.
  • With the AWS Amplify JavaScript library.

Let’s look at both options.

Sending real-time events with the AWS SDK

This is a very easy process: we can simply use the PutEvents API to send either a single event, or a list of up to 10 events. Of course, we could use any of the AWS SDKs: since my favourite language is Python, this is how we can send events using the boto3 SDK.

import boto3
personalize_events = boto3.client('personalize-events')
personalize_events.put_events(
    trackingId = <TRACKING_ID>,
    userId = <USER_ID>,
    sessionId = <SESSION_ID>,
    eventList = [
      {
          "eventId": "event1",
          "sentAt": 1549959198,
          "eventType": "rating",
          "properties": """{\"itemId\": \"123\", \"eventValue\": \"4\"}"""
      },
      {
          "eventId": "event2",
          "sentAt": 1549959205,
          "eventType": "rating",
          "properties": """{\"itemId\": \"456\", \"eventValue\": \"2\"}"""
      }
    ]
)

In our application, we rated movie 123 as a 4, and movie 456 as a 2. Using the appropriate tracking identifier, we sent two Events to our event tracker:

  • eventId: an application-specific identifier.
  • sentAt: a timestamp, matching the timestamp property defined in the schema. This value seconds since the Unix Epoch (1 January 1970 00:00:00.000 UTC), and is independent of any particular time zone.
  • eventType: the event type, matching the event_type property defined in the schema,
  • properties: the item id and event value, matching the item_id and event_value properties defined in the schema.

Here’s a similar code snippet in Java.

List<Event> eventList = new ArrayList<>();
eventList.add(new Event().withProperties(properties).withType(eventType));
PutEventsRequest request = new PutEventsRequest()
  .withTrackingId(<TRACKING_ID>)
  .withUserId(<USER_ID>)
  .withSessionId(<SESSION_ID>)
  .withEventList(eventList);
client.putEvents(request)

You get the idea!

Sending real-time events with AWS Amplify

AWS Amplify is a JavaScript library that makes it easy to create, configure, and implement scalable mobile and web apps powered by AWS. It’s integrated with the event tracking service in Amazon Personalize.

A couple of setup steps are required before we can send events. For the sake of brevity, please refer to these detailed instructions in the Amazon Personalize documentation:

  • Create an identity pool in Amazon Cognito, in order to authenticate users.
  • Configure the Amazon Personalize plug-in with the pool id and tracker id.

Once this is taken care of, we can send events to Amazon Personalize. We can still use any text string for event types, but please note that a couple of special types are available:

  • Identify lets you send the userId for a particular user to Amazon Personalize. The userId then becomes an optional parameter in subsequent calls.
  • MediaAutoTrack automatically calculates the play, pause and resume position for media events, and Amazon Personalize uses the position as event value.

Here is how to send some sample events with AWS Amplify:

Analytics.record({
    eventType: "Identify",
    properties: {
      "userId": "<USER_ID>"
    }
}, "AmazonPersonalize");
Analytics.record({
    eventType: "<EVENT_TYPE>",
    properties: {
      "itemId": "<ITEM_ID>",
      "eventValue": "<EVENT_VALUE>"
    }
}, "AmazonPersonalize");
Analytics.record({
    eventType: "MediaAutoTrack",
    properties: {
      "itemId": "<ITEM_ID>",
      "domElementId": "MEDIA DOM ELEMENT ID"
    }
}, "AmazonPersonalize");

As you can see, this is pretty simple as well.

Creating a recommendation solution

Now that we know how to ingest events, let’s define how our recommendation solution will be trained.

We first need to select a recipe, which is much more than an algorithm: it also includes predefined feature transformations, initial parameters for the algorithm as well as automatic model tuning. Thus, recipes remove the need to have expertise in personalization. Amazon Personalize comes with several recipes suitable for different use cases.

Still, if you’re new to machine learning, you may wonder which one of these recipes best fits your use case. No worry: as mentioned earlier, Amazon Personalize supports AutoML, a new technique that automatically searches for the most optimal recipe, so let’s enable it. While we’re at it, let’s also ask Amazon Personalize to automatically tune recipe parameters.

All of this is very straightforward in the AWS console: as you’ll probably want to automate from now on, let’s use the AWS CLI instead.

$ aws personalize create-solution \
  --name jsimon-movieclick-solution \ 
  --perform-auto-ml --perform-hpo \
  --dataset-group-arn $DATASET_GROUP_ARN

Now we’re ready to train the solution. No servers to worry about, training takes places on fully-managed infrastructure.

$ aws personalize create-solution-version \
  --solution-arn $SOLUTION_ARN 

Once training is complete, we can use the solution version to create a recommendation campaign.

Deploying a recommendation campaign

Still no servers to worry about! In fact, campaigns scale automatically according to incoming traffic: we simply need to define the minimum number of transactions per second (TPS) that we want to support.

This number is used to size the initial fleet for hosting the model. It also impacts how much you will be charged for recommendations ($0.20 per TPS-hour). Here, I’m setting that parameter to 10, which means that I will initially be charged $2 per hour. If traffic exceeds 10 TPS, Personalize will scale up, increasing my bill according to the new TPS setting. Once traffic drops, Personalize will scale down, but it won’t go below my minimum TPS setting.

$ aws personalize create-campaign \
  --name jsimon-movieclick-campaign \
  --min-provisioned-tps 10 \
  --solution-version-arn $SOLUTION_VERSION_ARN

Should you later need to update the campaign with a new solution version, you can simply use the UpdateCampaign API and pass the ARN of the new solution version.

Once the campaign has been deployed, we can quickly test that it’s able to recommend new movies.

Recommending new items in real-time

I don’t think this could be simpler: just pass the id of the user and receive recommendations.

$ aws personalize-rec get-recommendations \
--campaign-arn $CAMPAIGN_ARN \
--user-id 123 --query "itemList[*].itemId"
["1210", "260", "2571", "110", "296", "1193", ...]

At this point, we’re ready to integrate our recommendation model in your application. For example, a web application would have to implement the following steps to display a list of recommended movies:

  • Use the GetRecommendations API in our favorite language to invoke the campaign and receive movie recommendation for a given user,
  • Read movie metadata from a backend (say, image URL, title, genre, release date, etc.),
  • Generate HTML code to be rendered in the user’s browser.

Amazon Personalize in action

Actually, my colleague Jake Wells has built a web application recommending books. Using an open dataset containing over 19 million book reviews, Jake first used a notebook hosted on Amazon SageMaker to clean and prepare the data. Then, he trained a recommendation model with Amazon Personalize, and wrote a simple web application demonstrating the recommendation process. This is a really cool project, which would definitely be worthy of its own blog post!

Available now!

Whether you work with historical data or event streams, a few simple API calls are all it takes to train and deploy recommendation models. Zero machine learning experience is required, so please visit aws.amazon.com/personalize, give it a try and let us know what you think.

Amazon Personalize is available in the following regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Singapore), and EU (Ireland)

The service is also part of the AWS free tier. For the first two months after sign-up, you will be offered:
1. Data processing and storage: up to 20 GB per month
2. Training: up to 100 training hours per month
3. Prediction: up to 50 TPS-hours of real-time recommendations per month

We’re looking forward to your feedback!

Julien;

from AWS News Blog https://aws.amazon.com/blogs/aws/amazon-personalize-is-now-generally-available/

Meet the Newest AWS Heroes! June 2019

Meet the Newest AWS Heroes! June 2019

At the heart of the global AWS community are the builders and influencers whose passion for AWS leads them to actively share their technical know-how with others. The most prominent of these AWS community leaders are recognized as AWS Heroes. Heroes regularly share their extensive AWS knowledge online via blogs, social media, and open source contributions, and in person by speaking at industry conferences and running workshops or AWS-focused User Groups.

Today we are thrilled to introduce the latest cohort of AWS Heroes:

Anton Babenko – Oslo, Norway

Community Hero Anton Babenko is a long time developer and CTO who runs a consulting company Betajob AS in Norway. He helps companies around the globe build solutions using AWS, and specializes in infrastructure as code, DevOps, and reusable infrastructure components. Anton spends a large amount of his time as an open-source contributor on various Terraform & AWS projects, including terraform-aws-modules and modules.tf. He enjoys solving real cloud architecture tasks, figuring out generic solutions by getting to the core, and making them available as open source to let the whole AWS community benefit. Anton also co-founded and co-organizes AWS, DevOps, and HashiCorp User Groups in Norway, DevOpsDays Oslo, and often speaks at various technical Meetups and conferences.

 

 

 

 

 

Bhuvaneswari Subramani – Bengaluru, India

Community Hero Bhuvaneswari Subramani is Director Engineering Operations at Infor. She has almost two decades of IT experience, specializing in Cloud Computing, DevOps, and Performance Testing. She holds the AWS Certified Solution Architect Professional certification, is a co-organizer of the AWS User Group Bengaluru, and is instrumental in organizing Meetups & AWS Community Day Bengaluru. Bhuvaneswari is also an active speaker at AWS community events, industry conferences and delivers guest lectures on Cloud Computing for staff & students at engineering colleges affiliated to Anna University. She is a technophile & IT Blogger, who meticulously and picturesquely depicts the events that inspires & influences her. Her passion for technical writing is exemplified in the form of tech blog DevOps and CloudComputing for over a decade and of late, she constantly writes about AWS conferences and Meetups on the AWS User Group Bengaluru Blog.

 

 

 

Colin Percival – Vancouver, Canada

Community Hero Colin Percival is the founder of Tarsnap, a secure online backup service which combines the flexibility and scriptability of the standard UNIX “tar” utility with strong encryption, deduplication, and the reliability of Amazon S3 storage. Having started work on Tarsnap in 2006, Colin is among the first generation of users of Amazon Web Services, and has written dozens of articles about his experiences with AWS on his blog. Colin has been a member of the FreeBSD project for 15 years and has served in that time as the project Security Officer and a member of the Core team; starting in 2008 he led the efforts to bring FreeBSD to the Amazon EC2 platform, and for the past 7 years he has been maintaining this support, keeping FreeBSD up to date with all of the latest changes and functionality in Amazon EC2.

 

 

 

 

Francesco Pochetti – Luxembourg

Machine Learning Hero Francesco Pochetti first got in touch with Machine Learning back in 2013, taking Stanford’s ML MOOC by Andrew Ng. Now he leverages the wonders of AWS AI infrastructure, plays around with new services, builds ML solutions and lets the world know about his projects on his blog. This is where he regularly posts all his experiments in Machine Learning and Deep Learning. Most notably, within the AWS domain, Inferring movie genre from its poster in AWS SageMaker, Analyzing IMDb reviews sentiment with Amazon Comprehend or Running Neural Style Transfer with Lambda and GPU-powered EC2s.

 

 

 

 

 

 

 

Guy Ernest – Tel Aviv, Israel

Machine Learning Hero Guy Ernest is busy in taking machine learning and AI to the masses to three audiences. The main audience he engages are software developers (SDE) and converting them to machine learning engineers (MLE), using the popular fastai library, PyTorch, and Amazon AI services. The next audience is business people in large enterprises that are learning the applicability of machine learning to their business and the way to conduct the AI transformation of their organization. Finally, Guy works with kids who are starting their AI/ML learning journey by enabling them with Alexa skills, computer vision, and robots in after school and summer camp activities.

 

 

 

 

 

 

 

Kesha Williams – Atlanta, USA

Machine Learning Hero Kesha Williams has over 20 years of experience in software development. She successfully transformed her engineering skills from Java Developer to ML Practitioner by leaning hands on with AWS AI solutions like AWS DeepLens, Amazon Rekognition, and Amazon SageMaker. Kesha believes that as we all benefit from integrating technology into our everyday lives, we still struggle to make meaningful relationships with each other. To solve this, Kesha develops ML models on Amazon SageMaker using computer vision and natural language processing to help us better connect with people around us. Kesha is also very active in the community helping others find their path to machine learning. She authors courses for learning platforms such as Manning Publications, Packt, LinkedIn Learning, A Cloud Guru, and Cloud Academy.

 

 

 

 

Manoj Fernando – Sri Lanka

Community Hero Manoj Fernando is a Technical Lead at 99X Technology in Sri Lanka and the CTO of Whatif AS in Norway. He is passionate about designing scalable and cost-effective cloud architectures on the AWS cloud platform. His team was one of the early adopters of AWS Lambda back in 2015, and he is one of the co-organizers of Serverless Sri Lanka Meetup. During his leisure time, he creates cloud training videos for the community on his YouTube channel. The training videos are focused on Modern Application Development, Cloud Certifications, and Cutting-edge cloud services. He is also a technical blogger, blogging on medium as well as on his website, and a public speaker, conducting cloud workshops for university students in Sri Lanka.

 

 

 

 

 

 

Margaret Valtierra – Chicago, USA

Community Hero Margaret Valtierra is a Program Manager for the Cloud Service team at Morningstar. She is responsible for managing the AWS relationship and promoting cloud skills and best practices. She has organized and led the Chicago AWS user group since 2013. She is a member of the Global AWS Community Day Core Team, promoting user groups and organizing the annual Midwest AWS Community Day. Margaret is also a co-organizer for DevOpsDays Chicago and is an AWS Certified Solutions Architect Associate.

 

 

 

 

 

 

 

Marco Viganò – Milan, Italy

Community Hero Marco Viganò is the CTO of Condé Nast Italy. He has more than 20 years of experience in IT, with a specific focus on media and publishing sector. He is a frequent speaker at AWS Summits and events, sharing design patterns for developing and operating highly scalable cloud solutions for the media and publishing industry. He is focused on Serverless and Machine Learning and one of his main topics is finding new technologies to improve systems. Also, he operates as Voice First evangelist inside the company using Alexa and AWS services.

 

 

 

 

 

 

 

Pavlos Mitsoulis – London, United Kingdom

Machine Learning Hero Pavlos Mitsoulis has 7 years of Machine Learning and Software Engineering experience. Currently, he is a Staff Software Engineer (Machine Learning) at HomeAway (an Expedia Group brand), leading Machine Learning initiatives to support growth marketing. Additionally, he is the creator of Sagify, an open-source library that simplifies training, tuning, evaluating, and deploying ML models to SageMaker. Recently Pavlos authored a Packt video course, “Hands-On Machine Learning Using Amazon SageMaker“.

 

 

 

 

 

 

 

Vicky Seno – Los Angeles, USA

Container Hero Vicky “Tanya” Seno is a Computer Science Professor at Santa Monica College. At SMC she teaches numerous AWS courses covering Computing Services, Containers, Kubernetes, ECS, Serverless, Networking and Security. Vicky has helped develop an AWS Cloud Computing College Degree and is part of a team that helps train and mentor faculty from nineteen local colleges in AWS, to help expand AWS course offerings in the Los Angeles area. She is also a co-organizer of AWS Cloud Day Conference at SMC that includes SA speakers, AWS workshops and a AWS CTF attended by over 130+ students at each event. In an effort to increase female representation in this field, Vicky has been involved in various speaking and training activities. Vicky hosts a YouTube Channel with over 34,000 followers and 100+ beginners tech tutorials. She has also spoken at AWS Summits on Containers, Kubernetes, and Amazon EKS.

 

 

 

 

 

 

You can learn about all the AWS Heroes from around the globe by checking out the Hero website.

from AWS News Blog https://aws.amazon.com/blogs/aws/meet-the-newest-aws-heroes-june-2019/

Learn about AWS Services & Solutions – June AWS Online Tech Talks

Learn about AWS Services & Solutions – June AWS Online Tech Talks

Join us this June to learn about AWS services and solutions. The AWS Online Tech Talks are live, online presentations that cover a broad range of topics at varying technical levels. These tech talks, led by AWS solutions architects and engineers, feature technical deep dives, live demonstrations, customer examples, and Q&A with AWS experts. Register Now!

Note – All sessions are free and in Pacific Time.

Tech talks this month:

AR/VR

June 20, 2019 | 1:00 PM – 2:00 PM PTCreate a Digital Asset with Amazon Sumerian and AWS IoT – Learn to create a “digital asset” of a physical machine using Amazon Sumerian and AWS IoT to create simulations, alerts, and diagnostic instructions.

Compute

June 25, 2019 | 9:00 AM – 10:00 AM PTRunning Enterprise CI/CD workloads with Amazon EC2 Spot Instances and CloudBees – Learn how to launch your CI/CD pipelines in the AWS cloud more quickly, and cost-effectively with CloudBees Core and Amazon EC2 Spot Instances.

June 26, 2019 | 11:00 AM – 12:00 PM PTGain Control Over Your License Usage and Reduce Costs Using AWS License Manager – See how you can gain control over your license usage and reduce costs using AWS License Manager through effective and proactive license management.

Containers

June 18, 2019 | 9:00 AM – 10:00 AM PTServerless E-commerce Application Using AWS AppSync – Learn how to get started with AWS Amplify, create a GraphQL API using AppSync, and see how easy it is to collect and visualize analytics data using Amazon Pinpoint, Amazon Kinesis Firehose, Amazon Redshift, and Amazon QuickSight.

June 20, 2019 | 11:00 AM – 12:00 PM PTContainer Security 101 and Beyond – Join a discussion on container-specific security considerations, from container image hygiene to running containers securely, handling sensitive data, permission management, as well as auditing.

Data Lakes & Analytics

June 17, 2019 | 11:00 AM – 12:00 PM PTBuild ETL Processes for Data Lakes with AWS Glue – Learn how you can build, automate, and manage ETL jobs for your data lake, using AWS Glue as a scalable, serverless platform for Apache Spark and Python shell jobs.

June 19, 2019 | 9:00 AM – 10:00 AM PT Ditching the Overhead: Moving Apache Kafka Workloads into Amazon MSK – Learn how to remove the complexity of managing Apache Kafka by moving your workloads to Amazon MSK with minimal downtime.

June 20, 2019 | 9:00 AM – 10:00 AM PTUnify Your Data Warehouse and Data Lake with AWS – Learn how to break data silos among analytical engines and between various groups in an organization by combining data warehouse and data lake architectures with AWS.

Databases

June 18, 2019 | 1:00 PM – 2:00 PM PTAchieve Continuous Availability with Amazon Aurora Multi-Master – Learn how to achieve Continuous Availability with Amazon Aurora Multi-Master.

DevOps

June 19, 2019 | 1:00 PM – 2:00 PM PTRoot Cause and End-user Impact Analysis Using AWS X-Ray – Learn how to pinpoint application performance bottlenecks and assess the root cause of discovered issues using AWS X-Ray.

End-User Computing

June 24, 2019 | 9:00 AM – 10:00 AM PTReplace Your On-Premises VDI with End-User Computing on AWS – Learn how you can improve the flexibility and scalability of your EUC solutions by moving to EUC services on AWS.

Enterprise & Hybrid

June 25, 2019 | 11:00 AM – 12:00 PM PTHow to Build Your Cloud Enablement Engine with the People You Already Have – Learn how to build the skills, organization, and operating model to accelerate cloud adoption.

IoT

June 24, 2019 | 11:00 AM – 12:00 PM PTSecuring Your Devices from the Edge to the Cloud – Learn how to use AWS IoT services to reduce the complexity of keeping data secure, restricting access to devices and cloud resources, securely connecting to the cloud, and auditing device usage.

Machine Learning

June 18, 2019 | 11:00 AM – 12:00 PM PTBuilding Secure Machine Learning Environments Using Amazon SageMaker – Learn how to build secure machine learning environments using Amazon SageMaker.

June 26, 2019 | 1:00 PM – 2:00 PM PT – Fraud Detection Using Machine Learning with Amazon SageMaker – Learn how to do fraud detection using machine learning with Amazon SageMaker.

Migration

June 27, 2019 | 9:00 AM – 10:00 AM PTHow AWS Migration Hub Helps You Plan, Track, and Complete Your Application Migrations – Learn how to use AWS Migration Hub and AWS Application Discovery Service to discover, plan and track your application migrations.

Networking & Content Delivery

June 17, 2019 | 1:00 PM – 2:00 PM PTCustomer Showcase: Exploiting Multi-Region Data Locality with [email protected] – Learn how to run your applications in multi-regions and how serverless can help.

Robotics

June 26, 2019 | 9:00 AM – 10:00 AM PT Developing Intelligent Robots with AWS RoboMaker – Learn how to develop, test and deploy intelligent robotic applications with AWS RoboMaker.

Security, Identity, & Compliance

June 17, 2019 | 9:00 AM – 10:00 AM PTContinuous Compliance with AWS Security Hub – Learn how AWS Security Hub provides you with a comprehensive view of your security state within AWS and helps you with continuous compliance checks.

June 19, 2019 | 11:00 AM – 12:00 PM PTCustomer Showcase: Learn How Amazon.com Uses Amazon GuardDuty to Protect its Infrastructure – Learn how one of AWS largest customers, Amazon.com, uses Amazon GuardDuty to protect their infrastructure.

Serverless

June 27, 2019 | 11:00 AM – 12:00 PM PTServerless Streams, Topics, Queues, & APIs! How to Pick the Right Serverless Application Pattern – Learn how to choose between streams, topics, queues, or APIs with AWS Lambda – pick right design pattern for your serverless application.

Storage

June 24, 2019 | 1:00 PM – 2:00 PM PT Build Your Data Lake on Amazon S3 – Learn why Amazon S3 is the best destination for your data lake.

from AWS News Blog https://aws.amazon.com/blogs/aws/learn-about-aws-services-solutions-june-aws-online-tech-talks/

Amazon Managed Streaming for Apache Kafka (MSK) – Now Generally Available

Amazon Managed Streaming for Apache Kafka (MSK) – Now Generally Available

I am always amazed at how our customers are using streaming data. For example, Thomson Reuters, one of the world’s most trusted news organizations for businesses and professionals, built a solution to capture, analyze, and visualize analytics data to help product teams continuously improve the user experience. Supercell, the social game company providing games such as Hay Day, Clash of Clans, and Boom Beach, is delivering in-game data in real-time, handling 45 billion events per day.

Since we launched Amazon Kinesis at re:Invent 2013, we have continually expanded the ways in in which customers work with streaming data on AWS. Some of the available tools are:

  • Kinesis Data Streams, to capture, store, and process data streams with your own applications.
  • Kinesis Data Firehose, to transform and collect data into destinations such as Amazon S3, Amazon Elasticsearch Service, and Amazon Redshift.
  • Kinesis Data Analytics, to continuously analyze data using SQL or Java (via Apache Flink applications), for example to detect anomalies or for time series aggregation.
  • Kinesis Video Streams, to simplify processing of media streams.

At re:Invent 2018, we introduced in open preview Amazon Managed Streaming for Apache Kafka (MSK), a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data.

I am excited to announce that Amazon MSK is generally available today!

How it works

Apache Kafka (Kafka) is an open-source platform that enables customers to capture streaming data like click stream events, transactions, IoT events, application and machine logs, and have applications that perform real-time analytics, run continuous transformations, and distribute this data to data lakes and databases in real time. You can use Kafka as a streaming data store to decouple applications producing streaming data (producers) from those consuming streaming data (consumers).

While Kafka is a popular enterprise data streaming and messaging framework, it can be difficult to setup, scale, and manage in production. Amazon MSK takes care of these managing tasks and makes it easy to set up, configure, and run Kafka, along with Apache ZooKeeper, in an environment following best practices for high availability and security.

Your MSK clusters always run within an Amazon VPC managed by the MSK service. Your MSK resources are made available to your own VPC, subnet, and security group through elastic network interfaces (ENIs) which will appear in your account, as described in the following architectural diagram:

Customers can create a cluster in minutes, use AWS Identity and Access Management (IAM) to control cluster actions, authorize clients using TLS private certificate authorities fully managed by AWS Certificate Manager (ACM), encrypt data in-transit using TLS, and encrypt data at rest using AWS Key Management Service (KMS) encryption keys.

Amazon MSK continuously monitors server health and automatically replaces servers when they fail, automates server patching, and operates highly available ZooKeeper nodes as a part of the service at no additional cost. Key Kafka performance metrics are published in the console and in Amazon CloudWatch. Amazon MSK is fully compatible with Kafka versions 1.1.1 and 2.1.0, so that you can continue to run your applications, use Kafka’s admin tools, and and use Kafka compatible tools and frameworks without having to change your code.

Based on our customer feedback during the open preview, Amazon MSK added may features such as:

  • Encryption in-transit via TLS between clients and brokers, and between brokers
  • Mutual TLS authentication using ACM private certificate authorities
  • Support for Kafka version 2.1.0
  • 99.9% availability SLA
  • HIPAA eligible
  • Cluster-wide storage scale up
  • Integration with AWS CloudTrail for MSK API logging
  • Cluster tagging and tag-based IAM policy application
  • Defining custom, cluster-wide configurations for topics and brokers

AWS CloudFormation support is coming in the next few weeks.

Creating a cluster

Let’s create a cluster using the AWS management console. I give the cluster a name, select the VPC I want to use the cluster from, and the Kafka version.

I then choose the Availability Zones (AZs) and the corresponding subnets to use in the VPC. In the next step, I select how many Kafka brokers to deploy in each AZ. More brokers allow you to scale the throughtput of a cluster by allocating partitions to different brokers.

I can add tags to search and filter my resources, apply IAM policies to the Amazon MSK API, and track my costs. For storage, I leave the default storage volume size per broker.

I select to use encryption within the cluster and to allow both TLS and plaintext traffic between clients and brokers. For data at rest, I use the AWS-managed customer master key (CMK), but you can select a CMK in your account, using KMS, to have further control. You can use private TLS certificates to authenticate the identity of clients that connect to your cluster. This feature is using Private Certificate Authorities (CA) from ACM. For now, I leave this option unchecked.

In the advanced setting, I leave the default values. For example, I could have chosen here a different instance type for my brokers. Some of these settings can be updated using the AWS CLI.

I create the cluster and monitor the status from the cluster summary, including the Amazon Resource Name (ARN) that I can use when interacting via CLI or SDKs.

When the status is active, the client information section provides specific details to connect to the cluster, such as:

  • The bootstrap servers I can use with Kafka tools to connect to the cluster.
  • The Zookeper connect list of hosts and ports.

I can get similar information using the AWS CLI:

  • aws kafka list-clusters to see the ARNs of your clusters in a specific region
  • aws kafka get-bootstrap-brokers --cluster-arn <ClusterArn> to get the Kafka bootstrap servers
  • aws kafka describe-cluster --cluster-arn <ClusterArn> to see more details on the cluster, including the Zookeeper connect string

Quick demo of using Kafka

To start using Kafka, I create two EC2 instances in the same VPC, one will be a producer and one a consumer. To set them up as client machines, I download and extract the Kafka tools from the Apache website or any mirror. Kafka requires Java 8 to run, so I install Amazon Corretto 8.

On the producer instance, in the Kafka directory, I create a topic to send data from the producer to the consumer:

bin/kafka-topics.sh --create --zookeeper <ZookeeperConnectString> \
--replication-factor 3 --partitions 1 --topic MyTopic

Then I start a console-based producer:

bin/kafka-console-producer.sh --broker-list <BootstrapBrokerString> \
--topic MyTopic

On the consumer instance, in the Kafka directory, I start a console-based consumer:

bin/kafka-console-consumer.sh --bootstrap-server <BootstrapBrokerString> \
--topic MyTopic --from-beginning

Here’s a recording of a quick demo where I create the topic and then send messages from a producer (top terminal) to a consumer of that topic (bottom terminal):

Pricing and availability

Pricing is per Kafka broker-hour and per provisioned storage-hour. There is no cost for the Zookeeper nodes used by your clusters. AWS data transfer rates apply for data transfer in and out of MSK. You will not be charged for data transfer within the cluster in a region, including data transfer between brokers and data transfer between brokers and ZooKeeper nodes.

You can migrate your existing Kafka cluster to MSK using tools like MirrorMaker (that comes with open source Kafka) to replicate data from your clusters into a MSK cluster.

Upstream compatibility is a core tenet of Amazon MSK. Our code changes to the Kafka platform are released back to open source.

Amazon MSK is available in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Tokyo), Asia Pacific (Singapore), Asia Pacific (Sydney), EU (Frankfurt), EU (Ireland), EU (Paris), and EU (London).

I look forward to see how are you going to use Amazon MSK to simplify building and migrating streaming applications to the cloud!

from AWS News Blog https://aws.amazon.com/blogs/aws/amazon-managed-streaming-for-apache-kafka-msk-now-generally-available/