Tag: SDK

Setting up an Android application with AWS SDK for C++

Setting up an Android application with AWS SDK for C++

The AWS SDK for C++ can build and run on many different platforms, including Android. In this post, I walk you through building and running a sample application on an Android device.

Overview

I cover the following topics:

  • Building and installing the SDK for C++ and creating a desktop application that implements the basic functionality.
  • Setting up the environment for Android development.
  • Building and installing the SDK for C++ for Android and transplanting the desktop application to Android platform with the cross-compiled SDK.

Prerequisites

To get started, you need the following resources:

  • An AWS account
  • A GitHub environment to build and install AWS SDK for C++

To set up the application on the desktop

Follow these steps to set up the application on your desktop:

  • Create an Amazon Cognito identity pool
  • Build and test the desktop application

Create an Amazon Cognito identity pool

For this demo, I use unauthenticated identities, which typically belong to guest users. To learn about unauthenticated and authenticated identities and choose the one that fits your business, check out Using Identity Pools.

In the Amazon Cognito console, choose Manage identity pools, Create new identity pool. Enter an identity pool name, like “My Android App with CPP SDK”, and choose Enable access to unauthenticated identities.

Next, choose Create Pool, View Details. Two Role Summary sections should display, one for authenticated and the other for unauthenticated identities.

For the unauthenticated identities, choose View Policy Document, Edit. Under Action, add the following line:

s3: ListAllMyBuckets

After you have completed the preceding steps, the policy should read as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "mobileanalytics:PutEvents",
                "cognito-sync:*"
            ],
            "Resource": "*"
        }
    ]
}

To finish the creation, choose Allow.

On the next page, in the Get AWS Credentials section, copy the identity pool ID and keep it somewhere to use later. Or, you can find it after you choose Edit identity pool in the Amazon Cognito console. The identity pool ID has the following format:

<region>:<uuid>

Build and test the desktop application

Before building an Android application, you can build a regular application with the SDK for C++ in your desktop environment for testing purpose. Then you must modify the source code, making the CMake script switch its target to Android.

Here’s how to build and install the SDK for C++ statically:

cd <workspace>
git clone https://github.com/aws/aws-sdk-cpp.git
mkdir build_sdk_desktop
cd build_sdk_desktop
cmake ../aws-sdk-cpp \
    -DBUILD_ONLY="identity-management;s3" \
    -DBUILD_SHARED_LIBS=OFF \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX="<workspace>/install_sdk_desktop"
cmake --build .
cmake --build . --target install

If you install the SDK for C++ successfully, you can find libaws-cpp-sdk-*.a (or aws-cpp-sdk-*.lib for Windows) under <workspace>/install_sdk_desktop/lib/ (or <workspace>/install_sdk_desktop/lib64).

Next, build the application and link it to the library that you built. Create a folder <workspace>/app_list_all_buckets and place two files under this directory:

  • main.cpp (source file)
  • CMakeLists.txt (CMake file)
// main.cpp
#include <iostream>
#include <aws/core/Aws.h>
#include <aws/core/utils/Outcome.h>
#include <aws/core/utils/logging/AWSLogging.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/identity-management/auth/CognitoCachingCredentialsProvider.h>
#include <aws/s3/S3Client.h>

using namespace Aws::Auth;
using namespace Aws::CognitoIdentity;
using namespace Aws::CognitoIdentity::Model;

static const char ALLOCATION_TAG[] = "ListAllBuckets";
static const char ACCOUNT_ID[] = "your-account-id";
static const char IDENTITY_POOL_ID[] = "your-cognito-identity-id";

int main()
{
    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;
    Aws::InitAPI(options);

    Aws::Client::ClientConfiguration config;
    auto cognitoIdentityClient = Aws::MakeShared<CognitoIdentityClient>(ALLOCATION_TAG, Aws::MakeShared<AnonymousAWSCredentialsProvider>(ALLOCATION_TAG), config);
    auto cognitoCredentialsProvider = Aws::MakeShared<CognitoCachingAnonymousCredentialsProvider>(ALLOCATION_TAG, ACCOUNT_ID, IDENTITY_POOL_ID, cognitoIdentityClient);

    Aws::S3::S3Client s3Client(cognitoCredentialsProvider, config);
    auto listBucketsOutcome = s3Client.ListBuckets();
    Aws::StringStream ss;
    if (listBucketsOutcome.IsSuccess())
    {
        ss << "Buckets:" << std::endl;
        for (auto const& bucket : listBucketsOutcome.GetResult().GetBuckets())
        {
            ss << "  " << bucket.GetName() << std::endl;
        }
    }
    else
    {
        ss << "Failed to list buckets." << std::endl;
    }
    std::cout << ss.str() << std::endl;
    Aws::ShutdownAPI(options);
    return 0;
}
# CMakeLists.txt
cmake_minimum_required(VERSION 3.3)
set(CMAKE_CXX_STANDARD 11)

project(list_all_buckets LANGUAGES CXX)
find_package(AWSSDK REQUIRED COMPONENTS s3 identity-management)
add_executable(${PROJECT_NAME} "main.cpp")
target_link_libraries(${PROJECT_NAME} ${AWSSDK_LINK_LIBRARIES})

Build and test this desktop application with the following commands:

cd <workspace>
mkdir build_app_desktop
cd build_app_desktop
cmake ../app_list_all_buckets \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_PREFIX_PATH="<workspace>/install_sdk_desktop"
cmake --build .
./list_all_buckets # or ./Debug/list_all_buckets.exe for Windows

The output should read as follows:

Buckets:
  <bucket_1>
  <bucket_2>
  ...

Now you have a desktop application. You’ve accomplished this without touching anything related to Android. The next section covers Android instructions.

To set up the application on Android with AWS SDK for C++

Follow these steps to set up the application on Android with the SDK for C++:

  • Set up Android Studio
  • Cross-compile the SDK for C++ and the library
  • Build and run the application in Android Studio

Set up Android Studio

First, download and install Android Studio. For more detailed instructions, see the Android Studio Install documentation.

Next, open Android Studio and create a new project. On the Choose your project screen, as shown in the following screenshot, choose Native C++, Next.

Complete all fields. In the following example, you build the SDK for C++ with Android API level 19, so the Minimum API Level is “API 19: Android 4.4 (KitKat)”.

Choose C++ 11 for C++ Standard and choose Finish for the setup phase.

The first time that you open Android Studio, you might see “missing NDK and CMake” errors during automatic installation. Ignore these warnings for the moment. You install Android NDK and CMake manually. Or you can accept the license to install NDK and CMake within Android Studio. Doing so should suppress the warnings.

After you choose Finish, you should get a sample application with Android Studio. This application publishes some messages on the screens of your devices. For more details, see Create a new project with C++.

Starting from this sample, take the following steps to build your application:

First, cross-compile the SDK for C++ for Android.

Modify the source code and CMake script to build list_all_buckets as a shared object library (liblist_all_buckets.so) rather than an executable. This library has a function: listAllBuckets() to output all buckets.

Specify the path to the library in the module’s build.gradle file so that the Android application can find it.

Load this library in MainActivity by System.loadLibrary("list_all_buckets"), so that the Android application can use the listAllBuckets() function.

Call the listAllBuckets() function in OnCreate() function in MainActivity.

More details for each step will be given in the following sections.

Cross-compile the SDK for C++ and the library

Use Android NDK to cross-compile the SDK for C++. In this example, you are using version r19c. To find whether Android Studio has downloaded NDK by default, check the following:

  • Linux: ~/Android/Sdk/ndk-bundle
  • MacOS: ~/Library/Android/sdk/ndk-bundle
  • Windows: C:\Users\<username>\AppData\Local\Android\Sdk\ndk\<version>

Alternately, download the Android NDK directly.

To cross-compile SDK for C++, run the following code:

cd <workspace>
mkdir build_sdk_android
cd build_sdk_anrdoid
cmake ../aws-sdk-cpp -DNDK_DIR="<path-to-android-ndk>" \
    -DBUILD_SHARED_LIBS=OFF \
    -DCMAKE_BUILD_TYPE=Release \
    -DCUSTOM_MEMORY_MANAGEMENT=ON \
    -DTARGET_ARCH=ANDROID \
    -DANDROID_NATIVE_API_LEVEL=19 \
    -DBUILD_ONLY="identity-management;s3" \
    -DCMAKE_INSTALL_PREFIX="<workspace>/install_sdk_android"
cmake --build . --target CURL # This step is only required on Windows.
cmake --build .
cmake --build . --target install

On Windows, you might see the error message: “CMAKE_SYSTEM_NAME is ‘Android’ but ‘NVIDIA Nsight Tegra Visual Studio Edition’ is not installed.” In that case, install Ninja and change the generator from Visual Studio to Ninja by passing -GNinja as another parameter to your CMake command.

To build list_all_buckets as an Android-targeted dynamic object library, you must change the source code and CMake script. More specifically, you must alter the source code as follows:

Replace main() function with Java_com_example_mynativecppapplication_MainActivity_listAllBuckets(). In the Android application, the Java code calls this function through JNI (Java Native Interface). You may have a different function name, based on your package name and activity name. For this demo, the package name is com.example.mynativecppapplication, the activity name is MainActivity, and the actual function called by Java code is called listAllBuckets().

Enable LogcatLogSystem, so that you can debug your Android application and see the output in the logcat console.

Your Android devices or emulators may miss CA certificates. So, you should push them to your devices and specify the path in client configuration. In this example, use CA certificates extracted from Mozilla in PEM format.

Download the certificate bundle.

Push this file to your Android devices:

# Change directory to the location of adb
cd <path-to-android-sdk>/platform-tools
# Replace "com.example.mynativecppapplication" with your package name
./adb shell mkdir -p /sdcard/Android/data/com.example.mynativecppapplication/certs
# push the PEM file to your devices
./adb push cacert.pem /sdcard/Android/data/com.example.mynativecppapplication/certs

Specify the path in the client configuration:

config.caFile = "/sdcard/Android/data/com.example.mynativecppapplication/certs/cacert.pem";

The complete source code looks like the following:

// main.cpp
#if __ANDROID__
#include <android/log.h>
#include <jni.h>
#include <aws/core/platform/Android.h>
#include <aws/core/utils/logging/android/LogcatLogSystem.h>
#endif
#include <iostream>
#include <aws/core/Aws.h>
#include <aws/core/utils/Outcome.h>
#include <aws/core/utils/logging/AWSLogging.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/identity-management/auth/CognitoCachingCredentialsProvider.h>
#include <aws/s3/S3Client.h>

using namespace Aws::Auth;
using namespace Aws::CognitoIdentity;
using namespace Aws::CognitoIdentity::Model;

static const char ALLOCATION_TAG[] = "ListAllBuckets";
static const char ACCOUNT_ID[] = "your-account-id";
static const char IDENTITY_POOL_ID[] = "your-cognito-identity-id";

#ifdef __ANDROID__
extern "C" JNIEXPORT jstring JNICALL
Java_com_example_mynativecppapplication_MainActivity_listAllBuckets(JNIEnv* env, jobject classRef, jobject context)
#else
int main()
#endif
{
    Aws::SDKOptions options;
#ifdef __ANDROID__
    AWS_UNREFERENCED_PARAM(classRef);
    AWS_UNREFERENCED_PARAM(context);
    Aws::Utils::Logging::InitializeAWSLogging(Aws::MakeShared<Aws::Utils::Logging::LogcatLogSystem>(ALLOCATION_TAG, Aws::Utils::Logging::LogLevel::Debug));
#else
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Debug;
#endif
    Aws::InitAPI(options);

    Aws::Client::ClientConfiguration config;
#ifdef __ANDROID__
    config.caFile = "/sdcard/Android/data/com.example.mynativecppapplication/certs/cacert.pem";
#endif
    auto cognitoIdentityClient = Aws::MakeShared<CognitoIdentityClient>(ALLOCATION_TAG, Aws::MakeShared<AnonymousAWSCredentialsProvider>(ALLOCATION_TAG), config);
    auto cognitoCredentialsProvider = Aws::MakeShared<CognitoCachingAnonymousCredentialsProvider>(ALLOCATION_TAG, ACCOUNT_ID, IDENTITY_POOL_ID, cognitoIdentityClient);

    Aws::S3::S3Client s3Client(cognitoCredentialsProvider, config);
    auto listBucketsOutcome = s3Client.ListBuckets();
    Aws::StringStream ss;
    if (listBucketsOutcome.IsSuccess())
    {
        ss << "Buckets:" << std::endl;
        for (auto const& bucket : listBucketsOutcome.GetResult().GetBuckets())
        {
            ss << "  " << bucket.GetName() << std::endl;
        }
    }
    else
    {
        ss << "Failed to list buckets." << std::endl;
    }

#if __ANDROID__
    std::string allBuckets(ss.str().c_str());
    Aws::ShutdownAPI(options);
    return env->NewStringUTF(allBuckets.c_str());
#else
    std::cout << ss.str() << std::endl;
    Aws::ShutdownAPI(options);
    return 0;
#endif
}

Next, make the following changes for the CMake script:

  • Set the default values for the parameters used for the Android build, including:
    • The default Android API Level is 19
    • The default Android ABI is armeabi-v7a
    • Use libc++ as the standard library by default
    • Use android.toolchain.cmake supplied by Android NDK by default
  • Build list_all_buckets as a library rather than an executable
  • Link to the external libraries built in the previous step: zlib, ssl, crypto, and curl
# CMakeLists.txt
cmake_minimum_required(VERSION 3.3)
set(CMAKE_CXX_STANDARD 11)

if(TARGET_ARCH STREQUAL "ANDROID") 
    if(NOT NDK_DIR)
        set(NDK_DIR $ENV{ANDROID_NDK})
    endif()
    if(NOT IS_DIRECTORY "${NDK_DIR}")
        message(FATAL_ERROR "Could not find Android NDK (${NDK_DIR}); either set the ANDROID_NDK environment variable or pass the path in via -DNDK_DIR=..." )
    endif()
if(NOT CMAKE_TOOLCHAIN_FILE)
    set(CMAKE_TOOLCHAIN_FILE "${NDK_DIR}/build/cmake/android.toolchain.cmake")
endif()

if(NOT ANDROID_ABI)
    set(ANDROID_ABI "armeabi-v7a")
    message(STATUS "Android ABI: none specified, defaulting to ${ANDROID_ABI}")
else()
    message(STATUS "Android ABI: ${ANDROID_ABI}")
endif()

if(BUILD_SHARED_LIBS)
    set(ANDROID_STL "c++_shared")
else()
    set(ANDROID_STL "c++_static")
endif()

if(NOT ANDROID_NATIVE_API_LEVEL)
    set(ANDROID_NATIVE_API_LEVEL "android-19")
    message(STATUS "Android API Level: none specified, defaulting to ${ANDROID_NATIVE_API_LEVEL}")
else()
    message(STATUS "Android API Level: ${ANDROID_NATIVE_API_LEVEL}")
endif()

    list(APPEND CMAKE_FIND_ROOT_PATH ${CMAKE_PREFIX_PATH})
endif()

project(list_all_buckets LANGUAGES CXX)
find_package(AWSSDK REQUIRED COMPONENTS s3 identity-management)
if(TARGET_ARCH STREQUAL "ANDROID")
    set(SUFFIX so)
    add_library(zlib STATIC IMPORTED)
    set_target_properties(zlib PROPERTIES IMPORTED_LOCATION ${EXTERNAL_DEPS}/zlib/lib/libz.a)
    add_library(ssl STATIC IMPORTED)
    set_target_properties(ssl PROPERTIES IMPORTED_LOCATION ${EXTERNAL_DEPS}/openssl/lib/libssl.a)
    add_library(crypto STATIC IMPORTED)
    set_target_properties(crypto PROPERTIES IMPORTED_LOCATION ${EXTERNAL_DEPS}/openssl/lib/libcrypto.a)
    add_library(curl STATIC IMPORTED)
    set_target_properties(curl PROPERTIES IMPORTED_LOCATION ${EXTERNAL_DEPS}/curl/lib/libcurl.a)
    add_library(${PROJECT_NAME} "main.cpp")
else()
    add_executable(${PROJECT_NAME} "main.cpp")
endif()
target_link_libraries(${PROJECT_NAME} ${AWSSDK_LINK_LIBRARIES})

Finally, build this library with the following command:

cd <workspace>
mkdir build_app_android
cd build_app_android
cmake ../app_list_all_buckets \
    -DNDK_DIR="<path-to-android-ndk>" \
    -DBUILD_SHARED_LIBS=ON \
    -DTARGET_ARCH=ANDROID \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_PREFIX_PATH="<workspace>/install_sdk_android" \
    -DEXTERNAL_DEPS="<workspace>/build_sdk_android/external"
cmake --build .

This build results in the shared library liblist_all_buckets.so under <workspace>/build_app_android/. It’s time to switch to Android Studio.

Build and run the application in Android Studio

First, the application must find the library (liblist_all_buckets.so) that you built and the standard library (libc++_shared.so). The default search path for JNI libraries is app/src/main/jniLibs/<android-abi>. Create a directory called: <your-android-application-root>/app/src/main/jniLibs/armeabi-v7a/ and copy the following files to this directory:

<workspace>/build_app_android/liblist_all_buckets.so

  • For Linux: <android-ndk>/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libc++_shared.so
  • For MacOS: <android-ndk>/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/arm-linux-androideabi/libc++_shared.so
  • For Windows: <android-ndk>/toolchains/llvm/prebuilt/windows-x86_64/sysroot/usr/lib/arm-linux-androideabi/libc++_shared.so

Next, open the build.gradle file for your module and remove the externalNativeBuild{} block, because you are using prebuilt libraries, instead of building the source with the Android application.

Then, edit MainActivity.java, which is under app/src/main/java/<package-name>/. Replace all native-libs with list_all_buckets and replace all stringFromJNI() with listAllBuckets(). The whole Java file looks like the following code example:

// MainActivity.java
package com.example.mynativecppapplication;

import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.widget.TextView;

public class MainActivity extends AppCompatActivity {

    // Used to load the 'list_all_buckets' library on application startup.
    static {
        System.loadLibrary("list_all_buckets");
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        // Example of a call to a native method
        TextView tv = findViewById(R.id.sample_text);
        tv.setText(listAllBuckets());
    }

    /**
    * A native method that is implemented by the 'list_all_buckets' native library,
    * which is packaged with this application.
    */
    public native String listAllBuckets();
}

Finally, don’t forget to grant internet access permission to your application by adding the following lines in the AndroidManifest.xml, located at app/src/main/:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.mynativecppapplication">
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
    ...
</manifest>

To run the application on Android emulator, make sure that the CPU/ABI is armeabi-v7a for the system image. That’s what you specified when you cross-compiled the SDK and list_all_buckets library for the Android platform.

Run this application by choosing the Run icon or choosing Run, Run [app]. You should see that the application lists all buckets, as shown in the following screenshot.

Summary

With Android Studio and its included tools, you can cross-compile the AWS SDK for C++ and build a sample Android application to get temporary Amazon Cognito credentials and list all S3 buckets. Starting from this simple application, AWS hopes to see more exciting integrations with the SDK for C++.

As always, AWS welcomes all feedback or comment. Feel free to open an issue on GitHub if you have questions or submit a pull request to contribute.

from AWS Developer Blog https://aws.amazon.com/blogs/developer/setting-up-an-android-application-with-aws-sdk-for-c/

Developing and testing GraphQL APIs, Storage and Functions with Amplify Framework Local Mocking features

Developing and testing GraphQL APIs, Storage and Functions with Amplify Framework Local Mocking features

This article was written by Ed Lima, Sr. Solutions Architect, AWS and Sean Grove, OneGraph

In fullstack application development, iteration is king. At AWS, we’re constantly identifying steps in the process of shipping product that slow iteration, or sap developer productivity and happiness, and work to shorten it. To that end, we’ve provided cloud APIs, serverless functions, databases, and storage capabilities so that the final steps of deploying, scaling, and monitoring applications are as instantaneous as possible.

Today, we’re taking another step further in shortening feedback cycles by addressing a critical stage in the application cycle: local development.

Working closely with developers, we’ve seen the process of delivering new product features to production:

  1. Prototyping changes locally
  2. Committing and pushing to the cloud resources
  3. Mocking/testing/debugging the updates
  4. Returning to step 1 if there are any fixes to incorporate

In some cases, this can be an incredibly tight loop, executed dozens or hundreds of times by a developer, before new features are ready to ship. It can be a tedious process, and tedious processes make unhappy developers.

AWS AppSync gives developers easy, and convenient access to exactly the right data they need at a global scale via its flexible GraphQL APIs. These APIs, among other data sources, can be backed by Amazon DynamoDB for a scalable key-value and document database that delivers single-digit millisecond performance at any scale. Applications can also use Amazon Simple Storage Service (S3) for an object storage service that offers industry-leading scalability, data availability, security, and performance. On top of it, developers can run their code without provisioning or managing servers with AWS Lambda. All of these services live in the cloud, which is great for production – highly available, fault tolerant, scaling to meet any demand, running in multiple availability zones in different AWS regions around the planet.

In order to optimize and streamline the feedback loop between local and cloud resources earlier in the development process, we talked to many customers to understand their requirements for local development:

  • NoSQL data access via a robust GraphQL API
  • Serverless functions triggered for customized business logic from any GraphQL type or operation
  • Developer tooling, including a GraphiQL IDE fully pre-integrated with open-source plugins such as those from OneGraph, customized for your AppSync API
  • Simulated object storage
  • Instantaneous feedback on changes
  • Debugging GraphQL resolver mapping templates written in Velocity Template Language (VTL)
  • Ability to use custom directives and code generation with the GraphQL Transformer
  • Ability to mock JWT tokens from Amazon Cognito User Pools to test authorization rules locally
  • Work with web and mobile platforms (iOS and Android)
  • And, the ability to work offline

With the above customer requirements in mind we’re happy to launch the new Local Mocking and Testing features in the Amplify Framework.

As a developer using Amplify, you’ll immediately see the changes you make locally to your application, speeding up your development process and removing interruptions to your workflow. No waiting for cloud services to be deployed – just develop, test, debug, model your queries, and generate code locally until you’re happy with your product, then deploy your changes to the scalable, highly available backend services in the cloud as you’ve always done.

Getting Started

To get started, install the latest version of the Amplify CLI by following these steps, and follow along with our example below. Use a boilerplate React app created with create-react-app and initialize an Amplify project in the app folder with the default options by executing the amplify init command. Note, the local mocking and testing features in the Amplify CLI will also work with iOS and Android apps.

Next, we add a GraphQL API using the command amplify add api with API Key authorization and the sample schema for single object with fields (Todo):

When defining a GraphQL schema you can use directives from the GraphQL Transformer in local mocking as well as local code generation from the schema for GraphQL operations. The following directives are currently supported in the local mocking environment:

  • @model
  • @auth
  • @key
  • @connection
  • @versioned
  • @function

The sample GraphQL schema generated by the Amplify CLI has a single “Todo” type defined with @model which means the GraphQL Transformer will automatically create a GraphQL API with an extended schema containing queries, mutations, and subscriptions with built-in CRUDL logic to access a DynamoDB table, also automatically deployed. It basically creates a fully-fledged API backend in seconds:

type Todo @model {
  id: ID!
  name: String!
  description: String
}

At this point, your API is ready for some local development! Fire up your local AppSync and DynamoDB resources by executing either the command  amplify mock to test all supported local resources or amplify mock api to test specifically the GraphQL API and watch as a local mock endpoint starts up. Code will be automatically generated and validated for queries, mutations, subscriptions and a local AppSync mock endpoint will start up:

Collaborating with the Open Source community is always special, it has allowed us to improve and better understand the use cases that customers want to tackle with local mocking and testing. In order to move fast and ensure that we were releasing a valuable feature, we worked for several months with a few community members. We want to give a special thanks to Conduit Ventures for creating the AWS-Utils package, as well as allowing us to fork it for this project and integrate with the Amplify new local mocking environment.

Prototyping API calls with an enhanced local GraphiQL IDE

The mock endpoint runs on localhost and simulates an AWS AppSync API connected to a DynamoDB table (defined at the GraphQL schema with the @model directive), all implemented locally on your developer machine.

We also ship tools to explore and interact with your GraphQL API locally. In particular, the terminal will print out a link to an instance of the GraphiQL IDE, where you can introspect the schema types, lookup documentation on any field or type, test API calls, and prototype your queries and mutations:

We’ve enhanced the stock GraphiQL experience with an open-source plugin that OneGraph have created to make your developer experience even nicer. In the Amplify GraphiQL Explorer, you’ll notice an UI generated for your specific GraphQL API that allows to quickly and easily explore, build GraphQL queries, mutations, or even subscriptions by simply navigating checkboxes. You can create, delete, update, read, or list data from your local DynamoDB tables in seconds.

With this new tooling, you can go from exploring your new GraphQL APIs locally to a full running application in a few minutes. Amplify is leveraging the power of open source to integrate the new local mocking environment with tools such as AWS-Utils and the GraphiQL Explorer to streamline the development experience and tighten the iteration cycle even further. If you’re interested in learning more about how and why the explorer was built, check out OneGraph’s blog on how they on-board users who are new to GraphQL.

What if you need to test and prototype real-time subscriptions? They also work seamlessly in the local environment. While amplify mock api is running, open another terminal window and execute yarn add aws-amplify to install some client dependencies then run yarn start.  In order to test, paste the code bellow to the src/App.js file in the React project, replacing the existing boilerplate code generated by the create-react-app command:

import React, { useEffect, useReducer } from "react";
import Amplify from "@aws-amplify/core";
import { API, graphqlOperation } from "aws-amplify";
import { createTodo } from "./graphql/mutations";
import { listTodos } from "./graphql/queries";
import { onCreateTodo, onUpdateTodo } from "./graphql/subscriptions";

import config from "./aws-exports";
Amplify.configure(config); // Configure Amplify

const initialState = { todos: [] };
const reducer = (state, action) => {
  switch (action.type) {
    case "QUERY":
      return { ...state, todos: action.todos };
    case "SUBSCRIPTION":
      return { ...state, todos: [...state.todos, action.todo] };
    default:
      return state;
  }
};

async function createNewTodo() {
  const todo = { name: "Use AppSync", description: "Realtime and Offline" };
  await API.graphql(graphqlOperation(createTodo, { input: todo }));
}
function App() {
  const [state, dispatch] = useReducer(reducer, initialState);

  useEffect(() => {
    getData();
    const subscription = API.graphql(graphqlOperation(onCreateTodo)).subscribe({
      next: eventData => {
        const todo = eventData.value.data.onCreateTodo;
        dispatch({ type: "SUBSCRIPTION", todo });
      }
    });
    return () => {
      subscription.unsubscribe();
    };
  }, []);

  async function getData() {
    const todoData = await API.graphql(graphqlOperation(listTodos));
    dispatch({ type: "QUERY", todos: todoData.data.listTodos.items });
  }

  return (
    <div>
      <div className="App">
        <button onClick={createNewTodo}>Add Todo</button>
      </div>
      <div>
        {state.todos.map((todo, i) => (
          <p key={todo.id}>
            {todo.name} : {todo.description}
          </p>
        ))}
      </div>
    </div>
  );
}
export default App;

Open two browser windows, one with the local GraphiQL instance and another one with the React App. As you can see in the following animation, you’ll be able to create items, see the mutations automatically triggering subscriptions and displaying the changes in the web app with no need to reload the browser:

 

If you want to access your NoSQL local data directly, as DynamoDB Local uses SQLite internally you can also access the data in the tables by using your IDE extension of choice:

Seamless transition between local and cloud environments

In the screenshot above you’ll notice the GraphQL API is in a “Create” state in the terminal section at the bottom, which means the backend resources are not deployed to the cloud yet. If we check the local “aws_exports.js” file generated by Amplify, which contains the identifiers of the resources created in different categories, you’ll notice the API endpoint is accessed locally and we’re using a fake API Key to authorize calls:

const awsmobile = {
    "aws_project_region": "us-east-1",
    "aws_appsync_graphqlEndpoint": "http://localhost:20002/graphql",
    "aws_appsync_region": "us-east-1",
    "aws_appsync_authenticationType": "API_KEY",
    "aws_appsync_apiKey": "da2-fakeApiId123456"
};

export default awsmobile;

What about testing more refined authentication requirements? You can still authenticate against a Cognito User Pool. The local testing server will honor the JWT tokens generated by Amazon Cognito and the rules defined by the @auth directive in your GraphQL schema. However, as Cognito is not running locally, you need to execute the command amplify push first to create the user pool and easily test users access with, for instance, the Amplify withAuthenticator higher order component on React. After that you can move back to the local environment with the command amplify mock api and authenticate calls with the generated JWT tokens. If you want to test directly from GraphiQL, after your API is configured to use Cognito, the Amplify GraphiQL Explorer provides a way to mock and change the username, groups, and email for a user and generate a local JWT token just by clicking the “Auth” button. The mocked values are used by the GraphQL Transformer @auth directive and any access rules:

After pushing and deploying the changes to the cloud with amplify push, the “aws_exports.js” file will be updated accordingly to point to the appropriate resources:

const awsmobile = {
    "aws_project_region": "us-east-1",
    "aws_appsync_graphqlEndpoint": "https://eriicnzxxxxxxxxxxxxx.appsync-api.us-east-1.amazonaws.com/graphql",
    "aws_appsync_region": "us-east-1",
    "aws_appsync_authenticationType": "API_KEY",
    "aws_appsync_apiKey": "da2-gttjhle72nf3pbfzfil2jy54ne"
};

export default awsmobile;

You can easily move back and forth between local and cloud environments as the identifiers in the exports file are updated automatically.

Local Debugging and Customizing VTL Resolvers

The local mocking environment also allows to easily customize and debug AppSync resolvers. You can edit VTL templates locally and check if they contain errors, including the line numbers causing problems, before pushing to AppSync. In order to do so, with the local API running, navigate to the folder amplify/backend/api/<your API name>/resolvers. You will see a list of resolver templates that the GraphQL Transformer automatically generated. You can modify any of them and, after saving changes, they are immediately loaded into the locally running API service with a message Mapping template change detected. Reloading. If you inject an error, for instance adding an extra curly brace, you will see a meaningful description of the problem and the line where the error was detected as shown below:

In case you stop the mock endpoint, for instance to push your changes to the cloud, all of the templates in the amplify/backend/api/<your API name>/resolvers folder will be removed except for any that you modified. When you subsequently push to the cloud these local changes will be automatically merged with your AppSync API.

As you are developing your app, you can always update the GraphQL schema located at amplify/backend/api/<your API name>/schema.graphql. You can add additional types and any of the supported GraphQL Transform directives then save your changes while the local server is still running. Any updates to the schema will be automatically detected and validated, then immediately hot reloaded into the local API. Whenever you’re happy with the backend, pushing and deploying the changes to the cloud is just one CLI command away.

Integrating Lambda Functions

Today you can already create and invoke Lambda functions written in Node.js locally with the Amplify CLI. Now how can you go even further and integrate lambda functions with GraphQL APIs in the new local mocking environment? It’s very easy to test customized business logic implemented with Lambda in your local API. Let’s start by creating a lambda function for your Amplify project with the command amplify add function to create a function called “factOfTheDay” as follows:

The function calls an external API to retrieve a fact related to the current date. Here’s the code:

const axios = require("axios");
const moment = require("moment");

exports.handler = function(event, _, callback) {
  let apiUrl = `http://numbersapi.com/`;
  let day = moment().format("D");
  let month = moment().format("M");
  let factOfTheDay = apiUrl + month + "/" + day;

  axios
    .get(factOfTheDay)
    .then(response => callback(null, response.data))
    .catch(err => callback(err));
};

Since the function above uses both the axios and moment libraries, we need to install them in the function folder amplify/backend/function/factOfTheDay/src by executing either npm install axios moment or yarn add axios moment. We can also test the function locally with the command amplify mock function factOfTheDay:

In our API we’ll add a field to the “Todo” type so every time we read or create records the Lambda function will be invoked to retrieve the facts of the current day. In order to do that we’ll take advantage of the GraphQL Transformer @function directive and point it to our lambda function by editing the file amplify/backend/api/localdev/schema.graphql:

type Todo @model {
  id: ID!
  name: String!
  description: String
  factOfTheDay: String @function(name: "factOfTheDay-${env}")
}

In order to test, we execute amplify mock to test locally all the mocked categories (in this case, API and Function) and access the local instance of the GraphiQL IDE in the browser:

As you can see, the GraphQL query is successfully invoking the local lambda function as well as retrieving data from the local DynamoDB table with a single call. In order to commit the changes and create the lambda function in the cloud, it’s just a matter of executing amplify push.

Integrating S3 storage

Most apps need access to some sort of content such as audio, video, images, PDFs and S3 is the best way to store these assets. How can we easily bring S3 to our local development environment?

First, let’s add storage to our amplify project with amplify add storage. If you have not previously added the “Auth” category in your project, the “Storage” category will also ask you to set this up and it is OK to do so. While this doesn’t impact local mocking as there are no authorization checks at this time for the Storage category, you must configure it first for cloud deployment to make sure the S3 bucket is secured according to your application requirements:

To start testing, execute amplify mock. Alternatively, you can run amplify mock storage to only mock the Storage category. If you have not pushed Auth resources to the cloud, you’ll need to do so by executing amplify auth push to create/update the Cognito resources as they’ll be needed to secure access to the actual S3 bucket.

You can use any of the storage operations provided by the Amplify library in your application code such as put, get, remove or list as well as use UI components to sign-up/sign-in users and interact with the local content. Files will be saved to your local Amplify project folder under amplify/mock-data/S3. When ready, execute amplify push to create the S3 bucket in the cloud.

Conclusion

With the new local mocking environment, we want to deliver a great experience to developers using the Amplify Framework. Now you can quickly spin up local resources, test, prototype, debug and generate code with open source tools, work on the front-end and create your fullstack serverless application in no time. On top of that, after you’re done and happy with your local development results, you can commit the code to GitHub and link your repository to the AWS Amplify Console which will provide a built-in CI/CD workflow. The console detects changes to the repository and automatically triggers builds to create your Amplify project backend cloud resources in multiple environments as well as publish your front-end web application to a content delivery network. Fullstack local development, testing, debugging, CI/CD, code builds and web publishing made much easier and faster for developers.

It’s just Day 1 for local development, mocking and testing on Amplify, what else would you like to see in our local mocking environment? Let us know if you have any ideas, feel free to create a feature request in our GitHub repository. Our team constantly monitors the repository and we’re always listening to your requests. Go build (now locally in your laptop)!

from AWS Mobile Blog

Configuring boto to validate HTTPS certificates

Configuring boto to validate HTTPS certificates

We strongly recommend upgrading from boto to boto3, the latest major version of the AWS SDK for Python. The previous major version, boto, does not default to validating HTTPS certificates for Amazon S3 when you are:

  1. Using a Python version less than 2.7.9 or
  2. Using Python 2.7.9 or greater and are connecting to S3 through a proxy

If you are unable to upgrade to boto3, you should configure boto to always validate HTTPS certificates. Be sure to test these changes. You can force HTTPS certification validation by either:

  1. Setting https_validate_certificates to True in your boto config file. For more information on how to use the boto config file, please refer to its documentation, or
  2. Setting validate_certs to True when instantiating an S3Connection:
    >>> from boto.s3.connection import S3Connection
    >>> conn = S3Connection(validate_certs=True)

To get the best experience, we always recommend remaining up-to-date with the latest version of the AWS SDKs and runtimes.

from AWS Developer Blog https://aws.amazon.com/blogs/developer/configure-boto-to-validate-https-certificates/

Supporting backend and internal processes with AWS AppSync multiple authorization types

Supporting backend and internal processes with AWS AppSync multiple authorization types

Imagine a scenario where you created a mobile or web application that uses a GraphQL API built on top of AWS AppSync and Amazon DynamoDB tables. Another backend or internal process such as an AWS Lambda function now needs to update data in the backend tables. A new feature in AWS AppSync lets you grant the Lambda function access to make secure GraphQL API calls through the unified AppSync API endpoint.

This post explores how to use the multiple authorization type feature to accomplish that goal.

Overview

In your application, you implemented the following:

  1. Users authenticate through Amazon Cognito user pools.
  2. Users query the AWS AppSync API to view your data in the app.
  3. The data is stored in DynamoDB tables.
  4. GraphQL subscriptions reflect changes to the data back to the user.

Your app is great. It works well. However, you may have another backend or internal process that wants to update the data in the DynamoDB tables behind the scenes, such as:

  • An external data-ingestion process to an Amazon S3 bucket
  • Real-time data gathered through Amazon Kinesis Data Streams
  • An Amazon SNS message responding to an outside event

For each of these scenarios, you want to use a Lambda function to go through a unified API endpoint to update data in the DynamoDB tables. AWS AppSync can serve as an appropriate middle layer to provide this functionality.

Walkthrough

An Amazon Cognito user pool authenticates and authorizes your API. Keep this in mind when considering the best way to grant the Lambda function access to make secure AWS AppSync API calls.

Choosing an authorization mode

AWS AppSync supports four different authorization types:

  • API_KEY: For using API keys
  • AMAZON_COGNITO_USER_POOLS: For using an Amazon Cognito user pool
  • AWS_IAM: For using IAM permissions
  • OPENID_CONNECT: For using your OpenID Connect provider

Before the launch of the multiple authorization type feature, you could only use one of these authorization types at a time. Now, you can mix and match them to provide better levels of access control.

To set additional authorization types, use the following schema directives:

  • @aws_api_key — A field uses API_KEY for authorization.
  • @aws_cognito_user_pools — A field uses AMAZON_COGNITO_USER_POOLS for authorization.
  • @aws_iam — A field uses AWS_IAM for authorization.
  • @aws_oidc — A field uses OPENID_CONNECT for authorization.

The AWS_IAM type is ideal for the Lambda function because the Lambda function is bound to an IAM execution role where you can specify the permissions this Lambda function can have. Do not use the API_KEY authorization mode; API keys are only recommended for development purposes or for use cases where it’s safe to expose a public API.

Understanding the architecture

Suppose that you have a log viewer web app that lets you view logging data:

  • It authenticates its users using an Amazon Cognito user pool and accesses an AWS AppSync API endpoint for data reads from a “Log” DynamoDB table.
  • Some backend processes publish log events and details to an SNS topic.
  • A Lambda function subscribes to the topic and invokes the AWS AppSync API to update the backend data store.

The following diagram shows the web app architecture.

The following code is your AWS AppSync GraphQL schema, with no authorization directives:

type Log {
  id: ID!
  event: String
  detail: String
}

input CreateLogInput {
  id: ID
  event: String
  detail: String
}

input UpdateLogInput {
  id: ID!
  event: String
  detail: String
}

input DeleteLogInput {
  id: ID!
}

type ModelLogConnection {
  items: [Log]
  nextToken: String
}

type Mutation {
  createLog(input: CreateLogInput!): Log
  updateLog(input: UpdateLogInput!): Log
  deleteLog(input: DeleteLogInput!): Log
}

type Query {
  getLog(id: ID!): Log
  listLogs: ModelLogConnection
}

type Subscription {
  onCreateLog: Log
    @aws_subscribe(mutations: ["createLog"])
  onUpdateLog: Log
    @aws_subscribe(mutations: ["updateLog"])
  onDeleteLog: Log
    @aws_subscribe(mutations: ["deleteLog"])
}

Configuring the AWS AppSync API

First, configure your AWS AppSync API to add the new authorization mode:

  • In the AWS AppSync console, select your API.
  • Under the name of your API, choose Settings.
  • For Default authorization mode, make sure it is set to Amazon Cognito user pool.
  • To the right of Additional authorization providers, choose New.
  • For Authorization mode, choose AWS Identity and Access Management (IAM), Submit.
  • Choose Save.

Now that you’ve set up an additional authorization provider, modify your schema to allow AWS_IAM authorization by adding @aws_iam to the createLog mutation. The new schema looks like the following code:

input CreateLogInput {
  id: ID
  event: String
  detail: String
}

input UpdateLogInput {
  id: ID!
  event: String
  detail: String
}

input DeleteLogInput {
  id: ID!
}

type ModelLogConnection {
  items: [Log]
  nextToken: String
}

type Mutation {
  createLog(input: CreateLogInput!): Log
    @aws_iam
  updateLog(input: UpdateLogInput!): Log
  deleteLog(input: DeleteLogInput!): Log
}

type Query {
  getLog(id: ID!): Log
  listLogs: ModelLogConnection
}

type Subscription {
  onCreateLog: Log
    @aws_subscribe(mutations: ["createLog"])
  onUpdateLog: Log
    @aws_subscribe(mutations: ["updateLog"])
  onDeleteLog: Log
    @aws_subscribe(mutations: ["deleteLog"])
}

type Log @aws_iam {
  id: ID!
  event: String
  detail: String
}

The @aws_iam directive is now authorizing the createLog mutation. Add the directive to the log type. Because directives work at the field level, also give AWS_IAM access to the log type. To do this, either mark each field in the log type with a directive or mark the log type with the @aws_iam directive.

You don’t have to explicitly specify the @aws_cognito_user_pools directive, because it is the default authorization type. Fields that are not marked by other directives are protected using the Amazon Cognito user pool.

Creating a Lambda function

Now that the AWS AppSync backend is set up, create a Lambda function. The function is triggered by an event published to an SNS topic, which contains logging event and detail information in the message body.

The following code example shows how the Lambda function is written in Node.js:

require('isomorphic-fetch');
const AWS = require('aws-sdk/global');
const AUTH_TYPE = require('aws-appsync/lib/link/auth-link').AUTH_TYPE;
const AWSAppSyncClient = require('aws-appsync').default;
const gql = require('graphql-tag');

const config = {
  url: process.env.APPSYNC_ENDPOINT,
  region: process.env.AWS_REGION,
  auth: {
    type: AUTH_TYPE.AWS_IAM,
    credentials: AWS.config.credentials,
  },
  disableOffline: true
};

const createLogMutation =
`mutation createLog($input: CreateLogInput!) {
  createLog(input: $input) {
    id
    event
    detail
  }
}`;

const client = new AWSAppSyncClient(config);

exports.handler = (event, context, callback) => {

  // An expected payload has the following format:
  // {
  //   "event": "sample event",
  //   "detail": "sample detail"
  // }

  const payload = event['Records'][0]["Sns"]['Message'];

  if (!payload['event']) {
    callback(Error("event must be provided in the message body"));
    return;
  }

  const logDetails = {
    event: payload['event'],
    detail: payload['detail']
  };

  (async () => {
    try {
      const result = await client.mutate({
        mutation: gql(createLogMutation),
        variables: {input: logDetails}
      });
      console.log(result.data);
      callback(null, result.data);
    } catch (e) {
      console.warn('Error sending mutation: ',  e);
      callback(Error(e));
    }
  })();
};

The Lambda function uses the AWS AppSync SDK to make a createLog mutation call, using the AWS_IAM authorization type.

Defining the IAM role

Now, define the IAM role that this Lambda function can assume. Grant the Lambda function appsync:GraphQL permissions for your API, as well as Amazon CloudWatch Logs permissions. You also must allow the Lambda function to be triggered by an SNS topic.

You can view the full AWS CloudFormation template that deploys the Lambda function, its IAM permissions, and supporting resources:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
  GraphQLApiEndpoint:
    Type: String
    Description: The https endpoint of an AppSync API
  GraphQLApiId:
    Type: String
    Description: The id of an AppSync API
  SnsTopicArn:
    Type: String
    Description: The ARN of the SNS topic that can trigger the Lambda function
Resources:
  AppSyncSNSLambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Description: A Lambda function that invokes an AppSync API endpoint
      Handler: index.handler
      Runtime: nodejs8.10
      MemorySize: 256
      Timeout: 10
      CodeUri: ./
      Role: !GetAtt AppSyncLambdaRole.Arn
      Environment:
        Variables:
          APPSYNC_ENDPOINT: !Ref GraphQLApiEndpoint

  AppSyncLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action: sts:AssumeRole
      Policies:
      - PolicyName: AppSyncLambdaPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Resource: arn:aws:logs:*
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:PutLogEvents
          - Effect: Allow
            Resource:
            - !Sub 'arn:aws:appsync:${AWS::Region}:${AWS::AccountId}:apis/${GraphQLApiId}*'
            Action:
            - appsync:GraphQL

  SnsSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Endpoint: !GetAtt AppSyncSNSLambda.Arn
      Protocol: Lambda
      TopicArn: !Ref SnsTopicArn

  LambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref AppSyncSNSLambda
      Action: lambda:InvokeFunction
      Principal: sns.amazonaws.com
      SourceArn: !Ref SnsTopicArn

Deploying the AWS CloudFormation template

Use the following two commands to deploy the AWS CloudFormation template. Make sure to replace all the CAPS fields with values specific to your AWS account:

aws cloudformation package --template-file "cloudformation.yaml" \
  --s3-bucket "<YOUR S3 BUCKET>" \
  --output-template-file "out.yaml"

aws cloudformation deploy --template-file out.yaml \
    --stack-name appsync-lambda \
    --s3-bucket "<YOUR S3 BUCKET>" \
    --parameter-overrides GraphQLApiEndpoint="<YOUR GRAPHQL ENDPOINT>" \
      GraphQLApiId="<YOUR GRAPHQL API ID>" \
      SnsTopicArn="<YOUR SNS TOPIC ARN>" \
    --capabilities CAPABILITY_IAM

Testing the solution

After both commands succeed, and your AWS CloudFormation template deploys, do the following:

1. Open the console and navigate to the SNS topic that you specified earlier.
2. Choose Publish message.
3. For the raw message body, enter the following:

{
   "event": "sample event",
   "detail": "sample detail"
}

4. Choose Publish message.

Navigate to the Log DynamoDB table that is your AWS AppSync API’s data source. You should see a new “sample event” record created using the CreateLog mutation.

Conclusion

With its new feature, AWS AppSync can now support multiple authorization types. This ability demonstrates how an AWS AppSync API serves as a powerful middle layer between multiple processes while being a secure API for end users.

As always, AWS welcomes feedback. Please submit comments or questions below.

Jane Shen is a cloud application architect in AWS Professional Services based in Toronto, Canada.

 

 

 

 

from AWS Mobile Blog

Introducing the ‘aws-rails-provisioner’ gem developer preview

Introducing the ‘aws-rails-provisioner’ gem developer preview

AWS is happy to announce that the aws-rails-provisioner gem for Ruby is now in developer preview and available for you to try!

What is aws-rails-provisioner?

The new aws-rails-provisioner gem is a tool that helps you define and deploy your containerized Ruby on Rails applications on AWS. It currently only supports AWS Fargate.

aws-rails-provisioner is a command line tool using the configuration file aws-rails-provisioner.yml to generate AWS Cloud Development Kit (CDK) stacks on your behalf. It automates provisioning AWS resources to run your containerized Ruby on Rails applications on AWS Fargate with a few commands. It can also generate a CI/CD AWS CodePipeline pipeline for your applications when you enable its CI/CD option.

Why use aws-rails-provisioner?

Moving a local Ruby on Rails application to a well-configured web application running on the cloud is a complicated task. While aws-rails-provisioner doesn’t change this into a “one-click” task, it helps ease the most monotonous and detail-oriented aspects of the job.

The aws-rails-provisioner gem shifts your primary focus to component-oriented definitions inside a concise aws-rails-provisioner.yml file. This file defines the AWS resources your application needs, such as container image environment, a database cluster engine, or Auto Scaling strategies.

The new gem handles default details—like VPC configuration, subnet placement, inbound traffic rules between databases and applications—for you. With CI/CD opt-in, aws-rails-provisioner can also generate and provision a predefined CI/CD pipeline, including a database migration phase.

For containerized Ruby on Rails applications that you already maintain on AWS, aws-rails-provisioner helps to keep the AWS infrastructure for your application as code in a maintainable way. This ease allows you to focus more on application development.

Prerequisites

Before using the preview gem, you must have the following resources:
• A Ruby on Rails application with Dockerfile
• A Docker daemon set up locally
• The AWS CDK installed (requires `Node.js` >= 8.11.x) npm i -g aws-cdk

Using aws-rails-provisioner

Getting started with aws-rails-provisioner is fast and easy.

Step 1: Install aws-rails-provisioner

You can download the aws-rails-provisioner preview gem from RubyGems.

To install the gem, run the following command:


gem install 'aws-rails-provisioner' -v 0.0.0.rc1

Step 2: Define your aws-rails-provisioner.yml file

The aws-rails-provisioner.yml configuration file allows you to define how and what components you want aws-rails-provisioner to provision to run your application image on Fargate.

An aws-rails-provisioner.yml file looks like the following format:

version: '0'
vpc:
	max_azs: 2
services:
	rails_foo:
    source_path: ../sandbox/rails_foo
    fargate:
      desired_count: 3
      public: true
      envs:
        PORT: 80
        RAILS_LOG_TO_STDOUT: true
    db_cluster:
      engine: aurora-postgresql
      db_name: myrails_db
      instance: 2	
	  scaling:
		max_capacity: 2
		on_cpu:
			target_util_percent: 80
			scale_in_cool_down: 300
	rails_bar:
		...

aws-rails-provisioner.yml file overview

The aws-rails-provisioner.yml file contains two parts: vpc and services. VPC defines networking settings, such as Amazon VPC hosting your applications and databases. It can be as simple as:

vpc:
  max_az: 3
  cidr: '10.0.0.0/21'
	enable_dns: true

Left unmodified, aws-rails-provisioner defines a default VPC with three Availability Zones containing public, private, and isolated subnets with a CIDR range of 10.0.0.0/21 and DNS enabled. If these default settings don’t meet your needs, you can configure settings yourself, such as in the following example, which defines the subnets with details:

vpc:
	subnets:
		application: # a subnet name
 			cidr_mark: 24
			type: private
		...

You can review the full range of VPC configuration options to meet your exact needs.

The services portion of aws-rails-provisioner.yml allows you to define your Rails applications, Database cluster, and Auto Scaling policies.
For every application, you can add their entry with identifiers like:

services:
	my_awesome_rails_app:
	  source_path: ../path/to/awesome_app # relative path from `aws-rails-provisioner.yml`
		...
	my_another_awesome_rails_app:
	  source_path: ./path/to/another_awesome_app # relative path from `aws-rails-provisioner.yml`
		...

When you run aws-rails-provisioner commands later, it takes the configuration values of a service—under fargate:, db_cluster:, and scaling: to provision a Fargate service fronted by an Application Load Balancer (DBClusters resource and Auto Scaling policies are optional for a service).

Database cluster

The db_cluster portion of aws-rails-provisioner.yml defines database settings for your Rails application. It currently supports Aurora PostgreSQL, Aurora MySQL, and Aurora. You can specify the engine version by appending engine_version to the command. You can also choose to provide a user name for your databases; if not, aws-rails-provisioner automatically generates username and password and stores it in AWS Secrets Manager.

To enable storage encryption for the Amazon RDS database cluster, provide kms_key_arn with the AWS KMS key ARN you use for storage encryption:

 my_awesome_rails_app:
	  source_path: ../path/to/awesome_app
	  db_cluster:
		engine: aurora-postgresql
      db_name: awesome_db
		username: myadmin

You can review the full list of db_cluster: configuration options to meet your specific needs.

AWS Fargate
The fargate: portion of aws-rails-provisioner.yml defines which Fargate services and Tasks that running your application image, for example:

my_awesome_rails_app:
	source_path: ../path/to/awesome_app
	fargate:
		public: true
		memory: 512
		cpu: 256
		container_port: 80
		envs:
			RAILS_ENV: ...
			RAILSLOGTO_STDOUT: ...
  ...

For HTTPs applications, you can provide certificate with a certificate ARN from AWS Certificate Manager. This automatically associates with the Application Load Balancer and sets container_port to 443. You can also provide a domain name and domain zone for your application under domain_name and domain_zone. If you don’t provide these elements, the system provides a default DNS address from the Application Load Balancer.

When providing environment variables for your application image, you don’t have to define DATABASE_URL by yourself; aws-rails-provisioner computes the value based on your db_cluster configuration. Make sure to update the config/database.yml file for your Rails application to recognize the DATABASE_URL environment variable.

You can review the full list of fargate: configuration options to meet your specific needs.

Scaling
You can also configure the Auto Scaling setting for your service. In this prototype stage, you can configure scaling policies on_cpu, on_metric, on_custom_metric, on_memory, on_request, or on_schedule.

my_awesome_rails_app:
	source_path: ../path/to/awesome_app
	scaling:
	  max_capacity: 10
    on_memory:
      target_util_percent: 80
      scale_out_cool_down: 200
    on_request:
      requests_per_target: 100000
      disable_scale_in: true
    ...

You can review the full list of scaling: configuration options to meet your specific needs.

Step 3: Build and deploy

With aws-rails-provisioner.yml defined, you can run a build command. Doing so bootstraps AWS CDK stacks in code, defining all the necessary AWS resources and connections for you.

Run the following:


aws-rails-provisioner build

This command initializes and builds a CDK project with stacks—installing all required CDK packages—leaving a deploy-ready project. By default, it generates an InitStack that defines the VPC and Amazon ECS cluster, hosting Fargate services. It also generates a FargateStack that defines a database cluster and a load-balanced, scaling Fargate service for each service entry.

When you enable --with-cicd, the aws-rails-provisioner also provides a pipeline stack containing source, build, database migration, and deploy stages for each service defined for you. You can enable CI/CD with the following command:


aws-rails-provisioner build --with-cicd

After the build completes, run the following deploy command to deploy all defined AWS resources:


aws-rails-provisioner deploy

Instead of deploying everything all at the same time, you can deploy stack by stack or application by application:


# only deploys the stack that creates the VPC and ECS cluster
aws-rails-provisioner deploy --init

# deploys the fargate service and database cluster when defined
aws-rails-provisioner deploy --fargate

# deploy the CI/CD stack
aws-rails-provisioner deploy --cicd

# deploy only `my_awesome_rails_app` application
aws-rails-provisioner deploy --fargate --service my_awesome_rails_app

You can check on the status of your stacks by logging in to the AWS console and navigating to AWS CloudFormation. Deployment can take several minutes.

Completing deployment leaves your applications running on AWS Fargate, fronted with the Application Load Balancer.

Step 4: View AWS resources

To see your database cluster, log in to the Amazon RDS console.

To see an ECS cluster created, with Fargate Services and tasks running, you can also check the Amazon ECS console.

To view the application via DNS address or domain address, check the Application Load Balancing dashboard.

Any applications with databases need rails migration to work. The generated CI/CD stack contains a migration phase. The following CI/CD section contains additional details.
To view all the aws-rails-provisioner command line options, run:


aws-rails-provisioner -h

Step 5: Trigger the CI/CD pipeline

To trigger the pipeline that aws-rails-provisioner provisioned for you, you must commit your application source code and Dockerfile with AWS CodeBuild build specs into an AWS CodeCommit repository. The aws-rails-provisioner gem automatically creates this repository for you.

To experiment with application image build and database migration on your own, try these example build spec files from the aws-rails-provisioner GitHub repo.

Conclusion
Although aws-rails-provisioner for RubyGems is currently under developer preview, it provides you with a powerful, time-saving tool. I would love for you to try it out and return with feedback for how AWS can improve this asset before its final launch. As always, you can leave your thoughts and feedback on GitHub.

from AWS Developer Blog https://aws.amazon.com/blogs/developer/introducing-the-aws-rails-provisioner-gem-developer-preview/

Announcing the new Predictions category in Amplify Framework

Announcing the new Predictions category in Amplify Framework

The Amplify Framework is an open source project for building cloud-enabled mobile and web applications. Today, AWS announces a new category called “Predictions” in the Amplify Framework.

Using this category, you can easily add and configure AI/ML uses cases for your web and mobile application using few lines of code. You can accomplish these use cases with the Amplify CLI and either the Amplify JavaScript library (with the new Predictions category) or the generated iOS and Android SDKs for Amazon AI/ML services. You do not need any prior experience with machine learning or AI services to use this category.

Using the Amplify CLI, you can setup your backend by answering simple questions in the CLI flow. In addition, you can orchestrate advanced use cases such as on-demand indexing of images to auto-update a collection in Amazon Rekognition. The actual image bytes are not stored by Amazon Rekognition. For example, this enables you to securely upload new images using an Amplify storage object which triggers an auto-update of the collection. You can then identify the new entities the next time you make inference calls using the Amplify library. You can also setup or import a SageMaker endpoint by using the “Infer” option in the CLI.

The Amplify JavaScript library with Predictions category includes support for the following use cases:

1. Translate text to a target language.
2. Generate speech from text.
3. Identify text from an image.
4. Identify entities from an image. (for example, celebrity detection).
5. Label real world entities within an image/document. (for example, recognize a scene, objects and activity in an image).
6. Interpret text to find insights and relationships in text.
7. Transcribe text from audio.
8. Indexing of images with Amazon Rekognition.

The supported uses cases leverage the following AI/ML services:

  • Amazon Rekognition
  • Amazon Translate
  • Amazon Polly
  • Amazon Transcribe
  • Amazon Comprehend
  • Amazon Textract

The iOS and Android SDKs now include support for SageMaker runtime which you can use to call inference on your custom models hosted on SageMaker. You can also extract text and data from scanned documents using the newly added support for Amazon Textract in the Android SDK. These services add to the list of existing AI services supported in iOS and Android SDKs.

In this post, you build and host a React.js web application that uses text in English language as an input and translates it to Spanish language. In addition, you can convert the translated text to speech in the Spanish language. For example, this type of use case can be added to a travel application, where you can type text in English and playback the translated text in a language of your choice. To build this app you use two capabilities from the Predictions category: Text translation and Generate speech from text.

Secondly, we go through the flow of indexing images to update a collection from the Amplify CLI and an application when using Amazon Rekognition.

Building the React.js Application

Prerequisites:

Install Node.js and npm if they are not already installed on your machine.

Steps

To create a new React.js app

Create a new React.js application using the following command:

$ npx create-react-app my-app

To set up your backend

Install and configure the Amplify CLI using the following command:

$ npm install -g @aws-amplify/cli
$ amplify configure

To create a new Amplify project

Run the following command from the root folder of your React.js application:

$ amplify init

Choose the following default options as shown below:

? Enter a name for the project: my-app
? Enter a name for the environment: dev
? Choose your default editor: Visual Studio Code
? Choose the type of app that you're building: javascript
? What javascript framework are you using: react
? Source Directory Path:  src
? Distribution Directory Path: build
? Build Command:  npm run-script build
? Start Command: npm run-script start
? Do you want to use an AWS profile? Yes
? Please choose the profile you want to use: default

To add text translation

Add the new Predictions category to your Amplify project using the following command:

$ amplify add predictions

The command line interface asks you simple questions to add AI/ML uses cases. There are 4 option: Identify, Convert, Interpret, and Infer.

  • Choose the “Convert” option.
  • When prompted, add authentication if you do not have one.
  • Select the following options in CLI:
? Please select from of the below mentioned categories: Convert
? You need to add auth (Amazon Cognito) to your project in order to add storage for user files. Do you want to add auth now? Yes
? Do you want to use the default authentication and security configuration? Default configuration
? How do you want users to be able to sign in? Username
? Do you want to configure advanced settings? No, I am done.
? What would you like to convert? Convert text into a different language
? Provide a friendly name for your resource: translateText6c4601e3
? What is the source language? English
? What is the target language? Spanish
? Who should have access? Auth and Guest users

To add text to speech

Run the following command to add text to speech capability to your project:

$ amplify add predictions
? Please select from of the below mentioned categories: Convert
? What would you like to convert? Convert text to speech
? Provide a friendly name for your resource: speechGeneratorb05d231c
? What is the source language? Mexican Spanish
? Select a speaker Mia - Female
? Who should have access? Auth and Guest users

To integrate the predictions library in a React.js application

Now that you set up the backend, integrate the Predictions library in your React.js application.

The application UI shows “Text Translation” and “Text to Speech” with a separate button for each functionality. The output of the text translation is the translated text in JSON format. The output of Text to Speech is an audio file that can be played from the application.

First, install the Amplify and Amplify React dependencies using the following command:

$ npm install aws-amplify aws-amplify-react

Next, open src/App.js and add the following code

import React, { useState } from 'react';
import './App.css';
import Amplify from 'aws-amplify';
import Predictions, { AmazonAIPredictionsProvider } from '@aws-amplify/predictions';
 
import awsconfig from './aws-exports';
 
Amplify.addPluggable(new AmazonAIPredictionsProvider());
Amplify.configure(awsconfig);
 
 
function TextTranslation() {
  const [response, setResponse] = useState("Input text to translate")
  const [textToTranslate, setTextToTranslate] = useState("write to translate");

  function translate() {
    Predictions.convert({
      translateText: {
        source: {
          text: textToTranslate,
          language : "en" // defaults configured in aws-exports.js
        },
        targetLanguage: "es"
      }
    }).then(result => setResponse(JSON.stringify(result, null, 2)))
      .catch(err => setResponse(JSON.stringify(err, null, 2)))
  }

  function setText(event) {
    setTextToTranslate(event.target.value);
  }

  return (
    <div className="Text">
      <div>
        <h3>Text Translation</h3>
        <input value={textToTranslate} onChange={setText}></input>
        <button onClick={translate}>Translate</button>
        <p>{response}</p>
      </div>
    </div>
  );
}
 
function TextToSpeech() {
  const [response, setResponse] = useState("...")
  const [textToGenerateSpeech, setTextToGenerateSpeech] = useState("write to speech");
  const [audioStream, setAudioStream] = useState();
  function generateTextToSpeech() {
    setResponse('Generating audio...');
    Predictions.convert({
      textToSpeech: {
        source: {
          text: textToGenerateSpeech,
          language: "es-MX" // default configured in aws-exports.js 
        },
        voiceId: "Mia"
      }
    }).then(result => {
      
      setAudioStream(result.speech.url);
      setResponse(`Generation completed, press play`);
    })
      .catch(err => setResponse(JSON.stringify(err, null, 2)))
  }

  function setText(event) {
    setTextToGenerateSpeech(event.target.value);
  }

  function play() {
    var audio = new Audio();
    audio.src = audioStream;
    audio.play();
  }
  return (
    <div className="Text">
      <div>
        <h3>Text To Speech</h3>
        <input value={textToGenerateSpeech} onChange={setText}></input>
        <button onClick={generateTextToSpeech}>Text to Speech</button>
        <h3>{response}</h3>
        <button onClick={play}>play</button>
      </div>
    </div>
  );
}
 
function App() {
  return (
    <div className="App">
      <TextTranslation />
      <hr />
      <TextToSpeech />
      <hr />
    </div>
  );
}
 
export default App;

In the previous code, the source language for translate is set by default in aws-exports.js. Similarly, the default language is set for text-to-speech in aws-exports.js. You can override these values in your application code.

To add hosting for your application

You can enable static web hosting for our react application on Amazon S3 by running the following command from the root of our application folder:

$ amplify add hosting

To publish the application run:

$ amplify publish

The application is now hosted on the AWS Amplify Console and you can access it at a link that looks like http://my-appXXXXXXXXXXXX-hostingbucket-dev.s3-website-us-XXXXXX.amazonaws.com/

On-demand indexing of images

The “Identify entities” option in Amplify CLI using Amazon Rekognition can detect entities like celebrities by default. However, you can use Amplify to index new entities to auto-update the collection in Amazon Rekognition. This enables you to develop advanced use cases such as uploading a new image and thereafter having the new entities in an input image being recognized if it matches an entry in the collection. Note that Amazon Rekognition does not store any image bytes.

Here is how it works on a high level for reference:

Note, if you delete the image from S3 the entity is removed from the collection.
You easily can setup the indexing feature from the Amplify CLI using the following flow:

$ amplify add predictions
? Please select from of the below mentioned categories Identify
? You need to add auth (Amazon Cognito) to your project in order to add storage for user files. Do you want to add auth now? Yes
? Do you want to use the default authentication and security configuration? Default configuration
? What would you like to identify? Identify Entities
? Provide a friendly name for your resource identifyEntities5a41fcea
? Would you like use the default configuration? Advanced Configuration
? Would you like to enable celebrity detection? Yes
? Would you like to identify entities from a collection of images? Yes
? How many entities would you like to identify 50
? Would you like to allow users to add images to this collection? Yes
? Who should have access? Auth users
? The CLI would be provisioning an S3 bucket to store these images please provide bucket name: myappentitybucket

If you have already setup storage from the Amplify CLI by running `amplify add storage`, the bucket that was created is reused. To upload images for indexing from the CLI, you can run `amplify predictions upload` and it prompts you for a folder location with your images.

After you have setup the backend through the CLI, you can use an Amplify storage object to add images to S3 bucket which will trigger the auto-indexing of images and update the collection in Amazon Rekognition.

In your src/App.js add the following function that uploads image test.jpg to Amazon S3:

function PredictionsUpload() {
  
 function upload(event) {
    const { target: { files } } = event;
    const [file,] = files || [];
    Storage.put('test.jpg', file, {
      level: 'protected',
      customPrefix: {
        protected: 'protected/predictions/index-faces/',
      }
    });
  }

  return (
    <div className="Text">
      <div>
        <h3>Upload to predictions s3</h3>
        <input type="file" onChange={upload}></input>
      </div>
    </div>
  );
}

Next, call the Predictions.identify() function to identify entities in an input image using the following code. Note, that we have to set “collections: true” in the call to identify.

function EntityIdentification() {
  const [response, setResponse] = useState("Click upload for test ")
  const [src, setSrc] = useState("");

  function identifyFromFile(event) {
    setResponse('searching...');
    
    const { target: { files } } = event;
    const [file,] = files || [];

    if (!file) {
      return;
    }
    Predictions.identify({
      entities: {
        source: {
          file,
        },
        collection: true
        celebrityDetection: true
      }
    }).then(result => {
      console.log(result);
      const entities = result.entities;
      let imageId = ""
      entities.forEach(({ boundingBox, metadata: { name, externalImageId } }) => {
        const {
          width, // ratio of overall image width
          height, // ratio of overall image height
          left, // left coordinate as a ratio of overall image width
          top // top coordinate as a ratio of overall image height
        } = boundingBox;
        imageId = externalImageId;
        console.log({ name });
      })
      if (imageId) {
        Storage.get("", {
          customPrefix: {
            public: imageId
          },
          level: "public",
        }).then(setSrc); 
      }
      console.log({ entities });
      setResponse(imageId);
    })
      .catch(err => console.log(err))
  }

  return (
    <div className="Text">
      <div>
        <h3>Entity identification</h3>
        <input type="file" onChange={identifyFromFile}></input>
        <p>{response}</p>
        { src && <img src={src}></img>}
      </div>
    </div>
  );
}

To learn more about the predictions category, visit our documentation.

Feedback

We hope you like these new features! Let us know how we are doing, and submit any feedback in the Amplify Framework Github Repository. You can read more about AWS Amplify on the AWS Amplify website.

from AWS Mobile Blog

Real-time streaming transcription with the AWS C++ SDK

Real-time streaming transcription with the AWS C++ SDK

Today, I’d like to walk you through how to use the AWS C++ SDK to leverage Amazon Transcribe streaming transcription. This service allows you to do speech-to-text processing in real time. Streaming transcription uses HTTP/2 technology to communicate efficiently with clients.

In this walkthrough, you build a command line application that captures audio from the computer’s microphone, sends it to Amazon Transcribe streaming, and prints out transcribed text as you speak. You use PortAudio (a third-party library) to capture and sample audio. PortAudio is a free, cross-platform library, so you should be able to build this on Windows, macOS, and Linux.

Note

Amazon Transcribe streaming transcription has a separate API from Amazon Transcribe, which also allows you to do speech-to-text, albeit not in real time.

Prerequisites

You must have the following tools installed to build the application:

  • CMake (preferably a recent version 3.11 or later)
  • A modern C++ compiler that supports C++11, a minimum of GCC 5.0, Clang 4.0, or Visual Studio 2015
  • Git
  • An HTTP/2 client
    • On *nix, you must have libcurl with HTTP/2 support installed on the system. To ensure that the version of libcurl you have supports HTTP/2, run the following command:
      • $ curl --version
      • You should see HTTP2 listed as one of the features.
    • On Windows, you must be running Windows 10.
  • An AWS account configured with the CLI

Walkthrough

The first step is to download and install PortAudio from the source. If you’re using Linux or macOS, you can use the system’s package manager to install the library (for example: apt, yum, or Homebrew).

  1. Browse to http://www.portaudio.com/download.html and download the latest stable release.
  2. Unzip the archive to a PortAudio directory.
  3. If you’re running Windows, run the following commands to build and install the library.
$ cd portaudio
$ mkdir build
$ cd build
$ cmake. 
$ cmake --build . --config Release

Those commands should build both a DLL and a static library. PortAudio does not define an install target when building on Windows. In the Release directory, copy the file named portaudio_static_x64.lib and the file named portaudio.h to another temporary directory. You need both files for the subsequent steps.

4. If you’re running on Linux or macOS, run the following commands instead.

$ cd portaudio
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_BUILD_TYPE=Release
$ cmake --build .
$ cmake --build . --target install

For this demonstration, you can safely ignore any warnings you see while PortAudio builds.

The next step is to download and install the Amazon Transcribe Streaming C++ SDK. You can use vcpkg or Homebrew to do that step. But I show you how to build the SDK from source.

$ git clone https://github.com/aws/aws-sdk-cpp.git
$ cd aws-sdk-cpp
$ mkdir build
$ cd build
$ cmake .. -DBUILD_ONLY=”transcribestreaming” -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF
$ cmake --build . --config Release
$ sudo cmake --build . --config Release --target install

Now, you’re ready to write the command line application.

Before diving into the code, I’d like to explain a few things about the structure of the application.

First, you need to tell PortAudio to capture audio from the microphone and write the sampled bits to a stream. Second, you want to simultaneously consume that stream and send the bits you captured to the service. To do those two operations concurrently, the application uses multiple threads. The SDK and PortAudio create and manage the threads. PortAudio is responsible for the audio capturing thread, and the SDK is responsible for the API communication thread.

If you have used the C++ SDK asynchronous APIs before, then you might notice the new pattern introduced with the Amazon Transcribe Streaming API. Specifically, in addition to the callback parameter invoked on request completion, there’s now another callback parameter. The new callback is invoked when the stream is ready to be written to by the application. More on that later.

The application can be a single C++ source file. However, I split the logic into two source files: one file contains the SDK-related logic, and the other file contains the PortAudio specific logic. That way, you can easily swap out the PortAudio specific code and replace it with any other audio source that fits your use case. So, go ahead and create a new directory for the demo application and save the following files into it.

The first file (main.cpp) contains the logic of using the Amazon Transcribe Streaming SDK:

// main.cpp
#include <aws/core/Aws.h>
#include <aws/core/utils/threading/Semaphore.h>
#include <aws/transcribestreaming/TranscribeStreamingServiceClient.h>
#include <aws/transcribestreaming/model/StartStreamTranscriptionHandler.h>
#include <aws/transcribestreaming/model/StartStreamTranscriptionRequest.h>
#include <cstdio>

using namespace Aws;
using namespace Aws::TranscribeStreamingService;
using namespace Aws::TranscribeStreamingService::Model;

int SampleRate = 16000; // 16 Khz
int CaptureAudio(AudioStream& targetStream);

int main()
{
    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
    Aws::InitAPI(options);
    {
        Aws::Client::ClientConfiguration config;
#ifdef _WIN32
        config.httpLibOverride = Aws::Http::TransferLibType::WIN_INET_CLIENT;
#endif
        TranscribeStreamingServiceClient client(config);
        StartStreamTranscriptionHandler handler;
        handler.SetTranscriptEventCallback([](const TranscriptEvent& ev) {
            for (auto&& r : ev.GetTranscript().GetResults()) {
                if (r.GetIsPartial()) {
                    printf("[partial] ");
                } else {
                    printf("[Final] ");
                }
                for (auto&& alt : r.GetAlternatives()) {
                    printf("%s\n", alt.GetTranscript().c_str());
                }
            }
        });

        StartStreamTranscriptionRequest request;
        request.SetMediaSampleRateHertz(SampleRate);
        request.SetLanguageCode(LanguageCode::en_US);
        request.SetMediaEncoding(MediaEncoding::pcm);
        request.SetEventStreamHandler(handler);

        auto OnStreamReady = [](AudioStream& stream) {
            CaptureAudio(stream);
            stream.flush();
            stream.Close();
        };

        Aws::Utils::Threading::Semaphore signaling(0 /*initialCount*/, 1 /*maxCount*/);
        auto OnResponseCallback = [&signaling](const TranscribeStreamingServiceClient*,
                  const Model::StartStreamTranscriptionRequest&,
                  const Model::StartStreamTranscriptionOutcome&,
                  const std::shared_ptr<const Aws::Client::AsyncCallerContext>&) { signaling.Release(); };

        client.StartStreamTranscriptionAsync(request, OnStreamReady, OnResponseCallback, nullptr /*context*/);
        signaling.WaitOne(); // prevent the application from exiting until we're done
    }

    Aws::ShutdownAPI(options);

    return 0;
}

The second file (audio-capture.cpp) contains the logic related to capturing audio from the microphone.

// audio-capture.cpp
#include <aws/core/utils/memory/stl/AWSVector.h>
#include <aws/core/utils/threading/Semaphore.h>
#include <aws/transcribestreaming/model/AudioStream.h>
#include <csignal>
#include <cstdio>
#include <portaudio.h>

using SampleType = int16_t;
extern int SampleRate;
int Finished = paContinue;
Aws::Utils::Threading::Semaphore pasignal(0 /*initialCount*/, 1 /*maxCount*/);

static int AudioCaptureCallback(const void* inputBuffer, void* outputBuffer, unsigned long framesPerBuffer,
    const PaStreamCallbackTimeInfo* timeInfo, PaStreamCallbackFlags statusFlags, void* userData)
{
    auto stream = static_cast<Aws::TranscribeStreamingService::Model::AudioStream>(userData);
    const auto beg = static_cast<const unsigned char*>(inputBuffer);
    const auto end = beg + framesPerBuffer * sizeof(SampleType);

    (void)outputBuffer; // Prevent unused variable warnings
    (void)timeInfo;
    (void)statusFlags;
 
    Aws::Vector<unsigned char> bits { beg, end };
    Aws::TranscribeStreamingService::Model::AudioEvent event(std::move(bits));
    stream->WriteAudioEvent(event);

    if (Finished == paComplete) {
        pasignal.Release(); // signal the main thread to close the stream and exit
    }

    return Finished;
}

void interruptHandler(int)
{
    Finished = paComplete;
}
 
int CaptureAudio(Aws::TranscribeStreamingService::Model::AudioStream& targetStream)
{
 
    signal(SIGINT, interruptHandler); // handle ctrl-c
    PaStreamParameters inputParameters;
    PaStream* stream;
    PaError err = paNoError;
 
    err = Pa_Initialize();
    if (err != paNoError) {
        fprintf(stderr, "Error: Failed to initialize PortAudio.\n");
        return -1;
    }

    inputParameters.device = Pa_GetDefaultInputDevice(); // default input device
    if (inputParameters.device == paNoDevice) {
        fprintf(stderr, "Error: No default input device.\n");
        Pa_Terminate();
        return -1;
    }

    inputParameters.channelCount = 1;
    inputParameters.sampleFormat = paInt16;
    inputParameters.suggestedLatency = Pa_GetDeviceInfo(inputParameters.device)->defaultHighInputLatency;
    inputParameters.hostApiSpecificStreamInfo = nullptr;

    // start the audio capture
    err = Pa_OpenStream(&stream, &inputParameters, nullptr, /* &outputParameters, */
        SampleRate, paFramesPerBufferUnspecified,
        paClipOff, // you don't output out-of-range samples so don't bother clipping them.
        AudioCaptureCallback, &targetStream);

    if (err != paNoError) {
        fprintf(stderr, "Failed to open stream.\n");        
        goto done;
    }

    err = Pa_StartStream(stream);
    if (err != paNoError) {
        fprintf(stderr, "Failed to start stream.\n");
        goto done;
    }
    printf("=== Now recording!! Speak into the microphone. ===\n");
    fflush(stdout);

    if ((err = Pa_IsStreamActive(stream)) == 1) {
        pasignal.WaitOne();
    }
    if (err < 0) {
        goto done;
    }

    Pa_CloseStream(stream);

done:
    Pa_Terminate();
    return 0;
}

There is one line in the audio-capture.cpp file that is related to the Amazon Transcribe Streaming SDK. That is the line in which you wrap the audio bits in an AudioEvent object and write the event to the stream. That is required regardless of the audio source.

And finally, here’s a simple CMake script to build the application:

# CMakeLists.txt
cmake_minimum_required(VERSION 3.11)
set(CMAKE_CXX_STANDARD 11)
project(demo LANGUAGES CXX)

find_package(AWSSDK COMPONENTS transcribestreaming)

add_executable(${PROJECT_NAME} "main.cpp" "audio-capture.cpp")

target_link_libraries(${PROJECT_NAME} PRIVATE ${AWSSDK_LINK_LIBRARIES})

if(MSVC)
    target_include_directories(${PROJECT_NAME} PRIVATE "portaudio")
    target_link_directories(${PROJECT_NAME} PRIVATE "portaudio")
    target_link_libraries(${PROJECT_NAME} PRIVATE “portaudio_static_x64”) # might have _x86 suffix instead
    target_compile_options(${PROJECT_NAME} PRIVATE "/W4" "/WX")
else()
    target_compile_options(${PROJECT_NAME} PRIVATE "-Wall" "-Wextra" "-Werror")
    target_link_libraries(${PROJECT_NAME} PRIVATE portaudio)
endif()

If you are building on Windows, copy the two files (portaudio.h and portaudio_static_x64.lib) from PortAudio’s build output that you copied earlier, into a directory named “portaudio” under the demo directory as such:

├── CMakeLists.txt

├── audio-capture.cpp

├── build

├── main.cpp

└── portaudio

    ├── portaudio.h

    └── portaudio_static_x64.lib

If you’re building on macOS or Linux, skip this step.

Now, “cd” into that directory and run the following commands:

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_BUILD_TYPE=Release
$ cmake --build . --config Release

These commands should create the executable ‘demo’ under the build directory. Go ahead and execute it and start speaking into the microphone when prompted. The transcribed words should start appearing printed on the screen.

Amazon Transcribe streaming transcription sends partial results while it is receiving audio input. You might notice that the partial results changing as you speak. If you pause long enough, it recognizes your silence as the end of a statement and sends a final result. As soon as you start speaking again, streaming transcription resumes sending partial results.

Clarifications

If you’ve used the asynchronous APIs in the C++ SDK previously, you might be wondering why the stream is passed through a callback instead of setting it as a property on the request directly.

The reason is that Amazon Transcribe Streaming is one of the first AWS services to use a new binary serialization and deserialization protocol. The protocol uses the notion of an “event” to group application-defined chunks of data. Each event is signed using the signature of the previous event as a seed. The first event is seeded using the request’s signature. Therefore, the stream must be “primed” by seeding it with the initial request’s signature before you can write to it.

 Summary

HTTP/2 opens new opportunities for using the web and AWS in real-time scenarios. With the support of the AWS C++ SDK, you can write applications that translate speech to text on the fly.

Amazon Transcribe Streaming follows the same pricing model as Amazon Transcribe. For more details, see Amazon Transcribe Pricing.

I’m excited to see what you build using this new technology. Send AWS your feedback on GitHub and on Twitter (#awssdk).

from AWS Developer Blog https://aws.amazon.com/blogs/developer/real-time-streaming-transcription-with-the-aws-c-sdk/

Announcing AWS Toolkit for Visual Studio Code

Announcing AWS Toolkit for Visual Studio Code

Visual Studio Code has become an enormously popular tool for serverless developers, partly due to the intuitive user interface. It’s also because of the rich ecosystem of extensions that can customize and automate so much of the development experience. We are excited to announce that the AWS Toolkit for Visual Studio Code extension is now generally available, making it even easier for the development community to build serverless projects using this editor.

The AWS Toolkit for Visual Studio Code entered developer preview in November of 2018 and is open-sourced on GitHub, allowing builders to make their contributions to the code base and feature set. The toolkit enables you to easily develop serverless applications, including creating a new project, local debugging, and deploying your project—all conveniently from within the editor. The toolkit supports Node.js, Python, and .NET.

Using the AWS Toolkit for Visual Studio Code, you can:

  • Test your code locally with step-through debugging in a Lambda-like environment.
  • Deploy your applications to the AWS Region of your choice.
  • Invoke your Lambda functions locally or remotely.
  • Specify function configurations such as an event payload and environment variables.

We’re distributing the AWS Toolkit for Visual Studio Code under the open source Apache License, version 2.0.

Installation

From Visual Studio Code, choose the Extensions icon on the Activity Bar. In the Search Extensions in Marketplace box, enter AWS Toolkit and then choose AWS Toolkit for Visual Studio Code as shown below. This opens a new tab in the editor showing the toolkit’s installation page. Choose the Install button in the header to add the extension.

After you install the AWS Toolkit for Visual Studio Code, you must complete these additional steps to access most of its features:

To use this toolkit to develop serverless applications with AWS, you must also do the following on the local machine where you install the toolkit:

For complete setup instructions, see Setting Up the AWS Toolkit for Visual Studio Code in the AWS Toolkit for Visual Studio Code User Guide.

Building a serverless application with the AWS Toolkit for Visual Studio Code

In this example, you set up a Hello World application using Node.js:

1. Open the Command Palette by choosing Command Palette from the View menu. Type AWS to see all the commands available from the toolkit. Choose AWS: Create new AWS SAM Application.

2. Choose the nodejs10.x runtime, specify a folder, and name the application Hello World. Press Enter to confirm these settings. The toolkit uses AWS SAM to create the application files, which appear in the Explorer panel.

3. To run the application locally, open the app.js file in the editor. CodeLenses appears above the handler function, showing options to Run Locally, Debug Locally, or Configure the function. Choose Run Locally.

This uses Docker to run the function on your local machine. You can see the Hello World output in the console window.

Step-through debugging of the application

One of the most exciting features in the toolkit is also one of the most powerful in helping you find problems in your code: step-through debugging.

The AWS Toolkit for Visual Studio Code brings the power of step-through debugging to serverless development. It lets you set breakpoints in your code and evaluate variable values and object states. You can activate this by choosing Debug Locally in the CodeLens that appears above your functions.

This enables the Debug view, displaying information related to debugging the current application. It also shows a command bar with debugging options and launch configuration settings. As your function executes and stops at a breakpoint, the editor shows current object state when you hover over the code and allows data inspection in the Variables panel.

For developers familiar with the power of the debugging support within Visual Studio Code, the toolkit now lets you use those features with serverless development of your Lambda functions. This makes it easier to diagnose problems quickly and iteratively, without needing to build and deploy functions remotely or rely exclusively on verbose logging to isolate issues.

Deploying the application to AWS

The deployment process requires an Amazon S3 bucket with a globally unique name in the same Region where your application runs

1. To create an S3 bucket from the terminal window, enter:

aws s3 mb s3://your-unique-bucket-name --region your-region

This requires the AWS CLI, which you can install by following the instructions detailed in the documentation. Alternatively, you can create a new S3 bucket in the AWS Management Console.

2. Open the Command Palette by choosing Command Palette from the View menu. Type AWS to see all the commands available from the toolkit. Choose AWS: Deploy a SAM Application.

3. Choose the suggested YAML template, select the correct Region, and enter the name of the bucket that you created in step 1. Provide a name for the stack, then press Enter. The process may take several minutes to complete. After it does, the Output panel shows that the deployment completed.

Invoking the remote function

With your application deployed to the AWS Cloud, you can invoke the remote function directly from the AWS Toolkit for Visual Studio Code:

  1. From the Activity Bar, choose the AWS icon to open AWS Explorer.
  2. Choose Click to add a region to view functions… and choose a Region from the list.
  3. Click to expand CloudFormation and Lambda. Explorer shows the CloudFormation stacks and the Lambda functions in this Region.
  4. Right-click the HelloWorld Lambda function in the Explorer panel and choose Invoke on AWS. This opens a new tab in the editor.
  5. Choose Hello World from the payload template dropdown and choose Invoke. The output panel displays the ‘hello world’ JSON response from the remote function.

Congratulations! You successfully deployed a serverless application to production using the AWS Toolkit for Visual Studio Code. The User Guide includes more information on the toolkit’s other development features.

Conclusion

In this post, I demonstrated how to deploy a simple serverless application using the AWS Toolkit for Visual Studio Code. Using this toolkit, developers can test and debug locally before deployment, and modify AWS resources defined in the AWS SAM template.

For example, by using this toolkit to modify your AWS SAM template, you can add an S3 bucket to store images or documents, add a DynamoDB table to store user data, or change the permissions used by your functions. It’s simple to create or update stacks, allowing you to quickly iterate until you complete your application.

AWS SAM empowers developers to build serverless applications more quickly by simplifying AWS CloudFormation and automating the deployment process. The AWS Toolkit for Visual Studio Code takes the next step, allowing you to manage the entire edit, build, and deploy process from your preferred development environment. We are excited to see what you can build with this tool!

from AWS Developer Blog https://aws.amazon.com/blogs/developer/announcing-aws-toolkit-for-visual-studio-code/

Serverless data engineering at Zalando with the AWS CDK

Serverless data engineering at Zalando with the AWS CDK

This blog was authored by Viacheslav Inozemtsev, Data Engineer at Zalando, an active user of the serverless technologies in AWS, and an early adopter of the AWS Cloud Development Kit.

 

Infrastructure is extremely important for any system, but it usually doesn’t carry business logic. It’s also hard to manage and track. Scripts and templates become large and complex, reproducibility becomes nearly impossible, and the system ends up in an unknown state, that’s hard to diagnose.

The AWS Cloud Development Kit (AWS CDK) is a unified interface between you, who defines infrastructure, and AWS CloudFormation, which runs infrastructure in AWS. It’s a powerful tool to create complex infrastructure in a concise and, at the same time, clean way.

Let’s quickly go through the phases of how a company’s infrastructure can be handled in AWS.

1. You start with the UI of AWS, and create resources directly, without any idea of stacks. If something goes wrong, it’s hard to reproduce the same resources in different environments, or even recreate the production system. This approach works only to initially explore the possibilities of the service, and not for production deployments.

2. You begin using the AWS API, most commonly by calling it using the AWS Command Line Interface (AWS CLI). You create bash or Python scripts that contain multiple calls to the API, done in imperative way, trying to reach a state where resources are created without failing. But the scripts (especially bash scripts) become large and messy, and multiply. They’re not written in a standard way, and maintenance becomes extremely tough.

3. You start migrating the scripts to AWS CloudFormation templates. This is better, but although AWS CloudFormation is a declarative and unified way of defining resources in AWS, the templates also get messy pretty quickly, because they mirror what happens in the scripts. You have to create the templates, usually by copying parts from other existing templates, or by using the AWS CloudFormation console. In either case, they grow big pretty fast. Human error occurs. Verification can only be done at the time of submission to AWS CloudFormation. And you still need scripts to deploy. Overall, it’s better, but still not optimal.

4. This is where the AWS CDK makes the difference.

Advantages of using the AWS CDK

The AWS CDK is a layer on top of AWS CloudFormation that simplifies generation of resource definitions. The AWS CDK is a library that you can install using npm package manager, and then use from your CLI. The AWS CDK isn’t an AWS service, and has no user interface. Code is the input and AWS CloudFormation templates are the output.

The AWS CDK does three things:

  • It generates an AWS CloudFormation template from your code. This is a major advantage, because writing those templates manually is tedious and error prone.
  • It generates a list of changes that would happen to the existing stack, if you applied the generated template. This gives better visibility and reduces human error.
  • It deploys the template to AWS CloudFormation, where changes get applied. This simplifies the deployment process and reduces it to a call of a single AWS CDK command.

The AWS CDK enables you to describe your infrastructure in code. Although this code is written in one of these imperative programming languages — TypeScript, Python, Java, C# — it’s declarative. You define resources in the style of object-oriented programming, as if they were simply instances of classes. When this code gets compiled, an engineer can quickly see if there are problems. For example, if AWS CloudFormation requires a parameter for a certain resource, and the code doesn’t provide it, you see a compile error.

Defining resources in a programming language enables flexible, convenient creation of complex infrastructure. For example, one can pass into a function a collection of references to a certain resource type. Additional parameters can be prepared before creating the stack, and passed into the constructor of the stack as arguments. You can use a configuration file, and load settings from it in the code wherever you need it. Auto-completion helps you explore possible options and create definitions faster. An application can consist of two stacks, and output of the creation of one stack can be passed as an argument for the creation of the other.

Every resource created by the AWS CDK is always part of a stack. You can have multiple stacks in one application, and those stacks can be dependent. That’s necessary when one stack contains resources that are referenced by resources in another stack.

You can see how convenient it is. Now let’s have a look at a few common patterns we encountered while using the AWS CDK at Zalando.

Common patterns – reusable constructs

The AWS CDK has four main classes:

  • Resource – Represents a single resource in AWS.
  • Stack – Represents an AWS CloudFormation stack, and contains resources.
  • App – Represents the whole application, and can contain multiple stacks.
  • Construct – A group of resources that are bound together for a purpose.

The following diagram shows how these classes interact.

Every resource in AWS has a corresponding class in the AWS CDK that extends class Resource. All those classes start with the Cfn prefix, which stands for CloudFormation Native, and comprise the so called level 1 of the AWS CDK. Level 1 can be used, but usually it isn’t, because there is level 2.

Level 2 contains standard constructs that represent groups of resources usually used together. For example, a construct Function from the @aws-cdk/aws-lambda package represents not just a Lambda function, but also contains two resources: a Lambda function and its IAM role. This simplifies creation of a Lambda function, because it always requires a role to run with. There are many other level 2 constructs that are handy, and that cover most standard use cases.

There is also level 3, which is a way to share your own constructs between your projects, or even with other people anywhere in the world. The idea is that one can come up with a generic enough construct, push it to the npm registry — the same way, as, for example, @aws-cdk/aws-lambda package is pushed — and then, anyone can install this package and use that construct in their code. This is an extremely powerful way of sharing definitions of resources.

Now I am going to talk about a few patterns we uncovered so far working on data pipelines. These are:

  • Periodic action
  • Reactive processing
  • Fan-out with Lambda functions
  • Signaling pub/sub system
  • Long-running data processing

Periodic action

The first pattern I want to show is a combination of a cron rule and a Lambda function. Lambda gets triggered every N minutes to orchestrate other parts of the system, for example, to pull new messages from an external queue, and push them into an internal Amazon SQS queue.

The following code example shows how the AWS CDK helps you manage defining infrastructure. In the code, a class CronLambdaConstruct extends class Construct of the AWS CDK. Inside there’s only a constructor. Every resource, defined in the constructor, will be created as part of the stack containing this construct. In this case the resources are a Lambda function, its IAM role, a rule with rate, and a permission to allow the rule to trigger the function.

import { Construct, Duration } from '@aws-cdk/core'
import { Function, S3Code, Runtime } from '@aws-cdk/aws-lambda'
import { Rule, Schedule } from '@aws-cdk/aws-events'
import { LambdaFunction } from '@aws-cdk/aws-events-targets';

export interface CronLambdaProps {
  codeReference: S3Code,
  handlerFullName: string,
  lambdaTimeout: number,
  lambdaMemory: number
}

export class CronLambdaConstruct extends Construct {
  constructor(parent: Construct, id: string, props: CronLambdaProps) {
    super(parent, id);

    const myLambda = new Function(this, 'MyLambda', {
      runtime: Runtime.JAVA_8,
      code: props.codeReference,
      handler: props.handlerFullName,
      timeout: Duration.seconds(props.lambdaTimeout),
      memorySize: props.lambdaMemory,
      reservedConcurrentExecutions: 1
    })

    new Rule(this, 'MyCronRule', {
      enabled: true,
      schedule: Schedule.rate(Duration.minutes(1)),
      targets: [new LambdaFunction(myLambda)]
    })
  }
}

The following picture depicts the output of the cdk diff, a command showing the plan of changes. Although only Function and Rule objects are explicitly initiated in the code, four resources will be created. This is because Function hides creation of its associated role, and Rule object results in creation of the rule itself together with the permission to trigger Lambda function.

Reactive processing

 

One of the major patterns useful in data pipelines is to connect an Amazon SQS queue to a Lambda function as its source, using event source mapping. This technique allows you to immediately process everything coming to the queue with the Lambda function, as new messages arrive. Lambda scales out as needed. You can detach the queue to stop processing, for example, for troubleshooting purposes.

The most interesting part is that you can attach another SQS queue to the source queue, known as a dead letter queue. A retry policy specifies how many attempts to process a message must fail before it will go to the dead letter queue. Then the same Lambda function can use the dead letter queue as its second source, normally in a detached state, to reprocess failed messages later. As soon as you fix the problem that caused Lambda to fail, you can attach the dead letter queue to the Lambda function. Then it can try to reprocess all the failed messages. This activity takes a click of a button in the Lambda console.

import { Construct, Duration } from '@aws-cdk/core'
import { Queue } from '@aws-cdk/aws-sqs'
import { Function, S3Code, Runtime, EventSourceMapping } from '@aws-cdk/aws-lambda';

export interface QueueDLQLambdaProps {
  codeReference: S3Code,
  handlerFullName: string,
  lamdbaTimeout: number,
  lambdaMemory: number
}

export class QueueDLQLambdaConstruct extends Construct {
  constructor(parent: Construct, id: string, props: QueueDLQLambdaProps) {
    super(parent, id);
 
    const inputQueue = new Queue(this, 'InputQueue', {
      maxMessageSizeBytes: 262144,
      retentionPeriod: Duration.days(14),
      visibilityTimeout: Duration.seconds(props.lamdbaTimeout)
    })

    const dlq = new Queue(this, 'DLQ', {
      maxMessageSizeBytes: 2621244,
      retentionPeriod: Duration.days(14),
      visibilityTimeout: Duration.seconds(props.lamdbaTimeout)
    })

    const myLambda = new Function(this, 'MyLambda', {
      runtime: Runtime.JAVA_8,
      code: props.codeReference,
      handler: props.handlerFullName,
      timeout: Duration.seconds(props.lamdbaTimeout),
      memorySize: props.lambdaMemory,
    reservedConcurrentExecutions: 1
    })

    new EventSourceMapping(this, 'InputQueueSource', {
      enabled: true,
      batchSize: 10,
      eventSourceArn: inputQueue.queueArn,
      target: myLambda
    })

    new EventSourceMapping(this, 'DLQSource', {
      enabled: false,
      batchSize: 10,
      eventSourceArn: dlq.queueArn,
      target: myLambda
    })
  }
}

Fan-out with Lambda functions

Now, let’s reuse the two previous constructs to get another pattern: fan-out of processing.

First there is an orchestrating Lambda function that runs with reserved concurrency 1. This prevents parallel executions. This Lambda function prepares multiple activities that can vary, depending on certain parameters or state. For each unit of processing, the orchestrating Lambda function sends a message to an SQS queue that is attached as a source of another Lambda function with high concurrency. Processing Lambda then scales out and executes the tasks.

A good example of this pattern is processing all objects by a certain prefix in Amazon S3. The first Lambda function does the listing of the prefix, and for each listed object it sends a message to the SQS queue. The second Lambda gets invoked once per object, and does the job.

This pattern doesn’t necessarily need to start with a cron Rule. The orchestrating Lambda function can be triggered by other means. For example, it can be reactively invoked by an s3:ObjectCreated event. I’ll talk about this next.

Signaling pub/sub system

Another important approach, which we use across Zalando, is signaling to others that datasets are ready for consumption. This appeared to be a great opportunity to use AWS CDK constructs.

In this use case, there is a central Amazon SNS topic receiving notifications about all datasets that are ready. Anyone in the company can subscribe to the topic by filtering on the dataset names.

import { Construct, Duration } from '@aws-cdk/core'
import { Queue } from '@aws-cdk/aws-sqs'
import { Topic } from '@aws-cdk/aws-sns'
import { Function, S3Code, Runtime, EventSourceMapping } from '@aws-cdk/aws-lambda'
import { PolicyStatement } from '@aws-cdk/aws-iam';

export interface QueueLambdaTopicProps {
  codeReference: S3Code,
  handlerFullName: string,
  lambdaTimeout: number,
  lambdaMemory: number,
  teamAccounts: string[]
}

export class QueueLambdaTopicConstruct extends Construct {

  public readonly centralTopic: Topic

  constructor(parent: Construct, id: string, props: QueueLambdaTopicProps) {
    super(parent, id);

    const inputQueue = new Queue(this, 'InputQueue', {
      maxMessageSizeBytes: 262144,
      retentionPeriod: Duration.days(14),
      visibilityTimeout: Duration.seconds(props.lambdaTimeout)
    })

    const enrichmentLambda = new Function(this, 'EnrichmentLambda', {
      runtime: Runtime.JAVA_8,
      code: props.codeReference,
      handler: props.handlerFullName,
      timeout: Duration.seconds(props.lambdaTimeout),
      memorySize: props.lambdaMemory
    })

    new EventSourceMapping(this, 'InputQueueSource', {
      enabled: true,
      batchSize: 10,
      eventSourceArn: inputQueue.queueArn,
      target: enrichmentLambda
    })

    this.centralTopic = new Topic(this, 'CentralTopic', {
      displayName: 'central-dataset-readiness-topic'
    })

    this.centralTopic.grantPublish(enrichmentLambda)

    const subscribeStatement = new PolicyStatement({
      actions: ['sns:Subscribe'],
      resources: [this.centralTopic.topicArn]
    })

    props.teamAccounts.forEach(teamAccount => {
      subscribeStatement.addArnPrincipal('arn:aws:iam::${teamAccount}:root')
    })

    this.centralTopic.addToResourcePolicy(subscribeStatement)
  }
}

To subscribe, there is a construct that contains an SQS queue and an SNS subscription, which receives notifications from the central SNS topic and moves them to the queue.

import { Construct, Duration } from '@aws-cdk/core'
import { Queue } from '@aws-cdk/aws-sqs'
import { Topic } from '@aws-cdk/aws-sns'
import { SqsSubscription } from '@aws-cdk/aws-sns-subscriptions';

export interface SubscriptionQueueProps {
  visibilityTimeoutSec: number,
  centralTopicARN: string
}

export class SubscriptionQueueConstruct extends Construct {

  public readonly centralTopic: Topic

  constructor(parent: Construct, id: string, props: SubscriptionQueueProps) {
    super(parent, id);

    const centralTopic = Topic.fromTopicArn(this, 'CentralTopic', props.centralTopicARN)

    const teamQueue = new Queue(this, 'CentralTopic', {
      maxMessageSizeBytes: 262144,
      retentionPeriod: Duration.days(14),
      visibilityTimeout: Duration.seconds(props.visibilityTimeoutSec)
    })

    centralTopic.addSubscription(new SqsSubscription(teamQueue))
  }
}

Here we have two constructs. The first construct contains the input SQS queue, the Lambda function that enriches messages with MessageAttributes (to enable filtering), and the central SNS topic for publishing about ready datasets. The second construct is for the subscriptions.

Long-running data processing

The last pattern I want to describe is a long-running processing job. In data engineering this is a Hello World problem for infrastructure. Essentially, you want to run a batch processing job using Apache Spark or a similar framework, and as soon as it’s finished, trigger downstream actions. In AWS this can be solved using AWS Step Functions, a service where you can create state machines (in other words – workflows) of any complexity. The picture below depicts the pattern being described.

To do this using the AWS CDK, you can create a StateMachine object from the @aws-cdk/aws-stepfunctions package. It triggers a job, waits for its completion or failure, and then reports the result. You can even combine it with the previous pattern, to send a signal to the queue of the central SNS topic, informing it that the dataset is ready for downstream processing. For this purpose, integration of Step Functions with SQS via CloudWatch event Rules can be used.

import { Construct, Duration } from '@aws-cdk/core'
import { IFunction } from '@aws-cdk/aws-lambda'
import { Condition, Choice, Fail, Task, Wait, Errors, StateMachine, Succeed, WaitTime } from '@aws-cdk/aws-stepfunctions'
import { InvokeFunction } from '@aws-cdk/aws-stepfunctions-tasks';

export interface StepfunctionLongProcessProps {
  submitJobLambda: IFunction,
  checkJobStateLambda: IFunction
}

export class StepfunctionLongProcessConstruct extends Construct {

  constructor(parent: Construct, id: string, props: StepfunctionLongProcessProps) {
    super(parent, id);

    const retryProps = {
      intervalSeconds: 15,
      maxAttempts: 3,
      backoffRate: 2,
      errors: [Errors.ALL]
    }

    const failedState = new Fail(this, 'Processing failed', {
      error: 'Processing failed'
    })

    const success = new Succeed(this, 'Successful!')

    const submitJob = new Task(this, 'Sumbit job', {
      task: new InvokeFunction(props.submitJobLambda),
      inputPath: '$.jobInfo',
      resultPath: '$.jobState'
    }).addRetry(retryProps)

    const waitForCompletion = new Wait(this, 'Wait for completion...', {
      time: WaitTime.duration(Duration.seconds(30))
    })

    const checkJobState = new Task(this, 'Check job state', {
      task: new InvokeFunction(props.checkJobStateLambda),
      inputPath: '$.jobInfo',
      resultPath: '$.jobState'
    }).addRetry(retryProps)
 
    const isJobCompleted = new Choice(this, 'Is processing done?')

    const stepsDefinition = submitJob
      .next(waitForCompletion)
      .next(checkJobState)
      .next(isJobCompleted
        .when(Condition.stringEquals('$.jobState', 'RUNNING'), waitForCompletion)
        .when(Condition.stringEquals('$.jobState', 'FAILED'), failedState)
        .when(Condition.stringEquals('$.jobState', 'SUCCESS'), success)
    )

    new StateMachine(this, 'StateMachine', {
      definition: stepsDefinition,
      timeout: Duration.hours(1)
    })
  }
}

Conclusion

The AWS CDK provides an easy, convenient, and flexible way to organize stacks and resources in the AWS Cloud. The CDK not only makes the development lifecycle faster, but also enables safer and more structured organization of the infrastructure. We, in the Core Data Lake team of Zalando, have started migrating every part of our infrastructure to the AWS CDK, whenever we refactor old systems or create new ones. And we clearly see how we benefit from it, and how resources become a commodity instead of being a burden.

 

About the Author

Viacheslav Inozemtsev is a Data Engineer at Zalando, has 8 years of data/software engineering experience, a Specialist degree in applied mathematics and a MSc degree in computer science, mostly on the topics of data processing and analysis. In the Core Data Lake team of Zalando he is building an internal central data platform on top of AWS. He is an active user of the serverless technologies in AWS, and an early adopter of the AWS Cloud Development Kit.

 

Disclaimer

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

from AWS Developer Blog https://aws.amazon.com/blogs/developer/serverless-data-engineering-at-zalando-with-the-aws-cdk/

Amplify Framework adds support for AWS Lambda Triggers in Auth and Storage categories

Amplify Framework adds support for AWS Lambda Triggers in Auth and Storage categories

The Amplify Framework is an open source project for building cloud-enabled mobile and web applications. Today, we’re happy to announce that you can set up AWS Lambda triggers directly from the Amplify CLI.

Using Lambda triggers, you can call event-based Lambda functions for authentication, database actions, and storage operations from other AWS services like Amazon Simple Storage Service (Amazon S3), Amazon Cognito, and Amazon DynamoDB. Now, the Amplify CLI allows you to enable and configure these triggers. The CLI further simplifies the process by providing you with trigger templates that you can customize to suit your use case.

The Lambda trigger capabilities for Auth category include:

  1. Add Google reCaptcha Challenge: This enables you to add Google’s Captcha implementation to your mobile or web app.
  2. Email verification link with redirect: This trigger enables you to define an email message that can be used for an account verification flow.
  3. Add user to a Amazon Cognito User Pools group: This enables you to add a user to an Amazon Cognito User Pools group upon account registration.
  4. Email domain filtering: This enables you to define email domains that would like to allow or block during sign up.
  5. Custom Auth Challenge Flow: This enables you add custom auth flow to your mobile and web application by providing a basic skeleton which you can edit to achieve custom authentication in your application.

The Lambda trigger for Storage category can be added when creating or updating the storage resource using the Amplify CLI.

Auth Triggers for Authentication with Amazon Cognito

The Lambda triggers for Auth enable you to build custom authentication flows in your mobile and web application.
These triggers can be associated with Cognito User Pool operations such as sign-up, account confirmation, and sign-in. The Amplify CLI provides the template triggers for capabilities listed above which can be customized to suit your use case.

A custom authentication flow using Amazon Cognito User Pools typically comprises of 3 steps:

  1. Define Auth Challenge: Determines the next challenge in the custom auth flow.
  2. Create Auth Challenge: Creates a challenge in the custom auth flow.
  3. Verify Auth Challenge: : Determines if a response is correct in a custom auth flow.

When you add auth to your Amplify project, the CLI asks you if you want to add capabilities for custom authentication. It generates the trigger templates for each step in your custom auth flow depending on the capability chosen. The generated templates can be edited as per your requirements. Once complete, you push your project using ‘amplify push’ command. For more information on these capabilities, refer to our documentation.

Here is an example of how you add one of these custom auth capabilities in your application.

Adding a new user to group in Amazon Cognito

Using Amazon Cognito User Pools, you can create and manage groups, add users to groups, and remove users from groups. With groups, you can create collections of users to manage their permissions or to represent different user types.

You can now use the Amplify CLI to add a Lambda trigger to add a user to a group after they have successfully signed up. Here’s how it works.

Creating the authentication service and configuring the Lambda Trigger

From the CLI, create a new Amplify project with the following command:

amplify init

Next, add authentication with the following command:

amplify add auth

The command line interface then walks you through the following steps for adding authentication:

? Do you want to use the default authentication and security configuration? Default configuration
? How do you want users to be able to sign in? Username
? Do you want to configure advanced settings? Yes, I want to make some additional changes.
? What attributes are required for signing up? Email
? Do you want to enable any of the following capabilities?
 ◯ Add Google reCaptcha Challenge
 ◯ Email Verification Link with Redirect
❯◉ Add User to Group
 ◯ Email Domain Filtering (blacklist)
 ◯ Email Domain Filtering (whitelist)
 ◯ Custom Auth Challenge Flow (basic scaffolding - not for production)
 ? Enter the name of the group to which users will be added. STUDENTS
 ? Do you want to edit the local PostConfirmation lambda function now? No
 ? Do you want to edit your add-to-group function now? Yes

The interface should then open the appropriate Lambda function template, which you can edit in your text editor. The code for the function will be located at amplify/backend/function/<functionname>/src/add-to-group.js.

The Lambda function that you write for this example adds new users to a group called STUDENTS when they have an .edu email address. This function triggers after the signup successfully completes.

Update the Lambda function add-to-group.js with the following code:

const aws = require('aws-sdk');

exports.handler = (event, context, callback) => {
  const cognitoidentityserviceprovider = new aws.CognitoIdentityServiceProvider({ apiVersion: '2016-04-18' });

  const email = event.request.userAttributes.email.split('.')
  const domain = email[email.length - 1]

  if (domain === 'edu') {
    const params = {
      GroupName: 'STUDENTS',
      UserPoolId: event.userPoolId,
      Username: event.userName,
    }
  
    cognitoidentityserviceprovider.adminAddUserToGroup(params, (err) => {
      if (err) { callback(err) }
      callback(null, event);
    })
  } else {
    callback(null, event)
  }
}

To deploy the authentication service and the Lambda function, run the following command:

amplify push

Now, when a user signs up with an .edu email address, they are automatically placed in the STUDENTS group.

Integrating with a client application

Now that you have the authentication service up and running, let’s integrate with a React application that signs the user in and recognizes that the user is part of the STUDENTS group.

First, install the Amplify and Amplify React dependencies:

npm install aws-amplify aws-amplify-react

Next, open src/index.js and add the following code to configure the app to recognize the Amplify project configuration:

import Amplify from 'aws-amplify'
import config from './aws-exports'
Amplify.configure(config)

Next, update src/App.js. The code recognizes the user groups of a user after they have signed in and displays a welcome message if the user is in the STUDENTS group.

// src/App.js
import React, { useEffect, useState } from 'react'
import logo from './logo.svg'
import './App.css'
import { withAuthenticator } from 'aws-amplify-react'
import { Auth } from 'aws-amplify'

function App() {
  const [isStudent, updateStudentInfo] = useState(false)
  useEffect(() => {
    /* Get the AWS credentials for the current user from Identity Pools.  */
    Auth.currentSession()
      .then(cognitoUser => {
        const { idToken: { payload }} = cognitoUser
        /* Loop through the groups that the user is a member of */
        /* Set isStudent to true if the user is part of the STUDENTS group */
        payload['cognito:groups'] && payload['cognito:groups'].forEach(group => {
          if (group === 'STUDENTS') updateStudentInfo(true)
        })
      })
      .catch(err => console.log(err));
  }, [])
  return (
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        { isStudent && <h1>Welcome, Student!</h1> }
      </header>
    </div>
  );
}

export default withAuthenticator(App, { includeGreetings: true })

Now, if the user is part of the STUDENTS group, they will get a specialized greeting.

Storage Triggers for Amazon S3 and Amazon DynamoDB

With this release, we’ve also enabled the ability to setup Lambda triggers for Amazon S3 and Amazon DynamoDB. This means you can execute a Lambda function on events such as create, update, read, and write. When adding or configuring storage from the Amplify CLI, you now have the option to add and configure a storage trigger.

Resizing an image with AWS Lambda and Amazon S3

Let’s take a look at how to use one of the new triggers to resize an image into a thumbnail after it has been uploaded to an S3 bucket.

From the CLI, create a new Amplify project with the following command:

amplify init

Next, add storage with the following command:

amplify add storage

The interface then walks you through the add storage setup.

? Please select from one of the below mentioned services: Content (Images, audio, video, etc.)
? You need to add auth (Amazon Cognito) to your project in order to add storage for user files. Do you want to add auth now? Yes
? Do you want to use the default authentication and security configuration? Default configuration
? How do you want users to be able to sign in? Username
? Do you want to configure advanced settings? No, I am done.
? Please provide a friendly name for your resource that will be used to label this category in the project: MyS3Example
? Please provide bucket name: <YOUR_UNIQUE_BUCKET_NAME>
? Who should have access: Auth and guest users
? What kind of access do you want for Authenticated users? create/update, read, delete
? What kind of access do you want for Guest users? read
? Do you want to add a Lambda Trigger for your S3 Bucket? Y
? Select from the following options: Create a new function

The CLI then generates a code template for the new Lambda function, which you can modify as needed. It will be located at amplify/backend/function/<functionname>/src/index.js.

Replace the code in index.js with the following code:

const gm = require('gm').subClass({ imageMagick: true })
const aws = require('aws-sdk')
const s3 = new aws.S3()

const WIDTH = 100
const HEIGHT = 100

exports.handler = (event, context, callback) => {
  const BUCKET = event.Records[0].s3.bucket.name

  /* Get the image data we will use from the first record in the event object */
  const KEY = event.Records[0].s3.object.key
  const PARTS = KEY.split('/')

  /* Check to see if the base folder is already set to thumbnails, if it is we return so we do not have a recursive call. */
  const BASE_FOLDER = PARTS[0]
  if (BASE_FOLDER === 'thumbnails') return

  /* Stores the main file name in a variable */
  let FILE = PARTS[PARTS.length - 1]

  s3.getObject({ Bucket: BUCKET, Key: KEY }).promise()
    .then(image => {
      gm(image.Body)
        .resize(WIDTH, HEIGHT)
        .setFormat('jpeg')
        .toBuffer(function (err, buffer) {
          if (err) {
            console.log('error storing and resizing image: ', err)
            callback(err)
          }
          else {
            s3.putObject({ Bucket: BUCKET, Body: buffer, Key: `thumbnails/thumbnail-${FILE}` }).promise()
            .then(() => { callback(null) })
            .catch(err => { callback(err) })
          }
        })
    })
    .catch(err => {
      console.log('error resizing image: ', err)
      callback(err)
    })
}

You can trace the execution of the code above in Amazon CloudWatch Logs on an event such as upload to the S3 bucket.

Next, install the GraphicsMagick library in the Lambda function directory. This ensures that you have the needed dependencies to execute the Lambda function.

cd amplify/backend/function/<functionname>/src

npm install gm

cd ../../../../../

To deploy the services, run the following command:

amplify push

Next, visit the S3 console, open your bucket and upload an image. Once the upload has completed, a folder named thumbnails will be created and the resized image will be stored there.

To learn more about creating storage triggers, check out the documentation.

Feedback

We hope you like these new features! As always, let us know how we’re doing, and submit any requests in the Amplify Framework GitHub Repository. You can read more about AWS Amplify on the AWS Amplify website.

from AWS Mobile Blog