Tag: AWS Startups Blog

Singapore’s Flourishing Startup Scene & What Helped Build It

Singapore’s Flourishing Startup Scene & What Helped Build It

Nighttime aerial view of Singapore

Since gaining its independence just over 50 years ago, the city-state of Singapore has quickly developed into a hub for startups. Just look at the cumulative venture capital raised by companies headquartered within its boundaries over the last 5 years: ~$5 billion. By that measure, Singapore ranks among the world’s top 10, next to locales that have had a much longer head start, like Germany, France, and the UK. So what’s their secret? After speaking with over a dozen founders of Singapore-based startups, a few themes emerged.

A Founder-friendly Government

Perhaps more so than for other types of companies, startups see time as money. Getting your business set up and scaling as quickly as possible can make or break an early-stage company. This fact is not lost on the Singaporean government, which has taken steps to save much of this precious resource for its startups by cutting down the time needed to get up-and-running.

Rob Roach, CTO and Co-founder of Perx, a SaaS startup that offers a platform for both loyalty and channel marketing management, points to this time savings as a main benefit for early-stage companies that call Singapore home. “Overall, it’s a very business-friendly city, so it’s quick and easy to get things started,” he says. In other words, you can count how long it takes to set-up a company in days, versus weeks.

Similarly, Roach argues the speediness of getting approval from the government for foreign employees is a huge help. Time from submission to approval is typically less than 4 weeks, per Roach. When compared with the nightmare scenarios seen in many other countries’ immigration policies, it’s no wonder founders increasingly look to Singapore.

This thoughtful approach is also exemplified by the Monetary Authority of Singapore (MAS), which acts as the central bank and financial regulatory authority for the city-state. Stemming from its position as a finance and trade powerhouse, many Singaporean startups find themselves looking to innovate within those industries, which means oversight from MAS—not necessary a bad thing in Singapore.

“MAS has done a tremendous amount of groundwork on regulatory frameworks to show us how best to work with them, what to do, and what not to do,” says Aananth Solaiyappan, CTO and Co-founder of WeInvest, a provider of robo-advisory tech.

That said, it’s not always entirely easy to meet the high bar set by MAS for operating. It can still be confusing at points, but in those cases many companies turn to AWS for help. Acknowledging that the process can be opaque at times, AWS has written a free guide to help startups navigate working with MAS, which can be found here.

The guide has been leveraged by companies including CCR Manager, which is building a digital platform for managing the behind-the-scenes work needed for international shipping. Per the startup’s CTO and Co-founder Andre Siregar, “The point-by-point guide provided by AWS was very useful, helping us work with MAS to scale to the 91+ financial institutions currently on our platform.”

So, what’s one way to attract founders and startups to your city? Do the upfront work to save them time and play to your strengths.

A Diverse Mix of People

“…if you’re looking for pound-for-pound the most food, best food, and most diverse selection of food maybe anywhere on the planet, you are most definitely talking about Singapore.” -the late Anthony Bourdain

Replace the word “food” with “entrepreneurs,” and the same would apply. The city-state’s unique mix of people is often cited as a unique advantage for its companies. Just as the melting pot of cultures creates new flavor profiles and dishes, the combination also is a boon for startups, a theme that Ronnie Tan, Managing Director at gaming company gumi Asia, identifies.

“For the people of Singapore, you grow up many different races. This multiculturalism makes it really easy to work alongside people from different countries,” he says. “With talent coming from all over the globe, it’s a great advantage.”

And while Singapore has historically been very multicultural, the aforementioned government support for bringing in a solid talent pool has played a large role in maintaining that spirit into the new digital age.

Robert Ross, CTO at Singapore Life, a digital insurance startup, exemplifies just that, both with his background and his company. Hailing from Illinois, Ross found himself in Singapore for the past three years after spending time in New York and Hong Kong.

For Ross, Singapore’s intrinsic diversity enables him to focus on attracting and developing the best talent, with a unique mix of people and backgrounds coming naturally with that process. And for his startup, this means they get a variety of perspectives on how to solve the many problems faced with early-stage companies.

Looking back, it’s no wonder that during our recent trip to the southeast asian city we talked with founders from over seven countries, including Finland, France, India, Indonesia, Poland, and Japan, all now living in Singapore.

What’s Next?

Although Singapore has found a groove as a hub for startups in Southeast Asia, there is still work to be done to take it to the next level. A topic mentioned throughout our interviews were the lack of lighthouse wins for the community, which many would argue is the most important indicator of success. All the venture capital invested must lead to outsized outcomes for the model to work, after all.

That said, exits take time and the Singapore startup scene is developing. While the huge acquisitions and IPOs have yet to start rolling in, the earlier stages are ripe with activity, led by the likes of the aforementioned companies: Perx, Singapore Life, WeInvest, and CCRManager.

And while Singapore can serve as a great example for how to build an entrepreneurial ecosystem similar to the one seen half a world away in Silicon Valley, it’s perhaps better looked at as an example that there is more than one way to do so. Just look at Singapore’s signature strength, diversity, something that is by no means an area where companies in the Bay Area excel.

So, for cities or countries looking to emulate the magic in Northern California, perhaps it’s best to look within to see what your part of the world is uniquely positioned to offer. There’s more than one way to build a community.

from AWS Startups Blog

Bambu’s B2B Robo-advising Platform Democratizes Financial Planning

Bambu’s B2B Robo-advising Platform Democratizes Financial Planning

Historically, financial planning was considered an option only for the white-collar crowd—those with lots of money would find a personal financial advisor to help them decide how much of their wealth to disburse to which asset classes.

That being said, over the past decade, a host of fintech startups have popped up around the world, looking to both capitalize on existing business and bring investing ability to new markets. Bambu, a Singapore-based company that built a robo-advising platform, is focused on the latter.

Founded in 2016 by Ned Phillips and Aki Ranin, Bambu is on a mission to apply the latest technology to financial advisory, thereby lowering the cost of such services and making them available to people who previously would’ve dismissed them as exclusively for the rich and powerful.

But while other global leaders in the robo-advising industry have found success going directly to the consumer (think Betterment or Wealthfront), Bambu is taking a different route by targeting established financial institutions.

Why? It’s really a product of where and how the company started, per Ranin.

“If you look at the places robo-advising took off, it’s large domestic markets: the USA, Europe, and some in China. At the center of this is the ease of moving through regulations and getting approval. For example, in Europe they have pan-European licensing processes that make it easy to get regulated in one country but do business across many. Asia, on the other hand, is fragmented, making the independent model a lot more difficult.”

This fragmentation led to Bambu working directly with banks on platform adoption, as the banks already have the necessary regulatory approvals. They then use the startup’s robo-advisory services to better serve their customers.

Using that model, Bambu was able to be the first robo-advisor to launch in its home of Singapore, relying heavily on AWS from the get-go, both from a technical perspective and a partnership perspective to help navigate the regulatory barriers surrounding financial services.

While today it seems like a given that most companies work on the cloud in some capacity, it wasn’t always the case, especially in emerging markets. Ranin describes how as recently as three years ago when Bambu started, the environment was less than welcoming, if not entirely skeptical.

“At the time, it wasn’t clear path for companies to launch utilizing cloud services, especially in highly regulated industries like ours. AWS and its local team in Singapore were hugely helpful not only in getting us ramped up, but also in understanding regulations, and making a case for why using the cloud actually increases security. Once that’s established there really isn’t any downside.”

With a solid stance in Singapore and activity underway in other Southeast Asian countries, including Vietnam, Malaysia, and Indonesia, Bambu has its sights set on geographic expansion. The platform was initially built for Asian countries, but it has since garnered much interest from the global community, a development that surprised Ranin and his team.

“We’ve seen inquiries come in from places all over the world, such as the U.S., South America, Europe, and the Middle East. Moving forward, Bambu will put more focus on building the platform for scale, making it easily launchable wherever the customer may be.”

Once completed, the startup’s next iteration will further the company’s goal of bringing financial advisory services to the masses, not only on home soil, but also for people around the world.

from AWS Startups Blog

RINGS.TV’s ‘Loops’ Connects Arabs Through Its Live Social Platform

RINGS.TV’s ‘Loops’ Connects Arabs Through Its Live Social Platform

For Bryan Loh and the team at RINGS.TV, jumping at the opportunity to build a social live streaming platform specifically for Arabian people seemed like a no brainer. The success of various similar products around the world — Periscope and Facebook Live, for example — clearly showed that people were ready and waiting for this type of service. What those platforms lacked, however, was knowledge of the Arabian market, something that Loh and his team had in spades.

Bryan Loh, Cofounder of Rings.tv and LoopsPrior to beginning the work on what would eventually become his current company, Loh spent years as a manager at MOZAT, a mobile-development company that’s been serving the Middle Eastern market for over a decade. During his time there, Loh and the team kicked off the development of a live streaming platform, eventually spinning it out of the parent company in 2016 after receiving interest for investors.

Since then, RINGS.TV’s platform, dubbed Loops, has exploded in popularity, much of which Loh attributes to the company’s focus on building specifically for the Arabian market and their unique needs.

Demonstrating this focus are culturally informed features that Loops currently has to facilitate interaction. Loh points out, “When you observe gatherings in Arab countries, games are a frequent activity. We looked to capture that same experience, but put it online.” This observation led the company to integrate the five games it now offers to its users, which include chess and Baloot, a popular card game.

The idea to build a live streaming platform specifically for the Arabian market seems to have fueled massive growth, too. After spinning out just 3 years ago, RINGS.TV’s Loops platform is at nearly 6 million users, a feat of scale that AWS was critical in achieving.

Per Loh, “A main benefit we’ve seen from partnering with AWS is the ability to scale-up quickly. Since launching, we’ve increased our usage to roughly 130 virtual machines, and we’re leveraging the CDN offering as well. Also, historically we’d operated using the Singapore availability zone, but are very excited to leverage the newly launched Dubai AZ to better serve our customers.”

As for what’s next for the growing startup? For Loh, doubling down on what’s working is front-of-mind.

“Our platform currently supports nine people virtually hanging out,” says Loh, “but we have plans to further expand that. We’re also looking to increase the number of games we offer from five to ten by the end of 2019.”

from AWS Startups Blog

Knorex Helps Advertisers Find the Right Person at the Right Time

Knorex Helps Advertisers Find the Right Person at the Right Time

The rapid spread of technology over the past couple of decades has been a boon to advertisers. Gone are the days of relying only on traditional mediums like television or billboards. The internet, and rise of the mobile web, made available countless new touch points for companies to reach their customers.

This expansion has not led to exclusively positive results for the industry, however. With more options than ever to choose from, advertisers need to be diligent in researching and understanding the right time and the way to reach their customers.

That’s where Knorex comes in.

Founded in 2010, the company offers an online bidding platform that enables companies to buy advertising on many different channels, including web, mobile, TV, and email. Leveraging mountains of data and machine learning, Knorex’s XPO platform can also offer guidance to its customers on what to buy and when. That’s not all, though. Wrapped up in this experience are also tools to build ad creative and manage campaigns at scale.

Justin Choo, Founder and CEO

The root of the company can be traced back to founder Justin Choo’s previous startup, where he co-built a similar real-time stock trading system. Now, he and the team of ~130 at Knorex are applying a similar model to the advertising industry in an effort to help companies serve relevant ads to their target markets.

Currently, the team sees the platform being largely utilized by enterprises in industries including travel and hospitality, both of which, Justin points out, rely heavily on online sales and performance marketing.

To fuel this rapidly developing platform, Knorex turned to AWS. As Justin puts it, “We decided to be cloud-based early on, and AWS was the clear choice. The company continually introduces new features or services, and we’re sure to have our team evaluate each.”

And how does the company decide which service to implement? It’s actually simple.

Per Justin, “We’re always evaluating to see if a service saves us time or money on something that is non-core to our advertising business. It’s a constant calculus we’re conducting.”

As for what’s next for the Singapore-based company, Justin has his eyes set on continuing the global expansion the team has been experiencing over recent years. Knorex opened up its first US office in mid 2018, and is showing no signs of slowing down.

from AWS Startups Blog

How Daivergent Is Providing Autistic People With a Stepping-Stone to a Career in Tech

How Daivergent Is Providing Autistic People With a Stepping-Stone to a Career in Tech

While most people find artificial intelligence pretty interesting, much of the work that constitutes the foundation of such technology would bore most of them to tears. Take, for example, building training datasets, a job that requires a person to do the same thing—tagging images and video, identifying and labeling text sentiment—over and over without allowing their mind to wander. “Anytime you have any models, you need large numbers of training sets. When it comes to training-set generation, you can have a great model, but if your training set’s not great, your model’s going to struggle no matter what,” says Rahul Mahida, co-founder and CTO of Daivergent. “When I worked as a data engineer, we tried to solve it ourselves; basically, we got bored and then our quality slipped.” Outsourcing the labor tended to produce a similarly mediocre result.

As a data scientist, Byran Dai also knew how difficult it was to reliably generate high-quality training datasets. He and Mahida found a solution to this workplace problem in a shared aspect of their personal lives: “We both [have] family members with autism. We know the type of things they enjoy doing,” says Mahida. “People on the autism spectrum tend to prefer jobs that are defined but repetitive, but they’re still able to hit that same level of complexity as anyone else…They don’t get that falloff based on repetitiveness where they’re getting bored, where their quality’s dropping. They can stick with it at a high quality for much, much longer.” Founded in 2017 by Dai and Mahida, Daivergent is a platform that connects companies with high-volume data needs—most of them artificial intelligence-related—to a remote workforce comprised of people with autism.

While building the platform, Daivergent consulted The Arc, an organization that provides services to people with autism, on how to tailor its interface and communication for its workforce; Mahida says that they’re still in “constant contact” with the group’s experts and social workers. After signing up for Daivergent, members (the title the company gives its contractors) “can put in as little or as much time as they want,” a few hours or all day. “It’s truly full flexibility because for us the goal is, in order to get the maximum spread amongst this population, we want to lower the barrier as much as possible,” says Mahida. He adds that a fair wage for members is priced in to every project Daivergent takes on, and that the company has a matching process intended to ensure that members are only given assignments to which they are well-suited, so they don’t end up “spending a lot of time just to do a few tasks and then not make much money.”

According to Mahida, the unemployment rate among autistic people, including those who are educated, is around 90 percent. He says that the typical Daivergent member is either not working or working very part-time, generally doing something like stocking shelves. Though Mahida is quick to note that there’s nothing wrong with that type of job, he points out that, “Often [autistic people] have gone to school. They’ve done high school, some have college degrees, often in tech. They want to work in tech. They want to work doing things for AI and for the web and they, up until now, have been just completely shut out from that possibility.”

“Our ethos as a company is to have people on the autism spectrum be able to get jobs, to be able to find meaningful work, and gain a meaningful life from it. For this, we actually see kind of a bifurcation. We see two sides,” says Mahida.

For some members, Daivergent can serve as a stepping-stone to a career. In addition to facilitating work experience for people who might not otherwise get the opportunity to demonstrate their aptitude and abilities, the platform offers video-based skills courses, job-interview coaching, and groups to help members get to know each other. Mahida says that members have told him that Daivergent “provides them a lot of stability, provides them a lot of confidence, to the point where it improves their chances when they are going for other jobs.”

For other members, Daivergent can simply serve as a venue for earning money on their own terms. “Some people, they don’t want in-person employment, or they don’t want a full-time job, and they want this variety of tasks. That’s okay, too. And for them, we might see people being longtime members of Daivergent, and just doing that as their primary day-to-day.”

Regardless of which path Daivergent members choose, Mahida believes that tech firms would be well-served by learning to integrate autistic people into their staff. “At its heart, autism is a social and communication disorder, and that makes it harder for someone on the spectrum to maybe work in a standard environment. But it also makes it harder for a manager who’s not maybe aware or used to working with someone in this population in how to communicate and how to work with them. And, for them, it’s a risk that they’d just rather not take.” But tech, with its general openness to risk and willingness to “edit [processes] to be focused more on getting things done rather than looking for a certain kind of criteria for a person,” is a sector where people with autism—a large number of whom are interested in the industry—should be able to thrive. Daivergent hopes to prove that “if you lower those barriers on social and communication, their technical ability shows through at the highest level.”

from AWS Startups Blog

Eko Offers an All-in-One Platform for Managing Frontline Workforces

Eko Offers an All-in-One Platform for Managing Frontline Workforces

For startups, it’s rarely a straight line from ideation to scaling successfully. The road to product-market fit and beyond is frequently laden with twists and turns, pivots and adjustments based on customer feedback.

When looking back at the founding of Eko, the provider of a mobile-first platform for managing frontline workforces, company CEO and Co-founder Korawad Chearavanont describes just such a road. “It all started back in 2012 with a group messaging service for students. That didn’t end up working out, so we decided to switch to secure messaging for corporates, which set us on a path to where we are today.”

That pivot has proved successful, with Eko currently boasting staff across three continents. And while the Bangkok-based startup has the lion’s share of its workforce in Thailand, it also has a sizable presence in London, Amsterdam, Berlin, New York, and Austin.

The secret to the scale? Chearavanont points to how his team concentrates on listening to its customers, most of whom use Eko to help manage their staff in the retail and hospitality industries. Per the CEO, “When expanding Eko’s functionalities, we looked to our customers to help guide what we developed. Their direct feedback is where you can trace the origins of many of our features back to, like the fielding of leave requests and the overall workflow engine.”

Through this feedback loop, Chearavanont and his team were able to build an all-in-one solution for its users, reducing the number of apps needed to manage front-line staff from five down to one: Eko.

As for plans for the future, it’s helpful to look where they’ve been. As Chearavanont puts it, “Eko historically has seen much success around its home region, but we’re increasingly shifting our focus to Europe and the US.”

To help fuel both past and future expansion, Eko looks to AWS, a partner that has been with the company since the beginning.

“One of the things that’s been really great about AWS is the ability to easily scale and deploy in every region in which we operate. For example, in 2017 we started selling a large number of license into China, which was hard to support with our existing infrastructure based in Singapore. Through AWS, we were able to quickly set up servers in Beijing to support this new customer base. We then mirrored that approach in Europe and the [United] States.”

from AWS Startups Blog

Meeting SLAs for Data Pipelines on Amazon EMR With Unravel

Meeting SLAs for Data Pipelines on Amazon EMR With Unravel

Guest post by George Demarest, Senior Director of Product Marketing, Unravel

A household name in global media analytics – let’s call them MTI –  is using Unravel to support their data operations (DataOps) on Amazon EMR to establish and protect their internal service level agreements (SLAs) and get the most out of their Spark applications and pipelines.  Amazon EMR was an easy choice for MTI as the platform to run all their analytics. To start with, getting up and running is simple. There is nothing to install, no configuration required etc. and you can get to a functional cluster in a few clicks. This enabled MTI to focus most of their efforts in building out analytics that would benefit their business instead of having to spend time and money on acquiring the skillset needed for setting up and maintaining Hadoop deployments by themselves. MTI was very quickly able to get a point that they were running 10’s of thousands of jobs per week. About 70% of which are Spark, with the remaining 30% of workloads running on Hadoop, or more specifically Hive/MapReduce.

Among the most common complaints and concerns about optimizing big data clusters and applications is the amount of time it takes to root-cause issues like application failures or slowdowns or to figure out what needs to be done to improve performance. Without context, performance and utilization metrics from the underlying data platform and the Spark processing engine can laborious to collect and correlate, and difficult to interpret.

Unravel employs a frictionless method of collecting relevant data about the full data stack, running applications, cluster resources, datasets, users, business units and projects. Unravel then aggregates and correlates this data into the Unravel data model and then applies a variety of analytical techniques to put that data into a useful context. Unravel utilizes EMR bootstrap Actions to distribute (non-intrusive) sensors on each node of a new cluster that are needed for collecting granular application level data which in turn is used to optimize applications.

Diagram of Unravel Data's Amazon AWS/EMR architecture

Unravel’s Amazon AWS/EMR architecture

MTI has prioritized their goals for big data based on two main dimensions that are reflected in the Unravel product architecture: Operations and Applications.

Optimizing Data Operations

For MTI’s cluster level SLAs and operational goals for their big data program, they identified the following requirements:

●      Reduce time needed for troubleshooting and resolving issues.

●      Improve cluster efficiency and performance.

●      Improve visibility into cluster workloads.

●      Provide usage analysis

Reducing Time to Identify and Resolve Issues

One of the most basic requirements for creating meaningful SLAs is to set goals for identifying problems or failures – known as Mean Time to Identification (MTTI) – and the resolution of those problems – known as Mean Time to Resolve (MTTR). MTI executives set a goal of 40% reduction in MTTR.

One of the most basic ways that Unravel helps reduce MTTI/MTTR is through the elimination of the time-consuming steps of data collection and correlation.  Unravel collects granular cluster and application-specific runtime information, as well as metrics on infrastructure, resources using native Hadoop APIs and via lightweight sensors that only send data while an application is executing. This alone can save data teams hours – if not days – of data collection by, capturing application and system log data, configuration parameters, and other relevant data.

Once that data is collected, the manual process of evaluating and interpreting that data has just begun. You may spend hours charting log data from your Spark application only to find that some small human error, a missed configuration parameter, and incorrectly sized container, or a rogue stage of your Spark application is bringing your cluster to its knees.

General overview of Unravel Data's operations dashboard

Unravel’s top level operations dashboard

Improving Visibility Into Cluster Operations

In order for MTI to establish and maintain their SLAs, they needed to troubleshoot cluster-level issues as well as issues at the application and user levels. For example, MTI wanted to monitor and analyze the top applications by duration, resources usage, I/O, etc. Unravel provides a solution to all of these requirements.

Cluster Level Reporting

Cluster level reporting and drill down to individual nodes, jobs, queues, and more is a basic feature of Unravel.

Overview Example of Unravel Data's cluster infrastructure dashboard

Unravel’s cluster infrastructure dashboard

One observation from reports like the above was that the memory and CPU usage in the cluster was peaking from time to time, potentially leading to application failures and slowdowns. To resolve this issue, MTI utilized EMR Automatic scaling feature so that instances were automatically added and removed as needed to ensure adequate resources at all times. This also ensured that they were not incurring unnecessary costs from underutilized resources.

Application and Workflow Tagging

Unravel provides rich functionality for monitoring applications and users in the cluster.  Unravel provides cluster and application reporting by user, queue, application type and custom tags like Project, Department etc. These tags are preconfigured so that MTI can instantly filter their view by these tags. The ability to add custom tags is unique to Unravel and enables customers to  tag various applications based on custom rules specific to their business requirements (e.g. Project, business unit, etc.).

Donut Graph of Unravel Data's application tagging by department example

Unravel application tagging by department

Usage Analysis and Capacity Planning

MTI wants to be able to maintain service levels over the long term, and thus require reporting on cluster resource usage, and data on future capacity requirements for their program. Unravel provides this type of intelligence through the Chargeback/showback reporting.

Unravel Chargeback Reporting

You can generate ChargeBack reports in Unravel for multi-tenant cluster usage costs associated by the Group By options: application type, user, queue, and tags. The window is divided into three (3) sections,

●      Donut graphs showing the top results for the Group by selection.

●      Chargeback report showing costs, sorted by the Group By choice(s).

●      List of Yarn applications running.

Donut graph of Unravel Data's chargeback reporting graphs

Unravel Data’s chargeback reporting

Improving Cluster Efficiency and Performance

MTI wanted to be able to predict and anticipate application slowdowns and failures before they occur. by using Unravel’s proactive alerting and auto-actions so that they could, for example, find runaway queries and rogue jobs, detect resource contention, and then take action.

Unravel Auto-actions and Alerting

Unravel Auto-actions are one of the big points of differentiation over the various monitoring options available to data teams such as Cloudera Manager, Splunk, Ambari, and Dynatrace. Unravel users can determine what action to take depending on policy-based controls that they have defined.

Example of how Unravel Data's auto-actions tool simplifies autonomous remediation of application slowdowns and failures

Unravel Auto-actions set up

The simplicity of the Auto-actions screen belies the deep automation and functionality of autonomous remediation of application slowdowns and failures. At the highest level, Unravel Auto-actions can be quickly set up to alert your team via email, PagerDuty, Slack or text message. Offending jobs can also be killed or moved to a different queue. Unravel can also create an HTTP post that gives users a lot of powerful options

Unravel also provide a number of powerful pre-built Auto-action templates that can give users a big head start on crafting the precise automation they wish for their environment.

Examples of Unravel Data's auto-action templates

Pre-configured Unravel auto-action templates

Applications

Turning to MTI’s application-level requirements, the company was looking at improving overall visibility into their data application runtime performance, and to encourage a self-service approach to tuning and optimizing their Spark applications.

Increased Visibility Into Application Runtime and Trends

MTI data teams, like many, are looking for that elusive “single pane of glass” for troubleshooting slow and failing Spark jobs and applications. They were looking to:

●      Visualize app performance trends, viewing metrics such as applications start time, duration, state, I/O, memory usage, etc.

●      View application component (pipeline stages) breakdown and their associated performance metrics

●      Understand execution of Map Reduce jobs, Spark applications and the degree of parallelism and resource usage  as well as obtain insights and recommendations for optimal performance and efficiency

Because typical data pipelines are built on a collection of distributed processing engines (Spark, Hadoop, et al.), getting visibility into the complete data pipeline is a challenge.  Each individual processing engine may have monitoring capabilities, but there is a need to have a unified view to monitor and manage all the components together.

Unravel Monitoring, Tuning, and Troubleshooting

Userflow example of drilldown from Spark application list to an individual data pipeline stage

Intuitive drill-down from Spark application list to an individual data pipeline stage

Unravel was designed with an end-to-end perspective on data pipelines. The basic navigation moves from the top level list of applications to drill down to jobs, and further drill down to individual stages of a Spark, Hive, MapReduce or Impala applications.

Unravel Data's Gantt chart view of a Hive query

Unravel Gantt chart view of a Hive query

Unravel provides a number of intuitive navigational and reporting elements in the user interface including a Gantt chart of application components to understand the execution and parallelism of your applications.

Unravel Self-service Optimization of Spark Applications

MTI has placed an emphasis on creating a self-service approach to monitoring, tuning, and management of their data application portfolio. They are for development teams to reduce their dependency on IT and at the same time to improve collaboration with their peers. Their targets in this area include:

●      Reducing troubleshooting and resolution time by providing self-service tuning

●      Improving application efficiency and performance with minimal IT intervention

●      Provide Spark developers performance issues and relate directly to the lines of code associated with a given step.

MTI has chosen Unravel as a foundational element of their self-service application and workflow improvements, especially taking advantage of application recommendations and insights for Spark developers.

Examples of Unravel Data's self-service capabilities

Unravel self-service capabilities

Unravel provides plain language insights as well as specific, actionable recommendations to improve performance and efficiency. In addition to these recommendations and insights, users can take action via the auto-tune function, which is available to run from the events panel.

Overview example of how Unravel Data provides intelligent recommendations and insights as well as auto-tuning.

Intelligent recommendations and insights as well as auto-tuning

Optimizing Application Resource Efficiency

In large scale data operations, the resource efficiency of the entire cluster is directly linked to the efficient use of cluster resources at the application level. As data teams can routinely run hundreds or thousands of job per day, an overall increase in resource efficiency across all workloads improves the performance, scalability and cost of operation of the cluster.

Unravel provides a rich catalog of insights and recommendations around resource consumption at the application level. To eliminate resource wastage Unravel can help you run your data applications more efficiently by providing you AI driven insights and recommendations to do show:

Underutilization of Container Resources, CPU, or Memory

Underutility example of how Unravel Data provides optimization suggestions, insights, and recommendations around resource consumption

Memory utility example of how Unravel Data provides optimization suggestions, insights, and recommendations around resource consumption

Too few partitions with respect to available parallelism

Partitions example of how Unravel Data provides optimization suggestions, insights, and recommendations around resource consumption

Mapper/Reducers Requesting Too Much Memory

too much memory example of how Unravel Data provides optimization suggestions, insights, and recommendations around resource consumption

Too Many Map Tasks and/or Too Many Reduce Tasks

too many map tasks example of how Unravel Data provides optimization suggestions, insights, and recommendations around resource consumption

Solution Highlights

Work on all of these operational goals is ongoing with MTI and Unravel, but to date, they have made significant progress on both operational and application goals. After running for over a month on their production computation cluster, MTI  were able to capture metrics for all MapReduce and Spark jobs that were executed.

MTI also got great insights on the number and causes of inefficiently running applications. Unravel detected a significant number of inefficient applications.  Unravel detected 38,190 events after analyzing 30,378 MapReduce jobs that they executed. They were also able to detect 44,176 events for 21,799 Spark jobs that they executed. They were also able to detect resource contention which causing Spark jobs to get stuck in “Accepted” state, rather than running to completion.

During a deep dive on their applications, MTI found multiple inefficient jobs where Unravel provided recommendations for repartitioning the data. They were also able to Identify many jobs which waste CPU and memory resources.

from AWS Startups Blog

Improving Medical Diagnoses With Health Tech App Healint

Improving Medical Diagnoses With Health Tech App Healint

We’ve all been there. You go to your semi-annual doctors visit and they ask, “How’s everything been going?” You then try to remember and recount your health over the last few months, in hopes that they can understand and diagnose a central issue to fix the various symptoms.

This process has more than a couple issues, however, not the least of which is that it relies on a person’s memory being foolproof and the person having the ability to clearly articulate his health problems. Working off of a semi-reliable, oral list of health issues can make it difficult for doctors to provide a great diagnosis, which leads to a higher chance of the wrong treatment being suggested.

Not to fear, however; the team at Healint, a healthtech company based out of Singapore, is here to fix that whole process. And it all starts with a mobile app.

Using Healint’s app, users are able to both actively and passively chronicle issues pertaining to their health. This data collection can take a handful of different forms, including the tracking of sleep patterns, potential triggers related to pain, as well as activity levels.

With the app, instead of having to try and recall months worth of information on the spot, patients can offer a report of how things have been going directly to their doctors, giving the medical professional more reliable information upon which to base his diagnosis.

The company’s ambitions don’t stop at just enabling the easy logging for users and reporting for doctors, though. Leveraging machine learning, Healint crunches the data from its 1.5 million users to understand correlations between reported symptoms and triggers. From there, based on a person’s medical profile, the app can proactively suggest recommendations.

Healint CTO Nicolas Paris

From a technical perspective, Healint has built its infrastructure utilizing microservices, a methodology that has seen increasing adoption as of late. Each feature has its own dedicated microservice, utilizing AWS services such as EC2 and SQS, to enable the collection of data from the application and storing of said data automatically at scale.

What’s next for the healthtech company? Per Healint CTO Nicolas Paris, it is eyeing expansion to support more chronic conditions. Previously, the company had been focused on working with people that experience chronic migraines or headaches. They’ve since started developing features for those with chronic pain and skin conditions like fibromyalgia, psoriasis, and eczema.

from AWS Startups Blog

Enterprises Use Crayon Data to Better Understand Their Customer’s Tastes

Enterprises Use Crayon Data to Better Understand Their Customer’s Tastes

When looking at the monstrous success of Google’s and Facebook’s ad businesses, most would point out that the core driving factor is their data and how they utilize it. These companies have built massive platforms around offering customized experiences to their users based upon mountains of proprietary information.

To date, that strategy has seemed somewhat limited to new-age technology companies. Crayon Data, a Singapore-based big data and AI startup, is working to change that.

Founded in 2012, Crayon believes that many companies are sitting on valuable data that can be used to better understand and engage customers, they just don’t know it. For example, Crayon’s Ravishankar Krishna points to how their main customer segment, financial institutions have historically interacted with their data to engage customers.

“Previously, much of personalization would be derived from actions taken on a company’s digital assets, like a website or mobile app like clickstream data, what you’ve liked, where you dropped off, etc.”

Crayon, however, looks beyond the obvious and dives deeper into the data each company has. This dive is not just limited to online interaction data; it also includes offline transaction data.

Krishna, who manages sales and business development activities in Asia, continues with the bank example and points to spend data, and the metadata that comes along with it.

“Imagine you have the bank’s credit card. You go use that card at coffee shops on Monday mornings right before heading to work. You then use it for dining, travel, and a host of other activities. Within those purchases are valuable signals that had before been disregarded.”

Using their proprietary AI engine—dubbed maya.ai—Crayon cleans and analyzes all of the data to create detailed profiles of each customer and his or her unique tastes. Companies can then use these profiles to offer more relevant content to users and drive multiple business cases, ranging from increased card spends to engaging at-risk customer segments.

As another member of the sales and business development team, Samarth Bansal, puts it, “Banks are looking for ways to stand out and not be commoditized as another payment mechanism. With Crayon, they’re able to look at the data they already have and bring new offerings to customers based on what those customers enjoy.”

The origin of the company can be traced back to a previous startup Krishna built with Crayon Founder and CEO Suresh Shankar. Prior to this enterprise, the two of them built Red Pill Solutions, a strategy consulting firm that focused on analytic services like marketing and risk analytics. After years of operating and growing that company, they completed an acquisition by IBM in 2010 and rolled the startup into Big Blue’s CRM offering. Through that experience, the team saw how quickly data was exploding and how the old model was not built to handle it, which led to the idea that would become Crayon Data.

Beyond the team, Red Pill’s influence on Crayon can also be seen in the customer base. Red Pill found much success working with financial institutions, a market that Crayon mainly serves as well.

As for what’s next for Crayon, Krishna points to geographic and industry expansion, leveraging AWS’s global reach along the way.

Per Krishna, “Currently, we find most of our clients are in Asia and the Middle East. Soon, we’ll be moving to enter the US market in a big way, as well as expand into other Asian geographies that we previously haven’t focused on, such as Australia and New Zealand. And while we’ve found much success in the financial services industries, we’re actively expanding into other sectors, including e-commerce, as well as travel and hospitality.”

from AWS Startups Blog

Completely All Connected Together: Talking with Fuze Leaders Aaron Evans, Doug Jones, and Khoder Shamy

Completely All Connected Together: Talking with Fuze Leaders Aaron Evans, Doug Jones, and Khoder Shamy

Communication Tech Startup Fuze leaders talk about growing the business on AWS

Perhaps it’s not surprising that the minds behind Fuze—a company the Aragon Report recently named as a Leader in Unified Communication and Collaboration for the third year in a row—would enthusiastically expound on, well, unified communication and collaboration. It’s what they’re all about. The company acts as a bridge, says VP of engineering Aaron Evans, bringing people in global enterprises “completely all connected together.”

What may be a little surprising, however, is learning just how much unification is needed. If Fuze is a bridge, it’s not crossing a babbling brook. It’s traversing a stormy stretch of open ocean.

“Our platform encompasses voice, collaboration including video conferencing and screen share, chat, presence, and some elements of contact center,” Evans says. “[For] many of our clients, we are replacing whatever sort of voice PBX solution that they’ve been using, but in addition, replacing other point solutions that they’re using for chat and collaboration to reduce their overall costs for communications needs.”

In essence, Fuze is doing the work of multiple platforms in one—it works to solve the problems associated with everything from connecting colleagues across great distances to consolidating telecom costs, from accommodating traditional office employees’ interest in using a voice channel and physical handset to meeting the desires of voice- and voicemail-averse Millennials. On average, Fuze replaces four to eight different communication and collaboration solutions for its customers. Evans expands on the pain points Fuze aims to solve for its customer base, explaining the tremendous need for streamlining and speed. “There’s a modernization aspect to this as well. [We’re] transforming how customers are communicating.”

It’s not just the customers’ communication that’s evolving, however. The Fuze origin story starts in what can be considered the Middle Ages of tech-time: the company was founded thirteen years ago. Back then, it managed its own data centers. As Doug Jones, VP of cloud infrastructure and operations, puts it, “We’re a global company, and therefore what we had was a lot of small, little data centers all over the world that we had to build, run, and maintain.”

But soon, Fuze stumbled upon one of the better problems for a communications provider to have: galloping success outpaced its resources. Fuze attracts larger enterprise customers, the kind with a global presence and a far-flung employee base. Fast connection is imperative; there’s no time to spend the necessary five to ten months building a data center. Jones realized a change needed to be made.

“And so, we decided to leverage AWS and just use virtual capacity from AWS,” he says. What this has enabled, adds Jones, is that necessary speed.

That doesn’t mean Fuze has abandoned its own global, private network backbone. For Khoder Shamy, Director of Cloud Platform at Fuze, that backbone is vital. He says, “We want to control the backbone for our services and our intercommunication between different regions. So, it will be a hybrid of architecture between AWS and our own network points of presence and backbone.”

This backbone-grounded hybrid approach helped during the early transition process, which was a period that, according to Evans, “made many existing customers nervous.” There were the usual questions about data security and privacy, but Evans is proud to report that Fuze didn’t suffer a loss of customers. “When we show them that we’re actually more secure and some of the controls and services that we utilize, then they become very comfortable.”

The reality of Fuze post-transition has been one of growth, both for customers opening new offices and for Fuze itself. And for now, continuing this growth is Fuze’s main goal, says Jones.

“We’re going to expand the footprint,” he says, adding that Fuze aims to reduce its number of colocation centers “to the point that we won’t have colos anymore.”

The goal is to be colo-free in a year’s time, at which point Fuze can both truly focus its energies on leveraging networking features and, as Evans explains, “continue to scale the platform and be able to support ever larger customers and a larger number of customers.” But ultimately, Fuze’s aims will remain the same: It’s still about bridging communication gaps and making sure people have a seamless connection to the data that they need.

from AWS Startups Blog