Honeycomb: How We Used Serverless to Speed Up Our Servers

Abstract

Honeycomb is the state of the art in observability: customers send us lots of data and then compose complex, ad-hoc queries. Most are simple, some are not. Some are REALLY not; this load is both complex, spontaneous, and urgent. It would be prohibitively expensive to size a server cluster to handle these big queries quickly, so we took a different approach: farm the work out to Lambda, Amazon's serverless offering.

In this model, Lambda becomes an on-demand accelerator for our always-on servers. The benefits are immense, improving response times by an order of magnitude. But the challenges are numerous and often unexpected. In this talk, I'll review the benefits (user experience on demand!) and constraints (everything in AWS has a limit!) of serverless-as-accelerator, and give practical advice based on our own hard-won experience.

Speaker

Jessica Kerr

Principal Developer Evangelist @honeycombio

Jessica Kerr (@jessitron on twitter) is fascinated by how software doesn’t get easier as she gets better at it: it gets harder, and also more valuable. Jessitron has developed software in Java, C, TypeScript, Clojure, Scala, and Ruby. She keynoted software conferences in Europe, the US, and Australia. She ran workshops on Systems Thinking (with Kent Beck) and Domain Driven Design (with Eric Evans). Now she works at Honeycomb, making complexity navigable with observability. Find her at jessitron.com, or systemsthinking.dev, or at home in St. Louis, MO.

Read more

Date

Wednesday Oct 26 / 11:50AM PDT ( 50 minutes )

Location

Ballroom A

Topics

Architecture Observability Serverless

Share

Share Share

From the same track

Session Architecture

Azure Cosmos DB: Low Latency and High Availability at Planet Scale

Wednesday Oct 26 / 01:40PM PDT

Azure Cosmos DB is a fully-managed, multi-tenant, distributed, shared-nothing, horizontally scalable database that provides planet-scale capabilities and multi-model APIs for Apache Cassandra, MongoDB, Gremlin, Tables, and the Core (SQL) APIs.

Speaker image - Mei-Chin Tsai

Mei-Chin Tsai

Partner Director of Software Eng Manager @Microsoft, one of the original developers on .NET

Speaker image - Vinod Sridharan

Vinod Sridharan

Principal Software Engineering Architect @Microsoft

Session Architecture

From Zero to A Hundred Billion: Building Scalable Real Time Event Processing At DoorDash

Wednesday Oct 26 / 02:55PM PDT

At DoorDash, real time events are an important data source to gain insight into our business but building a system capable of handling billions of real time events is challenging.

Speaker image - Allen Wang

Allen Wang

Software Engineer @DoorDash, previously Lead for real-time data infrastructure team @Netflix

Session Architecture

Magic Pocket: Dropbox’s Exabyte-Scale Blob Storage System

Wednesday Oct 26 / 04:10PM PDT

Magic Pocket is used to store all of Dropbox’s data.

Speaker image - Facundo Agriel

Facundo Agriel

Software Engineer / Tech Lead @Dropbox, previously @Amazon

Session Architecture

Amazon DynamoDB: Evolution of a Hyper-Scale Cloud Database Service

Wednesday Oct 26 / 10:35AM PDT

Amazon DynamoDB is a cloud database service that provides consistent performance at any scale. Hundreds of thousands of customers rely on DynamoDB for its fundamental properties: consistent performance, availability, durability, and a fully managed serverless experience.

Speaker image - Akshat Vig

Akshat Vig

Distinguished Engineer @MongoDB, Previously Senior Principal Engineer NoSQL@AWS