Data storage and access at the edge delivers massive performance gains by reducing location-sensitive latency. Cloudflare operates a distributed cross-region database architecture at the edge, serving over 50TB of online transactional data over millions of requests across our entire security and web performance product suite. Storing and managing relational data at the edge carries a unique set of challenges dictated by timeless CAP constraints and highly variable load conditions, requiring careful tradeoffs.
How can we minimize replication lag, choose replication strategies (asynchronous, streaming, logical, custom tooling) that balance consistency and performance, optimize request routing, and even speed up split brain cluster resynchronization? Achieving high availability storage performance at the edge demands innovative techniques involving custom performance optimizations across our anycast BGP network, cluster management layer, and down to the core Postgres recovery system.
So, you still want to build your database architecture at the edge?
You’ll learn about Cloudflare’s entire edge database architecture, unique challenges and practices for data replication, failover and recovery, and custom performance techniques. We'll also peek into the exciting future landscape of edge storage.
Interview:
What's the focus of your work these days?
Building a managed Postgres environment at Cloudflare, working on building edge applications, and contributing to Postgres and related open-source projects.
What's the motivation for your talk at QCon San Francisco 2023?
To help everyone understand the challenges of data at the edge and what are the ways to solve them.
How would you describe your main persona and target audience for this session?
Someone who is a Systems Engineer, SRE or even a developer who is curious in understanding edge computing and how to enable it through Postgres.
Is there anything specific that you'd like people to walk away with after watching your session?
I would expect attendees will learn something about the limitations/challenges of data at the edge, and opportunities if done right.
Speaker
Justin Kwan
Software Engineer Intern - iCloud Edge @Apple, Previously @Cloudflare
Justin Kwan is a Software Engineer at Apple. He is currently focused on building high-throughput DDoS mitigation and privacy-preserving features protecting Private Relay and iCloud edge networks. Formerly at Cloudflare, Justin focused on multitenant resource isolation, storage reliability, chaos experimentation across distributed storage infrastructure, and patching the performance of large open-source systems such as Postgres and PgBouncer. His excitement lies in high-performance systems software, distributed networking and storage, kernel-layer observability, and the intricacies of database query execution.
Find Justin Kwan at:
Speaker
Vignesh Ravichandran
Engineering Manager @Cloudflare, Contributor to Postgres, Previously at Ticketmaster
Engineering Manager of Databases at Cloudflare. Manages a team of engineers across the world who is responsible for ensuring the databases are available, reliable, and highly performant 24*7*365. Created spinup and other projects. He is a frequent author and speaker at various open-source conferences.
Find Vignesh Ravichandran at:
Date
Monday Oct 2 / 11:45AM PDT ( 50 minutes )
Location
Ballroom BC
