Beyond Durability: Enhancing Database Resilience and Reducing the Entropy Using Write-Ahead Logging at Netflix

In modern database systems, durability guarantees are crucial but often insufficient in scenarios involving extended system outages or data corruption. Additionally, challenges arise when a single user mutation request is distributed across different database nodes or across multiple databases, necessitating a robust mechanism for mutation replay.

At Netflix, we have developed an innovative architecture that not only addresses these issues but also enhances the system's overall functionality. This talk will outline how we leverage Write-Ahead Logging (WAL) as a service to ensure all database mutations are durable and maintain ordering guarantees, even during periods when a downstream database is temporarily unavailable.

A key design principle in our approach is the separation of concerns, which helps manage system complexity and facilitates higher adoption. By employing a pluggable architecture, we enable the development of high-leverage capabilities such as secondary indexes, delayed queues, and generic replication services. These capabilities ensure data consistency across various non-distributed data stores like RocksDB, Redis, and Memcached. 

This talk explores the key considerations that must be taken into account when building WAL for various use cases.

Key Takeaways

• How building systems with pluggable architecture helps in higher adoption and building high-leverage capabilities.
• Keeping separation of concerns as a core design principle to manage the complexity of the system.
• How to invest in building newer systems/architectures as the use cases evolve.