Speaker: Jun He

He / him / his

Staff Software Engineer @Netflix, Managing and Automating Large-Scale Data/ML Workflows, Previously @Airbnb and @Hulu

Jun He is a Staff Software Engineer in the Big Data Orchestration team at Netflix, where he leads the effort to build Netflix's workflow orchestrator, a.k.a. Maestro, to manage and automate large-scale Data/ML workflows at Netflix. He also made contributions to multiple open source projects, such as Apache Iceberg. Prior to Netflix, He spent a few years building distributed systems and search infrastructure at Airbnb. He was the main contributor to design and build message bus and search pipeline at Airbnb. 
 

Find Jun He at:

Session

Efficient Incremental Processing with Netflix Maestro and Apache Iceberg

Incremental processing, an approach that processes only new or updated data in workflows, substantially reduces compute resource costs and execution time, leading to fewer potential failures and less need for manual intervention.

Read more

Date

Tuesday Nov 19 / 03:55PM PST ( 50 minutes )

Location

Ballroom A

Topics

Data Pipelines Apache Iceberg Data Workflow Data Accuracy

Share