Percona Live Online

At Adobe Experience Platform, we ingest TBs of data every day and manage PBs of data for our customers as part of the Unified Profile Offering. At the heart of this is a bunch of complex ingestion of a mix of normalized and denormalized data with various linkage scenarios power by a central Identity Linking Graph. This helps power various marketing scenarios that are activated in multiple platforms and channels like email, advertisements etc. We will go over how we built a cost-effective and scalable data pipeline using Apache Spark and Delta Lake and share our experiences.

* What are we storing?
* Multi Source - Multi Channel Problem
* Access Pattern to optimize for
* Custom High Performance Query engine
* Data Representation and Nested Schema Evolution
* PerformanceTrade Offs with Various formats
* Go over anti-patterns used
* (String FTW)
* Data Manipulation using UDFs
* Writer Worries and How to Wipe them Away
* Gotchas
* Concurrency
* Column size
* Update frequency
* Transaction Management for A Healthy State
* Staging Tables FTW
* Why we can't live without them
* Datalake Replication Lag Tracking
* Instrumentation of the data pipeline gives more confidence to the reader
* Downstream Data Pipelines
* Showcase easy building of incremental versions of applications
* Maintenance Jobs
* Go over essentials of compaction and vacuuming
* Performance Time!
* What scale are we operating at?
* Settings like autoCompact and optimizeWrite
* Timings With and Without Delta
* Cost