Lumen Grocer is a fictional company used in this demonstration site. The story below shows the kind of result a streaming and batch data team can expect and is not a real customer account.
The problem
Lumen’s order and delivery events flow through Kafka into Spark jobs and then into 1,400 dbt models in Redshift. Nine teams read from that graph. When a model broke, the engineer who owned it knew what the model did, but not who depended on it, because the dependencies crossed team boundaries and many were added by people who had since moved on.
Every incident started with a long thread. People asked the same questions: what is affected, who owns it, is it urgent.
What the team did
- One lineage graph for everyone. Tidewater reads the dbt manifest, Redshift query logs and the Spark SQL listener, so the graph includes tables that are only read by ad hoc queries and scheduled reports.
- Impact before merge. A check in pull requests lists the tables, models and dashboards touched by a changed model. Reviewers see the owners and can ask them before merging.
- Streaming freshness. Kafka topic freshness and consumer lag sit beside batch freshness, so a producer that goes quiet shows up as a late downstream table with the cause attached.
The result
The median time to produce a complete list of affected assets after an alert is now 2 minutes. Failed overnight runs fell by 38 percent over the next two quarters, mostly because the impact check in pull requests catches breaking changes before they run.
The graph also found clean up work. In the first month the team identified 212 tables with no readers and no recent writes, and retired 160 of them.
What they would tell another team
Give lineage to the people who review changes, not only to the people who get paged. Most of the saving came from incidents that never happened.