Freight and logistics, 1,400 employees, 9 person data team

How a freight marketplace stopped finding out about late shipment data from its own customers.

Northfold Freight
  • Apache Airflow
  • dbt Core
  • Snowflake
  • Looker
  • 6 min Median time to detect late data Down from 3 hours 20 minutes, measured over the 90 days before and after rollout.
  • 71% Fewer late data incidents reaching customers Incidents that a customer reported before the data team knew, per quarter.
  • 4 Hours a week saved on triage Estimated by the team from their on-call hand-over notes.

Northfold Freight is a fictional company used in this demonstration site. The story below shows the kind of result a data platform team can expect and is not a real customer account.

The problem

Northfold runs a marketplace that matches shippers with carriers. Customers see estimated arrival times in a dashboard that is rebuilt from about 400 Airflow DAGs and 900 dbt models. Most nights everything finished on time. A few times a month, one upstream feed arrived two hours late, every job still reported success, and the arrival estimates on Monday morning were wrong.

The data team had tests and Airflow alerts. Neither fires when a job succeeds on yesterday’s data. The first signal was usually a customer success manager forwarding a complaint.

What the team did

They connected Airflow, dbt and Snowflake to Tidewater in an afternoon, and imported their existing meta.owner tags so every table had an owner from day one.

  1. Freshness before success. Each customer facing table got a freshness expectation: “updated by 06:00 Eastern on weekdays”. Tidewater learned the rest from history.
  2. Impact in the alert. Every late data alert lists the downstream models and the Looker dashboards that depend on the table, so whoever is on call can decide in a minute whether it matters.
  3. Routing by owner. Alerts go to the owning team’s channel, with a daily digest for low urgency items. The shared “data-alerts” channel was archived after three weeks.

The result

In the first quarter the team caught 31 late feeds before they reached a dashboard. The median time from a feed going late to someone knowing dropped from 3 hours 20 minutes to 6 minutes.

The change in habits mattered as much as the numbers. The on-call engineer now starts the day by reading one digest instead of checking eleven dashboards, and the weekly data quality review moved from “what happened” to “what do we change so it cannot happen again”.

What they would tell another team

Start with the ten tables that customers see, not the whole warehouse. Add owners first, because an alert with no owner is only noise. Look at the lineage view before you set thresholds: most teams discover tables nobody remembers building.

We used to learn that the shipment tables were stale from a customer email. Now the first message is in our own channel, and it already names the dashboards it affects.

Marisol Vance, Director of Data Platform, Northfold Freight

Tidewater for data teams

See your own pipelines in Tidewater

Book a 30 minute walkthrough. We connect a sample of your setup and show you what it would have caught last month.