A data team once told us their quality record was excellent. Every pipeline had tests, every test passed, and the orchestrator showed a wall of green. The same week, the finance dashboard showed Monday’s revenue on Wednesday.
Nothing had failed. A vendor feed had arrived two days late, the transformation job ran on the previous file, and every check agreed that the output was valid. It was valid. It was also old.
The failure that no test catches
Tests describe what correct data looks like: no nulls in this column, ids are unique, totals match. They say nothing about whether the data is the data you expected today. A table can pass every test while holding last week’s rows.
Job status has the same blind spot. A job succeeds when it finishes without an error. It does not know that its input was a day late.
There are three common ways to be late without failing:
- A source is late. The file, API or event stream arrives after the deadline, and the job runs on what is there.
- A schedule slips. An upstream task waits for a resource, and everything behind it starts later than usual but still finishes.
- A job runs on an empty window. The query is correct, the window has no rows yet, and zero rows is a valid result.
Freshness is a property of data, not of jobs
The question to ask is not “did the job run” but “how old is the newest data in this table, and is that acceptable”. That is freshness, and you can measure it directly.
For a table with an event timestamp, the measurement is a query:
select
max(event_time) as newest_event,
datediff('minute', max(event_time), current_timestamp) as minutes_behind
from analytics.orders;For a table without a reliable event time, use the load time that your warehouse already records. Most warehouses keep a last modified time in their metadata views, which is cheaper to read than the table itself.
Once you can measure freshness you can compare it with an expectation. Tidewater stores that expectation next to the table, and shows the answer as one of three states: on time, late, or late and affecting something downstream.
Setting an expectation people will trust
A freshness expectation is a promise, and a promise that is wrong too often is ignored. Three habits keep them credible.
Write the deadline as the consumer would. “The orders table is complete by 06:00 on weekdays” is a statement a finance director can check. “Max lag 240 minutes” is not.
Use history, then round up. Look at the last 30 days of arrival times. Set the deadline just beyond the slowest normal day, not the average. If the table arrives between 04:10 and 05:20, a deadline of 06:00 will rarely fire for no reason.
Separate late from broken. A table that is 20 minutes late is a warning. A table that is four hours late on a close day is an incident. Two thresholds let the system say the difference.
| Signal | What it tells you | What it misses |
|---|---|---|
| Job succeeded | The code finished without an error | Whether the input was current |
| Test passed | The data is shaped as expected | Whether the data is from today |
| Row count in range | Volume looks normal | Whether the rows are new |
| Freshness within deadline | The newest data is recent enough | Whether the values are right |
No single signal is enough. Freshness fills the gap that the other three leave, which is why we treat it as the first check on any table that people read.
What to do when a feed is late
When an alert arrives, the first useful fact is what is affected. A late staging table that nothing reads can wait. A late table that feeds the revenue dashboard cannot.
You can ask the Tidewater API for the downstream assets of a table directly:
curl -s "https://api.tidewater.example/v1/lineage/analytics.orders?direction=downstream&depth=3" \
-H "Authorization: Bearer $TIDEWATER_TOKEN"The response lists tables, models and dashboards with their owners. The second useful fact is whether the source is late or only your copy of it. That decides whether you wait, ask the vendor, or re-run a job.
A starting checklist
- List the ten tables that people outside the data team read most.
- For each one, write the deadline as the reader would say it.
- Check 30 days of arrival times and move any deadline that would have fired for no reason.
- Give each table an owner, and send its alerts to that owner.
- Review the alerts that fired after two weeks, and tighten or loosen the deadlines.
Teams that do this usually find, within the first month, one feed that has been quietly late for a long time. That alone tends to pay for the effort.