The Problem
A recent data-platform problem made our team rethink what initially appeared to be a very simple question: “When was this Gold table last updated?”
At first glance, querying
information_schema.tables.last_altered
looks like the obvious answer.
But there is a hidden trap: a table can be modified without receiving new business data. Maintenance activity can make a table appear recently altered while the actual ingestion has stalled for days.
Evaluating the Standard Approaches
last_altered
Extremely cheap and easy to query.
Represents table/object modification rather than necessarily representing new business data.
MAX(timestamp)
Closest to the actual meaning of data freshness.
A full-table aggregation can become expensive on large Gold tables.
Iceberg Metadata / History
Provides a transaction-level view without scanning the business data.
Running history-based checks indiscriminately across frequently updated tables introduces its own scalability considerations.
The Hybrid Approach
Instead of asking “Which one of these methods should we use?”, we reframed the problem:
The result is a tiered detection framework designed to balance accuracy, cost, scalability and visibility.
Step 1: Automatically Discover Gold Tables
We use Unity Catalog metadata to automatically identify the tables belonging to the Gold layer.
This is important because the framework should not contain manually maintained monitoring logic for every individual table.
information_schema.tables
│
▼
Gold Tables
│
▼
Freshness EngineStep 2: Progressive Partition Scanning
Our Gold tables are partitioned by date. Instead of scanning the entire table to determine the latest ingestion timestamp, we exploit that partitioning strategy.
The freshness engine uses configurable lookback windows:
We start with the smallest window. If a valid ingestion timestamp is found, the evaluation is complete. If not, the engine progressively expands the lookback window.
Most healthy Gold tables have regular ingestion patterns. For these tables, we only need to inspect a very small portion of the data rather than repeatedly scanning the entire table.
Step 3: History as an Exception Handler
What happens when the progressive partition search cannot identify a valid ingestion timestamp?
Only then do we fall back to table history.
This fundamentally changes the role of history inspection. Instead of treating it as the default freshness mechanism for every table, we use it as an exception-resolution mechanism.
The Architecture
What We Capture
Rather than storing only a single timestamp, the freshness result contains enough information to explain how the platform arrived at that result.
Retaining last_altered for Diagnostics
We didn't throw last_altered away. It simply answers a
different question.
By retaining it alongside the actual freshness signals, we can identify discrepancies instead of hiding them.
The discrepancy immediately tells us that recent table activity should not automatically be interpreted as recent business-data ingestion. This is exactly the type of false-positive signal we wanted the new framework to eliminate.
Beyond Freshness: Root-Cause Analysis
A freshness dashboard that only tells us “this table is stale” is useful—but it is only the beginning.
For tables that remain stale beyond a defined threshold, such as 30 days, we want to understand why.
The investigation can determine whether the upstream job is paused, the pipeline has failed, the source has stopped producing data, or the table is intentionally static.
Instead of simply reporting “This table is stale”, the platform moves toward explaining: “This table is stale because its upstream source has stopped producing data.”
The Bigger Picture
Data freshness isn't just a timestamp problem. It is a balance between correctness, cost, scalability and explainability.
Use the cheapest reliable signal first, and invoke more expensive mechanisms only when the cheaper signal cannot provide a confident answer.
Final Thought
The goal isn't to find one perfect freshness metric.
The goal is to build a system that can make a reliable freshness determination across a large and evolving data platform while keeping compute under control—and, just as importantly, explain how it reached that conclusion.
Freshness should tell us not only whether data is fresh, but also give us the evidence needed to understand when it isn't.