You've Always Known the Data Was There
You've had years of data run through your inbox, your shared drives, your job folders. Cost reports, change logs, daily reports, sub buyouts, closeouts. You've always known it should be useful.
You've tried to capture it. Built a tracking sheet, asked PMs to log production, tried to standardize cost codes. For a few weeks, maybe a month, it held together.
Then the job picked up. People got busy. Data came in half-complete—missing quantities, blended costs, no context. Cleaning it took longer than collecting it. Eventually it wasn't worth the effort, and the system defaulted back to what it always does: skim a few past jobs, pull numbers that feel close, make the call.
Not because anyone stopped believing in data. Because collecting it properly has always cost more time than anyone actually has.
What the Drift Actually Looks Like
Consider a $20M ground-up. Estimate built off historical data from before the last labor run-up. Contingency set at 4% because that's what the company always uses. At 50% complete, the run rate is already 8% over. The PM knew at 25% something felt light but didn't say it loudly enough. The estimator never got the field report on productivity. The owner saw the variance for the first time on a monthly billing summary.
Nobody lied. Nobody was lazy. The process let the drift hide.
But look closer at the historical data the estimate was actually built on. A drywall number pulled from a past job with no note on ceiling height, crew mix, or whether that project was running Saturdays. Labor rates from 18 months ago, blended across conditions that don't match. Productivity assumptions that were never measured—just carried forward because nobody had time to question them.
That's not bad estimating. That's unverifiable input. And unverifiable input produces confident-sounding numbers that have no real floor.
The Math Keeps Getting Better. The Overruns Don't.
Current estimating approaches tend to underestimate expected cost and overestimate variance, particularly on larger projects. Parametric approaches reduce some of the structural bias. Ensemble models have pushed prediction accuracy into the 60% range on certain public datasets. The math keeps getting better—and the overruns keep happening.
Because the problem isn't only in how you predict. It's in what you're feeding the prediction.
"Feed actuals back into the system" only works if the actuals are usable. Most aren't. Costs coded one way in estimating, another in accounting. Labor, material, and equipment blended because that's how invoices hit. Quantities not tracked against hours. Cost codes repurposed mid-job to make the report read cleaner. By closeout, the data exists—but it can't tell you where you actually made or lost money.
A usable feedback loop requires some basic discipline that most firms haven't enforced: date your assumptions—labor rate, escalation, productivity, contingency basis—so they can be evaluated later. Track variance by trade, not just total job, because that's where the patterns actually live. Pull operations in before the number locks, not to review everything, but to pressure-test the assumptions that actually drive cost. And stop carrying unit costs forward without knowing what conditions produced them. A number without context isn't data—it's a guess with a source citation.
The Pressure That Doesn't Show Up in the Model
Optimism bias is the obvious problem, but it shows up in quieter ways than people tend to admit. Scope gets blended so numbers land where they need to. Assumptions don't get documented because documented assumptions can be challenged. Cost codes get adjusted late so the story reads cleaner than reality. The estimate doesn't just reflect the job—it reflects the pressure around the job, and that pressure has always been invisible to any tool trying to improve accuracy.
Why the Data Problem Has Persisted
What's actually changed isn't the understanding of what good data looks like. That's been understood for years. What's changed is the cost of collecting it.
Manual collection breaks before it becomes useful. Foremen don't have bandwidth to track production cleanly. PMs don't reconcile cost to scope in real time. Estimators get fragments after the fact, not structured feedback. The work to capture data well has always consumed more capacity than field and office teams have available—so it doesn't happen, and the feedback loop stays theoretical.
Field reports, photos, and logs can now be parsed automatically. Timecards can be tied to scope and location instead of generic codes. Cost, schedule, and estimate structures can be mapped once so incoming data lands in the right place. More importantly, data doesn't have to be cleaned at closeout anymore—it can be corrected as it's created. Flagged when cost codes drift, when dollars appear without quantities, when one project behaves differently than the last five. It won't be perfect. But it will be consistent, and consistency is what every manual system eventually fails to sustain.
The Tradeoff Nobody Talks About
There's a tradeoff in this that doesn't get discussed much. Better data doesn't just improve estimates—it makes the misses harder to explain away. When you can see that sitework productivity ran 20% off, that MEP rough-in consistently drifts on compressed schedules, that your standard contingency hasn't covered reality across multiple jobs—it gets harder to call those outcomes bad luck.
That's part of why this problem has persisted. Not because the solution is technically out of reach. Because operating with that level of clarity requires an organizational honesty that's uncomfortable to sustain. The plausible deniability that lives inside vague data has always been useful to someone.
Where This Leaves the Estimator
None of this replaces what an experienced estimator actually does. They still decide what's comparable, which subs to trust, when a number doesn't pass the smell test regardless of what the model says. That judgment doesn't go away.
What changes is the environment around it. Assumptions become visible. Data becomes traceable. The feedback loop arrives early enough to affect the next decision instead of just explaining the last one. The estimator with clean data and a real feedback loop isn't doing a different job—they're doing the same job with fewer places for the drift to hide.