The Data Is the Only Living Spec


giarec: “Everything has been implemented as documented and requested, so it should work.” This can't be more wrong. If your company has been collecting/authoring a dataset for years, possibly decades, with tens to hundreds of different sources, curators, and tools, the only real “documentation” is the data itself.

OpZ: That’s the first pushback you hear when you tell engineers to slow down and expose raw data to users on day one. What’s the smallest change you’ve seen a team make to that handoff process that actually cut the number of “invisible” bugs in half?

giarec: Use real data early on, better if the entire dataset, and assume every step will be dense with bugs you can’t notice without the help of someone intimately familiar with the data, ideally using those data every day.

OpZ: You’re describing a handoff problem—developers building pipelines in the dark, then throwing the output over the wall to users who can’t debug what they can’t see. What’s the first concrete signal in a team’s backlog that tells you they’re about to repeat this mistake?

giarec: The most obvious sign is pipelines designed without a user interface allowing an analyst to inspect data at any major step. Dealing with data, there are tons of scenarios that can break the final output even without any visible error in the code (empty fields, wrong case, different encoding, missing links, corrupted information, misplaced fields… most of these issues affect the final output but may, or better, will traverse many steps of the output undetected. QA never has enough intimate knowledge of the data to detect these issues. The only solution is to involve the actual users of the data, business analysts, editors, curators, much earlier in the process to rely on them to validate, spot-check potential issues, and guide the developers by surfacing undocumented nuances and critical details not to miss.

OpZ: You’re saying the data is the only living spec, and QA can’t test what it doesn’t understand. What’s one thing QA teams could do tomorrow to stop pretending they can validate data without domain knowledge?

giarec: Platforms and pipelines should be QAed to ensure deterministic and reliable behavior. While many first-time issues can’t be detected without domain knowledge, most should be avoided on subsequent pipeline enhancements. To this extent, creating golden datasets including any identified nuance and implementing a deterministic pipeline is critical to intercept regression bugs early on. Fundamentally, you need to work under the assumption that neither the developer nor the QA engineer can ever remember all possible scenarios and shapes real data can take. The best approach is providing interfaces for users with domain knowledge to check and run reports, and collect dataset slices that can be used as golden images for regression tests.

OpZ: That golden-dataset idea turns domain knowledge into executable tests—so the next engineer doesn’t have to relearn the same edge cases. If you had to pick one metric to track over the next six months to prove this shift is working, what would it be?

giarec: The most telling sign is how long it takes from code complete to a successful go-live. If you do things right, you will be able to code in parallel while users review the data, leading to consistently faster successful deployments.

OpZ: That’s the finish line: not fewer bugs, but faster safe launches. Cheers for the hard truths.

Worth your time?10
  • data pipelines
  • software development
  • domain knowledge
  • qa testing
  • data validation
  • bug detection
  • playful
  • serious