The Real-Time Big Data Mistake That Costs Cash and Sleep
giarec: Engineers severely underestimate the complexity of real-time big data pipelines. They are complex to set up, debug and test, and do not have an obvious way to reprocess data when something breaks. At Gracenote, we learned this early on: on projects with very tight deadlines and new features added weekly, we were unable to make them reliable, and we still needed to reprocess data manually all the time, forcing us to always support both real-time and batch.
OpZ: So you’re saying the pipeline had to be both from day one—doubling the surface area for bugs. What’s the rule of thumb you now use to decide whether a feature actually needs second-level updates, versus one where users won’t notice a 30-minute delay?
giarec: Anything hourly, daily or monthly: use batch. If you can tolerate a 5-minute or more delay between when an event is generated and its effect has to be seen/accounted for at the end of the pipeline, use batch. Between 1 and 5 minutes, let's talk, micro-batches are still a better approach. Less than 1 minute and multiple steps? Probably real-time. Aside from the accepted delay, a real-time approach is also needed if you truly have massive scale (million+ events/sec) and you can't delay, accumulate too much data, and/or want to partition/organize your data early on. Even in this case, not all of the pipeline may need to be designed for real-time processing.
OpZ: You just named two exceptions—sub-minute latency and million-plus events per second—where real-time still makes sense. What’s the simplest test you give a team to prove they’re in one of those, not just chasing the real-time label?
giarec: Have you already successfully implemented the batch version of the pipeline? Do you need to reduce latency to sub-minute or increase scale? If the answer is not a solid yes to both questions, stick to batch.
OpZ: That’s a hard gate: no real-time unless batch already works. What’s the most common pushback you hear from engineers who still want to start with real-time, and what do you say to shut it down fast?
giarec: Most engineers come from non-big-data experience. A real-time architecture is perceived as a better design, very much like event-driven vs. polling. Many projects leverage big-data technologies for not-so-big data, which makes things even more confusing. My pushback is always: if you decided to go for big-data technologies, even if your data is not so big, master batching first. 99% of projects work better as a well-defined batch pipeline.
OpZ: Before they even touch a big-data tool, what’s the one core principle you wish every engineer remembered?
giarec: The core principle is to start by designing how you will partition your data. Big-data architectures scale pipelines by partitioning the data. Most engineers spend way too little time on this core aspect of the design and land on suboptimal architectures and implementations.
OpZ: Partition first, pipeline second. Cool, that’s the takeaway. Appreciate the clarity.

