Three Architectures, One Decision
The three architectures - batch, streaming, and hybrid - each carry different costs, complexity profiles, and operational demands. Engineering teams pick one when they should be picking based on workload requirements rather than on familiarity or vendor enthusiasm. The decision is consequential because data pipeline architecture is hard to change once it has accreted workloads.
Confluent's 2024 customer research found roughly 60 percent of streaming infrastructure carried workloads that did not require streaming, with the streaming architecture chosen because the team had built it or because the vendor enthusiasm exceeded the use case (Confluent, "State of Data in Motion 2024"). The result is unnecessary cost and complexity.
The architecture decision should follow workload analysis. Three architectures, mapped against workload requirements, produce a defensible answer.
The Three Architectures
The three architectures are well-documented but worth describing in operational terms rather than marketing terms.
Batch architecture processes data in discrete chunks on a schedule. Hourly, daily, weekly. The data lands somewhere, accumulates, and gets processed during a window. Tools include Airflow, Dagster, Prefect for orchestration. Spark, Databricks, BigQuery, Snowflake for processing. Operational characteristics: predictable resource usage, well-understood failure modes, longer end-to-end latency from event to availability.
Streaming architecture processes data continuously as it arrives. No accumulation, no scheduled window. Events flow through and produce derived data within seconds. Tools include Kafka, Kinesis, Pulsar for transport. Flink, Spark Streaming, Materialize, Beam for processing. Operational characteristics: lower end-to-end latency, higher infrastructure complexity, more difficult failure modes (exactly-once semantics, state management, recovery from outages).
Hybrid architecture combines batch and streaming for different purposes. Often a Lambda or Kappa architecture pattern. Streaming layer handles fresh data for low-latency consumers. Batch layer handles historical accumulation and reprocessing. The combination serves both real-time and historical queries from the same underlying data flow. Operational characteristics: more complex than either alone, more flexible than either alone.
A workload assigned to the wrong architecture produces operational pain. A workload assigned correctly fits the architecture's strengths.
The Decision Map
Three questions about the workload point toward the right architecture.
Question one is freshness requirement. How fresh does the data need to be when consumers read it. If the answer is seconds, streaming or hybrid. If minutes, near-real-time micro-batch or hybrid. If hours or days, batch.
Question two is consumer pattern. Who reads the data and how. Operational systems that act on data in real time need streaming-fresh data. Analytical systems that produce reports periodically need batch-acceptable freshness. Mixed consumer base needs hybrid or careful workload partitioning.
Question three is processing complexity. What transformations happen on the data. Simple transformations work well in streaming. Complex transformations involving multiple data sources, complex windowing, or historical comparisons usually fit batch better. Hybrid lets each transformation run in the architecture that fits.
A simple decision matrix from these three questions usually identifies the right architecture. Workloads requiring sub-minute freshness with simple transformations and operational consumers are streaming workloads. Workloads tolerating hours of staleness with complex transformations and analytical consumers are batch workloads. Workloads with mixed requirements are hybrid workloads.
What Streaming Actually Costs
The streaming architecture cost premium over batch is substantial and worth understanding before committing.
Streaming infrastructure runs continuously. Batch infrastructure runs during processing windows. Pure compute cost can be 3-5x higher for streaming for equivalent data volume because batch infrastructure can scale to zero between runs.
Streaming engineering complexity is higher. Exactly-once semantics, state management, recovery from partial failures, schema evolution under load. The skills required are scarcer and more expensive than batch engineering skills.
Streaming observability is harder. Failure modes in streaming systems often present as silent degradation (a stream slows but does not error) rather than as clean failures. Catching these requires more sophisticated monitoring than batch needs.
Streaming operational headcount is typically 30-50 percent higher than equivalent batch operations. The combination of complexity and observability difficulty translates to staffing.
These costs are justified for workloads that genuinely need streaming. They are wasted for workloads that do not.
What Batch Actually Limits
Batch architecture has limits too. The limits matter for workloads that approach the edges.
End-to-end latency from event to availability is bounded by the batch interval plus processing time. For hourly batches with 15 minutes of processing, the typical latency is 30-75 minutes from event to availability. Workloads requiring faster than that are batch failures.
Throughput is bounded by the processing window. If the data exceeds what the batch infrastructure can process in the window, the pipeline falls behind. Catching up requires either scaling the infrastructure for the catch-up period or extending the window.
Schema changes are harder to handle gracefully. Streaming systems can adapt to schema changes in flight. Batch systems usually require coordinated migration of producers, schema registries, and consumers.
Workloads near these limits often benefit from hybrid architecture even when their primary mode is batch.
The Hybrid Pattern That Works
Hybrid architectures get over-engineered as often as they get under-engineered. The pattern that produces value rather than complexity is workload-partitioned hybrid: each workload runs in the architecture that fits, and the architectures coexist without being entangled.
Workload-partitioned hybrid looks like this. The streaming layer carries the workloads that need it (operational analytics, fraud detection, real-time recommendations). The batch layer carries the workloads that do not need streaming (daily reports, periodic ML model training, historical analysis). Both layers read from a common upstream data source. Neither layer depends on the other for runtime correctness.
The alternative hybrid pattern (Lambda architecture where every workload runs in both layers) is much more complex to operate. Most teams who tried Lambda in 2018-2022 have moved toward workload-partitioned hybrid because the operational cost of Lambda exceeded its theoretical benefits.
What This Costs at Scale
A mid-market enterprise running mixed workloads in 2026 typically has data pipeline infrastructure costs in the $50K-$300K monthly range depending on volume and architecture choice. Pure-streaming environments tend toward the higher end of this range. Pure-batch environments tend toward the lower end. Hybrid environments can be either depending on workload partition.
The operational headcount typically scales as: pure batch needs 1-3 data engineers for moderate workloads, hybrid needs 2-5, pure streaming needs 3-7. The headcount is more variable than infrastructure cost because team skill distribution affects efficiency.
The total cost picture (infrastructure plus operations plus opportunity cost of engineering time) is what should drive the architecture decision rather than infrastructure cost alone.
What Logiciel Does Here
Logiciel works with data engineering teams whose pipeline architecture has accreted workloads beyond what the original architecture was designed for. The work is typically structured around workload-architecture fit analysis followed by sequenced migration to fit-appropriate architectures.
The Real-Time Data Architecture framework covers the five use profiles that complement the three-architecture decision. The Data Pipeline Cost Optimization framework covers the cost dimension that often drives the architecture conversation.
A 30-minute working session is enough to assess your current architecture against your workload profile.
Frequently Asked Questions
Can I start with batch and add streaming later?
Yes, and this is the pattern most enterprises actually follow. Start with batch for workloads that fit. Add streaming when specific workloads require it. The resulting hybrid is usually workload-partitioned rather than Lambda-style. Easier to evolve into than starting with streaming first.
How do I know my workload needs streaming?
Three signals. The workload's value drops sharply with latency above some threshold. The consumer acts on the data automatically rather than reading reports. The data source produces continuously rather than in batches. Workloads with all three signals usually need streaming.
What is the right team composition for hybrid?
Data engineers with both batch and streaming skills, plus a data architect who can adjudicate workload assignments to architectures. The architect role is what most teams under-staff. The engineers can build both architectures; the assignment decisions need someone with cross-architecture perspective.
How does this work with vector databases and AI workloads?
Vector databases sit alongside the pipeline architectures, not inside them. Embedding generation can run in batch or streaming depending on requirement. Vector storage and query is operational, not pipeline. Most AI workloads use batch pipelines for offline embedding work and live queries against vector stores.
When does Apache Beam or other unified frameworks help?
When the same processing logic genuinely needs to run in both batch and streaming modes. The frameworks reduce code duplication. They do not reduce operational complexity meaningfully. Useful for specific patterns and over-applied generally. Sources: - Confluent, "State of Data in Motion 2024" - Snowflake, "Cost Optimization Patterns," 2024
