The stack between an event and action
became the problem.
Organisations do not lack policies, processing agreements or data tools. They lack one operating layer that keeps the approved meaning, quality, rules, action and evidence together while data moves.
Operational reality
More components created more handovers, not more control.
Each tool may solve a technical task and each contract may document responsibility. The combined live operation still leaves the organisation accountable for the changing gaps between them.
Too many moving parts
Collectors, queues, pipelines, transforms, orchestration and monitoring each create another boundary to operate.
Context arrives too late
The business situation is reconstructed in a warehouse after the moment for the decision has already passed.
The agreement cannot control the code
A processing agreement records intent, but cannot by itself prove that every pipeline version, rule and destination still follows it.
Architecture follows the provider
Business operations become entangled with one cloud platform’s services, formats and commercial model.
The customer inherits operations
Buying components still leaves sizing, upgrades, retries, observability and recovery with the customer team.
Agents create another shadow path
Direct agent integrations bypass shared context, deterministic policy and the evidence required for accountability.
The old question
Which products do we need to build the pipeline?
This starts with infrastructure and makes the business operation adapt to the stack.
The Datastreams question
What must happen while this data is moving?
This starts with source, meaning, quality, purpose, rules, context, outcomes and evidence. The approved runtime definition becomes the operating agreement.
The component tax
Every new obligation became another product, code path and bill.
Transport was simple. Then came validation, transformation, identity, consent, policy, orchestration and observability. The architecture grew faster than the business operation.
A typical chain
collector → queue → schema → transform → identity → consent → policy → orchestrator → loader → monitor
Code sits beside the data, between integrations and inside manipulation jobs. No single definition explains the complete operation.
What the invoice hides
- Another service licence and commercial model
- Integration code between two more boundaries
- More data movement, copies and retention work
- Monitoring, retry and recovery ownership
- Consent and purpose logic repeated in multiple places
- Evidence assembled after the fact for audits
The Big Tech distinction
The issue is not whether an AWS or Microsoft component works. It is who owns the complete operation.
Cloud platforms offer capable building blocks. Datastreams is different because it starts with the governed business outcome and accepts an agreed responsibility for the runtime that executes it.
A component price is not an operation price
Ingestion, execution, storage, networking and observability are metered separately. A low price for one step says little about the cost of the complete outcome.
Managed components do not manage the chain
A provider may operate each product reliably while your team still owns contracts, handovers, ordering, retries, permissions and evidence between them.
Open source is a building choice, not a cost model
An open-source engine can reduce licence dependency, but hosting, managed-service margins, integration code, upgrades and specialist operations remain part of the service total.
Replace the operational stack one stream at a time.
Select one recurring flow, define the required outcome and make every current component, cost, handover and governance obligation visible.
Map the first stream