The layer underneath the model
Models rarely fail first — the data underneath them does
We build the ingestion, contracts, lakehouse and semantic layer that AI systems depend on. The test is simple: every number traces back to a source column, and an upstream schema change breaks a build rather than a board meeting.
Engagement at a glance
- Capabilities
- 05
- Deliverables
- 08 artefacts
- Stack groups
- 04
- Ways to start
- 03
Engagement options
- Platform Assessment2–3 weeks
- Foundation Build10–14 weeks
- Embedded Platform Team6–12 months, rolling
What breaks
The failure modes we keep meeting
Cheap to design around at the start. Expensive to discover once the system is carrying real traffic.
The silent schema break
A service team renames a column on a Tuesday and ships it. CDC keeps replicating happily — the new column simply arrives as a new field. The join in the feature pipeline now matches nothing, the model scores against nulls, and the first real signal is a support ticket eleven days later. Nothing failed loudly because nothing had agreed to fail.
Nobody can answer where the number came from
An auditor, a regulator or a CFO asks how a figure was derived. The honest answer is a chain of untracked notebooks, a spreadsheet one analyst maintains, and a view whose author left in 2024. Meanwhile four systems each define “active customer” differently, so every review burns its first twenty minutes on reconciliation instead of decisions.
Nightly batch, real-time promises
The roadmap wants fraud scoring, live personalisation and agents that know what happened an hour ago. The warehouse loads at 02:00. That gap is architectural, not a tuning problem — with no event backbone, no stream processing and no online store, the whole class of real-time features is unbuildable regardless of which model is chosen.
Capabilities
What the work actually consists of
01
Ingestion and change data capture
Getting data out of operational systems without coupling to them, and without a nightly full-table scan that the DBA quietly hates.
- Log-based CDC from PostgreSQL, MySQL, SQL Server and MongoDB using Debezium and Kafka Connect, with snapshot-then-stream cutover and replication-slot monitoring
- SaaS and third-party sources via Airbyte or Fivetran connectors, with incremental cursors, rate-limit backoff and reconciliation counts per sync
- An immutable landing zone in Iceberg — raw payload, ingestion timestamp, source offset — so any downstream model can be rebuilt from history rather than re-extracted
- Dead-letter topics and replay tooling for malformed records, so one bad message never halts a partition
- PII tagged at the point of landing, not discovered later during an audit
02
Event backbone and stream processing
A durable log the whole estate can read from, so real-time features stop being a special project each time someone asks for one.
- Kafka or Redpanda topic design — keying, partition count, retention and compaction chosen against the actual read patterns, not defaults
- Avro or Protobuf schemas in a registry with enforced compatibility mode, so a producer cannot deploy an incompatible change
- Apache Flink jobs for windowed aggregation, enrichment joins and sessionisation, with checkpointing and exactly-once sinks where the cost is justified
- Late and out-of-order event handling with explicit watermarks, plus a stated policy for what happens to data that arrives after the window closes
- Online feature serving so a model reads the same computed feature in training and at inference
03
Lakehouse and modelled transformation
One storage layer that analytics, ML and ad-hoc SQL all read, with transformation that is version-controlled, tested and reviewable like any other code.
- Apache Iceberg tables with partition and sort strategy derived from real query predicates, plus scheduled compaction and snapshot expiry so small files do not strangle scan performance
- A layered dbt or SQLMesh project — staging, intermediate, marts — with incremental models, snapshots for slowly changing dimensions and documented grain on every table
- Engine choice made on workload: Trino or DuckDB for interactive, Spark for heavy backfills, ClickHouse where sub-second aggregation matters
- Orchestration in Dagster or Airflow with asset-level dependencies, retries and SLAs, so a late upstream fails visibly instead of producing a stale mart
- Warehouse cost instrumented per model and per query pattern, reviewed as an engineering metric
04
Contracts, quality and lineage
The part most teams skip, and the reason their pipelines are fragile. Producers commit to a shape; consumers get a build failure instead of a silent null.
- Data contracts as versioned schemas with owner, SLA and semantics, enforced by a CI check that fails the producer’s pull request on a breaking change
- Column-level lineage via OpenLineage emitters on every job, surfaced in DataHub, OpenMetadata or Unity Catalog so impact analysis takes minutes
- Quality tests at the boundary — uniqueness, referential integrity, accepted ranges, freshness — with Great Expectations, Soda or dbt tests wired into the orchestrator
- Circuit breakers that stop a bad load from propagating, with a documented decision on when to serve stale data rather than wrong data
- Freshness and completeness SLOs per dataset, with a named owner and an alert that pages someone real
05
Serving: semantics, features, vectors and activation
Where the platform earns its keep — one governed definition of each entity, exposed to BI, to models and to retrieval systems alike.
- A semantic layer in Cube or the dbt Semantic Layer holding a single governed definition of revenue, active customer and order, consumed by BI and by agents through the same interface
- Embedding pipelines with content hashing and incremental re-embedding, so a document edit does not trigger a full corpus rebuild and model upgrades have a migration path
- Vector storage sized to the corpus — pgvector where the data already lives in PostgreSQL, Qdrant or LanceDB where filtered ANN at scale is the requirement
- Reverse ETL through Hightouch or Census to push governed segments back into CRM, support and marketing tools, with idempotent writes and sync observability
- Row and column access policies plus masking, so the same table serves an analyst and an auditor under different rules
Outcomes
Targets, and where each number comes from
The basis line under every figure is the point of this section. Where a number is a design standard rather than a measured delivery, it says so — we would rather be checkable than impressive.
- Freshness on the tables that feed live decisions
- Minutes, not overnightFreshness on the tables that feed live decisionsDesign target for the streaming path we build; batch-only estates typically sit at 6–24 hours
- Producer schema changes caught before they reach consumers
- Break the build, not the reportProducer schema changes caught before they reach consumersDesign principle enforced by contract checks in producer CI — not a measured client result
- Per core entity, held in the semantic layer and reused everywhere
- One definitionPer core entity, held in the semantic layer and reused everywhereArchitectural principle of the engagement; we make no claim about reconciliation time saved
- Warehouse spend commonly recoverable on a first tuning pass
- 20–40%Warehouse spend commonly recoverable on a first tuning passTypical range for untuned Snowflake, BigQuery or Databricks estates — partitioning, materialisation and warehouse sizing. Entirely dependent on starting state; not a guarantee
Stack
What we reach for, and when
Defaults, not dogma. The list below is what we would propose on a blank page; an existing estate, a procurement constraint or a team's operating experience all legitimately move it.
- Debezium
- Kafka Connect
- Apache Kafka
- Redpanda
- Confluent Schema Registry
- Airbyte
- Fivetran
- AWS DMS
- Apache Iceberg
- Delta Lake
- Apache Spark
- Apache Flink
- Trino
- DuckDB
- ClickHouse
- Snowflake
- BigQuery
- Databricks
- dbt Core
- SQLMesh
- Dagster
- Apache Airflow
- Cube
- dbt Semantic Layer
- Apache Polaris
- Apache Arrow
- OpenLineage
- DataHub
- OpenMetadata
- Unity Catalog
- Great Expectations
- Soda Core
- Apache Ranger
- Feast
- pgvector
- Qdrant
- Hightouch
Deliverables
What you keep
Everything below lands in your repositories and your accounts, under your licence, with the reasoning written down. There is no runtime you have to keep renting from us.
08 artefacts, handed over
- A written architecture decision record set covering table format, catalogue, the batch/streaming boundary and engine choice — each with the tradeoff stated, so the next team knows why
- Terraform modules for the whole platform — storage, catalogue, compute, IAM, network — with separate dev and production workspaces and no click-ops in the path
- A running ingestion layer across the agreed source systems, with CDC cutover notes, backfill procedure and replay tooling
- A layered dbt or SQLMesh repository with tests, documented grain, exposures and a generated docs site your analysts can actually navigate
- Versioned data contracts for the critical tables, plus the CI check that enforces them in producer repositories
- A populated catalogue with column-level lineage, dataset owners and freshness SLOs wired to alerting
- A PII register: tagged columns, masking and access policies, retention rules and the residency decision written down
- A cost model and dashboard broken down by pipeline and query pattern, with the three most expensive things named and a plan for each
Engagements
Three honest ways to start
Sized so the first one can end. Each option is designed to produce something usable even if we never work together again.
Platform Assessment
2–3 weeks
You have a warehouse and a stalled AI initiative, and no agreement inside the team about which one is at fault.
Includes
- Source and pipeline inventory with a real dependency map, not the one on the wiki
- Lineage spike on two critical metrics, traced end to end by hand to find where it breaks
- Warehouse cost and query teardown
- Readiness review against the specific AI use case you want to ship
- Prioritised remediation plan with effort bands and the decisions recorded as ADRs
Foundation Build
10–14 weeks
You need a platform that works in production, for one or two domains, before scaling it across the business.
Includes
- Lakehouse and catalogue stood up in your cloud account, managed in Terraform
- CDC ingestion for the agreed source systems with backfill and replay proven
- Marts for two business domains, tested and documented, with grain agreed in writing
- Data contracts on the critical tables plus CI enforcement in producer repositories
- One streaming path end to end, chosen for a use case that genuinely needs it
- Catalogue, lineage, freshness SLOs, alerting and an on-call runbook handed over
Embedded Platform Team
6–12 months, rolling
The platform exists and now has to absorb new domains, new consumers and an AI roadmap without accumulating debt.
Includes
- A standing squad working inside your repositories and your on-call rotation
- New domain onboarding at an agreed cadence, each with contracts and tests from day one
- Continuous cost and performance tuning, reviewed monthly against a stated budget
- Semantic layer curation and governance review as definitions evolve
- Deliberate knowledge transfer — pairing, written runbooks and an exit plan that does not depend on us
Questions
The awkward ones
The questions that decide whether this is worth starting — answered the way we would answer them on a call.
A warehouse is storage and compute. It is not contracts, lineage, a semantic layer or an event backbone. An estate can hold every row it needs and still be unable to answer where a number came from, or guarantee that answer survives next sprint’s deploy. The warehouse is the easy half.
Adjacent work
These are the practices this one leans on and the reference architectures that show it assembled.
Related solutions
Reference architectures
Next step
Bring us the problem you keep deferring.
A 45-minute working session on your Data & Platform Engineering work. We will tell you what we would build, what we would not build, and roughly what it costs. No deck.