Choosing slice boundaries when a pipeline grows past one repo
Teams split pipelines when merge queues slow down or when compliance wants tighter access boundaries. A bad slice creates duplicate dimension logic and opaque failures.
Slice on failure domains
Group jobs that share a scheduler pool and on-call rotation. Marketing experiments and general ledger loads should not share a deploy token if their blast radius differs.
Keep shared dimensions in one owner repo
Customer, product, and calendar dimensions need a single maintainer team. Downstream repos consume published artefacts — versioned tables or packages — instead of copying SQL.
Observability travels with the slice
Each slice should expose a health dashboard: last success, rows processed, and open SLA breaches. Splitting code without splitting alerts recreates monolith confusion in Slack.
When to pause splitting
If two squads still debate grain weekly, fix modelling before drawing repository lines. Structure follows agreed semantics, not the other way around.
Pipeline design sessions often cover this topic in the first hour.