Data pipelines and automation
Data pipelines in dbt, migrations with reconciliation, and integrations across Microsoft 365, Salesforce, Genesys Cloud and more, so a process runs unattended.
Pipelines fail quietly. Reports drift. A migration finishes and nobody is quite sure whether the numbers still reconcile. Most of the cost in data engineering is not writing the first version; it is keeping it trustworthy after the people who built it have moved on.
We build four kinds of thing: the automation that replaces a manual process, the integration that moves data between systems without anyone re-keying it, the migration that reconciles before it cuts over, and the model that turns all of it into numbers someone can plan on. Each comes with the tests, documentation and handover that keep it running after we have left.
Figure 1: one refresh, before and after
Before · the Monday refresh, by hand
- 08:10 Export last week from the case system 12 min
- 08:25 Paste into the planning workbook, fix the dates by hand 35 min
- 09:05 Re-key the totals into the finance template 40 min
- 09:50 Rebuild the pack, check it by eye 55 min
- 10:50 Email the pack 5 min
Elapsed 2 h 45 m · Owner: one analyst · Runs: when there is time
After · the same refresh, scheduled
- 06:00:00 Run started on the weekday schedule
- 06:00:04 Extract from the case system to staging 18,204 rows
- 06:01:37 Transform in dbt 41 tests passed
- 06:02:10 Reconcile against the manual pack 0 differences
- 06:02:15 Load to the warehouse, refresh Power BI
- 06:07:12 Run complete · pack published · no alerts
Elapsed 7 m 12 s · Owner: the team · Runs: every weekday
-
The schedule replaces the person. It runs whether or not anyone is in, and an alert goes to a channel rather than an inbox.
-
Tests catch the silent failure. The join that multiplied rows, the filter that dropped a region, the date that slipped a financial year.
-
Reconciliation before cut-over. Old and new run side by side, and every difference is explained before the manual version is switched off.
-
Handover is in the scope. Documentation and working sessions with the people who will own it, so the run does not depend on us.
Where this fits
Three situations this is built for
-
A process runs on exports, re-keying and one person's spreadsheet.
Data leaves one system by hand and enters another by hand. The refresh happens when that person has time, and when they are away it does not happen.
-
Reporting exists, but nobody trusts the numbers or knows how they are made.
Dashboards were built fast and drifted. Two reports disagree and the fix is a conversation rather than a test. It needs modelling, tests and documentation underneath.
-
A migration, integration or model needs building, and there is nobody to build it.
A warehouse move, a Salesforce or Microsoft 365 integration, a forecasting pipeline: defined work with an end point, needing someone who has shipped it before and will hand it over.
What you get
Delivered, not recommended
- Automation and integrations across Microsoft 365, Salesforce, Genesys Cloud and internal systems, so a process runs unattended.
- Warehouse and semantic models in dbt with tests, documentation and lineage.
- Migration plans with reconciliation: the old and new numbers are compared, and the differences are explained before cut-over.
- CI for data: pull-request checks, scheduled runs and alerting through GitHub Actions or the platform's native tooling.
From the work
Automated
Forecast refresh, previously manual
Forecasting pipelines that replaced manual work
New Zealand government · same engagement
Forecasts were produced through manual processes: exports, re-keying and hand-built refreshes that depended on the people doing them and left little time for the forecast itself.
The manual processes were replaced: refreshes ran unattended, and the time that used to go into the mechanics went into the forecast instead.
Read the noteQuestions
Asked before the first call
Which platforms do you work with?
Most of our work has shipped on Snowflake and Databricks for warehousing, dbt for modelling, Power BI for reporting, Power Automate and Microsoft Graph for automation, and Salesforce and Genesys Cloud as source systems. The methods are not tied to any of them. If your stack is different, say so in the scoping call; a different warehouse or reporting tool is rarely the thing that decides whether we can help.
How do you handle a migration without disrupting reporting?
Old and new run side by side for a period. Every report that matters is reconciled between the two, and the differences are documented and explained before anything is switched off.
Will my team be able to maintain what you build?
Yes. That is the point of the build. Models are documented and tested, the orchestration is visible, and handover includes working sessions with the people who will own it.