For analytics and data platform leads
Your warehouse is where the organisation agrees what is true, and a supplier whose numbers arrive as a manual extract sits outside that agreement permanently. What is needed is a scheduled load into storage you control, a documented schema that changes with notice, and figures that reconcile against the operating screens without a person adjudicating.
A data platform exists so that an organisation stops arguing about numbers. Once a source lands there, its figures join the same definitions, the same refresh, the same lineage and the same governance as everything else. A source that does not land there is permanently outside that agreement, and every conversation about it begins by establishing whose number is right.
The usual arrangement is a monthly file. Somebody in operations exports it, somebody in analytics reshapes it, and by the time it reaches a report the numbers have passed through two people’s judgement about what a column meant. It is nobody’s fault and it is not repeatable, and the person who built the reshaping step is the reason it still works.
Where an interface exists, the shape of it frequently defeats a bulk load. Endpoints designed for an application ask you to page through records at a rate that turns a full pull into an overnight job, or several, and offer no way to ask for only what changed. A pipeline built on that is a pipeline that gets slower every quarter.
The schema is the part that breaks quietly. A field changes meaning, a code list gains a value, a column that was always populated starts arriving empty for a subset of records — and nothing errors. The load succeeds, the dashboard renders, and a number is wrong for a fortnight until somebody senior notices it does not match what they saw on a screen.
And the deletion problem nobody plans for: a record removed at the source stays in the warehouse forever unless the load knows about removals. A retention obligation satisfied in one system is quietly unsatisfied in the copy the analytics team made, which is a compliance finding waiting for an auditor who thinks to look.
A data platform needs a load it can schedule, a way to ask only for what changed, a schema that announces its changes, and removals it can apply — and an interface designed for an application supplies none of those.
The bulk path is separate from the application interface for exactly that reason. It is designed to be pulled by a pipeline rather than by a screen: whole tables at once, with a watermark so a subsequent run asks only for what changed since the last one. That single property is what keeps a pipeline’s cost flat as the record count grows, and it is what an application endpoint cannot give you.
Removals travel with it. A record deleted at the source appears in the change feed as a removal rather than simply ceasing to appear, so the warehouse can apply it. Without that, a deletion is invisible to a pipeline and the copy silently diverges — which is both a correctness problem and, where retention obligations exist, a compliance one.
Every field is described in a dictionary written in ordinary language rather than in ours, and the dictionary is versioned alongside the data. A field description that lives in somebody’s memory is the reason two teams compute the same measure differently, and a dictionary that is not versioned cannot tell you when a meaning changed.
Schema changes are additive and announced. Fields are added rather than repurposed, code lists grow rather than change meaning, and a change reaches your nominated contact before it appears in a load rather than being discovered in a dashboard. The failure this prevents is the expensive one: not a pipeline that breaks loudly, but a pipeline that keeps working and is quietly wrong.
And reconciliation is designed for rather than hoped for. The figures a pipeline computes are meant to agree with what the operating screens show, and where a definition differs — a count taken at a different moment, a status that means something narrower — the difference is documented rather than left for two teams to discover in a meeting. A supplier who cannot explain why their screen and their export disagree has two definitions and knows about neither.
Whether a load can be scheduled without a person — measured by running the pipeline twice unattended and confirming both runs complete.
Whether a second run is incremental — measured by comparing the volume and duration of the first run against the second.
Whether deletions reach the warehouse — measured by deleting a record at the source and confirming the next run removes it downstream.
Whether a field can be understood without asking us — measured by handing the dictionary to an analyst who has never used the product and watching them build a measure.
Whether a schema change would be announced — measured by the written notice commitment and the nominated contact it reaches.
Whether the export reconciles with the screens — measured by computing one figure both ways and confirming they agree or that the difference is documented.
no claim that the bulk path reproduces application latency — it is designed for a pipeline rather than for a screen. No claim to model your business definitions for you; the dictionary describes fields rather than deciding what your organisation means by a measure. No claim that a historical figure will never be restated, because a correction upstream is a correction. An independent SOC 2 Type II attestation is in progress and no report exists yet.
The load target is yours — your object storage, your warehouse, your lake — and the credentials are ones you issue and can revoke. Nothing here requires a copy of your warehouse credentials to be held on our side, which is a smaller claim than it sounds and it removes an entire category of question from your review.
Formats are the ones a warehouse already ingests, with the dictionary shipped alongside the data rather than living on a documentation site that ages separately. A dictionary that is not in the same place as the data is one that will eventually describe a different version of it.
And where your organisation already has a preferred orchestration tool, it should be the thing running this rather than a scheduler of ours. A pipeline your data team cannot see inside their own tooling is one they will not be able to debug at the moment it matters.
A pipeline that breaks is cheap: somebody is paged, the cause is found, the load reruns. A pipeline that keeps working while a field quietly changed meaning is expensive, because the wrong number survives long enough to be used in a decision, and the discovery is usually somebody senior noticing that a dashboard disagrees with a screen.
So the commitments here are weighted towards announcement rather than towards throughput: additive changes only, notice before a change reaches a load, a versioned dictionary, and documented differences where a screen and an export legitimately disagree. Those are less impressive than a performance figure and they are the ones that determine whether the numbers can be trusted.
What is deliberately not claimed is that a historical figure will never move. Corrections happen upstream, and a supplier claiming perfect immutability of reported figures is describing a system that cannot fix its own mistakes. What is committed is that a restatement is visible rather than silent.
Every property here is designed rather than independently attested. A SOC 2 Type II attestation is in progress and no report exists yet; it is an attestation with a scope and a period rather than a certification.
Run the load twice and compare the second run against the first. If the second run costs what the first did, there is no watermark and your pipeline expense grows with your record count forever. This is the single most consequential test on this page and it takes two runs.
Delete a record at the source and confirm it leaves the warehouse on the next load. A copy that never forgets is a retention obligation your organisation is quietly failing in a system nobody thought to include in the scope.
Hand the dictionary to an analyst who has never used the product and ask them to build one measure. If they have to ask a question, the dictionary is a field list rather than a dictionary, and the difference will cost you every time somebody new joins.
And compute one figure both ways, from the export and from an operating screen. Agreement is good; a documented explanation of a legitimate difference is equally good; a shrug is the finding.
A trial load into storage your team controls, run twice, followed by deleting one record at the source and running it a third time.
Three runs and one deletion settle almost everything on this page. The second run reveals whether the load is incremental; the third reveals whether your copy can forget. Both are properties nobody can establish from a document.
Do it into your own storage with your own orchestration from the start rather than watching a demonstration, because the thing being tested is whether your team can own the pipeline rather than whether a pipeline exists.
Then hand the dictionary to somebody new and have them build one measure end to end. That is the test of whether this source can join the warehouse or will remain a thing one person understands.
Because an interface designed for a screen is designed to return a small number of records quickly, and a pipeline needs the opposite — whole tables, and then only what changed. Paging an application endpoint for a bulk load turns a full pull into an overnight job that gets slower every quarter, and it usually cannot tell you what changed at all. The two are different problems and a supplier who offers one surface for both has optimised for the demonstration rather than for the pipeline you will run every night for five years.
That is the most common quiet compliance failure in an analytics estate, and it is caused by change feeds that represent a deletion as an absence — a pipeline cannot distinguish a deleted record from one that simply was not in this page of results. Deletions travel as explicit removals here so your load can apply them. Test it directly: delete one record at the source and run the load again. If it survives, your retention obligations are satisfied in one system and broken in the copy, and an auditor who thinks to look will find it.
Usually because two definitions exist and nobody wrote either of them down — a count taken at a different moment, or a status that means something narrower in one place. Where an export and an operating screen legitimately differ here, the difference is documented rather than left for two teams to discover in a meeting. Test it by computing one figure both ways during the trial. Agreement is a good answer, a documented explanation is an equally good answer, and a shrug is the finding you were looking for.
Changes are additive — fields are added rather than repurposed and code lists grow rather than change meaning — and notice reaches a contact you nominate before the change appears in a load. That combination targets the expensive failure rather than the cheap one: a pipeline that breaks gets fixed the same day, while a pipeline that keeps working after a field quietly changed meaning produces a wrong number that survives a fortnight and gets used in a decision.
Nor should you, and none are held here. The load runs from your own orchestration tool into your own storage, using credentials your organisation issues and can revoke, and there is no copy of a warehouse credential on our side. That removes an entire category of question from your review and it also means your data team can see inside the pipeline in their own tooling — which matters at the moment it fails, because a pipeline they cannot inspect is one they cannot debug.