No one recordsdata a ticket that claims “our structure has an abstraction drawback.” They file tickets saying the information is fallacious, or lacking, or late. So engineering spends two weeks chasing a data-quality challenge that doesn’t exist, fixes nothing, and the identical ticket comes again the next quarter sporting a barely totally different hat.
That was us. Essentially the most helpful factor I realized from the entire effort is that the bug was by no means within the knowledge. It was in what we had been asking the information to be.
We had an on-premises MongoDB occasion serving because the registered golden supply for enterprise reference knowledge. Codes, classifications, id lookups, the unglamorous shared knowledge that quietly underpins buyer onboarding, regulatory reporting, and a dozen different issues individuals solely discover once they break. It was well-maintained, authoritative, the real single supply of fact. The crew that owned it was rightly pleased with it. By each cheap measure, the system was wholesome.
And but each time an analytics crew or a downstream product group wanted one thing from it, the expertise was depressing. They reverse-engineered the operational schema. They wrote one-off queries towards nested JSON they solely half understood. They tracked down whoever nonetheless carried the institutional reminiscence of the gathering construction, waited, after which repeated all the ritual three months later when the requirement shifted by an inch.
The analysis took longer than it ought to have
I watched this play out for months earlier than it clicked. The info was high-quality. We had been asking an operational retailer to moonlight as an analytical platform, and it was dangerous on the second job. Not by way of any flaw of its personal. It was merely by no means constructed for that.
Operational shops optimise for correctness and life cycle administration. Analytics groups want one thing else completely: secure shapes, fields which might be really documented, a refresh cadence you may predict, and a method to choose whether or not a dataset is match for objective with out reverse-engineering another person’s schema. These aren’t the identical necessities, and conflating them is exactly how you find yourself with a system that’s technically excellent and virtually ineffective. Wholesome uptime, depressing customers.
So we stopped asking individuals to eat reference knowledge instantly from MongoDB. We began treating every dataset as an information product: one thing with a named proprietor, a definition, high quality gates, ruled entry, and an actual path to publication. The technical pipeline, MongoDB by way of Kafka Join into Touchdown, Bronze and Silver layers as Iceberg tables on S3, Athena on high, publication by way of the Knowledge Market, adopted from that call moderately than driving it. Twenty-one reference knowledge merchandise finally shipped down that single path.
Determine 1: The complete pipeline. MongoDB because the authoritative golden supply, occasions flowing by way of Kafka into Touchdown, Bronze and Silver layers as Iceberg tables on S3, Athena offering the question floor, and the enterprise Knowledge Market because the publication endpoint. Airflow orchestrates all the things; DPPS UI offers operational visibility.
What “knowledge product” really pressured us to resolve
“Knowledge product” is a type of phrases that may imply nearly something, which normally means it means nothing. So we made it imply one thing particular and non-negotiable: a dataset couldn’t be revealed till it had a named proprietor, an information dictionary, enterprise and technical metadata, documented audit expectations, high quality gates, and a ruled route into the Market. Compliance with all energetic requirements at deployment time was necessary, enforced at publication, not requested in a evaluation assembly.
That framing instantly surfaced questions that ought to have been answered years earlier. What’s the precise boundary of this product? Which attributes matter to customers, and that are operational plumbing no person exterior the proudly owning crew cares about? What does “present” imply for this dataset, and the way would a shopper know if it had gone stale? How does anybody uncover it with out submitting a ticket and ready for a human to level them on the proper S3 path?
None of that was governance overhead bolted on for present. Answering these questions was the structure. The Kafka connectors and Iceberg tables had been nearly the simple half by comparability.
The three choices that formed all the things else
The primary choice was to maintain MongoDB because the golden supply. No rip-and-replace. Authority stayed the place it belonged, with the crew that understood the information’s lifecycle and had maintained it accurately for years. The enterprise requirement was express: no business-logic transformation, a one-to-one mapping from supply to vacation spot, devoted preservation moderately than enrichment. The temptation to crown a shiny new system because the supply of fact lurks in each modernisation challenge, and it’s nearly all the time fallacious. MongoDB did its job properly. We had been constructing a supply layer, not changing a basis, and complicated the 2 is how good migrations flip into eighteen-month disasters.
The second was to construct one supply mannequin as a substitute of tolerating 4. Earlier than this work, a minimum of 4 groups had independently extracted roughly the identical reference knowledge, every with its personal refresh logic, its personal studying of the sphere semantics, and its personal personal definition of “present.” The diplomatic phrase for that state of affairs is “decentralised.” The trustworthy phrase is chaos. Occasions flowing from MongoDB by way of Kafka Join into the pipeline, Airflow orchestrating a month-to-month batch on the fifth at 07:00 UTC with no dependency on working days or vacation calendars, schema validation firing earlier than something touched S3, changed all 4 personal empires with a single path anybody might motive about.
The price of these 4 pipelines was by no means the compute or the storage, which was trivial. It was the reconciliation tax. Every time two copies disagreed, they usually did, somebody senior and busy needed to work out which one to consider. Multiply a half-day investigation by each quarter and each consuming crew and also you arrive at a genuinely costly behavior that by no means appeared on any price range line, as a result of it was hidden inside everybody’s unusual work. Collapsing 4 pipelines into one didn’t simply simplify the diagram. It deleted a complete recurring class of argument.
The third was to deal with publication as an actual pipeline stage moderately than an afterthought. Knowledge that reached Silver obtained revealed into the Knowledge Market with metadata, a Kitemark high quality rating, documentation, and subscription behaviour already hooked up. Consumption occurred completely by way of the Market subscription mannequin, by no means by handing somebody an S3 path. Customers might discover a product, choose whether or not it match, and subscribe to it without having to know which bucket to ask about or which Slack channel to beg in. Publication meant the product went dwell. It didn’t imply a file quietly appeared in storage and somebody hoped the fitting individuals would discover.
The boring stuff turned out to be the laborious stuff
I saved ready for the laborious issues to indicate up within the pipeline itself. Kafka connector configuration, Iceberg desk upkeep, Athena partition tuning, all of it wanted consideration, and all of it obtained sorted sooner or later. However the hole between “a pipeline that works” and “a platform individuals belief” got here from the issues I used to wave off as housekeeping. Naming conventions. Audit column requirements. Documentation templates somebody would really open. Possession that was actual moderately than nominal.
Naming is an efficient instance of how unglamorous and the way decisive this will get. A shopper looking out {the catalogue} has to discover a dataset utilizing enterprise-standard terminology, not the inner shorthand that made sense to the crew that constructed it. The metadata framework mapping to the enterprise customary is tedious work that exhibits up on no demo. It’s also all the distinction between a listing individuals can navigate and an inventory of cryptic desk names solely the authors perceive.
Right here is the uncomfortable half I didn’t respect moving into: shared enterprise knowledge tends to fail socially earlier than it fails technically. The Kafka connector can be high-quality. What corrodes is the shared understanding of what “authoritative” means in observe, whether or not a given dataset is the actual one or a duplicate any individual made eighteen months in the past and forgot to deprecate. No quantity of Iceberg optimisation touches that. You repair it on the layer the place customers resolve whether or not to belief a dataset, which is the product layer, and nowhere else.
A concrete instance of how social this will get. Early on, two groups disagreed about which currency-code dataset was appropriate. Each had been internally constant. Each had been “proper” sooner or later. The distinction got here right down to a refresh one crew had quietly stopped operating a yr earlier, and neither crew might show which copy mirrored the dwell supply, as a result of nothing in both dataset recorded the place it got here from or when. We didn’t repair that with a greater connector. We mounted it by making provenance a first-class column. Each Silver document now carries SOURCE_SYSTEM, JOB_RUN_ID, VALID_FROM and VALID_TO, so the query “is that this the actual one, and is it present?” has a documented reply as a substitute of a hallway debate.
Storage will not be the product
I’ve watched groups land knowledge in S3, declare victory on self-service, after which spend six months baffled that no person is utilizing it. The reply is sort of all the time the identical. “The info is in S3” will not be a product. It’s a location. Individuals have to know the information exists, work out what it means, choose whether or not it suits their objective, and discover out who to contact when one thing appears fallacious. A path offers them none of that.
The Market addressed this greater than any particular person pipeline element did. It turned a scattered set of S3 paths right into a ruled catalogue of subscribable merchandise, every with documentation, a top quality rating, and clear possession. That’s the distinction between handing somebody a warehouse handle and handing them a store. And since subscription is the one sanctioned path to the information, {the catalogue} stays the one entrance door moderately than one possibility amongst a number of personal again channels.
Separate fact, transport, and consumption
If I had 5 minutes with somebody beginning this work, I’d spend all of it on one concept. Separate fact, transport, and consumption, and deal with them as three totally different considerations owned by three totally different components of the system. MongoDB holds fact, and stays authoritative. The pipeline, Touchdown by way of Bronze to Silver, strikes that fact reliably and proves it arrived intact with checksum reconciliation and inter-layer record-count checks. The product layer, Silver tables, Athena, and the Market, makes fact consumable by individuals who have no idea and may by no means have to understand how MongoDB organises its collections.
Determine 2: The identical knowledge, three separated planes. Fact stays within the operational golden supply; transport strikes it and proves it arrived intact; consumption exposes it as ruled, subscribable merchandise. Separating the three considerations, every with its personal proprietor, is what removes the friction between producers and customers.
When these three are genuinely separate, an unlimited quantity of organisational friction merely evaporates. Producers cease getting dragged into ad-hoc reporting. Customers cease reverse-engineering operational intent. The ops crew can evolve the MongoDB schema with out shattering six downstream jobs. And a brand new crew that wants nation codes or forex classifications can discover them within the Market, learn the documentation, and be achieved in a day as a substitute of 1 / 4.
The info was all the time high-quality. What we really constructed was the boundary that permit everybody cease arguing about it.