However this results in our fifth drawback, which is similar-yet-different information units. Why are there multiples? Which one ought to I take advantage of? Is that this information set nonetheless maintained, or is it a zombie information set that’s nonetheless commonly up to date however with out anybody overseeing it? The issue involves a head when you have got essential computations that disagree with one another, as a result of counting on information units that ought to be an identical however should not. Offering conflicting reviews, dashboards, or metrics to clients will end in a lack of belief, and in a worst-case situation, lack of enterprise and even authorized motion.
Even in the event you kind out all of those issues—decreasing latency, decreasing prices, eradicating duplicate pipelines and information units, and eliminating break-fix work—you continue to haven’t offered something that operations can use. They’re nonetheless on their very own, upstream of your ETLs, as a result of the entire cleansing, structuring, transforming, and distribution work is just actually helpful for these within the information analytics area.
Shift left for a headless information structure
Constructing a headless information structure requires a rethink of how we flow into, share, and handle information in our organizations—a shift left. We extract the ETL->bronze->silver work from downstream and put it upstream inside our information merchandise, a lot nearer to the supply.