Why do data warehouses take forever?

Is your company’s data warehouse working the way you want it to? Are you happy with your dashboards? Do all the sources of data you produce feed into it?

It’s possible that if you’re using the ‘old methods’, the answer is ‘probably not’. Sadly this is going to change anytime soon if your business changes with any pace. Indeed, you likely have a series of overlapping change requests and a reluctance to introduce more until you can have assurance of some progress, or even some semblance of currency.

Rising above the change requests, a more pertinent question is: ‘Does your data warehouse reflect the current reality of your business?’ Certainly not if you are still waiting on those pending changes. In fact, I’ll bet you have probably resigned yourself to all of this lag and labelled it ‘business as usual’. You might be, if you are proactive, be looking out for technology solutions, but it need not be like that.

Using just the ‘tried and tested’ approaches, anytime you change one thing in your data warehouse there are usually knock on effects such as dependent systems. As a result your data warehouse must usually alter (and test!) iteratively until its data model can cope. This happens a lot, and many have resigned themselves to accepting this as the price of change in ‘modern’ data warehouse technology. We know it’s not true.

New regulations, compliance requirements, connected external systems, business mergers, process changes or simply expanding requirements introduce a need for modifying how you capture, store and analyse your data. In fact the most significant challenge of any data warehouse is to capture change while maintaining continuity with historical data so that reporting is consistent and accurate. Sounds impossible? Not really.

Making changes using traditional methods will nearly always require a review of data models, impact assessments and even complete rewrites when new requirements are implemented – it’s all very slow, tedious and sometimes painful. And when you are done, the model changes must be tested for assurance that nothing has been broken, before moving from staging to live analytics. Weeks turn into months with all of this, and it involves a huge amount of time and effort.

A better approach

It’s complicated keeping up with today’s pace, which is why some clever people came up with the Data Vault discipline. It’s a method of separating core data structure from the data content. The result is you can quickly and accurately adapt to change without suffering lag times and the inaccuracies of traditional data warehouse methodologies. You build the structure ONCE and then hang your data off of it in ‘satellites’.

Need to change? No problem, just add additional structure without breaking what was already there. If you need to include more data, you simply hang more off of the core structure (‘hubs’ and ‘links’ ).

Data Vault is a methodology, not a technology. It is a way of thinking about data, rather than a shiny new trend. It involves separating the structure of your data from the changes. It simplifies and stabilises your data model, so that it becomes a fixed entity into which your raw data is locked and never changes. It is a vault of truth of your business – warts and all.