Monday 22 September 2008

Databases no longer shared resources?

One interesting trend that I've noticed in many of the organizations that I've been into is that increasingly databases are being built to serve single applications. The early visions of databases shared amongst multiple applications is no longer the first choice. To a certain extent this has always been the case for certain operational systems, but now the reach of single application databases has grown. You'll even find data replicated across multiple multi-terabyte data warehouses to support different business intelligence solutions.

One reason for this trend could be that disk is seen as a cheap resource and there is no longer the cost constraint for minimizing the number of copies of data. Obviously this is not the full story as there is a cost to keep these different versions of data in sync, although this may not always be necessary. Given the synchronization cost, it seems the real driver for why single application databases are becoming more and more popular is the need for business agility. I think businesses can no longer be held to ransom by the database when they perceive the need to update an application to improve business performance.

In a shared database environment, making a change to an application that requires changes to the underlying database can be an extremely costly and time-consuming business. Just take a look at the excellent book Refactoring Databases: Evolutionary Database Design if you want to better understand the various impacts of making changes at the data layer. By moving to single application databases, all these complexities can be removed and business can update their business applications at the rate that the market demands rather than at the pace the database allows.

This move also fits in with the agile development practices which have been coming to the fore in the development community in recent years. I'm sure that time and again business guys have asked for updates to applications and the developers have said "sure" only to be blocked by what was going on in the database layer. That is not to say that changes are not possible in a traditional shared database environment, as Refactoring Databases shows, but without introducing single application databases you are not going to be able to run as fast as the business guys want.

Thursday 18 September 2008

Big Data

Take a look at the special edition of Nature on Big Data. Although this is focused on the continued explosion of scientific data and the problems with processing all this data, it's a fair commentary on what is happening across all industries. There is too much data for current tools to efficiently process. More often than not, organizations are finding themselves limited by their ability to process and understand all the data that they hold. Google, when is it not, is a stellar example of the payback you can get if you can tame the data deluge. If only ordinary organizations could do half as good a job of getting to grips with their data. Fortunately, that means work for the rest of us.


Another article worth read is The Claremont Report on Database Research. This reports on a workshop held by some eminent database bigwigs and their conclusions on the challenges facing the database industry in this new climate. The report picks up on the "Breadth of excitement about Big Data". Again Google has probably been key in stirring up this excitement. They have certainly opened peoples eyes to the value of getting to grips with this explosion of information. This is probably why there are so many new venture-backed database companies starting out at the moment. It's all about the data. What a great time to be a database practitioner.