Why short projects are part of agile business intelligence

One of the key ideas in agility is the importance of delivering real, testable results without delay. In fact, the Agile Manifesto recommends delivering working software frequently, from a couple of weeks to a couple of months.

Delivery working software with two months may sound a bit extreme, but there is good evidence that short projects are more successful than long projects. In fact, our research in the BI Survey shows that the application should be rolled out to the users less than six months after the product has been selected. We have found the same result year after year in the ten year history of the Survey. Amazingly, project benefits start to fall as early as a month after the product is selected, and continue to fall thereafter. And of the many project parameters we study, none shows as clear an effect on project success as project length.

These results from the BI Survey provide clear empirical support for the idea of using agile methods in business intelligence projects. The results have remained also consistent since we started the Survey ten years ago, long before the idea of agile development or agile business intelligence became popular.

But why do short projects work so much better? Our research shows that the main problems that arise in longer projects are organizational, not technical. Needs change over the course of time, and end users lose interest in the project. Disagreements over the project goals arise. Lack of interest and disappointed end users are a major issue in business intelligence.

And needs certainly do change quickly in business intelligence. For example, another study we carried out shows that three quarters of European companies modify their planning processes one or more times a year. In an environment like this, a project that takes a year to implement is quite likely to be obsolete before it is finished. Even a six month wait can push potential users to look around for a more agile solution.

The problem this creates is that not all business intelligence projects can be carried out within a few months. This is especially true when major data management issues need to be addressed. The agile solution to this is to find way of splitting large projects into smaller ones. The usual argument against this approach is that it creates the risk of reducing efficiency in the long term. But the agile methodology is to measure success in terms of working software delivered in the short term, instead of attempting to meet nebulous long-term goals.

Business Intelligence, the semantic Web and Alltop

My blog is including on the Alltop business intelligence page, and at the time of writing I display their badge on the blog.

According to the site 'The purpose of Alltop is to help you answer the question, “What’s happening?”' Alltop is a 1990's style Web registry maintained by human editors.

But Alltop's business intelligence page has several problems that make it less useful than it could be. The page has fallen victim to a semantic Web style gotcha. Like many other phrases, business intelligence means different things to different people. If you don't disambiguate somehow, a web registry based on a phrase may make no sense.

There are three distinct meanings of the phrase "business intelligence". The first is something about software for analyzing business data -- like my blog. The second is news about businesses, which is interesting but unrelated. These are some of those blogs:

MEED NEWS, EMARKETS.DE, B2B TRADE, DEALBOOK, ARUNDEL BUSINESS NEWS, FOREX TRADING INFO, THE FINANCE KID, ARBITRAGE MAGAZINE, SMALL BUSINESS SUPPORT

The third meaning is based on a completely different meaning of intelligence -- intelligence as in IQ, as opposed to intelligence as in information for analysis. In the sense business intelligence just means being smart about business, which could men just about anything.

So Alltop's business intelligence page contain sites that are not at all related to the #businessintelligence tag on Twitter. A lot of these seem to be advice about sales and entrepreneurship or general management consulting blogs. A few are just political blogs, or blogs about general market or marketing trends. They're fine in their way, I guess, just misplaced. Here's a list:

STATSPOTTING!, WSJ: THE NUMBERS GUY, MANAGE BY WALKING AROUND, BUILDING BUSINESS VALUE, CORPORATE MANAGEMENT STRATEGIES, KNOWLEDGE WORKS, HUSTLEKNOCKIN', THE THINK HERE BLOG, LEAD VIEWS, THE SOLOPRENEUR LIFE, SMEDIO, INTERCALL BLOG, GLOBAL INSTITUTE FOR INSPIRATION, SMALL BUSINESS SUPPORT, RED WING SOFTWARE BLOG, FRED67 B2B REFERRAL CLUB, RESULTS.COM BUSINESS GROWTH TIPS

I'm not criticizing any of these guys, just saying they seem to be improperly categorized.

Also Alltop is syndicating advertising material thinly disguised as blogs. Of course, maybe they're getting paid, what do I know? If not they should be. The following are BI vendors, which may or may not be problem:

RED WING SOFTWARE BLOG, LOGIXML, BIME - SAAS BUSINESS INTELLIGENCE (BI), BLOG.JINFONET.COM, MICROSTRATEGY 101, PANORAMA BUSINESS INTELLIGENCE

In addition, there are several aggregators -- Yahoo, and Beyenetwork twice. These guys can be seen as comptetitors, I guess.

In the end I think that the lack of careful stewardship reduces the usefulness of the site. The problem is that business intelligence is a vague term and needs a semantic Web to be useful. Manual editing in a web registry is a workaround, but it is not being used here to much effect.

Discovering hierarchies in columnar databases

I recently blogged about columnar and Wayne Eckerson asked me for a clearer explanation of what I mean by columnar databases "discovering hierarchies".

For example consider the approach of two well known products, IBM Cognos TM1, which is multidimensional, and QlikView, which is columnar.

My definition of a data model is a structure that is informed by an administrator, or set down in the master data. To me this is different to a structure derived from analyzing the transactions. In the following simple example, let's say I have two sales teams, one for dental hygiene products and one for soap.

If I were designing a data model in TM1, then I could create a hierarchy, which is a set of parent child relationships between the departments and the products they sell. If the soap people cross-sold some toothpaste, it would have no effect on the hierarchy, because it is predetermined by my idea of how my company is supposed to work.

If I were to import the same data in QlikView I could create a report that showed me the relationship between the sales teams and the products without defining the model. Once the data is imported, QlikView recognizes the relationships automatically.

When the soap guys cross-sell toothpaste, QlikView discovrs the new relationship, but the hierarchies stay the same in TM1, because that's how I defined the model. To me this is the key difference. On the one hand the structures are coming directly from the actuals, and on the other hand they reflect my predefined perception (or "model") of what is going on.

So columnar databases typically discover the relationships automatically, and multidimensional databases allows you to define the relationships as you want them. Another way to look at this is that the transactional data drives the master data structure in a colunmar database, but those structures are wired into the multidimensional model.

So which approach is better? It depends on the application.

Is the cloud really the future of business intelligence?

The idea of hosting software off-premises is not new. In fact, software as a service (SaaS) is just a rewording of an older term referring to companies providing the service, application service provider (ASP). ASPs were a very popular investment target during the dotcom era, but not much ever came of the business model. Like so many Internet-based startup ideas in this era, most of these companies disappeared.

In fact, the only major exception to this rule was Salesforce.com, which used to refer to itself as an ASP, before the SaaS term became popular. And indeed Salseforce.com is often taken as an example in discussions of the future of BI as a service, or cloud BI.

But why was the ASP wave a failure and why should cloud computing be different? Even the often quoted success of Salesforce.com is far removed from the company’s once loudly trumpeted goal of replacing all the ERP features of SAP and PeopleSoft. Furthermore, do the forces that could make SaaS more successful than ASP also apply to the BI market?

The main argument offered by vendors in favor of cloud BI is that it is less expensive than the more typical on-premises software. The upfront capital costs of hardware contribute to these savings. The other major factor is that the company is not required to operate separate servers for the BI application.

However, both of these savings result from the investment required to own and operate the hardware for BI software. Similar savings can also be realized with strictly in-house approaches such as server virtualization. Furthermore, the use of Web technology has already eliminated one of the key costs of BI initiatives, which is the cost of rolling out the front-ends to large numbers of users.

Another argument is that mobile devices are driving SaaS BI, but as HTML5 spreads, it is hard to see why mobile devices should require a cloud solution.

We a conducting a survey to find out more about BI in the cloud. If you take part you could win an iPad 2.

I'd also be interested to hear any comments on what I have talked about here.

Data modeling and agile BI

One of the advantage that some analytical tools such as QlikView Spotfire or Tableau claims to offer over the products they call "Traditional BI" is that they can be used without data modeling. According to this claim, data modeling is a major obstacle to agile business intelligence and not needed anyway.

Is it true that data modeling is dead? Has technology found a workaround?
The need for data modeling depends upon the application. Products that promise user friendly analysis without any data analysis are usually intended for a specific type of analysis that does not require any previously specified structure.

A good example of data does not require modeling that retailers gather about their customers. This data comes in big flat tables with many columns, and the whole point to the analysis is to find unexpected patterns in this unstructured data. In this case adding a model is adding assumptions that may actually hinder the analysis process.

However, some types of analyses only make sense with at least some modeling. Time intelligence is an example of a type of analysis that is supported by a data model. Also analyzing predefined internal structures such as cost accounts or complex sales channels is usually more convenient based on predefined structures. The alternative method of discovering the structures in the raw data may not be possible.

Planning is a common area of agile BI, and planning is rarely possible without predefined structures. It is no coincidence that the tools that promise analysis without data modeling do not offer planning features. Planning requires adding new data to an existing data set. In some cases, this includes adding new master data, for example when new products are being planned. Furthermore, there is often a good deal of custom business logic in a planning application that cannot be defined automatically. Most financial planning processes, and the analysis and simulation that goes along with them cannot be carried out on simple table.

In my view the new generation columnar databases are a welcome addition to agile BI. But I also think that their marketing is sometimes a little over the top when it comes to dismissing existing BI solution in this area.