In-memory databases and mobile business intelligence

I was looking at some results of a benchmark test MicroStrategy did on its product. The benchmark was about supporting large numbers of users. MicroStrategy is pushing into the mobile business intelligence business and it has identified (correctly in my opinion) user scalability as one of the key challenges in this area.

Anyway, the benchmark tests to see how MicroStrategy would do with up to 100K users or so. Impressively, (if not surprisingly, considering they ran the tests themselves) the product did well. They ran the test both in Linux and Windows.

There were two things I found intriguing about the test: The hardware configuration of the application server, (144 GB of RAM), and the fact that they allowed the system four hours to load the data into cache, which is basically an in-memory cube.

In other words the Oracle database that was holding the data probably wasn't working very hard during the actual test. It worked during the four hours of loading, and was probably done for the day. Most of the heavy lifting was going on in MicroStrategy's in-memory database solution.

I think this is what we should expect more and more in the future: large scale database solutions of this type will make use of the falling RAM prices to squeeze better performance out of their solutions. Mobile applications with large numbers of users will need in-memory technology.

Flash and business intelligence

I have heard quite a bit recently about the demise of Adobe Flash. In fact I recently had a discussion with some programmer who were decidedly of the opinion that the future belongs to HTML5.

Apple does not like Flash, and has managed to keep it off the iPhone and iPad. Steve Jobs also claims that Flash is the most common cause for crashes on Macs.

Microsoft is not as strongly opposed to Flash as Apple is, but Silverlight is certainly intended to compete with it, and Microsoft has also thrown in support for HTML5, as as Google.

But I certainly don't see any move by business intelligence companies to move away from Flash in their reporting tools. In fact IBM Cognos released Cognos 10 fairly recently with more support for Flash than in previous versions. I also notice that Information Builders has actually more towards Flash in its Active Reports, away from the original DHTML version. And MicroStrategy doesn't seem likely to move away from its popular Flash dashboards, even if its current interest in mobile business intelligence will means more support for Apple platform.

I think people tend to underestimate the role that momentum plays in the IT business. I can understand the arguments that the markets are moving away from Flash, but I wouldn't expect it to disappear overnight.

Do companies really want transparency and collaboration?

Just finished our annual conference on planning tools. As usual, I talked to a lot of customers and vendors, and one theme came up several times: Some of the great features that vendors have on the roadmaps for their tools and that analyst list high in their discussions of trends for the new year may not be customer requirements at all.

I'm not talking about platform issues like mobile BI. This is about actual features of the tool. In particular, features that shed light on the thought processes that lead to the final planning numbers. Here are some specific remarks:

• Fine grained comments, for example comments on individual numbers, are rarely used.
• Detailed collaboration is often better done off line. Is this a tool issue or a cultural issue?
• Comments from large user groups tend to be too low in quality to be of much use.
• Middle management sometimes opposes processes that encourage transparency, even if they save time and effort.
• In some cases middle managers feel that storing data locally until it has been verified is better for them. So (to get back to a platform issue) Web-based data collection is not as popular as you might expect, a trend I have mentioned before.

Of course these are not universal truths. But it is interesting to hear how often they are mentioned.

Business intelligence and the iPad

I have been observing the wave of BI vendors coming out with applications for the iPad with some interest. QlikTech and MicroStrategy, who are always quick to move with the latest market trends, were among the first. IBM Cognos waited for their new release but also have a client as well. And with SAP having just spent billions on Sybase for its mobile technology, it’s not surprising to see SAP has also jumped into the fray with a new iPad client for Crystal Reports and the BusinessObjects Explorer – though not, as far as I know, using Sybase technology.

But the point here is not to list all the iPad BI clients available – I am sure I have missed a few. What interest me more is the question of how these interfaces are designed. The key to making a mobile reporting interface work is to squeeze interface onto a very small screen. There are lots of tricks for doing this, most of them involving navigations tricks with simple gestures to make traversing the interface feel natural.

But the iPad really does not have that small a screen. In fact, with a 9.7 inch diagonal, it is about the same size as an Asus Eee screen. It does not weigh a lot less either. But despite Eee’s success, business intelligence companies were not falling all over themselves to create a new interface of the Eee.

Furthermore, in my opinion, the interface that Apple gave the iPad is not quite appropriate. The first time I saw it I just thought, “Oh cool, it’s like the biggest iPhone ever”, and so did many other people, I suspect. But frankly, the iPhone interface does not expand all that well. In short, the iPhone interface is a good interface because it is visually attractive and it handles the severe size constraint it is subject to quite well. To make this possible, the users’ choices are intentionally limited, but users do not mind since the device is so small. But the IPad does not really have this constraint, and so the restrictions are more of an inconvenience. Furthermore, the bigger screen gives the user way of interacting with the software, so looking simple becomes more and more cryptic.

So iPad interfaces tend to discard the typical desktop standards that have been accepted over the decades (for better or worse) and replace them with a concept that is finely tuned for a completely different format. But most users – especially users of business tools as opposed to toys – are more interested in the content of the application than in fancy interaction techniques. Business intelligence vendors may be better off just making sure their Web applications work well on Safari running on the IPad than spending time developing iPad specific applications.

Panopticon

I had another look at Panopticon today. I had seen it before, but they have a new version out. Panopticon is probably most noticeable at first glance because of its spectacular data displays. Their marketing material usually features very busy charts of two types – tree maps (heat maps with a drill down) and what Panopticon calls horizon graphs, which only they offer, as far as I know. Horizon graphs are too busy for most applications and only seem to me to be useful for trying to get an overview of a large quantity of data. It sees itself as a competitor of Tableau and Spotfire.

Similarly, Panopticon’s data layer is specialized in supporting large quantities of multivariate data, often in real time, or near real time. Typical applications include real time monitoring of telecom networks and financial trades. Product is notable for its ability to access disparate data sources including column-based data sources and enterprise buses. This makes a bit more like middleware than most BI. The company is particularly active in the financial services industry, where about 80% of its customers are found. Panopticon offers semantics in what it calls a Stream Cube, a multidimensional model with data access and caching functions.

Panopticon’s newest release includes an easier way for end users to design dashboard that are embedded in third party systems. This product, which the company refers to as the Rapid Development Kit(RDK), is a reminder that the product is often used in very technical applications, such as as an embedded dashboard in a custom solution. The RDK is intended to allow business users to deliver content to such technical environments.

Cognos 10

I had look at Cognos 10 today when IBM’s Andrew Popp and Andrew MacNeill dropped by in Würzburg on a whistle-stop tour of Europe. On the whole I was pretty positively impressed.

Cognos says it is moving back towards the business user. That was the main message I picked up from the meeting. I have been noticing this interesting trend with many vendors. About ten years ago there was a big swing away from the business user, and vendors started focusing more and more on the enterprise. The main reason for doing this was that they were chasing the bigger deals.

But vendors often forgot about the end user in their rush to establish themselves as enterprise standards. In fact, the BI Survey 9 found that buying choosing BI software for a project because it is the enterprise standard leads to the worst project results. It is refreshing to see vendors making a real effort to address this issue.

Cognos is focused on making the product easier to get started with. They have identified the issue of jumping between multiple studios as the key issue for end users. Removing the “studio hop” is a big part of the new product. Actually this never seemed to me to be such a big deal. However, I have often seen users confused about it in the field – at least during sales presentations. I wonder if it is also an issue for users with a day’s training.

Be that as it may, the new version of Cognos provides a new way of accessing the system. Now the user starts in a Workspace that provides easy self service features and can add to that by clicking a button automatically attached to objects that brings the user to the appropriate authoring tool. Cognos is also moving away from the recursive grid concept that is the basis of Report Studio. This is probably a good idea as well, because you need quite a bit of practice to get a complicated layout right.

Cognos also introduced disconnected analytics with Report Studio’ ability to create prepackaged reporting applications. These single file packets include a slice of data and one or more interactive formatted reports. They are called Active Reports and are a lot like Information Builder’s Active Reports. They have also developed a set of query optimizations called the Dynamic Query Mode.

Another interesting side remark that IBM made was that they were considering using the name Dashboard for the Workspace. In the end they decided against it, and I think it was a good decision, although it is probably easier to get attention using the totally overhyped term dashboard which seems to be used to mean just about anything.

Visual Mining

I had a look at Visual Mining's Performance Dashboards a few days ago. Visual Mining has a product call NetCharts which comes in different editions for different customer groups. The core version is a set of developer tools that allow Java developers to create HTML 5 or Flash charts. NetCharts Server provides additional features for delivering the charts to end users. The NetCharts Performance Dashboards add an additional layer of user interfaces and data access tools to allow users with relatively few technical skills to create their own dashboards.

Strictly speaking dashboards do not include tabular data. Well anyway, that was the original idea, even if most vendors do not stick to it. and most of the presentation I got of the Performance Dashboards actually revolved around the features of the table object that is provided. The tables offer of sorting and filtering as well as conditional formatting.

Each table is made up of a single hierarchy and one or more columns, either taken directly from the database or calculated on the fly as a variances, including simple time intelligence features such as previous year. The hierarchy has good presentation features - it shows children as indented elements, the parent can be above or below the children, level based formatting is available and so on. Furthermore the end user can add or remove levels and change their order on the fly.

The product also has drill-down feature in the tables and the charts, so users can see the transactions the underlie the aggregate data. Navigating in the finished dashboard is relatively simple, and more emphasis is put on creating the dashboards. The end user can be given the rights to modify the charts freely, even changing chart types.

Chris Webb on the BI Survey

Chris Webb made a few comments on the results for SSAS in The BI Survey 9. I think the most interesting point he brings up is the poverty of front-end tool use, particularly third party tools, for a low-priced database with an open interface. You would think third party vendors would make up a larger market share, but that doesn't seem to be the case.

After all one would expect third party vendors to flock to the tool, especially since it is already available at so many sites. (The latter claim is my speculative inference based on the fact that SSAS appears so seldom in competitive evaluations.) Of course there are plenty of third party vendors on SSAS, but they seem to make a relatively small dent in the market. We do see quite a few third party planning tools using Microsoft as a platform though.

I don't think that this is limited to SSAS however. Use of third party front-ends for other multidimensional databases such as TM1 and Essbase also seems relatively seldom.

Web Business Intelligence is not a key issue

One of the questions we ask in the BI Survey is which criteria people use to choose BI products. Buyers in The BI Survey don't seem to find Web support very important. Specifically relatively few mention it as a reason for selecting the product they elected. In fact, less than one in ten do.

Of course that does not mean that Web based BI tools are no longer used. It just means that as a product differentiator, a Web platform really is not of any great interest to buyers.

This is interesting because Web architecture is a priority areas for business intelligence vendor R&D, and marketing departments often emphasize the Web capabilities of new releases.

John Schwarz was never going to be CEO of SAP

If you are interested in business intelligence or follow the doings of SAP, you probably already know that the former BOBJ CEO John Schwarz has left the company. The temptation is to delve into the details of all the post-takeover events at SAP and try to puzzle out the proximate cause of this move. Then you could do a little Kremlinology, trying to guess what is going on in the minds of various execs.

But I prefer to stick to a few general observations.

• When business intelligence companies get taken over, the senior management almost always leaves after a few years.
• There has been some discussion that Schwarz was shooting for the CEO job, but that hardly seems likely to me. Remember SAP had about ten times the revenues of BOBJ. Also, as SAP has learned the hard way, selling business intelligence software is a very different business than selling ERP.
• The main thing that keeps an executive of the acquired company on board is a clause in the contract. As soon as the clause expires, the executive jumps ship.

That's about it, really.

OK, like Oscar Wilde, I can resist anything but temptation. Look at the press release. Schwarz says "I strongly believe that SAP BusinessObjects will play a vital role in SAP's future success" and that he is "confident about the future direction of the company". It's hardly a ringing endorsement of SAP's ERP business, is it?

Mixing Excel versions

The last partner presentation I saw at the the Microstrategy was given by Microsoft's Donald Farmer on the subject of PowerPivot. The presentation didn’t seem terribly relevant to Microstrategy, but it was a solid gee-whiz run-through of PowerPivot's features accompanied by a distracting post-modern slide deck featuring grainy black-and-white photos and classic ads. Donald also mentioned one intriguing tidbit that surprisingly no one really seemed to pick up on.

Afterwards I had dinner with Donald and Neil Raden in an unobjectionable casino restaurant. Neil is a devoted dad and regaled us the entire evening with amusing with anecdotes about his family. And as usual on such occasions we spent a lot of time gossiping and reminiscing. It was good fun.

The tidbit was that Excel 2010 will be able to run on the same machine as Excel 2007. Am I the only one who missed this? I’ve been discussing Microsoft’s new Office version and how it fits Microsoft’s BI strategy with just about anyone who will listen for months and I don’t think I’ve heard anyone talk about this.

Microsoft’s BI strategy has always been platform oriented. By this I mean that Microsoft is really interested in pushing its platform and the BI tools have always been an appendix to this goal. When it comes to Excel, Microsoft’s main challenge is to get users to upgrade and BI features are offered as a gambit to drive upgrades.

The problem here is that Microsoft’s argument doesn’t quite add up. Customers are supposed to use Microsoft’s BI tools because they have them anyway, but they are supposed to upgrade to the newest Office version because then they get the BI tools. I have heard this called synergy, but it isn’t – it is circular logic.

Most large companies will reluctant to take advantage of the BI features built into Office 2010 if it requires upgrading the Excel version. Microsoft often competes on price in the BI space, but there are simply too few BI users as a percentage of total Office users to justify a costly Office upgrade to get free BI features.

But all this assumes the company will only run one version of Excel. But if I understood Donald Farmer correctly, you can now run two versions of Excel on the same machine, meaning Excel 2010 can be positioned as a “normal” BI tool that runs alongside Excel 2007. If Microsoft follows this strategy in the field it could bring a lot of clarity into its BI strategy.

A better word for marketing is demarketization

The word market means different things to different people. Adam Smith's idea was that markets are about competition, and competition auomatically keeps tabs on vendors and consumers. So actually markets are about competition.

However, if you are in the marketing business, market is just another word for opportunity, and opportunities tend to arise in places where there is little or no competition.

Check out famed internet marketeer Seth Godin's blog entry on Groucho Marx.

Seth states "It's extremely difficult to repair the market." What is he actually saying?

He's saying get out of situations where you have to compete directly. So for him, a market is a place where there is no competition, and when competition comes, it's time to get out.

That is the point to vendor lock-in. When you lock in your customer, you lock out the competition. And that is why software vendors love it so much. It demarketizes the customer relationship.

Lock-in

Switching costs are the costs incurred from changing from one brand to another. Tangible switching costs are things like laying a new telephone line. Intangible costs are things like the cost of switching to a new telephone number.

Vendor lock-in is when a customer prefers a different product than the one he uses, but not enough to pay the switching costs. Many vendors of high tech products work to increase their own lock-in and decrease the lock-in of their competitors. In some cases, for example computer hardware, switching costs tend to decrease with time.

Brand specific training for users of computer software is a lock-in that tends to increase with time. Users do not like changing from software they are familiar with, and the longer they use the software, and more proficient they become, the less willing they are to change. With enterprise software, the cost and risk of switching from one product to another is a powerful force maintaining the status quo.

Lock-in is the key issue in software marketing. The usual marketing ideas that apply to selling high margin consumer goods like fresh fish simply do not apply. In fact the normal rules our supply and demand are so skewed in the IT market that they are almost impossible to recognize.

Competing commodity software packages immediately become freeware. The reason is that the high initial cost of creating software combined with the low marginal costs per customer encourages vendors to increase market share by price cutting. If there is no brake, the market ends up spiralling down to freeware. If a new standard is introduced into the market, and it succeeds in becoming a real standard, your package may become a commodity and you may find yourself having to give it away.

But by locking customers into a solution, software vendors can reap sizeable profits even if their products are more or less the same as the competition. The reason is that the switching cost, and not the license fees for the software itself –- which tend to be a small part of the total cost of ownership -– is what is keeping the customer paying.

Alea developers move to Jedox

Interesting story behind Jedox hiring the remaining Alea developers from Prague. As far as I know there are three left, but once there were thirty. The MIS hotheadedly fired most of them after the disastrous Alea 4.0 project back in 2001 (or maybe 2002, I have forgotten).

Alea was a clone of TM1 developed by MIS GmbH from scratch in Prague back in the mid 90s. After squandering its IPO money, MIS ended up as part of Infor. Comshare was there too, and the two product lines started to compete internally and merge. In my opinion MIS had better technology than Comshare, but I am not a neutral observer.

One upshot of all this is Alea was renamed Infor OLAP and development was moved to Ann Arbor. My guess is that this is one reason why Matthias Krämer went to Jedox, though I haven't asked him. He won't need to take much time to get to know the products.

The MIS front-end tools are still being developed in Prague. The Web application development tool was originally created by Intellicube, which Christian Raue (founder of Jedox) sold to MIS. The development team was partly made up of redundant Alea developers. I was later product manager for that product, now known as Application Studio. Unsurprisingly, the Jedox front-end tools are very similar to Application Studio.

PowerPivot as a database

A couple of people disagreed with my somewhat pessimistic analysis of PowerPivot's (Gemini) prospects. So I decided to jot down a few more details on how the product works as a database. I still stand by my original comments, but it is interesting to look at what the advantages of the product are as well.

PowerPivot is an interesting approach to dealing with the problem of data storage in Excel. Microsoft Excel users often store data in Excel as if it were a database. But Excel is not a good database for several reasons:

• It fails to separate formats and raw data in an orderly way.
• It does not provide an easy way to share data among users.
• It has no straightforward means of applying calculations to more than a single cell.
  

As a result, there are lots of issues that come up in projects making heavy use of Excel. In particular performance, accuracy, reliability and versioning are compromised. Gemini deals with some of these issues.

Gemini addresses the issue of data storage in Excel in the most simple and obvious way. It provides a new method of storing data in an Excel file that is separate from the formatting and calculations in Excel. It does not, by default. store the data outside the Excel file. But although PowerPivot data is stored in the same file as Excel it is stored separately from the rest of the Excel data.

To understand exactly what is going on we need to take a step back and look at how the latest Office products store data. Basically the new Office format is a zip file which contains several XML files. One Excel file contains the Excel spreadsheet Excel itself. Other objects such as charts are stored separately in the same zip file as the spreadsheet file. (Actually this is a simplification. Each sheet is stored in a separate file, and there are other complications involving shared content.) The zip file has the format of a normal zip file even though it has the extension xlsx and not the .zip extension. But don't take my word for it -- you can see the exact internal format of Excel for yourself by simply opening any Excel 2007 file with a data compression program like WinRAR.

In Office 2010 it will be possible to store additional data in another separate file in that same zip file and to access that data using the database interfaces provided by Excel. These database interfaces are the pivot table and the database spreadsheet formulas. From the outside the file won't look any different -- except maybe a little bigger. Or maybe a lot bigger. The much ballyhooed column compression only works in memory.

Office 2010 also addresses the issue of sharing data between users. I guess this option is really intended for Word more than for Excel, but it works across te entire product line. In any event an Excel sheet stored on a server will be available to multiple concurrent users with a local installation of Excel or in Sharepoint. The idea is that you create an Excel file and put it on a server where all your colleagues can access it, and the product allows you to edit the file simulataneously, even updating changes made by others. This sounds like a rough and ready solution to the second problem listed above, although it does not provide any security.

The final issue that seems important to me is applying a calculations to a large number of elements. And here, too, PowerPivot does offer some respite. Calculated columns can be added to the table. This is a Good Thing because it save the user from the difficult and error prone procedure of copying the formula down the entire column. It is not a replacement for multidimensionality, because the "database explosion" effect means that you will need to define a lot of columns to imitate multidimensional aggregation. But plenty of Excel-based solutions get by without this so I guess it won't always be an issue.

The upshot is that the product really does offer departmental users some relief from Excel chaos by treating the worst symptoms. And it does so without being disruptive, a key design consideration in viewof the size of the Excel user base. So I can understand some people's enthusiasm.

The Limitations of Gemini

Information is still a bit sketchy on Gemini,but it is already fairly clear where Microsoft is going with it. The clarity comes partly from Microsoft's public statements and partly from a sober analysis of the constraints Microsoft's BI program is subject to.

Microsoft will be offering three ways to view Gemini data. The first way is the grid they have been presenting in there demos. Take a close look at that grid. They always start it in Excel, subliminally -- or not so subliminally -- suggesting that the data is in Excel, but that is not the case at all. In fact it is just a separate grid that floats above the Excel window. The Gemini grid has nothing to do with Excel and is pretty basic as a reporting tool.

The second and third ways of viewing Gemini data are not new at all. In fact they are just the same tool that are now used for viewing Analysis Services data -- Reporting Services and the Excel Pivot Tables. Although both are adequate tools for some purposes, they tend not to score particularly well in customer satisfaction surveys. Perhaps more important they do not offer much in the way of features that are specific to the data model of Gemini.

The marketing role of the front end is very different in QlikTech. QlikTech's main marketing argument is "associative analysis" which it claims is a revolution in OLAP. Whether or not you agree with that claim, there is no doubt that the product delivers a specific set of features to make good on it. Specifically, the selection behavior of the list objects with their characteristic green shading, automatic hiding of unassociated elements in other objects and so on are specialties of the product that visualize the concept of associative analysis.

Arguably the real reason for this is that the front end is out of the hands of the creators of Gemini. After all Microsoft's BI front end is Excel. Excel has perhaps a half a billion users, and most of them would just be confused by any radical change in the interface.

The data import tools for end users look pretty simple. This is an issue that often gets forgotten in BI sales presentations, because data import is not a topic that end users like to get involved with. It is also a key weakness in QlikView, which offers a few automatic options but quickly switches to a proprietary scripting language. So far Gemini looks even weaker. But strengthening them would be to compete with the tools SQL Server already offers.

So with Gemini Microsoft isn't offering anything new in the front end or data import. What this means is that Microsoft is betting on the in-memory feature itself to make Gemini attractive to the end user. Microsoft has very little in the way of analogous features for the end users. Instead it has emphasized the issue of scaleablility, always a popular marketing claim but not the key issue in a self service environment.

So Gemini will have to sell itself on the technical merits of in-memory data management.

IBM acquires SPSS

Still digesting Cognos, IBM is now investing in high end analytics -- Reuters says it splashed out $1.2 bn for the data mining and predictive analytics specialist SPSS. That is a remarkable sum, considering that it’s a 40% premium on the value of the stock and about 25 times the company's annual profit. The takeover is great news for veteran CEO Jack Noonan who joined the company in 1992 and is now approaching retirement age. This is especially true considering that the success of the open source R language is squeezing proprietary data mining vendors.

This isn't IBM's first foray into data mining. In 2000 BARC published a study showing that IBM's Intelligent Miner was one of the best products on the market from a functional point of view, right up there with SAS Enterprise Mining. Ultimately however, IBM never had a lot of success with Intelligent Miner. IBM has been moving out of the market step by step and now only offers a few specialized solutions. And when IBM announced the SPSS deal Intelligent Miner was only mentioned in passing as an execution engine for SPSS solutions.

Maybe the Cognos acquisition changed things for IBM. SPSS is a Cognos OEM-Partner providing advanced analysis features to Cognos. Cognos is more focused on reporting and basic analysis so the SPSS products show no real overlap. On the other hand Cognos has also made its forays into the data mining business, including its purchase of Forethought, part of a once fashionable but now forgotten desktop data mining strategy.

Data mining and advanced analytics companies have carved out an interesting niche for themselves. Bringing data mining to the masses in the form of simplified data mining has not been a very successful idea, probably because the concepts are just to complex for the average business user. And BI companies have worked for decades to replace custom modeling with pre-built applications, but have had little success. But both SPSS and SAS successfully sell applications such as analytic CRM and vertical solutions for the financial industry.

SAS is strong on data management and advanced analysis. IBM now has assets strengths in both areas. SAS is still stronger in specialized analytic applications, but SPSS has been successful in this area as well. Furthermore IBM is likely to emphasize pre-built applications – as they have for Cognos.

SAP Business Objects is another company that will have to adjust to this acquisition. It had a very close relationship with SPSS, because SAP -- like IBM Cognos -- does not have much to offer in this market segment. SAP will have to look around for a new partner, perhaps KXEN or another smaller vendor.

Running BI on Chrome OS

The Google Chrome operating system would basically be a browser and some drivers to connect to the underlying hardware. It would be the realization of Marc Andreessen's dream in the 90s reducing Windows to a device driver.

It only make sense for a client. The idea would be to run programs with a Web client that don't require any local installation. That way it doesn't matter what the underlying system actually is.

Google has been pushing HTML 5 pretty hard, and Chrome OS is a good reason to. For example, HTML 5 has a tag to show vidoes, which means you would need a plugin. That means shifting the burden to the browser itself. The more the browser does the more viable the Chrome OS idea is.

It's notable that a lot of BI tools would work on Chrome. For example, a lot of the sprawling Information Builders would work fine, as would MicroStrategy -- they run on DHTML. So would Business Objects WebI, which is Java based. I'm guessing Java will work. Cognos would have some issues because some of the studios require ActiveX.

And so on. I'm not attempting to do a complete survey here, just point out that BI vendors have already done quite a bit for Web support, and ultimately, a Web strategy is a Windows-free strategy.

Cognos and the midmarket

At the IBM event in Berlin last week (the "IOD") Cognos spent a good deal of time and energy briefing analysts on its midmarket strategy. To make a long story short, Cognos is planning to bring out a new product to address the midmarket.

One take is that it is at least partly a step towards alignment with the policies of IBM's service organization. IBM Services has had a mid-market offering for years now. And my impression is that Cognos is focusing a lot of energy on aligning itself to towards IBM Services as a sales channel. The alignment towards IBM Services also shows in Cognos's increased focus on prepackaged business content.

However the midmarket idea is in line with what appears to be an internal shift at Cognos back towards the end user. Cognos 8 shifted the product line towards the needs of the enterprise. The shift has been going on for some time, probably as a reaction to customers who prefer to stick with Cognos 7. "The pendulum is swinging back" was the message in Berlin, and recent versions of Cognos 8 Bi have revived the PowerPlay product family.

Targeting the midmarket is not exactly the same thing as targeting the business user. Targeting business users can also mean creating departmental solutions in large companies. However there certainly is an overlap. The new product is intended to be easy to install and administer and to cover planning reporting and analysis in one package. It will have Web access and an Excel front-end. I'm guessing its also going to be relatively cheap.

Cognos is adamant that it is building a new purpose built solution, not simply bundling existing tools. But it also says the new product is "based on existing proven technology like IBM Cognos TM1 and IBM Cognos 8". It will be interesting to find out exactly what that means.

Issue they should be looking at is

• Compatibility with existing products. Customers using TM1 might want to upgrade to this product. If they can't they might start worrying about the future of the existing product.
• Organizational isues. Will IBM Cognos have a sales team willing to make the effort to sell this product?
• Easy installation. Self service is a key goal in the project, and installation needs to be automatic.
• Automatic metadata exchange. TM1 models need to be automatically visible in the Cognos 8 bits of the offereing, even at the cost of flexibility.
  

Twitter's limited API

When I first looked at the Twitter API I was surprised to see how simple it is to use. The next thing I noticed is how little it can do. In particular, as a business intelligence guy I was struck by the lack of sophisticated query methods.

The API does not provide a way to do simple things like get a sorted list of the people you follow, which is probably ok. It's easy enough to do your own sort in your client. But what if you want to sort the people you follow by say the number of people THEY follow? To do that, you need to make an API call for each person you follow to find who he follows, and then do the sort in the client. This is pretty much hopeless for analysis purposes, especially considering the calls per hour limitations the API has and the fact that there are lots of Tweeters following 20,000+ people.

Twitter doesn't deliver some API features because it can't afford to. The "firehose" is the jargon for a real time feed of all tweets from everyone worldwide. Twitter has been promising this for some time but keeps delaying it. One reason may be that they are afraid there would be too many takers. I suspect the reason Google can offer so much storage to gmail users is that no one uses it -- like a bank hoping there won't be a run. Using an add on to store lots of data there is possible, but I think Google frowns on those shenanigans and will even block your account if you up load too fast... So as long as Twitter keeps growing at its current breakneck speed, there are some things the API won't offer because offering it could break their overstrained servers.

Another reason that Twitter might not offer some functions to their API is that they want to sell analyses as an added value service. Twitter still doesn't have a business model (except "Microsoft or Google") but analytics is an obvious option. In fact I think that providing analytics is the only real prospect that Twitter has.

In particular, Twitter has a mechanism for providing that favorite business intelligence feature -- the real time alert. For example, sighting a yellow headed blackbird in Connecticut is unusual, but the news needs to be immediate for an ornithologist to profit from it.

Twitter is also about people, not just about information. It provides relatively detailed information about who knows who. In fact I see this as the keep features of the service, so I am surprised that so little effort is invested in suppressing spam.

But whatever the specific application, the ability to analyze Twitter's database is too valuable to give away, so I suspect Twitter will not do too much to make their API better for analytics in the near future.