Chris Webb's BI Blog: On the internet

Kalido Universal Information Director now generates Analysis Services cubes

Mon, 21 Jul 2008 16:04:00 GMT

Here's the press release from Kalido:
http://www.kalido.com/5a04ea36-c50e-4617-b25d-85ea56a22690/news-and-events-press-center-press-releases-detail.htm

I don't have any direct experience with Kalido's products (although I've heard good things) but I'd be interested to see the cubes it generates. I wonder to what extent it's possible to optimise automatically generated cubes?

Interview on the SQL Down Under Show

Thu, 19 Jun 2008 08:47:40 GMT

For your listening pleasure: the latest edition of the SQL Down Under Show is me talking to Greg Low about MDX and Analysis Services. You can download it here -

http://www.sqldownunder.com/PreviousShows/tabid/98/Default.aspx

BI Survey 8

Sat, 14 Jun 2008 06:21:22 GMT

Is it that time of year again? Yes, here's the link for the latest BI Survey:
http://www.eu-survey.com/BI_8/BI.asp?lan=1&lin=23

Here's the blurb:

We would very much welcome your participation in The BI Survey, conducted annually by Nigel Pendse. This is the largest independent survey of OLAP users worldwide. The Survey will obtain input from a large number of organizations to better understand their buying decisions, the implementation cycle and the business success achieved. Both business and technical users, as well as vendors and consultants, are welcome.

The BI Survey is strictly independent. While vendors assist by inviting users to participate in the Survey, the vendors do not sponsor the survey, nor influence the questionnaire design or survey results. You will be able to answer questions on your usage of a BI product from any vendor. Your data will only be used anonymously, and no personal details will be passed to vendors or other third parties.

As a participant, you will not only have the opportunity to ensure your experiences are included in the analyses, but you will also receive a summary of the results from the full survey. You will also have a chance of winning one of ten $50 Amazon vouchers.

Interview with me on Cristian Lefter's blog

Wed, 23 Apr 2008 15:45:56 GMT

While I was at PASS Europe last week Cristian Lefter videoed an interview with me on what I think is cool in AS2008, for the Romanian SQL Server user group. You can watch it on his blog here:

http://sqlserver.ro/blogs/cristians_blog/archive/2008/04/22/pass-europe-2008-interview-with-chris-webb.aspx

He also interviewed a load of other people, including Marco Russo.

Google App Engine

Tue, 08 Apr 2008 11:33:25 GMT

I see Google have announced their own web application platform:
http://www.techcrunch.com/2008/04/07/google-jumps-head-first-into-web-services-with-google-app-engine/#comments
which includes BigTable as part of it (remember this post?). There's speculation over whether Microsoft has something similar up its sleeve:
http://blogs.zdnet.com/microsoft/?p=1320This and the Panorama and Good Data stuff I blogged about over the last few weeks make me quite excited. What will the next generation of OLAP/BI tools be like? Surely it's a mistake to think of them as hosted versions of what we've got today. As I've said before, the attribute-based approach of databases like BigTable remind me of Analysis Services dimensions; wouldn't it be cool just to be able to grab data from a number of these stores and use them as dimensions and fact tables? Maybe through a front-end that was something like Lotus Improv (thanks to Andew Wiles for directing me to this in a recent conversation) on the web? But with an XMLA interface too? And since in the cloud hardware scale-out will presumably be just a matter of paying a bit more cash, you'd want an engine that could handle that transparently in the way I understand something like Teradata does? Ahh, if only I had a couple of million USD$s of VC funding to waste I'd hire Mosha and set him to work on this...

Good Data

Mon, 24 Mar 2008 07:47:51 GMT

Possibly a little early to blog as this startup seems to be a while away from RTM, but it looks cool and BI as a service is very topical...

Yesterday a guy called Roman Stanek linked to my blog, so naturally I checked out his blog and his company:
http://www.gooddata.com/

What they've got is a "a complete, on-demand business intelligence platform combining analytics, reporting, data warehousing and data integration". The workflow seems to be that you upload your data, then you have an online environment where you have OLAP and collaboration tools; similar, I guess, to what Panorama are working on with Google docs? I guess they're going to be making their money designing the initial data warehouse/ETL/OLAP design for each customer as well as subscription costs?

This raises some fundamental questions about BI as a service in my mind:

Will companies be willing to upload their most valuable and secret data to a third party on the web? If they're willing to use online data stores like Amazon S3 or SQL Server Data Services or other hosted solutions then I suppose so, but that's a big if - and these online data stores aren't necessarily positioned to hold the kind of ultra-sensitive data that you work with in a BI system.
However complete a platform you build, the old adage that all data ends up in Excel needs to be considered. Remember it's Excel I said here, not any old online spreadsheet. Will the guys in Finance abandon Excel to do spreadsheets on the net? I'm not sure. This is where Microsoft have a locked-in advantage, but it's not an insurmountable problem for BI service-providers. Why not expose an XMLA interface for your hosted OLAP and let people connect to it directly from Excel? It's possible for Analysis Services and SAP BW, and I'm sure companies like Simba will be only too happy to Excel-enable other OLAP tools.
Unless BI-as-a-service companies let customers or partners design and build their own solutions on their platform, will these companies be able to scale out to meet the demand of designing enough apps for potentially hundreds of customers? That's a lot of BI projects for any one consultancy to run. More importantly, will they be able to do an adequate job of designing each app? One of the arguments that's always used against outsourcing BI projects is that face-to-face contact between designer and business users is essential, and that's a view I tend to agree with.

Panorama and Google do BI!

Wed, 19 Mar 2008 22:32:39 GMT

Clever old Panorama... just when you thought that it was about time for them to roll over and die, having been jilted by Microsoft and SAP, they team up with Google to do BI. From their blog, here's the press release:
http://www.panorama.com/blog/?p=98
and more information is here:
http://google-pivot-tables.blogspot.com/
and there's a tutorial here:
http://www.panorama.com/google/pivot-table/tutorial/

This is part of the wider Google announcements on Google Gadgets:
http://googledocs.blogspot.com/2008/03/collaboration-goes-one-level-deeper.html
and the Google Visualization API:
http://google-code-updates.blogspot.com/2008/03/introducing-google-visualization-api.html
http://code.google.com/apis/visualization/documentation/gadgetgallery.html

I need some time to digest this, but my first impression is that this is BIG. The PowerApps stuff sounds very interesting indeed.

BI Survey 7 Findings

Fri, 08 Feb 2008 16:20:17 GMT

One of the benefits of being a blogger is all the free stuff that gets sent your way by people looking for some publicity. Unfortunately in my case the stuff I get isn't all that sexy, ie no free XBoxes, but it's still interesting - the books are always welcome, the free licences for AS client tools are useful too, and yesterday I got a freebie copy of the results of Nigel Pendse's "BI Survey 7" with an invitation to blog about the contents. Now I'm sure over the next few weeks you'll see the marketing machines of all the BI vendors crank into action, cherry-picking the findings so they can say things like "XSoft is the most reliable BI platform" and "YWare offers the best query performance" etc etc. But what does it actually have to say about Analysis Services?

Firstly, some characteristics of Analysis Services deployments. This year 34% of respondents were still using AS2K, with 66% now on AS2005 (with no-one using OLAP Services any more). This is almost a complete inversion of the usage figures from the previous survey, indicating most people migrated from AS2K to AS2005 in the last year. This is certainly what I've seen in my consultancy work, but I'm still surprised so many people have migrated given that the two platforms are so different. Another interesting point made is that while you'd expect AS to be the dominant OLAP tool when SQL Server is the data source, it's also the most common OLAP tool for Oracle, Sybase and DB2 sites and it comes in second (behind Microstrategy) for Teradata. Oracle I can understand, because I work with Oracle data sources as much as I work with SQL Server (though in the past I used to work more often with Oracle, in fact) but I've never yet worked with anyone using DB2 or Teradata and only one person using Sybase, which isn't in fact officially supported as a data source for AS. Maybe I don't get around enough. I bet this is uncomfortable reading for Oracle, especially, since the only Oracle BI product that features in the top ten BI tools used against an Oracle data source is Essbase and that's at #10.

One of the things that people often bring up about AS is that people only use it because it's bundled free with SQL Server. That's certainly a big part of why it's chosen, and the survey shows it's very common that when it's used it doesn't go through a formal evaluation process or that it's the only product evaluated. But that certainly doesn't explain all the Oracle, IBM and Sybase shops that use it and the survey also shows that when a formal, multi-product eval is conducted then AS wins 75% of the time - putting it ahead of all of its major competitors. Again, that's consistent with my experience: I was working with a customer recently where they'd done an eval comparing AS with Oracle's OLAP option and Essbase, and neither of the latter could handle the data volumes and dimension sizes that AS2005 could.

Regarding the problems faced by BI projects, it's not surprising that slow query performance is the most common across the board for all products. AS is slightly above average in that 20% of respondents complained about this (maybe they need to get a specialised consultant on board to do some tuning? Now, who could do a job like that?); TM1 does the best at only 6.6%, SAP BI does worst at 37.5%. One thing that AS does worse on than average is 'Security Weaknesses', which to be honest is a bit strange given that I've never found any holes in the product that would lead to values being shown to people who shouldn't see them. What I suspect is happening here is that although AS has a great set of options for security it's still way too difficult to configure, especially for complex scenarios (see here for my experiences, although I wonder whether other products could meet these requirements at all) and for large numbers of users. There's a real need for a better, more end-user friendly interface for managing roles, and perhaps it would be good to be able to use dynamic security right out of the box rather than have to implement it manually.

Lastly, there's a good section on the client tools used with AS. I cannot believe that as many as 6.6% of people are still using Data Analyzer as a client tool - what planet are they living on? The high percentage using pivot tables I can understand, and there's a very long tail of people using tools that have only got 0-4% of market share. The survey makes the interesting point that fewer AS users use any kind of Excel front-end than users of TM1, Essbase or SAP BI; maybe that'll change as more companies move to Office 2007.

Anyway, hopefully I won't get into trouble for divulging too many details but I can honestly say it's a fascinating read and at 420 pages long there's a lot that I haven't mentioned here, and it's more substantial than a lot of IT books. If you're a vendor of any kind I'm sure you'll be buying this anyway, but I'd also recommend it if you're a consultancy (lots of juicy facts to quote to prospective customers to trash the competition) or you're about to embark on a BI project in-house. Overall, AS comes out of it very well which is obviously good news for me and I guess the majority of people who read this blog. Of course Nigel has long been very positive about the MS BI stack, much more so than other analysts although Gartner has suddenly got very positive too. I don't know about anyone else, but sometimes I get a bit miffed by the way some BI journalists talk about Cognos and Business Objects as serious platforms but never mention Microsoft at all... hopefully that's starting to change.

SQL Down Under features Richard Tkachuk

Fri, 25 Jan 2008 22:43:26 GMT

The latest edition of the SQL Down Under podcast features Richard Tkachuk, currently of the SQLCat team and previously of the AS dev team. You can download it here:

http://www.sqldownunder.com/PreviousShows/tabid/98/Default.aspx

If you're rushed for time, the last twenty minutes are the most interesting with a discussion of some of the new features of AS2008.

SQL2008 coming Q3

Fri, 25 Jan 2008 22:39:32 GMT

Apparently SQL2008 will be shipping Q3 this year, according to Francois Ajenstat (and he should know):

http://blogs.technet.com/dataplatforminsider/archive/2008/01/25/microsoft-sql-server-2008-roadmap-clarification.aspx

FAST acquisition includes interesting BI extras

Mon, 14 Jan 2008 22:13:31 GMT

Interesting article by Seth Grimes in Intelligent Enterprise here:

http://www.intelligententerprise.com/blog/archives/2008/01/buying_fast_mic.html

He points out that Microsoft's recent acquisition of FAST includes some interesting BI-related products. Having a look on their website, I found this page:

http://www.fastsearch.com/thesolution.aspx?m=513

Here are some details:

FAST AIW

FAST AIW (Adaptive Information Warehouse) is an information management solution that integrates your structured, unstructured, and multi-media data to create a virtual intelligence library where any insight is a few clicks away. FAST AIW incorporates both quantitative and qualitative analytics through mining of your numeric and text data.

FAST Database Offloading Solution

The FAST Database Offloading Solution liberates eBusinesses from the artificial constraints of legacy structures by offloading data from the relational database to a search index. Now you can offer the same information, but in a more meaningful and intelligent context. The FAST Database Offloading Solution provides higher performance to eBusinesses at a dramatically lower TCO.

FAST Data Cleansing Solution

The FAST Data Cleansing Solution provides the ability to harvest meta-information from text and use linguistics to cleanse multiple structured data repositories into a clean master index. With the FAST Data Cleansing Solution structured data from multiple repositories can be merged to create a clean master index cost-effectively in a matter of weeks.

FAST Radar

FAST Radar is a personalized Business Intelligence solution that empowers decision makers to explore and view information that is most relevant to them in an efficient, graphically intelligent fashion. It puts the creative process back in the hands of the business user by providing a simple and effective approach to Business Intelligence exploration and monitoring, reducing process times from weeks to real-time and aggregating information from data sources that may have been previously unavailable.

I wonder if/how/when all this will get integrated in the MS BI stack?

Amazon SimpleDB... for BI?

Fri, 14 Dec 2007 23:36:50 GMT

I've just seen that Amazon has released a new web service database called SimpleDB:

http://www.amazon.com/b/ref=sc_fe_c_1_3435361_1?ie=UTF8&node=342335011&no=3435361&me=A36L942TSJ2AJA

Note that it's not a relational database, but has a very flexible attribute-based model; it also claims to need no tuning, data modelling or any of the boring stuff that us database people do for a living. While claims like that automatically make me somewhat sceptical, I also think about Google Bigtable and the new generation of COP databases and think, well maybe... Slap an MDX-like interface over this, make it accessible from Excel, and you've got something interesting and disruptive from a BI point of view.

IBM buys Cognos

Mon, 12 Nov 2007 16:01:17 GMT

As the pundits have been predicting, IBM is to buy Cognos:

http://news.yahoo.com/s/nm/20071112/bs_nm/cognos_ibm_dc_4

COP Databases

Tue, 30 Oct 2007 11:38:47 GMT

COP (Column Oriented Processing) databases seem to have been making something of a comeback recently. You may be wondering what a COP database is; the best short summary is from the OLAP Report's glossary:

This is a class of database that is quite different to, but is nevertheless sometimes confused with, OLAP. COP databases perform high speed row counting and filtering at a very detailed level, using column-based structures, rather than rows, tables or cubes. COPs analyze detail and then occasionally aggregate the results, whereas OLAPs are largely used to report and analyze aggregated results with occasional drilling down to detail.

Although less well known and recognized than OLAP, COP databases have also been in use for more than 30 years (for example, TAXIR), which means that, just like OLAP, they predate relational databases. COP products are used for very different applications to OLAP and examples include Alterian, Sand Nucleus, smartFOCUS, Sybase IQ, Synera, etc.

As Nigel says, they have been around for ages but have never seemed to be as popular as OLAP. I have no idea why this is because the technology is fundamentally good - my very first project using OLAP Services back in the SQL 7 days was an attempt to replace a proprietary COP database, still desktop-based and not even 32-bit code, and we still struggled to match its performance on some queries. Maybe it just needed a big software company to buy into the sector in the way that Oracle and Microsoft did with OLAP for it to take off. There are certainly some big companies using it though - for example Tesco, the biggest supermarket chain in the UK:

http://www.sand.com/resources/casestudies/tesco-dunnhumby/

Anyway, the prompt for this blog entry was several people asking me about COP databases over the last few months and then, this morning, reading this entry on Shawn Rogers' blog about a new COP product called Paraccel:

http://www.b-eye-network.com/blogs/rogers/archives/2007/10/paraccel_and_su.php

It got me thinking... I wonder if Microsoft should think about buying or developing a COP database? I have no idea whether it makes sense or not, but you could integrate it as another engine within Analysis Services and perhaps even create a hybrid of OLAP and COP. People often complain about how bad AS is at querying transaction-level data, and drillthrough at the moment is very difficult to get good performance from; I wonder if a COP would help here. Certainly Paraccel's AMIGO feature where it can synch with an existing relational database sounds very much like processing a cube (only faster); give it the ability to be queried in MDX and think of the fun... and in the long run, maybe all this stuff should have closer integration with the relational database, as Oracle are doing.

SAP to buy Business Objects

Mon, 08 Oct 2007 17:13:35 GMT

If you care about this, I'm sure you've already heard that SAP have agreed to buy Business Objects:

http://www.eweek.com/article2/0,1895,2192841,00.asp

The general consensus seems to be that this is bad news for Business Objects customers and that the last few remaining independents (eg Cognos) will go the same way too. Will Microsoft be buying anyone? MS aways gets mentioned as a potential buyer in these kind of situations (see for example Mark Rittman) but I just don't see it happening myself - it would be completely out of character. IBM on the other hand...

Details on MS MDM emerge

Mon, 24 Sep 2007 21:46:28 GMT

Kirk Haselden has revealed a few details about what's going to happen with Microsoft's new MDM stuff on his blog:

http://blogs.msdn.com/knight_reign/archive/2007/09/23/microsoft-master-data-management-website-is-now-live.aspx

As the url above suggests, there's also a new page on the Microsoft website too:

http://www.microsoft.com/mdm

Cognos buys Applix

Fri, 07 Sep 2007 09:20:29 GMT

Yet another acquisition, this time Cognos buying Applix... some commentaries:

http://www.olapreport.com/consolidations.htm#Applix

http://www.intelligententerprise.com/blog/archives/2007/09/deal_for_applix.html

As the second article notes, there's some overlap between Temtec Executive viewer (which I know a few people are using on top of AS) and Cognos' own products, so I hope this doesn't mean another third-party AS client tool will disappear. Also, TM1 currently supports MDX as a query language, but who knows if continuing this support will be a priority for Cognos. What with this and the acquisition of Essbase I hope support for MDX in the wider OLAP community won't start disappearing.

Post-holiday round-up

Wed, 22 Aug 2007 15:15:02 GMT

The worst thing about taking a holiday is coming back home and, despite having taken my laptop with me and checking my mails a few times, finding there are a gazillion emails waiting for me to follow-up. Hohum. Anyway, here are a couple of things that need mentioning...

First of all, registration is open for the next BI evening in London on September 26th:
http://www.sqlserverfaq.com/

There's also a whole load of new stuff up at the SQLBits site, including some sessions (though not mine yet) and speakers:
http://www.sqlbits.com/

You can register for the conference here:
http://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032349295&Culture=en-GB

I see Vidas Matelis has picked up some more useful snippets of new features in AS2008 from a recent webcast:
http://www.ssas-info.com/VidasMatelisBlog/29_ssas-2008-katmai-info-from-august-9th-webcast#more-29

I missed the main webcast on the wider changes (originally billed as just being about the time series algorithm but it seems to have had its scope widened) in AS2008 but it looks like the recording should be available soon:
https://connect.microsoft.com/SQLServer/content/content.aspx?ContentID=6194

PerformancePoint CTP4 has been released and can be downloaded from Connect. David Francis has a feature list:
http://blogs.conchango.com/davidfrancis/archive/2007/08/16/PerformancePoint-CTP4-now-available.aspx

I also received a copy of Marco Russo's new book "Introducing Microsoft LINQ". I won't be reviewing here because I'm not really qualified to do so, but from the point of view of someone whose dev skills are pretty basic it looks like a good introduction to the subject.

Metashare

Fri, 27 Jul 2007 20:18:11 GMT

Mark Garner has just dropped me a mail to let me know that Metashare, a tool he's been working on for generating metadata documentation from SQL 2005 data warehouses and and SSIS packages, has just got to the beta 1 stage. Here's his blog entry announcing this:

http://mgarner.wordpress.com/2007/07/27/metashare-beta-1/

...and you can find out more about it, and download it, from Sourceforge:

http://metashare.sourceforge.net/

Oracle 11g materialised views

Mon, 16 Jul 2007 10:58:38 GMT

You may (or may not) have noticed that Oracle 11g got released recently. One of the new features that caught my eye is discussed in these two posts by Seth Grimes and Doug Henschen on the Intelligent Enterprise blog:

http://www.intelligententerprise.com/blog/archives/2007/07/oracle_11g_and.html

http://www.intelligententerprise.com/blog/archives/2007/07/oracles_11g_lau.html;jsessionid=0WUDMNQVPHFZSQSNDLRCKH0CJUNN2JVN

Seems like Oracle have used a new OLAP engine (ie not Express or Essbase) as the basis for managing large numbers of materialised views. When I first read this I thought the kind of feature that only a die-hard relational-database-lovin' OLAP-denier could ever get excited about, but then I realised that that's probably the point. It would be cool if SQL Server could use Analysis Services in the same way, and if it could it would open the eyes of a lot of new people to the power of Analysis Services.

BI Survey

Fri, 15 Jun 2007 05:20:03 GMT

It's BI Survey time again! Wasn't it the OLAP Survey last year? Anyway, it always makes for a fascinating read when it comes out. Here's the blurb:

We would very much welcome your participation in The BI Survey. This is the largest independent survey of OLAP users worldwide. The Survey will obtain input from a large number of users to better understand their buying decisions, the implementation cycle and the business success achieved. Both business and technical respondents are welcome.

The BI Survey is strictly independent. While vendors assist by inviting users to participate in the Survey, the vendors do not sponsor the survey, nor influence the questionnaire design or survey results. As a participant, you will not only have the opportunity to ensure your experiences are included in the analyses, but you will also receive a summary of the results from the full survey. You will also have a chance of winning one of ten $50 Amazon vouchers. Click here to complete the survey on-line:

www.eu-survey.com/BI/9

Microsoft buys Stratature

Thu, 07 Jun 2007 22:55:49 GMT

Microsoft have been on a bit of a buying spree this week: after the Dundas deal they've now gone and bought the MDM vendor Stratature. Take a look at Stratature's site for more details:

http://www.stratature.com/

Jamie Thomson has some extra information here:

http://blogs.conchango.com/jamiethomson/archive/2007/06/07/Microsoft-purchases-Stratature.aspx

Yet another step closer to Microsoft having a complete BI stack...

SQL2008 Webcasts

Tue, 05 Jun 2007 06:05:17 GMT

Via Euan Garden, here are some upcoming webcasts on new SQL2008 (Katmai) functionality:

http://blogs.msdn.com/euanga/archive/2007/06/04/sql-server-2008-web-casts.aspx

...including one on new dimension design features in AS.

Katmai Public CTP / Dundas Deal

Mon, 04 Jun 2007 20:38:17 GMT

I'm not at TechEd so this is second-hand news, but two big announcements: first of all, the first public CTP of Katmai is available for download on Connect; secondly, and more interestingly, a deal has been done where Microsoft has licensed Dundas' data visualisation components. Russell Christopher has the best breakdown of this deal I've seen so far here:

http://blogs.msdn.com/bimusings/archive/2007/06/04/microsoft-acquires-dundas-s-data-visualization-components-sql-2008-news-download-ctp3.aspx

He's right in wondering whether the OLAP component is part of this deal - that's the big question from the AS point of view.

UPDATE: see Russell's comment below - the OLAP components (http://www.dundas.com/Products/Chart/NET/OLAP/index.aspx) are part of the deal! For me, the obvious use is as a replacement for OWC, not just instead BI Development Studio but as a free component that anyone developing a thick/thin client for Analysis Services can use. Next question: does licensing the components mean that Microsoft can now give them away for free like this? Or can it only use them embedded in its own products?

Batch Reporting With SSIS and OfficeWriter

Wed, 23 May 2007 21:18:56 GMT

I'm a long-standing fan of OfficeWriter, the tool that's just been licensed by Microsoft for possible inclusion in Katmai Reporting Services, and recently I was engaged by Softartisans to write a few articles for their website (yes, that means I was paid). Here's the first of them, on how to create a batch reporting solution using their components and Integration Services:

http://officewriter.softartisans.com/officewriter-409.aspx

Seems a bit of a weird thing to want to do when you can do the same thing in Reporting Services, but as I say in the article there are some advantages for using Integration Services for this task. I'm going to write another article next week on using OfficeWriter with Excel 2007 and Analysis Services which I've got some fun ideas for...

Microsoft BI Conference Podcasts

Mon, 14 May 2007 10:44:20 GMT

No, I didn't get round to recording a podcast but it seems like someone else did a whole load:

http://www.microsoft.com/bi/resources/biconference.aspx

(via Ben Tamblyn)

The Rumour Mill

Sat, 12 May 2007 05:07:58 GMT

Mark Whitehorn has an interesting non-story over at the Register:

http://www.regdeveloper.co.uk/2007/05/12/microsoft_bi_three/

What was going on here? Perhaps something was going to be announced that couldn't be announced at the last minute? Well, I did hear some rumours about... hold on, someone's knocking at my hotel room door. Who could it be at this time of night? Be back in a minute...

SAP to acquire Outlooksoft

Wed, 09 May 2007 22:42:34 GMT

Yet more consolidation:

http://www.sap.com/usa/company/press/press.epx?pressid=7671

OLAP or Relational?

Tue, 01 May 2007 12:37:10 GMT

If there's one big religious divide within the BI world, greater than the differences between vendors, then the question of whether to use an OLAP database or to query the relational database directly is it. Standing in the pro-OLAP camp as I do I probably have more in common with an Essbase guy than someone who wants to do BI with the SQL Server 2005 relational database exclusively. Anyway, here's an article by Ralph Kimball in Intelligent Enterprise that sums up the arguments on both sides pretty well:

http://www.intelligententerprise.com/showArticle.jhtml?articleID=199202330

I'm pleased to see that he makes much of MDX as a plus-point on the side of OLAP; even Oracle's supposedly all-SQL approach begins to look a bit MDXy when you see the details (see for example Mark Rittmann's post, also from today: http://www.rittmanmead.com/2007/04/30/obi-ee-time-dimensions-and-time-series-calculations/) although from what I've seen MDX still has some important advantages. It's just a pity that as a language it seems to be fragmenting into different vendor-specific implementations so quickly - for example, see this post from the Panorama blog on SAP BW MDX: http://www.panorama.com/blog/?p=44.

The future of Essbase

Mon, 23 Apr 2007 19:54:28 GMT

Interesting thoughts on which way the whole Hyperion/Oracle BI overlap might be resolved, from Cubegeek:

http://cobb.typepad.com/cubegeek/2007/04/hyperion_soluti.html

Business Objects to buy Cartesis

Mon, 23 Apr 2007 19:48:24 GMT

Industry consolidation continues - at this rate there'll be about three vendors left. Andy Hayler says pretty much everything that needs to be said here: http://andyonenterprisesoftware.com/2007/04/another-one-bits-the-dust/

AS2005 Design Best Practices White Paper

Tue, 27 Mar 2007 17:02:12 GMT

Via Darren Gosbell, I see that the best practices included in the forthcoming SQL2005 Best Practices Analyzer have been turned into a white paper which you can download here:

http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/olapdbpssas2005.mspx

Most of the recommendations are fairly obvious, some were new to me, some provoked a 'they need to fix that' response, and some I wasn't sure I agreed with 100%. For example:

Avoid creating diamond-shaped attribute relationships

I quite often find myself designing diamond-shaped relationships in my attributes; the example they give in the text isn't particularly good, but Day->Month->Year and Day->Week->Year is very common. OK most of the time I design user hierarchies along these paths anyway, but this need to design user hierarchies just for the sake of performance rather than because users actually want to see these hierarchies (and this isn't the first time I've heard this recommended) is something that I don't feel particularly happy about. It would be nice if we could make the choice between using attribute hierarchies and user hierarchies based on ease-of-use alone.

Avoid including unrelated measure groups in the same cube

This is the multiple measure groups and one cube vs multiple cubes with one measure group question that has been kicking around for at least a year. We were promised it was going to be addressed in the AS2005 Performance Guide but it wasn't; when are we going to get some details published? In my experience I have seen a slight improvement in performance when you split cubes up in this way but nothing major, yet clearly there must be some reason for MS to keep recommending this so perhaps I've not come across the right scenario yet.

Avoid having very big intermediate measure groups or dimensions of many-to-many dimensions

Obviously having large intermediate measure groups and/or dimensions in m2m relationships is going to slow things down, but in my experience I've always been pleasantly surprised with the performance of m2m dimensions. I've seen intermediate measure groups with much more than 1 million members in perform really well...

Avoid having partitions with more than 20 million rows

As an aside, I recently noticed that in Eric Jacobsen's recently blog entry on partitioning here he says you can also think about 2Gb (rather than a number of rows) as a rough guideline for the maximum size of a partition, and more interestingly that you should not have more than about 2000 partitions in your measure group. I asked him about this and it turns out that there are some internal limitations in AS at the moment, which hopefully will be fixed soon, that mean that slow down performance when you have large numbers of partitions. This does suggest that at the moment roughly speaking there's a maximum size for a MOLAP cube to perform well of around 4Tb or 30 billion rows... not that I've ever seen a AS2005 MOLAP cube that big, but interesting to note.

Do set the Slice property on partitions that are ROLAP or partitions that use proactive caching

Based on Eric's article mentioned in the last point, I think it's also a good idea to set the Slice property on MOLAP partitions where possible too given that the automatic detection of which members appear in which partitions gets confused by overlapping slices. Greg Galloway recently did some experiments on this which I saw where he showed that if you set the slice on a MOLAP partition the auto-slice information isn't used.

Avoid creating aggregations that are larger than one-third the size of the fact data

I quite frequently find myself designing aggregations which fail the one-third rule, although not by much, and for certain queries and calculations it can be a good idea. But I agree you should only do so as a last resort.

SQL2005 Samples on Codeplex

Thu, 22 Mar 2007 23:18:49 GMT

The SQL 2005 samples are now available for download from Codeplex:
http://www.codeplex.com/SqlServerSamples

Nothing you can't get from the usual sources, of course, but at least now you can download the bits you want individually; the site will also act as a pointer to other SQL Server projects on Codeplex too.

Oracle to buy Hyperion

Thu, 01 Mar 2007 11:11:49 GMT

See this article in the New York Times:
http://www.nytimes.com/2007/03/01/technology/01deal.html?_r=2&oref=slogin&oref=slogin

As Mark Rittman says (and he should know), "no doubt what Oracle are interested in here is Hyperion’s expertize in the area of analytic financial applications, and their no doubt high-value customer base."

What does this mean for Oracle BI and for Hyperion's toolset, especially Essbase? Given that Oracle already have their own OLAP tools, one wonders... I think this can only be a good thing for Microsoft - surely existing Essbase users will now be questioning its future and be looking for migration options, not necessarily from Oracle.

UPDATE: the news is confirmed, the price tag $3.3bn. Here are Nigel Pendse's initial thoughts:

http://www.olapreport.com/consolidations.htm#hyperion

Interesting that he thinks that "Essbase is likely to become Oracle’s primary OLAP server".

UPDATE #2: well, the consensus that's emerging is that Essbase is too valuable to be dropped. See for example:
http://cobb.typepad.com/cubegeek/2007/03/oracle_hyperion.html
http://andyonenterprisesoftware.com/2007/03/a-titan-no-more/
http://www.dbms2.com/2007/03/01/how-hyperion-will-change-oracle/

... but the fact remains that Oracle now has two MOLAP databases, and that's one too many in the long term. So perhaps it's the old Express users who need to be thinking about their migration options? And perhaps the Essbase people should remember that Oracle has bought a market-leading MOLAP before, and look what happened to it - not everyone is convinced that multidimensional is best.

UPDATE #3: I'll link, last of all, to Mark Rittman again, who shares his thoughts on what's happened and what it means for Oracle BI:

http://www.rittman.net/2007/03/04/off-to-iceland-and-thoughts-on-hyperion/

SP2 Coming Monday?

Fri, 16 Feb 2007 16:03:16 GMT

I've heard the rumours it was almost ready, and now according to Russell Christopher, SP2 could be ready for download on Monday:

http://blogs.msdn.com/bimusings/archive/2007/02/16/sql-2005-sp2-most-likely-available-this-monday-2-26.aspx

But then again, I've been told that SP2 was 'coming soon' since about October... From an AS point of view the thing to look out for will be the new version of the samples, which will include a tool for designing aggregations manually (discussed in the AS2005 Performance Guide) and also a bit of code showing how to run traces.

Microsoft Dynamics CRM Analytics Foundation

Thu, 15 Feb 2007 16:06:38 GMT

Earlier this week Microsoft released something called 'Microsoft Dynamics CRM Analytics Foundation':

http://www.microsoft.com/presspass/press/2007/feb07/02-12AnalyticsFoundationPR.mspx

The name's a bit of a mouthfull, but basically it's a shared-source solution (including AS2005 databases) for building BI solutions on top of Microsoft Dynamics CRM. It's available through Codeplex, and you can find out more about it on the project page:

http://www.codeplex.com/crmanalytics

I guess that there are going to be other similar packs available for other Microsoft products, Dynamics and otherwise, soon. Making it shared source is a nice idea too, although free stuff never quite has the same credibility as something you've paid for - not that customers won't be paying partners to implement and customise all this, I suppose.

Twelve 'Next Big Things' that didn't happen

Mon, 12 Feb 2007 14:32:14 GMT

On one of my periodic visits to the OLAP Report website I noticed this commentary, which struck such a chord I had to link to it:
http://www.olapreport.com/Faileddozen.htm

I don't think I've seen so many marketing concepts skewered so accurately in one place. Several of the dozen are relevant to current Microsoft product strategy, and while you could argue Nigel & co are unduly cynical I think a little cynicism is necessary to counter the overwhelming amount of hype that the BI software industry generates.

Microsoft/Teradata BI Announcement

Tue, 16 Jan 2007 10:05:53 GMT

It seems Microsoft and the newly-independent Teradata have announced they're going to 'optimise interoperability' between their platforms, with Analysis Services integration arriving in Q1 this year. Here's the full press release:
http://www.microsoft.com/presspass/press/2007/jan07/01-15MSTeradataPR.mspx

This has been in the pipeline for absolutely ages - I remember hearing something about an alliance like this about four years ago. In fact Teradata is already explicitly supported as a data source for AS, so I wonder what it is they're actually going to announce....?

Powershell script for backing up AS databases

Fri, 05 Jan 2007 21:19:52 GMT

Just seen on SQL Server Central, a useful-looking Powershell script that connects to an AS instance and back up all the databases on there:
http://www.sqlservercentral.com/scripts/viewscript.asp?scriptid=1850

Swivel

Sat, 09 Dec 2006 23:09:04 GMT

Just come across Swivel (http://www.swivel.com/), the 'YouTube for data'. There's a pretty good overview of what it's all about here:

http://www.techcrunch.com/2006/12/05/swivel-to-launch-this-week-communitize-your-data/

Apart from being a fun way to waste time for people who are into exploring data (which I assume most people who work in the BI industry are) I noticed you can also download data too - very useful for doing demos I think, so you can build a cube an explore some interesting data rather than show off Adventure Works for the 1000th time; even better for data mining demos.

Unfortunate name though...

Optimising Distinct Count

Sun, 19 Nov 2006 11:49:11 GMT

Great blog entry from Denny Lee on optmising distinct count measures here:

http://denster.spaces.live.com/Blog/cns!125D53A08EC75357!1156.entry

I'd seen a ppt deck of a presentation he did at PASS a few years ago where he first set out these ideas, but it looks like he's now updated them for AS2005.

On a similar topic, I've been working with a customer recently where we needed to optimise a calculated measure which did a count on a distinct list of customers (>1 million in the dimension) who met a certain criteria. The filter couldn't be designed into the cube or dimension completely so it had to be done with a filter function, and similar to what Denny describes here I was seeing in Profiler that the data was being read of the disk pretty quickly but it was the filter/count that was taking the time because only one CPU was being utilised. Unfortunately I found out that there's no way to design parallelism into the formula engine in the same way that Denny's technique above designs parallelism into the work the storage engine has to do - apparently AS prevents two queries executing at the same time unless one of them is a Cancel, and despite experimenting with sprocs to do this I had no luck because none of the code in the formula engine is thread safe. I was able to pass the raw data into a sproc using SetToArray and do the work outside the formula engine but that performed even slower. Hohum...

What's new in SP2

Tue, 07 Nov 2006 22:25:57 GMT

Just seen a link to a paper on what's new in SP2 on http://www.sqlis.com:

http://download.microsoft.com/download/2/B/5/2B5E5D37-9B17-423D-BC8F-B11ECD4195B4/WhatsNewSQL2005SP2.htm

Here's the bits relevant to Analysis Services:

Analysis Services

Microsoft Office 2007 requires the installation of SQL Server 2005 Analysis Services SP2 to support all of its business intelligence features. Features of Microsoft Office 2007 that require SP2 will be disabled when running against an instance of Analysis Services that does not have SP2 installed.
The performance of local cubes, grouping, and subselects have been substantially improved.
MDX query performance has been improved with respect to subselects, arbitrary shapes, running sum calculations, visual totals, ROLAP dimensions, cell writeback, many-to-many dimensions, 64-bit NUMA hardware, semi-additive measures and unary operators.
The functionality of subselects has changed.
A warning message now appears when a user-defined hierarchy is not defined as a natural hierarchy.
The MDX Drilldown* functions have a new argument that allows you to specify drilldown on specified tuples only.
The SCOPE_ISOLATION property has been added to the MDX CREATE MEMBER function.

This property enables session-scoped and query-defined calculations to be resolved before calculations in the cube rather than after.
Numerous functionality and performance-related bugs have been incorporated.

Specifically, improvements have been made to incremental processing, usage-based aggregation design algorithms, backward and forward compatibility, parent-child security, partition query scalability, cell writeback and the Time Intelligence Wizard.
The performance and functionality of the neural network viewer has been improved and support for multiple nested tables has been added.
The performance of naïve bayes predictions have been improved through caching of commonly used attributes.
Neural network training has been improved through better utilization of memory with sparse training data sets and better utilization of multiple threads during error computation (SQL Server 2005 Enterprise Edition feature).
Limited support for data mining viewers with local mining models has been added.
The redistribution of data mining viewer controls is now dependent upon ADOMD.NET.

All very vague... I hope they're planning on releasing some more detailed documentation on each of these. I know some people in Redmond are aware of how poor the documentation for AS has been in the past; let's hope they've been listened to.

UPDATE: you can download the CTP here:

http://www.microsoft.com/sql/ctp.mspx

UPDATE: ok, I take it back... at least, some of it. Here's the KB article containing a list of all of the bugs fixed in the CTP:

http://support.microsoft.com/default.aspx?scid=kb;en-us;921896

But there aren't just bug fixes, there are some quite important changes in behaviour too. At least Mosha is blogging about them.

SAP BI Accelerator

Mon, 25 Sep 2006 13:11:20 GMT

Via Mark Rittman, news (and good analysis of) SAP's recent announcement of its BI Accelerator product:

http://www.rittman.net/archives/2006/09/saps_new_bi_accelerator.html

Anyone who's read the OLAP Report or the OLAP Survey on SAP's BI offerings over the last few years, or indeed talked to anyone on the sharp end of a SAP BI implementation, will probably know that query performance has been a real problem for SAP BI. If, as Mark suggests, SAP have gone down the route of using COP database technology (see the entry here http://www.olapreport.com/glossary.htm for some more info) then perhaps they are going to solve this problem at last - which, from a competitor's point of view, is pretty scary because even though SAP's BI has got poor reviews for its technology, SAP have a formidable sales force and have been able to sell it regardless.

The only place I disagree with Mark is when he says "Where it falls short, I'd say, compared to products such as the OLAP Option is probably around calculations, time-series analysis, multi-dimensional queries and so on" - isn't it the case that you can query SAP Infocubes with MDX? In which case, problem solved.

This is good news too for Panorama, who, as you probably know, have fled to the arms of SAP after Microsoft acquired Proclarity. Here are three interesting blog postings from earlier this year from Oudi Antebi, ex-Microsoft and now of Panorama, on Microsoft, SAP BW and BI:

http://oudiantebi.spaces.live.com/blog/cns!B46B87F7D36F0A7C!1385.entry

http://oudiantebi.spaces.live.com/blog/cns!B46B87F7D36F0A7C!1404.entry

http://oudiantebi.spaces.live.com/blog/cns!B46B87F7D36F0A7C!1505.entry

He makes some very valid points here, especially the one about SAP 'owning the data'.

Note to self: time to start learning SAP-flavour MDX too...

Many-to-Many Dimension White Paper

Mon, 25 Sep 2006 05:59:57 GMT

Marco Russo has just released his white paper on many-to-many dimensions and all the cool things you can do with them. Here's the announcement on his blog:

http://sqljunkies.com/WebLog/sqlbi/archive/2006/09/24/23652.aspx

...and you can download it here:

http://www.sqlbi.eu/Default.aspx?tabid=80

It's really detailed and full of good information, definitely worth a look.

Attribute Relationships article

Mon, 11 Sep 2006 06:51:38 GMT

I check Richard Tkachuk's site every few weeks for new material (Richard, if you're reading, any chance you could implement an RSS feed?) and have just noticed he's put up a great article that explains the behaviour of attribute relationships in a lot of detail; there also seems to be a new version of his CellsetGrid control. His site is here:

http://www.sqlserveranalysisservices.com/default.htm

The MDX Performance tips article, also updated, is definitely worth a read too. One thing I've been wondering about recently to do with the whole multiple measure groups vs multiple cubes issue (see Teo's post on this here http://prologika.com/CS/blogs/blog/archive/2006/06/28/1331.aspx for more details) is whether using the VALIDMEASURE function, which Richard suggests is good for performance, is going to give as good performance as splitting a cube with multiple measure groups up into multiple cubes that use linked measure groups. Anyone got any thoughts?

Google Bigtable

Fri, 01 Sep 2006 20:10:25 GMT

A lot has been said about Google doing BI over the last few months, mostly fluff, but I just saw the following post on Cubegeek's blog and was intrigued:

http://cobb.typepad.com/cubegeek/2006/09/google_bi.html

I'd never heard of Bigtable - there's not much information available about it, but this post from last year gives some details:

http://andrewhitchcock.org/?post=214

And it turns out that Google have just released a paper about it:

http://labs.google.com/papers/bigtable.html

Hmm, now although there's no suggestion anywhere that they are planning this, if this was exposed as a service (and if it ever got out of beta!) it would be interesting...

Timeline sue Microsoft again

Fri, 01 Sep 2006 10:26:12 GMT

I see Timeline are suing Microsoft again about Microsoft's BI tools (principally AS, I assume) infringing their patents...

http://www.tmln.com/docs/PR_8-21-06_Microsoft_contract_termination.pdf

hohum, this happened a few years ago... here's an example of the kerfuffle that happened last time they sued:

http://www.theregister.co.uk/2003/02/20/sql_server_developers_face_huge/

Not the best advertisement for the concept of software patents in my opinion. There must be plenty of examples of prior art out there - see Nigel Pendse's history of OLAP, for example: http://www.olapreport.com/origins.htm