Chris Webb's BI Blog: MDX

Named Sets, AutoExists and Katmai

Tue, 15 Jul 2008 16:08:20 GMT

A couple of months ago Radim Hampel pointed out to me some very weird stuff happening with named sets and the Where clause. Since it turned out that Darren had run into the same issue and also been thrown by it, and since I tested it out on Katmai CTP6 and could see that it was behaving differently from AS2005, I opened an item on Connect:
https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=331186

And now, after a long and detailed email thread involving Mosha, Edward Melomed, Marius Dumitru, Darren and Deepak we've got to the stage where I understand what's going on, Katmai RC0 does roughly what I want, and I can blog about it!

Let me explain what I saw first. What would you expect the calculated member in following query to return?

--Query 1
WITH
SET mytestset AS
    PeriodsToDate
    (
      [Date].[Calendar].[Calendar Year]
     ,[Date].[Calendar].[Month].&[2004]&[6]
    )
MEMBER measures.test AS
    SetToStr(mytestset)
SELECT
measures.test ON 0
FROM [adventure works]
WHERE
[Date].[Calendar].[Month].&[2004]&[7]

Just from looking at the code I would have set the calculated member should return the string representation of the set from January 2004 to June 2004. But if you run the query you will in fact see that it returns the set containing the member July 2004 on both AS2005 and Katmai. To me that made absolutely no sense... Now, take a look at this query:

--Query 2
WITH
SET mytestset AS
[Date].[Calendar].[Month].&[2004]&[6]
MEMBER measures.test AS
SetToStr(mytestset)
SELECT
measures.test ON 0
FROM [adventure works]
WHERE
[Date].[Calendar].[Month].&[2004]&[7]

On AS2005 the calculation returns June as I would expect, on Katmai it returns an empty set. Now run this query:

--Query 3
WITH
SET mytestset AS
    {
      [Date].[Calendar].[Month].&[2004]&[5]
     ,[Date].[Calendar].[Month].&[2004]&[6]
    }
MEMBER measures.test AS
    SetToStr(mytestset)
SELECT
measures.test ON 0
FROM [adventure works]
WHERE
[Date].[Calendar].[Month].&[2004]&[7]

and this query:

--Query 4
WITH
SET mytestset AS
    (
      [Date].[Calendar].[Month].&[2004]&[5]
     :
      [Date].[Calendar].[Month].&[2004]&[6]
    )
MEMBER measures.test AS
    SetToStr(mytestset)
SELECT
measures.test ON 0
FROM [adventure works]
WHERE
[Date].[Calendar].[Month].&[2004]&[7]

...which to me should do the same thing. On AS2005 query 3 returns the set May and June but Katmai returns an empty set; query 4 returns an empty set on both AS2005 and Katmai. At this point I could see that something funny was happening that I didn't like!

What are the practical implications of this? Take the following query from Mosha's blog entry on ranking:

WITH
SET OrderedEmployees AS
    Order
    (
      [Employee].[Employee].[Employee].MEMBERS
     ,[Measures].[Reseller Sales Amount]
     ,BDESC
    )
MEMBER [Measures].[Employee Rank] AS
    Rank
    (
      [Employee].[Employee].CurrentMember
     ,OrderedEmployees
    )
SELECT
[Measures].[Employee Rank] ON 0
,[Employee].[Employee].[Employee].MEMBERS ON 1
FROM [Adventure Works]

Run it and you'll see that the Employee with the key 46, A Scott Wright, has a rank of 18. Now let's slice by this Employee:

WITH
SET orderedemployees AS
    Order
    (
      [Employee].[Employee].[Employee].MEMBERS
     ,[Measures].[Reseller Sales Amount]
     ,bdesc
    )
MEMBER measures.[employee rank] AS
    Rank
    (
      [Employee].[Employee].CurrentMember
     ,orderedemployees
    )
SELECT
measures.[employee rank] ON 0
FROM [adventure works]
WHERE
[Employee].[Employee].&[46]

Run this and you'll see that A Scott Wright has now supposedly got a rank of 1. Whatever the logic behind this, it doesn't make sense from an end user perspective does it?

So how can we explain what's happening here? It's all to do with autoexists: in some cases it makes sense to apply autoexists to named sets, but in others (mostly when the set is intended for use in a calculation) then it doesn't. Let's forget about trying to understand what AS2005 does because it tries to guess when it should apply autoexists and gets very confused, but Katmai is mostly consistent and logical: by default it applies autoexists to all named sets. That explains why queries 2,3 and 4 all return empty sets on Katmai: May and June don't exist with July.

After my initial item on Connect was opened those nice people in Redmond (who agreed with me that the way things were working wasn't ideal) added a new connection string property, Autoexists, which can have the following values:

0 – default (same as 1)
1 – Apply deep autoexists for query axes and named sets (with WHERE clause and subselects)
2 – Apply deep autoexists for query axes and no autoexists for named sets (with WHERE clause and subselects)
3 – Apply shallow autoexists for query axes with WHERE clause, deep autoexists for query axes with subselects, no autoexists for named sets with WHERE clause and deep autoexists for named sets with subselects

Here's the explanation I got from Marius about what's meant by 'deep' and 'shallow' autoexists:

Suppose a query axis or named set involves a set expression of the form F(G(s)), with F and G being set functions (e.g. TopCount, Tail etc.)
Let SSW denote the Where clause slice and Subselect restrictions present in the query.
“Shallow autoexists” evaluates the set expression as Exists(F(G(s)), SSW).
“Deep autoexists” evaluates the set expression as Exists( F( Exists( G( Exists(s, SSW) ), SSW ), SSW) – it applies autoexists with the Where and Subselects at every intermediary step. Deep autoexists applies to all set-valued subexpressions being evaluated in the root context of the query (i.e. the context of default member coordinates, or Where clause coordinates, if a Where clause is present).
For many/most set functions (e.g. Union), the rules above produce the same result.
For others (e.g. TopCount, Head/Tail), the results differ in the general case.

So the behaviour I describe above for RC0 is also what you get when you put Autoexists=1 in the connection string. But what about Query 1 - why does the PeriodsToDate function return July with this setting? Hmm, well I think this is a bug and it should return an empty set. I opened another Connect about this:
https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=356193
and I'll update this post when I get an answer on it.

What happens with Autoexists=2 and Autoexists=3 then? In both cases, Query 1 returns the set of months from January to June; Query 2 returns June, Query 3 and Query 4 both return May and June. To see the difference between these two settings take a look at this query:

WITH
SET mytestset AS
    (
      [Date].[Calendar].[Month].&[2004]&[5]
     :
      [Date].[Calendar].[Month].&[2004]&[6]
    )
MEMBER measures.test AS
    SetToStr(mytestset)
SELECT
measures.test ON 0
FROM (SELECT
[Date].[Calendar].[Month].&[2004]&[7] ON 0
FROM [adventure works] )

With Autoexists=1 and Autoexists=3 the set here is empty; with Autoexists=2 then it contains May and June.

Dynamically generating session calculated members in code with Analysis Services 2008

Thu, 03 Jul 2008 17:25:56 GMT

One problem I come across from time to time is the need to be able to group members on a dimension into buckets, for example if you have a Customer dimension you may want to group your Customers together by age group and run a query that shows sales for the 0-9 age group, the 10-19 age group and so on. If you know what buckets you want then you can simply build this into your dimension in advance by creating another attribute; but what if your users want to be able to change the definition of the buckets themselves from query to query - perhaps they wanted to 5 year age groups rather than 10? I described a bit of a hack you could use on AS2K to do this a while ago, but it doesn't work any more on AS2005 or AS2008 because it uses the unsupported CreatePropertySet function and in any case it was pretty nasty; but the only alternative is to be able to have your client tool dynamically create large numbers of calculated members (one for each bucket) with the appropriate definition and then use these calculated members in your query.

However with Analysis Services 2008 there's now another possibility - you can create a stored procedure that can create calculated members within the current session. The "Analysis Services Personalization Extensions" sample shows off a lot of the new possibilities for server-side coding and Michel Caradec blogged about them a while ago too, but neither show how to create calculated members so I thought I'd put an example up here.

The following class generates the calculated members needed to draw a Lorenz curve (as always please excuse the code - I'm no great C# coder and I just threw it together to make the point):

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.AnalysisServices.AdomdServer;

namespace testdc
{
    public class testdc
    {
        public static void Lorenz(string cubeName, int incr, string setToGroup, string numericValue)
        {
            AdomdCommand cmd = new AdomdCommand();
            try
            {
                //drop set
                cmd.CommandText = "drop set [" + cubeName + "].[Lorenz]";
                cmd.ExecuteNonQuery();
            }
            catch
            {
                //if the set isn't there we'll get an error we can ignore
            }
            StringBuilder s = new StringBuilder();
            s.Append("{");
            //create the calculated member for the total of the set
            //try to drop the member first, in case it exists
            try
            {
                cmd.CommandText = "drop member [" + cubeName + "].measures.LorenzTotal";
                cmd.ExecuteNonQuery();
            }
            catch
            {
                //if the member isn't there we'll get an error we can ignore
            }
            cmd.CommandText = "create member [" + cubeName + "].measures.LorenzTotal ";
            cmd.CommandText += "as aggregate(" + setToGroup + " , " + numericValue + ")";
            cmd.CommandText += ", visible=false;";
            cmd.ExecuteNonQuery();
            //create the calculated members for the Lorenz curve
            for (int i = incr; i <= 100; i += incr)
            {
                //try to drop the member first, in case it exists
                try
                {
                    cmd.CommandText = "drop member [" + cubeName + "].measures.Lorenz";
                    cmd.CommandText += i.ToString();
                    cmd.ExecuteNonQuery();
                }
                catch 
                {
                    //if the member isn't there we'll get an error we can ignore
                }
                cmd.CommandText = "create member [" + cubeName + "].measures.Lorenz"; 
                cmd.CommandText +=  i.ToString();
                cmd.CommandText += " as aggregate(bottomcount(" + setToGroup ;
                cmd.CommandText += ", count(" + setToGroup + ") * " + i.ToString() + "/100 , ";
                cmd.CommandText += numericValue + "), " + numericValue + ")/measures.LorenzTotal";
                cmd.CommandText += ", visible=false, format_string='percent';";
                cmd.ExecuteNonQuery();
                s.Append("measures.Lorenz" + i.ToString() + ",");    
            }
            s.Append("{}}");
            //Create set containing all the calculated members
            cmd.CommandText = "create hidden set [" + cubeName + "].[Lorenz] as " + s.ToString();
            cmd.ExecuteNonQuery();
            cmd.Dispose();
        }
    }
}

You need to call the as follows:

call
testdc!Lorenz(
//The name of the cube you're using
"Adventure Works",
//The % increment for each grouping member
//For example, 5 will result in 100/5=20 members
5,
//The set to group
"[Customer].[Customer].[Customer].Members",
//The measure to use
"[Measures].[Internet Sales Amount]")

When it executes it creates a number of calculated members (the exact number depends on the second parameter) and a named set called [Lorenz] with them all in. Having done that you can run a query and request all the members in the named set:

select [Lorenz] on 0 from [adventure works]

You can call the sproc many times in the same session with different parameters and it should always work, although I'll admit it could do a better job of dropping old calculated members. It would also have been able to create a function that did much the same thing but which returned a set containing all these calculated members, so you could call it direct from your query and not have to call the sproc beforehand, but I couldn't get that to work unfortunately.

Using the Caption property with Calculated Members in AS2008

Tue, 17 Jun 2008 17:51:20 GMT

One of the many minor improvements in AS2008 MDX is the ability to specify a separate caption for calculated members. Vidas already did a good writeup of the other new properties that you can specify on calculated members, but for me the ability to specify captions is interesting because we should really be setting captions on calculated members as a matter of best practice.

First, how does it work? Well, it's very straightforward - here's an example query and output:

You can see that although we've declared a calculated measure whose unique name is measures.test, when the query is run the end user sees the caption "This is a test measure". Note you have to use a hard-coded string as a caption, you don't seem to be able to use an MDX expression.

But why should we be doing this? Because calculated member captions are something that change quite often, especially during development, and if you don't use the caption property then you'll end up having to change the unique name of the calculated member all the time - and this of course will break any other calculations in the MDX Script that refer to this calculation, and worse it will also break any existing queries in reports that refer to this calculation. Also since calculated member captions can be quite long, the ability to write MDX expressions that refer to a much shorter unique name means your code will be much more concise and readable.

Of course with real members on non-measures dimensions, if you have specified different columns for the Key and Name properties of the attribute hierarchy you already get this separation between the unique name of the member and the caption that the user sees. The only missing bit of functionality now is a separate Caption property for real (as opposed to calculated) measures.

Oracle and MDX

Wed, 04 Jun 2008 20:25:49 GMT

Meanwhile, in the parallel universe inhabited by Oracle users, a few weeks ago Mark Rittmann interviewed the architect for Oracle Business Intelligence Enterprise Edition, Phil Bates, and asked readers to suggest some questions for him. Of course I couldn't resist asking about MDX, even though I got a very nondescript response:
http://www.rittmanmead.com/2008/06/04/phil-bates-answers-your-questions/

Interesting that many of the other questions concerned data access (eg "Will Oracle reverse Hyperion’s undocumented strategy for making it almost impossible to get data out of Essbase successfully?") as well.

IBM Cubing Services and MDX support

Fri, 30 May 2008 09:11:45 GMT

Via Amyn Rajan of Simba Technologies, I see that IBM's Cubing Services OLAP tool now supports OLEDB for OLAP:
http://blogs.simba.com/simba_technologies_ceo_co/2008/05/ibm-cubing-serv.html

Another boost for the MDX language then! It's the lure of Excel compatibility that is driving all this of course, but it also opens the way for other MDX-friendly client tools as well.

Meanwhile MDX support is also proceeding in other, less direct ways. I mentioned before here that I hoped some of the new breed of data warehouse appliance vendors would start to support MDX as well and in a way it's starting to happen: Vertica and Pentaho announced a partnership a few months back and I've already heard of one instance of a company using Mondrian on top of Vertica. Hopefully other vendors will begin to realise that raw query performance is not much use unless matched with a language that allows you to easily express the queries and calculations you need.

LINQ to MDX

Tue, 27 May 2008 21:38:18 GMT

Seeing that Marco Russo has released his book "Programming Microsoft LINQ" reminded me of a conversation I had with him a while ago about something I've heard various people ask about over the last year - will there be a LINQ to MDX?

Before we go on, I should state that it's my opinion that there isn't a big enough market out there for anyone (Microsoft or a third party) to justify spending time developing LINQ to MDX. That's not to say that I wouldn't want to see it - I would - just that I doubt anyone much would use it. MDX remains too much of a niche language, and off-the-shelf tools work well most of the time so there's less need to write custom MDX-generation code. As far as I know Microsoft isn't planning on developing LINQ to MDX and I'd be surprised if it ever did, so this will remain a theoretical discussion.

But for the sake of argument if you were to implement LINQ to MDX the main problem you'd have to tackle would be the same one you have with using MDX in Reporting Services and Integration Services: MDX can't guarantee fixed column names for any given query. However I had an idea on how to avoid this, and that is to think in terms of LINQ to MDX sets rather than LINQ to MDX queries. So for example if you take the following SQL query on a dimension table:

Select Year, Quarter, Month
From TimeDim
Where Year=2008 and (Month=March or Month=April)

that would then translate easily into the following MDX set expression:

{([TimeDim].[Year].[2008], [TimeDim].[Quarter].[Q3], [TimeDim].[Month].[March]), ([TimeDim].[Year].[2008], [TimeDim].[Quarter].[Q3], [TimeDim].[Month].[April])}

MDX sets have to be made up of tuples containing the same dimensionality, and if you think of a set's dimensionality in terms of columns in a SQL SELECT statement then you can see how that might map onto LINQ concepts. Instead of using LINQ to create an MDX SELECT statement directly, you'd use LINQ to create MDX sets and then pass all of these sets into another function which would then run the query using the sets created as axes.

Since I'm no LINQ expert this was the point where I dropped a mail to Marco to ask him his opinion; he thought it was feasible and even came up with an idea of what the code might look like in C#:

var timeSet = from period in cubeBudget.TimeDim.Members
            where period.Year == 2008
                    && (period.Level == TimeDim.Level.Year
                        || period.Month == "March"
                        || period.Month == "April" )
              select period;

Var measures = from measure in cubeBudget.Measures
               Where new string[] { "Sales", "Quantity", "Price" }.Contains( measure.Name )
               select measure;

Var query = from measure in measures
            from period in timeSet
            select new { period.Name, measure.Name, measure.Value };

var cellset = from cell in cubeBudget
              where cell.Columns( measures )
                    && cell.Rows( timeSet )
              select cell;

In the code above, timeset would define the set of members on the Period dimension you wanted to use and measures would return the set of measures. Then you could use them in two ways: query would return a flattened rowset and cellset would return a cellset. But this all seems very convoluted and probably just as confusing to the average developer as raw MDX.

Another alternative approach would ignore MDX altogether and query Analysis Services using SQL directly (see http://cwebbbi.spaces.live.com/blog/cns!7B84B0F2C239489A!751.entry for more details on this topic), although I'm sure you'd run into the limitations of the SQL that is supported very quickly and in any case you lose all of the flexibility and functionality of the MDX language when you do this.

Maybe if we're looking for a way to programmatically generate MDX then LINQ isn't the way to do it. It's something olap4j is working towards and they have taken an approach that is much more in tune with the multidimensional nature of MDX. One of the 'open issues' that caught my eye in the section of the olap4j spec that deals with this functionality is the question "Is this API at the right level, or is it too close to MDX?", meaning I guess that it would be all too easy to come up with an interface that is just as complex as MDX itself. Where would you draw the line between ease-of-use and functionality? Is it the MDX language that is confusing for people, or the multidimensional concepts (concepts that any interface would have to reflect) that underpin it? Can you abstract MDX to an interface to make it easier to use? I wish I knew more Java so I could test drive olap4j - is there anyone reading this who is using it? As always, I'd be interested to hear anyone's thoughts on this matter so please leave some comments...

Currency formats: should they be tied to language?

Thu, 24 Apr 2008 18:31:11 GMT

One of the most commonly asked questions on the AS MSDN Forum is how to format measures that contain values in different currencies with the correct currency symbol. I've never blogged about this because a lot of people have already written up the solution in detail, for example Mosha:
http://www.sqljunkies.com/WebLog/mosha/archive/2005/10/13/mdx_format_currency.aspx

and Vidas:
http://www.ssas-info.com/analysis-services-faq/27-mdx/244-how-change-currency-symbol-based-on-selected-currency-dimension-member

However I was thinking about this recently and in my opinion there's a big problem with using the Language property to do this. And that is that when you set the Language of a cell, you not only change the currency symbol but you also change other ways that the number is formatted, for example the symbols used as thousands separators and decimal separators. In the US and UK of course, we use full stops (I think they're called 'periods' in the US?) as decimal separators and commas as thousands separators, but in continental Europe the roles are reversed. So the value:
100,001
would be interpreted as one hundred thousand and one in the UK, but one hundred point zero zero one in Germany, say, and the value:
100.001
would be interpreted in the opposite way. Borrowing one of the screenshots from Vidas's post you can see how the Language property respects these conventions:

So you can see what the potential problem is - what happens if you have values in Euros, USDs and GBPs in your cube? However much you educate your users you can guarantee that someone at some time is going to get confused or worse not realise what's going on and interpret the values incorrectly.

What's the alternative then? I think using Format_String has to be the way to go. If you alter Vidas's example so that instead of locale ids you put currency symbols inside the currency dimension named query, for example:

SELECT     CurrencyKey, CurrencyAlternateKey, CurrencyName,
                      CASE CurrencyAlternateKey WHEN 'GBP' THEN '£' WHEN 'EUR' THEN '€' WHEN 'JPY' THEN '¥' WHEN 'USD ' THEN '$ ' END AS LocaleID
FROM         DimCurrency
WHERE     (CurrencyKey IN
                          (SELECT DISTINCT CurrencyKey
                            FROM          FactCurrencyRate))

and then change his MDX assignment to be something like:

SCOPE ([Destination Currency].[Destination Currency].[Destination Currency].Members);
Format_String(This) = [Destination Currency].[Destination Currency].[Symbol].MemberValue + "#,#.00";
END SCOPE;

Then you get the desired result. However, one thing I did notice when I was experimenting with this is that if you try to use more than one character for your symbol (for example you might want to use CHF for Swiss Francs) you sometimes get the following error:

#Error The following system error occurred: Out of present range. .

Not good. Here's the bug logged on Connect:
https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=339913

Following on from all this, it also makes sense that users connecting from different locales automatically see numbers (but not currency symbols) formatted in the convention of their locale. So a German person might connect to the cube and see Euros with a € and USDs with a $, but see commas used as decimal separators, whereas a user in the UK would still see €s and $s with the correct symbol but full stops used as decimal separators. Now AS2K I seem to remember used to be able to handle this perfectly well - it could display the appropriate decimal separator and thousand separator depending on the client locale. However AS2005 RTM had a problem in that it worked ok for calculated measures but not for real measures; this was 'fixed' in SP2 so both calculated measures and real measures always got displayed in the locale of the server. American software, eh? And to think that so many members of the dev team are from Europe too. Here's the Connect:
https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=218858

and here's a forums thread on the subject:
http://forums.microsoft.com/msdn/showpost.aspx?postid=1488729&siteid=1&sb=0&d=1&at=7&ft=11&tf=0&pageid=1

Hmmmmm.... I need to check if this has been fixed properly in Katmai.

Can your sum be a subtraction? Or can you avoid it altogether?

Thu, 03 Apr 2008 17:21:32 GMT

Quite often you'll find yourself writing calculations that need to sum up large sets; in fact, they might be summing up all of the members on a level apart from one or two. In that situation it's worth using the structure of your dimension to your advantage. Take the following query, which sums up the Internet Sales of all customers apart from one and then does a TopCount based on this:

WITH
MEMBER MEASURES.TEST AS
SUM(
EXCEPT(
[Customer].[Customer].[Customer].MEMBERS
, {[Customer].[Customer].&[20075]})
, [Measures].[Internet Sales Amount])

SELECT MEASURES.TEST ON 0,
TOPCOUNT(
[Date].[Date].[Date].MEMBERS
, 10, MEASURES.TEST) ON 1
FROM [Adventure Works]

On my laptop using AS2005 the query runs in 2 minutes 2 seconds on a cold cache (Katmai does no better with this query, incidentally). Yuck. But if we change the calculation around, so that rather than summing the customers we do want we take the sales for all customers and subtract the sales for the customer we don't want (which is fine if the measure we're looking at has an aggregate function of Sum) then we can do the following:

WITH
MEMBER MEASURES.TEST AS
[Measures].[Internet Sales Amount] -
([Customer].[Customer].&[20075], [Measures].[Internet Sales Amount])

SELECT MEASURES.TEST ON 0,
TOPCOUNT(
[Date].[Date].[Date].MEMBERS
, 10, MEASURES.TEST) ON 1
FROM [Adventure Works]

...which executes in 1 second on a cold cache. Taking this further, if you have a set like this you're frequently summing up in calculations it might be a good idea to create a new attribute on your dimension to avoid having to do any work in MDX at all. In the Adventure Works example above, maybe Customer 20075 is the Sultan of Brunei and he ordered 5000 new bikes for all his friends - which means that including him in your calculations would skew the results. What you could do is create a new boolean attribute on Customer called something like 'Exclude from Calculations', which would then mean you could rewrite the query like this:

SELECT [Measures].[Internet Sales Amount] ON 0,
TOPCOUNT(
[Date].[Date].[Date].MEMBERS
, 10, [Measures].[Internet Sales Amount]) ON 1
FROM [Adventure Works]
WHERE ([Customer].[ExcludeFromCalculations].&[False])

and probably get even better performance.

Parameterising Calculated Measure Definitions

Tue, 26 Feb 2008 21:06:07 GMT

I'm currently holed up in my hotel room somewhere in Sweden facing up to the fact that I've barely started work on my presentation for SQLBits on Saturday. It's on the subject of using Analysis Services as a data source for Reporting Services, and as it happens one of the tips I'll be talking about came up at work today - handling parameterised calculated measure definitions - so I thought I'd blog about it too.

RS reports commonly use calculated measures in the WITH clause to get around that annoying problem that RS needs to know about its column names in advance. If you're creating a parameterised report which allows your users to choose which columns appear when it renders, you might be tempted to write your query something like this:

with member measures.test as
//this would actually be strtomember(@MyParameter) in the query
strtomember("[Measures].[Internet Sales Amount]")
select measures.test on 0,
topcount(
[Customer].[Customer].[Customer].members
*
[Date].[Calendar].[Calendar Quarter].members
, 100, measures.test) on 1
from [adventure works]

But performance isn't great here: the query executes in 23 seconds cold cache on my laptop. When you compare it with the un-parameterised version:

with member measures.test as
[Measures].[Internet Sales Amount]
select measures.test on 0,
topcount(
[Customer].[Customer].[Customer].members
*
[Date].[Calendar].[Calendar Quarter].members
, 100, measures.test) on 1
from [adventure works]

...which executes in a mete 2 seconds cold cache, then you'll be right in thinking we need to find a better approach. The culprit is of course the strtomember function; using any of the StrToX functions in a calculation is surefire way of killing query performance. Unfortunately because MDX parameters return strings (maybe we could get typed parameters in some future release?) we have to use strtomember to cast the uniquename string our parameter is returning to a member. The best way around this I've found is to create a named set in the With clause and parameterise that instead: it only gets evaluated once, and after that you can reference that set in your calculation so:

with
set mymeasure as {strtomember("[Measures].[Internet Sales Amount]")}
member measures.test as
mymeasure.item(0).item(0)
select measures.test on 0,
topcount(
[Customer].[Customer].[Customer].members
*
[Date].[Calendar].[Calendar Quarter].members
, 100, measures.test) on 1
from [adventure works]

This runs in a much more respectable 7 seconds on my laptop. But there's one more trick you can use though, our old friend non_empty_behavior. If we set it to the measure whose value we're actually displaying like this:

with
set mymeasure as {strtomember("[Measures].[Internet Sales Amount]")}
member measures.test as
mymeasure.item(0).item(0), non_empty_behavior= strtomember("[Measures].[Internet Sales Amount]")
select measures.test on 0,
topcount(
[Customer].[Customer].[Customer].members
*
[Date].[Calendar].[Calendar Quarter].members
, 100, measures.test) on 1
from [adventure works]

We're back down to 3 seconds on a cold cache.

Calculated members vs assignments to real members

Sat, 23 Feb 2008 23:04:07 GMT

One design technique I see used quite frequently on my travels, usually in financial applications and usually by people who have experience with OLAP tools other than AS, is the creation of members on a dimension which have no real data associated with them in the fact table and whose value is later overwritten with an assignment in the MDX Script. Imagine the following scenario, for example: you have a chart of accounts dimension which has a member Total Sales, which represents the total sales of all of the operating units in your company; you also want another member, Sales Plus Tax, which is the Total Sales value multiplied by a constant representing the rate of tax. To implement this you could either create a calculated member on the Account dimension which did the calculation, or in your Account dimension table create an extra real member and then in the MDX Script use a scoped assignment to write the value of the calculation to that member.

Creating the calculated member is the simplest option, so why not do that? Well, the 'real member' option has several advantages, such as:

You have absolute control over where the member appears in the hierarchy, unlike with a calculated member
Real members can have children, calculated members can't
Real members can have member properties associated with them, calculated members can't
It's easier to manage all of your dimension members in one place, ie in the dimension table itself
It's easier to write scoped assignments which cover only real members; writing scoped assignments that cover multiple calculated members can be a pain (see my blog entry here for example).
Some client tools (ok, Excel 2007) have problems with displaying and selecting calculated members on non-measures dimensions. For another practical example of the 'real member' technique, see Marco Russo's blog entry on using it to create a time utility dimension here to get around exactly this issue.

Hmm, so that's great, are there any disadvantages to using it? I'm going to have to be vague here, unfortunately, because I don't understand what's going on inside AS to be able to say for sure, but in many cases where I've seen the 'real member' technique used it's been associated with poor query performance. Even where the calculation involved has been trivial and there has been no further aggregation of the resulting value involved, I've seen situations where queries are slow and Profiler has gone nuts. It certainly doesn't happen for every calculation and every query, but when it has and I've created a calculated member that contains the same calculation performance has been much better. I wonder what's going on here? Maybe someone from Microsoft can comment? As you can probably guess, I'm writing this while onsite with a customer experiencing exactly this problem and I'm hoping to provoke some discussion on this topic...

In the meantime, all I can say is that be wary of using the 'real member' technique and test to see if a calculated member performs better. And hopefully in Katmai+1 we'll see some of the limitations of calculated members addressed so the provide a stronger alternative.

Cell Security and the Formula Engine Cache

Tue, 19 Feb 2008 16:46:39 GMT

I've been looking at a security implementation this week for a customer and investigating what impact it's having on their cube's performance. A number of interesting points have come up which are all worthy of a blog entry, but I thought I'd start with what I noticed about cell security. I'd heard from Mosha that cell security was just about the worst feature to use on a cube from a performance point of view because it stops the AS engine from knowing whether a cell is empty or not - and I can see this on my customer's cube because cold cache queries take a lot longer to run for users who have cell security compared to users who don't. But I also found another bad side-effect: it seems to stop the AS engine caching the results of calculations.

Here's the steps I took to repro this in Adventure Works (I used the Simple version but I'm sure it works just the same on the full version):

Comment out the entire MDX Script apart from the Calculate statement
Add the following calculated measure to the Script:
CREATE MEMBER CURRENTCUBE.MEASURES.[Test] AS
[Measures].[Internet Sales Amount]+1;
Add a new role to the database and give it 'Read' permission on the Adventure Works cube
Create a new Windows user (without administrator permissions on the cube) and add it to the new role
Start a Profiler trace, making sure you include the 'Get Data From Cache' event.
Connect to the cube using SQL Management Studio as this user (I used the 'Run As' option on the right-click menu) and run the following query:
SELECT {[Measures].[Internet Sales Amount], [Measures].[Test]} ON 0,
[Date].[Calendar Year].MEMBERS ON 1
FROM [ADVENTURE WORKS]
In Profiler you'll notice everything's as you'd expect when you're running a query on a cold cache: the disk is being hit, data is being read from partitions.
Run the query again, ie on a warm cache
In Profiler you'll see that this time the data used to answer the query is read from cache. There are four 'Get Data From Cache' events: two reading data from the measure group cache, which is raw data from the cube; and one each from the flat cache and the calculation cache, which are formula engine caches containing the results of calculations. For more information on these types of cache, see chapter 28 of "Microsoft SQL Server Analysis Services 2005".
Now, go back to your role and on the Cell Data tab check the Enable Read Permissions checkbox and enter the following expression:
IIF(1=1, TRUE, FALSE)
Incidentally, there's a bug to watch out for here: sometimes you need to click the Edit MDX button and then OK on the resulting dialog to make BIDS aware that the cell security expression has actually been edited. Also although this is a pretty trivial expression, I found that I could not repro the behaviour if I just used the expression:
TRUE
for cell security. Clearly AS is able to work out that this expression always returns true, even if if can't do the same for the first one!
Next, clear the cache and watching Profiler rerun the query twice.
Notice that on the second run, you now only see three 'Get Data From Cache' events and they are all for the measure group cache. So AS has been able to cache the raw data but not the results of the calculation.

If you imagine a scenario where there are hundreds of users, many connecting through roles with cell security, very complex MDX calculations and queries that take 5-15 seconds to run on a cold cache then you can imagine the kind of impact that cell security can have on performance. Queries that should run instantaneously on a warm cache are consistently taking almost as long to run as they did on a cold cache because the calculations have to be re-evaluated every time. So the moral of this tale is: don't use cell security unless you absolutely have to.

Dynamic named sets in AS2008 - not as fun as you might think

Mon, 14 Jan 2008 21:59:05 GMT

I got all excited when, last summer, Mosha blogged about dynamic named sets in Katmai: just when it looked like there weren't going to be any cool new features to play with in AS2008, here was a juicy new MDX thing. However I didn't really play with them properly until a bit later when I came to prepare my presentation for last autumn's SQLBits conference, and that's when I realised that they weren't quite as cool or useful as I had thought they were going to be.

The key thing I hadn't picked up on from my initial reading of Mosha's blog entry was that they are evaluated once per query. While that means they are still useful in some scenarios, one of the key examples that Mosha describes in his blog entry is a bit misleading and that's the ranking example. We all know, or at least we should all know, that the key to optimising rank calculations is to declare a named set which gets ordered just once and which is referenced inside the calculated member that returns the rank. Mosha accurately points out that the big drawback to this approach is that "It can only be used when the user can write his own MDX query" but then says that dynamic named sets are a solution - and my point here is that, in my opinion, they aren't really.

The problem can be seen if you change Mosha's example query slightly by adding Ship Date Calendar Years to the Columns axis:

WITH
MEMBER [Measures].[Employee Rank] AS RANK([Employee].[Employee].CurrentMember, OrderedEmployees)
SELECT
[Ship Date].[Calendar].[Calendar Year].MEMBERS
*
[Measures].[Employee Rank] ON 0
,[Employee].[Employee].[Employee].MEMBERS ON 1
FROM [Adventure Works]

If you run this query, you'll see that instead of seeing different ranks for different years, you get the same ranks repeated across every year - which is what you'd expect, because remember our dynamic named set is only evaluated once per query. I'm not saying this is a bug or something that should be fixed, however, because if the set was not evaluated once per query and evaluated every time it was called you'd be back where you started with poor performance; it's just that dynamic sets aren't very useful in this particular scenario. If the user can't write their own MDX then it follows that they're going to be in the situation where both the dynamic set and the rank calculated member are defined on the server and they'll be querying with a tool like Proclarity or Excel, so you'd expect them to be able to generate whatever query they wanted and have it work as they would expect, but as you can see it isn't going to.

Incidentally, if you're playing around with this there is a bug in the November CTP that Mosha told me about: if you have a calculated member that references a dynamic named set then it should appear in the MDX Script before the dynamic named set definition. If the calculated member definition comes after the named set definition you seem to get some problems with caching and strange results are returned.

Multiple Assignments Bug

Thu, 22 Nov 2007 19:03:16 GMT

I don't usually blog about the bugs I find in AS because, well, bugs usually get fixed pretty quickly and are usually only seen in very specific scenarios. However I'm going to make an exception here because this bug is very easy to reproduce, it can lead to inconsistent values being returned and it doesn't look like it's going to be fixed any time soon.

Here's the repro on Adventure Works (I used the Simple version, but I'll assume it works on the regular version too):

Comment out the MDX Script apart from the Calculate statement
Add the two following assignments:
([Measures].[Reseller Sales Amount])=1;
([Measures].[Reseller Sales Amount], [Employee].[Employees].&[291])=2;
Run the following query on a cold cache and note that the value returned for the employee Stephen Y. Jiang is 11 (all other non-leaf members show values aggregated up from their leaf members too):
select [Measures].[Reseller Sales Amount] on 0,
[Employee].[Employees].members
on 1
from [Adventure Works]
Now run the following query which only returns data for Stephen Y. Jiang without clearing the cache and note that the value returned is now 1:
select [Measures].[Reseller Sales Amount] on 0,
[Employee].[Employees].&[272]
on 1
from [Adventure Works]
Now rerun the query from step 3 and note that the value for Stephen Y. Jiang is still 1.

I have a case with PSS open about this, and from what I can gather the problem is to do with making multiple assignments to the same cell; I've only reproed it with a parent/child hierarchy, but I'm told it could appear on a regular hierarchy as well. In my opinion it's the values returned in Step 3 that are incorrect since an assignment on the member Jae B. Pak should only trigger an aggregation from leaf level for that member's siblings; the values from Step 4 for Stephen Y. Jiang and the other non-leaf members are right.

The job I'm working on at the moment is a financial application, and the only workarounds seem to be liberal use of FREEZE or making sure that you don't try to assign values to the same cell more than once, neither of which are exactly feasible in my situation: I'm already dealing with a vast amount of code in the MDX Script and getting either approach to work is next to impossible. I would assume that most financial applications will have parent/child hierarchies (eg on an Accounts dimension) and multiple overlapping assignments in them, so be warned! I wonder if this is going to be a problem for PerformancePoint applications?

AS2008 Block Computation

Wed, 21 Nov 2007 18:25:10 GMT

A lot has been said about block computation as it's the biggest feature coming in AS2008. Basically it means better performing MDX, but as you'd expect it doesn't mean that all your MDX queries and calculations are going to run faster, just some. But how do you know whether you're going to benefit? Well, digging through the latest update of AS2008 BOL today I found the following, very detailed description of when you will benefit from these performance improvements:

http://msdn2.microsoft.com/en-us/library/bb934106(SQL.100).aspx

Very interesting... the list of set functions is a bit limited (where is Filter?) but in general it looks like all common calculations are going to get faster.

Will MDX Go Mainstream?

Mon, 22 Oct 2007 19:12:59 GMT

I always look out for Mark Whitehorn's articles in the Register, if only because I'm tickled to see any mention of MDX on the front page of the UK's foremost IT news site. Here's his latest:
http://www.regdeveloper.co.uk/2007/10/22/mdx_intro/

While I agree with his judgement that "MDX will become a highly saleable skill" (it already is for me!), and while I agree with with all his arguments that MDX is a good thing and better than SQL for BI queries, I can't agree with his central argument that application developers are going to start learning it on a massive scale. I would really, really, really like to believe it but I can't. A small minority of developers who develop specialist analytical applications will need to learn a bit but everyone else will rely on third-party products like Dundas OLAP Services (what's going to happen about that with the MS acquisition of the Dundas products? Anyone know?) or the Intelligencia OLAP Controls. It's not that MDX is difficult per se, but that people who are used to thinking in SQL - and developers are always going to need to know some SQL - find it very difficult to start thinking in MDX, and that's a big hurdle to overcome. And as the existence of not just the tools I mentioned but the entire AS client tool market proves, it's also relatively easy to write a generic MDX query generator that will work well on just about any cube whereas you can't just write a generic SQL query generator that will work on any set of tables without building a metadata layer over the top (eg in the way Report Builder needs its Report Models); and once you've built that metadata layer you might as well have built a cube anyway.

Asymmetric Sets on Columns in Reporting Services

Wed, 10 Oct 2007 18:04:45 GMT

Now I don't know if I've blogged about this before - apologies if I have, but I can't find any trace of it on Google and since this is a fairly common performance-related problem it deserves to be mentioned. It's not rocket science MDX either but sometimes I think I focus too much on fun but obscure problems at the expense of real-world scenarios...

As you know, Reporting Services only allows you to have members from the Measures dimension on columns in a dataset. This is rubbish at the best of times since you need to use a matrix control to pivot your results, but it can sometimes present a performance problem. Consider the following query:

select
{([Measures].[Internet Sales Amount], [Date].[Calendar Year].&[2004]),
([Measures].[Internet Sales Amount], [Date].[Calendar Year].&[2003]),
([Measures].[Internet Tax Amount], [Date].[Calendar Year].&[2004])}
on 0,
non empty
[Customer].[Customer].[Customer].members
*
[Product].[Product].[Product].members
on 1
from [Adventure Works]

On the AdventureWorks Simple db (kind of like Adventure Works but with some things removed - you can download it here: http://www.microsoft.com/downloads/details.aspx?FamilyID=975c5bb2-8207-4b4e-be7c-06ac86e24c13&DisplayLang=en) this runs in around 11 seconds and returns 55361*3=166083 cells on a cold cache. It's typical of the kind of large reports people for some reason like to run in SSRS but the fact that you have an asymmetric set on columns becomes a problem when you try to convert it for use with a matrix: when you pivot Year to appear on rows you are automatically returning extra data you didn't want to use in your query, in this case values for Internet Tax Amount and 2003:

select
{[Measures].[Internet Sales Amount],[Measures].[Internet Tax Amount]}
on 0,
non empty
[Customer].[Customer].[Customer].members
*
[Product].[Product].[Product].members
*
{[Date].[Calendar Year].&[2003],[Date].[Calendar Year].&[2004]}
on 1
from [Adventure Works]

In this case it doesn't make much of a difference, but it can result in a big increase in the size of the resultset and a corresponding increase in query time especially if you have measures from different measure groups and your new query touches extra partitions as a result of the pivot.

You may think that the answer is to create a calculated measure to display the value of Internet Sales Amount for 2003 and put that on columns along with the regular Internet Sales Amount and Internet Tax Amount measures, and put the Year in the Where clause:

with member measures.inetsalesprevyear as
([Measures].[Internet Sales Amount],
[Date].[Calendar Year].currentmember.prevmember)
select
{[Measures].[Internet Sales Amount],
measures.inetsalesprevyear,
[Measures].[Internet Tax Amount]}
on 0,
non empty
[Customer].[Customer].[Customer].members
*
[Product].[Product].[Product].members
on 1
from [Adventure Works]
where([Date].[Calendar Year].&[2004])

But if you try running this you'll find that performance is much, much worse: in fact on my laptop I killed the above query after it had run for more than a minute. The problem is that AS can't optimise this query in the same way as it has the previous queries, and a quick look in Perfmon confirms this as you can see the Total Cells Calculated counter going through the roof when it runs.

What can we do? You might think Non_Empty_Behavior is the way to go but I've not got it to make any difference even with a hard-coded tuple; in fact we what we need to do is optimise the Non Empty rather than the calculated measure. Instead of looking for non empty rows where one of the columns represents a calculated measure, we can use the NonEmpty function to be specific about what rows we want to appear:

with
member measures.inetsalesprevyear as
([Measures].[Internet Sales Amount],
[Date].[Calendar Year].&[2003])
select
{[Measures].[Internet Sales Amount],
measures.inetsalesprevyear,
[Measures].[Internet Tax Amount]}
on 0,
nonempty(
[Customer].[Customer].[Customer].members
*
[Product].[Product].[Product].members
, {([Measures].[Internet Sales Amount],
[Date].[Calendar Year].&[2004].prevmember)
, ([Measures].[Internet Sales Amount],
[Date].[Calendar Year].&[2004])
, ([Measures].[Internet Tax Amount],
[Date].[Calendar Year].&[2004])})
on 1
from [Adventure Works]
where([Date].[Calendar Year].&[2004])

What I'm doing here is sticking with the same calculated measure but in the filter set for NonEmpty using the same set of tuples that I originally wanted on columns; the calculated measure is then only evaluated when we know that the row contains a non empty value for one of the three columns. Performance is exactly the same as the original query as far as I can see, and it can all be parameterised nicely.

Now if Reporting Services had proper support for MDX we wouldn't need to go into all this, but don't get me started on that topic again...

SCOPE and calculated members

Thu, 30 Aug 2007 16:08:29 GMT

This post on the MSDN forum:

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2064643&SiteID=1

...reminded me of something interesting I found out a few months ago. It seems that whereas you can't mix regular and calculated measures in a set used in the SCOPE statement, you can rewrite the assignment to avoid using SCOPE and do a direct assignment instead. So, for example, if you were trying to assign to a regular measure and a calculated measure using a SCOPE statement like this:

SCOPE({[Measures].[RegularMeasure],[Measures].[CalculatedMeasure]});
this=1;
END SCOPE;

You would get the following error:

A set has been encountered that cannot contain calculated members.
MdxScript(Cube1) (line, col) A set has been encountered that cannot contain calculated members.
The END SCOPE statement does not match the opening SCOPE statement.
MdxScript(Cube1) (line, col) The END SCOPE statement does not match the opening SCOPE statement.

You could rewrite the assignment as follows using two SCOPEs:

SCOPE([Measures].[RegularMeasure]);
this=1;
END SCOPE;

SCOPE([Measures].[CalculatedMeasure]);
this=1;
END SCOPE;

...but this is clearly a pain, as you're duplicating your assignment logic. What you can do instead is this:

({[Measures].[RegularMeasure],[Measures].[CalculatedMeasure]})=1;

Not as easy to read as using SCOPE, I know, especially if you're doing something complex, but at least it works! Now I wonder why SCOPE has this problem? Probably something worth opening an issue on Connect about...

White paper on resolving query performance problems

Wed, 29 Aug 2007 09:45:16 GMT

Another great and very detailed white paper from the SQLCat team entitled "Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services". You can download it here:

http://www.microsoft.com/downloads/details.aspx?FamilyId=975C5BB2-8207-4B4E-BE7C-06AC86E24C13&displaylang=en

There's loads of information here I've not seen before, for example on Perfmon counters. A must read!

When are named sets in the MDX Script evaluated?

Wed, 04 Jul 2007 21:22:45 GMT

Interesting discovery I made last week: I was tuning a cube and noticed that all my queries, when they were run on a cold cache, were much slower than I was expecting - even the most basic query seemed to take at least 40 seconds. After a lot of head-scratching I looked in Profiler and found the answer to what was going on, and it turned out to be the fact that there were two named sets in the MDX Script that used very complex expressions and which together took 40 seconds to evaluate. Commenting them out reduced all my query times by 40 seconds. The problem was that I wasn't referencing these sets in any of my queries...!

I had thought that named sets in the MDX Script were evaluated the first time they were actually used in a query but this is demonstrably not the case. Consider the following query:

with set mytest as bottomcount(
{[Department].[Departments].&[2],[Department].[Departments].&[3]}
*
{[Scenario].[Scenario].&[1],[Scenario].[Scenario].&[2] }
*
[Account].[Account].[Account].members
*
[Date].[Date].[Date].members
, 10,[Measures].[Amount])
select {} on 0,
mytest on 1
from [Adventure Works]

Executed on a cold cache on the Adventure Works cube it returns in 13 seconds on my laptop. Consider also the following very basic query:

select {[Measures].[Internet Sales Amount]} on 0,
[Date].[Calendar Year].members on 1
from [Adventure Works]

This returns in under a second on a cold cache on the Adventure Works cube. Now go into Visual Studio to edit the Adventure Works cube and add the set from the first query as a named set at the end of the MDX Script so:

create set myexpensiveset as
bottomcount(
{[Department].[Departments].&[2],[Department].[Departments].&[3]}
*
{[Scenario].[Scenario].&[1],[Scenario].[Scenario].&[2] }
*
[Account].[Account].[Account].members
*
[Date].[Date].[Date].members
, 10,[Measures].[Amount]);

If you then clear the cache and rexecute the second query, which used to complete in only 1 second, it will now take 13 seconds despite the fact it doesn't reference this set in any way. If you take a look in Profiler you can see that the Execute MDX Script event, which is fired the first time you run a query on a cold cache, is now taking 13 seconds and that's what's causing the query as a whole to take so long. So named sets in the MDX Script are evaluated when the MDX Script is executed, and that takes place the first time you run a query after the cache has been cleared - either by running a ClearCache command or by processing your cube.

Using Dynamically-Generated Sets in the MDX Script

Tue, 26 Jun 2007 21:32:18 GMT

Even more fun: I've just found that you can use the technique for dynamically generating sets within a cube's MDX Script. For example, add the following set declarations to the end of the Adventure Works cube's MDX Script:

create hidden set myyears as [Date].[Calendar].[Calendar Year].members;
create hidden set mytest as
generate(myyears,
strtoset("
intersect({},
{topcount(descendants([Date].[Calendar].currentmember,[Date].[Calendar].[Date]), 10,[Measures].[Internet Sales Amount])} as
[Top days for "
+ myyears.current.item(0).item(0).name + "])"
));

Then connect to the cube with your favourite client tool and, bingo! you see all the sets available to use in your queries:

The only thing to watch out for is that, for some reason, you need to declare the set you pass in to the first parameter of the Generate function separately as I have above, rather than inline.

I can see this as being quite useful - if you need to create a large number of very similar sets on your cube it's much more manageable than declaring each set individually since you only need to write the set expression once.

Advanced Ranking and Dynamically-Generated Named Sets in MDX

Mon, 25 Jun 2007 19:37:03 GMT

Calculating ranks is a classic MDX problem, and as is usually the case Mosha has an excellent blog entry on the subject which I would encourage you to read if you haven't done so already:
http://sqljunkies.com/WebLog/mosha/archive/2006/03/14/mdx_ranking.aspx

His advice can be summarised fairly simply - where possible declare a named set which is ordered to use in the Rank function rather than try to do this ordering every time you want to calculate a rank because it'll give you much better performance. This is all you need to know for 99% of the rank calculations you'll ever write; however, I've been doing a bit of thinking around the remaining 1% of scenarios and here's what I've come up with.

First of all it's worth pointing out that the Order function isn't always the most efficient way of ordering a set. As Mosha points out in this blog entry:
http://sqljunkies.com/WebLog/mosha/archive/2007/04/19/stored_procs_best_practices.aspx
... there's a performance bug in the Order function which makes it very slow. Mosha's own example query:

with member y as
count(
Order( [Customer].[Customer].[Customer].MEMBERS
*[Product].[Category].[Category].MEMBERS
*[Product].[Style].[Style].MEMBERS
, Measures.[Internet Sales Amount], BDESC))
select y on 0 from [Adventure Works]

...runs in 40 seconds on my laptop, and while he shows how a sproc can improve performance he doesn't mention that if you rewrite to use the TopCount function you get even better performance. The following query runs in 4 seconds on a cold cache on my machine:

with
member y as count(topcount(
{[Customer].[Customer].[Customer].MEMBERS
*[Product].[Category].[Category].MEMBERS
*[Product].[Style].[Style].MEMBERS} as myset
, count(myset), Measures.[Internet Sales Amount]))
select y on 0
from [Adventure Works]

The TopCount function uses a different algorithm to the Order function and it's optimised to return the top n members of an ordered set; for a relatively small set like the one in this example it performs better than the existing buggy implementation of Order but that may not always be the case. And of course it's highly likely that the Order function will be fixed in a future service pack so at some point it will start performing better than TopCount. As a result, use this approach at your own risk and test thoroughly!

But let's get back onto the subject of ranking proper. The obvious problem that Mosha doesn't deal with in his article is what happens when you have to calculate a rank on more than one criteria? To take Mosha's example, what about calculating the rank of Employees by Reseller Sales Amount for several Months, where those Months are to appear on Columns? If you know what those Months are going to be in advance it's fairly straightforward because you can create multiple named sets to use; the worst problem you're going to have is that your query is going to be pretty long. But what if you don't know what those Months are going to be or how many of them there are? For example, you might be filtering months on another criteria or using the TopPercent function. There's no way you can create the named sets you need to get good performance if you don't know how many sets you're going to need, is there? If you had complete control over the code of your client tool then you could dynamically generate your MDX query string to give you all the named sets you needed, but that would be a pain even if it was possible at all (and it wouldn't be with many off-the-shelf tools like Proclarity Desktop). Well, one solution to this problem simply uses one named set for all your Months:

WITH
SET MyMonths as TopPercent([Date].[Calendar].[Month].Members, 20, [Measures].[Reseller Sales Amount])
SET MyEmployees as [Employee].[Employee].[Employee].MEMBERS
SET OrderedEmployees AS
GENERATE(MyMonths,
ORDER(
[Employee].[Employee].[Employee].members
, ([Measures].[Reseller Sales Amount],[Date].[Calendar].CurrentMember), BDESC)
, ALL)
MEMBER [Measures].[Employee Rank] AS
RANK([Employee].[Employee].CurrentMember,
SubSet(
OrderedEmployees
,(RANK([Date].[Calendar].CurrentMember, MyMonths)-1) * Count(MyEmployees)
, Count(MyEmployees)
))

SELECT
MyMonths
*
[Measures].[Employee Rank]
ON 0
,MyEmployees ON 1
from [Adventure Works]

The set MyMonths contains the months that I'm interested in, and as I said because it uses the TopPercent function I don't know in advance how many Months it will contain. However I do know that there's a static number of Employees that I want to rank so in my OrderedEmployees set I use the Generate function to create a concatenated list of ordered Employees for each Month (note the use of the ALL flag here to make sure Generate doesn't do a distinct on the set it returns). In my Employee Rank calculation I can then use the Subset function to pick out the section of this set which returns the ordered list of Employees for the current month: it's the subset that starts at index (RANK([Date].[Calendar].CurrentMember, MyMonths)-1) * Count(MyEmployees) and is Count(MyEmployees) members long.

BUT... of course this only works because we know there are the same amount of Employees each month. What happens if we change the calculation and ask if the current Employee is in the top 75% of Employees by Reseller Sales Amount for each month? In this case, 8 Employees make up the top 75% for August 2003 but there are 10 for September 2003 and so on so this approach isn't any use.

The solution to this problem came to me while I was driving home down the motorway on the way back from my in-laws' house on Saturday afternoon, and when it did my wife asked me why I had suddenly started smiling so broadly - this is something that will get all you MDX fetishists out there (all three of you) equally excited. Basically it's a way of dynamically generating named sets within a query. Let's take a look at the whole solution first:

WITH
SET MyMonths as TopPercent([Date].[Calendar].[Month].Members, 20, [Measures].[Reseller Sales Amount])
SET MyEmployees as [Employee].[Employee].[Employee].MEMBERS
SET MyMonthsWithEmployeeSets as
Generate(
MyMonths
, Union(
{[Date].[Calendar].CurrentMember}
,
StrToSet("
Intersect({},
{TopPercent(MyEmployees, 75, ([Measures].[Reseller Sales Amount],[Date].[Calendar].CurrentMember))
as EmployeeSet" + Cstr(MyMonths.CurrentOrdinal) + "})")
))

MEMBER [Measures].[TopEmployee] AS
iif(
RANK([Employee].[Employee].CurrentMember,
StrToSet("EmployeeSet" + Cstr(Rank([Date].[Calendar].CurrentMember, MyMonths)))
)<>0, "Yes", "No")

SELECT
MyMonthsWithEmployeeSets
*
[Measures].[TopEmployee]
ON 0
,MyEmployees ON 1
from [Adventure Works]

This executes in 2 seconds on a cold cache on my laptop, compared to 52 seconds for the equivalent query which evaluates the TopPercent for every single cell, so it's definitely a big improvement. What I'm doing is in the set declaration for MyMonthsWithEmployeeSets using a Generate function to iterate over the set of Months I'm going to display on columns and return exactly the same set, but while doing so find the set of the top 75% Employees for each Month and store it in a named set declared inline. The way I do this is to return a set containing the current Month from the iteration and union it with an expression which returns an empty set; the top 75% set is evaluated and stored inside the expression which returns the empty set. The fun bit is that the expression which returns the empty set is inside a string which is passed into StrToSet, and as a result I can dynamically generate the names of my named sets using a static string plus the result of the currentordinal function. In the example above I end up creating five named sets called EmployeeSet1 to EmployeeSet5, one for each Month. Thanks to the fact that I can reference these sets in another calculated member so long as it's evaluated further down the execution tree (see http://www.sqlserveranalysisservices.com/OLAPPapers/ReverseRunningSum.htm), assuming I construct my SELECT statement appropriately I can then get at the contents of these sets within my TopEmployee calculated member using another call to StrToSet and some string manipulation to determine the name of the set that I'm after. How cool is that?

New release of the Analysis Services Stored Procedure Project

Sat, 23 Jun 2007 21:04:51 GMT

After months of procrastination, there's finally a new release of the Analysis Services Stored Procedure Project available here:

http://www.codeplex.com/ASStoredProcedures

Thanks to Darren Gosbell for doing the build and tidying up the wiki. Several of the new features have already been blogged about here and on Darren's blog so I won't go into too many details, but there's a lot of cool new stuff there to enjoy. My favourite is Greg Galloway's CreatePartitions function that allows you to automatically create partitions with slices based on the members of an MDX set - great for creating proof of concept cubes; his functions to dump AS memory and disk usage information are also impressive. Darren's XMLADiscover class contains a load of useful stuff, particularly the ClearCache function which clears the cache on a database or a particular cube. There's also some stuff that hasn't even been included in the project yet but hopefully will be soon, so stay tuned...

Want to work for Microsoft?

Thu, 24 May 2007 09:23:04 GMT

The AS user education team are having a hard time trying to recruit someone who knows MDX. Is this something you, dear reader, would be interested in? Here's the job description I got from Neil Orint:

Do you have a background in Analysis Services and MDX? Are you looking to put your mark on the next version of SQL Server content deliverables? Want to work on a writing team where your development team counterparts are as passionate about your content as you are? If so, the Analysis Services User Education team is looking for an experienced technical writer to assist us in delivering top-notch customer facing technical documentation.

The successful candidate will have a strong background in technical writing, a working knowledge of MDX and OLAP; possess solid project management and planning skills and a passion for learning new technologies. Strong communication skills are a must. As a member of our team you can expect opportunities to:
· Help define and execute upon content strategies and prioritizes
· Listen, analyze and respond to customer feedback
· Learn the entire spectrum of Microsoft’s business intelligence offering - Integration Services, Analysis Services, Reporting Services, Office and more.
Qualifications
· A practical understanding of MDX and OLAP.
· A history of developing assistance content for end users in a variety of delivery formats
· The ability to learn new tools, technologies, and processes quickly and independently

Please contact: neilor@microsoft.com

Please email Neil on the address given if you want to discuss it further.

AS2005 MDX Course Now Available

Mon, 21 May 2007 12:48:21 GMT

As I've mentioned before, I do all of my training activities through Solid Quality Mentors (also known as Solid Quality Learning) and earlier this year they persuaded me to write an MDX course for them. You can see the outline on their website here:

http://learning.solidq.com/na/CourseDetail.aspx?IdCourse=300

If you're interested in booking this as a private course please contact Solid Quality through the address on the site. If you're in the UK you'd definitely get me teaching it and I'd probably cover certain other European countries too, but the good thing about Solid Quality is that they have a network of top-notch BI people not only in the US but also in Central America and continential Europe too, many of whom will be teaching this class as well.

I think there's a real need for in-depth MDX training out there - even people who know Analysis Services really well sometimes struggle with it, and you can only get so far without understanding the fundamentals. With PerformancePoint on the way the market is only going to grow and I've deliberately kept the first day or so as platform-agnostic as possible with a view to adapting the material for SAP BW, Essbase, TM1 and Mondrian MDX at some point in the future.

One last thing to mention: Solid Quality have also got a lot of other Microsoft BI courses if you're not just interested in MDX. Take a look:

http://learning.solidq.com/na/BI.aspx

They've even got an Analysis Services data mining course written by Dejan Sarka:

http://learning.solidq.com/na/CourseDetail.aspx?IdCourse=177

</Shameless Advertising Plug>

Using Linear Regression to Calculate Growth

Fri, 18 May 2007 23:01:00 GMT

A few blog entries back I showed the MDX I used to calculate a seasonally-adjusted total in my chalk talk at the BI conference. This is useful but if we're looking for a calculation that we can use for the Trend property of a KPI it's not the whole story - we still need to find a way of expressing how much a value is growing or shrinking over time. Although previous period growth calculations are a lot more useful with seasonally-adjusted values, we can use simple linear regression (and it has to be simple because, as I said, I'm no statistician) to do a better job.

The starting point for understanding how to use linear regression in MDX is (surprise, surprise) Mosha's blog entry on the subject:

http://sqljunkies.com/WebLog/mosha/archive/2004/12/21/5689.aspx

However, the function that's going to be most useful here is the linregslope function. If we're looking at the values in our time series and trying to find a line of best fit for those values with the equation y=ax+b, linregslope returns the value of a in that equation, ie the gradient - when the value of x increases by 1, y increases by the value of a. Here's an example of how to use it:

with member measures.gradient as
linregslope(
lastperiods(3, [Date].[Calendar].currentmember) as last3
, [Measures].[Internet Sales Amount]
,rank([Date].[Calendar].currentmember, last3)
)

select {[Measures].[Internet Sales Amount], measures.gradient} on 0,
[Date].[Calendar].[Month].members on 1
from [Adventure Works]

The trick with using this function in MDX with a time series is to be able to work out what values you want to pass in for the x axis. Here I've used the lastperiods function to get a set containing the current member on the Calendar hierarchy, the previous member on the Calendar hierarchy and the member before that, in the first parameter of the function; at the same time I've declared a named set and then used that with a rank function in the third parameter to return the values 1, 2 and 3 for each of these three members.

This gets us the slope, then, but I was thinking it would be better to express this value as a percentage - but of what? The current period's value? Or one of the preceding two periods values? I have to admit I don't know which would be correct. Can someone help me out here? Please leave a comment..

Modelling Goals and Thresholds in Measure Groups

Fri, 18 May 2007 22:52:11 GMT

Before I carry on with my chalk talk series, I have to own up to something: I didn't actually want to present on the topic of KPIs, and when I found out that I was going to have to talk on the subject I fired off a few emails to people who spend more time with KPIs than I do to ask them if they could suggest some interesting things to talk about. One of these people was Nick Barclay, co-author of 'The Rational Guide to Business Scorecard Manager 2005' (which I shall be reviewing very soon - it's a good book), and he pointed out that while all the examples of KPIs he'd seen hard-coded goals and thresholds into the MDX code this was not a good thing - users want to change their values all the time and ideally you'd want to be able to let them do this themselves. Why not store these values in a measure group, allow users to change the values using writeback, and then use these values within the KPI definition somehow?

Actually modelling how the values should be stored in measure groups was very straightforward. In my demo I showed two fact tables, one for the Goals and one for the Thresholds, with one measure each. I also created a KPI dimension for both of them to allow multiple goals and thresholds for different KPIs to be stored in the same measure group; for the Goal measure group I added the Date dimension at the granularity of Calendar Year (so there was a column in the fact table containing year names) and for the Threshold fact table I also created a Threshold dimension. This Threshold dimension contained one member for each threshold to be used: Very Bad, Quite Bad, OK, Quite Good and Very Good; there was also a numeric column containing the values -1, -0.5, 0, 0.5 and 1 which represented the numeric values each threshold gets normalised to and which I assigned to the ValueColumn property of my sole attribute when I built the dimension.

One this was done and the measure groups were added to the Adventure Works cube, I showed some ways to allocate the Goal values down from the Year granularity at which they were stored. Here's the scoped assignment for the simple allocation which simply splits the values equally by the number of time periods in the year, so for example each month shows 1/12 of the year total:

SCOPE([Measures].[Goal]);
SCOPE([Date].[Calendar Semester].[Calendar Semester].MEMBERS, [Date].[Date].MEMBERS);
THIS=[Measures].[Goal]/
COUNT(
DESCENDANTS(
ANCESTOR([Date].[Calendar].CURRENTMEMBER,[Date].[Calendar].[Calendar Year])
, [Date].[Calendar].CURRENTMEMBER.LEVEL
)
);
END SCOPE;
END SCOPE;

Here's the code for doing the weighted allocation by the previous year's Internet Sales Amount measure values:

SCOPE([Measures].[Goal]);
SCOPE([Date].[Calendar Semester].[Calendar Semester].MEMBERS, [Date].[Date].MEMBERS);
THIS=[Measures].[Goal]
*
(
(PARALLELPERIOD([Date].[Calendar].[Calendar Year],1,[Date].[Calendar].CURRENTMEMBER), [Measures].[Internet Sales Amount])
/
(ANCESTOR([Date].[Calendar].CURRENTMEMBER,[Date].[Calendar].[Calendar Year]).PREVMEMBER, [Measures].[Internet Sales Amount])
);
END SCOPE;
END SCOPE;

A few things to note here:

In both cases, because I've set IgnoreUnrelatedDimensions to false on the measure group, to get the year's Goal measure value I can simply reference [Measures].[Goal] - the values for the year are copied down automatically to the attributes below the granularity attribute.
Although normally when you assign a value to a regular measure with an additive aggregation function the assigned value gets aggregated up, when you assign to a regular measure below the granularity attribute of a dimension no aggregation happens, similar to what you get with a calculated measure.
The assignment SCOPE([Date].[Calendar Semester].[Calendar Semester].MEMBERS, [Date].[Date].MEMBERS) means 'scope on everything from the Date attribute (I've included the whole attribute here, All Member and the leaf level - everything on a dimension exists with either the All Member or the leaf members of the key attribute) up to and including the Calendar Semester attribute but no higher'.

Moving onto the thresholds, we need to find a way to apply the threshold values we've got in our measure group to the measure we're interested in. Here's a calculated member definition that does this:

CREATE MEMBER CurrentCube.[Measures].[Internet Sales To Goal Status] AS
TAIL(
{[Threshold].[Threshold].&[1],
FILTER(
[Threshold].[Threshold].&[2]:null
, ([Measures].[Threshold],[KPI].[KPI].&[1])
<
([Measures].[Internet Sales Amount]/[Measures].GOAL)
)}
,1).ITEM(0).MEMBERVALUE
;

What I'm doing here is creating a set that always contains the first member of the Threshold dimension, the 'Very Bad' member, and then filtering on the set that contains every other threshold to return the members for whom the threshold measure is less than the value of Internet Sales Amount. I then get the last member in that set, which represents the threshold with the highest value that is less than Internet Sales Amount, and use the MemberValue function to get the normalised value (the value between -1 and 1) that I assigned to that member.