Chris Webb's BI Blog: Analysis Services

Show Hidden Objects in the Calculations tab

Wed, 23 Jul 2008 16:05:36 GMT

Here's a new button I've just noticed in the toolbar on the Calculations tab in BIDS 2008: Show Hidden Objects. Quite often when you're writing MDX you want to reference hierarchies that are hidden to the end user, and in BIDS 2005 you had to unhide them to be able to see them (and so find out their unique name) in the metadata pane, which was a pain; now in BIDS 2008 you just need to click the new 'Show Hidden Objects' button to be able to see them. For some bizarre reason it doesn't seem to allow me to see hidden sets or calculated members though - why?

Caching: can there be too much of a good thing?

Wed, 09 Jul 2008 16:40:07 GMT

I was doing some load testing on a cube recently and noticed an interesting thing. When my load tests started, the average query response time was high for the first minute or so, which is reasonable given that all queries were being run on a cold cache; subsequently the average query response time fell dramatically, as you would expect when the cache warmed up. But over a long period (and I was running a lot of queries over several hours) I noticed that the average query response time started creeping up again. Not by a massive amount, it's true, but enough to be noticeable: say a rise from three to four seconds.

Naturally I fired off an email to various people who I thought might help, and as usual Mosha came up with the goods. It turns out he'd noticed the same thing happening on the stress tests that MS run internally but come to the conclusion that it wasn't going to be a problem in the real world. I won't try to repeat too much of the technical detail he gave in case I end up garbling it but essentially there were two reasons why this performance degradation was happening:

Most stress tests use template queries into which random parameters are passed. So, for example, I had captured several 'typical' queries with four or five dimensions in the Where clause which in my test I parameterised;I then selected a random member on each of these dimensions to pass in to each query. This resulted in a lot of queries being run which returned no data at all, which in turn meant that the cache filled up with entries in the index that were very small and not much use, which in turn made cache lookups slower. In the real world the proportion of queries that return no data is much smaller.
The AS cache is optimised for ad-hoc query patterns, where a user starts in one place then drills down/up, slices, dices and so on, and where the data that a query returns is very closely connected to the data retrieved by the last query that was run. This is very different from randomised queries, which as a result perform worse.

So all in all, it's nothing to worry about. I guess if you had a large cube, lots of users, random query patterns and didn't process your cube very often then it might be a slight problem, but that's a lot of ifs.

As an aside, if you're thinking of doing stress testing I wouldn't waste any time trying to get the AS Stress tool I blogged about here working - I found it a real pain and ended up building my own load test tool in SSIS using an approach similar to my cache warmer package.

Connection String Properties in SQLMS RC0

Mon, 09 Jun 2008 16:04:33 GMT

I heard this was going to be possible back in March, but now I've seen it in RC0 - you can now set connection string properties in SQL Management Studio when opening an MDX query window. Just click the new MDX query button, then click Options and you'll see a third tab:

Although it's much less common to need to set connection string properties, there are still a few scenarios where it's useful. One example is the 'Cube' connection string property, which allows you to write session scoped script assignments in the way Mosha does here:
http://sqlblog.com/blogs/mosha/archive/2008/06/08/mdx-answer-to-nextanalytics-challenge.aspx

Another good example is when you're using the Roles or EffectiveUserName properties to test out how queries behave with security.

SQL Server 2008 RC0 is out

Sat, 07 Jun 2008 06:35:48 GMT

Andrew Fryer has the details here:
http://blogs.technet.com:80/andrew/archive/2008/06/06/sql-server-2008-rc0.aspx

If we've got a release candidate, then this means RTM isn't too far away...

Cell Security: when Read permissions are actually Read Contingent

Tue, 20 May 2008 16:59:16 GMT

I usually avoid using cell security like the plague for the good reason that it absolutely kills query performance. But sometimes there's no alternative but to use it and I'm working on one such project right now. However I've found a new gotcha: the behaviour of cell security changed between AS2005 SP1 and SP2 and in my opinion it didn't change for the better. The short explanation is that when you are using just Read permissions in cell security, if you have MDX Script assignments at a lower level of granularity you'll find that the higher level members whose values are affected by the assignments have Read Contingent permissions applied to them, even though according to the MDX expression I used in for the Read permissions these values should not be secured.

Here's the repro I've got for Adventure Works using AS2005 SP2:

In AW, comment out everything in the MDX Script apart from the Calculate statement
Add the following calculations to the Script:
CREATE MEMBER CURRENTCUBE.MEASURES.[Show Value] AS
IIF(
[Date].[Calendar].CURRENTMEMBER IS
[Date].[Calendar].[Month].&[2004]&[1],
FALSE, TRUE);

CREATE MEMBER CURRENTCUBE.MEASURES.[Show Internet Sales Amount] AS [Measures].[Internet Sales Amount];
As an Administrator you see the following results for a query in the cube browser:
Now create a role with the following Read permissions in the cell security tab:
IIF(
[Date].[Calendar].CURRENTMEMBER IS
[Date].[Calendar].[Month].&[2004]&[1]
AND
[Measures].CURRENTMEMBER IS [Measures].[Internet Sales Amount],
FALSE, TRUE)
Display the same query in the browser when connecting through this new role, and as expected you see the following:
Now add the following assignment to the MDX Script:
([Date].[Calendar].[Month].&[2004]&[8])=[Date].[Calendar].[Month].&[2004]&[7];
And refresh the query, so you see this:
The big change is that the value for CY2004 and the All Member are now secured too. So we’re now in the situation where we’ve made an assignment that displays a value that wasn’t secured anyway, but we now can’t see the CY2004 value even though the expression in the Read permissions returns true. If you were to change the assignment so it refers to the value for January 2004, as follows:
([Date].[Calendar].[Month].&[2004]&[8])=[Date].[Calendar].[Month].&[2004]&[1];
You can see the January value either while looking at August or the ‘Show Internet Sales Amount’ calculated measure, as I would expect, because neither are aggregated from anything and the expression for the Read permission always returns True for them:

For my customer, who is migrating from AS2005 SP1 direct to Katmai, this is potentially a major problem. And to be honest as my repro shows I don't think this new behaviour makes any sense at all anyway - if I wanted to use Read Contingent permissions I'd have used Read Contingent permissions!

SqlSpec

Thu, 15 May 2008 20:42:13 GMT

I've just come across SqlSpec from Elasoft (http://www.elsasoft.org/) a tool that can create documentation for Analysis Services databases. You can see some sample output here:
http://www.elsasoft.org/samples.htm

I've downloaded it and given it a quick try, and it seems to do the job pretty well. This is the second documentation tool for SSAS that I'm aware of, the other being BI Documenter. Which one is better? Well I guess it depends on what you want to do since the two tools support different data sources and have slightly different features (http://www.elsasoft.org/compare.htm vs http://www.bidocumenter.com/Public/Features.aspx). You can see sample output for BI Documenter here:
http://www.bidocumenter.com/Public/SampleOutput.aspx

You pays your money, you takes your choice...

Analysis Services Browser Views addin

Wed, 07 May 2008 11:33:02 GMT

Via Russell Christopher, news of a new addin for BIDS created by Yossi Elkayam and Eran Sagi of Microsoft Israel that allows you to save the Analysis Services queries created in the browser tab:

http://blogs.microsoft.co.il/blogs/bei/archive/2008/04/25/analysis-services-browser-views-add-in.aspx

Looks very useful... Greg, Darren, perhaps this can be integrated with BIDS Helper?

Transact SQL Server Analysis Services Metadata

Thu, 01 May 2008 14:10:46 GMT

Via http://www.ssas-info.com/ (which is a great site, if you haven't seen it then definitely check it out) I've just come across a new project on Codeplex that aims to allow you to browse and manage AS metadata through T-SQL from Leandro Tubia:

http://www.codeplex.com/TSSASM

Kind of like what Darren Gosbell's doing with Powershell, only for people who don't want to learn Powershell, and what Darren also did in the Analysis Services Stored Procedure Project but which the full flexibility of T-SQL. Nice idea; it's still early days for the project but I hope it develops.

Things I'd like to see in Analysis Services 2011

Wed, 30 Apr 2008 15:33:52 GMT

It's that time in the release cycle. I know what the new features are going to be for Analysis Services (for a full list see here) and it's rather like Boxing Day: I've just got my presents and I'm starting to wonder to what Santa will bring on his next visit. And to a certain extent I'm feeling as though I should have been a bit more specific when I was writing my Christmas list last time, given that I'm a teeny bit underwhelmed by the features I did get in AS2008. So here, for the benefit of the elves working away in Redmond, is a list of things I'd love to see in Analysis Services 2011 or whatever it will be called:

Auto-partitioning
Building and maintaining partitions is a job that almost everyone needs to do and yet it's something that involves way too much hard work at the moment. As such it's an ideal candidate for some form of automation, and indeed a while back Jamie Thomson posted the following suggestion on Connect: https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=127378. Other possibilities include the use of data mining to determine how you should set the slices for your partitions (which I blogged about a long time ago) and something like the MDX driven auto-partitioning that Greg Galloway implemented for the Analysis Services Stored Procedure Project.
Do away with the arbitrary shaped sets error
This is a particular bugbear of mine. The more I work on cubes with complex calculations, the more I find that I want to scope calculations on arbitrary shaped sets, which of course I'm not allowed to do. That means I end up making several identical assignments to get the same result, which leads to way too much duplicate code and a maintenance nightmare. If I can break up an arbitrary shaped set into something acceptable to Analysis Services, why can't Analysis Services do this itself? Here's the Connect I logged on this: https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=339861
Fix parent-child hierarchies
Parent-child hierarchies are a pain. For a start they cause performance problems so what I would want to see is the ability for AS to create aggregations at intermediate levels in a parent-child hierarchy at the very least; I'm sure there are a lot of other potential engine improvements that could be made for them. Secondly, scoping calculations on them is a real nightmare and on a related note trying to use dimension security with them is an equally thankless task. Given that there are some situations where you have no option but to use them, I think some time needs to be invested in making them better.
Build parallelism into the calculation engine
What with all the block computation improvements in AS2008, I'm finding that many calculations are performing a lot better than in AS2005. However I'm still finding scenarios where the calculation engine is the bottleneck and just one CPU is being used on the server when I'm running a query (something I discussed here). Surely it's possible to get the calculation engine to do some calculations in parallel when a query is being run?
Be able to rename attributes in role-playing dimensions
If there was one feature I wanted to see in AS2008 it was this. It would have been so easy to do, I know loads of other people wanted it, and I did log it in Connect ages ago: https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=144500. And it wasn't included. All I want to do is to be able to rename the individual attributes of a role-playing dimension instead of just being able to change the dimension name - for example if I have Order Date and Ship Date role-playing dimensions based on a single Date dimension, then I want my 'Calendar Date' hierarchy in Date to appear as 'Calendar Order Date' and 'Calendar Ship Date' to my end users. Not being able to do this has stopped me using role-playing dimensions on so many occasions simply because seeing multiple hierarchies with the same name is too confusing.
Support for cross-dimensional user hierarchies
Sometimes I find myself in the situation where I've got a large dimension, such as a Customer dimension, where the lower-level attributes are queried much less often than the higher level attributes. Now AS2005 handles large dimensions pretty well, in my experience, but it would still be useful to be able to store less useful lower-level attributes in ROLAP mode and more useful higher-level attributes in MOLAP mode. One solution to this could be the ability to set the storage mode on individual attributes rather than the whole dimension but I'm not sure that's practical; instead it would be great if you could split the attributes up into a ROLAP dimension and a MOLAP materialised reference dimension and be able to build a user hierarchy that spans both. So you'd be able to drill down from Country to City to Post Code to Customer, and when you were querying the first three levels you'd be hitting a MOLAP dimension and when you queried Customer you'd be hitting a different ROLAP dimension. In fact I'm sure the ability to set up cross-dimensional drill paths would be useful in other scenarios too and make cubes much easier to use.
Support for partitioned dimensions
Dimensions are getting bigger and bigger, and I wonder if at some stage it would be useful to be able to partition dimensions as well as measure groups to get better performance and manageability? Just a thought.
Be able to manage unary operators and custom member formulas from the MDX Script
Perhaps this is just me, but sometimes I find it's a bit of a pain when you're using unary operators and custom member formulas that their values have to be stored in the dimension table. Can't we have the option to be able to set or override a unary operator or custom member formula from the MDX Script - after all, that's meant to the one central repository for all my calculations. I wonder if this is possible anyway with an ALTER CUBE statement? Hmm, I don't think so.
New features for calculated members
As I've mentioned here before, quite often I see people creating real members in dimensions then overwriting their values with scoped assignments when I'm sure they would be better off with using calculated members. So to meet that and other challenges, here's a list of things it would be cool to allow with calculated members:
- Allow calculated members to have other calculated members as children
- Be able to control where calculated members appear in a hierarchy
- Allow calculated members to have member properties, either returning static values or the results of MDX expressions
- Following on from that, if calculated members can have calculated member property values, real members should have that too. For example, on a Customer dimension I might know a date of birth but I might also want to be able to calculate a customer's age and show that as a property
- Following on from that, it would be great to be able to create entire hierarchies based on calculated members or properties. So if I can calculate a customer's age, and then have another calculated property that placed that customer into a bucket based on their age (eg 0-9, 10-19, 20-29 and so on), I'd also like to be able to determine the number of buckets based on some MDX as well (maybe using server parameters - see below) and create a user hierarchy where the top level would consist of calculated members representing these buckets and where I could drill down from these buckets to the individual ages and then down to the real customer members themselves.
Server parameters for MDX calculations
The ability to parameterise an MDX query is cool, but it would also be great if you could also parameterise calculations on the server. For example you could set up a kind of server-wide variable which was the rate of Income Tax, do calculations using that value and then allow users to have the option of changing its value and explore 'what-if' scenarios. This would clearly have an impact on AS's ability to cache calculations but so long as people knew about this then I'm sure it would be ok. These parameters would also be visible through the metadata so client tools would know to show users they could change them.
Be able to define either whole queries or 'selectable chunks' of MDX on the server
After my posting on how dynamic sets actually work in AS2005 a few months back, I got to thinking about why I was disappointed and what I really wanted to see. The problem is that however well you model your cube, and however easy-to-use your client tool is, there's always a gap between what the tools are capable of and what the user is able to accomplish with them. For example, you might know that your end users want to be able to create reports showing the top 10 customers with an extra value underneath that represents the aggregated values of all other customers outside the top ten. This is certainly possible in MDX and perhaps there are client tools out there that support this, but wouldn't it be cool if you could create this as a kind of pre-canned selection that was parameterisable (eg show the top 20 instead of the top 10, or show products not customers) and visible through the metadata so any client tool could display it to the end user and allow it to be used? This would be a selection that would be designed to work wherever the user placed it in his or her query; it's not the same thing as a dynamic set, more of a parameterisable custom set function. The concept could be extended from pre-canned selections to entire queries too - and if I haven't convinced you (or been entirely successful in explaining) the concept of pre-canned selections then I think the case for entire queries stored on the server is much easier to make. Again these would be parameterisable and discoverable through metadata, so that any client tool would (hopefully) be able to use them; for Reporting Services alone this would be extremely useful, so you could have one query displayed in many reports with a single point of maintenance. I'm sure there would also be an opportunity to introduce some performance benefits for these stored queries too, perhaps in terms of being able to cache entire cellsets rather than just the values of individual cells.
Better management tools for security
As I mentioned before here, the tools we've got at the moment are not up to the task of managing large numbers of security roles and some serious work needs to be done in the UI. It would also be great if the tools supported dynamic security and the creation of all the supporting objects, perhaps through a wizard.
Better support for Analysis Services in SQL Management Studio
This is an easy one to specify: I want all of the functionality in MDX Studio put into SQL Management Studio. And a hundred other little things, like in the metadata pane I want to be able to open a level and expand a member to drill down as well instead of always having to start at the All Member of a hierarchy; oh, and I'd like to have a Reconnect button for when I've got a query and I've killed my session because of cube changes, instead of having to disconnect and reconnect. And the ability to display...
Query execution plans
A lot of people have been asking for this already; here's the Connect: https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=321161. This would be soooo useful in performance tuning.
Support for other RDBMSs
Analysis Services is useful not only as a database engine in its own right, but as a query/calculation layer on top of other database engines. OK no-one uses ROLAP on SQL Server because performance is rubbish compared to MOLAP, but for years people have been experimenting with using AS in ROLAP mode on top of Teradata (see here) and I'm sure that it could be used in a similar way on top of other data warehouse appliances or the new generation of COP databases like Vertica. You'd be selling it as a way of OLAP-ifying these databases, getting the incredible scalability and performance they (allegedly) offer in combination with the power of MDX for calculations and access to the data via Excel. Writing new cartridges for these databases should be pretty easy; I suppose the problem comes with trying to generate SQL optimised for each of these platforms.
Writeback with no allocation
MOLAP writeback in AS2008 certainly does improve performance, but the need always to allocate your values down to the granularity of the fact table is always going to lead to a performance hit. Wouldn't it be useful if you could write a value back to a cell without having to allocate it down always? You'd get instant writeback. I'm sure that would be useful in many cases, and in fact it would work in the same way as if you'd assigned a value to that cell in the MDX Script. I wrote some code in the Analysis Services Stored Procedure Project that did this, in fact, although it wasn't really anything more than a proof-of-concept exercise.

Anyway, this post has gone on quite long enough now, although I'm sure if I thought about it I could go on for even longer. I need to do some work! But what would you like to see in Analysis Services 2011? Answers in a comment, please...

ASCMD Beta

Sat, 19 Apr 2008 11:01:44 GMT

Dave Wickert of Microsoft has asked me to relay the following message about a new version of ascmd he's working on:

“In preparation of SQL Server 2008 Analysis Services RTM’ing, I am starting a small beta program for the next version of ‘ascmd’. This version will have two important new features:

1) Built and verified against Analysis Services 2008 – but also directly compatible with AS2K5 – thus I need both AS2K8 and AS2K5 participates.

2) New multi-user capabilities – the Microsoft SQLCAT team uses ascmd as the multi-user load facility for its Analysis Services benchmarks. Ever want to load up a few hundred users against a SSAS server? We are going to be retrofitting those capabilities (plus a few others) back into the released version of ascmd.

I am looking for users to test out this new version: 1) to, at least, slipstream this version into their current use to see if we’ve broken anything (we don’t think so, but we want to fix anything if we did), and 2) test out the new features if you can.

We are also actively soliciting new features and improvements for ascmd at this time – if we can sneak them in, we will.

The “readme” for the current version of ascmd is located here: http://msdn2.microsoft.com/en-us/library/ms365187.aspx

If you are an avid user of ascmd and would like to be included in this beta, please contact me directly (dwickert@microsoft.com).”

Panorama, Google and Analysis Services Support

Sat, 19 Apr 2008 06:35:14 GMT

New developments on the Panorama/Google front: users of Panorama's pivot table for Google docs will be able to connect direct to Analysis Services via http (if they want to pay) and also upload local cube files and query them (which will be free). Here's the announcement:
http://www.panorama.com/news/news/archives/2008/apr-17-2008.html

This comes hot on the heels of Google's Salesforce link-up:
http://blogs.salesforce.com/blogs/2008/04/announcing-sale.html

They're slowly joining the dots...

I was asked not to talk about this until the press release went out, but then I found that Nigel Pendse has already beaten me to it and done an excellent job of summing everything up:
http://www.olapreport.com/Comment_Panorama_Google.htm

I agree with Nigel that I don't think there will be many people at the moment who will be willing to expose their SSAS server via http. However there are a small number of companies who provide hosted SSAS solutions and this I think will become a much more popular option over the coming years. I do think, though, that the local cube option has some very interesting possibilities especially because it's free. I can imagine it would be a great way of distributing data to disconnected employees like salespeople, or to partners, for small companies on a tight budget. Tim Peterson must be pleased - this should cause a resurgence of interest in local cubes.

There will be a SQL2005 SP3!

Wed, 16 Apr 2008 14:45:42 GMT

A triumph for people power! Microsoft have announced that there will be a SP3 for SQL2005 and it will be delivered this calendar year. More details here:

http://blogs.technet.com/dataplatforminsider/archive/2008/04/15/sql-server-2005-sp3-coming-soon.aspx

There's also been yet another cumulative update released, CU7:

http://support.microsoft.com/kb/949095/en-us

UPDATE: there's a blog entry from the guys in PSS on how CU7 changes what happens when backups and processing overlap, here:
http://blogs.msdn.com/psssql/archive/2008/04/16/analysis-services-changes-introduced-in-cu-7-may-impact-backups.aspx

Monitoring examples from Richard Lees

Fri, 28 Mar 2008 12:15:27 GMT

Continuing the theme of monitoring AS and other applications, I've just come across some of the great live demos Richard Lees has on his site. For example, here are some details on a cube built from Perfmon data:

http://richardlees.com.au/Pages/PerfmonCube.aspx

...and you can see it in action, in a live PerformancePoint dashboard showing SQL and Analysis Services counters here:

http://richardlees.com.au/sites/Demonstrations/Shared%20Documents/SQL%20Server%20Database%20and%20OLAP%20Activity/SQL%20Server%20Database%20Activity%20and%20Health.aspx

There's also an Excel Services pivot table querying the same cube here:

http://richardlees.com.au/sites/Demonstrations/Pages/ExcelServices.aspx

SSAS Monitoring Sample App

Tue, 25 Mar 2008 07:16:43 GMT

After my moan the other week about the lack of monitoring tools for the Microsoft BI stack, Carl Rabeler of the SQLCat team contacted me with news of a whole load of new, related samples that he and his colleagues have been working on. Most relevant is a sample app for capturing SSAS trace information, storing it in SQL Server then using SSRS to report on it:

http://sqlcat.com/toolbox/archive/2008/03/24/sql-server-analysis-services-performance-monitoring-and-visualization-methodology.aspx

I know several people have already done something similar (a friend of mine, Hugo Lopes, told me he'd done the same thing only gone the exta step of putting the data into a cube) and hopefully now all this is in Codeplex the community can pul together and develop it into something more sophisticated.

There are also some sample Powershell scripts for querying SSAS008 DMVs:

http://sqlcat.com/toolbox/archive/2008/03/24/powershell-scripts-for-sql-server-2008-analysis-services-dmvs.aspx

Visualising Analysis Services Trace Information in Reporting Services

Tue, 18 Mar 2008 17:10:16 GMT

More fun with Reporting Services. The other week I was working with a customer who I suspected was experiencing query performance problems as a result of the issue described here:

http://blogs.msdn.com/sqlcat/archive/2007/10/19/ssas-small-big-query-interaction.aspx

Certainly in Profiler I could see some fairly trivial queries which were taking much longer than they should have done, and which when I ran them on a test system ran much faster. And certainly when I ran a 'big query' and a 'small query' simultaneously I saw the 'small query' run slow, and when I got the secret ini file settings from PSS my 'small queries' ran much faster. But there was still some doubt in my mind over the extent to which the customer was suffering this problem - just by looking at my Profiler traces from the Production servers I couldn't tell whether lots of small queries were overlapping with the big queries.

What I then thought was that it would be useful to be able to visualise the information from a Profiler trace in some way. I did some experiments with different chart types and different tools, mainly Excel 2007, but in the end I went with Reporting Services 2005 and the Dundas Gantt chart control. I'm not sure whether the Gantt chart has made it into the current RS2008 CTP but I guess it will soon if it hasn't; in any case the data was coming from a 2005 system so that's what I used.

The first step was to get the data out of Profiler into some useful format; the easiest way turned out to be to get Profiler to output directly to a SQL Server table and for the problem here all I needed was the Query End events, and specifically the Duration, SPID, TextData, ConnectionID, EndTime, EventClass, EventSubClass, and NTCanonicalUserName columns. I could now run my Profiler trace against the Production system and capture some data. The next problem was to manipulate the data into a useable format. Gantt charts need a start time and a duration for each event and the problem with trace data is that you only know how long something takes when it's finished, ie when you have an 'End' event. Subtracting the EndTime from the duration gives the derived start time of the event, which is one way of dealing with the problem of working out when queries started and finished. There are Query Begin and Query End events and perhaps another, better, solution would have relied on the knowledge that you can only have one query executing on any given session at any one time and matching, so it should be possible to match each Query End event to a Query Begin event. In fact I started off more ambitiously by including other types of events, such a Progress Reports, but the problem here I found was that the StartTimes and EndTimes given by Profiler were rounded to the nearest second which meant that subtracting the Duration from the EndTime didn't give accurate derived start times and I was finding events were coming out in what was clearly the wrong order when I tried to order them by derived Start Time (some of the SQL I'll give below is more complex than necessary for just the Query End event because it was written for these other event types too). It would be nice to visualise the various internal events that are generated when you run an individual query, such as reads from partition and cache, but I need to do some more work before that's possible.

Anyway, with my Query End events I wrote the following SQL query to use in Reporting Services which finds the derived start time for each query, then makes the first query to start at time 0 and calculates all other derived start times relative to that first query. As always I need to apologise for my SQL in advance:

with RelativeStartTimes (OriginalRowNumber, RelativeStartTime, EventClass, DurationMSecs
, TextData, ConnectionID, EventSubClass, NTCanonicalUserName)
as
(
select
a.Rownumber,
--Find the Relative Start Time of each event
--by finding the difference in seconds between the End Time
--of the event and the Earliest End Time in ms
--then subtracting the Duration
(datediff(ss,
c.EarliestEnd,
isnull(a.EndTime, a.StartTime)
) * 1000) - isnull(a.Duration, 0) as RelativeStartTime,
a.EventClass,
isnull(a.duration, 0) as DurationMSecs,
a.TextData,
a.ConnectionID,
a.EventSubclass,
a.NTCanonicalUserName
from dbo.[102QE] a cross join
--Get the earliest End Time in the Trace
(select min(isnull(b.EndTime, b.StartTime)) as EarliestEnd
from dbo.[102QE] b
where b.Rownumber>0
) c
where a.Rownumber>0
)

select
row_number() over(order by (r.RelativeStartTime - x.EarliestRelStartTime) asc) as row,
r.OriginalRowNumber,
r.RelativeStartTime,
x.EarliestRelStartTime,
r.RelativeStartTime - x.EarliestRelStartTime as BeginTime,
r.RelativeStartTime - x.EarliestRelStartTime + r.DurationMSecs as EndTime,
r.EventClass,
r.DurationMSecs,
r.TextData,
r.ConnectionID,
r.EventSubclass,
r.NTCanonicalUserName
from RelativeStartTimes r
cross join
(--Find the Earliest Relative Start Time
select
min(RelativeStartTimes.RelativeStartTime) as EarliestRelStartTime from
RelativeStartTimes) x

I could then use this as the basis for my Reporting Services report. As always with these things I spent most of my time working out how to use the Gantt chart but when I did, and after I'd done some other funky stuff that allowed me to filter by row number and start/end time, and click on an item in the Gantt chart to view the MDX for the query that a particular line represented, I got a report looking like this:

Another problem to mention here is that, as with this report, when you've got a lot of queries to look at the Gantt chart takes a loooooong time to render. Anyway, from this initial view you can see that almost all queries executed very quickly but fairly early on there was a very long query, represented by the longish brown line at around event 250 that I've circled. Filtering the view down to show this query and those around it in more detail gets you this:

Although the vast majority of queries here execute immediately (they're just small queries the front end uses to retrieve metadata etc and are represented as single blobs) I've circled a few queries around the main query that look like they are taking a few seconds to run. And indeed, running at least one of these queries on the test server showed it runs faster on its own than it did when this Profiler trace was running. Which proved my point.

Moving away from this particular exercise, I'm fairly surprised that none of the big SQL Server tool vendors have tackled the problem of monitoring Analysis Services performance and activity. The only tool I know of that does this is Companion for Analysis Services:

http://www.sqlminds.com/Products/CompanionforMSAnalysisServer/tabid/119/Default.aspx

Where are the others though? I would have though Analysis Services would be the obvious next step for them, given that many SQL Server relational shops also use the MS BI stack too. Whenever I talk to enterprise customers they always ask me what tools are available, certainly... I guess there's three things going on here: first of all the all-too-common ignorance of Analysis Services in the wider SQL Server community; second the problem that no-one (not even the SSAS experts) has really sat down and thought about what needs to be monitored in an Analysis Services deployment; and thirdly, at least before AS2008, the hooks for getting at data that can be used for monitoring either haven't been available or have been well hidden. This and my last post on viewing and killing queries in Reporting Services have given a few ideas on what's possible, though, and hopefully within the next few years Microsoft and/or third party vendors will give us the tools we need.

Killing Queries From Reporting Services

Fri, 14 Mar 2008 23:39:41 GMT

I guess anyone who has had to demo the new DMVs in Analysis Services 2008 has, like me, put together a few Reporting Services reports to show off what they can do. Now I don't have anything much to add about the content of the DMVs and schema rowsets beyond what Darren and Vidas have already blogged, but I did discover an interesting trick while I was creating my reports that I thought would be good to share.

One of my example reports was displaying the results of the following DMV query:

select * from $system.discover_commands

What this does is display information on currently executing command or the last command that executed on every open session on the server (see here for more details). Pretty useful information in that if someone is running the query from hell on your AS server you'll be able to see it here; what you'd really want to do though is act on this information and be able to kill the query. You can certainly get the same information and can kill the query if you're using the Activity Viewer sample app that comes with the SQL Server samples (Jesse Orosz blogged about its features and shortcomings here); but can you do the same thing from within Reporting Services without writing any code? It turns out you can.

Here's what my report looks like in BIDS 2008:

So far, so straightforward. Notice the column on the far right that contains the text "Kill This": this is static text and is a link to another report which takes one parameter, the SPID of the session that represents the current row. When you click on this link you jump to another report and it's the act of running this report that kills the query on the session whose SPID you pass in. This second report looks like this in BIDS:

You'll see that the design surface of the report contains a single text box with the message that your query has been killed successfully. The killing is done by the query in the sole dataset, and the query is generated by a Reporting Services expression that looks like this:

="<Cancel xmlns=""http://schemas.microsoft.com/analysisservices/2003/engine"">
<SPID>" + Parameters!SPID.Value.ToString() + "</SPID>
</Cancel>"

It's an XMLA Cancel command with the SPID injected into it from the report parameter mentioned above. Two things to know to make it work:

You have to execute it on an OLEDB-type connection, ie not Reporting Services' "Analysis Services" connection
The command doesn't return any data and RS expects all its datasets to return at least one field. To work around this, you need to create a new dummy calculated field on the dataset that returns any value you like (you may see some error messages along the way but they can safely be ignored):

And bingo, you've got a basic activity monitor implemented in RS2008! I haven't tested it, but you should be able to do exactly the same thing in RS2005 if you use the functionality in the XMLADiscover class Darren put into the Analysis Services Stored Procedure Project; the reason I like the approach I've just described more than this option is that whenever I've wanted to use some ASSP functionality in a production environment the customer has, for obvious reasons, not been comfortable uploading a dll onto their server.

Analysis Services and Trimming

Mon, 03 Mar 2008 19:52:07 GMT

Interesting post here on the PSS SQL Server Engineers blog on SQL Server working set trim problems:
http://blogs.msdn.com/psssql/archive/2008/03/03/sql-server-working-set-trim-problems-consider.aspx

There's also a related KB article here:
http://support.microsoft.com/default.aspx/kb/938486
And a representative blog entry on the subject here:
http://weblogs.asp.net/omarzabir/archive/2007/10/19/a-significant-part-of-sql-server-process-memory-has-been-paged-out-this-may-result-in-performance-degradation.aspx

I'm working with a customer where PSS think this is happening with Analysis Services. Certainly Perfmon shows some massive drops in memory usage for no obvious reason (ie the system is nowhere near the LowMemoryLimit). Something to look out for... I'll post more details if/when I get them.

Using Non_Empty_Behavior With YTD calculations

Wed, 27 Feb 2008 18:30:47 GMT

I always believe in giving credit where credit's due, and before I go on with this blog entry I have to declare that this isn't my idea. I heard about the general approach from Eric Jacobsen (or possibly Richard Tkachuk) at the BI Conference last year and he said it originally came from Thomas Kejser who is now on the SQLCat team; I'll be working with him on a project fairly soon so I'll ask him about it! I've not actually seen it implemented or written about anywhere though, and since I've only just got around to testing it out and seen how useful it can be I thought I'd blog about it.

Let's start by taking a look at the following Adventure Works query:

with member measures.ytdsales as
sum(
periodstodate([Date].[Calendar].[Calendar Year],[Date].[Calendar].currentmember)
, [Measures].[Internet Sales Amount])

select {[Date].[Calendar].[Month].&[2003]&[12].children} on 0,
non empty
[Customer].[Customer Geography].[Customer].members
*
[Promotion].[Promotion].[Promotion].members
on 1
from [adventure works]
where(measures.ytdsales, [Product].[Subcategory].&[2])

It runs in 1 minute 10 seconds on my laptop and returns 33 columns and 2585 rows. YTD calculations are now very efficient in AS2005 SP2 and as far as I can see the real cause of the problem is the extremely large crossjoin on rows and the associated non empty. Relatively few customers actually bought anything in 2003 in this particular subcategory and when they did it was usually associated with no discount, so there's a lot of empty tuples that we are going to need to remove on rows; but in order to be able to filter them out, at the moment, we have to calculate the ytd sales for every single one of them. Now if we had control over the MDX of this query we could probably do some clever stuff with the NonEmpty function that would improve performance a lot, but what if users are querying with an ad hoc tool like Excel or Proclarity? It would be nice if we could set the Non_Empty_Behavior property on this calculated member, but that involves having a real measure on the cube whose value is null when ytd sales is null, and we don't have one of those...

So why don't we build one? In fact it's much easier to do this than you think especially when you remember that we're not interested in the value of this measure at all, just whether it's null or not. What we need to do is take our existing fact table, do a group by to find the minimum date for every key combination and then generate a row in the new fact table from that date to the end of the year. So to take a simplified example if you had a fact table with two dimensions, Product and Time, and the granularity of Time was Month, you would start with this:

Product	Time	Sales
Apples	October	100
Oranges	November	150

and want to get to this:

Product	Time
Apples	October
Apples	November
Apples	December
Oranges	November
Oranges	December

Here's a SQL query which does just that for the FactInternetSales fact table in Adventure Works DW:

select
f.ProductKey, f.DueDateKey, f.ShipDateKey,
f.CustomerKey, f.PromotionKey, f.CurrencyKey, f.SalesTerritoryKey,
c.TimeKey, f.SalesOrderNumber, f.SalesOrderLineNumber
from
(select i.ProductKey, i.DueDateKey, i.ShipDateKey,
i.CustomerKey, i.PromotionKey, i.CurrencyKey, i.SalesTerritoryKey
, i.SalesOrderNumber, i.SalesOrderLineNumber
, min(i.OrderDateKey) as mindate, t.CalendarYear
from
dbo.FactInternetSales i inner join dbo.DimTime t
on i.orderdatekey = t.timekey
--where SalesOrderNumber='SO43701'
group by t.CalendarYear, i.ProductKey, i.DueDateKey, i.ShipDateKey,
i.CustomerKey, i.PromotionKey, i.CurrencyKey, i.SalesTerritoryKey, i.SalesOrderNumber, i.SalesOrderLineNumber) f
inner join
DimTime c on f.CalendarYear = c.CalendarYear
and f.mindate<=c.TimeKey

You'll have to excuse my SQL - it's pretty ropey - but even though I'm sure this could be optimised it still runs very quickly; if you uncomment that line with the where clause in the middle, so the table is filtered down to one SalesOrderNumber, you'll see what it's doing more easily. Now you'll probably be worried, quite rightly, that doing this creates a fact table that's much larger than your original fact table and that's certainly true. In this case FactInternetSales is just over 60000 rows and the query above returns almost 7.5 million rows; however depending on the granularity of your fact table and the nature of your data the explosion could be a lot less, and we're not interested in any of the measure columns in the original fact table either so this new table will not be as wide.

The next step is to take the query above and use it as the basis of a named query in the Adventure Works dsv, and then build a new measure group in the Adventure Works cube from it containing a single Count measure. Partitioning and aggregation might be useful here too, but I didn't bother with them and just went ahead and processed; I was pleasantly surprised at how quickly it processed, I guess because of the lack of measures.

Anyway, now we've got a measure in our cube we can use for our Non_Empty_Behavior property. I called it [Fact NEB Count] and here's the new version of the calculation in the original query:

with member measures.ytdsales as
sum(
periodstodate([Date].[Calendar].[Calendar Year],[Date].[Calendar].currentmember)
, [Measures].[Internet Sales Amount])
, non_empty_behavior=[Measures].[Fact NEB Count]
select {[Date].[Calendar].[Month].&[2003]&[12].children} on 0,
non empty
[Customer].[Customer Geography].[Customer].members
*
[Promotion].[Promotion].[Promotion].members
on 1
from [adventure works]
where(measures.ytdsales, [Product].[Subcategory].&[2])

The query now runs in, wait for it, in 7 seconds on a cold cache in SQLMS and if you look in Profiler you'll see that in fact the majority of that time is taken up by SQLMS rendering the resultset - it takes just over 2.5 seconds to actually run on the server. A lot of effort, certainly, but a big improvement. Of course I've cheated a little bit and used a query that shows a particularly large change - the larger percentage that rows with data make up of the overall number of rows in the crossjoin, the less the difference will be. That said, I'm working with a customer at the moment for whom this is a very valid scenario: they have a lot of customers and a lot of products, but a customer only tends to buy one or two products and the users love to run big queries with YTD measures.

CTP6 and CU6 are now available

Wed, 20 Feb 2008 16:35:27 GMT

The February CTP of SQL2008, aka CTP6 is now available for download here:
http://www.microsoft.com/downloads/details.aspx?familyid=749bd760-f404-4d45-9ac0-d7f1b3ed1053&displaylang=en&tm

Also now available, although not something you can download freely, is cumulative update 6 of SQL 2005 SP2:
http://support.microsoft.com/kb/946608/en-us

Several of those bugs there are ones I found (chest swells up with pride)... Meanwhile, in the SQL Server community, the calls for Microsoft to reverse their decision to not release a SP3 are getting louder. As I've said before, Analysis Services SP2 contained way too many bugs in my opinion and given that many shops either don't know about CU releases, can't get hold of them or won't install anything other than an official service pack the fact that these bugs have since been fixed is irrelevant until the fixes get rolled up into an official service pack. It just sends a message to the customer that Microsoft is more concerned with shipping a profitable new release than making the product that everyone's actually using at the moment (and most people will be using for a long time to come) more reliable.

Dimension Security Tips, Tricks and Problems

Tue, 22 Jan 2008 17:07:22 GMT

I've been doing some work with complex dimension security recently and come across a number of issues which I've either known about but never blogged about, or only just become aware of. It's a bit of a random list but hopefully it'll help someone out there...

First of all, how do you use dimension security to deny access to an entire dimension? It was one of those things that I thought was possible out of the box even back in the days of AS2K but it turned out my memory was playing tricks with me and it's a lot harder to do than you might think. Dimension security lets you deny access to individual members, so to deny access to the entire dimension you have to deny access to all the members on every single attribute hierarchy on the dimension - which is a right pain in the **** to do. From reading the relevant pages in the book "Microsoft Analysis Services 2005" (which is very good on the subject of security) I had thought it would be enough to set the Allowed Set on the key attribute to be an empty set, ie {}, and indeed this is what I'd been told would work. However this only has the effect of denying access to the key attribute itself and as I said you need to set the Allowed Set to {} for every attribute. Furthermore, once you've done that you'll still see the dimension itself visible even though all you'll see on each attribute hierarchy is an All Member, and there isn't a really a good way of hiding it apart from creating a perspective and trying to get all users who are denied access to the dimension to connect using that perspective. Not very nice. I posted something on Connect here and even though it's been closed as Won't Fix I had an email conversation with various people in Redmond a while ago and I'm hopeful there'll be a good solution to this one day.

Then there's the problem of calculations and security. If you read Mosha's blog entry here on the subject of cube initialisation you'll see that the MDX Script is executed after dimension security is applied - so if you reference a member in your MDX Script that has been secured, then it could result in an error. For example, imagine you have the following calculated measure defined on Adventure Works:

Create Member Currentcube.Measures.Test as Measures.[Internet Sales Amount];

If you create a role that denies access to [Internet Sales Amount] and then browse the cube, you'll still see this calculated measure but it will return an error. The stock answer to this problem is that you should set the cube's ScriptErrorHandlingMode property from IgnoreNone to IgnoreAll, so that when the MDX Script executes when someone connects through the role the calculated measure doesn't get created because there's an error in its definition. I'm not too keen on this solution, and nor is anyone else I've talked to who has faced this problem, because what happens when there's a real bug in your MDX Script? You run the risk of calculations you actually want being ignored and no error being displayed, and in the worse case scenario this could result in incorrect data being returned. Imagine you've got a script assignment like this on Adventure Works:

Scope({Measures.[Internet Sales Amount],Measures.[Internet Tax Amount]});
This=Measures.Currentmember * 2;
End Scope;

If you connect to the cube through a role where [Internet Sales Amount] is denied and if ScriptErrorHandlingMode is set to IgnoreNone, you'll see a "END SCOPE statement does not match the opening SCOPE statement" error. Set ScriptErrorHandlingMode to IgnoreAll though, and you don't get an error because the Scope block is ignored - but the value of [Internet Tax Amount] will be half the value that is displayed when someone connects through a role which has access to [Internet Sales Amount].

The best solution to this problem is one I found in this thread on the MSDN forum suggested by Deepak and Mosha. While it doesn't seem to be possible to conditionally hide calculated members (again using a perspective seems to be the best option although still not ideal) it is possible to make sure they don't return errors, and similarly it is possible to write scoped assignments that can still work if one or more members they're scoped on is secured. The key is to use named sets in scoped assignments, as follows:

Create Member Currentcube.Measures.Test as null;

Create Set myset1 As
iif(iserror(strtomember("Measures.[Internet Sales Amount]")), {},
{Measures.Test});

Scope(myset1);
This=Measures.[Internet Sales Amount];
End scope;

Create Set myset2 As
iif(iserror(strtomember("Measures.[Internet Sales Amount]")), {Measures.[Internet Tax Amount]},
{Measures.[Internet Sales Amount],Measures.[Internet Tax Amount]});

Scope(myset2);
This=Measures.Currentmember * 2;
End Scope;

For both the calculated measure and the scoped assignment, a named set is created that uses the IsError function to check to see whether [Internet Sales Amount] is secured or not - if it is, then an attempt to use the StrToMember function with its unique name will return an error. The outcome of this then determines the contents of a named set; for the calculated member this named set is used with a scoped assignment that assigns a definition to the member, and in the second example the named set is used instead of the hard-coded set in the Scope statement. This approach is a bit fiddly but it will work even when ScriptErrorHandlingMode is set to IgnoreNone because it avoids an error being raised.

Another thing to note is that if you are using role-playing dimensions in your cube, you will need to set dimension security on the cube dimensions rather than the underlying database dimension. For example, in Adventure Works, if you go to the Date dimension in the list of database dimensions (in the dimensions drop down on the Dimension Data tab of the Roles editor, database dimensions are listed first, then cube dimensions second) and set dimension security say to allow access to one year in the Calendar Year attribute, you'll get an error like "The 'Calendar Year' attribute on the 'Ship Date' dimension has a generated dimension security expression that is not valid". Setting the same permissions on each of the role-playing dimensions in the cube based on Date, ie Ship Date, Delivery Date and Date does not cause a problem.

Lastly, what I will say after spending a week or so playing around with dimension security is that even in SP2 it is way more buggy that it should be. I've had no end of problems setting up the roles I want: usually everything works for a while and then suddenly the roles stop accepting any changes I make to them, ie you'll make a change, save it, then test it and find that the change you've made has been forgotten about. I suspect something is getting corrupt somewhere. I've not managed to get a clear repro so that I can open a case with Microsoft about this, unfortunately, but it's something to do with large dimensions and/or membership of multiple roles and/or visualtotals and/or securing measures and it's happening enough to force us to rethink our entire approach to security. Hohum.

ASSP 1.2

Sun, 23 Dec 2007 22:32:04 GMT

Darren Gosbell has just done a new release of the Analysis Services Stored Procedure Project; he's blogged about it here:
http://geekswithblogs.net/darrengosbell/archive/2007/12/22/ssas-assp-1.2-released.aspx

And of course, you can download it from Codeplex here:

http://www.codeplex.com/Release/ProjectReleases.aspx?ProjectName=ASStoredProcedures&ReleaseId=7228

It includes the partition healthcheck code that I blogged about here, as well as various other cool stuff.

M2M Dimension Optimisation Techniques Paper

Fri, 21 Dec 2007 23:43:45 GMT

Another new paper from the SQLCat team, this time on techniques that can be used to optimise cubes that use many-to-many relationships:

http://www.microsoft.com/downloads/details.aspx?FamilyID=3494E712-C90B-4A4E-AD45-01009C15C665&displaylang=en

I've only skimmed through it so far, but I thought I'd mention two other tricks to try with m2m dimensions. One is breaking up large dimensions if you don't ever need to see them, which I blogged about in detail here: http://cwebbbi.spaces.live.com/Blog/cns!7B84B0F2C239489A!777.entry, and which can be very powerful. The other is introducing other dimensions to the intermediate measure group: for example, imagine you have a measure group showing the relationship between Products and the Components that make them up, you could try adding the Time dimension to that measure group; even though it will increase the size of the measure group because you are repeating the Product/Component combinations for each Time period, if all of your queries are sliced by a Time period and you know that the majority of your Products were only sold for a short space of time then it can be beneficial, although if neither of these conditions is true then doing this can have a negative impact on performance too.

Comparing MDX Query Results With Tablediff

Mon, 10 Dec 2007 23:18:16 GMT

Recently I was working on a project where the customer wanted to upgrade from SP1 to SP2 but found that when they did so the results of some of the calculations on their cube changed. Unfortunately these changes weren't easy to spot: their MDX Script was >1500 lines long and the bugs they were running into were dependent on the query being run and whether the cache was cold or warm (in fact one of them was the bug I blogged about here). When testing various workarounds and hotfix builds, the problem I ran into was a problem I've run into before on numerous occasions: when I changed something on the MDX Script, how did I know it didn't break something else?

In the past what I've done was take a few MDX queries, run them, copy and paste the results into Excel, make my changes and then rerun the same queries and paste the new results into Excel again and use Excel formulas to compare the old and new resultsets. However in this case this approach just wasn't feasible and I started looking for a better way of comparing large numbers of MDX resultsets. Now there is definitely a tool to be written here, probably one which runs batches of queries, serialises the cellsets and saves them to disk, and then compares the cellsets and highlights any cells with differences, but I didn't have time to write it. I was talking the problem over with Didier Simon of PSS, though, and he made a great suggestion - use tablediff.exe, the command line utility you get with SQL Server to compare relational tables. You can read more about it here:
http://msdn2.microsoft.com/en-us/library/ms162843.aspx

This turned out to be the key. I cannibalised bits of my Analysis Services cache warmer and created two SSIS packages, the first of which ran a batch of MDX queries retrieved from a relational table (originally saved from a Profiler trace) through SQL Server using OpenRowset and saving the results in a table using a SELECT INTO. Here's what the control flow looked like:

And here's an example of the kind of SQL query I was generating:

select identity(int, 1,1) as idcolumn, * into Query1
from openrowset('MSOLAP', 'Datasource=localhost; Initial Catalog=Adventure Works DW',
' with cell calculation test for ''(*)'' as ''cstr(round(measures.currentmember,5))''
select Measures.[Internet Sales Amount] on 0,
[Date].[Calendar Year].Members on 1
from [Adventure Works]')

A couple of things to note - Tablediff needs a column which can uniquely identify each row in the tables its comparing, hence the identity column. I also found that I was running into the precision problems described here and as a result, tablediff was flagging up cells as having different values when in fact the values were differing only by a miniscule amount, so I created a calculated cell assignment in the WITH clause which rounded up every value to five decimal places. I could do this because I knew none of my queries had WITH clauses, but another solution would have been to create a session calculated cell assignment which did the same thing.

After running this package I had a SQL Server database with as many tables containing results as I had queries. I could then make my changes to the cube and rerun the same package to dump a new set of results to a different database, and then run my second package to compare the results. This was, again, a fairly simple package to put together:

All I did was use a ForEach loop to loop over the tables in the database containing the first set of results, and then use an Execute Process task to call tablediff to compare each table with the equivalent table in the second database; if it found any, I used the -et argument to save the output to another table. I also used the -c argument to get column-level differences and the -b argument to specify the number of bytes to compare for large objects, necessary because the SELECT INTO creates an ntext column for all the member names on the Rows axis.

Overall, it's not an ideal solution (coding a proper app is definitely the way to go) but it did the job, and hopefully this will be useful to someone else...

Partitions and Aggregations

Mon, 03 Dec 2007 23:32:23 GMT

Something that's easy to forget (at least I'm always forgetting it) when creating an aggregation design is the impact that partitioning can have on it. Aggregations are only created on a per-partition basis - you can't aggregate across partitions - so that means, for example, when you're partitioning by Month then there's little point building aggregations at granularities higher than Month; after all, if you only have one Month of data in your partition and you aggregate that data to the Month, Quarter or Year levels then the figures for each aggregation will be identical! In fact building aggregations above the granularity of Month can be a bit of a waste: an aggregation at Month granularity can be used by queries at Quarter and Year, but an aggregation for the same partition at Year granularity that is in all other respects identical can only be used by queries at the Year granularity.

The best thing to do to ensure this doesn't happen is to set the AggregationUsage property to None for all attributes above the granularity that you're partitioning by, and also to make sure your partition row counts are set correctly, before you run either of the Aggregation Design wizards. This should ensure that your aggregation design properly reflects your partitioning strategy.

One of the questions I've always meant to research further on this topic is whether partitioning could hurt query performance, given that it effectively prevents certain aggregations being built. Taking the example we've been using so far, what happens if you're partitioning by Month and you've got a slow query at Year granularity that you'd like to build an aggregation for - would partitioning by Year rather than Month, so you could build a true Year-level aggregation, make sense? Well, I've just had a chance to test this out on a fairly large cube (with approx 100 million rows in the main fact table, and Product and Customer dimensions with 1 million+ members on their key attributes) and interestingly partitioning seems to make no difference at all. I created two measure groups, one partitioned by Month and one partitioned by Year, and then created one aggregation on the former at Month/Product and one aggregation on the latter at Year/Product. I then ran a query that returned the top 10 Products at Year on each measure group and the performance was identical. Clearly this is not something to be worried about, thank goodness...

[Thanks are due to Eric Jacobsen of the SQLCat team for talking through some of these issues after the BI Conference earlier this year. He accurately predicted that partitioning wouldn't hurt performance because the AS query engine is very efficient at merging the data caches of partitions.]

SQL2008 November CTP released

Tue, 20 Nov 2007 19:14:40 GMT

I'm a bit late on this (ie about a day, but that's late in the blog world) but the November CTP of SQL2008 has been released and you can download it here:

http://www.microsoft.com/downloads/details.aspx?FamilyId=3BF4C5CA-B905-4EBC-8901-1D4C1D1DA884&displaylang=en

I won't go into the details of what it contains because various other people have already done so; the best round up I've seen is from Vidas Matelis:

http://www.ssas-info.com/VidasMatelisBlog/38_ms-sql-server-analysis-services-2008-november-ctp-quick-review#more-38

Apparently the only feature missing from this CTP for AS is the ability to share a single AS database between multiple instances; there don't seem to be any new features coming that I didn't know about apart from some new Perfmon counters and some new Profiler trace events, and the rumoured query plans have not made the cut. Am I alone in feeling a little... underwhelmed by the new features in AS2008? Maybe that's because I haven't had a chance to test out the block computation changes yet (I might do this week though) - they're not very obvious or demoable but they should hopefully make a big, big improvement to the performance of many calculations.

Viewing Partition Slice Information

Thu, 08 Nov 2007 22:10:46 GMT

The other week I discovered that it was possible to view the partition slice information discussed here using the DISCOVER_PARTITION_DIMENSION_STAT schema rowset. I've always thought it would be cool to be able to visualise this information so you could easily check to see whether your dataid ranges overlapped or not; to do this, though, you would need to see the slice information for all your partitions in one resultset and DISCOVER_PARTITION_DIMENSION_STAT only returns data for one partition. So this week I dusted off my coding skills and wrote a sproc that loops through every partition in a measure group, calls DISCOVER_PARTITION_DIMENSION_STAT and concatenates the results as well as checking to see which partitions have ranges which overlap each other. I've added it to the source code for the Analysis Services Stored Procedure Project, which you can get hold of here:

http://www.codeplex.com/ASStoredProcedures

Note that the function isn't part of the latest release, you will have to download the code and compile it yourself. As an example of what it does, in SQLMS if you connect to the Adventure Works database and then run the following statement:

call assp.DiscoverPartitionSlices("Adventure Works", "Internet Sales")

...you'll get a resultset showing all the partition slice information for every attribute on every partition in the Internet Sales measure group.

Having done this I was able to put together a simple SSRS report to display this information. I say simple, it probably took me longer to create the report than to write the sproc because I couldn't work out how to visualise the ranges in a useful way; in the end I used the RangeColumn chart style from Dundas, which seemed to be the best fit. Here's an example of what the report returned for the Date attribute on the Date dimension and the Internet Sales measure group:

I cheated a bit by setting the IndexBuildTheshold very low so that indexes would be created for the 2001 and 2002 partitions, but it shows what a healthy set of slices looks like (the Internet Sales measure group is partitioned by Calendar Year). Another attribute worth looking at is the Product attribute on the Product dimension:

What we see here is that in 2001 and 2002 only a small number of products were actually sold compared to 2003 and 2004. In this situation it might be worth trying to reorder the members or set a slice on 2001 and 2002 so that AS knows about this distribution: it would mean that if you were running a query for the sales of a Product that was only sold in 2003/4 for all time periods, AS would not need to scan the 2001 and 2002 partitions because it would know in advance that there would be no data for that product in those partitions. One of the improvements I would like to make to the sproc is to display member names and unique names as well as dataids so that the slice ranges are easier to understand; Mosha also had the idea that it would be good to be able to take these unique names and use them to set a slice on a partition for exactly the scenario I've just described.

Lastly, I found what is almost certainly a bug in AS when I was looking at the Calendar Semester attribute on the Date dimension:

Calendar Semester has, as you would expect, a one-to-many relationship with the Year attribute and we would expect to see no overlaps as a result -but as the report shows, that isn't the case. When you run a query on this attribute, for example:

select measures.[internet sales amount] on 0,
{[Date].[Calendar Semester].&[2004]&[1]} on 1
from [adventure works]

You see partitions being hit when they shouldn't be hit - in this case the 2003 and 2004 partitions - and this even sometimes seems to happen when you have a slice explicitly set on the partition using the Year attribute, which is pretty bad. Greg Galloway did a bit of digging on this and discovered that the SQL generated returned rows in the same mixed-up order as the dataids of the members, and fixing the SQL to return members in the order you want the dataids stops this happening to a certain extent. Mosha says that it's a problem with the decode tables created within the dimensions, but whatever the cause you may want to check your dimensions to see if it's happening to you. This kind of problem has been noticed before and I knew about the Disable Prefetch Facts connection string property that was introduced to deal with it, so I wonder if this is related in some way? Anyway, I logged the bug on Connect and I'll blog about any new information that I get on it:

https://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=308793