This is in fact a double double post. Double because it is composed of 2 parts and double because I am writing about two things today:

1 – Open Live Writer (OLW) – The open source version of Windows Live Writer – can be downloaded from there

2 – Cozyroc great SSIS components who had just saved my bacon (speaking of which I could use a BLT sandwich now that I am seating at 30000 ft above the north sea on my way to SQL Saturday in Ljubljana). Cozyroc SSIS+ components can be downloaded from this page

Along the way I will also comment the writing experience from using OLW.

The use case

A customer of mine needed help to determine the best way to do data alerts in the Microsoft BI stack. What they want to achieve is:

– have the Business defining alert rules without IT interference

– this rules will always return : a report name/url , an email address and the name of the email recipient.

– the data to get is either located in a cube or in a relational database

What they have at their disposal:

– SharePoint 2013 Enterprise and Reporting Services in SharePoint mode

– SSAS 2012 Enterprise

– SQL Server 2012 Enterprise edition

– the ability to copy paste some MDX or SQL queries that generates the alert rules.

– Reports built in Excel with pivot tables

For example once they have more than 10 unpaid bills for a given customer they to alert the Key Account Manager for that customer with a link to a report in an email.

My first idea was to use Reporting Services data alerts. Alas it was not possible for 2 reasons:

1 – data alerts require a data feed from a report, which in turn will require them to build SSRS reports for every of their Excel reports.

2 – data alerts don’t work against cubes, only against SQL Server databases. (which I learned the hard way after recommending that solution to the customer)

The solution

• 2 SharePoint lists:
  • One for alert definitions (MDX/SQL query, column name for email and column name name, report name and URL and a connection string for the query)
  • One for saving the result of the query
  • A workflow in SharePoint sending emails to the right recipient for every new item in the list
• a SSIS package reading from the SharePoint alert definition list, executing the queries and saving the result into the second list.

The problem

The whole point about building SSIS packages is to get the metadata of the source for your query before you can run the query. When the query source and text changes the metadata changes and the metadata needs to be recompiled. This cannot be done at runtime. The only way to do it was to build one data flow per possible query source, which is not terrbly dynamic and will require IT intervention each time Business build a new report. Not a great solution.

This is where I remembered Cozyroc great set of SSIS components. And particularly two of them:

– Dynamic Data flow

– SharePoint List Destination (and Source)

So armed with a NFR license of Cozyroc component I went on building the solution. I will start by saying that I’m in no way paid or endorsed by Cozyroc for writing this post. And the following are my findings goods and bads.

The use of the component is not very well documented on the internet. There are some videos by Jamie Thompson and some people from Cozyroc out there, but none covering my exact situation. Luckily the developers at Cozyroc are really ready to give a hand. So I got in touch with Ivan Peev who helped me a lot with the errors I was encountering and the way to solve them. Thanks for your patience Ivan !

The solution (again)

After installing cozyroc SSIS components (which was a painless experience) chosing the SSIS version to install for I created a new Integration Services Solution and had immediately access to all of Cozyroc SSIS components.

Figure 1 – Some of the tasks available

Figure 2 – the overall package

First thing first, I need a data flow task to get my data for the alerts out of SharePoint. My original idea was to have all the alert definition items into recordset to be able to loop over them, but unfortunately the components didn’t allow me to save the recordset to a variable. So I had to create a table for storing the data. This is this table I start by truncating in the Clean up Execute SQL task.

The Sharepoint Data flow task is straightforward. (no auto-correction in Open Live Writer as it seems, nice!). I have a SharePoint Source pointing at my SharePoint website (via a Cozyroc SharePoint connection) and from the component I pick

1. the list to get data from
2. the CAML query to pass to the SharePoint list – great so I can filter columns and rows.
3. the view to use when querying the list.

Figure 3 – Component properties page for the SharePoint source

As expected the columns I get back a list of columns and have to pick the ones I want to go on with.

I could have written a more specific CAML query to restrict the columns returned but due to the fact that it was a POC with only 10 hours for presenting a working prototype I chose not to. Lazy me !

So here I am with the columns I need for moving on. The next step is to create a SQL Server table to hold all these columns. Here is the script for that:

CREATE TABLE [dbo].[ST_SharePoint_Worktable_Alerts](
[QueryLanguage] [nvarchar](4000) NULL,
[QueryText] [nvarchar](4000) NULL,
[Report Link] [nvarchar](255) NULL,
[Alert Name] [nvarchar](255) NULL,
[AlertDefinitionID] [smallint] NULL,
[Connection String] [nvarchar](255) NULL,
[Query Column Email] [nvarchar](255) NULL,
[Query Column Name] [nvarchar](255) NULL
) ON [PRIMARY]

Using an Execute SQL Task I can now read from the table above and save the Full Result set into a user defined variable of type object : User::AlertDefinitions

Now I am almost ready. Since I want to only handle one alert at a time, I need a For Each loop. I setup the loop the following way:

• Collection Enumerator : For Each ADO enumerator
• ADO object source variable = User::AlertDefinitions
• Variable mappings (I need to create a bunch of variables here – all strings except for the AlertDefinitionID) in the same order than they are queried in the previous step for ease of use
  • User::ConnectionString
  • User::QueryType (MDX or SQL)
  • User::Query (the query to use)
  • User::ReportLink (the url to the report that the end user should check when they get an alert)
  • User::AlertName
  • User::AlertDefinitionID
  • User::ColumnEmail (the column name in the query which contains the recipient email address)
  • User::ColumnName (the name of the column with the recipient name)

Figure 4 – a bunch of variables

So for each alert in my alert definition list I will get an iteration of the loop with these variables set to the right values.

Now keep in mind that what I want to do is to be able to execute a query against a datasource that I don’t have some metadata about. I still want to keep track of my AlertDefinitionID, AlertName and ReportLink. So I will need to add them with a derived column.

Now it took me some time to figure that out but I think that the Cozyroc’s data flow task plus uses a user variable to be able to dynamically populate the derived column component.

In the next blob post I will try to go into the details about the Data flow plus and how to configure the different components of the task.

Stay tuned for part II coming really soon!!