Microsoft Business Intelligence – Steve Novoselac's Blog

10 years ago, it was SSIS/SSAS/SSRS

Then in 2007 SharePoint, PerformancePoint/SSRS

Then in 2010 Power Pivot in Excel/SharePoint, then Power View in SharePoint

Then in 2013 Power BI … Power Pivot, Power View, Power Query, Power Map.. In Excel and Office 365.

Now in 2015 Power BI Version 2. Not in Office 365, separate. Power BI Designer, or use the Power BI web site to set up your dashboards, mobile, etc.

All the while, the existing solutions that have been available previously are still there and available, making things… well, confusing to say the least.

Most shops .. It all depends on when they started going heavy BI with the Microsoft tools, on where they land. Also, how well they could move when things change, as well as how much they want to stay up to date with the tools.

If you started 10+ years ago, you probably have a good base of ETLs written in SSIS, as well as many multi-dimensional (MD) OLAP cubes in SSAS, and SSRS reports off your cubes and data warehouse, running in SSRS Native Mode. You started with SQL 2000 if you were lucky, with cubes and dts packages, but then SQL 2005, then 2008, 2008 R2, 2012, 2012 R2 and now are on 2014. You really liked 2005 SP2 and 2008 R2 for the BI features :). This setup is like the VB6 or .NET Winforms of BI. It will probably be around forever in some way shape or form but not a ton of updates and Microsoft has moved on.

If you started a little later you might have SSRS in SharePoint mode, and some Performance Point dashboards. You might have even used Performance Point for planning/budgeting (and loved it?) until Microsoft killed it. Then you had to look for alternatives for that, or use OLAP Cube Writeback. In my opinion, SSRS in SharePoint and Performance Point are dead. Not dead as in they don’t work or won’t be supported, but I see them as the wrong path, life supported direction. If you are still using these heavy I would look for alternatives.

Now it gets interesting. You started with Excel 2010 and PowerPivot (no space!) and had SharePoint 2010 setup and Power View in SharePoint. You created V1 Power Pivot models, they were limited, you could do some things, but still it was limited. You still needed to get data somewhere so SSIS ETL’s or something to get data in tables you can use. If you are using Power View in SharePoint, I would hurry up and look for alternatives, it is dead (my definition of dead like SSRS/PP in SharePoint). Excel 2010 is long past and V1 PowerPivot is dead too. Seems like this era was short lived and just a stepping stone.

Then, in 2013, Power BI. So they added a space to Power Pivot 🙂 .. And made it better, v2. Added missing features, Tabular SSAS cubes even! And Power View could be used in Excel. They both came by default in Excel (depends on version) but turned off. Power Query came out of nowhere and is awesome and Power Map, while buggy, was better than nothing. But what do you do with all these solutions you build? Where to publish? Not SharePoint on prem? But Power BI in SharePoint Online.. So you need Office 365 and Power BI subscription. You set up Data Management Gateway so you can get to your on prem data sources. You can refresh once a day or manually. You can do some pretty cool things, create workbooks with pivots and Power Views.

But you are missing things. Missing things like the ability to schedule a report to run and email someone, like SSRS. You are missing awesome formatting abilities for every pixel like SSRS. You wonder when SSRS is going to come to Power BI or what your options are… you hope you see iterations and features released to Power BI as that is the path, but then..

New Power BI Preview comes out in 2015. It has a standalone Power BI Designer (reminiscent of the Performance Point designer) that lets you create reports, dashboards and save a file to publish to the NEW Power BI portal. So you have two Power BI portals.. New and old. They don’t overlap or talk to each other, the licensing is different, etc. The old Power BI lets you connect to SQL on prem with refresh with the DMG and other data sources, etc. The new one does not. The new one lets you connect to GitHub and SalesForce and Marketo, but not other data sources that the old Power BI did. The new Power BI lets you connect to on-prem TABULAR SSAS cubes with refresh, but not MD ones (yet). The new Power BI lets you connect to excel data in OneDrive/OneDrive for Business. So could one publish a data file out to ODFB to faux refresh? I have yet to try. The new Power BI lets you publish dashboards to the iOS Mobile apps and also embed (up to 10 MB – which needs change to be bigger) on websites. New Power BI has an API that lets you create your own connectors / REST API for things. And the list goes on and on.

So where does that leave us? Well, of you invested time and money in BI the last 10 years, you might feel like Microsoft is abandoning you. It kind of seems that way. You need to change or get left behind. But what do you change to? Change your MD cubes to Tabular? Rethink your architecture? Sync data to Azure? Power Pivot/Power Query? Abandon SharePoint as a BI tool? Move your reports from SSRS to something else or Power BI (if you can?) I am unsure. Still trying to figure it all out.

One thing for sure is, it will always keep evolving. Me, I would say, tabular first if you are on prem. Try to use Power BI where you can. Minimize SSRS reports. Use SSRS native instead of SharePoint. Stop using PerformancePoint if you are still using it or thinking about it. I bet at some point SSRS comes to the new Power BI – there is an item on the UserVoice forum already asking for it. Try the Power BI Designer and Website and see what you can do. Always be trying to get something going in the newest and latest technology/tools available.

Have Fun with Microsoft BI now and what is yet to come!

Today I received an email asking to some extent best practices on development with SQL Server Integration Studio (SSIS) and Business Intelligence Developer Studio (BIDS) in a team environment. Here is part of the email:

Me and another DBA belong to the same team, we have a SQL server with SSIS running. We use the SSIS transfer data among multiple data sources. In SQL 2000 DTS, both of us can save the package on the server and open/edit it in the enterprise manager. In SQL 2005, I can see the package on server, but can’t open it directly. We came out a solution: create a shared folder on the server called ‘SSIS Projects’, both of us can access to it. We run the ‘SQL Server Business Intelligence Development Studio’ on local PC, to open the project in that shared folder. When done with the change, save the package to the SSIS server. Now, we have more than 50 packages in a project. Problem is: it’s very slow when open a project, ‘Business Intelligence Development Studio’ tends to open/verify every single package inside a project, takes up to 10 mins and getting worse. We really miss the SQL 2000 DTS, but we have to turn to SQL 2005.

Are we doing the right thing? Is there any better solution for SSIS developing in a team environment?

When open a project, does ‘Business Intelligence Development Studio’ has to open/verify every package?

This got me thinking, and I figured instead of write an email back, it would be good info for a blog post. So here is what I think and some things I have done that have worked.

First, yes, SQL 2000 DTS allows you to just edit on the server, do more than SSIS, is just way better than SSIS. Wait, what? Well, yeah some people will say that, because it does one thing that might be a little rigmarole in SSIS, but no, SQL 2000 DTS is not better than SSIS, just wanted to clear that up.

So, the is meant to be a starting point, by no means all encompassing, and as always, YMMV.

One thing that I first thought about is this: Yeah, if BI devs and SQL devs have never really worked in a team environment, developing software, how would they know what to do, or best practices? They would just go about “making it work” until everything breaks or who know what.

So how to develop Microsoft Business Intelligence Solutions in a team environment?

1) Standardize on Versions

First, figure out what “versions” you are going to support, and what you are going to use, and get standardized on them. I am guessing majority of BI devs right now are on the 2005 stack. Yeah, there is still probably a bit of 2000 legacy stuff out there, and some people are now getting into the 2008 stuff, but 2005 is pretty much the norm from what I see, at least at this point.

So, 2005. Get all your dev’s on 2005 on their machine – same patch level, etc. Get BIDS up to the same level. Get BIDS helper installed everywhere. Strive to get all your ETL packages in SSIS 2005, get all your cubes to SSAS 2005, etc, etc. Come to a consensus on things like config files for SSIS, naming conventions, within your development and on disk – folder structure is key! With a smaller number of versions of things floating around, it makes it easy for anyone on the team to open up a solution and start hammering away without tons of setup.

2) Get Source Control

This is crucial! I have talked about source control in the past, and also about some that aren’t so great. Really it doesn’t matter what you use, I prefer SVN. I install Tortoise SVN, SVN proper (to do scripting etc if I need to using cmd line) and also purchase Visual SVN, an add on to Visual Studio that integrates with SVN. for 50 bucks you have your source control system. Visual Source Safe works but is outdated, honestly I hate it. Team Foundation Server is good, but expensive. Other solutions might be using something like GIT, etc. Whatever you do, just get a source control system going, and learn it well. Learn how to create repos, commit, update, revert, merge, etc. Set up a user for each BI dev and make sure they commit often, and make sure they leave comments in the source control log when they commit, history is your lifeline to go back to something if you need to! Note: exclude .suo, bin, obj directories, .user files, etc. Anything that changes every time you build, open, etc, you want to exclude from source control.

3) Development Box

You now have your version standardized, and your source control setup. You can get most of your work done on your machine, but you need somewhere to test deployments, run scenarios, etc, etc. Make sure you have a comparable box to your production server. Set it up the same, same software etc. Make sure its backed up. Let all the devs know its a dev box, it can be wiped at any time for any reason if need be. It can be rebooted 5 times a day if need be. Its a dev box! But you can test and develop and tweak and change settings to your hearts content and not have to worry about breaking Mr. Executives reports.

4) Developing, Merging, Committing, Collaborating, Communicating.

So now you have your setup, well.. setup. Start creating stuff. SSIS Packages, ETL’s, SSAS Cubes, SSRS Reports, the whole MSFT BI Solution. This is where stuff can start to get tricky in a team environment though. SSIS/SSAS/SSRS isn’t as clean cut as something like C#/VB.NET, etc. Everything is in some form of XML behind the scenes, and with graphical based editing, you can move stuff around and it changes the files. Things like that are going to be your enemy. This is why you need to collaborate and communicate. Usually one person should be working on one project at a time. You can get really good at communicating and then in SSIS at least have multiple people working on different packages. Also in SSAS dimension editing and stuff can be done by multiple people at the same time as long as the dim is already hooked up to the cube. But you want to make sure that you communicate, “Hey, I am checking this in, you might want to do an update”, or “Is anyone working on this or are they going to? I want to modify something, and I will check it in so you all can see it”

You want to make sure you have your folder structure, and solution/project structure set up well. C:Projects .. and then maybe a folder for each major project “CompanySales” and under that, “ETL”, “Cube”, “Reports” and have a solution under each with 1 project of each type. You can also have a generic SSRS solution with many projects, which might work well for you. In any case, just come up with a standard and stick to it. Trust me it will make your life easier. The question from the email above, it sounds like they have every package in one solution, one project. Sounds like it needs to be split, multiple solutions, multiple projects.

5) Deployment Scenarios and Strategies

Now that you have everything developed, tested, checked in, what do you do?

Personally for SSIS I like xcopy deployments. One folder on the server, not on the C drive, but another drive, lets say “E:SSIS” under that a folder for each project. Put your dtsx and configs in the same folder. 99% of the time you are going to call the dtsx from a SQL Agent Job, and most likely you are going to run into a scenario
where you need uber rights to execute it, so learn how to create a proxy/credential in SQL security so you can run the step as that. Once you have this folder and subfolders setup, you can use something like Beyond Compare to compare the folder on the server to the one you have locally that matches. Remember to copy files from the bin directory of your project after you build it, not the files directly on your project. As far as BIDS validating every package, there are workarounds out there you can do, here is one.

For SSAS, I try to lean towards using the Deployment Wizard that comes with SQL Server. You can use BIDS deployment, but if you start doing anything advanced with roles, partitions, etc, you are going to run into trouble. Take control and use the deployment wizard. I usually like to deploy, and then process manually when developing. And then later use SQL Agent or and SSIS package to actually do processing when it comes to a scheduled processing.

SSRS, I have become used to the auto deployment from Visual Studio. To really do this though, you need a project for every folder in SSRS, which can become a pain. You can always just upload the .RDL file and connection and do it manually, but if you start off right with using the deployment from BIDS, it can make your life easier.

So that is just a 10 minute overview of everything to kind of get started. Everything depends on your infrastructure and the way your team is setup, etc. But I think the biggest thing to take from all of the above is to standardize on things. If you standardize on as much as possible, SQL versions, setup of machines, naming conventions, layouts, design patterns, etc, everyone can do things faster and pretty soon it will start running like a well oiled machine!