Categories
Business Intelligence Geeky/Programming SharePoint

SharePoint 2007 2nd Stage Recycle Bin and Content DB Size

Recently, have been running into space issues from a content database perspective in SharePoint (MOSS) 2007. The DB server is still running SQL 2000 on a semi-old box, but since there are physical hard drives, hard to extend without a lot of surgery. Anyways, did some digging and found some things that may be useful for others..

1. You need to be Site Collection Admin and “God” on the servers.

I thought I was, but was mistaken. Was set up a while ago by consultant and I wasn’t involved, and added later. I had some pretty good rights, but not what I needed. I had Central Admin “admin” and other but not on the root site. Also some of the domain users that have rights I don’t have passwords for anywhere as an oversight, so kind of at a loss. I had to get that first before moving forward. I found a solution on this blog. I ran the following from the command line on the central admin server for my site and domain and got the access I needed to run what I needed to run.

stsadm -o siteowner -url http://sharepoint.test.com/sites/[sitename] -ownerlogin [DOMAINnetID]

 

2. I ran Quest’s free server reporting tool.

This is pretty cool, you can go here: http://sar.ondemand.quest.com and install a little WCF service and then run the reports right from your browser, you just have to have the admin rights to the SharePoint site collection to do it. I ran this and found only around 5 GB of usage, but my database was at 222 GB with 278 MB free space not utilized.

image

image

After digging around, I found the issue. First, the recycle bin in SharePoint was full, but also, the “Second Stage Recycle Bin” was ultra full. and Ultra being 217 GB full. What is the second stage recycle bin? Well, users can delete from the site, and it basically changes a db flag to hide the file, then, after 30 days (by default) it moves it to the 2nd stage recycle bin. There it sits and sits and sits. You need to clean this one up, but if you don’t you end up in a situation where I am at.

You can’t turn off the 2nd Stage Recycle Bin because there is too much stuff in it!

3. Delete Items from the 2nd Stage Recycle Bin using PowerShell.

I found this cool PowerShell script here and modified it. First I changed the rowlimit to 10k, and I added a date variable to output the date so I could see that progress was being made. I also am checking my DB size as I run it and watching the free space go up, up up!

#################################################
#
# flushrecyclebin.ps1
#
# Invoke this from a Powershell prompt by calling "./flushrecyclebin.ps1 http://webapp/siteurl".
# This script will only delete items in the second stage (site collection) recycle bin, so you
# will need to manually flush items you wish to delete from the first stage into the second
# stage.
#
#################################################

# we accept this parameter on the command line. If you have several sites' recycle bins to
# flush, you could easily turn this into a foreach loop
param($param_site);

# set this to be an acceptable number of records to delete in one batch. If your recycle bin
# has more items in it than this, you will need to rerun it
$rowlimit = 10000;

#################################################
#
# Don't change below here
#
#################################################

[void] [System.Reflection.Assembly]::Load(”Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c”)

$site = new-object Microsoft.SharePoint.SPSite("$param_site");

$query = new-object Microsoft.SharePoint.SPRecycleBinQuery;
$query.ItemState = "SecondStageRecycleBin";
$query.RowLimit = $rowlimit;

$itemcoll = $site.GetRecycleBinItems($query);
foreach ($item in $itemcoll) {
    $a = Get-Date
    $id = $item.ID;
    write-host -nonewline $a " ... Deleting .. " $item.Title " ... ";
    $itemcoll.Delete($id);
    write-host "Done";
}

 

So where does this leave you? Well, first, on a new SharePoint 2007 setup you want to setup the Recycle Bin Settings so you don’t get caught like this. If you go to Central Administration > Application Management > Web Application General Settings and at the bottom you will see the Recycle Bin settings:

image

Like I said, with a full 2nd Stage Recycle Bin, you are stuck and can’t really do much here. But once clean, you probably want to reduce the number of days to like 7 and also possibly turn off the 2nd stage Recycle bin or reduce the size.

After cleanup like I did, you will want your DBA to probably shrink the file, this is one of the times that shrinking makes sense.

Happy SharePoint’ing!

Categories
Business Intelligence Geeky/Programming SharePoint SQLServerPedia Syndication

PowerPivot, Excel Services, SharePoint 2010 Farm, and You

The last few days I have spent an exorbitant amount of time (5-6 hours?) getting PowerPivot and Excel Services working on a SharePoint 2010 Farm. I just want to get out there some of the things I had to do to get it all working, and why (at least what I think is why).

First off, most dev setups are using one box for SharePoint, which, in my eyes, masks most every issue you will run into.

Most prod environments are multiple boxes, Web Front Ends, App Servers, etc. This leads to the most pain when setting up these services (PowerPivot, Excel Services) as there are hundreds of different configurations and setups. You need to get it juuuust right.

Anyways, I will explain as best I can.

First, assuming you have PowerPivot running on an app server and Excel Services running to. You upload your PowerPivot workbook to a PowerPivot gallery and you go home happy. But wait, does the data refresh correctly? Can you even open it?

1. If your site is running https, you have to tweak the Excel Services settings.

By default, only http:// shows up here (I think?) so you need to add https:// or you can’t even really get into your PowerPivot.

image

2. You probably are going to have to up the upload size limit

Some PowerPivot workbooks are big, like 80-100 MB big. Default in SharePoint 2010 for Excel Services is something like 10 MB, and SharePoint default is 50 MB, you need to change for both settings.

3. If you don’t have Kerberos set up, it is tougher.

PowerPivot refreshes data from some system somewhere to its own “cube”, then Excel Services refreshes data from that cube into Excel Web. You have to setup a way for the data to get refreshed into PowerPivot’s cube. No Kerberos? Then you need to use the Secure Store Service and set up a credential, and set that up as the unattended service account for PowerPivot refresh. Then, at least you can get data from somewhere else into PowerPivot’s “cube”

Second step is getting Excel Services to refresh from that cube into Excel web. What I had to end up doing was creating another credential in the Secure Store service for Excel Services Refresh (set up as farm account for now, it has the stroke it needs). And then set that up in Excel Services settings as well for Excel Services unattended refresh account. But also! – you need to change your workbooks before you upload or whatever.

In your workbook, go to your connection, properties, and get to the authentication, and change to “none” instead of “windows authentication”, then your data will refresh from the PowerPivot cube to Excel in SharePoint.

4. Same thing goes for the PowerPivot Management Dashboards

The management dashboards are set to “windows authentication” so they wont work either, you need to change to “none”, in the Management Dashboard site, goto all site content, PowerPivot Management, <some guid> folder, and then 1033 (US English), edit the two workbooks to use “none” and save, and your management dashboard will actually work!

I’m sure there is a ton more I go delve into here, but this is the high level. As Rob (@powerpivotpro) would say – “make sure to click on the slicers!”

Categories
Geeky/Programming

Japanese Character Sets – Input, Identification and Comparison

As a byproduct of some work our team is doing, we needed to test and handle different character sets for Asian languages (Katakana, Hiragana, etc). A guy on the team came up with a pretty sweet proof of concept test application so we can trial combinations and see how things work based on ignoring widths, etc. even some database interaction. We figured this would be useful to others in the .NET community so we are giving back

Check out http://jpcharsets.codeplex.com/ for more info, but basically you can play around with the different character sets to your hearts content and look at the source code behind the comparisons of different sets. Enjoy!

Categories
Agile Geeky/Programming SQLServerPedia Syndication

Agile: Creating an SSRS Burndown Chart Part 2

In the previous post in this series, Agile: Creating an SSRS Burndown Chart Part 1, I explained what data you would need to prepare to create an SSRS Burndown Chart (Sprint_Dates, Stories, Story_History). In this part of the series I will explain how to get a basic burndown report in SSRS.

First, fire up Report Builder 3.0 and create a new report (if the wizard pops up, just pick “Blank Report”). You need to add a Data Source to your report. In my example, I am just using a database on my localhost called Agile, so I connect to that and create a report Data Source.

image

 

We then need to add 3 Datasets to the report. (Burndown, Sprints, and CurrentSprint), and one parameter (Sprint) and we can then format our report.

 

Sprints (this will be a dropdown of Sprints for a user to choose from)

image

CurrentSprint (this will get the current sprint based on what day we view the report, default param for the Sprint parameter we will create)

image

 

For the Burndown, do the same thing, but since the query is so large, no screenshot, just the query:

;WITH DayHistory AS
(
SELECT
	 bd.[Date]
	,bd.PointsScheduled
	,bd.PointsLeft
	,bd.PointsScheduled - ((ROW_NUMBER() OVER (ORDER BY bd.[Date]) - 1) * (CAST(bd.PointsScheduled AS DECIMAL(15,6))/10.0)) AS 'Goal'
	,ROW_NUMBER() OVER (ORDER BY bd.[Date]) AS [DayNumber]
FROM (
	SELECT tot.Sprint,tot.LogDate AS [Date],
		CASE WHEN SUM(tot.PointsScheduled) = 0 THEN (SELECT SUM(Points)
		FROM dbo.Stories st
		WHERE Sprint = 'Sprint01') ELSE SUM(tot.PointsScheduled) END AS 'PointsScheduled',
		SUM(tot.PointsLeft) AS 'PointsLeft'
	FROM (
			-- Get History for the Current Sprint
			SELECT Sprint,LogDate,SUM(Points) AS 'PointsScheduled', SUM(PointsLeft) AS 'PointsLeft'
			 FROM
				 dbo.Story_History st
				WHERE Sprint = @Sprint
			GROUP BY Sprint,LogDate
			UNION
			-- Get the Current Day
			SELECT	Sprint AS 'Sprint',CAST(GETDATE() AS DATE) AS 'LogDate',
				SUM(Points) AS 'PointsScheduled',
				SUM(PointsLeft) AS 'PointsLeft'
				FROM dbo.Stories
				WHERE Sprint = @Sprint
			GROUP BY Sprint
			UNION
			-- Get zero's for all days in sprint to round out our dataset
			SELECT 'Sprint01' AS 'Sprint',WorkDate,0,0
			FROM dbo.Sprint_Dates
			WHERE Sprint = @Sprint
		) tot
	GROUP BY tot.Sprint,tot.LogDate
) bd
)
SELECT
	 a.[Date]
	,ISNULL(b.PointsScheduled, a.PointsScheduled) AS [PointsScheduled]
	,ISNULL(b.PointsScheduled, a.[PointsLeft]) AS [PointsLeft]
	,ISNULL(b.PointsScheduled, a.[Goal]) AS [Goal]
FROM DayHistory a
	LEFT OUTER JOIN DayHistory b
		ON a.DayNumber = b.DayNumber - 1
			AND b.DayNumber = 2
ORDER BY Date

 

This query is where all the magic happens. First, you need to get your story point values for the days, from your history, and also from the current day, you also need to get all days for that sprint with zero’s so that your graph will have all days and not just days with burndown. The CTE around the main query calculates the burndown by day so you end up with 4 columns, Date, PointsScheduled, PointsLeft, Goal

Now that you have your Datasets, we need to create a parameter, and then the graph!

Create a new parameter called “Sprint”, and set up the available values. Remember the Dataset we created to get all the sprints? Here is where you use it, like this:

image

Next, we want to setup the default values. Remember the query to get the “Current Sprint” – that is used to set our default.

image

Once you have that all setup, it is time to build the graph!

We are really close to having a working report here, and check back for part 3 of the series to get the graph working correctly, and part 4 for beautification!

Categories
Geeky/Programming

VMWare Workstation Lost All USB Devices – Fix

I blogged earlier about VMWare and Lost Ethernet connections as Emily has her pc running as VM on my dev machine at home. Well recently another weird thing happened. All USB devices dissapeared. Not sure why or what happened, but I had to edit the .vmx file directly in notepad and set the option

usb.present = “TRUE”

it was set to FALSE, but neither of us changed it, somehow VMWare lost the ability to know it could process usb devices for this VM. Odd, but resolved.

Categories
Agile Geeky/Programming

Agile: Creating an SSRS Burndown Chart Part 1

The burndown chart. A must have for any ScrumMaster and Agile team. What it should show you is the rate at which you are “burning” down story points.

image

As you can see from the chart above, 3 lines. Red is your “points scheduled”, Green is the “goal” and blue is “points left”. While it is easy enough to create this chart and track the burndown manually in Excel, many teams after using Excel turn towards other systems to track their points and sprints. Right now I have one team using Unfuddle, one team using TFS, there are others that use this chart that use Footprints and really you can use whatever, and this chart can be built off of any database as long as it has the right data.

First, you need a table with your stories in it. You need to have some key columns – Sprint, Points and PointsLeft.

CREATE TABLE [dbo].[Stories](
	[Sprint] [varchar](50) NULL,
	[Points] [int] NULL,
	[PointsLeft] [int] NULL,
	[StoryId] [int] NOT NULL,
	[StoryText] [varchar](max) NULL
) ON [PRIMARY]

Now you may have others, like StoryId, StoryText, Assignee, etc but we aren’t concerned about those for this chart.

You then need at least 2 or tables, and a SQL job. 1 table to hold your Sprint and Dates and one to hold your “Story History”

 

CREATE TABLE [dbo].[Sprint_Dates](
	[Sprint] [varchar](50) NOT NULL,
	[WorkDate] [date] NOT NULL
) ON [PRIMARY]

CREATE TABLE [dbo].[Story_History](
	[LogDate] [date] NOT NULL,
	[Sprint] [varchar](50) NOT NULL,
	[Points] [int] NULL,
	[PointsLeft] [int] NULL
) ON [PRIMARY]

 

You will need a SQL Agent Job to run at 11:55 PM to capture the history, which should run this query:

 

INSERT INTO dbo.Story_History (LogDate,Sprint,Points,PointsLeft)
SELECT CAST(GETDATE() AS DATE),Sprint,SUM(Points),SUM(PointsLeft)
FROM dbo.Stories
GROUP BY Sprint

 

Remember you might not need all 3 tables, just the history and dates. You can get your actual stories off of wherever your stories are stored in the database. Now that you have your data in place, you can get ready to write the actual report! Look for the next part in this series.

Categories
Geeky/Programming

Windows 7 Winforms ActiveX DEP and TFS

Wow, what a title. Deserving for the debacle it involves. If you are using an older ActiveX/COM component that doesn’t support DEP (Data Execution Prevention) – you might run into this.

Your app will compile file, and when users try to run it, it runs fine, yet when you use the control you get an error about shared and corrupt memory. The ActiveX control might not support DEP and you don’t have any way to go around it. You can have the users turn off DEP but that isn’t a viable solution.

What you can do is set a post-build step to turn off DEP for your exe which works great.

here is the post build step from the link:

call $(DevEnvDir)..toolsvsvars32.bat
editbin.exe /NXCOMPAT:NO $(TargetPath)

But then you go to check it into TFS and your build fails. Why? Because there is no environment variable for $(DevEnvDir) when team build (msbuild) builds your solution.

In order to fix that, you need to add an environment variable on the build controller to

64bit: C:Program Files (x86)Microsoft Visual Studio 10.0Common7IDE  

32bit: C:Program FilesMicrosoft Visual Studio 10.0Common7IDE

and then reboot your build controller, and it should build.

Categories
Geeky/Programming Product Reviews

UserVoice: Using A Customer Service Tool To Democratize Technical Debt

Not sure if you anyone has heard of or used “UserVoice” – It is a site that allows you to create “forums” for your products and then submit ideas, give users votes and they can vote them up, and an admin can say things are started, merge ideas, or mark when the idea has been completed (and the votes go back to the users).

UserVoice is (sorta) along the same lines as GetSatisfaction (another cool customer service 2.0 app). Pretty cool tools. If I was in a customer service role, especially with any type of user based or public product, I would be running these tools to gather ideas and feedback from my users.

I am in a technical role, so what I decided to do was “democratize” the development area of our product one of my teams is working on. We have a ton of technical debt (as do most teams, it is just a matter of what level of debt you have) – but what should we work on next from a technical perspective?

In comes UserVoice. Let’s throw out ideas on UserVoice, give everyone 50 votes, and the ideas that bubble to the top will become our next set of things to work on. One “idea” may become several “user stories” (we are agile). Our goal is to have 20-25% of our stories focused in on paying down our technical debt. If we didn’t, the debt would never get down to a low enough point to where we are very comfortable.

What is cool is that it really shows what the team wants to focus on next. People can have others vote up their ideas, etc. Also, getting the votes back at the completion of an idea is key. As you can imagine, our forum is private. The one cool thing about UserVoice is you can create multiple forums, with different ranges of settings, so you could also have a public forum, or a different private forum for a select group of users, etc.

One thing I wish I could do is maybe give different # of votes to different users. Integration out of the box with TFS or other systems would be nice too, I haven’t looked to much into that though.

If you have a team that ranges from medium to large, I would suggest checking out UserVoice to get the ideas and opinions of the members out on the table regarding your technical debt. You may be surprised as to what gets voted to the top!

Categories
Business Intelligence Geeky/Programming SQLServerPedia Syndication

#sqlpass Summit General Thoughts

Was at #sqlpass last week. Great conference/trainings, and people. Hopefully I will go next year (I am planning on it!) General thoughts/overview..

1. Should have done pre/post cons
2. Drink more water (funny thing is, no water during refreshment breaks? just juice/coffee/soda)
3. Leave room in your luggage for schwag
4. Bring more business cards.
5. Introduce yourself to more people
6. Realize that the level #’s on the courses aren’t always indicative of the content.
7. Go to sessions with ambiguous titles, they are usually the new stuff/undercover stuff
8. The Daily Grill (restaurant attached to the Sheraton) is the most expensive place in the world
9. Plan flights with more time in between if connecting. We missed our connecting flight because of delays.
10. Put the thing on your door from Sheraton to not upkeep your room, you get 5 $ coupon a day
11. Wifi sucks, use your 3G or EVFO/Mifi or whatever
12. They keynotes each day can be hit or miss, but still make sure to go to all of them.
13. Leave sessions 2 min early if you want to get a good seat at a highly anticipated next session
14. Take the back stairs from level 6 to level 4 instead of the escalator for a shortcut
15. “game” gameworks – my team knows what I mean 🙂
16. Bush Garden is small. But you can still rock #sqlkaraoke . Just don’t sit on the couch in the bathroom.
17. You can walk pretty much anywhere (Pike St, Space Needle, etc)
18. Go talk to vendors, but don’t let them suck you in. Stick mainly to the PASS area and Microsoft area (at least that is what I think)
19. Try to take advantage of the games and contests held throughout (Foursquare, Vendor check ins, Wheel of SQL) but don’t let it consume you.
20. If you go into a session that you find boring or not relevant, don’t feel afraid to leave and go to another – the sooner the better.
21. Order the DVD’s
22. Take advantage of BOF luncheon.
23. Follow the twitter stream .. #sqlpass
24. Tweet your encounters and info
25. Blog about it after 🙂

So much more to talk about, I will try to decompress it all and throw up a few posts, but yeah, well worth it, so much information and people and just thinking outside the box, and it just gets you to think and become motivated. Great time!


Categories
Geeky/Programming

VB6 to .NET Migration: Decision and Analysis

Recently completed a huge project migrated a fairly decent size code base from Visual Basic 6 (VB6) to C# .NET. Yes, I said “migrated”, not rewrite – let’s get that out of the way up front. The code was migrated.

How did we go about doing this? Well, I’d like to detail it out over a few posts here and in the near future.

First, you need to make the decision to migrate instead of rewrite. Not very often do dev shops get the chance to do a rewrite, and even when they do, most rewrites go over time and budget and end up losing business logic and functionality, that is why the decision to migrate instead of rewrite is a such a good one.

So, after making that decision, which hopefully you do early, or before you are in the middle of a rewrite (as we did), you can get started. There are a few options, and I won’t go into complete detail on them, but just go over them high level..

1. Use the Visual Studio Migration tool and convert the code yourself and go at it alone

2. Look at the various companies out there that do migrations and pick one, and create a project that way.

We went with #2. After some research it was pretty obvious who the main player was out there, ArtInSoft. After researching them and watching their webcasts and reading and doing as much looking into as we could, we contacted them.

Their model is pretty simple, they send someone out to you for a week (on your dime) and they analyze the VB6 project to see how much it would take to migrate.

What I learned shortly after talking with ArtInSoft and bringing one of their analysts here is that ArtInSoft actually created the VB6->.NET conversion wizard in Visual Studio. What it is, is a lightweight version of the VBUC (Visual Basic Upgrade Companion) that they licensed to Microsoft. The VBUC in Visual Studio is very limited though. It will only migrate to VB.NET and has many other limitations, but it does still work. If you had a very small project (lines of code, complexity) – you could easily use the VBUC in Visual Studio, but most projects aren’t simple and have many lines of code, that is where you would want to use ArtInSoft and their “enterprise” version of the VBUC, as well as assistance from ArtInSoft.

Back to analysis. What they do is actually migrate your project, find any ancillary projects you might have and make sure everything gets converted. Their VBUC puts in EWI’s, Errors, Warnings and Issues, within comments in the newly converted code.

They then go throw the code and do find’s on all the EWI’s, dump them and parse them out, put into excel and pivot them to analyze how many of each EWI you might have. That way they can estimate how much effort/time/money it might be to do your project. They have done this many times, so they have it pretty much down to a science.

An example would be, you have 2500 warnings for warning XYZ – we know it takes 2 minutes to fix one of those, and they are usually all unique, so then we know the total time it would take to resolve them all.

After doing their analysis, they give you a project estimation, and then it is up to you to decide if you want to move forward, and when you would want to move forward..

 

To Be Continued…