Small PowerShell Script

July 6, 2010 at 9:59 am (PowerShell, SQL Server 2008, SQLServerPedia Syndication) (, , , )

I’m still trying to learn PowerShell better. The opportunity to answer simple questions and problems with the tool is hard to pass up. We had a need to clean up data directories where data files were left behind or people put inappropiate files, so I wrote the following Powershell script:


Set-Location $filelocation

foreach($file in get-childitem)

{$base = $file.Name;

$result = Invoke-Sqlcmd -ServerInstance $sqlinstance -Query "SELECT DB_NAME(mf.database_id) AS db FROM sys.master_files mf WHERE RIGHT(mf.physical_name,LEN('$Base')) = '$Base' UNION ALL SELECT 'NoDb' AS db WHERE NOT EXISTS (SELECT DB_NAME(mf.database_id) AS db FROM sys.master_files mf WHERE RIGHT(mf.physical_name,LEN('$Base')) = '$Base');" ;

if($result.DB -eq "NoDb" -and $file.Extension -ne ".cer"){Remove-Item $base}}

It’s a very simple script. It takes a UNC and a server instance and then walks through the files in the UNC and validates whether or not those files exist within databases on the server. If they don’t exist, it deletes them. That’s it.

I’ve published this to the Technet Script Center Repository right over here. I’m going to work on making it a bit better, so for updates, go there.

Permalink 1 Comment

More Free Training

September 15, 2009 at 9:59 am (SQL Server 2005, SQL Server 2008, TSQL) (, , , , )

Quest Connect 2009, taking place in October 21 for 24 hours, looks like it’s going to have 64 different sessions, live and recorded, by a variety of the names in the industry. It’s another chance to dig in and learn the details on a variety of topics from some of the top names in the business. Can you say Tom LaRock? How about Tim Ford? I know you want to hear from Brent Ozar. Those are just some of the featured speakers. There are a whole slew of others, it’s worth pursuing, and did I mention, the price is right.

I recorded a session for them last night. It’s on the basics of understanding execution plans.

Permalink 2 Comments

More Spatial Headaches

June 11, 2009 at 2:38 pm (spatial data, SQL Server 2008, SQLServerPedia Syndication, TSQL) (, , , , )

I keep thinking I’ve got a handle on spatial data and I keep finding out I’m wrong. I rewrote the cursor that my co-worker used into a query that joined the list of test criteria onto our list of locations. I used an OUTER APPLY to more or less run a check for each of the test criteria since, except for the proximity to the locations, there’s no actual relationship between the test criteria and the location data for me to join on.

The query looked something like this:

,l.[Location].Lat AS LocationLatitude
,l.[Location].Long AS LocationLongitude
FROM dbo.Alert AS a
		FROM dbo.MyLocs x
		WHERE x.OrgID = 42
		AND x.[Location].STDistance(a.AlertLocation) < 50000) AS l

The cursor was taking almost a full minute to return just under 400 rows worth of data. This query took about 25 seconds. SCORE!

Unfortunately when I looked at the execution plan, the optimizer was not able to recognize that my spatial index would increase the performance. So I modified the query to provide a hint:

FROM dbo.MyLocs x WITH (INDEX(xsHigh))

Now the query returned in 5 seconds. BIG SCORE! Then I remembered my co-worker and his low density index. So I quickly rebuilt it and changed the INDEX hint to use the low density index. It ran in just over 2 seconds. Yeah, it’s a score, but why? I thought the high density index was the way to go with point locations. I decided I just wanted to see the middle behavior and see if a medium density index took 3.5 seconds about mid-way between the low and high density indexes… It ran in just over 1 second. WHAT THE F…

I then ran a whole series of experiments with my query. I kept changing the density and the cells per object value for the three indexes and recording times. In every test, the medium density index outperformed the low and high density indexes. It never outperformed the low density index by a whole lot, usually running ~50% faster as opposed to running 5 times faster than the high density index. But it really did run better.

My current working theory, and I’m trying to figure out how I can validate this, is that the low and medium density indexes were more efficient at eliminating the negative values, the places where the criteria wasn’t in proximity to my locations. Then, the medium density index was just better at retrieving the data than the low density index. I base this on the fact that the primary filter efficiency on the medium density index is  better than the low (although not as good as the high).

Permalink 2 Comments

Spatial Data

April 21, 2009 at 7:33 am (spatial data, SQL Server 2008, TSQL) (, )

I work for an insurance company. If you think that maybe, we might be interested in the physical location of the things we insure, you’d be right. Actually, we’re an insurance company predicated on the idea that risk can be managed. That means that not only do we know where your factory is located. We know the wind zone, earthquake zone, flood zone, rain zone and temperature zone it’s in. We send engineers out to the site to inspect it and recommend upgrades. We track the upgrades and the condition of your facility.

With all that location specific information, just how important do you think it is that with SQL Server 2008 we’re finally getting a spatial data type? Yeah, exactly.

We’re in the process of launching our first full implementation of the spatial data type and, quite frankly, I was not ready. I had read several of the BOL entries and a few articles online, but nothing had given me enough information for me to say I understood how spatial data worked. Now I at least feel like I’ve got a basis for understanding. That’s because of Alistair Aitchison‘s book Beginning Spatial.

This was not an easy read for me. The first few chapters are frankly difficult. The concepts behind spatial data are not exactly simple. However, Mr. Aitchison did a great job of presenting the information in well explained, digestible chunks. I was able to get through. Coming out the other side, I’ve still got a lot of questions and concerns (especially around spatial indexes, he could have done another three or four chapters on just that topic), but I’m ready to support our project going forward now.

If you’re dealing with Spatial data and you’re not already a guru on the topic, I’d strongly suggest you pick this up. If you’re an expert, this book isn’t for you. There are plenty of examples and Mr. Aitchison walks you through some important concepts, such as importing spatial data. Did you realize that Microsoft didn’t include any mechanisms for dealing with in SSIS? Instead you either need to get creative with TSQL or, better still, use a third party product. I got a copy of Shape2SQL and I’ve tried it out. It seems to do what we need. We still might end up buying a commercial product (several were listed in the book). But it was the information in the book that told me what I needed to know to get started working with our spatial data.

I need to track down more info on spatial indexes now.

Permalink Leave a Comment

Reading to Learn

March 30, 2009 at 10:55 am (spatial data, SQL Server 2008, SQLServerPedia Syndication) (, , , , , , )

I just finished chapter 1 of Alastair Aitchison’snew book on SQL Server spatial data, “Beginning Spatial with SQL Server 2008.” If this is the beginners book… oh boy. The advanced book must be insane. Seriously though, Mr. Aitchison seems to have written a fantastic book. I’m going to tear through it as fast as I can because I’ve got two projects that are looking to start using spatial data and quite frankly, I’m a bit lost.

There’s a great discussiongoing on over at SSC as to the worth of technical books for DBA’s. It’s based on this editorialby Tony Davis. I’m surprised by the number of people who say they don’t use books. It seems that a lot more people use blogs and articles and discussion groups to learn. Maybe I’m showing my age a bit, but I don’t think a blog post or an article is going to get the depth and knowledge that Mr. Aitchison is displaying in this book. I know I’m regularly opening Kalen Delaney’s Inside SQL Server 2005 (and the new one for 2008 just came out) to look up bits & pieces of information that just isn’t as readily available on the web. Also, it’s worth pointing out, except for the editing that comes from people who read this blog, no technical review is done of this information. I might be right about the things I post, but I could be VERY wrong. Same with any other blog you read, including blogs by the big names. Despite the errors that creep into books (and trust me, they do), books are very carefully scrutinized by multiple sets of eyes to try to catch those errors prior to publication. They miss some, but they try not to miss any. Few blogs are like that. Not that many technical publications are terribly strict about technical accuracy either. I generally find more good information in the right books than anywhere else.

End of rant. I need to get back to reading this excellent book.

Permalink 4 Comments

Execution Plan Estimated Operator Cost

March 19, 2009 at 1:09 pm (SQL Server 2008, SQLServerPedia Syndication) (, , , , , , )

I’ve said it over and over again, the costs on operators in execution plans, even in actual execution plans are estimates.  You need to understand that when looking at your execution plans. It’s vital because you need to be able to distinguish between the truly costly parts of a plan and the less costly parts of a plan. Don’t believe me? Take a look at this picture and see if you can spot the discrepancy:


Spot it yet?

Add up the costs for the operators visible in the part of the plan…

Yep 125%.  And there’s more to the plan that I’m not showing. I think this one must total near 200%. The statistics are up to date and there’s no consistency errors in the database. These estimates are just off sometimes.

This is a rather costly query being run against Microsoft Dynamics CRM. There were two missing queries identified by the optimizer and about four key lookup operations that I fixed with two index changes. This jumped out and I wanted to share. The fact is, right now, the query is working well. That particular table has one scan and some reads, identified from looking at the STATISTICS I/O, and it’s not even close to 100% of the cost of the query, but I would sure be in a panic if I believed the estimated operation cost.

Permalink 14 Comments

Missing Index Information and Query Stats

February 12, 2009 at 3:49 pm (SQL Server 2005, SQL Server 2008, SQLServerPedia Syndication, TSQL) (, , , , )

So the goal was to find a way to pull information from sys.dm_exec_query_stats so that we could identify poor performing procedures that were in cache at the moment and combine it with missing index information from sys.dm_db_missing_index_details. We’re combining these because we’re working with a Microsoft Dynamics CRM database that is almost all ad hoc queries and lots of them are against tables with missing indexes. The hope was to identify necessary indexes merely by looking at the longest running queries.

Unfortunately there is no way to combine data from the missing indexes set of DMV’s and all the execution DMV’s that show query stats, execution plan, etc. None of the missing index tables has a plan handle or a plan hash column that would allow you to combine that data with the query data to identify which queries would directly benefit from the index if it were created.

But, if you look at the query plans in sys.dm_exec_query_plan, you can see the missing index information there. What to do? XQuery.

Since the query_plan is stored as XML, simply writing a small XQuery exist() function will do the trick:

FROM sys.dm_exec_query_stats s
   CROSS APPLY sys.dm_exec_query_plan(s.plan_handle) AS p
WHERE  p.query_plan.exist(
‘declare default element namespace “;;
/ShowPlanXML/BatchSequence/Batch/Statements/StmtSimple/QueryPlan//MissingIndexes’) = 1
ORDER BY s.total_elapsed_time DESC

This is a pretty simple example, and yes, in reality you would not want to use SELECT *.  You would want to specify those columns that you were really interested in. Also, this allows you to get performance information from queries that show a MissingIndexes element in the XML of the showplan,  but I haven’t pulled the Missing Index data out and displayed it. That can be done, but I’ll leave it as homework for you for now (and because I’m still having a hard time with XQuery).

A quick nod to Tim Ford for the advice on the DMV’s. Please finish that book soon.

Permalink 10 Comments

SQL Server 2008 Management and Administration

February 3, 2009 at 8:50 am (SQL Server 2008) (, , , )

I just got a book in the mail from a friend, Todd Robinson, who was the technical editor. The name of the book is Microsoft SQL Server 2008 Management and Administration. The book was written by Ross Mistry and Hilary Cotter. I don’t know Ross Mistry and I think I met Hilary Cotter once, although I know who he is. However, since Todd was involved, I’m pretty sure this is going to be a high-end, must read. I just started and the initial chapter’s discussion on using Windows Server 2008 sparked a few questions for my local admin team. I’m looking forward to more.

Thanks Todd.

Permalink 2 Comments

SQL Server 2008 Upgrade Lab at Microsoft

January 14, 2009 at 9:11 am (SQL Server 2008, Tools) (, , , , , )

I got an invite to take part in a lab at the Microsoft Technology Center in Waltham. I took advantage of it. I’ll use this post to describe the experience so that anyone else with the opportunity will know what to expect. 
 They knew I was coming

They knew I was coming

First, you recieve a very explicit set of pre-requisites. You need to install the SQL Server Upgrade Assistant, a tool that Microsoft licensed Scalability Experts to create for them. You have to run this against a small database, >25gb. The tool backups up all the databases from the server (so you need to put it on to a test box, rather than try to move an entire production system worth of databases). It then starts a trace that captures all the calls made to the database. I spent two days working with one of my application teams to get a server set up, the app connected, and a good set of tests run on the server to capture about an hour’s worth of trace data. It was at no point hard to meet the requirements, it just took time to get everything set up just right. They recommend you single thread the trace, meaning, just have one user run all the tests. This is because, when run without any extra work, Profiler, which replays traces, is single threaded. This can lead to unrealistic errors, especially blocking and deadlocks.

Once I had everything, I went to Waltham (a two-hour commute… the less said the better) to run the tests. The lab set up was impressive. They had a seperate room for each of the four companies that sent someone to the testing facility. We had a solid workstation (running Windows 7 by the way, fun stuff) and a set of servers on a seperate lan inside each room. The servers were running on HyperV, Microsoft’s virtual server software. Unfortunately, we did run into a snag here. Each server was supposed to have 100gb of space to accomodate the copy of the database as well as a restore of it and some more room besides. The virtual machines were configured to run on a system that only had 140gb of storage to start with. I filled that with my database during the initial set up (I ran the processes on three servers simultaneously). That put us out of commission while the MS techs went about fixing everything (which they did, quickly). It was just the pain of being first.

MS provided nice work stations

MS provided nice work stations

The documentation on the labs was very complete. Few steps were left to the imagination. Any where that had ambiguity, a second set of documentation cleared up nicely. With one exception. They did want to you to restore the System db’s. It made sense to do it, but I checked both sets of documentation and it wasn’t there, so I thought, hey, what I do know, MS is on top of this… Wrong. Had to restart, again.

Once all the initial configuration issues were done, it was simply a matter of walking through the lab. The first step was to establish a baseline, so I played back the trace on a 2000 server. Then I did an in place upgrade to 2008 and ran the trace and an upgrade to a new install using a restore and ran the trace there.  All the results could then be compared.

Over all, it was a good session. Rich Crane, Rob Walters and Sumi Dua from Microsoft were very helpful. I picked up a few tips on doing upgrade testing and got to do it away from managers and developers, making quite a few mistakes along the way. Now maybe I can do it in front of them with fewer mistakes. I liked the Upgrade Assistant tool since I’m pretty lazy, but it didn’t do anything earth shattering that you couldn’t do on your own.

One tip worth repeating, if you’re using the Upgrade Assistant to capture a trace, it doesn’t put filtering in place. You can open the trace file, filter out the databases, by ID, that you don’t need, and then save a new copy of the trace file, just for the database you’re interested in. Thanks for that one Rich.

Gentlemen, you did a nice job. I appreciate your time and your help. Sumi, nice to meet you. Rob, good to see you again. Rich, thanks again for everything, great chatting and good to see you again as well. Rich and I used to work at a dot com “back in the day.”

Permalink 1 Comment

SQL Server 2008 Upgrade Whitepaper

December 8, 2008 at 7:33 am (SQL Server 2008) (, , )

One of my tasks for the coming year is to evaluate each and every SQL Server 2000 system, identify all the databases, the applications they belong to, and provide an upgrade to SQL Server 2008 cost estimate. I’ve only started reading this new white paper from Microsoft, but I can already tell it’s going to be a huge help.

Permalink Leave a Comment

Next page »