I made a database blunder a few years back at a place I call Heavenly Hannover — and my manager sent out an email to the DBAs saying,

We made a mistake that requires a restore . . .

That’s what a manager should do — but now it’s my turn to do what I should do.

I replied to all by saying . . .

While I greatly appreciate the sentiment, “we” did nothing of the kind . . .

When I walked into his office a little bit later, he said, “I see you fell on your sword.”

Hannover’s SSIS work ranks with the best I’ve ever seen, but they had a key package that wouldn’t finish without adjusting the server memory. My manger asked me to look into it and I saw the issue right off. The culprit was a script component that was not the best way to aggregate 60 million rows. I removed it and used Window Functions in a procedure instead — which radically reduced the processing time.

I also discovered that they were 3 million dollars off in some key reporting.

Compounding the problem was a very long and convoluted procedure with no useful code commenting. While I could fix the issue, I had to be extremely careful not to create another one (especially since I planned to clean up the proc a good bit).

I met with the BA to review the business logic, and she wrote up some requirements to make sure we had it all covered. I ran both versions many times over a couple of weekends (logging every process to compare each state of my changes). I also wrote extensive code commenting (which took advantage of newer functionality that they had not been using yet).

We put it into production without a hitch.