So, here's my first post.

While at a previous company, I became enamoured with Extreme Programming and Agile Software Development . (Extreme Programming is a dumb name, which is why I call it "XP".) It didn't really fit into the culture there, so it wasn't a particular success. I did learn a lot about how people react to some of the methodologies, and how things can go wrong.

People mainly have an immediate knee-jerk reaction to pair programming. Some people even have a little prepared speech about it that they recite every time the subject comes up. To avoid that conflict, we tried having a "buddy" system where people could check in code and then had to have someone else review the diff and give the okay to the source control admin. I don't think most people were really doing anything, because I found insanely obvious bugs looking at the diffs myself. Oh well.

Then we tried writing unit tests. This is a rather large pain in the nuts if the code isn't structured for testing individual functionality and it's an even larger pain when that code isn't in a language that support reflection-like capabilities (read: C++). I hired my friend as a contractor to help with QA, and set him loose on cppunit and the C++ codebase. cppunit does the best it can, and it's still quite painful. On the upside, the core objects were relatively testable (due to the architect having modelled then during design, I imagine) and once we got over the initial hump, we found 5 hardcore functionality bugs (mainly relating to copy constructors not copying everything they should) for the 20 unit tests we wrote before his contract was up. I was mainly "porting" the product to VC++ 2002, fixing bugs the new debug CRT was finding, and optimizing the use of STL (this means not using std::string.c_str() and std::string(char*) six times when it wasn't necessary in the main performance critical loop. These optimizations turned into a screaming success. Packet output went from 300 bytes per second to 20,000 bytes per second. Initially I thought I had broken something. The unit tests and WinRunner tests along with a manual regression test proved otherwise.) The CTO and another developer helped review my changes and caught a few bugs that my usage of PC-Lint, Purify, and Insure++ didn't. I don't know if they ever released my improvements to the customers or not.

A new product was being written from scratch with several new team members, using C# and .NET. I had been learning ASP.NET as a research endeavor and using NUnit as part of that. I really liked the idea of unit testing, but I didn't totally understand all the intricacies of test-driven development. How do you deal with private methods? What about object dependencies? I still thought some unit tests were better than none at all, and I presented my ideas to the group at that time. Oddly, even the CEO was interested and asked for a demonstration. So the new project went on (I was generally not directly involved for various political reasons), and unit testing took a back seat. They did do pair programming most of the time, though some team members thought themselves above that. These same people also liked to rewrite other people's code without telling them, causing builds to fail, general havoc to ensue, etc. I call this "cowboy" coding. It also generally involves massive checkins with little or no comments, constantly wanting to rewrite the entire universe, and being very territorial. Some people can pull this off and produce great products. And some people cannot. Anyways. As the schedule slipped and slipped, unit testing took more and more of a backseat. I quit before they ever shipped or sold anything of the new product.

What I learned from this is that unit testing has to be part of the routine, regardless of if you're in crunch mode or not. In fact, the previous release I was involved with at that company was so stressful with such long hours that people were introducing more bugs than they were fixing near the end. (Code reviews helped some, but not totally.) Without tests to back things up, things degraded rather quickly. Even with tests, if the code coverage isn't very high, the confidence degrades even further. It makes you afraid to change anything because it might break something. So you get this sort of paralysis where you start rationalizing why you shouldn't fix a bug since it might introduce several more in the process. This is not an engineering or business situation anyone wants to be in, but I've seen it happen over and over again across many different companies.

When I quit that job, I took a small break and developed the first BugScan prototype in February of 2003. It took 2 weeks to get things going, with the backend written in C/C++ and the frontend being C#/ASP.NET (which I fucking love). I showed the prototype to the CTO of the previous company, and we decided to start a company around a product. When I started development, I totally threw away the prototype and started from scratch. (Not getting attached to code and throwing it away when appropriate is another XP thing.) I had stupidly convinced myself that I didn't have time to do Test-Driven Development (TDD), and since I was the only person coding, no pair programming was going on either. This didn't turn out to be too bad since I'd done QA for 7 years or so and done a lot of code and design reviews, so I tend to be pretty thoughtful and meticulous. This was the first full-time coding project I'd ever done, so I did make quite a few dumb mistakes that mostly related to not doing TDD. For details on those mistakes, see my BlackHat Windows 2004 slides .

I quickly learned my lesson and in March stopped all my development and wrote unit tests for everything. I refactored a lot of common code out altogether, took things out of the UI and put them into objects as I should have, and generally cleaned house. I then started work on a new feature totally using TDD. It was fucking awesome. I made an object that stood alone and was able to verify all of it's functionality with unit tests and didn't even begin to write the UI for the feature until it was all in the object. Later when I went to optimize something, one of my tests started failing. Fuck yeah. The one thing I had trouble with still was figuring out where to start. I'd read the Adventures in C# article on XProgramming.com, and that helped give me some perspective, but it's different when you have a blank page staring you in the face. I started with passing nulls, empty collection, and things like that into the interfaces and using NUnit's <code>[ExpectedException]</code> attribute, and that got the ball rolling.

When I hired my friend again for some contract work on the ASP.NET frontend and had him do TDD. He needed a little coaching, but got the hang of it pretty quickly. I was doing work on the analysis engine (adding the buffer iteration detection feature), but I still wanted to have a code review for a given feature before he checked things in. This combination worked out pretty well, and this is what I plan on doing for the new contractor we're about to hire as well. I firmly believe in getting the fuck out of people's way so that can do their work, unless they prove themselves unreliable. With TDD, you don't have to wait for QA/support complaints or slipped schedules before you smell what's going on.

Recently I picked up a really excellent TDD book (geared toward Java, but very applicable to any OO language) called TDD: A Practical Approach . The main thing this book opened my eyes to is usage of Mock Objects . I had heard of them before, especially among the design patterns XP wonks, but I didn't totally grasp how to make effective use of them. This book's introduction of Mock Objects usage as a manual process and then showing tools like EasyMock got me really excited. This completed the TDD/unit testing picture for me, and I immediately thought of where I could use it in my code right now. It also drove home the "favor composition over inheritance" mantra I'd been hearing in a very real way.

So I read the Mock Objects portion of the book last night before going to bed, and was totally jazzed this morning to find a .NET version. I quickly found EasyMock.NET . Hooray! Looking, they had switched from integrating with NUnit to integrating with csUnit. BOO! csUnit is one of those useless forks that happens in the opensource community. I tried to find a compelling reason to switch to csUnit, but was scared off when I saw they had mass checkins with generic comments. This is what XP and Refactorig types call a "smell". So I looked at the diffs of the EasyMock.NET code seeing that the changes between csUnit and NUnit were minimal. So I checked out the EasyMock.NET source from sourceforge, did a few Replace In Files (Control+Shift+H) in VS.NET, some manual cleanup, and got all of EasyMock.NET's unit tests passing with NUnit in about 15 minutes. Hooray!

I guess the thing that I see MO being useful for in the very near future is in reducing dependencies. For example, before a dataflow analysis object can be used, the program data needs to be supplied. That means writing a disassembly/reflection engine has to be done before any of the dataflow algorithms or signatures can really be checked out. With MO, I can define the interface (IDisassembly), and then use that mock object to supply "fake" program data to the data flow analysis engine. Once I see that code working, work can begin on the harder stuff. By using the object via unit tests this way, it helps me refine the interface and make it easier to use. This would also allow me to do work in parallel -- instead of the dataflow analysis team not being able to write any code until the disassembly team gets all their code fully working, they are able to parallelize their development quite a bit! This is just an example, by the way -- don't read anything into it.

There are some other really useful extensions JUnit has that NUnit doesn't at the time of this writing (in 2003), but I'm sure that's just a matter of time. There are xUnit tools for many other languages as well, but I'm told NUnit is the least due to it's use of C# attributes for annotation rather than inheritance and special method naming.

Back to work, now.