My friend Luis said I should make a blog posting each month telling one of my "QA nightmare" stories. it was during these experiences that I quickly became aware of and proficient in static and runtime analysis, modelling, and reverse engineering for the sake of security QA. The names are left out to protect the guilty parties and the company this was at. I have several stories all at this same company -- tune in next month!

I was working on a firewall product at the time, which for some reason had some of the most evil and shitty developers on it I have ever worked with (this was 7 years ago now). I got the QA Manager responsibility dumped on me because everyone knew that it was doomed. This is usually an instance where I think "yay, a challenge!".

Since I got to be QA Manager and decide what to do (for the most part), I decided to focus on security QA since none whatsoever had been done up until that time on this firewall product (which was a combination of proxies and packet filtering). In a few days time, I had found 3 different bluescreen bugs (this firewall was on NT) just using nmap (yes, I'm lame).

The developer tried moving the bugs around in the database (starteam sucks) so they couldn't be found, but since I wasn't like some of the other QA zombies, I noticed. When I kept re-entering the bug, he then tried to discredit me saying I didn't know what I was talking about and that I was just holding up the product ship (which was still a month away at the time). When he finally admitted the bluescreen was real and in his code (I had gotten the VP involved at that point), he then made up some rrrrreaaally dumb excuses that said nmap wasn't complying to the TCP spec ("this tool isn't sending ARP replies!") and the packets wouldn't be routable. I knew he was just spouting nonsense, so I put the firewall on an external network and did the nmap on that external IP from home, and demonstrated this for the programmer (who was being more cooperative since I got the VP involved). So, the bug was real, in his code, and was remotely "exploitable" by doing a normal portscan on the external interface of the firewall. Sweet whiskey jesus. At least it failed closed (or did it?).

This developer then proceeded to refuse to fix the bug, and threatened to quit if things went further. No shit. When they said they weren't fixing it for release, I started looking for a job. I didn't bother to tell them until my manager asked me a week before ship why I looked depressed. When I told him I was looking for another job since I couldn't fucking believe a security company would ship a product in that state. It was negligence as far as I was concerned, and it would just take a couple of advisories to totally ruin the brand, choking what was supposed to be a major revenue stream for the business unit. I even went as far as to test Firewall-1 to see if it had the same problem, which it didn't. (It had a totally different class of problems that only seemed to cause usermode funkiness. I think I posted them somewhere anonymously at one point and someone lifted that for a bugtraq posting without crediting the original source.)

In the end, the bug was finally fixed. I don't know what they said to this developer (who is still working with the same VP, but at a different company as of this writing in 2003) to make him finally take responsibility. The bug was actually that there were no timers on entries in the TCP state table maintained by the firewall's NDIS driver. Table entries were only cleared if a FIN was sent, so a SYN scan would create a fuckton of entries that would never be free'd, eventually causing the driver to fill up all of the nonpaged kernel memory pool, causing a kmalloc() equivalent to return NULL and the return value wasn't checked. b00m.

So, that's the first QA nightmare tale. Well, not the first one chronologically.. the first post. Like it? Want to hear other ones? This isn't even the worst one, but this is the one that got me started on the path I am currently on. I'll post next about my first experiences with PC-Lint, PreFix, Purify, BoundsChecker, and Insure++ in 1998.