went to linux world today (wednesday)

---

I haven't been to LinuxWorld in a long time. My co-workers Jake and Anthony were going up today, so I thought I would go along and take a look around. If nothing else, I thought, I could wander around for a bit and then write some code. Not much has changed in the last 5 years, it would seem, but I'm sure it serves its purpose.

Besides working at TurboLinux in 2000 and running Gentoo Linux on my laptop (and Geoff on his desktop), 2 years ago I started plans to move BugScan to an open source platform. Several customers were very adamant about their distrust of the security quality of the previous platform we were using, and I decided to act on that feedback. If we were going to move to <a href="http://www.kernel.org">Linux</a>, that also meant <a href="http://www.mono-project.com">mono</a>. Having used Linux in a similar commercial product setting before (WebShield at NAI) and worked for a Linux company (TurboLinux), I had some idea of the kind and amount of work we were in for, but there were some surprises.

At LinuxWorld, there was a KDE booth that had some copies of Xandros Linux Desktop they were "giving away" if you donated a suggested amount. My old college chum Cullen said he was using Xandros and liked it pretty well last time I spoke to him, and I thought it would be a good thing for my parenst to try. My dad only uses a web browser and email for his various autions and things, so I thought I would make a donation and get one. They had 3 different versions: a 'deluxe' desktop, a 'business' desktop, and a special version that came with skype and a USB headset/microphone. The last version was the coolest, but my parents don't really need skype. If someone were getting a new computer, though, I would totally recommend that version. For all of them, I was a little confused as to whether they could import bookmarks, email, and other things from Windows. I ended up getting the 'deluxe' one.

The guy at the KDE booth was adorable -- kind of shy, and apparantly a Scrabble champion. The Debian booth was next door, and when one of their representatives saw me looking at the Xandros boxes, he came over to harass the people at the KDE booth as to how much they were 'charging' for Xandros and its relative value to regular Debian. (Xandros is Debian based.) This person was extremely annoying and fairly hostile, reminding me quite a bit of the OpenBSD users I was resigned to dealing with at a previous job. After chasing off the Debian curmudgeon, I wished the Scrabble champion luck and then Anthony and I headed back to Mountain View. We walked Baxter to the train station and left Anthony there (Anthony lives in San Jose).

Back to moving BugScan to Linux. I had learned previously not to underestimate the difficulty of customizing a linux distribution, making the kernel work well with the chosen hardware, chasing down compiler+library bugs, in addition to making it all debuggable and supportable when trouble arises. We chose to use x86-64-based appliances for their superior performance and NX memory support (No eXecute, which helps reduce exploitability of stack-based buffer overflows).

Customizing the Linux distribution was only minorly problematic on the business side. I was able to hire my friend, Josh Rodman, who had previously worked at SuSe Linux in the support department. He was also a long-time Debian user and generally smart guy. I had originally wanted to use a SuSe-based bootable CD, mainly because I knew we were going to be using mono (BugScan 2.0 is written in C#), and partly due to Josh's previous experience. In the end, we couldn't get the details worked out with Novell in time for our development schedule.

We ultimately chose <a href="http://www.knoppix.org">Knoppix</a>. I knew I wanted to use a 2.6 kernel, and Knoppix 3.7 shipped with a decent one. (2.6.10 appears to be lucky for me on my Dell laptop and on our Sun Opteron hardware so far.) Of course, Knoppix is also primarily a bootable CD configuration, which is ideal for our needs. On top of that, it's based on Debian, which Josh had a lot of personal experience with. Customizing Knoppix and getting it to build automatically was easier than I thought it would be. My good friend <a href="http://www.canola-jones.com">Rodney Thayer</a> had warned me about building Knoppix, but Josh and Bryan were able to get it done on a SuSe 9/Opteron box pretty easily. We had bought the Knoppix Hacks book, which my friend Jake Applebaum contributed to, but I don't think we needed to refer to it (or him) much even though the book didn't have a lot of relevant content. In just over a week we had a 32-bit Knoppix booting on our Opteron hardware. w00h00!

The first step of customization was to remove packages we didn't need, which was easy. The next step of adding mono was non-trivial due to incorrect documentation of their build process. I'm going to skip that story for now and talk about customizing the Linux kernel for our hardware. Knoppix 3.7 shipped with 2.6.9, but 2.6.10 had some important x86-64 fixes that I thought we should take advantage of, and the kernel needed to be recompiled as 64-bit anyways. On top of that, I wanted the kernel to be stripped down so it would only boot properly on a very limited set of hardware that included our own. This turned out to be pretty trivial. 2.6.12 has some good stuff in it as well so far, we incorporated that once it was released. (2.6.12.3 worked fine on my laptop.) I look at the full kernel (and mono) changelogs to get a feel for what's relevant to us in the new version, and the SuSe guys are finding and fixing lots of good Opteron bugs. Sun sells SuSe support with the hardware we're using (SunFure v20z), so I guess it's part of SuSe's immediate testing and development rig. The <a href="http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html">kernel test matrix</a>

My previous experience with Linx on new-ish hardware wasn't nearly as smooth. Once at NAI (kernel 1.3.x) and later at TurboLinux (kernel 2.2.x), there were several times a full-time kernel engineer were very necessary just in reactive situations. For instance, under a ping flood, the kernel crashes; so you have to figure out why, fix the bug, and get it into the mainline kernel sources. The latter can sometimes be difficult for someone new to Linux kernel development in getting the code styling right and working with Linux folks on their mailing list. Besides the fact that vanilla kernels are generally good to go out of the box, this whole process is much easier now. Even so, free software is still not really "free" for use in this way. At the very least, you have to pay close attention tto the daily ChangeLogs and bug reports for the critical open source packages you integrate/ship/use and integrate any relevant changes. About 15 minutes of my work day is spent going through mailing lists, bug databases, and cvs/svn commit logs for this purpose. The diligence has paid off and it very much worth it.

Josh left the team and now Jake is working with us to help automate the build and tests a bit more than they already are. He is working out really well so far, and has come up to speed on quite a bit of the technology like mono and NAnt really quickly and with enthusiasm. I'm looking forward to the point when his work has the ripple effect of closing some quality and testing gaps we currently have. Even with pair programming, test-driven development with both unit tests and system tests, and as much testing as I can muster, there is still more to do. Hard to believe, but even all of these things don't equate to a silver bullet. Many customers and prospects have given us great detailed, actionable feedback that has allowed us to make these strides effectively.

Discussion: showing all 0 messages

(No messages)

>

Post a new message:

---