YaK:: WebLog #535 Topic : 2006-03-06 17.21.59 matt : a test suite for code analysis [Changes]   [Calendar]   [Search]   [Index]   [PhotoTags]   
  [Back to weblog: pretention]  

a test suite for code analysis

The static analysis market is not innovating fast enough, and I think I know what can be done to help things along.

Static analysis vendors are getting away with way too much nonsense as far as I am concerned.

To this end, I would like to suggest an open, independent project to create a test suite so that consumers of code analysis products can :

  • objectively evaluate code analysis tools against one another
  • make sure any upgrade doesn't regress bug detection functionality
  • stimulate the open source community to meet the challenge
  • force vendors to make finding real-world exploitable bugs with low false positive rates a real priority

    There has been a research paper or two in this area already. This one titled "ABM: A Prototype for Benchmarking Source Code Analyzers" (http://vulncat.fortifysoftware.com/benchmark/abm-ssattm.pdf) by my old co-worker Tim Newsham (http://www.lava.net/~newsham/) and Brian Chess. While Tim is one of my favorite people from my time at Network Associates, I feel this paper is not as useful as it could be. First, Brian Chess works for Fortify -- a vendor of these tools. Second, the test suite wouldn't challenge the original lint -- it's extremely basic. The second, was a paper and a sort-of implementation called SecuriBench (http://suif.stanford.edu/~livshits/securibench/). SecuriBench would be great, except that it is java-only, doesn't explicitly tell what bugs should be detected, and has no eye toward automation. I had some email conversations with the author, but he did not seem interested in moving in the same direction I was looking toward.

    I'm looking forward to good discussions and ideas from CONSUMERS of these tools who know what they need and what the vendors are not currently providing. Vendors can consider this a free service.


    showing all 0 messages    

    (No messages)

    Post a new message:


  • (unless otherwise marked) Copyright 2002-2014 YakPeople. All rights reserved.
    (last modified 2006-03-13)       [Login]
    (No back references.)