In the essay “The Cathedral and the Bazaar,” Eric Raymond introduced the idea that “with many eyeballs, all bugs are shallow,” known as Linus’ Law, named after Linus Torvalds, the creator of Linux. While the concept may seem straightforward, not everyone agrees. Michael Howard and David LeBlanc argue in their book “Writing Secure Code” that most individuals lack the expertise to spot vulnerabilities.
A recent study conducted by the Coverity Scan project reveals that a considerable number of people indeed possess the skills to identify software defects. The research suggests that open source software matches or even surpasses proprietary software in terms of defect detection. The Coverity Scan project, initiated in 2006 at the behest of the U.S. Department of Homeland Security, evaluates various open source projects annually, such as the Linux kernel, to assess code security and quality.
The evaluation method employed by the Scan project focuses on defect density, calculated by dividing the number of defects by the lines of code. This metric allows for a meaningful comparison across projects of different sizes. The defects identified, including Null Pointer Dereferences and Memory Corruption, are classified as “high-impact” and “medium-impact” by Coverity’s Static Analysis scanning suite.
Coverity’s findings demonstrate that software like the Linux 2.6 kernel, PHP 5.3, and PostgreSQL 9.1 exhibit significantly lower defect densities compared to the industry average. The report also delves into the reasons behind the higher defect density of the Linux kernel, attributing it to the cautious approach of developers in altering stable code sections.
Unlike previous reports that focused solely on numerical data, Coverity’s 2011 report compares open source and proprietary software directly. The proprietary code samples used in the scan were sourced from various industries such as finance, telecommunications, and medical devices, with most applications having been in development for 5-10 years.
Comparing the defect densities of proprietary and open source software of similar scale, the study found that the quality of open source software matches that of proprietary software. Notably, safety-critical industries exhibited lower defect densities than non-critical sectors.
The key findings of the report are:
1. Open source software quality is comparable to proprietary code, particularly among projects of similar size.
2. Organizations that prioritize software quality through development testing witness improvements in code quality over time.
While the truth of Linus’ Law has been acknowledged by open source enthusiasts, Coverity’s analysis provides objective validation. The report urges a revision of the Wikipedia entry on Linus’ Law to include this evidence as a response to critics.
Image credit: IXS_1916 by acme
