I think the run for penetration testing is about over. There are several reasons for this:

1. Instead of spending the money on pen testing, I see executives moving more toward education for system administrators and developers to help prevent and remediate the vulnerabilities in the first place.

2. The character and attitude of many penetration testers really doesn't fit within the corporate culture of most organizations. I'm not sure that the upper-leadership of many organizations (especially the more conservative ones) ever felt that they could trust their penetration testers (Are the wolves watching the hen house?).

It's interesting to think about the impications of this shift away from management of security problems and toward resolving them. If I were a security vendor, I'd be very concerned right now. It appears that most of the security products on the market attempt to help offset weaknesses in implementation of basic security practices (IDS/IPS, SEM/SIM, vulnerability and application scanners, firewalls, etc...) I'm not advocating that we get rid of these tools, but their role is greatly diminished if patch management, access control, system hardening, seperation of duties, use of coding best-practices, and other basic security best-practices are used.

So how are companies suppose to audit their production environments to see how these wonderful new Q&A tests are capturing defects before they are promoted?

I believe this article's view in narrow-minded that pen tests will be dead (or changed). Pen tests capture the entire end product state, where changes that made piece-meal to products may be ok by themselves. Pen tests look at the entire process and flow and should bring back recommendations to the QA group for remediation.

Maybe the way these guys were using pen tests will be dead, since the reason they were using them were wrong in the first place.

I have to disagree with the idea that penetration testing is dead. With the economic down turn, these projects may be delayed or smaller scoped to minimize costs, but they will never go away. Particularly in the public sector, current regulations require independent testing of controls that safeguard financial, client, and company information, i.e. control and technology audits. Unless these numerous regulations and industry standards are modified or repealed, external auditing is not going anywhere.

I took this article as an opportunity for this person to highlight his company and product by making an outrageous comment.

John Reeder
VP, Information Security & Compliance
CISSP, CISA, CISM, CIPP

This topic was the specific focus of my recent presentation at DEFCON. While the lackluster vuln-scans of 2004 (otherwise loosely dubbed as penetration tests) may in fact be a dead service line, the penetration test is still one of the few means by which to determine the effectiveness of security policies and deployed countermeasures. See the full video of my presentation at my blog.

Compliance is the WORST argument for pen-testing I have ever heard.

Every regulation can be met through alternate or compensating controls.

The demise of penetration testing is only one of the things Chess, Migues and I found in a current study still underway.

For more surprising results, see:

http://www.informit.com/articles/article.aspx?p=1315431

gem

There is so much I could pick-up on in there, but I’ll stick to the two main issues in there. The first was “Does Penetration Testing belong in the QA department?” to which the answer has always been “yes”. However, the discipline doesn’t exclusively belong to QA, it also belongs elsewhere, like audit/compliance too. This isn’t a popular opinion within the industry, but that’s ok. I don’t mind being unpopular; I’m not looking to be voted in as class president or anything.

The second issue was “is penetration testing dead in 2009?” and the answer to that is a resounding “no”. Now, before I ramble onward, it is only fair to point out the vested interests involved. I am the CEO of a company providing security services, such as penetration testing. And Brian Chess is the something-or-other of a company providing static code analysis tools; a QA driven product that is designed to find security flaws before they get to the real world. So, contextually, he is inclined toward thinking his approach is better (which is a trap that I, obviously, would never fall into).

Anyway, in an ideal world good design and development would eliminate all the flaws in a product before it gets to market. The reality though, is that even if you throw a huge amount of effort and money at this, then the best you will get is a reduction in the number of flaws, not elimination. The remaining flaws will still find their way into the products, and will be successfully attacked and exploited by people with a vested interest in doing so. It’s ok though, there is a tool in the corporate arsenal for identifying the flaws even after the product has been deployed; it’s called penetration testing.

But I would encourage you not to simply take my word for it; have a look at some of the statistics that have been released by others recently:

For Microsoft, a company that arguably has thrown more money at SDLC than any other in recent years, they have seen only a 50% reduction in flaws after years of effort. Not elimination. And not a “penetration testing is dead” milestone. http://blogs.msdn.com/david_leblanc/archive/2008/11/17/improvements-in-office-security.aspx

For companies considering swapping to a quality driven SDLC and looking to implement a static code analysis tool into this mix, the recent University of Wisconsin analysis (admittedly of only a few tools) makes interesting reading. http://pages.cs.wisc.edu/~kupsch/vuln_assessment/manual_vs_automated_vuln_assessment.html

Truth is: Conventional(network-based) Pen tests have become 'commoditized' with the average rate per day for a highly skilled pen testing gig down from $1500 per day to around $1100 per day, in some cases even $900 per day or less. Most organizations now own their very own 'pen testing and vulnerability scanning' solution using CoreImpact, Metaspolit, and traditional vulnerability scanning solutions like Qualys, Nessus, etc.

Highly skilled Application pen testers and security code reviewers will continue to still have a niche market to ply their trade to; focusing on Web 2.0 and application specific security issues.

Pen-testing: down but not out for the foreseeable future.

Be weary of security vendors bearing 'pearls of wisdom' and offering advise on the future; as very little of such advice in not pre-approved as part of bigger marketing plan and strategy ;-)

Penetration Testing of course is not going to die because of secure coding. And when we are saying secure coding of course we mean what we know now as attacking vectors in software. Buffer overflows, Format Strings, etc. New class of bugs might come soon if hackers and researchers try to be more innovating in their research methods and leave in the past the so loved and known class of bugs. New technologies are coming, with more complexity so you have to expect new things. Actually something tells me that new things are already out but most of us are still stucked with overflows...