When One Testing Solution Isn’t Enough

Go to any conference, attend some webinars, or just do a search for application security testing solutions and you can quickly see the sheer number of solutions out there. As in every situation, there are some that are great and some that are not so great. With such great marketing, it is often very difficult to determine what is the best solution. All too often people are looking for that silver bullet. That one testing tool or pen testing company that will find everything. Unfortunately, that solution doesn’t exist, or I just haven’t seen it.

Application testing usually falls into two categories: Automated scanning and manual assessments. To break that down further, automated scanning is broken down into two disciplines: Static and dynamic.

Static Analysis

Static scanning means that code is scanned without running it. Here is a quick post about static analyzers. Inputs are traced through the code to see what paths are taken and determine vulnerabilities. Static scans can be efficient at finding flaws such as cross-site scripting, injection flaws, configuration issues, etc. Static scans excel at finding code issues like hard coded passwords, cryptography issues, and other items not possible through a dynamic or manual test. They still do not do very well at finding business logic flaws which can be the most devastating.

Dynamic Analysis

Dynamic scanning means that the application is scanned as it is running. Unlike a static scan, it can be very difficult to identify hard coded passwords or cryptographic issues because they are not visible to the end user. Dynamic scans do a decent job of identifying other flaws such as cross-site scripting, injection flaws, insecure cookies, and some other configuration issues to name a few. Dynamic scanners do not do a good job identifying authentication, authorization, or business logic flaws in general. Properly tuned scanners may be able to do this, but that requires extra work.

Manual Assessments

Manual assessments are mostly a manual process of analyzing source code or using the application to find security flaws. Source code review, has many advantages of the static scanning mentioned above. Penetration testing is very similar to the dynamic scanning mentioned. In both cases, the testing is done manually vs. using just an automated tool. Manual assessments (pen tests) are much more efficient at finding authentication, authorization and business logic flaw types. Of course the quality of the assessment is heavily influenced by the skill of the person doing the assessment.

Some of the big vendors have started offering all three types of assessment for their customers. The big players (White Hat Security, VeraCode, HP, etc.) all offer static, dynamic and manual assessments for your applications. And with good reason. If you have worked in an enterprise where you are required to have multiple assessments by different groups you quickly realize how nothing provides a complete solution. Each assessment paints a portion of the security picture. Items found by one may not be found by the other. In some instances you may think that two should have found the same item, but that is the nature of security.. No one finds everything.

Multiple Solutions

While you may think that it is overkill to have multiple scanning solutions or testers looking at an application, they all bring a different viewpoint and help identify different types of vulnerabilities. Sort of like an enhanced second pair of eyes. Not only can they double check the work the other has done, but in may situations they can identify different types of vulnerabilities. Not only do different solutions find different vulnerabilities, they also find them at different phases of the development lifecycle. It is common to use the static scanners in the development phase. Dynamic scanners are more commonly seen in the verification phase. Manual assessments, especially 3rd party penetration testers, are also in the verification phase, but near the end of the cycle. The closer to development a flaw is found, the quicker it is to remediate it.

The right solution is the one that fits your environment. Take the time to analyze your setup to determine the best way to implement a solution. These solutions take time and should be carefully considered.