Business, as usual
No matter how intriguing the myths and conspiracy theories related to security testing might be, the scarce security testing resources stipulate pragmatism and business-driven approaches. In fact, a thing that came by surprise when I started doing product security is the diversity of sources that would guide the testing. For instance, given that Common Criteria and national security certifications are important for Tough Mobile customers, it´s important to support the evaluations and ensure that the penetration tests done by external laboratories are completed smoothly.
Prioritizing security testing based on threats is quite easy, theoretically. Just take the standard risk model, which states that the risk is equal to the likelihood (chance of a particular threat to happen) times the impact (severity of the consequences if the particular threat happens). However, one soon finds oneself wondering how to calculate the amount of likelihood and impact. DREAD, a popular mnemonic, could help in this task. It stands for Damage (how big would the damage be if the attack succeeded?), Reproducibility (how easy is it to reproduce an attack to work?), Exploitability (how much time, effort, and expertise is needed to exploit the threat?), Affected users (if a threat were exploited, what percentage of users would be affected?) and Discoverability (how easy is it for an attacker to discover this threat?). However, finding quantitative answers to these questions in practice requires a lot of imagination to estimate the unknown factors. This makes the prioritizing more of an art than an engineering task, especially if it is to be done is a reasonable time.
Different Shades of Grey
It has become already apparent that threats are not black and white but they have different shades of grey depending on which angle you look at them and who is looking. Therefore, some blunt facts are needed to tip the scales. What is the role of the test target in overall security posture? How are the customers using it and what are the governing security policies? What is the testing effort needed based on available resources and expertise? If there is a vulnerability found, can we mitigate it? Is there going to be code refactoring or design changes for the target in the near future? Can the testing be automated? What is the history of the tested code base and will it be maintained? What are the trends in vulnerability research? [3]
There seems to be a lot of work even before the actual testing can be started. So usually one has to increase contrast instead of defining the exact shade of grey. But none the less, defining the purpose of the testing is paramount since the persistent tester might put their whole soul in the search of the unknown.
From Black Box…
In addition of product security, I have been doing penetration and security testing for external networks and devices. It is usually done as black box (no prior information on the target) or gray box testing (some information e.g. architecture documents are used as prior information). Discovering information on the target (what technologies are used, how it is configured, what are the protected assets and even sometimes hints of the mind of the creators) through reverse-engineering and reconnaissance is exciting by its own right.
But this approach is even more open-ended than white box security testing (where you have the details of the most parts of the implementation available). Since it's hard to figure out how well you have done your job, e.g. in terms of test coverage, there is a constant pressure to complete the testing “honorably", e.g. by finding a critical vulnerability on the target. On the other hand this pressure is the fuel that could make one to push over the limits.
..to White Box
White box testing is a different ball game, but the lessons learned from black box testing are also valuable. For example, the white box targets tend nowadays be a mess of complex frameworks, build environments and interrelated dependencies. Therefore, sometimes even some basic parts of the code do not work as expected which can be discovered through a black box testing type of poking.
When I started doing product security it was no surprise how much more efficient the white box approach was. However, it did not mean that the testing could be done in less time although the same coverage could be achieved faster. When playing with open cards, there are also higher expectations for “honorable´ outcome. When the code is available to scrutiny, new testing techniques and tools such as source code auditing and static and dynamic code analysis are apparent. Fuzz testing (a testing technique using invalid, unexpected, or random data as inputs), which is also a crucial part of black box testers arsenal, could be utilized more efficiently through instrumentations that enable faster and coverage guided fuzzing as well as sanitizers that would reveal security issues that otherwise would be unnoticed.
Testing tools used in black box testing, such as vulnerability scanners, network traffic analysis and attack tools and Android application security scanners, are also valuable. One could in theory just rely on those tools (and related checklists) but usually the targets are so unique that the tools do not cover all the relevant aspects. Therefore, security testing is not always about breaking things but also about creating right tools for the job.
Enjoying the Journey
The security testing has been an enjoyable journey for me. Although the excitement of finding new vulnerabilities is still there after the years, the mix of learning new, challenge of open-ended questions, profound pondering mixed with myth and unknowns, pragmatism and co-operation makes the journey most enjoyable.
Well, that's a lot of topics that would deserve more pondering. But now I have to leave you with another quote from Dan Geer: "There is never enough time. Thank you for yours."
