Files
Abstract
Over the past decades, our society has become progressively reliant on system software. They are used in all sorts of different areas such as hospitals, nuclear plants and even to fly space crafts. One minor software vulnerability might be enough to lead to damage that is unrecoverable. Manual software testing, as the most prevalent technique used for software security analysis, is laborious and prone to human error. With the increase of our dependence on computer software, the desire of developing techniques that perform systematic check on the system software we use automatically for critical vulnerabilities increases.In this dissertation, we investigate the techniques of system software testing for identifying security vulnerabilities and then expand the system software testing techniques by addressing two problems: increasing code coverage and triggering vulnerability. We first explore the feasibility of identifying security vulnerabilities in the implementation of virtualization hypervisor by extracting the implementation of the virtual devices and modeling them with symbolic execution. Exercising virtual devices independently with symbolic execution engine without the full virtualization running enables the possibility of discovering vulnerabilities with the novel technique.In the second part of the dissertation, we show how different system software testing techniques can be integrated together to improve the effectiveness by investigating Sezzer, a framework that incorporates fuzzing and symbolic execution yet overcomes the major disadvantages from both of them. Our experiments on different benchmark binaries as well as real-world applications, shows that Sezzer not only outperforms modern system testing techniques in most cases, but also can find security-critical bugs in realworld applications. With the preliminary realworld testing, we have found 6 unique vulnerabilities in GNU-Binutils which resulted in 5 patches and 3 CVEs.