As the foundation of the trusted computing base, the operating system kernel is a valuable target for attackers of a computer system seeking maximum control and privilege. Furthermore, because the majority of modern security solutions rely on the correctness of at least some portion of the operating system kernel, skilled attackers who successfully infiltrate kernel memory can remain undetected indefinitely.

In this dissertation, we present an approach for detecting attacks against the kernel's integrity (commonly referred to as "rootkits"). Our approach, which we call property-based integrity monitoring, works by monitoring and analyzing the kernel's state at runtime. Unlike traditional security solutions, our monitor operates in isolation of, and independently from, the protected operating system and has direct access to the kernel's runtime state.

The basic strategy behind property-based monitoring is to identify a set of properties that are practical to check, yet are effective at detecting the types of changes an attacker might make - both known and yet-to-be-discovered. In this work, we describe a practical and effective property for detecting persistent control-flow modifications in running kernels, called state-based control-flow integrity (SBCFI). Furthermore, to address those data-only attacks that do not violate the kernel's control-flow, we introduce a high-level policy language system for enforcing semantic integrity constraints in runtime kernel data.

To evaluate the feasibility and effectiveness of our system, we have implemented two property-based integrity monitors for the Linux kernel - one using a virtual machine monitor and the other using a PCI-based coprocessor. We demonstrate that property-based monitoring is capable of detecting all publicly-available kernel integrity threats while imposing less than 1% overhead on the protected system. We conclude that property-based kernel integrity monitoring can be both practical and effective.