Host intrusion prevention systems for both servers and end-hosts must address the dual challenges of accuracy and performance. Researchers have mostly focused on addressing the former challenge, suggesting solutions based either on exploitbased penetration detection or anomaly-based misbehavior detection, but yet stopping short of comprehensive solutions that leverage merits of both approaches. The second challenge, however, is rarely addressed; doing so comprehensively is important since these systems can introduce substantial overhead and cause system slowdown, more so when the system load is high. We present Rootsense, a holistic and real-time intrusion prevention system that combines the merits of misbehaviorbased and anomaly-based detection. Four principles govern the design and implementation of Rootsense. First, Rootsense audits events within different subsystems of the host operating system and correlates them to comprehensively capture the global system state. Second, Rootsense restricts the detection domain to root compromises only; doing so reduces run-time overhead and increases detection accuracy (root behavior is more easily modeled than user behavior). Third, Rootsense adopts a dual approach to intrusion detection - a root penetration detector detects activities that exploit system vulnerabilities to penetrate the security perimeter, and a root misbehavior detector tracks misbehavior by root processes. Fourth, Rootsense is designed to be configurable for overhead management allowing the system administrator to tune the overhead characteristics of the intrusion prevention system that affect foreground task performance. A Linux implementation of Rootsense is analyzed for both accuracy and performance, using several real-world exploits and a range of end-host and server benchmarks.
