Unix privileged programs (e.g., setuid root programs) have contained numerous vulnerabilities enabling outside intruders to enter the system and normal users to obtain root privileges. With current misuse detection techniques, we need to know the vulnerabilities before we can detect intrusions that exploit them. However, these vulnerabilities are often very subtle and are very difficult to find.
We propose a specification-based approach to this problem. It is called ``specification-based'' because it involves writing security specifications for privileged programs. The specification of a program captures the desirable behavior and is written based on the functionality of the program, the system security policy, and security concepts. The specification is then used for monitoring of the actual execution behavior of the program. Behavior violates the specification is reported.
We developed a prototype Execution Monitor and the specifications for the problematic privileged programs in Unix. We tested the prototype with several known attacks; our Execution Monitor can detect them instantaneously. In addition, the specifications of the privileged programs are precise and concise.