MCF: a Malicious Code Filter

R.W. Lo, K.N. Levitt, R.A. Olsson

The goal of this research is to develop a method to detect malicious
code and security-related vulnerabilities in system programs. The
Malicious Code Filter (MCF) is a programmable static analysis tool
developed for this purpose. It allows the examination of a program before
installation, thereby avoiding damage a malicious program might inflict.
This paper summarizes our work that led us to develop MCF. We investigated
and classified malicious code. Based on this analysis, we developed a
novel approach to distinguish malicious code from benign programs. Our
approach is based on the use of tell-tale signs. A tell-tale sign is a
program property that allows us to determine whether or not a program is
malicious without requiring a programmer to provide a formal
specification. We generalized program slicing to reason about tell-tale
malicious properties. Program slicing produces a bona fide program-a
subset of the original program behaving exactly the same with respect to
the realization of a specified property. By combining the tell-tale sign
approach with program slicing, we can examine a small subset of a large
program to conclude whether or not the program is malicious. We
demonstrated the capabilities of the tell-tale sign approach and program
slicing to detect some common UNIX vulnerabilities. We determined how our
basic approach could be defeated and developed a countermeasure-the
well-behavedness check. Static analysis produces inaccurate slices on some
programs. The well-behavedness check applies flow analysis and
verification techniques to identify such problematic cases.