Network Security Via Reverse Engineering of TCP Code:
Vulnerability Analysis and Proposed Solutions*


Biswaroop Guha and Biswanath Mukherjee
Department of Computer Science
University of California
Davis, CA 95616, U.S.A.

e-mail: {guha,mukherje}@cs.ucdavis.edu
Tel: +1-916-{785-5863, 752-4826}
Fax: +1-916-752-4767

November 7, 1995

Abstract

The Transmission Control Protocol/Internet Protocol (TCP/IP)
suite is a very widely used technique that 
is employed to interconnect computing 
facilities in modern network environments. However, there
exist several security vulnerabilities in the TCP 
specification and additional weaknesses in a number of 
widely-available implementations of TCP.
These vulnerabilities may enable an intruder to ``attack'' TCP-based
systems, enabling him/her to ``hijack'' a TCP connection or cause denial
of service to legitimate users. We analyze TCP code via a
``reverse engineering'' technique called ``slicing'' to identify
several of these vulnerabilities, especially those that are related
to the TCP state-transition diagram. We discuss many of the flaws present
in the TCP implementation of many widely used operating systems, such as
SUNOS 4.1.3, SVR4, and ULTRIX 4.3. We describe the corresponding TCP attack 
``signatures'' (including the well-known 1994 Christmas Day Mitnick Attack)
and provide recommendations to improve the security state of a TCP-based
system, e.g., incorporation of a ``timer escape route'' from every TCP state.


Keywords and Phrases: Network Security, TCP, IP, Reverse Engineering,
 Slicing, Vulnerability Analysis, State Transitions, Timer Escape Route.
 

*This work has been supported by the Advanced Research Projects Agency
 (ARPA) under Contract No. DOD/DABT63-93-C-0045.
 A short, summarized version of this paper appeared in the
 Proceedings of the IEEE Infocom '96 Conference.