Once you've answered the question of why you need intrusion prevention on your network, you next need to take a...
moment to examine your definition of the security you hope to gain from an IPS and state it explicitly.
- Are you talking about integrity and availability of your network?
- Regulatory compliance?
- Application security?
- Leak protection?
Each could be a valid component, but getting your team on common ground is a good step at this point. You'll also need to consider coverage---what kinds of protocols and applications will the IPS be responsible for inspecting and understanding, and at what depth.
While the vendors' marketing departments make great efforts to distinguish the underlying technologies, there are fundamentally three approaches in current IPS products: signature-based (including protocol anomaly) IPS, rate-based IPS, and behavioral IPS. While the leading products may include some pieces from all three approaches, each follows one of these fundamental directions, with the other two approaches being secondary and tertiary.
The important part of this step is to decide which is most important and appropriate for your application (refer to your "Why" statement here for continuity.) Each approach to intrusion prevention provides a variant level of protection, and sits in a different spot on your network.
Signature-Based IPS Catches Common Exploits; Has its Limits
Signature-based IPS dominates the market. These products are readily available and range from remotely managed service-based devices, to standalone high-performance IPS, to embedded IPS technology in firewalls.
Signature-based IPS products do not rely entirely on signatures to detect malicious or improper behavior. Many also include other detection technologies. For example, a detection technology good at catching zero-day attacks is protocol anomaly detection, which looks for application or TCP/IP behaviors that are either non-standard or far from the normal behaviors (such as an SMTP "recipient" address with 500 characters in it, or TCP packets with malformed options in them). Most signature-based products will include some protocol anomaly measures in their repertoire as a means of thwarting zero-day attempts.
Signature-based IPS technology is critical to catching and blocking common exploits, but it's also important to understand that it has significant limitations. A signature-based IPS is only as good as its signatures, and writing signatures is a difficult art, made still more difficult to evaluate since very few vendors actually offer open signatures that can be inspected. Although a mantra of signature-writers is to "block the vulnerability, not the exploit," the reality is that many IPS signatures are only good at catching well-described exploits, and do not necessarily protect against the underlying vulnerability. Because most systems see many different data streams as equivalent, long considered a desirable attribute of a well-designed and interoperable Internet application, many IPS signatures have an Achilles heel in their inability to identify every possible permutation of an attack that will exploit a vulnerability.
Even with all these technologies brought to bear, most signature-based IPSes are best at detecting use of common exploits (for example, by attackers simply trying tools they've downloaded from the Internet) and not as capable in blocking a true, targeted attack. If your main worry is attacks that might exploit unpatched and unprotected systems, signature-based IPS will block the script kiddies' attempts to compromise your systems, but not someone who has insider information and is intentionally trying to evade the IPS.
Rate-Based IPS Detect and Shield Against DDoS Attacks
Rate-based IPS works by closely watching the rate at which connections come into high-performance application servers, most typically Web servers. The primary goal of rate-based IPS is to mitigate and protect against denial-of-service attacks (whether intentional, or unintentional, as misbehaving software might be a likely root cause). Rate-based IPSes are definitely in-line devices, because they take an active part in monitoring, controlling, and filtering connections. Rate-based IPS products can detect simple overloads (such as too many connections over a short period of time, typical of a Botnet- originated DoS attack) as well as attacks based on half-open connections (such as those that try to fill application server process tables or firewall state tables with incompletely established connections).
The best rate-based IPS will actually step in and shield servers from bad connections during periods of stress by proxying connections to be sure that there is someone on the other end. More sophisticated rate-based IPS, appropriate for huge application server farms, offer a myriad of fine-tuned controls, but the basics of rate-based IPS can be built into any in-line IPS device or firewall. These technologies scale down very well and can easily protect midmarket businesses with Internet-facing servers from many types of denial-of-service attacks.
Since rate-based IPS is best aimed at the perimeter of the network, embedding the technology into firewalls is the most appropriate strategy for all but the largest of data centers.
Behavioral IPS Sniffs Out Bots on Your Network
Behavioral IPS tracks the flows and traffic patterns of a network. When these change, the IPS alerts the security manager and, in extreme cases, blocks or throttles traffic. Behavioral IPS is poor at detecting or blocking specific incoming attacks because most attacks, based on a specific data stream embedded in a normal protocol transaction, are not actually changes in behavior. However, these systems are very good at identifying systems that have become infected and are now attacking other systems and users, or which have become bases of operation for hackers.
Behavioral IPS offers an interesting view to network managers, especially in large, complex networks where the actual flows are not fully understood as a general rule. For that reason alone, many behavioral IPS systems have become valuable tools. However, behavioral IPS is barely comparable in its ability to actually prevent intrusions to rate-based and signature-based IPS, and solves very different problems.
Joel Snyder is a senior partner at Opus One, an IT consulting firm specializing in security and messaging.