My organization is about to purchase an IDS device and we've narrowed down the choices to a couple of vendors. Would you recommend trying to get demo versions and testing each of the boxes ourselves, side by side? Or since we know the products have similar features, should we rely on benchmarks and just focus on getting the best price?
As I lay out in my Buying Security Products Guide, having multiple options that you know can do the job is important. Since intrusion detection (IDS) is a mature technology, there will be limited differences among commercial products. In fact, if you are only looking for IDS functionality, then you should probably look at the open-source tool Snort. It's consistently rated as a top 3 IDS offering, and the price is right (it's free).
Generally speaking, I'm not a fan of "benchmarks," product reviews or product certifications when making buying decisions. Those resources can certainly help security professionals understand what they need to look for and help to compare products at a surface level, but there is no substitute for installing and testing a product to be sure it works for a specific organization.
In this case, if time is not an issue, I would suggest implementing the products in your organization's lab. It's difficult to know how a product will perform in your environment until it's properly tested. You may hate the user interface, find the updating process ponderous or have some other trouble that should've been recognized before a check was signed. Unless the products are tested, it's impossible to know.
Once it's determined (hopefully through lab testing) which of the products will work in your environment, then it's time to start negotiating for the best price -- but don't let the cart get ahead of the horse. A organization needs to make sure the product will work before worrying about saving $10.
