Our company gets bids for penetration testing (the slang term is pen testing) all the time, and it's one of the least satisfying parts of the security business. But rather than riff about how bad pen testing is and how the results are so often misused, I'd like to encourage you to try some of your own penetration tests against your perimeter network security devices.
To get you started, let's pick one tiny piece of the picture: your antimalware tools. People usually start by asking "how well does my antimalware work?" That's a coverage test where you're looking to see how well your antimalware tool covers the attack space. Testing antimalware for coverage means throwing zillions of bits of badware against the defense and seeing what it catches. Most of us don't have the resources or patience for that kind of testing.
Let me suggest something different; step back and change the question to: Does my antimalware work at all? The answer may surprise you.
I like testing the effectiveness of antimalware in general, because this is a test we can do something about. If you find out that your antimalware tool only catches, say, 93% of the samples, how are you going to get that number up? There are not a lot of options short of changing vendors. However, if you do a validation test and discover that you've got a hole in your perimeter network security tools, this is often something under your control. You may find misconfigurations in your tools, or you may find that things don't behave exactly the way you thought.
Start your testing by getting a virus. If you're conservative, head to eicar.org and grab their sample test virus. Antimalware authors universally agree that they EICAR virus will detect as a virus -- but it's also completely harmless, just a text string. Working with live samples is exceptionally dangerous and I can't encourage it, although I will admit that I often get different results.
You should run these tests with your desktop antivirus turned on, and then again with it turned off. You think you have defense-in-depth? Let's find out for sure.
To do this testing, you will need a small Web and email server sitting on the Internet. An easy approach is to download a Unix-based virtual appliance with these tools preinstalled and leave it running at home with a static IP address. One very valuable technique I use is to have services running on both standard and non-standard ports. Since malware authors don't play by the rules, you shouldn't either. For example, when you test for POP protocol, run it on the normal port 110, but also on a non-standard port such as 1100 and on a port used by another protocol: 53 (the DNS port) is a particularly good choice, but so are 80 and 443 (the HTTP ports).
The goal here is to validate that the antimalware works, not how many viruses it can catch. So a single example virus is good enough. What we want to do is try and get that single virus into our organization through every single hole possible.
Now, think about all the vectors into your organization. Email is the obvious one, that's a push avenue. Try sending the virus to yourself from the server you set up at home to your normal corporate mail server. Now ZIP it, and retest. Now double-ZIP it. Now double-ZIP it and password protect the file. Try more unusual archive formats, such as RAR or GZIP. Renaming the file is a simple trick, but can help as well. It may be EICAR.COM, but try changing it to FOO.PDF, FOO.ZIP, FOO.DOC, FOO.CSV, FOO.TXT, FOO.JPG and FOO.DLL. You may end up with a couple dozen tests, but you may also discover some results that surprise you.
Continue testing any other push avenues. Can people on the Internet upload files to an FTP site at your company? Post them on a forum webpage (where others inside or outside could download them)? Attach them to an incoming Web-based customer support request?
Once you've exhausted push transmission from the Internet into your organization, look at the many ways your staff can pull data. The list is almost endless, but you should start with Web browsing. Put all of the test files, in all of their varying formats, onto a series of webpages. Now try and download them all. Run the Web server on port 80. Run it on port 443, unencrypted. Run it on ports 25, 110, 143, 53, 8080 and 7633.
Another nice testing strategy is to put the viruses in webmail. Most commercial webmail services will detect and block the virus, but if you install Squirrelmail or any other open source webmail tool, you can build your own webmail service and point it at the email server you just created. Try testing webmail -- don't forget those non-standard ports -- along with POP and IMAP as well as outbound SMTP on both standard and non-standard ports.
Some other hints on validating your own antimalware: Keep a notebook by your side (whether a paper one or electronic one) and make notes on each test you run. It's easy when you start talking about 50 or 100 tests to get lost in the details, but keeping notes on what did and didn't work can help you to see trends and analyze results.
If you find out something surprising, drop me a note and let me know what you learned about your own defenses!
Joel Snyder is a senior partner at Opus One, an IT consulting firm specializing in security and messaging.
Send comments on this technical tip firstname.lastname@example.org.