Got a whole lotta logs? Way down inside, you need them. Your network has a lot to tell you, and you're probably not listening to it all -- security, performance, the mundane and the exceptional -- a whole lot of information that most network managers throw their hands up at. Here are three tips on how to leverage your security event log data for greater awareness of the state of your company's security.
COLLECT LOG DATA IN A SINGLE PLACE
This seems obvious, but it should be your first step, and most of the companies I visit haven't done it properly. Start with a syslog server, which you can put on the operating system of your choice -- Unix or Windows. Generally, if you don't have strong religious feelings one way or the other, I recommend using Unix. The reason is simple: more built-in tools and less care and feeding. If you spin up any common Linux distribution, you'll find everything you need to get started.
The server doesn't have to be big or fast, and you can even use a virtual machine if that's convenient. About the only thing you need out of the ordinary is disk space, lots of it. Fortunately, that's about the cheapest thing you can buy today, with 1 terabyte (TB) drives coming in at the $100 range. You don't need RAID; you don't need SAS; all you need is a place to dump everything. (Of course, RAID and SAS wouldn't hurt, but don't let the high cost of super-fast, super-reliable disk space get in your way.)
Facility is simply a super-category, a way of dumping logs into large buckets so they can be filed, archived or searched easily. You should try and keep things together that make sense. For example, your mail server and antivirus/antispam gateway could log to a common facility, "mail" (see Common Log Facilities and Severities). In addition to about a dozen predefined categories, syslog messages can also be filed under eight different "local use" categories, so you can divide up things so they make sense.
Priority level isn't a choice, but a cutoff. Syslog messages are sorted from most important priority (emergency, level 0) down to a lowest level of importance (debug, level 7). In this case, smaller numbers are more important. Unless you are debugging a problem, you probably want to syslog everything with a priority of notice (level 5) down to emergency (level 0).
There is a missing piece, though: your Windows event logs. These are in Microsoft's own format, and of course there's no built-in tool to send Microsoft logs over to a syslog server. However, there are a number of open source agents that can be installed on Windows servers that will receive Windows Event Logs and retransmit them to a syslog server.
AUTOMATE MANAGEMENT AND REPORTING
Once you have logs flowing in, you'll want to put in place automated management and reporting tools. Depending on how much log data you generate, you'll want to roll over the log files periodically, and compress older log files. Weekly, monthly, or even daily if you get a lot of logs. Pick a time window that compromises between having a log file too large and having to pore through a lot of compressed archives to solve problems or answer questions.
You may want to segment different logs into different files. For example, you can separate email logs from firewall logs because you rarely look at both of them at the same time. This will also let you divide devices or servers which have a high logging rate from less verbose systems, which in turn simplifies log file management.
As you're thinking about your log management strategy, keep in mind that you're not just storing the logs to have them -- you want easy access for searching and reading them. Don't over-divide things (for example, by putting every log or every week in a different directory), because that makes searching through multiple log files more difficult.
Make your filenames self-explanatory, easy to type and easy to group. For example, you might name your mail logs "mail.yymmdd," if you divided them by the day. That's a lot more useful than "mail.dd-mm-yy," which looks nicer but doesn't make it easy to search subsets of the logs all at once. Since we're going for functionality over everything else, keep that in mind as you come up with an automation strategy.
AUTOMATED REPORTING BOOSTS VALUE OF CENTRALIZED LOGGING
Simply managing the logs is important, but having automated reporting is where the real value of centralized logging shows through. The goal of reporting is to generate information that helps alert you to problems and give you visibility into your security and system posture.
When I do this for a consulting client, I always start with two different reports -- one is looking for unusual or exceptional messages, and the other is to summarize counts so anomalies will show up.
Finding unusual messages usually means filtering out the usual messages. I turn to five common Unix applications (also available as add-on tools to Windows) to build small scripts to filter out the usual and look for the unusual (See Starter Tools for Processing Logs). You can find good tutorials for each of these on the Internet if you haven't already had some experience with them. Knowing what these little Swiss army knives of text processing can do for you is not just useful here, it's a way to save you time throughout the rest of your career.
I suggest you start by generating a few weeks' worth of logs. Then, work on your "interesting log lines" report. Using the log files you have as a starter, write some scripts that will sort through and weed out the most common lines until you're left with just the ones that are truly interesting for your report. For example, your firewall should be generating a log entry for every successful connection -- those don't need to go into the "interesting" report. Remember, you don't have to be brilliant or efficient at filtering out these logs -- this is something that's going to run daily or even just weekly, and if it takes 10 minutes to generate a report, it's not going to matter.
If you can get your daily or weekly "interesting" report down to a page of output or less, you're doing a good job.
The second report that I always start with is a count report -- trying to count log lines to get an idea of what is happening. For example, each connection through the firewall can be identified by the firewall rule that allows it, and looking at a nice information table that summarizes the rules and how many each rule allowed, is a great way to get a snapshot on how your network is being used. You may even want to edit your firewall rule set a bit to segment the traffic and get a slightly more granular report, such as separating servers from end-user systems.
You may need to add another tool, Perl, to your toolbox to generate this report. Yes, there's a language that was actually designed for generating reports from text. Perl stands for "Practical Extraction and Report Language," and for all its faults, reporting on large log files is one of the things that Perl does best. Perl has been widely misused in the world of Web applications to generate lots of insecure and buggy applications, but if you take Perl home to its roots, you'll find that it lets you do great things with your logs.
If you don't want to learn Perl, that's OK too -- you can still generate some very interesting reports using the simpler tools I mentioned above.
'YOU BEEN LEARNING, I BEEN LEARNING'
Many of the functions of log management and reporting I described in this tip are also present in commercial products, called SIMs (or SEMs or SEIMs), for security information management tools. As you add more important security elements to your logs, such as vulnerability management tools or intrusion detection/intrusion prevention systems, you may want to look into SIMs for their ability to automatically sort and correlate information across events.
However, as useful as SIMs are, don't feel that you can't do log management and get value out of your logs without one. Follow the steps laid out here, and you'll have a big head start on building your own tools to increase security awareness and visibility.
Joel Snyder is a senior partner at Opus One, an IT consulting firm specializing in security and messaging.
Send comments on this technical tip firstname.lastname@example.org.
This was first published in August 2009