Software development is complex and products are often rushed to market, making it virtually impossible to release perfect code. It's a safe bet patch management is a crucial part of your midmarket company's IT strategy and that you spend a good deal of time patching problems. But what if all that patching does some breaking?
Your first instinct may be to apply security patches immediately, but that may break production. Sometimes a workaround is the best approach. Your organization should address this in its policy for applying patches, and remember that workarounds should always be considered an exception to policy, and not a substitute.
 |
| Patch management policy must-haves |
Your patch management policy should address the following:
- What sources to monitor for releases and updates
- Who reviews and evaluates importance or urgency of patches
- Who tests to ensure compatibility
- How users are notified
- How to roll out patches
- Verification of success
- What users should do to ensure compliance
|
|
|
|
|
When workarounds should trump patching
The first reason to pursue a workaround is when it's been publicly announced a patch is problematic and will break production. For example, in July 2008 Microsoft released a security patch that essentially locked out ZoneAlarm users from the Internet. Oops. Although workarounds were posted quickly, it didn't solve the chicken-egg problem of how do you get to the Internet to find the solution to not being able to get to the Internet?
If you run custom applications internally, chances are that a programmer somewhere took a shortcut to deliver the functionality you expect. At the time, port 31337 might have looked like a good way to communicate. When malware is subsequently released that has this port as its signature, security patches might block all traffic on that port, including your application. Not good. Hence, another good reason for testing.
The second reason to pursue a workaround is that you may be at a critical phase of operations and can't risk downtime. For example, it may be the end-of-the-quarter sales rush, or just prior to a major deadline (imagine rolling out a patch on April 14 that breaks tax filing software). Important deadlines mean no surprises. Sometimes you have to make a risk-based decision to live with a security vulnerability long enough to complete an essential deliverable.
The third reason is that you just might not have the patch. Consider the ZoneAlarm example above; if you can't get to the patch, you have to entertain a workaround. It's important to stay current on application mailing lists and websites. Occasionally, vulnerabilities are discovered, posted, and near-term workarounds are recommended until a vendor releases a formal patch sometime in the future.
Workaround policy exceptions can become policy
There seem to be enough instances of exceptions to policy that they should become, well, a policy unto themselves. Take time to determine specific cases when a workaround may be needed (e.g., you rely heavily on custom applications that always seem to break when the latest security patch is installed.) You won't be able to precisely identify 100% of the cases, but if you can meet the 80/20 rule, you're ahead of the game.
I keep mentioning policy, and that's because it really is your playbook. It keeps your actions aligned and understandable. If you're on vacation, others can figure out what to do. Most importantly, it avoids constantly trying to wing it, and formalizes best practices that have been known to work more often than not.
Patch management options have varying degrees of reliability
In smaller organizations, you might leave users to their own devices. This is extremely cheap to implement, but also the least reliable solution. It usually doesn't work. People tend not to go looking for fixes, and even when downloaded automatically, often procrastinate. For example, the Navy and Marine Corps Intranet (NMCI) patch system allows users to defer patches up to three times before forcing it upon them. Most everyone I know uses all three delays, and would take many more if available.
If you're all in a single office or one building, you might walk around and supervise patch installation manually. This is time-consuming, but you absolutely know that it gets done. It also makes you visible to users, and helps reinforce the importance of patch or workaround implementation. After doing this once or twice, your user community may be sufficiently sensitized (and educated) to the process, after which you can…
Push patches to clients. There are many making a prioritized list of applications based on importance to the enterprise (and getting management to approve), you'll have a firm foundation on what to work on first when everyone starts yelling at once.
Assign responsibilities by either job description or name, and ensure that you have workarounds (i.e., alternates) trained and ready to assume duties in the event the primary person is unavailable. Airline pilots routinely let their co-pilots land the plane; you should be training your co-pilot as well.
Finally, constantly evaluate the effectiveness of your policy and its implementation. Track the number of times you were able to roll out patches without adverse consequences, and the occasions you needed to apply workarounds. You may start to see patterns that help you instinctively know when to resist the urge to patch immediately and head first to the test environment. Your expertise in patch management may make you a hero someday, so start working on that policy now.
G. Mark Hardy is president of National Security Corporation, and is the author of more than 100 articles and presentations on information security.
Send comments on this technical tip to editor@searchmidmarketsecurity.com.
