Home Patch Management List Charter Patch Management List FAQ Patch Management List Archive WSUS List Charter WSUS List FAQ Moderators Privacy Policy		Essentials of Patch Management Policy and Practice by Jason Chan, @stake Introduction A few years ago, patch management was barely a blip on the radar screens of most security and IT personnel. 'Install and forget' was a fairly common practice; once deployed, many systems were infrequently or never updated. Obviously, for a number of reasons, this approach is no longer an option. The rise of widespread worms and malicious code targeting known vulnerabilities on unpatched systems, and the resultant downtime and expense they bring, is probably the biggest reason so many organizations are focusing on patch management. Along with these threats, increasing concern around governance and regulatory compliance (e.g. HIPAA, Sarbanes-Oxley) has pushed enterprises to gain better control and oversight of their information assets. Add in increasingly interconnected partners and customers and the rise of broadband connections and remote workers, and you have the perfect storm that has thrust patch management to the forefront of many organizations' list of security priorities. It's obvious that patch management is a critical issue. What is also clear is the main objective of a patch management program: to create a consistently configured environment that is secure against known vulnerabilities in operating system and application software. Unfortunately, as with many technology-based problems, good, practical solutions aren't as apparent. Managing updates for all the applications and operating system versions used in a small company is fairly complicated, and the situation only becomes more complex when additional platforms, availability requirements, and remote offices and workers are factored in. Just as each organization has unique technology needs, successful patch management programs will vary in design and implementation. However, there are some key issues that should be addressed and included in all patch management efforts. This paper provides a technology-neutral look at these basic requirements. The tips and suggestions provided are rooted in best practice, so a given patch management program shouldn't be considered a failure if all items haven't been accounted for. Instead, use this overview as a means of assessing your current patch management efforts or as a framework for designing a new program from the ground up. Security and Patch Information Sources A key component of patch management is the intake and vetting of information regarding both security issues and patch release - you must know which security issues and software updates are relevant to your environment. An organization needs a point person or team that is responsible for keeping up to date on newly released patches and security issues that affect the systems and applications deployed in its environment. This team can also take the lead in alerting administrators and users of security issues or updates to the applications and systems they support and use. A comprehensive and accurate asset management system can help determine whether all existing systems are accounted for when researching and processing information on patches and updates. An organization should also have relationships with their key operating system, network device, and application vendors that facilitate the timely release and distribution of information on product security issues and patches. These relationships can range from weekly or monthly calls with the account manager to simple subscriptions to the vendor's security announcement list. In addition, public web sites and mailing lists should be regularly monitored. Such information sources include Bugtraq, the various SecurityFocus Focus lists, and patchmanagement.org. Patch Prioritization and Scheduling Several scheduling guidelines and plans should exist in a comprehensive patch management program. First, a patch cycle must exist that guides the normal application of patches and updates to systems. This cycle does not specifically target security or other critical updates. Instead, this patch cycle is meant to facilitate the application of standard patch releases and updates. This cycle can be time or event based; for example, the schedule can mandate that system updates occur quarterly, or a cycle may be driven by the release of service packs or maintenance releases. In either instance, modifications and customizations can and should be made based on availability requirements, system criticality, and available resources. The second scheduling plan deals more with critical security and functionality patches and updates. This plan helps the organization deal with the prioritization and scheduling of updates that, by their nature, must be deployed in a more immediate fashion. A number of factors are routinely considered when determining patch priority and scheduling urgency. Vendor-reported criticality (e.g. high, medium, low) is a key input for calculating a patch's significance and priority, as is the existence of a known exploit or other malicious code that uses the vulnerability being patched as an attack vector. Other factors that should be taken into account when scheduling and prioritizing patches are system criticality (e.g. the relative importance of the applications and data the system supports to the overall business) and system exposure (e.g. DMZ systems vs. internal file servers vs. client workstations). Patch Testing Ideally, the breadth and detail of an organization's patch testing will relate directly to the criticality of systems and data handled and the complexity of the environment (e.g. number of supported platforms and applications, number of remote offices). The patch testing process begins with the acquisition of the software updates and continues through acceptance testing after production deployment. The first component of patch testing will thus be the verification of the patch's source and integrity. This step helps ensure that the update is valid and has not been maliciously or accidentally altered. Digital signatures or some form of checksum or integrity verification should be a component of patch validation. This signature should be regularly verified, especially as an update is passed through an organization's technology operations (e.g. on the update server, in build images, in software repositories). Once a patch has been determined valid, it is typically placed in a test environment. While the perfect test environment will mirror production as closely as possible, it is important to at least account for the majority of critical applications and supported operating platforms in your patch testing infrastructure. Many organizations will use a subset of production systems as an ad hoc test environment; department-level servers and IT employee systems are typically used in these cases. Regardless of the available test equipment and systems, exposing the update to as many variations of production-like systems as possible will help ensure a smooth and predictable rollout. The actual mechanics of testing a patch vary widely by organization. This testing could be simply installing a patch and making sure the system reboots, or the test procedure could involve the execution of a battery of detailed and elaborate test scripts that validate continued system and application functionality. In the end, a suitable approach toward detailed patch testing will be dictated by system criticality and availability requirements, available resources, and patch severity. The initial phases of production rollout can be considered an additional component of the testing process. Rollouts are often done in tiers, with the initial tiers often involving less critical systems. Based on the performance of these stages of the patch deployment process, the entire environment will be updated, and the testing process can be considered finished with the completion of final acceptance testing. Change Management Change management is vital to every stage of the patch management process. As with all system modifications, patches and updates must be performed and tracked through the change management system. It is highly unlikely that an enterprise-scale patch management program can be successful without proper integration with the change management system and organization. Like any environmental changes, patch application plans submitted through change management must have associated contingency and backout plans. What are the recovery plans if something goes wrong during or as a result of the application of a patch or update? Also, information on risk mitigation should be included in the change management solution. For example, how are desktop patches going to be phased and scheduled to prevent mass outages and support desk overload? Monitoring and acceptance plans should also be included in the change management process. How will updates be certified as successful? There should be specific milestones and acceptance criteria to guide the verification of the patches' success and to allow for the closure of the update in the change management system (e.g. no reported issues within a week of patch application). Patch Installation and Deployment The deployment phase of the patch management process tends to be where administrators and engineers have the most experience. Installation and deployment is where the actual work of applying patches and updates to production systems occurs. And, while this stage is the most visible to the organization as a whole, the effort expended throughout the entire patch management process is what dictates the overall success of a given deployment and the patch management program in total. The most important technical factor affecting patch deployment is likely the choice of tools used. One key distinction between patch tools is a common system development issue - to buy or to build? Historically, many organizations have created custom solutions using scripting languages combined with available platform tools to distribute and apply patches. As the industry has matured and the need for comprehensive and automated updates has increased, many tools have become available to help manage the patch application process. These tools are often classified as being either agent-based or agentless systems, depending on whether they rely on software being installed on the target systems that are to be patched. Additionally, many existing system management tools have the capability to perform software and system updates. The correct choice of patch management tools for any organization depends on a number of issues, including: the number of platforms supported, the number of systems to be patched, existing expertise and personnel involved, and the availability of existing system management tools. While applying patches, and especially security updates, in a timely manner is critical, these updates must be made in a controlled and predictable fashion. Without an organized and controlled patch application process, system state will tend to drift rather quickly from the norm and compliance with mandated patch and update levels will diminish. In general, users and even administrators should not be permitted to apply patches arbitrarily. While this should be addressed initially at a policy and procedure level (e.g. with acceptable use policies, change management, and established maintenance windows), it may also be appropriate to apply additional technical controls to limit when and by whom patches can be applied. The type of controls enforced will vary by organization and requirement, but include items such as restricted user rights (the user does not have sufficient permissions to update the system) and network-based access controls (the system cannot access the resources needed to perform an update, for example Windows Update or RedHat Network). In smaller organizations, automated, user-driven tools such as Windows Update may be acceptable. However, groups that use these update methods will likely need to rely heavily on policy guidance and enforcement along with regular assessment to ensure that organizational goals for patch and configuration compliance are met. Audit and Assessment Regular audit and assessment helps gauge the success and extent of patch management efforts. In this phase of the patch management program, you are essentially trying to answer two questions: What systems need to be patched for any given vulnerability or bug? Are the systems that are supposed to be updated actually patched? The audit and assessment component will help answer these questions, but there are dependencies. Two critical success factors are accurate and effective asset and host management. Often, these related goals of asset and host management are addressed by a single product, such as with Tivoli, Unicenter, or SMS. The major requirement for any asset management system is the ability to accurately track deployed hardware and software throughout the enterprise, including remote users and office locations. Ideally, host management software will allow the administrator to generate reports (e.g. all clients without a given hot fix, all versions of particular applications) that will be used to drive the effort toward consistent installation of patches and updates across the organization. System discovery and auditing are also components of the audit and assessment process. While asset and host management systems can help you administer and report on known systems, there are likely a number of systems that have been either unknowingly or intentionally excluded from inventory databases and management infrastructures. System discovery tools can help uncover these systems and assist in bringing them under the umbrella of formal system management and patch compliance. Organizations typically use either their own discovery and assessment mechanisms or one of the various managed vulnerability assessment tools. Regardless of the tools used, the goal is to discover unknown systems within your environment and assess their compliance with organization update and configuration guidelines. Consistency and Compliance While the audit and assessment element of your patch management program will help identify systems that are out of compliance with your organizational guidelines, additional work is required to reduce non-compliance. Your audit and assessment efforts can be considered 'after the fact' evaluation of compliance, since the systems being evaluated will typically be already deployed into production. To supplement post-implementation assessment, controls should be in place to ensure that newly deployed and rebuilt systems are up to spec with regard to patch levels. System build tools and guidelines are the primary enforcement means of ensuring compliance with patch requirements at installation time. As new patches are approved and deployed, build images and scripts should be updated so that all newly built systems are appropriately patched, and associated build documentation should be updated to reflect these changes. In addition to updates to build tools and documentation, operational procedures must exist to facilitate ongoing compliance of newly built systems. If an engineering team typically builds servers (e.g. with the base operating system and applications) and a separate operations team then assumes management of the system, a process must exist to funnel operational changes back to the build and engineering stage of the system lifecycle. These modifications are most ideally and suitably handled via an enterprise-wide change management system. Any new patches and updates that are approved and installed by operations should also be integrated by the engineering team into new builds, with the change management system providing both an appropriate audit trail and suitable procedural guidelines for this implementation. Conclusion While the issue of patch management has technology at its core, it's clear that focusing only on technology to solve the problem is not the answer. Installing patch management software or vulnerability assessment tools without supporting guidelines, requirements, and oversight will be a wasted effort that will further complicate the situation. Instead, solid patch management programs will team technological solutions with policy and operationally-based components that work together to address each organization's unique needs. January 31, 2004
			PatchManagement.org website hosted by Shavlik Technologies, LLC