tape and scissors

In July 2017, PayPal purchased a payment processing company called TIO Networks for $238 million. However, a data breach forced PayPal to suspend TIO’s operations in November 2017. Hackers had exploited a vulnerability in the TIO Networks system, and information on more than 1.6 million customers may have been stolen.

The compromised data included information such as bank account details, Social Security numbers and payment card information. That event led PayPal to experience financial losses and a damaged reputation, and it also forced the company to offer free credit monitoring to their customers.

In a similar case, Uber confirmed that that it paid two hackers $100,000 to delete stolen data in 2016. Hackers gained access to confidential information of around 57 million users that was stored on Uber’s GitHub account in October 2016. Uber tried to cover the incident up, but the news finally broke in November 2017. The compromised data included details such as names, email addresses and phone numbers of customers, as well as the driver’s license numbers of approximately 600,000 people in the United States.

There have been plenty of breaches like these in the past, and there will undoubtedly be more in the future. Organizations need to have a plan ready to curtail these breaches.

Unpatched systems are the key avenue that hackers exploit when trying to gain access to networks. There’s an ongoing cycle of identifying open-source security vulnerabilities, developing patches and applying these patches. With the proliferation of open-source software and use of open-source components in commercial off-the-shelf hardware and software products, it’s more important than ever to scan, detect and patch vulnerabilities.

Let’s review the patch management life cycle and points that organizations should consider while drafting a patch management strategy.

Patch Management Life Cycle

With the exponential growth of security vulnerabilities we’re experiencing, we need a more methodical and controlled approach to managing them. Here are steps you can take that will help you assess and prioritize risks, apply patches and design a patch management program.

1. Define the Scope of Patching

Organizations often neglect to patch all of their vulnerable assets, which is a big mistake that can lead to disaster. Whether your data is stored on-premises or in the public cloud, a well-managed and 100 percent complete asset inventory is critical for a good patching strategy. The inventory includes version details of all operating systems, firmware versions of the network and storage infrastructure, and many other details about all the devices in your data center.

An incorrect inventory can worsen your problems. For example, if you don’t have an exact count of the Windows 2008 servers running in your environment, how will you patch all of them if a vulnerability is identified? You can start by maintaining an asset inventory in a spreadsheet, but as the number of devices increases, you should consider implementing an asset management tool and an asset discovery tool.

2. Risk Assessment and Prioritization

If your organization is large, it will be nearly impossible to patch all of your devices simultaneously with every release of an update or a fix. You’ll have to decide which patches are your highest priorities (e.g., a security patch or hotfix) and apply the critical ones first.

For example, if you’re an ecommerce company, your marketplace is the most critical element of your business. Any vulnerability in your marketplace systems must be patched immediately.

If you don’t patch your system quickly enough, you might experience financial losses, either through loss of business or penalties from regulatory authorities. So segregation of services on the basis of business importance should be another deciding factor in your patching strategy.

According to the SANS Institute, services can be split into three categories:

  1. Mission-critical services (e.g., ecommerce websites).
  2. Business-critical services (e.g., PeopleSoft servers and applications).
  3. Business operational services (e.g., internal ticketing tools).

3. Change Management, Rollback and Communication Plans

Hastily applying a patch without involving your organization’s change management board (CMB) is a bad idea. Every patch in the production environment should be applied only after the CMB has approved it.

Typically, the CMB centrally analyzes all incoming requests and approves them based on their importance and the impact they will have on other applications and the environment was a whole. In the event of an urgent patching requirement, your organization should have an emergency process to follow to seek the CMB’s approval.

The CMB is also responsible for questioning you about any rollback plans you have for handling situations in which patching activity fails. They board members will ask questions about whether a full backup is in place, if the patch’s testing was carried out in development or in a testing environment, etc. They can deny the activity if its timing is inappropriate. For example, if you’re planning a critical network activity during business hours, the CMB can deny the change and suggest an alternative schedule so the impact can be minimized in case of activity failure.

How will you inform the stakeholders about patching dates and times, patching success or failure, and any delays in activity? Specific communication templates pertaining to patching activities should be designed and used for efficient communication. These templates can also serve as proof of the number of times you asked the application team for downtime and how many times they denied.

4. Establish Organizational Roles and Responsibilities

A clear-cut separation of roles needs to be explicitly defined and agreed upon. For example, you have to decide who will be responsible for backing up applications, communicating with stakeholders, seeking downtime, deploying patches and reporting.

With predefined roles, necessary permissions can be granted to different teams or individuals according to their responsibilities. Having undefined roles can lead to confusion and failure at the time of activity.

5. Test

How much testing is enough to deploy a patch in a production environment?

Are you ready to deploy the patch in your production environment?

These are tough questions, and the answers depend on multiple factors, such as these:

  • The application’s status. Is it mission-critical, business-critical or business operational?
  • The nature of your testing environment. Is it an exact replica of production, or a subset of production?
  • The amount of testing you will do once the patch is deployed. Will you be testing each function of the application post-deployment, or will you just be doing “sanity testing”?
  • The severity of the flaw or vulnerability being patched. Are you deploying a a security patch or a small bug fix?

If patching activity fails in production, there can be catastrophic results. Therefore, it’s extremely important to create a testing setup that is an exact replica of the production systems for mission-critical applications.

The chronology of testing in different environments, starting with the development/testing environment and followed by the production environment, is another consideration that you should include in your patching strategy.

Additionally, you should carry out thorough testing with properly designed test cases (automated testing is preferable).

You should present the results to the CMB when seeking approval for deploying the patch in the production environment.

6. Patch Deployment and Verification

After the risk assessment, approvals and testing are complete, it’s time to deploy the patch. One of the important decisions you will need to make during deployment is whether to patch manually or automatically.

For mission-critical applications and critical servers, you should always patch manually so the ground team can take immediate action if problems arise.

There are various tools on the market that you can use to apply normal updates and patches.

You should verify post-deployment system states and carry out a final round of testing. Ground teams should monitor the system closely for a predetermined period of time so they can act quickly if there are abnormalities.

Points to Consider for Your Patch Management Program

Here are a few important points to consider before you design a patch management program.

Speed: The speed of vulnerability patching is critical. For example, the WannaCry ransomware attack of May 2017 exploited a vulnerability in the server message block service. As the response to that case demonstrates, disabling the service quickly helps avoid attacks.

Potential damage to the system: Patching servers and applications comes at a price. Patching not only consumes the bandwidth of your resources, but also carries the risk of damaging your systems. Do a thorough analysis before you decide to patch a server. Is it a critical security patch or bug fix, or is it something else that you don’t really need?

Immutable infrastructure: If you’re running your workloads in the public cloud, or if you have a large number of servers at your disposal in an on-premises environment, you should adopt the immutable infrastructure approach, meaning you should set up a new instance rather than applying a fix or patch to an old server. One of the advantages of the immutable infrastructure approach is that it enables you to avoid configuration drift and abnormal behavior caused by an old setup.

Shift-left security: It is a good idea to embed security in every step of the software development life cycle and continually monitor the various software components. Moving application quality and security tasks closer to the developer (i.e., shifting them to the “left” of the delivery chain) can decrease the cost of fixing security holes later. There are tools on the market that can integrate with your development process and identify open-source vulnerabilities with each build run.

Automated patch management: If your budget permits, use an automated tool for deploying patches. Such tools can help you track the status of end-to-end testing, starting from the development/test environment all the way to production. It will make the process smoother and save time, energy and resources.

You Can Do It

Patching is an important activity. It helps minimize the risks that come along with outdated software and vulnerabilities. Determining whether you really need to patch a server, a network device or an application is difficult, but with proper assessment and prioritization, you can do it.

Sound vulnerability assessment and automation tools and a well-crafted patch management program can further enhance your security if they’re employed in each phase of software development. With these strategies on your side, you can ensure that your organization’s innovations will be secure.