Best Practices for Patch Management

Best Practices to follow when using Qualys Patch Management. The best practices explained in this article facilitate effectively deploying patches on your assets.

By effective patch deployment, we mean remediating your assets from vulnerabilities, which is achieved by running optimum patch jobs with fewer reboots and by avoiding hindrances in the end user’s day-to-day work.

The key points that enable effective patch deployment are:

  • Asset categorization by using proper tagging
  • Appropriate Patch Selection
  • Job creation according to the best-suited options for your organization’s requirements

At times, the patch deployment fails due to causes, such as system failure, environmental failure, or user issues. The details provided about these causes are helpful in troubleshooting.

Refer to the following sections: 

Asset Categorization by Using Proper Tagging | Static and Dynamic Tags | Multilevel Tagging | Guidelines for tag assignment | Appropriate Patch Selection | Job Creation | Patch Window |Time Zone | Job Failures and Retry Criteria

Asset Categorization by Using Proper Tagging

Proper asset categorization plays a vital role in patch deployment. You can assign a tag to one or more assets. See more information about types of tags and multilevel tagging, which enables you to categorize your assets and make the patch job light.

Static and Dynamic Tags

You can tag your assets by using static and dynamic tags. Assign dynamic tags to assets based on rule-based criteria, such as time zone, OS, IP range, etc. Use out-of-the-box static tags (Business Units, Asset Groups, Cloud Agent, and Internet Facing Assets) or create your own to apply tags to your assets manually.

Multilevel Tagging

Multilevel tagging works best in asset categorization. Create a parent tag and multiple child tags. You can include or exclude child tags to light the patch jobs.
If you apply the patch job based on the parent tag, the assets that fall under the parent tag also get patched.

 Based on a patch's vulnerability severity or criticality, you can choose to deploy the patch on the parent or child tags.

Guidelines for tag assignment

Consider the following suggestions to categorize your assets:

 All the following categories are suggested to best replicate the natural categories in your business context. Choose the ones that closely depict your organizational structure. Some organizations might need more than one category. Keep in mind that the categories are not mutually exclusive.

  • Geographical location - Individual asset time zone: Categorize your assets based on the geographical location. It will enable you to decide the patch deployment schedule, patch window, and so on. 
    Example: Assets from Country 1 from X time zone can be tagged as c1 and assets from Country 2 from Y time zone can be tagged as c2. The patch deployment can be done easily in the respective time zones using the said tags.
  • Irrespective of the Time Zone: Categorize your assets differently using appropriate tags if you don’t want them to be time zone-dependent for patching. You can create a patch job with a fixed schedule for such assets. This only matters in the case of scheduling jobs and manifest distribution. Organizations with less than 1000 assets need not worry about these. For larger organizations, this would matter, especially if it is being done from a Western office.
  • Workstations and Servers: In some cases, categorizing your assets as servers and workstations may also help. Activities, where the job focus changes on the asset type, would benefit from tags of these kinds. As server reboot after patch deployment might have a larger impact, you can schedule the patch deployment on servers and workstations, taking the work-related and business-related priorities into consideration.
  • Weekdays or Weekends: Categorize your assets, considering whether it will work best to deploy patches on weekdays or weekends on your assets.
    Example: You can deploy patches on servers & databases on weekends and workstations on weekdays.
  • Work hours or non-work hours: Categorize your assets based on whether it will work best to deploy patches during work hours or non-work hours. 
  • Tagging newly added assets: For newly added assets, most endpoints need many patches to be deployed. We recommend adding a separate “dynamic” tag to track such newly added assets. Also, you can consider including the initial patch deployment as a part of the asset onboarding checklist.
  • Language-specific variants: Some applications have special patches, which might not need to be rolled out to a larger asset group at the same time. Keep language-specific patches distinct for each language and tag the assets based on languages. Also, test such patches on a smaller group of assets before deploying them across all desired assets. 

Appropriate Patch Selection

Before looking into the pointers to select appropriate patches, see details about an assessment profile. 
An assessment profile plays a vital role in getting an insight into the missing and installed patches on the assets.
The administrators create the assessment profile, which is the default assessment profile. By default, your cloud agents scan for missing and installed patches at a specific interval that is defined in the default Assessment Profile.
At first, a default assessment profile is applied to all agents, which scans the assets at an interval of 24 hours for a free subscription and 4 hours for a paid subscription.
You can create a custom profile to change the scan interval.
Based on the insight received about the missing patches, you can select appropriate patches and deploy them on the assets. Refer to the following points when you include patches in a job: 
Search the latest (published) patches from the catalog: Decide patches that you want to add to a patch job based on the published date. 

  • Search Patches by Publish date and apply in reverse chronological order

publishedDate
Use a date range or specific date to find when patches were last published.
Examples:
i. Show patches published within certain dates
publishedDate: [2018-02-01 ... 2018-02-12]

ii. Show patches published starting 2018-02-01, ending 1 month ago
publishedDate: [2018-02-01 ... now-1M]

iii. Show patches published starting 2 weeks ago, ending 1 second ago
publishedDate: [now-2w ... now-1s]

Show patches published on a certain date
publishedDate:'2018-02-22'

As mentioned in iii, the patches published between the “1 second ago to the last two weeks.”
timeslot will be considered for patching. The 2-week period is the sliding window in this case.

  • Prioritize Products for Patching: Patch the products with the highest number of vulnerabilities first. Candidates with critical vulnerability severity should be patched more frequently. Use the PRIORITIZED PRODUCTS tab for this.
  • Superseded patches: We recommend you to create QQL-based jobs to manage the superseded patches. The system will deploy all the latest superseded patches, which contain the cumulative fixes included in the previous patches.  
  • Dependencies: When certain patches are deployed on your assets due to dependency on other patches, additional patches might also be deployed. The job log file might show an entry for additionally installed patches in such cases.
    In rare cases, if a dependent patch is only partially installed, the job log file might still show an entry for the additional patch.
  • Rollback: Create a separate job to include the patches that can be rolled back. 

Job Creation

Refer to the following suggestions to create jobs according to the best-suited options for your organization’s requirements:

  • Asset reboot after patch deployment: Create separate jobs based on whether an asset reboot is required after the patch deployment. We recommend deploying a job that doesn’t require an asset reboot first and then deploying a job that needs one.
  • Balance number of patches and assets in a job: If you want to apply an extensive number of patches on an extensive number of assets, balance the number of patches and assets wisely. We recommend that you don’t create a job with a large number of patches and deploy them on a large set of assets at the same time.  Run an individual patch job only on 50 – 67% of assets at a time. Though a maximum of 2000 patches can be added in a single job, patching will take a long time and might diminish the patching experience too.
  • One-time and Recurring jobs: Choose to schedule a one-time (immediate or scheduled) or recurring job based on your requirements.
  •  On-demand job: Create on-demand jobs on the highest priority to apply critical patches that fix vulnerabilities. The on demand job runs immediately after it’s enabled. 
    Note: After the patch deployment, reboot the assets if it’s required. A patch job is effective only after the assets are rebooted.
  • Schedule job: Runs on the selected day and time. 
  • Recurring job: Runs based on a daily, weekly, or monthly basis. The Patch Tuesday scheduling option (Windows deployment job only) is available only for the monthly schedule option. 

Patch Window

You can configure a patch window to run the deployment job only within a particular time frame. Setting the patch window ensures that the agent starts the job within the specified patch window (e.g., start time + 6 hours). The job will time out if it does not start within this window.
If you don’t set a patch window, the cloud agent takes as much time as it needs to complete the patch job beyond the maintenance window too.

Time Zone

For scheduled jobs (one-time and recurring), the system uses the agent time zone by default. However, you can customize the time zone to fit your requirements. 

  • Server time vs Agent time: You can create jobs based on server time zone and agent time zone. We recommend that you create a job based on the server time zone for efficient patching. 
  • Based on your environment, create jobs based on appropriate asset tags, such as weekdays or weekends, working or non-working hours, and reboot criteria.
  • Job deployment duration: We recommend considering the job deployment duration when you create jobs. The duration depends on the number of patch jobs and the size of the download file (only for Windows deployment jobs).
    Tip: The patching time depends on fetching the patches (Window only), installation, and asset reboot. To expedite a Windows deployment job, you can download the installer file in advance.
    Note: If a reboot is required, we recommend that you reboot your assets ASAP, or in the worst case, within 24 hours after patching. A patch job is effective only after the assets are rebooted.
  • Conflicting or frequent jobs: Avoid conflicting or frequent jobs on the same assets. Create jobs based on tags, such as geography, business unit, and business-critical app-wise configuration. 

Job Failures and Retry Criteria

Patch deployment can fail due to system failure, environmental failure, or user issues.

  • Environmental failure: In case of environmental failure for Windows patch jobs, if the patch file can’t be downloaded, the agent retries to download it.  However, if the system cannot connect to the server, then the agent will not try to download the file after certain retries. 
    If the installer file crashes, the agent will not try to download and install the file again.
    Note: The agent resumes only a Windows patch job during the maintenance window.
    In case of environmental failure for Linux patch jobs, the agent doesn’t retry to download it. We report the job failures.
  • Nonavailability of asset: Patch deployment on an asset might be delayed or timeout if the agent (not an asset) is not online or unavailable when the job runs.