Part 3

The 12-Step Patching Solution That Will Save Your Sanity, and Your Weekend

In our first two blogs, we defined the three different types of SharePoint updates and then took a quick look at a typical patching experience. At this point, you know that a patch event has three components and if you don’t manage these, you’re at the mercy of the default update process. But with our 12-step patch installation process, we make that process parallel and multi-threaded. This is a long technical blog, but stick with me and we’ll save you time and take back your weekend! Ready?

Listed below are the steps I use to gain control of the patching process, save time, and make it a meaningful process with quantifiable benefits. The 12 steps to successful CU patching on a production SharePoint farm are:

1. Download the CU and document the issues it is purported to resolve
2. Install the CU on a non-production environment FIRST and TEST IT!!
3. Prepare the production SharePoint environment
4. Document the production environment BEFORE patching
5. Dismount ALL SharePoint content databases
6. Install the CU on each farm server
7. Run PSConfig on all servers
8. Re-mount all content databases
9. Ensure ALL content databases have been re-mounted successfully
10. Run content database upgrade script(s)
11. Document the production SharePoint environment AFTER patching
12. Verify production farm functionality

Let’s look at each of these steps, one by one:

1. Download the CU and document the issues it is purported to resolve

The point here is two-fold:

a. Determine if the CU is one you need to address an existing issue on your farm.

b. Once you’ve determined you need to install the CU, download it into your environment.

2. Install the CU on a non-production environment FIRST and TEST IT!!

The benefits seem obvious, but I have to mention that installing a CU on a test or staging environment allows for verification of the patch’s fixes as well as keeping your test, staging, and production environments in sync.

3. Prepare the production SharePoint environment

Here’s where we get the servers ready to handle the multi-threaded, parallel CU installation we’re going to oversee: Create patch and script folders on each SharePoint farm server:

• On each App server C:\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput. We’re going to place the CU (the June 2017 CU in this example) in the June 2017 folder and the scripts will go in the scripts folder. The output from the scripts will be stored in the scriptOutput folder.
• On each WFE server: C:\SP2013\PatchDir\CU\June2017. Similarly, the WFEs will have the CU installation .exe file in the June 2017 folder. We don’t need a Scripts or a ScriptOutput folder because we generally don’t run our patch installation scripts on WFE servers.

In Step 3 we first get a hint that we will be using PowerShell scripts to manage this process. Additionally, we get to use these scripts in the next step, documenting the farm before we begin the update.

4. Document the production environment BEFORE patching

In our CU installation, we chose to document all the sites in each web application, the content databases our farm supports and the Managed paths of the web applications. I could easily have chosen other settings to document, but since I’m mostly concerned with ensuring that I finish with the same content I started with, I chose these PowerShell commands/scripts to run:

• Run the 1-ListAllSitesAllWebs_before.ps1 PowerShell script:
  Get-SPWebApplication -Limit All | Get-SPSite | Format-Table -Property URL,ContentDatabase | out-file -filepath \\App01\c$\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput\APP01SitesAllWebs_before.txt
• Run the 2-ListMountedContentDatabases_before.ps1 PowerShell script:
  Get-SPContentDatabase -webapplication https://webapp.agency.gov/ | out-file -filepath \\APP01\c$\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput\APP01ContentDBs_before.txtGet-SPContentDatabase -webapplication https://my.agency.gov | out-file -filepath \\APP01\c$\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput\MySitesContentDBs_before.txt
• Run the 3-ListManagedPaths_before.ps1 PowerShell script:
  Get-SPManagedPath -webapplication https://sp.webapp.gov | out-file -filepath \\APP01\c$\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput\APP01ManagedPaths_before.txtGet-SPManagedPath -webapplication https://mysites.agency.gov | out-file -filepath \\APP01\c$\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput\MySitesManagedPaths_before.txt

5. Dismount ALL SharePoint content databases

This is the singularly most impactful step, which splits the CU update process into two segments—updating the servers, the service applications, the SharePoint configuration database in one activity, and separately, updating the content databases. Here’s the command I used to dismount the content databases all at once, well, once for each web application. I have a content web app and a MySites web app: Run the 4-DismountContentDatabases.ps1 PowerShell script:

Get-SPContentDatabase -webapplication https://sp.webapp.gov | Dismount-SPContentdatabase -Confirm:$false

Get-SPContentDatabase -webapplication https://mysites.webapp.gov | Dismount-SPContentdatabase -Confirm:$false

6. Install the CU on each farm server

Actually, you can complete this step at any point before you begin running PSConfig on the servers. In my government agency environment, a separate team installs the patches; so the time of CU installation can be out of the hands of the SharePoint administrator sometimes. Also, be aware that there’s really nothing you can do to speed up the installation of the CU bits onto the server.

• Logon to each farm server using the farm account in the following order:
  • Central Admin
  • Search, App Servers
  • WFEs
• Stop the following SharePoint Windows Server services, in this order:
  • SharePoint Timer Service (SPTimerV4)
  • SharePoint Server Search 15 (OSearch15)
  • SharePoint Search Host Controller (SPSearchHostController)
• Run the CU executable.
• Reboot each server after the CU executable installation completes.

7. Run PSConfig on all servers

Here’s the first place where we see some time savings. Since the content databases have been dismounted, we are only updating the SharePoint Configuration database and the service applications—instead of all that plus the content databases—we see a dramatic decrease in the time it takes to complete running PSConfig on each server. So, your process for running PSConfig is to:

• Logon to each farm server using the farm account in the following order:
  • Central Admin
  • Search, App Servers
  • WFEs
• Run PSConfig with the following command syntax:
  • Get-SPProduct -Local
  • Psconfig.exe -cmd upgrade -inplace b2b -wait
• Reboot each server after the installation completes.

Because we dismounted the content databases, the first server we run PSConfig on, the Central Admin server, takes the most time. Running PSConfig on the Central Admin server upgrades the configuration database as well as all the farm’s service applications. When we run PSConfig on the other servers, the time to completion is dramatically less.

8. Re-mount all content databases

Once you’ve completed running PSConfig on all the servers successfully, you’re ready to re-mount the content databases so that they can be upgraded. The content databases are upgraded separately from the rest of the farm to save time, as you will see detailed in Step No. 10. To re-mount the databases we dismounted, we create mount scripts based on the output from the script we ran in Step No. 2. Specifically, we’re going to take the output from the 2-ListMountedContentDatabases_before.ps1 script and create two mount scripts, one for each web application. A content database mount script could look like this:

Mount-SPContentDatabase -Identity WSS_ContentDBName -DatabaseServer DBserverName\InstanceName -WebApplication https://sp.webapp.gov

Mount-SPContentDatabase -Identity WSS_ContentDBName -DatabaseServer DBserverName\InstanceName -WebApplication https://mysites.webapp.gov

9. Ensure ALL content databases have been re-mounted successfully

Because end users are sticklers for having their content available to them, we must ensure that we re-mount ALL the databases. To help ensure that we do, complete the following tasks:

• Run .\6-ListMountedContentDatabases_after.ps1 PowerShell script:
  Get-SPContentDatabase -webapplication https://sp.webapp.gov/ | out-file -filepath \\APP01\c$\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput\ContentDBs_after.txtGet-SPContentDatabase -webapplication https://mysites.webapp.gov | out-file -filepath \\APP01\c$\SP2013\PatchDir\CU\June2017\Scripts\ScriptOutput\MySitesDBs_after.txt
• Compare the _before and _after.txt output files and manually mount any content databases that were not mounted by the mount scripts

10. Run separate content database upgrade script(s)

FINALLY, the time-saving part of the process!! In this step, we will create separate content database upgrade scripts, one for each App server in the farm. I guarantee you cut your time dramatically using this approach. Here’s how we accomplished that feat. In my example, I have two App servers:

• Create two separate upgrade scripts by splitting the list of all SHAREAPP01 and MySites web application content databases into two (or more) separate upgrade scripts.
• Run one upgrade script on each app server in the SHAREAPP01 farm.
  • RDP Logon to SHAREAPP01 server and run one content database upgrade script:
    Upgrade-SPContentDatabase WSS_ContentDatabaseName1 -Confirm:$false
    Upgrade-SPContentDatabase WSS_ContentDatabaseName2 -Confirm:$false
  • RDP Logon to SHAREAPP02 server and run one content database upgrade script:
    Upgrade-SPContentDatabase MySites_ContentDatabaseName1 -Confirm:$false
    Upgrade-SPContentDatabase MySites_ContentDatabaseName2 -Confirm:$false

Using multiple app servers for upgrading the content databases creates a multi-threaded, parallel upgrade process. So, instead of one server—typically the Central Administration server—running a single-threaded, sequential content database upgrade process on all your content databases, you now split the content database upgrade processing between all of your app servers. This is where you see your weekend coming back to you.

11. Document the production SharePoint environment AFTER patching

Once you reach this step, you’ve started to dream of whatever weekend activity you can do, many many hours before you thought you could. But, let’s not get lazy; let’s finish strong:

• Run the 12-ListALLSitesAllWebs_after.ps1 PowerShell script and compare the results with the _before script output from Step No. 2.
• Run the 13-ListManagedPaths_after.ps1 PowerShell script and compare the results with the _before script output from Step No. 2.

12. Verify production farm functionality

Every administrator should have her own checks for verifying the functionality of her farm. I’m not going to define those here. But suffice it to say, if you don’t have a list of checks you perform routinely to verify SharePoint functionality, you need to work with your admin and development team(s) to create one. Don’t let your desire to finish quickly compromise the hard work you’ve already done.

Takeaways: If you’ve gotten to this point in the article, thank you!! I believe you will see dramatically diminished update times. In the end, consider the following points:

• You don’t always need to patch just because Microsoft released a CU:
  • Do your due diligence and make sure any update you’re installing actually fixes something that needs to be fixed.
  • If you don’t install this month’s update, the same hotfixes will be there in the next CU.
• Script your update:
  • Create a checklist.
  • Use PowerShell scripts to document, dismount, remount, and upgrade.
  • Use the computing power of all your server resources to speed up the process.
• Use the CU patch activity as an opportunity to learn about your farm.

Good luck and hope to see you on the links by Sunday!