Using PowerShell Background Jobs can help you speed up Exchange Tasks. (Part 1)

Managing Exchange Server via PowerShell is always fun and effective but sometimes it might be slow and painful to get all the information you want.

Wouldn’t it be nice to speed thing up a bit? Background Jobs are perfect for tasks that will consume a long time to complete. ( Sometimes :) )

A Windows PowerShell background job runs a command “in the background” without interacting with the current session. When you start a background job, a job object is returned immediately, even if the job takes an extended time to complete. You can continue to work in the session without interruption while the job runs.

Which means we can run multiple jobs and get the results faster. (On some cases slower when working with Exchange as it requires remote session but it depends on what you are collecting)

In order to understand how jobs work we need to know which commands are available:

  • The Get-Job cmdlet gets objects that represent the background jobs that were started in the current session. You can use Get-Job to get jobs that were started by using the Start-Job cmdlet, or by using the AsJob parameter of any cmdlet.
  • The Start-Job cmdlet starts a Windows PowerShell background job on the local computer.
  • The Stop-Job cmdlet stops Windows PowerShell background jobs that are in progress. You can use this cmdlet to stop all jobs or stop selected jobs based on their name, ID, instance ID, or state, or by passing a job object to Stop-Job
  • The Receive-Job cmdlet gets the results of Windows PowerShell background jobs, such as those started by using the Start-Job cmdlet or the AsJob parameter of any cmdlet. You can get the results of all jobs or identify jobs by their name, ID, instance ID, computer name, location, or session, or by submitting a job object.
  • The Remove-Job cmdlet deletes Windows PowerShell background jobs that were started by using the Start-Job or the AsJob parameter of any cmdlet.
  • The Wait-Job cmdlet waits for Windows PowerShell background jobs to complete before it displays the command prompt. You can wait until any background job is complete, or until all background jobs are complete, and you can set a maximum wait time for the job.
  • The Suspend-Job cmdlet suspends (temporarily interrupts or pauses) workflow jobs. This cmdlet allows users who are running workflows to suspend the workflow. It complements the Suspend-Workflow activity, which is a command in the workflow that suspends the workflow.

In this post I will just go through a common task to show you an example on how you can utilize background jobs in your scripts to improve performance. As I mentioned before this is not for every task and there are certain things you need to be careful about before going down in this path.

  • Exchange throttling policy on both Exchange 2010 and Exchange 2013 PowerShellMaxConcurrency is set to 18 by default. Which means you can not have more than 18 concurrent connections to same Exchange Server via PowerShell session. (Unless you change it)
  • Background jobs will create temp files which in size are pretty large so it is important to set your temp location in your scripts and clean them up when you are done. I bet you do not want to run out of disk space.
  • Background jobs will consume Memory and CPU as they are all seperate powershell processes. Depending your data size they might consume a lot. Limit the number of Jobs you run and you should be good to go.

So lets see how we can use these background jobs.

In order to make it simple I will share a script to get calendar permission for default user from every mailbox to report. I am well aware that this is not something you do very often or ever at all but stay with me. I will also explain each line inside the code.

#Start the Clock
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
#Get All Mailboxes
$AllMailboxes = Get-Mailbox -ResultSize Unlimited
#Create an empty array for Calendar Permissions.
#This will be our overall result
$CalendarPermissions = @()
#We will get calendar permissions for each mailbox
foreach ($Mailbox in $AllMailboxes)
{
	#Set the Default calendar name to show up as Calendar on output.
	$Calendar = 'Calendar'
	#Get the calendar permission for this mailbox
	$CalendarPermission = Get-MailboxFolderPermission -Identity ($Mailbox.alias + ':\Calendar') -User Default -ErrorAction 'SilentlyContinue' | Select User, AccessRights, Identity
	#If the calendar name is not calendar the return will be empty.
	#If return is empty we need to check all calendar folders.
	if (!$CalendarPermission)
	{
		#If the return is empty find the calendar in the mailbox as the name might be localized.
		#Find the calendar folders
		$Calendars = Get-MailboxFolderStatistics -Identity $Mailbox.Identity -FolderScope Calendar | ? { $_.FolderType -eq "Calendar" } | Select -Expand Name
		#Create empty calendar permission array as we will go to every calendar folder and collect permissions.
		$CalendarPermission = @()
		#Go and check every calendar folder in the mailbox
		foreach ($Calendar in $Calendars)
		{
			#Get the calendar permissions on this calendar folder
			$CalendarPermission = Get-MailboxFolderPermission -Identity "$($Mailbox.Identity):\$($Calendar)" -User Default -ErrorAction 'SilentlyContinue' | Select User, AccessRights, Identity
			#Create a result object that includes our results
			$result = New-Object PSObject -Property @{ "Mailbox" = $Mailbox.Name; "User" = $CalendarPermission.User; "AccessRights" = $CalendarPermission.AccessRights; "Calendar" = $Calendar }
			#Add the result to Calendar Permissions array
			$CalendarPermissions += $result
		}
	}
	#if the Calendar folder exists than add the results to Calendar Permissions array
	else
	{
		$result = New-Object PSObject -Property @{ "Mailbox" = $Mailbox.Name; "User" = $CalendarPermission.User; "AccessRights" = $CalendarPermission.AccessRights; "Calendar" = $Calendar }
		$CalendarPermissions += $result
	}
}
#Show how long it took to complete
Write-Host "Total Elapsed Time: $($elapsed.Elapsed.ToString())"

On this example I have only around 1500 mailboxes and it took around 50 mins :(

1

The problem is, we are going through each mailbox one by one to get the calendar permission for default user. This process can take a very long time in an environment where you have thousands of mailboxes.

What if we divide this process to 10 different background jobs?

Tip: Keep in mind that maximum jobs you can run is 17 because you have already 1 session on Exchange and Throttling Policy will stop you at 18. If you need to run more background jobs on Exchange you need to change the throttling policy.

Here is how we can do it.

Note: This is not the only way or the best way to do it, Please be kind to remember that this is just an example.

First we need a function that we can use to split the array to the number of background jobs that we are going to run.

function Resize-Array ([object[]]$InputObject, [int]$SplitSize = 100)
{
	$length = $InputObject.Length
	for ($Index = 0; $Index -lt $length; $Index += $SplitSize)
	{
		, ($InputObject[$index .. ($index + $splitSize - 1)])
	}
}

We also need an initialization script that we need to pass to the job so we can do the Exchange PowerShell remote connection. This is one of the parameters for Start-Job command.

-InitializationScript
Specifies commands that run before the job starts. Enclose the commands in braces ( { } ) to create a script block.

$InitializationScript = {
	function Connect-myExchangeServer
	{
		Param
		(
			[Parameter(Mandatory = $true)] $ServerFQDN,
			[Parameter(Mandatory = $true)] $Credentials
		)
		#Get the My Documents Path from local computer
		$DefaultRootFolder = [environment]::getfolderpath("MyDocuments")
		#Create a folder for collecting temp data
		$TargetDIR = "$DefaultRootFolder\SRKNVRGL_Temp"
		if (!(Test-Path -Path $TargetDIR))
		{
			New-Item -ItemType directory -Path $TargetDIR
		}
		#Set Temp location for powershell sessions to this temp folder
		Set-Location $TargetDIR
		$env:Temp = $TargetDIR
		$env:Tmp = $TargetDIR
		#Create a new ps session to exchange server
		$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://$ServerFQDN/PowerShell/ -Authentication Kerberos -Credential $Credentials -WarningAction 'SilentlyContinue'
		#Connect and import module just incase if you are running it from Powershell and not from EMS
		$Connect = Import-Module (Import-PSSession $Session -AllowClobber -WarningAction 'SilentlyContinue') -Global -WarningAction 'SilentlyContinue'
	}
}

Few important notes about initialization script above:

  • In this initialization script you can see that there is function to connect to Exchange server. It requires ServerFQDN and your credentials. (Depending on your environment you might need to edit this part if you are planning to test the same script) Keep this in mind as we will need to provide them later in the script.
  • This initialization script creates a temp folder under logged on users my documents folder. It uses this directory for temp files that will be generated by PowerShell when job starts. Make sure you clear these items once you are done.

Now the rest of the script:

#Get Credentials so we can use it to open sessions.
$Credentials = Get-Credential
#Set the ServerFQDN variable with the FQDN of the Exchange Server we will use to connect
$ServerFQDN = "EX01.Get-Mailbox.Org"
#Start the Clock
$elapsed = [System.Diagnostics.Stopwatch]::StartNew()
#Get All Mailboxes
$AllMailboxes = Get-Mailbox -ResultSize Unlimited
#Calculate the split size that will be used to create 10 sessions
$SplitSize = $AllMailboxes.Count/10
#Split the array into 10 arrays
$MailboxesArray = Resize-Array -InputObject $AllMailboxes -SplitSize $SplitSize
#We need to start a background job for each array we created
foreach ($Array in $MailboxesArray)
{
	#Start Job using ArgumentList and InitializationScript.
	#Argument list is passed in as an array so we will use each array item to set our variables
	Start-Job -ScriptBlock {
		$ServerFQDN = $args[0]
		$Credentials = $args[1]
		$Mailboxes = $args[2]
		$Connection = Connect-myExchangeServer $ServerFQDN -Credentials $Credentials
		$CalendarPermissions = @()
		foreach ($Mailbox in $Mailboxes)
		{
			#Set the Default calendar name to show up as Calendar on output.
			$Calendar = 'Calendar'
			#Get the calendar permission for this mailbox
			$CalendarPermission = Get-MailboxFolderPermission -Identity ($Mailbox.alias + ':\Calendar') -User Default -ErrorAction 'SilentlyContinue' | Select User, AccessRights, Identity
			#If the calendar name is not calendar the return will be empty.
			#If return is empty we need to check all calendar folders.
			if (!$CalendarPermission)
			{
				#If the return is empty find the calendar in the mailbox as the name might be localized.
				#Find the calendar folders
				$Calendars = Get-MailboxFolderStatistics -Identity $Mailbox.Identity -FolderScope Calendar | ? { $_.FolderType -eq "Calendar" } | Select -Expand Name
				#Create empty calendar permission array as we will go to every calendar folder and collect permissions.
				$CalendarPermission = @()
				#Go and check every calendar folder in the mailbox
				foreach ($Calendar in $Calendars)
				{
					#Get the calendar permissions on this calendar folder
					$CalendarPermission = Get-MailboxFolderPermission -Identity "$($Mailbox.Identity):\$($Calendar)" -User Default -ErrorAction 'SilentlyContinue' | Select User, AccessRights, Identity
					#Create a result object that includes our results
					$result = New-Object PSObject -Property @{ "Mailbox" = $Mailbox.Name; "User" = $CalendarPermission.User; "AccessRights" = $CalendarPermission.AccessRights; "Calendar" = $Calendar }
					#Add the result to Calendar Permissions array
					$CalendarPermissions += $result
				}
			}
			#if the Calendar folder exists than add the results to Calendar Permissions array
			else
			{
				$result = New-Object PSObject -Property @{ "Mailbox" = $Mailbox.Name; "User" = $CalendarPermission.User; "AccessRights" = $CalendarPermission.AccessRights; "Calendar" = $Calendar }
				$CalendarPermissions += $result
			}
		}
		#Output the CalendarPermissions so we can collect once the job is completed.
		$CalendarPermissions
	} -InitializationScript $InitializationScript -ArgumentList @($ServerFQDN, $Credentials, $Array)
}
#Wait for the jobs to complete and than collect the output to CalendarPermissions variable
$CalendarPermissions = Get-Job | Wait-Job | Receive-Job
#We do not need the jobs any more so we can remove them.
Get-Job | Remove-Job
#Show how long it took to complete
write-host "Total Elapsed Time: $($elapsed.Elapsed.ToString())"

Note: This script uses PowerShell v2. It will work on Exchange Server 2010 and Exchange Server 2013. With PowerShell v3 on Exchange Server 2013 background jobs are a bit easier as you do not need to use -argumentlist parameter instead you can use $using: command to pull the variable into the job

This script opens 10 background jobs and will do the same thing for smaller number of mailboxes that has been used on the first script.

Keep in mind this approach is not for every single script or command. Background jobs are great if you can use them and utilize them in your Exchange scripts where you can divide the task into smaller parts and finish faster.

This script completed under 10 mins.

2

Start-Job wins the competition. :) at least for now. but a cost of high CPU and Memory usage due to having a seperate powershell process for each job.

3

In part 2 we will try to see if we can reduce the resource usage with Runspaces for our jobs.

Please drop a comment if you have questions or feedback.

 

About Serkan Varoglu

Serkan Varoglu is a Turkish IT Pro living in Ireland. Serkan has over 10 years experience and hold certifications including MCITP (EMA 2010 and Enterprise Admin), MCSE, MCSA, MCTS, ITIL and was awarded the Microsoft MVP Award (Exchange Server).