Fabric Circular Logging for Skype for Business 2019

The issue

If like me, you have been using Pat’s awesome Get-CSFeatures script for your Skype for Business builds. You may have noticed a feature that no longer works with 2019. “Misc Server Config > Set Fabric logging to circular” with the error “This server does not have Windows Fabric Installed”

This is no fault of Pat’s. The error message is correct. Windows fabric isn’t installed!

With the release of Skype for Business 2019 the backend fabric has been moved to Microsoft Azure Service Fabric.

With the move to a new app is a new path for log files.
Windows Fabric used to log to C:\programdata\Windows Fabric\Fabric\log\Traces
However, Azure Fabric logs to C:\ProgramData\Microsoft\SF\Log\Traces

I naively thought that with improvements to 2019 and Azure Service Fabric that logs couldn’t grow out of control like they used to. But within 2 months my logs were already up to 22GB (1 of 3 Enterprise Ed Servers, with 1326 users across the pool)

This also happens on Standard Edition Servers. Here’s one that’s only been up for 2 weeks with no users on it. It’s already at 10GB!

So why don’t you just update the logs to circular manually?

Tom and FlinchBot have written about this before, but I’ve never really needed to worry about it thanks to the feature in Set-CsFeatures

Unfortunately, the old “Logman update trace FabricLeaseLayerTraces -f bincirc –cnf” trick doesn’t work anymore. Even with “-ets” to force it to run straight on the scheduler.

If we look at Logman we can see the following traces are running (note the -ets flag)

And by default, these are all set to linear logging.

Additionally, it’s no longer just the trace’s in the folder that gets large.
But now we have PerfMon graphs to worry about!

A quick look at Logman and we can see that the performance counters are set to not overwrite or circularly record either.

Unfortunately, regardless of whatever voodoo magic I parse to Logman.exe I cant enable circular logging. (Here are some examples, I spent hours trawling through docs.microsoft and trying different combinations)

Digging further I can see something is “Cleaning up” these logs and archiving them anyway. But these Perfmon logs are what is growing out of control.

The real fix

I started writing a script that would automatically clean these logs and more specifically the perfmon data up and got stuck in feature creep like I always do… Automatic installers, self-updates.. blah blah

Then I got sick with pneumonia and stopped working on it. Because you know, breathing is a thing.

Little did I know Chris Hayward was also working on a solution to the fabric logging problem, see his script block over here for adjusting the config and cleaning up the actual Fabirc Logs.

What Chris’s solution doesn’t do yet, is to clean out the “Archived” perfmon data from these folders too.

C:\ProgramData\Microsoft\SF\Log\PerformanceCounters_WindowsFabricPerfCounter
C:\ProgramData\Microsoft\SF\Log\PerformanceCountersBinaryArchive

I’m still working on my own self-installing script, but for now, you can use this script block to clean it up.

	Function Clear-S4BFabricLogs
	{
	<#
	.SYNOPSIS
	Removes old Windows Fabric log files and Perfmon Data
	.DESCRIPTION
	Removes logfiles older than 14 days from the following locations
	C:\ProgramData\Microsoft\SF\Log\PerformanceCounters_WindowsFabricPerfCounter
	C:\ProgramData\Microsoft\SF\Log\PerformanceCountersBinaryArchive
	.EXAMPLE
	Clear-S4BFabricLogs
	Removes the logs
	.LINK
	https://www.UcMadScientist.com
	.INPUTS
	This function does not accept pipelined input
	.OUTPUTS
	This function does not create pipelined output
	#>
	[CmdletBinding()]
	$function = 'Clear-S4BFabricLogs'
	$SpaceBefore = (Get-WmiObject -Class Win32_logicaldisk -Filter "DeviceID = 'C:'" | Select-Object -Property @{L='FreeSpaceGB';E={"{0:N2}" -f ($_.FreeSpace /1GB)}})
	Write-Host "Cleaning up files"
	$CurrentDate = Get-Date
	$DatetoDelete = $CurrentDate.AddDays(-14)
	$FolderstoClean = @(
	"C:\ProgramData\Microsoft\SF\Log\PerformanceCountersBinaryArchive"
	"C:\ProgramData\Microsoft\SF\Log\PerformanceCounters_WindowsFabricPerfCounter"
	"C:\ProgramData\Microsoft\SF\Log\PerformanceCountersBinary"
	)
	ForEach ($folder in $FolderstoClean) {
	Write-Host "Checking $Folder"
	$files = (Get-ChildItem $Folder | Where-Object { $_.LastWriteTime -lt $DatetoDelete })
	Write-Host "Found $($Files.Count) in $Folder"
	Write-Host "Removing Files"
	Get-ChildItem $Folder | Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item
	}
	$SpaceAfter = (Get-WmiObject -Class Win32_logicaldisk -Filter "DeviceID = 'C:'" | Select-Object -Property @{L='FreeSpaceGB';E={"{0:N2}" -f ($_.FreeSpace /1GB)}})
	$SpaceSaved = (($Spacebefore.FreeSpaceGB – $SpaceAfter.FreeSpaceGB) – ($Spacebefore.FreeSpaceGB – $SpaceAfter.FreeSpaceGB)*2)
	Write-Host "Saved $SpaceSaved GB"
	}

view raw Clear-S4BFabricLogs.ps1 hosted with ❤ by GitHub

As with anything that fiddles with these settings, we aren’t sure if it’s supported by the product group but rest assured we have asked the right people so hopefully, there is an official fix in the works.

Hope this helps.