BizTalk DevOps: Monitor your BizTalk environment using PowerShell – Manage messages being archived locally into the hard drive

Posted: May 15, 2015 in BizTalk, PowerShell
Tags: , , , , , ,

Is very common in integration scenarios to see messages being archived locally into the hard drive – either by using a pipeline component like BizTalk Archiving – SQL and File or by simple using the default functionalities in BizTalk like filters:

This can happen for several reasons, for example:

  • The messages needs to be archived for legal reasons (normally (recommended) we use an “external” repository – not a local hard drive in the server)
  • or because it is practical and useful to have the file so it can easily be reprocessed in case of failure

The common problem with some of these approach is that normally we tend to implement the archive mechanism and we forget to implement a cleaning mechanism and we end up with several problems after a while:

  • The most critical –> disk full, causing processes to fail or other problems even more critical
  • Many files in a folder, causing performance problems when we are trying to perform operations on it à According to some articles, the lookup time of a directory increases proportional to the square of the number of entries and performance seems to drop between 1000 and 3000 files (the important is not the exact number itself but knowing that a high number may cause problems).

And if you have an integration process that processes hundreds of messages, you can quickly get this kind of problems.

In my demo scenario we have a common Message Archive folder:

  • D:\BizTalkApplications\Archive

This folder is organized by sub-folders describing the processes and the communication, for example:

  • System A
    • Inbound
    • Outbound
  • System B
    • Inbound
    • Errors
  • Canonical Messages
    • Inbounded
    • Archived

And so on… that I need to monitor and constant clean it.

So how can PowerShell help us?

With this script you can be able to automatically monitoring these folders on the servers using PowerShell.

This script allows you to set:

  • The numbers of hours, minutes or days that you want to archive the files, leaving only the recent ones – if the creation time of the file is older than this parameter then it will be archived and deleted from the folder
$Hours = "12"
$LastWrite = $Now.AddHours(-$Days)
  • The directory you want to monitor
$Directories = Get-ChildItem "D:\BizTalkApplications\Archive" -Recurse | ?{ $_.PSIsContainer } | Select-Object FullName
  • And for each subdirectory it will check if exist files to be archived
    • if so, it will create a zip file, using the 7-zip tool, in a different location “BackupFileWarehouse”
    • Otherwise will not create any file and move to the next directory
foreach ($directory in $Directories)
{
    $zipFile = "E:\BackupFileWarehouse\" + $directory.FullName.Substring($directory.FullName.LastIndexOf(":")+2).Replace("\","_") + "_" +  (get-date -f yyyyMMdd_hhss).ToString() + ".zip"

    $filelist = get-childitem $directory.FullName |
                                where-object {$_.LastWriteTime -le (get-date).AddHours(-$Hours)} |
                                where-object {-not $_.PSIsContainer}

    if($filelist.Count -gt 0)
    {
        $filelist | format-table -hideTableHeaders FullName | out-file -encoding utf8 -filepath lastmonthsfiles.txt
                    & 'C:\Program Files (x86)\7-Zip\7z.exe' a $zipFile `@lastmonthsfiles.txt

        $filelist | %{Remove-Item $_.FullName }
        rm lastmonthsfiles.txt
    }
}

Of course if this script will not move/create the archived files in a shared location or something similar, then you will need an additional mechanism to maintain and delete this zip files

Requirements:

The script can be found and download on Microsoft TechNet Gallery:
BizTalk DevOps: Manage messages being archived locally into the hard drive (1.0 KB)
Microsoft TechNet Gallery

Comments
  1. wfmichel says:

    Very timely. I was just thinking of putting together something like this. As always your posts are very useful. 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s