Posts Tagged ‘BizTalk’

In the past I wrote a series of post on Basics principles of Maps where I stated that testing should be a continuous process as you build your map, not only at the end of development, but when necessary or when an important mapping block is complete.

And the good thing is that this feature is available to developers in an easy manner and directly from our favorite development tool – Visual Studio – without the need to build and deploy the maps or even create and configure ports and test them in runtime.

Note that Visual Studio allows you also to Validate and Debug maps at design time.

For this we need to:

  • Open the Solution Explorer windows
  • And execute the test by right-clicking the map name and selecting “Test Map” option
  • Verify the results in the Output window.

By default an instance of the input schema is generated automatically with dummy values, according to their type, and tested in the selected map. At the end, the generated result or the errors that occurred are displayed in the output window.

01-Visual-Studio-Output-Window

However this scenario provably is not the ideal one and what we want is to test the map with an existing and real message and not a dummy one. And the good news is that, again, BizTalk Mapper allows us to specify a source document.

In addition, it also supports instances as native (CSV, EDI …) or Extensible Markup Language (XML) as input or output according to the scenarios.

For this we only need to configure the properties of the map before we execute the test:

  • Right-clicking the map name and select Properties option;
  • In the Properties window set “TestMap Input Instance” property with the path to the input instance message file to be used while testing the map.

02-Visual-Studio-Map-Properties-Windows

Other important properties that you can specify are:

  • TestMap Output Instance: Specifies the location where the Test Map should generate the output message.
  • TestMap Input: Specifies the format of the input instance message.
  • TestMap Output: Specifies the format for the output instance message.
  • Validate TestMap Input: Specifies whether you want to validate input instance messages against the source schema before you test the map.
  • Validate TestMap Output: Specifies whether you want to validate output instance messages against the destination schema after you test the map.

This last option, “Validate TestMap Output”, of the map properties is extremely important for the partial tests of maps. By setting this property as "False", allows us to test an incomplete map without being shown errors due lack of mandatory data, many of them associated with areas still to map.

This is nothing new, despite we often forget some of these properties. However I find myself constantly hear that, despite being easy to test one-to-one maps inside Visual Studio at design time, when we have multiple inputs to the map, many-to-one or many-to-many transformations, the same doesn’t happen, and it’s impossible to test this type of maps inside Visual Studio at design time because we only can specify an input message… and that it’s not true!

How can we test many-to-one or many-to-many maps inside Visual Studio?

Testing one-to-one maps is extremely simple and straightforward, however, many-to-one or many-to-many maps, despite being simple to test, it will required some additional steps.

First you have to understand that, Visual Studio doesn’t let us define multiple native instances in the “TestMap Input Instance” property, it only allows one instance message. But when a map has multiple source schemas, they are wrapped in a hierarchy multi-part message created by BizTalk.

So the only way for us to test this type of maps is to provide with an instance of the message in the “TestMap Input Instance” property that matched with the hierarchy multi-part message created by BizTalk.

The question here is how we can easily create an instance of multi-part message?

And the question is simple, by using the default behavior of the Test Map operation to make this work for us!

Let’s explain better, when we create a new map the “TestMap Input” and “TestMap Input Instance” proprieties are defined by default as: “Generate Instance” and “” (empty) respectively:

03-Visual-Studio-Map-Default-Properties-Windows

This means, as I explained earlier that if we test the map without changing this properties, one instance message will be generated automatically with dummy values. However in this scenarios, it will generate a dummy instance of the multi-part message expected by the map! AWESOME!

So what we need to do is:

  • Right-clicking the map name and selecting “Test Map” option
  • In the Output window the BizTalk Mapper will provide a link to the test’s output XML file and also to the input XML file.

04-Visual-Studio-Output-Window-input-and-output-file

  • Through the File Explorer, go to the directory folder specified folder in the Output window to have access to the dummy input file:
    • In this case: “C:\Users\Administrator\AppData\Local\Temp\1”
  • Copy the file to your project folder, in this case: “inputfile.xml”

This is a sample of this file:

05-multi-part-message-input-sample

As you can see, all the different input messages are there, with the write namespaces and so on… So what we need to do now is to:

  • Edit this file and put the User and Address messages that you want (you can do this easily with notepad)
  • And then specify the “TestMap Input Instance” property with this file.

As said earlier … we need some additional steps … but it is extremely easy to accomplish!

Microsoft Integration MVP 2013

Microsoft Integration MVP 2014

Thanks Microsoft for another wonderful start of New Year. I’m delighted to share that I have been awarded Microsoft Most Valuable Professional (MVP) 2014.

It’s always nice to wake up after a long night of New Year’s Eve, and check the email and see:

Congratulations! We are pleased to present you with the 2014 Microsoft® MVP Award!

For me this is the fourth consecutive time that I receive this award and as usual on such occasions, I would like to thank to all my blog readers, friends and members of BizTalk Community (there are too many to mention here). Also a big thanks to my MVP Lead Cristina Herrero, Alessandro Teglia, BizTalk Product Team: Mandi Ohlinger, Guru Venkataraman, Karthik Bharathy, Suma Sushilendra, Nitin Mehrotra, Paolo Salvatori and many more; Microsoft Portugal: Miguel Vicente, Paulo Mena and all other Microsoft employees; my fellow Microsoft Integration MVP’s: Michael Stephenson, Jon Fancey, Ben Cline, Mick Badran, Mikael Håkansson, Johan Hedberg, Bill Chesnut Richard Seroter, Dan Rosanova, Stephen W. Thomas, Kent Weare, Leonid Ganeline, Rick G. Garibay, Damir Dobric, Jon Fancey, Rohit Sharma and of course all Portuguese MVP’s.

I also like to make a very special thanks to my beautiful girlfriend Lígia and my family for their support during these years, to the BizTalkCrew (and also Microsoft MVP’s): Nino Crudele, Tord Glad Nordahl, Saravana Kumar and Steef-Jan Wiggers and finally to all my coworkers at DevScope, of which I wish to highlight: Miguel Silva, Vânia Braziela, Márcia Teixeira, José António Silva, Rui Quintino and Rui Barbosa, for all the support.

It’s a big honor to be in the program and I’m looking forward to another great year.

In my native language (Portuguese):

Obrigado Microsoft para outro maravilhoso início de Ano Novo. Estou muito feliz por poder partilhar com todos vocês que fui nomeado Microsoft Most Valuable Professional (MVP) 2014

Esta é a quarta vez que recebo este prémio e como é normal nestas ocasiões, gostaria de agradecer a todos os leitores do meu blog, amigos, a todos os membros da comunidade. Também gostaria de agradecer a minha MVP Lead Cristina Herrero, ao Alessandro Teglia, a todas as pessoas da Microsoft em especial à equipa do produto – BizTalk Product Team, Microsoft Portugal, a todos os meus colegas Microsoft Integration MVP’s e a todos os MVP’s Portugueses.

Eu também gostaria de fazer um agradecimento muito especial a minha linda namorada Lígia e há minha família por todo o apoio durante todos estes anos, ao membros do BizTalkCrew (e também da Microsoft Integration MVP’s): Nino Crudele, Tord Glad Nordahl, Saravana Kumar e Steef-Jan Wiggers e, finalmente, a todos os meus colegas de trabalho na DevScope, dos quais eu gostaria de destacar: Miguel Silva, Vânia Braziela, Márcia Teixeira, José António Silva, Rui Quintino e Rui Barbosa, por todo o apoio e suporte.

É uma grande honra estar no programa e espero continuar a contribuir para a comunidade BizTalk e a merecer o reconhecimento.

Exchanging or routing messages between existing applications, systems or external partners is one of the many common scenarios in Enterprise integration solutions. And sometimes they are build base on the same data model, however because they are built in different systems and teams they can have slight small differences like: the order of the elements, the name of the elements, the structure or even the namespace. As a result of that we need to be able to transformed the original message to the expect message by that systems.

BizTalk Mapper provides you with just-in-time assistance, through a shortcut menu, when you create links between two record elements of source and destination schemas:

  • You can create multiple links simultaneously if:
    • Relevant schema structures must be the same (node names are different but order is the same or similar;
    • Or Record, elements or field names must match;
  • You can manually create single links by dragging and dropping the source element to the destination element, if the destination schema don’t have a similar structures or the same names;
  • Or you can use Mass Copy Functoid to copy the element in the input instance message to a destination schema that use the <any> Element (some kind of generic schema). This functoid also copies any and all of its substructure, and re-creates it in the output instance message at the linked node in the destination schema.

The BizTalk Mapper provides you with just-in-time assistance, through a shortcut menu, when you create links between two record elements of source and destination schemas.

So for accomplish that, open your map and:

  • Drag-and-drop the root name of the source schema, in this case “PurchaseOrder” to the root name of the destination schema, which will be in this case “PO”, and release the direct button of the mouse.

just-in-time-assistance-BizTalk-Mapper

  • A windows assistance will pop up were you can select create record-to-record links automatically in the following ways:
    • Direct Link: Using this technique, the BizTalk Mapper links the record from source schema to the selected record in the destination schema. This will not copy any kind of hierarchy structure from the source to the destination, it will only link the record or node.
    • Link by Structure: Using this technique, the BizTalk Mapper attempts to match the Record and Field nodes within the Record nodes being linked according to the structures of those Record nodes, regardless of names of the corresponding nodes within those structures.
    • Link By Name: Using this technique, the BizTalk Mapper attempts to match the Record and Field nodes within the Record nodes being linked according to the names of the corresponding nodes, regardless of their structure, within the Record nodes being linked.
    • Mass Copy: The Mass Copy functoid enables your maps to use schemas that include any and anyAttribute elements. For information about the functoids available in BizTalk Mapper.

Our scenario will have a Purchase Order that will need to be mapped to 3 different systems with 3 different target schemas.

  • The System A have a similar schema of our system, with the same structure and the same names, but they are presented in a different order;
    • Because the destination schema have the same names of the source schema, the best approach in this scenario is to use the Link By Name option
  • The System B have also a similar schema of our system, we as the same order and the same schema structure, however the elements have different names;
    • Because the destination schema have the structure of the source schema, the best approach in this scenario is to use the Link By Structure option
  • The System C have a canonical schema (or generic schema) that will accept any structure.
    • Because the destination schema will accept any kind of structure, the best approach in this scenario is to use the Mass Copy option
Scenario A: How to link the record elements by name

To accomplished that we need:

  • Drag the mouse from the root name “PurchaseOrder” in the source schema, and then drop it to root name “PO” in the destination schema.
  • On the shortcut menu, click “Link by Name” option. The BizTalk Mapper will attempt to match all the Record, Elements and Field nodes under the “PurchaseOrder” node according to their names, regardless of their structure, within the node “PO”.

And the BizTalk Mapper will automatically map all the elements from the source schema to the target schema that have the same name, as the picture bellow will show:

Link-by-Name

Of course this technique as some limitations, the elements need to have the exact same name, and will only work in some scenarios.

Scenario B: How to link the record elements by structure

To accomplished that we need:

  • Drag the mouse from the root name “PurchaseOrder” in the source schema, and then drop it to root name “ExternalPO” in the destination schema.
  • On the shortcut menu, click “Link by Structure” option. The BizTalk Mapper will attempt to match all the Record, Elements and Field nodes under the “PurchaseOrder” node according to the structure of those elements, , regardless of names of the corresponding nodes within those structures, within the node “ExternalPO”.

And the BizTalk Mapper will automatically map all the elements from the source schema to the target schema that have the same structure, as the picture bellow will show:

Link-by-Structure

Again this technique have the same limitations of the previous one.

Scenario C: How to link using a mass copy functoid

To accomplished that we need:

  • Drag-and-drop the Mass Copy functoid from the Toolbox to the map grid page and then:
    • Link the source root name “PurchaseOrder” as input parameter for the Mass Copy functoid;
    • Link the Mass Copy functoid to the destination root name “CanonicalPurchaseOrder” to set the output parameter from the Mass Copy functoid;

Mass-Copy

The only problem in using the Mass Copy functoid is that it will move all the elements and values from the source schema to the destination schema but it also will include the targetNamespace of source schema in the destination schema and this behavior can be a problem in some scenarios.

Example of the map output:

<ns0:CanonicalPurchaseOrder xmlns:ns0="http://BizTalkMapperSemanticTranslatorPattern.CanonicalPurchaseOrder">
 <From xmlns:ns0="http://BizTalkMapperSemanticTranslatorPattern.PurchaseOrder">From_0</From>
 <To xmlns:ns0="http://BizTalkMapperSemanticTranslatorPattern.PurchaseOrder">To_0</To>
 <LineItems xmlns:ns0="http://BizTalkMapperSemanticTranslatorPattern.PurchaseOrder">
 <Item>
 <Product>Product_0</Product>
 <Description>Description_0</Description>
 <Price>10</Price>
 <Quantity>10</Quantity>
 </Item>
 </LineItems>
</ns0:CanonicalPurchaseOrder>

So how can we avoid this behavior?

However sometimes the destination schema expect a different message, without the namespaces in the element:

<CanonicalPurchaseOrder xmlns:ns0="http://BizTalkMapperSemanticTranslatorPattern.CanonicalPurchaseOrder">
  <From>From_0</From>
  <To>To_0</To>
  <LineItems>
     <Items>
        <Product>Product_0</Product>
        <Description>Description_0</Description>
        <Price>10</Price>
        <Quantity>10</Quantity>
     </Items>
  </LineItems>
</CanonicalPurchaseOrder>

I already saw this scenario many times. So how can we accomplish this?

There is no easy way to accomplished this without the need to use custom XSLT and you will need to customize the syntax to your scenario.

In this sample if we need to map everything without namespaces we need to “replace” the Mass Copy functoid for a Scripting functoid:

  • Drag-and-drop the Scripting functoid from the Toolbox to the map grid page and then:
    • And Link the Scripting functoid to the destination root name “CanonicalPurchaseOrder” to set the output parameter from the Scripting functoid;
    • This functoid will not have any input parameters;
  • Double-click in the Scripting functoid to show the Properties window, then:
    • Select the “Script Functoid Configure” tab, and then choose “Inline XSLT” as your selected script type
    • In the “Inline script” text box enter the following XSLT code without the comments (they are only for explaining the operation of the script):
<CanonicalPurchaseOrder>
      <xsl:for-each select="/s0:PurchaseOrder/*">
        <xsl:if test="local-name()!='LineItems'">
          <xsl:element name="{local-name(.)}">
            <xsl:value-of select="." />
          </xsl:element>
        </xsl:if>
      </xsl:for-each>

      <xsl:choose>
        <xsl:when test="count(/s0:PurchaseOrder/LineItems) &gt; 0">
          <LineItems>
            <xsl:for-each select="/s0:PurchaseOrder/LineItems/*">
              <Items>
                <xsl:for-each select="./*">
                  <xsl:element name="{local-name(.)}">
                    <xsl:value-of select="." />
                  </xsl:element>
                </xsl:for-each>
              </Items>
            </xsl:for-each>
          </LineItems>
        </xsl:when>
      </xsl:choose>
</CanonicalPurchaseOrder>

Basically what we are doing is:

  • The Scripting functoid will not add the root name, so we need to manually add it
  • The we are Getting all the element inside “PurchaseOrder” but because we need to travel all the elements inside the LineItems we need to treat this different, so we will get all the element except the ” LineItems”
  • The next step is check if there is “LineItems”
  • If exist we will travel all the element inside the LineItems (for each)
  • Because the “LineItems” only contain one record we will add this manually but we need an additional cycle this time to travel all the elements inside the record Item

And with that we create our own custom Mass copy Functoid

The sample code is available for download in Code Gallery:.

Automatically Link The Record Elements By Structure, Name or Using Mass Copy (148.5 KB)
Microsoft Code Gallery

We are getting close to the end of another year and this is the right time to do a personal end-of-year review and make some work/life balance. To do some of your own personal planning for the year ahead. What personal goals do you want to achieve over the next 12 months? But also to look back and reflect on what you achieved in this last year. So in the following of one of my previous posts, this is the list of my contributions in the last six months on MSDN Code Gallery and TechNet Gallery.

MSDN Code Gallery is your destination for downloading sample applications and code snippets, as well as sharing your own resources.

List of contributions on MSDN Code Gallery:

TechNet Gallery is your destination for finding resources written in VB Script, PowerShell, SQL, JavaScript, or other script languages, or other kind of resources (docs, Visio file and so on).

List of contributions on TechNet Gallery:

  • BizTalk Infrastructure and Solution Overview Samples: Creating documentation is boring and it can be a challenging task. However documentation is an important part on every BizTalk Integration projects or for infrastructures operations management and administration. They will help you in many critical situation and will reduce the time needed to carry out different procedures like finding were the machines are hosted, how many machines your environment have, what are the actors (systems, protocols) in a given solution and so on. This contribution is a list of diagrams samples that you can use in your documentation describing Infrastructure, Solution and Technical overview of your BizTalk Solution/Environment.
  • Monitoring Windows Updates and Pending Restarts on the servers using PowerShell: The script will monitor all the servers with Windows Updates ready to be installed, Windows Updates configured to be checked manually or servers that required a reboot. Servers that do not fall under these conditions will not be listed.
  • PowerShell to Configure BizTalk Server Host and Host Instances: One of the task that we need to do in all our new BizTalk environment over and over again is creating and configuring the Host, Host Instances and of course the adapter handlers. This PowerShell script will help you to automate this process according to some of the Best Practices.
  • Collection of Visio 2013 Stencil for BizTalk Server: New version of Visio 2013 Stencil for BizTalk Server with several modern flat Visio 2013 shapes for representing BizTalk Server physical architectures, Integration architectures or solutions diagrams in Visio 2013.
  • Collection of Visio 2010 Stencil for BizTalk Server: New version of Visio 2010 Stencil for BizTalk Server with several modern flat Visio shapes for representing BizTalk Server physical architectures, Integration architectures or solutions diagrams in Visio 2010.
  • Enable all the required IIS features for BizTalk Server with PowerShell: Microsoft Internet Information Services (IIS) provides a Web application infrastructure for many BizTalk Server features. BizTalk Server requires IIS for the following features: HTTP adapter, SOAP adapter, Windows SharePoint Services adapter, Secure Sockets Layer (SSL) encryption and BAM Portal. This PowerShell script to help you automate the process of configuring IIS with all the required features for BizTalk Server.

Some time ago, I do not remember when or where I was asked to provide some examples of diagrams that I use in my documentation or some diagrams representations in which I use the Visio Stencils for BizTalk Server

I know that no one likes to create documentation, is boring, it can be a challenging task, but documentation is an important part on every BizTalk Integration projects or for infrastructures operations management and administration. They will help you in many critical situation and will reduce the time needed to carry out different procedures like finding were the machines are hosted, how many machines your environment have, what are the actors (systems, protocols) in a given solution and so on. You can have at least three types of documentation when it comes to BizTalk:

  • Infrastructure Overview
  • Solution Overview
  • Technical Overview

Or as Steef-Jan mentioned in one of this posts:

  • BizTalk Environment (Design) document
  • Functional Design of BizTalk Solution
  • Technical (Design) of BizTalk Solution

Visio is great for creating multiple kinds of diagrams, unfortunately, the set of stencils and shapes, which out-of-the-box come with Visio provably are not sufficient to cover the kinds of diagrams we need with our BizTalk/Integration projects or they are not easy to use and read. Visio 2013 Stencil for BizTalk Server and Visio 2010 Stencil for BizTalk Server can easily help you creating documentation for the first two: BizTalk Environment (Design) document and Functional Design of BizTalk Solution. They are a collection of 73 modern flat Visio 2013 shapes that will help you representing BizTalk Server physical architectures, Integration architectures (Cloud or Hybrid scenarios) or solutions diagrams in Visio 2013 and 2010.

Infrastructure Overview Samples

This logical diagram describes the overview of your BizTalk environments where you can specify the Infrastructure of your production, UAT, testing or developer environments.

Here are some samples of:

  • Complex logical diagram of your Infrastructure

BizTalk-Infrastructure-Overview-Samples-1

  • Basic logical diagram of your Infrastructure

BizTalk-Infrastructure-Overview-Samples-2

  • Or for example a diagram representing a logical diagram of your BizTalk Group and Host Infrastructure

BizTalk-Host-Infrastructure-Overview-Sample

Solution Overview Samples

This logical diagram depicts a view of the solution. It will describe how solution will work, who are the actors, what type of protocols and so on.

Here are some samples of solution diagrams:

  • Complex logical diagrams

BizTalk-Solution-Overview-Sample BizTalk-Solution-Overview-Sample-2

  • Or basic logical diagram

BizTalk-Solution-Overview-Sample-3

This documentation should be available to all stakeholders within your organization: developer, operation and administration team.

Contrary to what many might think some kind of documentation is pretty straight forward and can be realized with minimal effort but have an enormous importance in the organization when you need to make changes inside your solution or infrastructure.

You can download BizTalk Infrastructure and Solution Overview Samples from:

BizTalk Infrastructure and Solution Overview Samples (11.4 MB)
Microsoft | TechNet Gallery

My last couple of posts I wrote about failures that can happen in the Backup BizTalk Server job for two reason:

  • First because I like to write about Errors, Warnings, Causes and Solutions, this is a good way to understand and have a good documentation of possible issues that can happen.
  • Second to create a TechNet Wiki article to demonstrate the best practices that you can have when you thinking in taking a question posted in BizTalk Server forums: Backup BizTalk Server job failed and convert it to a TechNet Wiki article.

In result I created 6 TechNet Wiki articles that will demonstrate some of the best practices, that I gathered after many conversations along with Steef-Jan Wiggers and Tord Gard Nordahl (some of the "Guardians of the BizTalk TechNet Wiki Articles"), that you can have and that will help you start creating your own articles in the TechNet Wiki:

  • Wiki: Best Practices For Converting a Forum Thread Into a New Wiki Article: This article is intended to be a list of all the best practices that you can have while converting a Forum thread into a new TechNet Wiki article.
  • Wiki: Templates For Converting a Forum Thread Into a New Wiki Article: This article is intended to be a list of all templates that can help TechNet Wiki members to easily create (you can copy the article structure and paste to your new article) and follow common layouts and guidelines when they try to convert a forum thread into a new Wiki article.
  • Wiki: Errors or Troubleshooting Template Format: If you are creating a new article about an Error or Troubleshooting, you can copy the article structure that exist in this article and paste to your new article. This template is intended to provide an initial path (model or pattern) for you to make the final article with a visual presentation of its structure and maybe input fields of data or description of the section that you should use.
  • Wiki: Questions and Solutions Template Format: If you are creating a new article about a solution to a question, you can copy the article structure that exist in this article and paste to your new article. This template is intended to provide an initial path (model or pattern) for you to make the final article with a visual presentation of its structure and maybe input fields of data or description of the section that you should use.
  • BizTalk Server: Backup BizTalk Server job failed Troubleshooting: This article has its origin in a question posted in BizTalk Server forums: Backup BizTalk Server job failed and will provide you with common problems, possible solutions, related solutions and how can you troubleshoot this problems. This article is also a demonstration how you can apply some of the best practices that you should have when taking a BizTalk Server questions from the forums and created a new BizTalk Server Wiki Articles.
  • BizTalk Server: How to map values from a repeating node into a single node using conditions: This article has its origin in a question posted in BizTalk Server forums: Reg BizTalk Mapping and will provide you with common possible solutions and try to explain what’s the best way to address and solve this problem. This article is also a demonstration how you can apply some of the best practices that you should have when taking a BizTalk Server questions from the forums and created a new BizTalk Server Wiki Articles.

In the last couple of months I have also finished many articles about BizTalk Server and Windows Azure BizTalk Services and this is the result!

List of contributions in English:

  • BizTalk Server Best Practices: Create and Configure BizTalk Server Host and Host Instances: BizTalk Server provides great flexibility for addressing high availability, because you can strategically dedicate logical hosts to run specific areas of functionality such as receiving messages, sending messages or processing orchestrations. Although a single BizTalk Host can contain items that receive, send, and process messages, it is considered a best practice to create different hosts for each function to create security boundaries and for easier management and scalability. In particular, we recommend that you use different hosts for processing and for receive/send operations, and that you separate trusted and non-trusted items. This is a list of some of the best practices that you can implement.
  • BizTalk Server: Webcasts and Videos: There are a few number of Web Casts and this article is intended to be a knowledge base of all BizTalk server Webcasts and Videos that are available.
  • List of Service Packs and Cumulative Updates available for BizTalk Server: This article lists all the available service packs and the cumulative updates for Microsoft BizTalk Server and for Microsoft BizTalk Adapter Pack.
  • BizTalk Server: Redistributable CAB Files: BizTalk Server relies on diverse products and technologies to provide essential services and features. If not already present, some software that is required by BizTalk Server is installed on your computer during the BizTalk installation process. This article lists all the available Redistributable CAB Files.
  • Windows Azure BizTalk Services Resources on the TechNet Wiki: This page contains links to all the Windows Azure BizTalk Services resources that are available on the TechNet Wiki.

Windows-Azure-BizTalk-Services-Resources-TechNet-Wiki

  • Windows Azure BizTalk Services: Webcasts and Videos: There are only a limited number of Web Casts and this article is intended to be a knowledge base of all Windows Azure BizTalk Services Webcasts and Videos that are available.
  • Windows Azure BizTalk Services: Code Gallery Samples: The MSDN Code Gallery enables you to download and share sample applications, code snippets, and other resources with the developer community where you can find a wide variety of all kinds of free sample code on different technologies ranging from general c# to advanced solutions. For Windows Azure BizTalk Services you can find a limited amount of sample solutions and code, however, if you got one that involves Windows Azure BizTalk Services, please add it here.
  • Windows Azure BizTalk Services: List of Errors and Warnings, Causes and Solutions: This article is intended to be a knowledge base of all Errors and Warnings, Causes and Solutions documented in all stages/components of Windows Azure BizTalk Services: different stages of development, deployment, runtime, setup and configuration…

List of contributions in Portuguese:

Feel free to edit, change and add content! And I hope you will find these articles useful.

Now I’m playing a little with this topic hehehe… but be with me, this have a final goal that I will reveal soon (probably two weeks from now since next week I will be away in the MVP Summit and BizTalk Summit in Seattle). Until then let’s try to find some of the “basic” possible errors that you can find in the Backup BizTalk Server job.

In my previous posts (here and here), I have explained two possible errors that you can find when trying to run this job, today we will talk about this one:

SQL Server Scheduled Job ‘Backup BizTalk Server (BizTalkMgmtDb)’ (0x4020953E49DEFA43B8FEC18D6AD9B062) – Status: Failed – Invoked on: 2013-11-12 08:50:46 – Message: The job failed. The Job was invoked by User BTS2013LAB01\Administrator. The last step to run was step 3 (MarkAndBackupLog). The job was requested to start at step 1 (Set Compression Option).

 

 

Backup-BizTalk-Server-job-warning-message-MarkAndBackupLog

Although the execution of job ‘Backup BizTalk Server (BizTalkMgmtDb)’ failed, no error is register in the event viewer only the warning showed above. This probably happens because the main task of this job: “BackupFull” responsible for performing a full database backups of the BizTalk Server databases, is properly executed. You can validate this in the tables “adm_BackupSettings” and “adm_BackupHistory”, or on the file system where the backup files are saved.

However, and as the message states, the job was not entirely performed and there is a problem in the third task: MarkAndBackupLog. Again the error message doesn’t help us much, so what you should do is go to the Backup BizTalk Server job history to try so see if you can obtain more information’s. In this case the log history show as much more information:

Log Job History (Backup BizTalk Server (BizTalkMgmtDb))
Step ID 3
Job Name Backup BizTalk Server (BizTalkMgmtDb)
Step Name MarkAndBackupLog
Sql Severity 15
Sql Message ID 105
Message
Executed as user: NT SERVICE\SQLSERVERAGENT. Incorrect syntax near ‘´’. [SQLSTATE 42000] (Error 102) Unclosed quotation mark after the character string ‘ /* location of backup files */ ‘. [SQLSTATE 42000] (Error 105). The step failed.

Backup-BizTalk-Server-job-error-history-MarkAndBackupLog

 

CAUSE

The MarkAndBackupLog step is responsible for marking the logs for backup, and then backing them up. This step execute the “sp_MarkAll” stored procedure that accepts two mandatory parameters:

  • @MarkName: Log mark name is part of the naming convention for backup files
  • @BackupPath:. You must change the destination path this to a valid one. It may be local or a UNC path to another server

And one optional parameter:

  • @UseLocalTime: This is an extra parameter that you can also add that tells the procedure to use local time

The mandatory parameters (MarkName and BackupPath) must be placed between single quotation marks:

  • MarkName: 'BTS', 'BTS2013LAB01' and so on
  • BackupPath: 'c:\backups'

The problem here is that for some reason I’m not using a single quotation mark:

exec [dbo].[sp_MarkAll] ‘BTS’ /* Log mark name */, C ….’ …

SOLUTION

Fix the job to ensure that all the parameters of the sp_MarkAll is between single quotation marks.

 

Another error that you can find in the Job history is:

Log Job History (Backup BizTalk Server (BizTalkMgmtDb))
Step ID 3
Job Name Backup BizTalk Server (BizTalkMgmtDb)
Step Name MarkAndBackupLog
Sql Severity 16
Sql Message ID 3013
Message
Executed as user: NT SERVICE\SQLSERVERAGENT. Cannot open backup device ‘K:\Backups\BizTalk Database\Logs\BTS2013LAB01_BAMAlertsApplication_Log_BTS_2013_11_12_17_51_07_990.bak’. Operating system error 3(The system cannot find the path specified.). [SQLSTATE 42000] (Error 3201) BACKUP LOG is terminating abnormally. [SQLSTATE 42000] (Error 3013). The step failed.

In this case you also get one or several errors in the Event Viewer:

BackupDiskFile::CreateMedia: Backup device ‘K:\Backups\BizTalk Database\Logs\BTS2013LAB01_BAMAlertsApplication_Log_BTS_2013_11_12_17_51_07_990.bak’ failed to create. Operating system error 3(The system cannot find the path specified.).

Backup-BizTalk-Server-job-error-message-MarkAndBackupLog

CAUSE

If you check the parameters of the “sp_MarkAll” stored procedure you will see that the second parameter is the location of backup files and this location must exist in the file system.

SOLUTION

You must remember that you need to ensure that all the paths specified in the BizTalk jobs must exist in the file system. In my case, we create the folder on the file system and run the job again with success.

After writing my last post I remembered that in the past I got a related error to this problem but with a different cause and solution. So I decided to emulate the error in my lab environment and make a second part of this topic.

I remember that on occasion, in order to be faster, I copied all the job configuration text from of my blog and pasted in the BackupFull step job window. However when I try to run I got a similar error:

SQL Server Scheduled Job ‘Backup BizTalk Server (BizTalkMgmtDb)’ (0x4020953E49DEFA43B8FEC18D6AD9B062) – Status: Failed – Invoked on: 2013-11-11 17:45:00 – Message: The job failed. The Job was invoked by Schedule 9 (MarkAndBackupLogSched). The last step to run was step 2 (BackupFull).

Backup-BizTalk-Server-job-error-message-MarkAndBackupLogSched

Again the error message doesn’t help us much, so what you should do is go to the Backup BizTalk Server job history:

  • Right click in “Backup BizTalk Server (BizTalkMgmtDb)” job and select “View History” option

And try so see if you can obtain more information about the error.

Unlike the event viewer and the error that I put in the previous post, in this case the log history show as much more information:

Date 11/11/2013 17:45:00
Log Job History (Backup BizTalk Server (BizTalkMgmtDb))
Step ID 2
Job Name Backup BizTalk Server (BizTalkMgmtDb)
Step Name BackupFull
Message
Executed as user: NT SERVICE\SQLSERVERAGENT. Incorrect syntax near ‘´’. [SQLSTATE 42000] (Error 102) Unclosed quotation mark after the character string ‘ /* location of backup files */ ‘. [SQLSTATE 42000] (Error 105). The step failed.

Backup-BizTalk-Server-job-error-history-Unclosed-quotation-mark

CAUSE

The copy of the text from my blog brought a wrong ASCII format transforming the single quotation mark character in an accent (´).

All the parameter of the sp_BackupAllFull_Schedule must be placed between single quotation marks:

  • Frequency: 'd', 'w' and so on
  • Name: 'BTS'
  • Location of backup files: 'c:\backups'

SOLUTION

Fix the job to ensure that all the parameters of the sp_BackupAllFull_Schedule is between single quotation marks.

For several years I’ve been setting up dozens of times the Backup BizTalk Server job without any kind of problems… but there’s a first time for everything and when I was trying to help remotely a friend configuring this job, we got stuck with this error when we try to start it:

Log Job History (Backup BizTalk Server (BizTalkMgmtDb))
Step ID 0
Job Name Backup BizTalk Server (BizTalkMgmtDb)
Step Name (Job outcome)
Message
The job failed. The Job was invoked by User domain\user. The last step to run was step 2 (BackupFull). The job was requested to start at step 1 (Set Compression Option).

Backup-BizTalk-Server-job-error-message

I knew that the job was configured correctly but unfortunately I didn’t have access to the SQL Server in order to see if I could find out more information about what was happening… Well, according to the error message we know that the job started in the first task “Set Compression Option” but failed in the second “BackupFull”. Beyond that the error message doesn’t help us much.

By going to the job history (right click in “Backup BizTalk Server (BizTalkMgmtDb)” job and select “View History” option)) we can validate that the problem in the BackupFull task but again the message described there doesn’t help us much.

Backup-BizTalk-Server-job-error-history

CAUSE

Unfortunately this error can happen for several reasons and we need to check what the best option for us is::

  • In Microsoft BizTalk Server 2004 and later, the Backup BizTalk Server job fails, and problems occur that relate to limited disk space. This happens when the backup files accumulate over time. This fills up the disk that houses the backup files.
    • This issue occurs because the SQL Server Agent job that is named “Backup BizTalk Server” does not provide functionality for deleting backup files that have accumulated over time. See: The “Backup BizTalk Server” job fails when backup files accumulate over time in the Microsoft BizTalk Server database server
    • Or you can create a Maintenance Plan to delete BizTalk Database Backups files see more here
    • I found also this great suggestion by Tord Glad Nordahl: “Verify that the compression option for the database is correct, after this open the management database located the table called “adm_BackupSettings” update this column “ForceFull” and set the value to 1. Then manually run the backup job again.” In this forum question.

But none of these suggestions work for me, in fact this was the first time that we were trying to run the job.

However sometimes solutions are simpler than it seems. If you check the parameters of the BackupFull step you will see that the third parameter is the location of backup files and this location must exist in the file system… so because I didn’t have access to the SQL Server machine, I asked to my friend to:

  1. Validate if the folder exist in the file system;
  2. See in the event viewer of the SQL Server if there was more information about the error;

The result was:

  1. The folder didn’t exist… never a good sign
  2. In the event viewer of the SQL Server Machine there was an entry saying the exact problem:

BackupDiskFile::CreateMedia: Backup device ‘K:\Backups\BizTalk Database\Full\BTS2013LAB01_BAMAlertsApplication_Full_BTS_2013_11_11_16_45_25_870.bak’ failed to create. Operating system error 3(The system cannot find the path specified.).

SOLUTION

You must remember that you need to ensure that all the paths specified in the BizTalk jobs must exist in the file system. In my case, we create the folder on the file system and run the job again with success.

Backup-BizTalk-Server-job-success

I have been receiving a few requests to release a new version of BizTalk Scheduled Task Adapter for BizTalk Server 2013. Actually I am planning a new version of this adapter, of course probably will be optimized for BizTalk Server 2013, but in reality this will not be the main focus.

BizTalk Scheduled Task Adapter Feature Roadmap

Here’s what you can expect to see in BizTalk Scheduled Task Adapter in the next few months:

  • Bring 1 or 2 really new features like JSON or REST support
  • Improve the existing features by solving some minor problems
  • Side-by-side installation
  • Improve documentation

 

But before I begin to develop a new version, I decided to perform a “battery of tests” with the current version of this adapter in order to know if there was any problem or if it was fully compatibility with latest platforms of Microsoft: BizTalk Server 2013, SQL Server 2012 and Windows Server 2012.

The good news is that for now I didn’t detect any kind of problems:

  • I was able to install and configure the adapter on a BizTalk Server 2013 running in a Windows Server 2012 with SQL Server 2012:

Scheduler-Adapter-running-BTS2013

  • And I was able to create a new receive location and specify a scheduler to run the XmlStringStreamProvider to send an XML message every one hour

Scheduler-Adapter-running-BTS2013-add-receive-locationScheduler-Adapter-running-BTS2013-configure-scheduler-receive-location

Scheduler-Adapter-running-BTS2013-configure-scheduler-receive-location-2

I have not done all the tests possible, but I can say that the bases features tested in the latest version of this adapter are fully compatible with BizTalk Server 2013.

If you found any problem running this version in BizTalk Server 2013 or 2010, feel free to email me with your problem or add a comment.