Posts Tagged ‘Mapping’

Sometimes I ask myself: Why is so hard to make a simple If-Then-Else Functoid, or even so painful to do an If-Then-Else operation, using BizTalk mapper?

I don’t mean to say that it is complicated, quite the opposite, is quite easy to make If…Then…Else statements using the Mapper. You can use If…Then…Else statements, to be completely correct, you can use something related to an If…Then…Else statements, to execute blocks of statements depending on the Boolean value of a condition by, normally, using:

  • One Logical Functoid (Logical Existence, Logical String, Logical Numeric, Equal, Greater Than, Less Than and so on) to determine:
    • whether the Record, Field Element, or Field Attribute node that is linked to it exists or have a valid value in a particular input instance message
    • or if a condition match.
  • One Logical NOT Functoid to negate the Logical Functoid
  • And two Value Mapping Functoids to returns the value that we want to linked based on the result of the condition

BizTalk-out-of-the-box-if-then-else-condition

In this sample, if the “Operation” element is equal to “Create”

  • Then: you will map the value of the element “ValueA” from the source schema to the element “Result” in the destination schema
  • Else (otherwise): you will map the value of the element “ValueB” from the source schema to the element “Result” in the destination schema

So, this functoid chain will provide a secondary path of execution when an “if” clause evaluates to false, the Else path. I said earlier: “something related to an If…Then…Else statement” because, programming speaking, If…Then…Else  statements are implemented by:

//C#
if(condition)
{
    //something if true
}
else
{
    //something if false
}

Or

<!-- XSLT -->
<xsl:choose>
  <xsl:when test="expression">
    ... something if true ...
  </xsl:when>
  <xsl:otherwise>
    ... something if false ...
  </xsl:otherwise>
</xsl:choose>

But in fact what this functoid chain does is two If statements

<!-- something if true -->
<xsl:if test="string($var:v1)='true'">
  <xsl:variable name="var:v2" select="ValueA/text()" />
  <Result>
    <xsl:value-of select="$var:v2" />
  </Result>
</xsl:if>
<!-- something if false (var:v3 contains the negation of var:v1) -->
<xsl:if test="string($var:v3)='true'">
  <xsl:variable name="var:v4" select="ValueB/text()" />
  <Result>
    <xsl:value-of select="$var:v4" />
  </Result>
</xsl:if>

This approach is good for small messages, but even with small messages, the Mapper will add many unnecessary code and operations. Because of that I decided to create a If-Then-Else Functoid to use in these situations and improve a little more the performance of the map, of course, undoubtedly, that the best option and with most performance to make conditions is using custom XSLT code (but this is another story).

The annoying things in using this out-of-the-box functois are:

  • We get these  annoying warnings (but they are useful in many scenarios) when we validate the maps:
    • warning btm1004: The destination node “Result” has multiple inputs. For a destination node to have multiple inputs, one of its ancestors should be connected to a looping functoid.
  • And for a simple if-then-else operation we need to use a minimum of 4 functoids. If we have several conditions inside the map, it will be easily filled with functoids and in result, will become a little confused and sometimes difficult to manager.
  • And, obvious, we will have several unnecessary operations.
So, why is so hard to make an If-Then-Else Functoid?

What I want to archive is create a custom functoid that accepts 3 inputs:

  • A Boolean – the result of a previous Logical Functoid (Logical Existence, Logical String, Logical Numeric, Equal, Greater Than, Less Than and so on)
  • And two inputs

Were the custom Functoid will return a value from one of two input parameters based on a condition.

  • If the condition (first input) is True, then the value of the second input parameter is returned;
  • Otherwise the Third input is returned.

Translating to C# code, will be something like this:

public string IfThenElseOperation(string condition, string trueValue, string falseValue)
{
    if (System.Convert.ToBoolean(condition))
        return trueValue;
    return falseValue;
}

Note: I will not address this topic here, this is content deserved a completely different post and dedicated to the topic, but the best option that you have is create a custom functoid belonging to the String functoid category (if you like it, put a comment in the post and I will address this topic in another time/post).

However, if you do that, you will find that, out-of-the-box, it is impossible to create a custom functoid based on a Logical Functoid and the reason why this is true is that, all Logical Functoids available out-of-the-box with BizTalk only accept the following outputs connection types:

  • ConnectionType.Element
  • ConnectionType.FunctoidAssert
  • ConnectionType.FunctoidNilValue
  • ConnectionType.FunctoidKeyMatch
  • ConnectionType.FunctoidTableLooping
  • ConnectionType.FunctoidValueMapping
  • ConnectionType.FunctoidScripter
  • ConnectionType.FunctoidLogical;

ConnectionType Enumeration: Specifies the types of connections that can be used as inputs or outputs for a functoid. This enumeration has a FlagsAttribute attribute that allows a bitwise combination of its member values.

ConnectionType.FunctoidString is not allowed in all the existing Logical Functoids. And that is the reason why you will find impossible to create a custom functoid based on a Logical functoid (don’t know the reason why Microsoft decide to implement this limitation)

How did I solve (or I overcame) this limitation?

To solve (work around) this limitation, so that I could create and use a custom if-then-else functoid, at the same time be fully compatible with existing functoids and don’t produce any more additional code, I was force to create my own personal custom Logical Functoids:

  • Advance Logical AND Functoid
  • Advance Equal Functoid
  • Advance Greater Than Functoid
  • Advance Greater Than or Equal To Functoid
  • Advance Less Than Functoid
  • Advance Less Than or Equal To Functoid
  • Advance Not Equal Functoid
  • Advance Logical NOT Functoid
  • Advance Logical OR Functoid

Toolbox-with-custom-logical-functoids

That have the same behavior as the out-of-the-box Logical Functoids:

  • Logical AND Functoid
  • Equal Functoid
  • Greater Than Functoid
  • Greater Than or Equal To Functoid
  • Less Than Functoid
  • Less Than or Equal To Functoid
  • Not Equal Functoid
  • Logical NOT Functoid
  • Logical OR Functoid

with the advantage that also accept the String output connection type.

base.OutputConnectionType = ConnectionType.FunctoidString | ConnectionType.Element | ConnectionType.FunctoidAssert | ConnectionType.FunctoidNilValue | ConnectionType.FunctoidKeyMatch | ConnectionType.FunctoidTableLooping | ConnectionType.FunctoidValueMapping | ConnectionType.FunctoidScripter | ConnectionType.FunctoidLogical;

By doing that, I can create my if-then-else custom functoid:

namespace BizTalk.CustomAdvanced.Functoids
{
    [Serializable]
    public class IfThenElse : BaseFunctoid
    {
        public IfThenElse()
            : base()
        {
            //ID for this functoid
            this.ID = 10900;

            // resource assembly must be ProjectName.ResourceName if building with VS.Net
            SetupResourceAssembly("BizTalk.Logical.Functoids.LogicalResources", Assembly.GetExecutingAssembly());

            //Setup the Name, ToolTip, Help Description, and the Bitmap for this functoid
            SetName("IDS_IFELSEFUNCTOID_NAME");
            SetTooltip("IDS_IFELSEFUNCTOID_TOOLTIP");
            SetDescription("IDS_IFELSEFUNCTOID_DESCRIPTION");
            SetBitmap("IDS_IFELSEFUNCTOID_BITMAP");

            //category for this functoid. This functoid goes under the String Functoid Tab in the
            this.Category = FunctoidCategory.String;

            // Set the limits for the number of input parameters. This example: 1 parameter
            this.SetMinParams(3);
            this.SetMaxParams(3);

            // Add one line of code as set out below for each input param. For multiple input params, each line would be identical.
            this.AddInputConnectionType(ConnectionType.AllExceptRecord); //first input
            this.AddInputConnectionType(ConnectionType.AllExceptRecord); //Second input
            this.AddInputConnectionType(ConnectionType.AllExceptRecord); //Third input

            // The functoid output can go to any node type.
            this.OutputConnectionType = ConnectionType.AllExceptRecord;

            SetScriptBuffer(ScriptType.CSharp, this.GetCSharpBuffer());
            HasSideEffects = false;
        }

        private string GetCSharpBuffer()
        {
            StringBuilder builder = new StringBuilder();
            builder.Append("public string IfThenElseOperation(string condition, string trueValue, string falseValue)\n");
            builder.Append("{\n");
            builder.Append("\tif (System.Convert.ToBoolean(condition))\n");
            builder.Append("\t\treturn trueValue;\n");
            builder.Append("\treturn falseValue;\n");
            builder.Append("}\n");
            return builder.ToString();
        }
    }
}

And have the ability to connect my logical functoids to a custom String functoid. Of course you need to use it wisely!

BizTalk-custom-if-then-else-condition

The beauty of this approach is that:

  • We reduce the functoids chain. By using less functoids we simplify the map visually, and this is very useful when you have a lot of conditions, even if they are simple conditions.
  • We will improve the map performance a little more, compared to the use of out-of-the-box functoids
    • Using Custom If-Then-Else Functoid:
<xsl:variable name="var:v1" select="userCSharp:LogicalEq(string(Operation/text()) , &quot;Create&quot;)" />
<xsl:variable name="var:v2" select="userCSharp:IfThenElseOperation(string($var:v1) , string(ValueA/text()) , string(ValueB/text()))" />

    <ns0:Output>
      <Result>
        <xsl:value-of select="$var:v2" />
      </Result>
    </ns0:Output>

public bool LogicalEq(string val1, string val2)
{
	bool ret = false;
	double d1 = 0;
	double d2 = 0;
	if (IsNumeric(val1, ref d1) && IsNumeric(val2, ref d2))
	{
		ret = d1 == d2;
	}
	else
	{
		ret = String.Compare(val1, val2, StringComparison.Ordinal) == 0;
	}
	return ret;
}

public string IfThenElseOperation(string condition, string trueValue, string falseValue)
{
	if (System.Convert.ToBoolean(condition))
		return trueValue;
	return falseValue;
}

public bool IsNumeric(string val)
{
	if (val == null)
	{
		return false;
	}
	double d = 0;
	return Double.TryParse(val, System.Globalization.NumberStyles.AllowThousands | System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out d);
}

public bool IsNumeric(string val, ref double d)
{
	if (val == null)
	{
		return false;
	}
	return Double.TryParse(val, System.Globalization.NumberStyles.AllowThousands | System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out d);
}
    • Using Out-of-the-box Functoids:
<xsl:variable name="var:v1" select="userCSharp:LogicalEq(string(Operation/text()) , &quot;Create&quot;)" />
<xsl:variable name="var:v3" select="userCSharp:LogicalNot(string($var:v1))" />

    <ns0:Output>
      <xsl:if test="string($var:v1)='true'">
        <xsl:variable name="var:v2" select="ValueA/text()" />
        <Result>
          <xsl:value-of select="$var:v2" />
        </Result>
      </xsl:if>
      <xsl:if test="string($var:v3)='true'">
        <xsl:variable name="var:v4" select="ValueB/text()" />
        <Result>
          <xsl:value-of select="$var:v4" />
        </Result>
      </xsl:if>
    </ns0:Output>
  
public bool LogicalEq(string val1, string val2)
{
	bool ret = false;
	double d1 = 0;
	double d2 = 0;
	if (IsNumeric(val1, ref d1) && IsNumeric(val2, ref d2))
	{
		ret = d1 == d2;
	}
	else
	{
		ret = String.Compare(val1, val2, StringComparison.Ordinal) == 0;
	}
	return ret;
}

public bool LogicalNot(string val)
{
	return !ValToBool(val);
}

public bool IsNumeric(string val)
{
	if (val == null)
	{
		return false;
	}
	double d = 0;
	return Double.TryParse(val, System.Globalization.NumberStyles.AllowThousands | System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out d);
}

public bool IsNumeric(string val, ref double d)
{
	if (val == null)
	{
		return false;
	}
	return Double.TryParse(val, System.Globalization.NumberStyles.AllowThousands | System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out d);
}

public bool ValToBool(string val)
{
	if (val != null)
	{
		if (string.Compare(val, bool.TrueString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return true;
		}
		if (string.Compare(val, bool.FalseString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return false;
		}
		val = val.Trim();
		if (string.Compare(val, bool.TrueString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return true;
		}
		if (string.Compare(val, bool.FalseString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return false;
		}
		double d = 0;
		if (IsNumeric(val, ref d))
		{
			return (d > 0);
		}
	}
	return false;
}
  • You don’t need to deploy any of these custom functoids (advance logical functoids or/and if-the-else functoid) to your production environment, you only need to have then in your development environment to be used inside Visual Studio because all of them are custom inline functoids, i.e., they will add the necessary code inside the XSLT file:
  • And finally, you can use all the custom advance Logical Functoids allow side with the out-of-the-box Logical Functoids without any impact and without producing additional code, because they will produce the exact same code of out-of-the-box Logical Functoids!

BizTalk-custom-and-out-of-the-box-if-then-else-condition-side-by-side

Generated code:

<!-- Custom Functoid -->
<xsl:variable name="var:v1" select="userCSharp:LogicalEq(string(Operation/text()) , &quot;Create&quot;)" />
<xsl:variable name="var:v2" select="userCSharp:IfThenElseOperation(string($var:v1) , string(ValueA/text()) , string(ValueB/text()))" />

<!-- out-of-the-box Functoids -->
<xsl:variable name="var:v3" select="string(Operation/text())" />
<xsl:variable name="var:v4" select="userCSharp:LogicalEq($var:v3 , &quot;Create&quot;)" />
<xsl:variable name="var:v6" select="userCSharp:LogicalNot(string($var:v4))" />

    <ns0:Output>
<!-- Custom Functoid -->
      <Result>
        <xsl:value-of select="$var:v2" />
      </Result>
<!-- out-of-the-box Functoids -->
      <xsl:if test="string($var:v4)='true'">
        <xsl:variable name="var:v5" select="ValueA/text()" />
        <Total>
          <xsl:value-of select="$var:v5" />
        </Total>
      </xsl:if>
      <xsl:if test="string($var:v6)='true'">
        <xsl:variable name="var:v7" select="ValueB/text()" />
        <Total>
          <xsl:value-of select="$var:v7" />
        </Total>
      </xsl:if>
    </ns0:Output>
  </xsl:template>

<!-- Both out-of-the-box Functoids and custom functoids will produce this LogicalEq code! -->
public bool LogicalEq(string val1, string val2)
{
	bool ret = false;
	double d1 = 0;
	double d2 = 0;
	if (IsNumeric(val1, ref d1) && IsNumeric(val2, ref d2))
	{
		ret = d1 == d2;
	}
	else
	{
		ret = String.Compare(val1, val2, StringComparison.Ordinal) == 0;
	}
	return ret;
}

public string IfThenElseOperation(string condition, string trueValue, string falseValue)
{
	if (System.Convert.ToBoolean(condition))
		return trueValue;
	return falseValue;
}

public bool LogicalNot(string val)
{
	return !ValToBool(val);
}

public bool IsNumeric(string val)
{
	if (val == null)
	{
		return false;
	}
	double d = 0;
	return Double.TryParse(val, System.Globalization.NumberStyles.AllowThousands | System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out d);
}

public bool IsNumeric(string val, ref double d)
{
	if (val == null)
	{
		return false;
	}
	return Double.TryParse(val, System.Globalization.NumberStyles.AllowThousands | System.Globalization.NumberStyles.Float, System.Globalization.CultureInfo.InvariantCulture, out d);
}

public bool ValToBool(string val)
{
	if (val != null)
	{
		if (string.Compare(val, bool.TrueString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return true;
		}
		if (string.Compare(val, bool.FalseString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return false;
		}
		val = val.Trim();
		if (string.Compare(val, bool.TrueString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return true;
		}
		if (string.Compare(val, bool.FalseString, StringComparison.OrdinalIgnoreCase) == 0)
		{
			return false;
		}
		double d = 0;
		if (IsNumeric(val, ref d))
		{
			return (d > 0);
		}
	}
	return false;
}

May not be the perfect solution, I agree on that but I found a very interesting and useful approach for many scenarios. I hope you enjoy it, because I have been used it a lot lately 🙂

BizTalk-custom-and-out-of-the-box-if-then-else-condition-side-by-side-2

The functoids will be available in the next version of BizTalk Mapper Extensions UtilityPack, until then you have all the source code and samples here:

BizTalk Mapper: How to create a custom If-Then-Else Functoid (18,6 KB)
Microsoft | Code Gallery

It’s always good to analyze code performed by others, please do not consider this a criticism, it is not my intention. Doing that you will, compare technics, learn new things, noted and we become aware of how some transformation rules are made inside the maps… and sometimes, most often when things are done in the wrong way, or not quite correct, we gain inspiration or idea to talk about it. And this is one of them, that in fact, I have seen often happening: Applying conditions (if-then-else) using sometimes complex Functoid chain.

So far nothing new, this is a trivial operation that we normally, or often, do. The thing is that often we do it wrong or not quite well.

I have already addressed the processing model of BizTalk maps on my BizTalk Mapping Patterns and Best Practices book but it is always useful to refer again. BizTalk maps follows the model below:

  • The BizTalk mapping engine traverses the destination schema from beginning to end;
  • The mapping rules are constructed and executed as links are encountered in the destination schema;
  • The information is extracted from the source when a link is encountered in the destination schema.

But you need to be extremely aware, and this is important, that when BizTalk mapper engine reach to a condition rules to construct, all the operation that are present downstream of the condition needs to be translated (executed) before the condition rule itself.

Let’s look to the following example:

Complex-Functoid-Chain-Condition-wrong

Where the basic rule that we want to implement is:

  • If the operation is equal to “insert”
    • Then we need to set, in the Cross Referencing tables, and return an identifier (Set Common ID Functoid), based on some content from the source schema and in some values present or not in the Cross Referencing tables (Get Common ID Functoid), and mapped to the “CommonId” element in the destination schema
  • If the operation is different of “insert”
    • Then we need to retrieve an identifier from the Cross Referencing tables (Get Common ID Functoid), again based on some content from the source schema, and mapped the result to the “CommonId” element in the destination schema

So, to be simple and clear, what the BizTalk mapper engine is doing in the picture above is:

  • It found 2 rules to translate, because we have two links connected with the “CommonId” element
  • In the first rule it will: If the operation is equal to “insert”
    • Get one element from the “keys” record in the source schema and put it into a variable
    • Try’s to retrieve two identifiers’, based on some content from the source schema or static data.
    • From the identifiers’ retrieve earlier, it applies some transformation rule (not important for this demo)
    • Set and return an identifier, based on the content of:
      • one element from the “keys”
      • and the transformation rule, associated to the identifiers’ retrieve earlier
    • And finally, will check if the operation is equal to “insert”
      • Then map the value of the identifier created in the previous step to the element “CommonId”
  • In the second rule it will: If the operation is different of “insert”
    • Try’s to retrieve two identifiers’, based on some content from the source schema or static data.
    • From the identifiers’ retrieve earlier, it applies some transformation rule (not important for this demo)
    • And finally, will check if the operation is different to “insert”
      • Then map the value of the identifier created in the previous step to the element “CommonId”

Are you already seeing the problem with this sample?

Let’s analyze the generated XSLT code and try to see if it makes more sense and detect the problem in a clear way:

<xsl:variable name="var:v14" select="userCSharp:LogicalNe(&quot;insert&quot; , string(@operation))" />
<xsl:variable name="var:v23" select="userCSharp:LogicalEq(&quot;insert&quot; , $var:v22)" />
<xsl:variable name="var:v18" select="ScriptNS1:GetCommonID(&quot;INPUT1&quot; , &quot;INPUT2&quot; , string($var:v17))" />
<xsl:variable name="var:v19" select="ScriptNS1:GetCommonID(&quot;INPUT1&quot; , &quot;INPUT3&quot; , string(keys/myelement/text()))" />
<xsl:variable name="var:v15" select="ScriptNS0:GetValue()" />
<xsl:variable name="var:v20" select="ScriptNS2:GetValue(string($var:v15) , string($var:v18) , string($var:v19))" />

<xsl:if test="string($var:v14)='true'">
     <xsl:variable name="var:v21" select="string($var:v20)" />
     <btsCommonId>
            <xsl:value-of select="$var:v21" />
     </btsCommonId>
</xsl:if>

<xsl:variable name="var:v25" select="ScriptNS1:SetCommonID(&quot;FamilyMemberVehicle&quot; , &quot;CRM&quot; , $var:v24 , string($var:v20))" />

<xsl:if test="string($var:v23)='true'">
     <xsl:variable name="var:v26" select="string($var:v25)" />
     <btsCommonId>
            <xsl:value-of select="$var:v26" />
     </btsCommonId>
</xsl:if>

And now, do you see the problem with this sample?

The problem is that independent of the type of operation, if is an insert or other operation, it will always set an identifier in the Cross Referencing tables! (which by the way in my scenario, will induce problems – violate key). And this happens because the condition was defined at the end of the rule (in the right corner)

Complex-Functoid-Chain-Condition-problem

How you should read the rules

Important Note: basically, there are some exceptions, you always need to read the rules:

  • from top to bottom in the order that they happier in the destination schema
  • and from the left to the right (from the source schema to the destination schema) for a specific link connected to an element, field or record in the destination schema;

So, in this case the solution here is very simple, we need to move the condition to a position further downstream, which will allow to be executed sooner, especially the condition is equal to “insert”, because the second part (get two identifiers’ from the Cross Referencing tables) is common for both rules (then and else).

In this case we need to place the condition before we execute the set and return an identifier from the Cross Referencing tables operation, as the picture bellow shows:

Complex-Functoid-Chain-Condition-right

If we check the XSLT code once again, now we will notice that:

  • We are doing all the common operations before the condition rule to e executed
  • And for each particular scenario we will map the result (not equal to “insert”) or apply more operations and map the result (equal to “insert”)
<xsl:variable name="var:v14" select="userCSharp:LogicalNe(&quot;insert&quot; , string(@operation))" />
<xsl:variable name="var:v18" select="ScriptNS1:GetCommonID(&quot;INPUT1&quot; , &quot;INPUT2&quot; , string($var:v17))" />
<xsl:variable name="var:v19" select="ScriptNS1:GetCommonID(&quot;INPUT1&quot; , &quot;INPUT3&quot; , string(keys/myelement/text()))" />
<xsl:variable name="var:v15" select="ScriptNS0:GetValue()" />
<xsl:variable name="var:v20" select="ScriptNS2:GetValue(string($var:v15) , string($var:v18) , string($var:v19))" />
<xsl:variable name="var:v23" select="userCSharp:LogicalEq(&quot;insert&quot; , $var:v22)" />

<xsl:if test="string($var:v14)='true'">
     <xsl:variable name="var:v21" select="string($var:v20)" />
     <btsCommonId>
            <xsl:value-of select="$var:v21" />
     </btsCommonId>
</xsl:if>

<xsl:if test="string($var:v23)='true'">
     <xsl:variable name="var:v24" select="string($var:v20)" />
     <xsl:variable name="var:v25" select="string(keys/siva_viaturaagregadoid/text())" />
     <xsl:variable name="var:v26" select="ScriptNS1:SetCommonID(&quot;FamilyMemberVehicle&quot; , &quot;CRM&quot; , $var:v25 , string($var:v24))" />
     <btsCommonId>
            <xsl:value-of select="$var:v26" />
     </btsCommonId>
</xsl:if>

Like this scenario there are several, more or less complex, more or less critical. Some of them we don’t “even notice” because everything works well but if we check careful, sometimes will make several unnecessary operations that can induce, or may induce, performance problems in our transformations.

Good practices

As a reference and good practice, when you are implement conditions using Functoids chains, you should make the conditions as early as possible, i.e., put the conditions (if-then-else) as possible as you can in the left side of the Functoid chain.

This will allow you:

  • To group and execute certain task only on the context of the condition (inside the <xsl:if> statement)
  • And better performance because we are reducing the “noise” produced in the XSLT generated code by the compiler.

Easy, isn’t it?
Hope you enjoy it.

Well let’s go back to the topic: you are doing crazy things with your maps!

I was thinking that I had already seen it all, I was thinking that I had already seen it all, but the reality is that I continue to be blown away. While testing a project migration I end up catching this error:

Xlang/s engine event log entry: Uncaught exception (see the ‘inner exception’ below) has suspended an instance of service ‘[MY_ORCHESTRATION_NAME] (1582c665-816d-07db-3dee-1c6a750944a0)’.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: 4e0d3e55-b481-442d-a2a4-9f4ee6e13fb7
Shape name: Create Client Msg
ShapeId: 0e4a8cba-c6a0-4a0f-ad94-574b2e8ab7ad
Exception thrown from: segment 1, progress 54

Inner exception: A failure occurred while evaluating the distinguished field [MY_FIELD_NAME] against the message part data. The message part data does not contain at least one of the nodes specified by the XPath expression (listed below) that corresponds to the distinguished field. The cause for this error may be that the message part data has not been initialized or that the message part data does not conform to the message part schema. Ensure that the message part data is initialized correctly. XPath expression: /*[local-name()=’ROOT_NAME’ and namespace-uri()=’NAMESPACE’]/*[local-name()=’MY_FIELD_NAME’ and namespace-uri()=”]

Exception type: XPathUpdateException
Source: Microsoft.XLANGs.Engine
Target Site: Void SetDistinguishedField(System.String, System.Object)
The following is a stack trace that identifies the location where the exception occurred
at Microsoft.XLANGs.Core.XSDPart.SetDistinguishedField(String dottedPath, Object val)
at Siva.CRM.EAI.SIVOrchestration.Client.SIV_Client_ClientAccountSubscriber_UpdDel.segment1(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)

I got curious to see that this problem, as the error specify clearly, was related to a distinguished field element and it was happening inside a Construct Message that was using a map.

CAUSE

The error details points for the cause of the problem and it is for sure something related to the XPath expression: /*[local-name()=’ROOT_NAME’ and namespace-uri()=’NAMESPACE’]/*[local-name()=’MY_FIELD_NAME’ and namespace-uri()=”], however, the cause may not be exactly the same.

In my case when I opened the map, I easily realize that the field that was promoted as distinguished field was being mapped twice, i.e., two source elements where being mapped to the same destination element what was causing this element to appear twice in the final message even though the destination element is defined as Max Occurs 1 and Min Occurs 0.

wrong-transformation-rule-causing-error-evaluating-distinguished-field

Of course you may know that distinguished fields are essentially XPath aliases, which simply point to the appropriate XML data field. They don’t have a size limitation but they only can appear once. You cannot promote a repeating element.

And you also should know that the map doesn’t validate the input or the output messages in runtime!

And the curious thing was that this particular distinguished field it was not being used inside the orchestration!

SOLUTION

The solution it easy for this case.

First:

  • Don’t promote elements that you are not really using or need to do some logic inside your orchestrations… If you don’t need them, don’t promote them. Distinguished fields cost less than Property Fields but they still have a cost for performance!

Second:

  • Fix the ma rule associated with this element. In my case I decide to remove the “nipc” rule and leave only the “accountId” direct link to the “EntityGuid”

In the last days I’ve been migrating old maps from BizTalk Server 2004 to the last versions of BizTalk Server: 2013 and 2013 R2 and I have seen several “styles/approaches” to address mappings, some of them great, some of them not so much and other completely crazy.

Today I will address a topic that I, unfortunately, have seen constantly: Reusing Scripting Functoids with Inline C# inside the same map. This is also something that I address in my book: “BizTalk Mapping Patterns & Best Practices

Inline scripts are convenient for custom code that you are unlikely to use elsewhere in your application or other maps. In addition to being convenient for one-time scripts, inline scripts are also useful for declaring global variables for use among a number of scripts and to use several times inside the same map.

In general, some of the main reasons to use Scripting Functoid are:

  • Perform functions otherwise not available with built-in Functoids
  • It also allows you to perform complex logic transformations that are impossible to make with built-in Functoids.
  • To simplify the map, making it sometimes more easy to read, instead of using a complex Functoid chain
  • Or simply to optimize the transformation rules

However, you need to have some precautions when using, or reusing the same, Custom Inline C# scripts inside the Scripting Functoid:

  • Function Names Limitation
  • Compiler limitations
  • Reusability
Function Names Limitation

For each function/method inside the Scripting Functoid you need to give different Method statements (name of the method + method parameters). If you have the same Method statement with different behaviors inside, for example:

  • In the first Scripting Functoid:
public string FormatDate(string inputDate, string inputFormat, string outputFormat)
{
    System.DateTime date;
    if (System.DateTime.TryParseExact(inputDate, inputFormat, 
        System.Globalization.CultureInfo.InvariantCulture, 
        System.Globalization.DateTimeStyles.AssumeLocal, out date))
    {
        return date.ToString(outputFormat);
    }
    return "";
}
  • And in the second Scripting Functoid:
public string FormatDate(string inputDate, string inputFormat, string outputFormat)
{
    if (String.IsNullOrEmpty(inputDate))
        return System.DateTime.Now.ToString(outputFormat);
    return inputDate;
}

Different-Scripting-Functoid-With-The-Same-Name-Not-Ok

Because both methods have the name and the same number of parameters, the mapper will interpret has the same, regardless if the code inside the functions are the same of not. In this case the first mapping rules to be reached is the first scripting functoid, which means that the second code inside the second functoid will be ignored and instead will be executed the exact same code that the first one, which is not what intended to do.

You can validate this map behavior if you validate the map and see the XSL produced:

<msxsl:script language="C#" implements-prefix="userCSharp"><![CDATA[ 
public string FormatDate(string inputDate, string inputFormat, string outputFormat) 
{
 System.DateTime date;
 if (System.DateTime.TryParseExact(inputDate, inputFormat, System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.AssumeLocal, out date))
 {
    return date.ToString(outputFormat);
 }
 return "";
} 
]]></msxsl:script>

To fixed this we need to give a different name to the second method, for example:

public string ValidateAndFormatDate(string inputDate, string inputFormat, string outputFormat)
{
    if (String.IsNullOrEmpty(inputDate))
        return System.DateTime.Now.ToString(outputFormat);
    return inputDate;
}

Nevertheless, you can give the same method name, if the method has a different number of inputs, for example this case is a valid example:

  • In the first Scripting Functoid:
public string FormatDate(string inputDate, string inputFormat, string outputFormat)
{
    System.DateTime date;
    if (System.DateTime.TryParseExact(inputDate, inputFormat, 
        System.Globalization.CultureInfo.InvariantCulture, 
        System.Globalization.DateTimeStyles.AssumeLocal, out date))
    {
        return date.ToString(outputFormat);
    }
    return "";
}
  • And in the second Scripting Functoid:
public string FormatDate()
{
    return System.DateTime.Now.ToString();
}

Different-Scripting-Functoid-With-The-Same-Name-Ok

Once again if you validate the map to see the XSL produced, you can see that in this case both functions are being generated:

 <msxsl:script language="C#" implements-prefix="userCSharp"><![CDATA[ 
public string FormatDate(string inputDate, string inputFormat, string outputFormat) 
{
 System.DateTime date;
 if (System.DateTime.TryParseExact(inputDate, inputFormat, System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.AssumeLocal, out date))
 {
   return date.ToString(outputFormat);
 }
 return "";
} 
public string FormatDate() 
{
 return System.DateTime.Now.ToString(); 
} 
]]></msxsl:script> 
Compiler limitations

When you create a script, for example a C# function, remember to give it a correct name because all scripts with the same name are treated as one by the compiler regardless of whether the code is equal or not.

You also have to take in consideration that BizTalk Mapper engine will save the inline scripts in the Extensible Stylesheet Language Transformations (XSLT) Stylesheet defining the map in the within the msxsl:script element. The following namespaces are supported by default: System, System.Collection, System.Text, System.Text.RegularExpressions, System.Xml, System.Xml.Xsl, System.Xml.Xpath, Microsoft.VisualBasic. You can add support for additional namespaces outside the listed using the namespace attribute of the child element, <msxsl:using>, of the <msxsl:script> element, but this will force us to use an External Custom XSLT file to accomplish the transformation. See more here: XSLT Stylesheet Scripting Using <msxsl:script> (http://msdn.microsoft.com/en-us/library/533texsx%28v=vs.110%29.aspx)

Reusability

Now let go to the main topic of this post Reusability of Scripting Functoids with Inline C# inside the same map.

What I normally see is people doing two things:

  • Copy and paste the entire method to all the Scripting Functoids – I’m calling that: Reusing in a Bad Way
    • In this case both scripting functoids have the exact same code
public string FormatDate(string inputDate, string inputFormat, string outputFormat)
{
    System.DateTime date;
    if (System.DateTime.TryParseExact(inputDate, inputFormat, 
        System.Globalization.CultureInfo.InvariantCulture, 
        System.Globalization.DateTimeStyles.AssumeLocal, out date))
    {
        return date.ToString(outputFormat);
    }
    return "";
}

Reuse-Scripting-Functoid-Bad-Way

This is a very simple case, now imagine a map with several grid pages and several functoids. The problem is this approach, as we explained earlier, is that if we for some reason change the code of one the scripting functoids, that is not the first to be translated by the mapper engine, the changes will not take effect!!!

  • Copy and paste the entire method to all the Scripting Functoids and give it a different name! – I’m calling that: Reusing in a Crazy Way
    • In this case the first scripting functoids have:
public string FormatDate(string inputDate, string inputFormat, string outputFormat)
{
    System.DateTime date;
    if (System.DateTime.TryParseExact(inputDate, inputFormat, 
        System.Globalization.CultureInfo.InvariantCulture, 
        System.Globalization.DateTimeStyles.AssumeLocal, out date))
    {
        return date.ToString(outputFormat);
    }
    return "";
}
  • And the second one have:
public string FormatDate2(string inputDate, string inputFormat, string outputFormat)
{
    System.DateTime date;
    if (System.DateTime.TryParseExact(inputDate, inputFormat, 
        System.Globalization.CultureInfo.InvariantCulture, 
        System.Globalization.DateTimeStyles.AssumeLocal, out date))
    {
        return date.ToString(outputFormat);
    }
    return "";
}

The one difference is that the first one is call FormatDate and the second FormatDate2. I’m not joking, I saw this approach being used nearly 8 times (Func1, Func2, Func3, Func4, … Func8) in a map! Of course this solve the problem of the first approach but now you have another! The artifact size deployed will be larger, not critical, but if you can avoid is better. This will happen because in this case all method will be embed in the XSL file that will be deployed to your environment.

  <msxsl:script language="C#" implements-prefix="userCSharp"><![CDATA[ 
public string FormatDate(string inputDate, string inputFormat, string outputFormat) 
{
 System.DateTime date;
 if (System.DateTime.TryParseExact(inputDate, inputFormat, System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.AssumeLocal, out date))
 {
    return date.ToString(outputFormat);
 }
 return ""; 
} 
public string FormatDate2(string inputDate, string inputFormat, string outputFormat) 
{
 System.DateTime date;
 if (System.DateTime.TryParseExact(inputDate, inputFormat, System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.AssumeLocal, out date))
 {
    return date.ToString(outputFormat);
 }
 return ""; 
} 
]]></msxsl:script>

Another problem, even worse, of this approach is that, if you need to make some changes in the method… you need to do the same changes in every single functoid! If you want them to work in a coherent and transparent way.

So what is the best way to reuse Scripting Functoids with inline C#?

Easy! You need to remember that when you have two or more Custom Inline C# scripts with the same Method statement (name of the method + method parameters), the compile will only take in consideration the first one linked in the map, even if the others have a different code inside they are ignored by the compiler.

Reuse-Scripting-Functoid-Good-Way

When we use Custom Inline C# scripts the best way to implement reusability is to specify the body function in only one Scripting Functoid and the remaining ones specify only the function declaration, in this sample in the first Scripting Functoid will have the following code:

public string FormatDate(string inputDate, string inputFormat, string outputFormat)
{
    System.DateTime date;
    if (System.DateTime.TryParseExact(inputDate, inputFormat, 
        System.Globalization.CultureInfo.InvariantCulture, 
        System.Globalization.DateTimeStyles.AssumeLocal, out date))
    {
        return date.ToString(outputFormat);
    }
    return "";
}

And the second one only the following declaration:

public string FormatDate(string inputDate, string inputFormat, string outputFormat)

Easy, isn’t it?
Hope you enjoy it.

You can download this tool here:

BizTalk Mapper Tips and Tricks: How To Reuse Scripting Functoids with Inline C# (18,6 KB)
Microsoft | Code Gallery

As most of you already know, last month I presented a session at BizTalk Summit 2015 London event about “BizTalk Server tips and tricks for developers and admins” (You can check the video recording of my session here)

It was a lightweight session (30 minutes) about useful tips that we can use in our daily work, because it was a small session I didn’t had enough time to cover everything, so I end up creating a “Director’s cut…” session with additional tips that you can check in the presentation slides here.

Today, and because I already received some emails regarding this topic, I will address the “Database Lookup functoid

We can use the Database Lookup functoid to extract information from a database and store it as a Microsoft® ActiveX® Data Objects (ADO) recordset. This functoid requires four input parameters in the following order:

  • Parameter 1: A value for which to search in the specified database, table, and column.
  • Parameter 2: The full connection string for the database with a provider, machine name, database and authentication (an ActiveX Data Objects .NET (ADO.NET) connection string)
  • Parameter 3: The name of the table in the database in which to search.
  • Parameter 4: The name of the column in the table in which to search.

The functoid is actually quite simple to use, however, the main problem that developers face when they use it refers to the second parameter: the connection string. And why?

First: What is the correct value for the connection string?

I always find hard to remember the correct value for the connection string to be used inside this functoid.

Database-Lookup-Functoid-sample-configuration

The Easiest way to make sure we are using the correct connection string value and for not having to remember this by head is to create a simple Universal Data Link (.udl) File… set OLE DB provider connection parameters and test the connection to check if everything is correct.

To accomplish this we need to:

  • · Navigate to a folder in your system, can be in the desktop or preferably in a folder under your BizTalk visual studio solution, let’s call it “Resources”.
  • Create a text file and name it “ODBCConnectionTest.udl”
    • The name of the file is not important, the important part is the extension, it must be “.udl”
  • Double click the file you just created.

create-Universal-Data-Link-udl-File

  • On the Provider Tab, select the appropriate OLE DB provider for the type of data you want to access and then Next
    • In my case it is “Microsoft OLE DB Provider for SQL Server”

Universal-Data-Link-udl-Properties-Provider-tab

  • In the Connection tab, specify:
    • Where your data is located, typically the server and database name.
    • How to connect to it using an OLE DB provider: Use Windows NT integrated security or Use a specific user name and password
    • In this particular case, you need to provide the SQL Server, the database name we want to connect.

Universal-Data-Link-udl-Properties-Connection-tab

  • Then you can and should click on “Test Connection…” button to attempt a connection to the specified data source. If no connection is made, review the settings. Otherwise click “Ok”

Universal-Data-Link-udl-Properties-Test-Connnection

After data Open the “ODBCConnectionTest.udl” file in notepad and you will find the connection string value that you can copy and use it in the second parameter of the Database Lookup Functoid.

Open-Universal-Data-Link-udl-File-notepad

This will lead us to the second problem that you can face using this functoid: using the connection string statically inside the Database Lookup Functoid

Hard-coding the SQL connection strings might lead to maintenance overhead and serviceability issues.

Important considerations:

  • You shouldn’t Hard-coding this value directly in the functoid otherwise it will be a nightmare when you deploy this to a different environment.
  • You can and you should store this parameter in a different storage location (SSO, Registry or others) and get this value using a scripting Functoid or custom functoid which can then be linked to the Database Lookup Functoid, like the:
    • BTSNTSvc Config Get Functoid: This functoid allows you to get configuration parameters from BTSNTsvc.exe.config. If there is no section specified, the functoid reads from the AppSettings.
    • Windows Registry Config Get Functoid: This functoid allows you to get configuration parameters from Windows Registry.
    • SSO Config Get Functoid: This functoid allows you to get configuration parameters from SSO Database.
    • Rule Engine Config Get Functoid: This functoid allows you to obtain a definition value from a Vocabulary in the Business Rules Engine.

In my personal opinion I advice you to use SSO or BRE to store this configuration parameters. All of these custom functoids are available in the BizTalk Mapper Extensions UtilityPack.

TIP-10-Database-Lookup-functoid

You can found and download the Source Code on MSDN Code Gallery:

BizTalk Mapper: How to use Database Lookup Functoid
Microsoft | MSDN Code Gallery

Back again to one of my favorites topics: transformations. This is something that several people asked me for help in the past but I never found time to reply… until now.

A quite note: if you want to know more about BizTalk Maps transformation check out my free book available at BizTalk360 website: BizTalk Mapping Patterns & Best Practices

While migrating an older BizTalk Map transformation, from a custom schema to a SAP IDOC schema, from an entire custom XSLT file to BizTalk Mapper (don’t get me wrong, I only made this migration because the source schema changed so I had the need to change the entire XSLT code, otherwise I would have used the existing external file), I had the need to implement some part of the transformation in an Inline XSLT code (using a Scripting Functoid) and I only used inline XSLT because I was transforming a flat structure from the source schema to a recursive node in the target schema base on some conditions (otherwise I would have used the Table Lopping Functoid)

BizTalk-Mapper-inline-scriptin-functoid-with-prefix

<ns1:E2EDKA1003>
  <ns1:DATAHEADERCOLUMN_SEGNAM>E2EDKA1003</ns1:DATAHEADERCOLUMN_SEGNAM>
  <ns1:PARVW>LF</ns1:PARVW>
  <ns1:PARTN>0000100000</ns1:PARTN>
</ns1:E2EDKA1003>

<ns1:E2EDKA1003>
  <ns1:DATAHEADERCOLUMN_SEGNAM>E2EDKA1003</ns1:DATAHEADERCOLUMN_SEGNAM>
  <ns1:PARVW>ZC</ns1:PARVW>
  <ns1:PARTN>
    <xsl:choose>
      <xsl:when test="@Transportation='1'">
        <xsl:value-of select="'0000100001'" />
      </xsl:when>
      <xsl:otherwise>
        <xsl:value-of select="'0000100009'" />
      </xsl:otherwise>
    </xsl:choose>
  </ns1:PARTN>
</ns1:E2EDKA1003>

Note that I’m using the prefix “ns1” on the element because the schemas is expecting that, and it should work because both the schema and the transformation file (XSL) contains the namespace

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
     xmlns:msxsl="urn:schemas-microsoft-com:xslt" 
     …
     xmlns:s1="http://unicer.pt/OrderUnigest01.xsd"
     xmlns:ns1="http://Microsoft.LobServices.Sap/2007/03/Types/Idoc/3/ORDERS01//700"
     xmlns:userCSharp="http://schemas.microsoft.com/BizTalk/2003/userCSharp">

BizTalk-Mapper-xslt-prefix-namespace

You can validate that by:

However, when you either compile the map or validate it, we get the following error message:

Validate the map or compile it, I get the following error:

Exception Caught: ‘ns1’ is an undeclared prefix. Line …, position …

 

CAUSE

Well unfortunately BizTalk Mapper Editor doesn’t like prefixes in inline XSLT code, although they exist he is not able to understand them without additional information in the code, which makes our work a little more difficult or we need to use a different strategy in our XSLT code.

Luckily, we can easily fix this strange behavior of the editor.

SOLUTION

The cleanest way to solve this strange behavior is to use the <xsl:element> element or <xsl:attribute> element to create the records, elements and attributes required, instead of manually create them <ns1:E2EDKA1003></ns1:E2EDKA1003> without the <xsl:element> element:

<xsl:element name="ns1:E2EDKA1003">
  <xsl:element name="ns1:DATAHEADERCOLUMN_SEGNAM">E2EDKA1003</xsl:element>
  <xsl:element name="ns1:PARVW">LF</xsl:element>
  <xsl:element name="ns1:PARTN">0000100000</xsl:element>
</xsl:element>

<xsl:element name="ns1:E2EDKA1003">
  <xsl:element name="ns1:DATAHEADERCOLUMN_SEGNAM">E2EDKA1003</xsl:element>
  <xsl:element name="ns1:PARVW">ZC</xsl:element>
  <xsl:element name="ns1:PARTN">
    <xsl:choose>
      <xsl:when test="@Transportation='1'">
        <xsl:value-of select="'0000100001'" />
      </xsl:when>
      <xsl:otherwise>
        <xsl:value-of select="'0000100009'" />
      </xsl:otherwise>
    </xsl:choose>
  </xsl:element>
</xsl:element>

If you don’t want to use the <xsl:element> element or <xsl:attribute> element, then you need to declare the prefix namespace in the record:

<ns1:E2EDKA1003 xmlns:ns1="http://Microsoft.LobServices.Sap/2007/03/Types/Idoc/3/ORDERS01//700">
  <ns1:DATAHEADERCOLUMN_SEGNAM>E2EDKA1003</ns1:DATAHEADERCOLUMN_SEGNAM>
  <ns1:PARVW>LF</ns1:PARVW>
  <ns1:PARTN>0000100000</ns1:PARTN>
</ns1:E2EDKA1003>

<ns1:E2EDKA1003 xmlns:ns1="http://Microsoft.LobServices.Sap/2007/03/Types/Idoc/3/ORDERS01//700">
  <ns1:DATAHEADERCOLUMN_SEGNAM>E2EDKA1003</ns1:DATAHEADERCOLUMN_SEGNAM>
  <ns1:PARVW>ZC</ns1:PARVW>
  <ns1:PARTN>
    <xsl:choose>
      <xsl:when test="@Transportation='1'">
        <xsl:value-of select="'0000100001'" />
      </xsl:when>
      <xsl:otherwise>
        <xsl:value-of select="'0000100009'" />
      </xsl:otherwise>
    </xsl:choose>
  </ns1:PARTN>
</ns1:E2EDKA1003>

In any of the case that you decide to implement, if then you try to compile or validate the map, everything will work fine and the error will disappear.

Let’s have a look to a little of inside information that will be in my upcoming Book about maps: demystify and clarify some features about the Table Looping Functoid.

First of all this functoid requires at least 3 inputs and a maximum of 100, in which the official documentation states that:

  • Parameter 1: A link from a repeating node in the source schema. The number of instances of this structure that occur in a particular input instance message defines the number of times that the associated table looping grid is processed.
  • Parameter 2: A constant input parameter that defines the number of columns in the associated table looping grid.
  • Parameters 3 – 100: A link from a node in the source schema or from another functoid, such as a Value Extractor functoid, or a constant input parameter. The relative order of parameters 3 – 100 is unimportant.

I think, at least for those who know the minimum about functoid, the last parameter(s), from 3 to 100, does not present any doubts. They are the values that we want to map from the source to the destination. And the order is not important because we need to define the Table Looping Grid property were we will define the order of appearance of these values.

The Table Looping Grid is basically a configurable table containing an arbitrary number of rows, configurable while editing the table, and a number of columns specified by the second input parameter of the Table Looping functoid. Each table cell is configured by using a drop-down list where the entries in the drop-down list are the values from to third through the last input parameters of the Table Looping functoid. These input parameters consist of a combination of the links into the functoid and any constant input parameters that have been defined. If link input parameters have been given a Label property value, that value is shown in the drop-down lists; otherwise, the value of the Link Source property is shown (generally, the former is friendlier than the latter). Constant input parameters are shown according to their constant value.

Important note: However the number of rows present in the Table Looping Grid are not defined in any input parameter from the Table Looping Functoid and in under no circumstances they are dynamic. The number of rows are statically defined during developing in order to apply a specific transformation rule and resolve a particular transformation problem.

Although people use this functoid correctly, especially if we are using it with a repeating record in the source schema, we will see in more detail further on in this chapter, is went we are trying to use this Functoid to transform a flat structure to a recursive structure that we realize that many developers don’t properly understand the first input parameter of this Functoid and to be honest, neither the explanation present in the official documentation will explain it correctly.

Although the Table Looping Functoid states that the first input must be a scoping element linked from a repeating group, that is not really true, in fact for me this description is complete false!

  • A link from a repeating node in the source schema – most common used but not entire true, you can also make use of a constant value, a simple node or an element.
  • The number of instances of this structure that occur in a particular input instance message defines the number of times that the associated table looping grid is processed. – I understand and in a way yes, but sometimes developers understand that this as the number of rows that they have to define in the table Looping Grid, with is not true, or that this input requires a number and in fact this is also not true.

So to be more clear, for me the best description of this parameter is:

  • Parameter 1: the first input parameter defines the action scope of the Table Looping Functoid and it can be defined by a link from a source tree node, repeating record, simple records or even elements, or by a constant value. This means:
    • If the scope is defined by a repeating record or repeating element, the Table Looping functoid will be created/execute in each occurrence of the record or element (in each iteration over the node). We can define this as multiple scope action.
    • If the scope is defined by a simple element or record (only occurs one time), than the Table Looping will be executed only one time. We can define this as simple scope action.
    • If the scope is defined by a simple constant value, regardless if it is an integer, alpha-numeric or string, than the Table Looping will be executed only one time. Again this is a simple scope action.

Table-Looping-Functoid-Scope-defined-by-a-simple-constant-value

Both this options are valid! More details will be found in my upcoming BizTalk Mapping Patterns and Best Practices free eBook.

Basically there are two properties inside the schema element decides whether an element can be absent from the document: Min Occurs and Nillable.

If Min Occurs property is set 0 then that element can be absent from the XML message but if it is set to 1 it has to be present though its value can be empty. This is useful to reduce the size of the document if only not all the elements are mandatory to end systems.

In other hand, if the Nillable property of the element is set to true, this will indicate that the value of an element in the document may be null. This NULL values will be expressed with xsi:nil = true attribute in the element, ex:

<IntExist xsi:nil="true" />

An element with the attribute xsi:nil = true explicitly means that the value is unavailable or unknown at that moment and sometimes the end system explicitly requires to be notified that the value of the element is NULL so that they can take appropriate action.

In this sample scenario we will have a 2 mandatory elements that can be nillable that we need to map to the destination schema. In this scenario all the destination elements are also mandatory and we need to fill them with a valid value or specify the nillable property as true: so if the element exist we need to map the correct source value otherwise we need to set the destination element as nillable.

The first element “DateExist” is a mandatory element that can be null. If null we need to set a null value in the destination element also as null, otherwise we need to map the source value. To accomplish that we need to:

  • Drag one IsNil Functoid from the Toolbox window onto the Grid.
    • Drag a link from the “NillValue” field element in the source schema to the IsNill Functoid
  • Drag one Nil Value Functoid from the Toolbox window onto the Grid.
  • Drag one Logical NOT Functoid from the Toolbox window onto the Grid.
  • Drag one Value Mapping Functoid from the Toolbox window onto the Grid.
  • To create a rule for mapping the value if the element is null
    • Drag a link from the IsNill Functoid to the Nil Value Functoid
    • Drag a link from the Nil Value Functoid to the “NillValueOutput” field element in the destination schema
  • Otherwise, To create a rule for mapping the value if the element different of null
    • Drag a link from the IsNill Functoid to the Logical NOT Functoid
    • Drag a link from the Logical NOT Functoid to the Value Mapping Functoid Functoid
    • Drag a link from the “NillValue” field element in the source schema to the Value Mapping Functoid
    • Drag a link from the Value Mapping Functoid to the “NillValueOutput” field element in the destination schema

Do the exact same logic for the second element present in the source schema.

BizTalk-Mapper-Working-With-Nillable-Values

Sometimes the maps are misunderstood and notorious for producing a lot of unnecessary code that may cause a in some cases lack of performance. So the question that we can and should ask is whether this is the best solution or not to address this type of operations. To respond this question we should also inspect the generated code produce by the BizTalk Mapper:

<xsl:variable name="var:v1" select="string(NillValue/@xsi:nil) = 'true'" />
    <xsl:variable name="var:v2" select="userCSharp:LogicalNot(string($var:v1))" />
    <xsl:variable name="var:v4" select="string(AnotherNilValue/@xsi:nil) = 'true'" />
    <xsl:variable name="var:v5" select="userCSharp:LogicalNot(string($var:v4))" />
    <ns0:OutputSchema>
      <xsl:if test="string($var:v1)='true'">
        <NillValueOutput>
          <xsl:attribute name="xsi:nil">
            <xsl:value-of select="'true'" />
          </xsl:attribute>
        </NillValueOutput>
      </xsl:if>
      <xsl:if test="string($var:v2)='true'">
        <xsl:variable name="var:v3" select="NillValue/text()" />
        <NillValueOutput>
          <xsl:value-of select="$var:v3" />
        </NillValueOutput>
      </xsl:if>
      <xsl:if test="string($var:v4)='true'">
        <AnotherNilValueOutput>
          <xsl:attribute name="xsi:nil">
            <xsl:value-of select="'true'" />
          </xsl:attribute>
        </AnotherNilValueOutput>
      </xsl:if>
      <xsl:if test="string($var:v5)='true'">
        <xsl:variable name="var:v6" select="AnotherNilValue/text()" />
        <AnotherNilValueOutput>
          <xsl:value-of select="$var:v6" />
        </AnotherNilValueOutput>
      </xsl:if>
    </ns0:OutputSchema>
  </xsl:template>

In fact is a pretty decent XSLT code but the reality is that it can be better, we don’t need to use any support variables and we can remove one if condition by replacing the xsl:if condition for one xsl:choose condition.

This is a very simple approach, easy to implement and readable that you should use even in small or large messages (transformations) but only if you have to deal with a small number of nillable element.

However applying this approach in transformation that will need to deal with a large number of nillable elements, can lead to two problems:

  • A lot of unnecessary XSLT code that can in fact and of course always depending in the size of the message can lead to some lack of performance
  • A lot of functoid shapes (4 Functoids) and links (7 links) for each element that can lead to lack of visual Readability

So can we improve this solution for transformations that needs to deal with a large number of nillable elements?

Well that’s the problem, there isn’t a simple solution for that. At the first look you may think that’s easy, just copy the XSLT code inside to a Scripting Functoid and optimize the XSLT code.

However by doing that you will receive an error:

error btm1050: XSL transform error: Unable to write output instance to the following <file:///C:\…\MapNillValuesWithCustomXSLT_output.xml>. Prefix ‘xsi’ is not defined.

The problem is that the nil attribute is defined in the XML Schema instance namespace, http://www.w3.org/2001/XMLSchema-instance (commonly associated with the prefix xsi) and this namespace is not declared by default in the XSL code generated by the BizTalk Mapper.

This namespace is automatically declare only if you use the Nil Functoids in the map.

So the normal solution here is… to bypass the BizTalk Mapper and generate an external XSLT code and add it to the map by specifying the Custom XSLT Path by:

  • Open the map
  • Click the grid zone and on the properties window there will be a “Custom XSLT Path” property.  Click the ellipses and navigate to the file containing the XSLT.

BizTalk-Mapper-Working-With-Nillable-Values-External-XSLT

You then can use a similar code to check and map the elements:

<xsl:choose>
  <xsl:when test="NillValue/@xsi:nil">
    <NillValueOutput>
      <xsl:attribute name="xsi:nil">
        <xsl:value-of select="'true'" />
      </xsl:attribute>
    </NillValueOutput>
  </xsl:when>
  <xsl:otherwise>
    <NillValueOutput>
      <xsl:value-of select="NillValue/text()" />
    </NillValueOutput>
  </xsl:otherwise>
</xsl:choose>

However applying this approach we have a major problem for me:

  • We lose all the BizTalk Mapper functionalities.
Workaround

Well, at least that I know, unfortunately there is no simple way to declared the xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance namespace to the stylesheet of the map.

However we can apply one small workaround, is not perfect but in most of the case it will solve my problems:

  • For only one of the nillable elements we need to use the Nil Functoids explained in the beginning of this post.
    • This will declare automatically the name xsi namespace for us.
  • In the rest of the elements we now can use Scripting Functoids with the optimized XSLT code described above

BizTalk-Mapper-Working-With-Nillable-Values-XSLT

I’m still working to find a better way but until then

You can download the source code from:

BizTalk Mapper: Working With Nillable Values (xsi:nil=”true”) (92.7 KB)
Microsoft | Code Gallery

I publicly announce for the first time that I was working in an eBook about BizTalk Mapping Patterns and Best Practices, and that I was published for free, this March during BizTalk Summit 2014 London…

BizTalk-Mapping-Patterns-and-Best-Practices-eBook-soon

… At the time I said it would be published in mid-April or May but since then I’ve been slightly quiet, a few tweets and that’s it!

So how’s the book and when will it be published? This is the question that some people have asked me and that you probably are asking too, at least if you attended the event in London

Well I have some good news Smile… the book will be a reality and will definitely be for free. I’ve already made ​​the commitment to the BizTalk community and I’ll fulfill it. And the first chapters had already been sent to my awesome team of reviewers… I’m waiting for their feedback Smile!

However there are still many things to finish, the structure of the book may still suffer some changes (depending on the reviewers feedback), although it is unlikely that I make some radical changes in its current structure. And I still need to finish the main chapter “BizTalk Mapper Patterns” and one additional final chapter.

You also need to remember that this eBook will be for free and that all those involved are spending their free time to make this happen! Personally, the last few months have been difficult for me to find some additional free time to engage and finish this project… and I also cannot ask and demand nothing more from my reviewers, we all are busy people. I prefer to take a little more time and provide some quality work (I hope).

State of art and key point to take:

  • The book will be a reality and will definitely be for free
  • 12 patterns addressed in the eBook, currently 233 pages…
  • Estimated that 85% of the work is done
  • All demos are finished, finally! (is hard to invent some good and practical scenarios)
  • I promise that I will push my reviewer to send me the feedback (but they are not the blocking point)
  • I’m currently have some free time to finish the eBook!!! SmileSmile

I hope that soon I will be sending all the chapter to the reviewers and then depending on the feedback: improve it, fix it and finally publish it… so I ask you to have a little more patience.

Unlike schemas that you will be able to directly take your BizTalk Server schemas and use them in you WABS solution, BizTalk Maps needs to be migrated because despite the Transform Designer (mapping tool used for WABS) and the BizTalk Mapper Designer looks mostly the same in terms of layout and user experience, they are quite different and the underlying map format is different. Basically when we are using BizTalk Mapper Designer the transformation are created using XSLT, however with Transform Designer the transformation are created using Extensible Application Markup Language (XAML).

Also the available functoids, that are now called Operations, are different as well. Not all the functoids available in BizTalk Server are available as map operations in Azure BizTalk Services and there are new Operations that doesn’t exist in BizTalk Server like: several List Operations, DateTimeReformat, GetContextProperty and so on. And despise XSLT and custom C# are supported the concept of “Scripting” map operation in Azure BizTalk Services Transforms are limited comparing to BizTalk Service for example:

  • You cannot use custom inline XSLT, you only can custom XSLT code directly to override WABS map, similar to the “Custom XSLT Path” map grid properties in BizTalk Server.
  • You can only use CSharp Scripting inside WABS maps.

Because of all this reasons you cannot directly use a BizTalk map in Azure BizTalk Services.

Basically you have two options that you can use to migrate BizTalk maps:

  • By using custom XSLT code gathered from your BizTalk Maps (…the easy way)
  • Or by using the BizTalk Map Migration Tool provided by Microsoft.

Using BizTalk Map Migration Tool

This is a command line tool, provided by Microsoft that takes a BTM File as input and that will try to produces the new mapper (.trfm) file as an output.

I’m saying “that will try” because there are many limitations present like:

  • Certain functoids may not get converted by this tool, for example some Scripting and Custom functoids from BTM way cannot be converted. In this case they are converted to arithmetic expression functoid in the TRFM map with the expression value as empty.
  • Links to functoids, which cannot be converted successfully, are denoted as empty inputs in the new TRFM functoids.
  • Certain features that are not supported, yet, in TRFM, will not be converted properly. For example, links from xs:extension node(s) are not converted to TRFM
  • Certain conditional mapping are not converted as expected as these features are not yet implemented in the Transforms
  • And so on.

At the end this tool will generate a conversion process report (Report.txt) listing out the failed conversions.

Also in terms of Organizing and Documenting Maps there are two drawbacks presents:

  • Labels and comments from the BTM functoids are not mapped onto the TRFM functoids
  • The multiple pages from BTM are translated to a single page in the TRFM map to ensure all operations within a scope are within a page.

The BizTalk Map Migration Tool is available on Codeplex at BTM Migration Tool, but before you use this tool you need to know that:

  • This tool expects that the machine on which it’ll run would have both products installed: BizTalk 2010 or above and Windows Azure Biztalk Services.
  • And the schemas should be present as expected by the BTM map (in the relative folders).

The BTM Migration Tool is a command line tool that uses the following parameters:

  • A BizTalk map (.btm)
  • Optional. Resulting BizTalk Service project Transform (.trfm)

BTM-Migration-Tool-cmd

Let’s look at one very easy example and analyze the end result. In this scenario we have two different records, Client and Employee, and we want to map all the man’s clients and employees to Person (Sex == M). This is look of the BizTalk Server maps:

BizTalk-Server-map-sample-to-WABS

When we apply the BTM Migration Tool it will generate this nice WABS map for us:

BizTalk-Services-map-sample-from-BTS

And the good news in this case, if we try the map it will successfully work and it will produce the tight output! (Yes I looked for a good sample). However is this the perfect output?

So let’s try change this to get only the female sex present in the input message. It seems an extremely easy change to make… we just need to go to the Logical Operation present in each MapEach Loop Operation and change the condition

BizTalk-Services-Logical-Operation-WABS-Condition

However surprisingly the output instance after this change is empty

WABS-map-empty-result

So why this happens? In this case, after looking to all the shapes, I found out that the MapEach Loop operation also as the same condition that is present in the Logical Operation that we need to change, as you can see in the picture bellow:

BizTalk-Services-MapEach-Loop-Operation-WABS-Condition

Can this map be improved?

The aim here is not to explain the functionalities of the new map and how the new operations works, however if you look at the map generated by the compiler, at least in this case, you can easily identify that we can simplify the map to be more easy to read and to maintain, by:

  • Maintaining the condition in the MapEach Loop operation
  • Suppress all the Logical and Conditional Assignment Operations that are inside the MapEach Loop operation
  • And use direct links to map the Name and Sex

BizTalk-Services-map-improved-from-BTS

BizTalk-Services-MapEach-Loop-Operation-WABS-Condition-2

So as a final note, even if the migrator tool did a decent job, you should always try to identify some bottlenecks, problems or overlaps and simplify it

Using Custom XSLT Code

Migrating maps from BizTalk Server solutions to Azure BizTalk Services could be one of the more complex artifacts to migrate (depending on map complexity)… or not!

The good news is that, you don’t necessary have to remake or migrate our BizTalk map to WABS map. As I mention earlier WABS maps support custom XSLT code, you can use a custom external XSLT file, similar to the “Custom XSLT Path” map grid properties in BizTalk Server to override WABS map and use custom XSLT to make the entire map transformation.

I try to avoid using Custom XSLT files for solving a mapping problem, for many reasons, however there are always exception. I do recommend using custom XSLT files:

  • If you are dealing with huge message and High Performance is one of the primary requirements (and even then this maybe can be avoid)
  • Or if you already have a custom XSLT file provided by another team, or from another system that you can re-use in your map. In this case don’t try to reinvent the wheel!

And because we are migrating a BizTalk Server map we already have the XSLT, so for me there is no point spending a few and in some case many hours trying to remake the same transformation in a different editor.

To extend your WABS maps to use custom XSLT code you must:

  • Go to the grid properties (or also call Transform design surface properties) in the Transform Designer, where you can see that there is a XSLT property in which we can extend the map to use custom XSLT code

BizTalk-Services-WABS-map-Transform-design-surface

Note: This time Microsoft did a nice work and if you notice in the grid page there is message that will help you.

Transform-design-surface-message

  • In Properties, click XSLT, and click the ellipses (…). A Configure XSLT dialog window is opened.

BizTalk-Services-configure-XSLT

  • To import an existing XSLT file, click Import XSLT. Browse to the .xslt or .xsl file and click Open. The imported XSLT populates the Configure XSLT dialog window. OR, enter the XSLT syntax in the Specify the XSLT below … window.

BizTalk-Services-configure-XSLT-import

  • Optional. To import an existing XML extension file, click Import EXTXML. Browse to the XML file and click Open. The imported XML populates the Specify the extension objects … dialog window. OR, enter the XML syntax in the Specify the extension objects … window.
  • Optional. Click Use XslCompiledTransform for better performance to transform XML data by compiling XSLT style sheets and executing XSLT Transforms. When the style sheet is compiled, it can be cached and reused. Otherwise, the XslTransform class is used; which is best when executing a Transform on one occurrence.
  • Click OK.

When XSLT is used, the Transform design surface is grayed out. XSLT can be specified for every Transform file (.trfm).

BizTalk-Services-WABS-map-Transform-design-surface-XSLT

How do I generate XSL output from the BizTalk Mapper Designer?

To generate a XSLT file that represent you BizTalk Map you just need to:

  • Open the BizTalk map in Visual studio
  • Right click on the map and select “Validate Map” option, this operation will generate the XSLT and you can see the path of generated XSLT in output window.

You can use this XSLT, without change it in most of the cases, to add in your WABS Maps.

Code Samples

All of this sample can be found and downloaded in Microsoft Code Gallery: