BizTalk Server: Transform text files (Flat Files) into XML – Conclusion (Part 4)

Posted: August 21, 2012 in BizTalk
Tags: , , , ,
Configuring the BizTalk application

This is the final step of this process. In order to properly test the solution we have been developing in BizTalk Server, we need to configure the application that was created in the publication.

For this, we will access the BizTalk Server Administration console to create and configure the following artifacts:

  • One receive port and location to process the input text files;
  • One send port to save the processed files in XML format..

Open the BizTalk Server Administration console under “Start –> Programs –> Microsoft BizTalk Server 2010”. On the left tree menu, maximize “BizTalk Server Administration –> BizTalk Group (…) –> Applications”, and look for the application called “TransfFlatFilesEmXML Demo” and maximize also this application.

maximize-BizTalk-Application

  • Press the right button on top of “Receive Port” menu and select “New –> One-Way Receive Port …” option;
  • A new window appears allowing us to define the port properties:
    • In the “General” tab, set the name of the receive port: “ReceiveTXT”.

New-receive-port

    • In the “Receive Locations” tab, select “New” and set the name to your receive location: “ReceiveTXT_to_XML”; in the property “Type” select the option FILE; and in the property “Receive pipeline” select from the drop down box the pipeline we created earlier: “ReceivePipelineCustomPessoas”;

New-receive-location

    • On the same tab select the “Configure” button. In the “FILE Transport Properties” window, set “Receive Folder” property, with the folder you created earlier “<solution>\PORTS\IN”. On property “File Mask” put the following value: “*. txt”, setting that will be processed only files with the extension. “txt”. Finally select the “Ok” button.

File-Adapter-Properties

    • To finish the process of creating the receive port press “Ok”.

The above steps mentioned, were needed to create one Receive Port and its Receive Location. Now we will create a Send Port to save messages on the file system, after they are processed by BizTalk:

  • Press the right button on top of “Send Ports” menu and select “New –> Static One-way Send Port…”
  • A new window appears allowing us to define the port properties:
    • In the “General” tab, set the name of the send port: ” SaveXMLFile “
    • In the property “Type” select the option FILE;
    • In the property “Send Pipeline” set the pipeline to “XMLTransmit”. This is a native pipeline that is used for processing XML messages.

New-send-port

    • On the same tab select the “Configure” button. In the “FILE Transport Properties” window, set the “Destination Folder” property, with the folder you created earlier “<solution>\PORTS\OUT”. On property “File Mask” put the following value: “%MessageID%.xml”, this is special tag (BizTalk Macro) that will set the name of each file written on the file system with the unique identifier of the message. Finally select the “Ok” button to return to previous window.

Send-port-File-Adapter-Properties

    • In order for the Send Port subscribes to all files, we need to set a filter on the tab “Filters” with the following expression: “BTS.ReceivePortName == ReceiveTXT”.
      • Note: This setting will force each message in the MessageBox which has originates from the Receive Port: ReceiveTXT, be routed and processed by the Send Port we are finished creating.

New-send-port-filter

    • To finish the process of creating the send port press “Ok”.

Finally, we need only to start our application. To do that press right button on top of the application name “TransfFlatFilesEmXML Demo” and select “Start …”.

start-BizTalk-application

Testing the solution

To test the BizTalk application we only need to copy a text file to the directory that is configured on the receive location: “<solution>\PORTS\IN”. The result should appear in the form of an XML document in the folder configured in the send port: “<solution>\PORTS\OUT”.

Note: The file disappears automatically from the input folder, so you should avoid doing drag & drop (move) with the test files at the risk of not being able to use it again.

Conclusion

As presented in this article, BizTalk Server helps us to solve many of the existing problems concerning message transformation: semantic and syntax Transformations, only with “out of the box” features within the product, in few steps and most of the time without developing any code.

The fact of all the definitions of the rules of parsing (transformation rules) are embedded in the form of annotations in XML Schema (XSD), such as the delimiter symbols, or the element size in a positional file, simplifies the reuse of all these schemes in different parts of the process. At any point, the document can be translated back into flat-file because the definition is declarative and symmetric, in contrast to what might happen with a “normal” programming language like C#, where we would have to create two components to translate text to XML and vice versa.

The model used by BizTalk schemas, , centralizes all the message definitions in a single artifact, simplifying the maintenance and evolution of the specification and therefore simplifying future reuse and dependencies management. When we set up this type of project, is exactly this kind of guarantees than keeps the systems operating properly for many years, and at the same time, they make life easier to those who one day have to correct and/or develop any of these processes

Related links

Source code

The solution of this example, as well all the code, is available on MSDN Code Gallery:
BizTalk Server – Transform text files (Flat Files) into XML (64.5 KB)
MSDN Code Gallery

About these ads
Comments
  1. Mauro says:

    Guide very detailed and precise.
    One question: if i need to work with file.csv out of Biztalk orchestration, the conversion is better to do to inside Biztalk or is better to convert outside?
    I think i leave all in Biztalk….Wrong?

    • Hi Mauro,

      I’m not sure I quite understand your question…

      However be noted that within BizTalk orchestrations you are not working with flat file format (file.csv) but with the representation of that file in XML format, the conversion of these formats should always be carried in pipelines.

      If you are you asking if these conversions must be made in BizTalk or in external systems, it depends on the architecture of the systems, and what is the cost that such operation represent or even if it is possible to change the format which these systems can communicate.

      Clear that if we can remove unnecessary transformations we will be improving the solution performance, but without reason in contrary, I will perform all these transformations in BizTalk.

  2. Mauro says:

    Hi Sandro and sorry for the ambiguity of my question! :)

    I know that Biztalk works only with xml files and my doubt is about this thing:
    the transformation is due to another external system that receive only csv files (that must be the output of my BT application).
    I think that this operation cannot be improved the performance, so following your idea, i try to perform these transformations in Biztalk…How can i do this?
    My idea was to work with maps, but how?
    Otherwise, can i personalize a receive pipeline that makes this transformations?

    Many Thanks

    • Hi Mauro,

      What you must do is:
      1- get a complete sample with the correct syntax of the CSV file that has to be consumed by the external system
      2- and use this file to create the Flat File Schema (as described in Part 3 of this guide) and of course in the end you should test that the Flat File Schema generated can correctly “translated” the CSV in its XML representation
      3- Create a Map that transform you input schema into the flat file schema generated.
      4- And use a send pipeline to “translate” the XML message to CSV format.

      and of course this also depends on the type of structure of CSV file (structure format), sometimes is not possible to accomplish this with this strategy, usually with complex flat files, we must develop a pipeline component for dealing with this transformation. But I believe that the majority of the scenarios can be implemented using the flat file schema and a pipeline without any custom pipeline component.

  3. samoth80 says:

    Hi Sandro,

    I have another question: in a BizTalk project (for example called General) i want to realize a sort of General Catalog within all the records that frequently i can use in my process (made as Data Structure Type), so that when i add other many projects (PROJECT A, PROJECT B…) i can recall the same records made in General without generate again them. This is possible or is completely wrong? And if yes, how can i recall my Data Structure Type?
    Sorry for my English, but my idea is a little bit complicated to explain :).

    Thanks

    • Hi,

      If I understand the question well, you want to create a “general project” with all common Data Structure Type like Phone (CountryCode, AreaCode, Number and Extension) and then reuse this structure in many projects.

      This is possible. Example scenario:
      Inside Visual Studio you need to:
      1. You need to create a “General Structure Project” with these schemas
      2. In your Visual Studio BizTalk Project A you need to
      2.1. Import the DLL of this “General Structure Project”.
      2.2. In your schema, select imports -> add XSD import and from the BizTalk Type picker select References->you DLL General Structure Project ->you schema with common Data Structure Types
      2.3. Then all the Data structure type which exist in your schema will be available
      3. Publish “General Structure Project” to a common BizTalk Application, let’s call “BizTalk Common Application”
      4. And Publish Project A, let’s call “BizTalk Application A”

      In BizTalk Administration Console you need to:
      1. Go to “BizTalk Application A” properties and add the reference to “General Common Application”

  4. Jordan says:

    Sandro,

    Do you think you can do a tutorial or give me guidance on using the Flat File Wizard to model complex data that meets this type of requirement.

    1. The tag identifier for each data line is not at the position 0 but 8.(Off Set Tag)
    2. Each type of data line can appear arbitrary number of time and in arbitrary order.(Random Ordering of Data)

  5. Onur says:

    Hi,

    I drag and drop my txt file to IN folder but OUT folder always be empty.

    Do you help me?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s