Following one of my last speaking engagement in the Porto.Data Community Meeting about Flat Files: How to process Flat Files documents (TXT, CSV …) in BizTalk Server where I address some of the following topics:

  • How to process Flat Files documents (TXT, CSV …) in BizTalk Server.
  • What types of flat files are supported?
  • How is the process of transforming text files (also called Flat Files) into XML documents (Syntax Transformations)
  • Where does it happen and which components are needed
  • Or how can I perform a flat file validation?
  • And so on

I decided to return writing about this topic in my blog, a topic that I already wrote in the past describing and explaining almost all of the topics mention above:

But this time transforming my previous session in a sequence of blog posts where I will address and explain how to solve some of the typical or common flat files structures and scenarios.

The simplest ones:

  • Text Files Delimited by symbols
    • For example:
Vânia;Braziela;1981;Vila Nova de Gaia;4400
    • Where we have: Name,Surname,Birthyear,Address,ZipCode
  • Positional Text Files
    • For example:
0120200300 01 PT        Sandro      Pereira         Rua Central, 133      Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        José        Silva           Crestuna              Crestuma    Vila Nova de Gaia  4415 Crestuma
0120200300 01 PT        Rui         Barbosa         Porto                 Porto       Porto              4400
0120200300 01 PT        Miguel      Silva                                 Porto       Porto              4415 Crestuma
    • Where we have:

0120|20|03|0|0 |01 |PT |Sandro |Pereira |Rua Central, 133 |Crestuma |Vila Nova de Gaia |4415 Crestuma

And evolve to more challenger scenarios like:

  • Handling Record Tag Identifiers, were we will treated each Tag Identifier differently
    • For example:
USERSandro Pereira,1978-04-04, Crestuma
USERCelso Pereira,1978-04-04, Crestuma
ITEM872-AA|Super Bock|Green
ITEM871-AA|Super Bock|Black
East Coast Facility
ERROR102|0|High|Sprocket query fails.|1999-05-31T13:20:00.000-05:00
ERROR16502|2|Low|Time threshold exceeded.|1999-05-31T13:20:00.000-05:00
    • Where we have:
      • Location
      • ERROR
        • ID|Type|Priority|Description|DateTime
      • BatchID
  • Or simple How can I remove a simple header from a Text File
    • For example
PA1,IN1900011,K1-N-A1,11/9/2015,BA511,15/9/2015,GTW,9871119,K,LTSGT 11A,12,PD806-D0,AGT;AKT;LOOP
PA1,IN1900012,K1-N-A1,11/9/2015,BA511,15/9/2015,GTW,9871119,K,LTSGT 11A,12,PM806-D0.0J,AGT;AKT;FLAT
    • Where the structure of the message is described in the header (first line) – This scenario for example is very common in CSV files.

Until we reach and address advance topic like:

  • Debatching Flat Files messages
  • Perform Deep Validation in flat files processes
  • Create generic Receive and Send Pipelines and how to configures this pipelines through BizTalk Administration Console
  • And so on

And hopefully teaching you something new (trick or tip) about Flat Files or at least trying to demystify some issues related with Flat Files.

Most of this things will be solved by explaining you how to create, for each scenario, flat file schema from flat file document instances using the BizTalk Flat File Schema Wizard.

Others how can we tuning some properties afterwards so that we can have the desired output.

Let’s the fun begin.

Last year I presented for the first time in a SQLSaturday (SQLSaturday #341 Porto – Porto-Wine edition) at the time the attendees chose a session about BizTalk Services – “Introduction to Microsoft Azure BizTalk Services”.


I’m proud to announce that, for the second consecutive year, I will present a session at SQLSaturday Porto – I’m extremely happy because it’s my hometown!

On Oct 3th SQLSaturday#429 Porto will be held at ISEP – Instituto Superior de Engenharia do Porto and my session this year will be about Logic & API apps: “Introduction to Microsoft Azure App Service (Logic and API Apps)

This will be a lighting talk session (15 to 20 minutes) that is intended to be an introduction to the new Azure Integration features: Logic Apps and API Apps. How can we get start using these new features? How can you easily use APIs that connect on-premises using hybrid connections or/and SaaS connectors from the gallery. And finally we will learn how you can use Azure Logic Apps to automate business processes without using code.

Note: You will not see my session in the scheduler, because it is included in the block “To be Announced”, between 03:50 PM – 05:00 PM, that will include 3 or 4 lighting talks.

I also will recommend you to attend my coworker, Rui Romano, session about Power BI: “Power BI for Developers” – “Are you a developer? Want to learn what PowerBI has for you? Then come to my session where you will learn how to enhance your applications with PowerBI experiences….”

And of course the DevScope sponsor session because we will give an awesome price to the attendees Smile

photo taken from DevScope blog.

Check the full event scheduler here: SQLSaturday#429 Porto Event Scheduler

Admittance to this event is free, all costs are covered by sponsorships. But please register quickly here: because seating is limited, and let friends and colleagues know about the event.

This post is for the BizTalk Server and other Portuguese Community, will be held on September 30, 2015 between 18:45 – 21:30 the XI Porto.Data Community meeting at the Science and Technology Park of University of Porto (UPTEC) in Oporto.

For me it is a pleasure to return, for the second time this year, to this community. This time I chose a topic, which at first sight it seems very simple and basic, but it hides some techniques in which even the most experienced ones are unaware of, forget or fail.

I will be speaking about: “How to process Flat Files documents (TXT, CSV …) in BizTalk Server

(Initially I was going to call this session: "Teach me something new about Flat Files")

Abstract: Despite over the year’s new protocols, formats or patterns emerged like Web Services, WCF RESTful services, XML, JSON, among others. The use of text files (Flat Files ) as CSV (Comma Separated Values) or TXT, one of the oldest common patterns for exchanging messages, still remains today one of the most used standards in systems integration and / or communication with business partners.

While tools like Excel can help us interpret such files, this type of process is always iterative and requires few user tips so that software can determine where is need to separate the fields/columns as well the data type of each field. But for a system integration (Enterprise Application Integration) like BizTalk Server, you must reduce any ambiguity, so that these kind of operations can be performed thousands of times with confidence and without having recourse to a manual operator.

In this session we will first address: How we can easily implement a robust File Transfer integration in BizTalk Server (using Content-Based Routing in BizTalk with retries, backup channel and so on).

And second: How to process Flat Files documents (TXT, CSV …) in BizTalk Server. Addresing what types of flat files are supported? How is the process of transforming text files (also called Flat Files) into XML documents (Syntax Transformations) – where does it happen and which components are needed. How can I perform a flat file validation?

XI Evento da Comunidade Porto.Data

A comunidade Porto.Data tem como objectivo a partilha de experiências e conhecimento entre os profissionais no Porto.

Para mim é um prazer voltar, pela segunda vez este ano, a esta comunidade. Desta vez eu escolhi um tema, que à primeira vista parece muito simples e básico, mas que esconde algumas técnicas ou segredos em que até mesmo os mais experientes desconhecem, se esquecem ou falham.

Desta vez vou vos falar sobre: "Como processar documentos Flat Files (TXT, CSV, …) em BizTalk Server"


Apesar de ao longo dos anos surgirem novos protocolos, formatos ou padrões: Web Services, WCF, RESTful services, XML, JSON entre outros. No entanto, um dos padrões mais antigos e comuns para a troca de mensagens, e que continua ainda a ser hoje em dia um dos padrões mais usados na integração de sistemas e/ou na comunicação com parceiros de negócio, é a utilização de arquivos texto (Flat Files) como: CSV (Comma Separated Values) ou TXT.

Embora ferramentas como Excel nos ajudem a interpretar um ficheiro destes (CSV), o processo é sempre interactivo e requer algumas dicas do utilizador para que o software consiga determinar onde separar os campos/colunas, bem como o tipo de dados de cada campo. Ora para um sistema de integração (Enterprise Application Integration) como o BizTalk Server, é preciso reduzir todas as ambiguidades, por forma a estas operações poderem ser efectuadas milhares de vezes com confiança e sem que seja necessário recorrer a um operador manual.

Nesta sessão vamos abordar, primeiro, como podemos facilmente efectuar uma integração do género File Transfer robusto em BizTalk Server. E em segundo, como é que o BizTalk Server processa Flat Files (CSV, TXT, …): Que tipos de flat files são suportados? Como é o processo de transformação de arquivos de texto (também chamados de Flat Files) em documentos XML (Transformação de sintaxe)? e onde essas transformações ocorrem, bem como na validação da informação neles contida.


O evento irá ocorrer no próximo dia 30 de Setembro no auditório da UPTEC, pelas 18:45 no Porto. As inscrições não tem custos mas é obrigatório que reserve o seu lugar aqui uma vez que os lugares são limitados.

Confira a agenda abaixo:

18:45 – Abertura e recepção.

18:50 – Community News.

19:00 – "Columnstore Indexes Revealed – Uncut & Unedited" – Niko Neugebauer MVP –

20:10 – Coffee break.

20:30 – "Como processar documentos Flat Files (TXT, CSV, …) em BizTalk Server" – Sandro Pereira MVP – DevScope

21:15 – Fim do evento.

21:20 – Sorteio de prémios.

21:30 – Jantar livre.

Reserve aqui o seu lugar! Estamos a sua espera.

Last Monday I presented my second session in the Integration Monday series (see my last post) and, as usually, I normally publish a post on my blog before the event – a way to help the organizers to advertise the event – or sometimes after the event sharing the resources… but I just now realized that I never created a post in my blog about my first session in the Integration Monday series: BizTalk Server Tips & Tricks for Developers and Admins (Deep Dive)

A session similar to the one I did in BizTalk Summit 2015 London event (also available online here) but a more open session addressing more tips and going into more detail – because I had more time, almost 82 minutes comparing to the 30 minutes I had in London Smile – and according to the Integration Monday organizers it has been a quite popular (or viewed) session.


So it is time to solve this "my big fault" and share with you the resources of this session – a session with a simple topic that I really love – I think that this (the topic) is the session that gave me more pleasure in making.


Session: BizTalk Server Tips & Tricks for Developers and Admins (Deep Dive)

Abstract: It’s critical to use good tools and techniques to produce working solutions as quickly as possible and at the same time, given the increase the requirements and number of applications organizations develop today. But at the same time, it’s also critical to maintain the health of the entire platform. In this session I’ll address and share some useful BizTalk Server Tips and Tricks (and Workarounds) both for developers and administrators that we can use in our daily work. And by doing so I’m hoping to simplify a little and/or automate some of the repeating tasks that we normally do and by sharing some unusual things or techniques that we can use I’m hoping to help you simplify your BizTalk solutions. Covering some topics like RosettaNet, SAP, database maintenance, debatching, out-of-the-box pipelines vs custom pipelines and many more

I hope you enjoy and find it an interesting session. Also I advise you to visit and view the sessions history that have taken place every Monday in the Integration User Group – Integration Monday series.

It was with great pleasure that I presented, last Monday – August 31, 2015 – my second presentation in the Integration Monday series. This time I was invited to talk about my personal experience about BizTalk NoS Ultimate and how this tool can improve our efficiency, productivity and experience while developing BizTalk projects.

And for this session I decided to take the risk of not creating a storyboard, as I normally do, preparing all the details about the entire session, test all the demos, train the session over and over to try to reach the perfection and so on… instead I decided to just spend some time on creating 5 or 6 slides that could visually express the way I see and feel this tool. And choose 2 or 3 real projects in which I could emulate the tool as I usually do in my daily work life.

The intention was only to prepare the support base for the entire presentation and then try to explain the impact of using BizTalk NoS Ultimate in the Developer Lifestyle, the advantages and how it allows me to improve my productivity/efficiency in my BizTalk projects… as if someone walked right now into my office and I had to explain him why to use and why I love it!

(That’s why some times you will see some pauses during the presentation and some “ahhh”’s and maybe some things note working so well – the risks inherent in doing this type of approach – hehhehe)


I hope you enjoyed and found this session informative and valuable, as I enjoyed doing it. And if you miss it or want to do see it again (I think anyone in their right mind wants to review) there are some good news… the video and slides from the session are now available in the Integration User Group site here:

I apologize for my voice, I was sick all week and my voice is a little husky (hoarse) – I don’t know very well the term in English – and despite being completely weird making a presentation to your computer, without seeing the people and their reactions – I’ve done four online sessions and it is impossible to get used to this format, I think I feel more comfortable being in front of an audience than making an online session hehehe – It was a pleasure to present once again in the Integration User Group.

Thank you for inviting me and see you all in a next time!

See also this amazing review made by Nino Crudele: BizTalk NOS Ultimate – BizTalk Assessment Power Up!

I’m glad to be back, once again, to Integration Monday Events! The first time I presented a session about “BizTalk Server Tips & Tricks for Developers and Admins (Deep Dive)” (June 22, 2015), a similar session that I also presented in BizTalk Summit 215 London, that you can found and watch online here.

This time I will present a session about BizTalk NoS Ultimate and how this tool can improve our efficiency, productivity and experience while developing BizTalk projects.


Session Name: Real Case Scenarios Where BizTalk NoS Ultimate Can Improve Our Efficiency

Session Overview: In this session, I will show, using real cases scenarios, how can BizTalk NoS Ultimate will help BizTalk developers to improve their efficiency, productivity and experience while developing BizTalk projects. BizTalk NoS Ultimate is a time-saving tool and, as we pass through its functionalities, we show where developers can save valuable time while working on their day-to-day tasks.

Note that I’m not a seller, and I will not try to sell this tool, instead, I will only focus in share my experience in real case scenarios where this tool can help us, at least it helped me, while developing BizTalk Projects. The rest It’s up to you if you want to use it or not.

I invite you all to join us next Monday. Grab your “seat” by registering in this session here.

In my last post I wrote about BAM Portal running in Internet Explorer 11, the browser we normally have in the BizTalk Server machine. However BAM Portal can and should be remotely accessed by the users and not directly from the server.

But this can leads us to other problems. Each user has his own preference regarding to browsers, some of them like Google Chrome, others Firefox and other IE. As recently I upgrade my laptop to Windows 10, I took this opportunity to validate the BAM Portal behavior in different browsers, including the new browser from Microsoft – Microsoft Edge

Microsoft Edge

As I expected BAM Portal does not perform well with the new Microsoft Edge browser. It ask the authentication credentials with a nice window, with a very clean layout


But after that you still have the same compatibility problem between Internet Explorer and the site: the menu have problems and you cannot select the columns to perform the queries


But unfortunately for us Microsoft Edge doesn’t have the compatibility view option, instead you have to use the option “Open with Internet Explorer”


Conclusion: BAM Portal is not supported on Microsoft Edge.

Internet Explorer 11

Again BAM Portal does not perform well with IE11 as I described in my last post: Running BAM Portal in Internet Explorer 11: Enable Compatibility view and to fix all the problems you need to add this site to the “Compatibility View Settings”

Once you turn on Compatibility View, Internet Explorer will automatically show the BAM Portal in Compatibility View each time you visit fixing all the previous behaviors problems.


Conclusion: IE is the best browser to run BAM Portal

Google Chrome

I did not have much hope that Google Chrome will be any different. And again when we try to access BAM Portal a popup asking the authentication credentials is presented


But after that you still have the same compatibility problem, again the menu have problems and you cannot select the columns to perform the queries


Google chrome doesn’t have the “Compatibility View Settings” option that exists in IE however there are some extension that you will find in chrome web store like “IE Tab” that will display web pages using IE within Chrome


After installing this extension you will have an option in the right corner of the browser to display BAM Portal using IE within Chrome


And using this technique you will be able to run BAM Portal in chrome


Mozilla Firefox

The exact same behavior happens in Firefox, the authentication popup is presented but after that you still have the same compatibility problem, again the menu have problems and you cannot select the columns to perform the queries



As with Chrome, Firefox also does not have the “Compatibility View Settings” option and also relies in extensions like “IE Tab” to render the page using IE


After installing this extension, by right click over the BAM Portal you can find the option “View Page in IE Tab” and this will do the trick for you


A new windows asking credentials is shown and after that you will have BAM Portal running fine in Firefox


In short, Microsoft Edge does not support BAM Portal. IE is the best browser to run BAM Portal and you can run BAM Portal in Chrome of Firefox using extensions to emulate IE.

Sometimes websites don’t look like you expect them to. Images might not appear, menus might be out of place, and text could be jumbled together. This might be caused by a compatibility problem between Internet Explorer and the site you’re on and most of the times this behavior can be fixed by adding the site to your Compatibility View list.

Compatibility View was introduced in Windows Internet Explorer 8 to help existing content continue to work with Windows Internet Explorer 7, while developers updated their content to support modern interoperable web standards. Well, unfortunately for us, BAM Portal never was updated and continues to have problems of compatibility.


As always one of the downsides of dealing with a new versions of products like IE and so on… is figuring out where some features/options have moved in this new release. And this exactly one of this cases…

“Starting with IE11, document modes are considered deprecated and should no longer be used. Webpages that require legacy document modes to display properly should be rewritten to use features defined by modern standards.”  – Source: Specifying legacy document modes

For this reason, in Internet Explorer 11, the compatibility view button is removed from the address bar. Microsoft felt that the button had outlived his purpose because it’s been many years now since Microsoft introduced X-UA-Compatible tags. Web developers were supposed to add these X-UA-Compatible meta tags to their website’s HTML header to indicate web page compatibility and not leave it to the user to depend on the button to correctly render the page.

However, and fortunately for us BizTalkers, the compatibility view feature still remains in the browser.

You can turn it on or off by:

  • Click on the Settings icon in the top right corner of IE11 and select the “Compatibility View Settings” item in the drop-down menu


  • In the “Compatibility View Settings” box, add the problematic website URL – aka BAM Portal – and then click Add.
    • Compatibility View is turned on for this single website, for this specific computer.
  • Enable both check box “Display intranet sites in Compatibility View” and “Use Microsoft compatibility lists”, and then click Close.


Once you turn on Compatibility View, Internet Explorer will automatically show the BAM Portal in Compatibility View each time you visit fixing all the previous behaviors problems.


Finally a new BizTalk Server book it’s available and just in time for my vacations! No I will not work during my vacations but unlike the Caribbean where we can eat and then jump right away into the water, or eat in the water, here in Europe we have to wait approximately 3 hours, depending on how much food you eat. So in that period I like to read a book and I usually take with me at least one technical book and this year will be: SOA Patterns with BizTalk Server 2013 and Microsoft Azure (Second Edition)


I like to be on the beach scratching technical books with notes and ideas Smile

This time I will not make any review on this book because I was pleased to be part of this project as a technical reviewer, so everything I can say may be suspect… and I take this opportunity to thank the authors (Richard Seroter, Mark Brimble, Johann Cooper, Colin Dijkgraaf, Mahindra Morar) for the invitation.

The book will address and will explain to you how to create and implement SOA strategies on the Microsoft technology stack using BizTalk Server 2013 and Azure Integration platforms

  • Understand how to implement SOA with BizTalk Server and the Azure platform
  • Consume and expose WCF services effectively via the use of Service Bus Relays and RESTful services
  • Implement effective schema design, including an introduction to various schema design patterns
  • Exploit various message exchange/endpoint patterns including request-response, fire and forget, and client callbacks
  • Leverage orchestration design patterns that maximize flexibility and reuse
  • Future-proof your BizTalk Server artifacts using well thought out versioning strategies
  • Build loosely-coupled BizTalk applications using the ESB Toolkit
  • Take a peek at API Apps, Logic Apps, and Azure API Management
  • And best of luck for those who are preparing to take this certification.

See Steef-Jan review about the book here and I hope you enjoy reading it. I know I will!

The book is available through Amazon and PACKT is self, so go to either of one and order it!

Have you had the time to think about what is some features like RosettaNet, ESB or UDDI have in common?

Well, all of them have custom databases and all of them are optional features.

But the most important question here, because they have custom databases, is: Do you think that these databases are being backedup? And the data inside are saved?

And the response is NO… Because these “custom” databases are not installed with BizTalk Server, they are not included in the default list of databases to be marked and backed up by the Backup BizTalk Server job. So if you want the Backup BizTalk Server job to back up RosettaNet, ESB or UDDI custom databases, you must manually add the databases to the Backup BizTalk Server job.


Fortunately for us, Microsoft provides two SQL Scripts:

  • Backup_Setup_All_Procs.sql
  • Backup_Setup_All_Tables.sql

That can be found in the Schema folder inside the BizTalk installation folder: “C:\Program Files (x86)\Microsoft BizTalk Server 2013 R2\Schema”, that you need to run against these databases in other to extend the standard backup mechanisms. However you also need to modify the adm_OtherBackupDatabases table (present in the BizTalk Management (BizTalkMgmtDb) database) to include a row for each these custom databases.

How to Back Up Custom Databases (RosettaNet, ESB or UDDI databases)

Note: I will not advise you to backup any custom application databases (database used to support BizTalk Application processes or others) with the Backup BizTalk Server job, with the exception of custom BizTalk database, like RosettaNet, UDDI or ESB.

To accomplish that you need to:

  • Browse to the “C:\Program Files (x86)\Microsoft BizTalk Server 2013 R2\Schema” directory, and then run against the RosettaNet, ESB or UDDI (custom) databases the following SQL scripts.
    • Backup_Setup_All_Procs.sql
    • and Backup_Setup_All_Tables.sql


  • So in this case we need to run these two SQL scripts against the following databases
    • RosettaNet
    • ESB Toolkit
      • EsbExceptionDb
      • EsbItineraryDb
    • UDDI
      • UDDI3

Note: This creates the necessary procedures, table, and role and assigns permissions to the stored procedures.

  • Modify the adm_OtherBackupDatabases table, present in the in the BizTalk Management (BizTalkMgmtDb) database, to include a row for each of your custom databases
    • Type the new server and database names in the corresponding columns
      • DefaultDatabaseName: The friendly name of your custom database.
      • DatabaseName: The name of your custom database.
      • ServerName: The name of the computer running SQL Server.
      • BTSServerName: The name of the BizTalk Server. This value is not used, but it must contain a value nonetheless.


The next time you run the Backup BizTalk Server job, it will back up your custom databases.


See this and more tips here: BizTalk Server Tips & Tricks for Developers and Admins (Deep Dive)