Friday, May 1, 2015

Begineer to BizTalk Expert : Interview with Nino Crudele

Welcome to eleventh interview of the series, today's expert is Nino Crudele.

He is full of passion for integration world and he likes to use all possible technologies to do that, Microsoft Azure, Event Hub, Stream Analytic, BizTalk Microservices, BizTalk Server, and all possible Integration technologies patterns.

Nino Crudele has been an BizTalk MVP for 7 years, besides that he is also a Virtual Technology Specialist for BizTalk Server, a role in which he works closely with Microsoft. He enjoys this role as it enables him to work with many different clients seeing BizTalk in different scenarios from a technical point of view, size, and criticality.

He developed the famous BizTalk Visual Studio NoS Addin.
All information here:
http://visualstudiogallery.msdn.microsoft.com/3b4d76e8-6875-4990-b986-6a28691da5a3

Nino has worked with most of the BizTalk accelerators, Rosettanet, HL7, SWIFT and has done some big projects on EDI and RFID. In the past he also had the opportunity to be part of a big integration project with a major Italian enterprise organization that integrates many technologies like SAP, AS400, TIBCO and others using BizTalk.

He is founder of the first Italian BizTalk communty, http://biztalkia.com
He is founder of the first Italian Connected System communty, http://ugics.org

Certifications:
MCTS BizTalk Server, MCT, MCAD and MCSD.


His blog at http://ninocrudele.me



Let's begin the interview.....


Mahesh: Who are you and what you do? 
Nino: I’m Nino Crudele, I have three amazing children Rebecca, Carolina and Riccardo, one wonderful wife, Grazia, and my life is totally focused on them.
I have been Application Integration MVP for 8 years, I’m working in Solidsoft Reply as Application Integration Specialist.
I like sports as Kayak and Skateboard, when I need to relax I just like to take a run with my skateboard, I also like to read about maths, science and play Chess.

Mahesh: When did you start working on BizTalk? 
Nino: I start many years ago using BizTalk Server 2002 version and I have a good memory about this time.

Mahesh: How did you mastered BizTalk (Learning path, amount of time)? 
Nino: I was really lucky about it because I worked for many years as Virtual Technology Specialist for Microsoft and I had the opportunity to do a lot of different things with BizTalk.
I used all the adapters and accelerators and I followed a lot of different critical missions, this was a good way to improve my knowledge.

Mahesh: Which are the major projects you handled so far? 
Nino: I worked in many different projects and for all the biggest companies in Italy, I have many different and interesting situations to speak about, some of them because using hundreds of CPUs or BizTalk boxes other because using a lot of different accelerators and adapters.

Mahesh: How do you see BizTalk compare to other integration platform? 
Nino: Good question and not so simple to answer, I think that BizTalk is the most complete and solid platform that Microsoft has developed, I really love BizTalk Server and my opinion is not because I’m Microsoft MVP but because it is real.
I used other integration platforms but BizTalk Server is the only one which able to cover all the most important aspects, for example reliable, correlation and transaction (SAGA), in very easy way.

Mahesh: What as per you is must to know to become an Integration(BizTalk) Expert? 
Nino: A BizTalk expert is a person who able to know all about the product and how to solve problem using it, he also needs to have a very good vision about integration scenarios and different technologies.
BizTalk runs on SQL Server, Windows resources, network, he has a lot of different features inside which use IIS, Disk I/O, and BizTalk has hundreds of adapters and thousands of setting to know.

An Integration expert needs to know a universe of things and BizTalk is one part of these.

Mahesh: What are your thoughts on forums,blogs and articles etc.? 
Nino: I think that all of these things are important to share knowledge, I like to write article and share, this is the best option we have to merge our passion.

Mahesh: Your suggestion to a newcomers? What should be approach to get sound knowledge in BizTalk? 
Nino: My best advices are:Keep calm, sit and start understanding the engine, BizTalk is not a complicate thing and there is a lot of good documentation and books in internet.

Start following the BizTalk engine flow, concept of port -> receive location -> adapter -> pipeline -> orchestration
How it works, Publish / Subscribing and Message context.

Mahesh: What is a BizTalk Host and Host Instance?
Nino: Don’t start developing because drag and drop and don’t force yourself using only BizTalk things, you are a developer, and you have also to use code.
Understand the BizTalk domain and what it means developing and using an own external library and how to manage it in the best way, in term of monitoring and errors.

Mahesh: What are your thoughts around BizTalk certification? 
Nino: Is a good option to certify your capabilities.

Mahesh: What is the future of BizTalk? 
Nino: BizTalk is a great product and I think it will be present for a long time.

Mahesh: Any thoughts on cloud? 
Nino: Microsoft released a lot of new interesting things in Azure , now we have a lot of more options and a lot of different technologies to use , I think That now the challenge is not about a Particular product but is the understanding how to use all of them in the best and convenient way ,

I like to say that the power of Azure is directly proportional with the summation of things that we are able to combine together.
I think about Azure as a little universe full of things and the collision between them is able to create energetic solutions.

The world of integration is huge and we have many different options to solve problems, I’m working in this new engine, JiTGate (Just in Time Gate), it is able to use all of these things together to integrate technologies in very easy and fast way, I’m in a good point with the development and I hope to release a version soon.

Mahesh: What motivates you to do the community work? 
Nino: I think the knowledge That Has to be free and not drove by the main actors, I'm not just speaking only about technology but about all, we have to be free to share and help people to understand and to live a better.

Mahesh: Being MVP, do you feel that responsibilities get added? What is your thought on MVP?
Nino: Yes I’m feeling the responsibilities to be a MVP, it is a big thing, the MVP family is full of great persons and I’m honoured to be part of this.


Thanks a lot Nino for great insights, this will surely benefit many !!!


Feel Free to ask questions to Nino in the comments!!!!!!!!



Related Post:



Monday, April 6, 2015

Fetching Data From Multiple Tables Through Dynamically Invoking Stored Procedure

There is a comment/question posted on following post:Polling data from SQL using WCF-SQL Adapter in BizTalk 2010 - TypedPolling(From Multiple table)  where Vinoth posted the scenario:

"Please help me, I have to poll the data from 15 Tables, Table by Table, should not join in to single select statement. For example, if I have 15 Tables to poll, I have to poll Table1 and store as XML and i have to poll Table 2, then 3, 4, 5...
Multiple Table polling by looping, how to do it dynamically poll the data Table by Table...
Please provide me the solution? its very urgent... "


There are situations where we have to configure WCF port dynamically,

1. The information for the configuration is available at runtime through some content in message or through some other way.

2. The configuration information is already known but it can change or has to be changed depending on the business needs.

To demonstrate how to configure WCF Port dynamically and how  to Invoke stored procedures dynamically, have used following scenario:

Scenario:

 I have to poll the data from Tables, one by one and save it on file system. The challenge is the number of tables can increase or can be reduced .For this demonstration , I have used 3 tables (Logic remains same for 15 tables as well).

Ok, as per the requirement posted the configuration of the port which fetches data has to be done dynamically and in a loop, so nothing has to be static inside orchestration related to the storedprocedure thus used BRE as it provides a way to do changes without having to change anything in orchestration after initial CallRules configuration. 


I have made some assumptions:

1. There is chances of changes in number of tables to be fetched - thus have used BRE which has provision of changing number of table without touching the Orchestration.
2. The response received is not to be mapped, instead it is to be saved as it is.
3. Invoking stored procedure can be anywhere in the designed process, thus have used dummy Order to instantiate the orchestration (which has nothing to do with logic)

Building the solution:


1. Database


a. Creating three tables

Figure1. Demo Database with three tables


Figure2. Order table design

 
Figure3. Order1 table design

Figure4. Order2 table design


b.Creating three stored procedures with simple query


Figure5. uspGetOrder1 stored procedure
Figure6. uspGetOrder2 stored procedure
Figure7. uspGetOrder stored procedure
  
Figure8. Three tables and respective stored procedures

2. BRE


a. Creating Policy GetConfigDetails


Figure9. GetConfigDetails Policy

b.Creating four rules under GetConfig

The first rule is used to set the number of tables to be fetched.


Figure10. GetConfigDetails Policy - Rule1

The rest three rules are used to set the properties i.e. Action,Address,Bindingtype and TransportType which are required to configure the port dynamically.


Figure11. GetConfigDetails Policy - Rule2


Figure12. GetConfigDetails Policy - Rule3


Figure13. GetConfigDetails Policy - Rule4

   

3. BizTalk Solution

For solution the name given is DynamicInvocationOfStoredProcedures which has schemas(intiating message, Configuration message , storedprocedure message), Orchestration, strong name key(used to sign the project) and binding files which are generated by wizard while creating schemas from stored procedures. Steps are as below.

Figure14. Solution structure

a. Create schema for message to instantiate the orchestration


b. Create schema for message to hold configuration details - which is updated using BRE

All the elements are promoted as distinguished field so as to get/set values from those in orchestration

c. Generate schemas from the stored procedures which are to be executed dynamically

We need to do this so that we can refer to the binding files generated to get configuration details.

1. Right click the project and  select Add-->Add Generated Items
2. Then select "Consume Adapter Service"
3. Now the Consume Adapter Service wizard wants us to provide it the required information, so firstly select "sqlbinding" against the label Select a binding.
4. Click the Configure button, on security tab select the Client credential type
5. On URI Properties tab:
      i.Type the name of the server where your database is located ( I have used "." as the database resides on same machine)
     ii. Provide the name of the database you want to use.
     iii. I have left InstanceName as blank because am using the default instance of SQL (If need to use some other instance then provide the name of that instance)


6. After done with above configuration,click on Connect then select Contract type as Client(Outbound operations)
7. Select the category as Procedures then select procedure based on which schema is to be generated, click ADD
8. Give Filename Prefix (it's optional - but it helps) and click ok
Figure. Generating schema from uspGetOrder storedprocedure

9. Repeat above steps for other stored procedures

d. Create Orchestration



Figure.PollMultipleOneByOneDynamically Orchestration

The orchestration as shown above has following variables and types used in it


Ports and Port Type
1. prtRcvStart of type prttypeRcvStart - to receive intial message thus binded to Order schema as intial message is based on it

2. prtExecuteSP of type prttypeExecuteSP - it is dynamic port to send message to invoke storedprocedure and to receive response from storedprocedure,binded to System.XMLDocument.
Why System.XmlDocument? - As per the requirement multiple storedprocedures are to be invoked but those are not static and are to be set dynamically so this port can't be binded to single schema. And as System.XmlDocument is the base schema for all schema it can be used in the scenario where generic approach is required.

3.prtSndInvokeSPResp of type prttypeSndInvokeSPResp - to send the response received from stored procedure to a folder


Figure.PollMultipleOneByOneDynamically Orchestration variables and types

Message and  Variables
1. msgstart - based on Order schema, used to instantiate the orchestration
2. msgConfig - based on Config schema, used to get config details from BRE
3. msgInvokeSP - based on System.XmlDocument, used to send message expected by StoredProcedure
4. msgInvokeSPResp - based on System.XmlDocument, used to get response from stored procedure
5. varLoop - Int variable to hold the value (number of times loop is to be run)
6. varAction - String variable to hold Action value set on msgConfig by GetConfigdetails policy
7. varAddress - String variable to hold Address value set on msgConfig by GetConfigdetails policy
8. varBinding - String variable to hold Binding value set on msgConfig by GetConfigdetails policy
9. varTransportType- String variable to hold TransportType value set on msgConfig by GetConfigdetails policy
7. xmlDoc - Variable of type System.XmlDocument

Steps in Orchestration

1. Receive start message
2. Construct the config message in message assignment shape, this message will be passed, here to note is that GetTableNumber element has values as Yes  

 3. Configure  the CallRules shape, select the GetConfigdetails policy and message expected by it  - msgConfig

4. Assign  the Loop variable to the value received in NumberOfTables field of msgConfig
5. Set the Loop condition

6. Inside loop , call the policy
7. Assign  the varAction,varAddress, varBinding,VarTransportType to the value received in msgConfig





8. Construct the message to invoke the stored procedure and set it's WCF.Action property - this is where we are dynamically setting the action - which we get from Policy we execute in step 6



9. Send the message to prtExecuteSP and collect the response
10. Send the collected response to the folder
11. So  now reset the loop variable by reducing it by one and also now next table is to be fetched so set TableNumber field of msgConfig respectively.










Allright, so why CallRules is used twice in above Orchestration?

Considering that number of tables to be fetched can be changed, the first callrule is used to tackle this scenario. So when Orchestration passes config message to rule engine having GetTableNumber as Yes, then  the NumberOfTables is set  and returned. And in loop, Orchestration is interested in knowing which StoredProcedure is to be executed(the WCF.Action property) and here second callrule is used to get the Action against the TableNumber.

If you see at Rules above, it will be more clear :)

e. Build, Deploy and Configure


After deployment the dynamic port is automatically added to the Application in the Admin console, whose configuration we did in orchestration dynamically.





So we just need to configure one Recieve Port, one static send port and bind it to Logical ports of Orchestration

Figure. After  Binding Orchestration

To test I dropped the instance of Order message (message which instantiates the orchestration) and three xml files were generated in the folder which is binded to the port which sends the response received from stored procedures.



Note : System.XmlDocument is used as base for the message  to communicate with stored procedures so it's untyped . There are two limitation that the response received can't be used in any map and if there is subscriber in your group for untyped message then copy of it will be passed to it as well.




Download Sample:

Wednesday, April 1, 2015

Begineer to BizTalk Expert : Interview with Michael Stephenson

Welcome to tenth interview of the series, today's expert is Michael Stephenson.

Michael is an independent integration & cloud specialist based in the UK. He is primarily focused around integration technologies in the Microsoft integration platform such as BizTalk, Windows Azure, and a number of other open source offerings. 

Michael has many years of technical leadership and coaching experience and has worked with customers to deliver a number of complex real-world hybrid integration solutions connecting business partners and SAAS applications to an organisations on premise line of business systems.  Prior to freelancing Michael also held a technical evangelist role with Edenbrook who were a large UK consultancy.

Michael is also heavily involved in community activities such as the UK Connected Systems User Group and blogging.  Michael also recently pioneered the BizTalk Maturity Assessment (http://www.biztalkmaturity.com) which is designed to help customers ensure they are doing the right things to deliver successful projects with BizTalk. Recognized by Microsoft as an Integration MVP - 7 Years.


Lets begin the Interview....

Mahesh: Who are you and what you do?
Mike:
I am Mike Stephenson, i am 38 years old and live in Newcastle in the North East of England.  I work as a freelance Consultant specializing in Integration and Cloud technologies on the Microsoft technology platform. I work with customers in the UK and abroad to help them create traditional integration solutions and also to understand how to leverage the cloud to create hybrid integration solutions.


 
Mahesh: When did you start working on BizTalk?
Mike:
I started working on BizTalk in 2002 with a company who specialized in supply chain outsourcing and have worked on every version of BizTalk since then.

Mahesh: How did you mastered BizTalk (Learning path, amount of time)?
Mike:
Personally I think BizTalk is a product where you always have something to learn.  There are so many different types of solution you can create whether it is integration patterns you have not used before or applications you have not integrated with.  Today technology is also changing very quickly so there are always new things to learn.
I also like to consider myself to be an Integration Specialist rather than just a BizTalk person.  I think its also very important to learn about the other integration technologies too and to understand where to use what.
For training I usually use Pluralsight and Technet wiki and peoples blogs to keep up to speed and also attend #IntegrationMonday web casts


Mahesh: Which are the major projects you handled so far?
Mike:
I have handled a number of projects over the years, back when I used to work for consultancies I spent time spent time supporting projects at many different customers and I have also worked on a number of projects with healthcare companies but more recently i am working in the higher education sector

Mahesh: How do you see BizTalk compare to other integration platform?
Mike:
If you compare BizTalk to other products within the microsoft integration technology suite then I consider BizTalk to be like the swiss army knife which has something for almost any integration problem.  This is why many customers use BizTalk at the heart of their integration solutions.
If you compare BizTalk (or microsoft integration) to other vendors then I think you will find it really depends upon what your use cases are.  I guess as a generalization there are two types of vendors those who provide the heavy weight tools which have been around for years and the newer breed of vendors who offer the lighter weight integration platforms where you can provide simpler ways to solve some of the newer world integration problems but often struggle to deal with the more complex integration solutions.
I think what is interesting about microsoft is it is moving quickly towards a good offering in both spaces. 

When it comes to comparing I think it is important to get past the marketing car sales man view of the products you will come across and to look into the real capabilities and how they are implemented.  This is why i contributed to the whitepaper by Kent Weare http://www.biztalk360.com/whitepaper/choosing-an-integration-platform/


Mahesh: What as per you is must to know to become an Integration(BizTalk) Expert?
Mike:
I am not a great fan of the term integration expert because expert implies you know everything and in integration there is such a wide range of things to consider that you can never know everything about everything and the best skill is to try and be humble and meet new people with different ideas and be learning all of the time.

To excel at integration i think you need to combine knowledge about the integration products you use with knowledge about best practices for application life cycle management and then the non technical skills such as how to work well with other people and how to understand managing dependencies.

I think you need to have a well rounded skill set of the above things to do well in the integration space


Mahesh: What are your thoughts on forums,blogs and articles etc.?
Mike:
I have always thought the biztalk community have an excellent set of blogs, forums etc.  I have my own preferences about how i like to interact but there is always opportunities to get involved and the community is very supportive of new contributors

 
Mahesh: Your suggestion to a newcomers? What should be approach to get sound knowledge in BizTalk?
Mike:
I think the most important thing is hands on experience and the best way to get that is working with lots of different people to discover the practices they use to decide which you like best.  Books, videos etc are all great but to get to the next level its really about hands on experience and appreciating that no matter how much experience you have you should be learning every day

Mahesh: What are your thoughts around BizTalk certification?
Mike:
I think BizTalk certifications have a value up to a point but over the years i have met people who had certifications who werent that great and i have met many people who have not considered certifications who were truly excellent.

If I was interviewing people a BizTalk certification would not be a factor which would make me choose on person over another for a role but is certainly a nice badge to have on the CV and I guess the best thing about certifications is that it can make you look at parts of BizTalk you may have never used before.


Mahesh: What is the future of BizTalk?
Mike:
Right now I am more excited about the future of BizTalk and Microsoft Integration than I have been at anypoint for a long time.  I think combining BizTalk with the new azure technologies will let us build some great solutions.

 


Mahesh: Any thoughts on cloud?
Mike:
I am a huge fan of cloud and like to work with customers who have a desire to consider where the cloud could benefit their organisation.

I would say to anyone now working commercially with cloud you absolutely must be learning about it in your own time or you will quickly fall behind which could affect your opportunities in the future.


Mahesh: What motivates you to do the community work?
Mike
: I think I like the challenge of doing interesting stuff and learning but also over the years i have developed some excellent friendships with the people in the integration community and its cool to work together on little projects and also to help and encourage each other

Mahesh: Being MVP, do you feel that responsibilities get added? What is your thought on MVP?
Mike:
I dont think the mvp programme adds extra responsibilities.  It is an excellent way to bring together community influencers and to help encourage people to participate in the community.  I think ive been in the programme for 7 or 8 years now and as result ive developed friendships with people in many countries around the world.  It rocks!


Thanks a lot Mike for great insights, this will surely benefit many !!!


Feel Free to ask questions to Mike in the comments!!!!!!!!



Related Post:



Friday, March 13, 2015

How to Access an Orchestration variable in Map

Many times while developing BizTalk application there arises a need to access some message context properties or some variables from orchestration in map. For message context properties you can manage with the help of message assignment shape,  for example: filename of incoming message(received via file or ftp adapter) is to be passed to a field in destination message. Consider incoming message is declared as msgIncoming and outgoing message is declared as msgOutgoing and it has field as Receivedfilename,in that case:
  • Mark the field ReceivedFileName as distinguished
  • In message assignment shape, add following code:
               msgOutgoing.ReceivedFileName=msgIncoming(File.ReceivedFileName);

Likewise any other message context property can be accessed provided that it is written in context either by adapter or pipeline, in above scenario if some other adapter is used then File.ReceivedFileName  won’t be available as this property  is only populated when file adapter is used.  But what about orchestration variable? Well there is no direct way to access it in map.


So how to access orchestration variables in map?


This question is very frequently asked on forums, also I have received questions directly around it along with following questions(answers of all I intend to cover in this post):
1. How to pass ReceivedFileName in map/How to insert ReceivedFileName in database
2. How to construct message using LoadXml Method
3. How to pass orchestration variable in map
4. How to create  map with multiple source to single destination
5. How to configure WCF-Custom adapter
6. How to generate schema from Table

To answer all questions above, have considered following scenario:

Scenario:

Order is received on file location and it is to be inserted in database along with FileName.

Creating the Solution:


Create Order table


Create Incoming Order Schema



Generate schema from Order table



Binding : sqlBinding as the database we are dealing is SQL Server
IntialCatalog : Demo as this is database which is to be connected and it holds the table we are dealing with
Server : “.” As database resides on same machine, if it was on other machine then it would have been that machine’s address.
Select contract type: Client (Outbound Operations) as message is going out of BizTalk
Operations: Insert as data will be inserted in the table



  • Select sqlBinding



  • Select how you would connect to SQL



  • Provide the database where the table reside against Initial catalog and on which server (here it is dot as database in on local machine)


  • Click Connect 




  • Select outbound operation as data is inserted into SQL from BizTalk (outbound in reference to BizTalk), if we were to receive then it would have been Inbound operation


  • Select the table and four operations will be visible


  • Provide filename prefix (not mandatory) but it helps in managing and reading when number of generated schemas are more in a project




  • As highlighted above, four files are added 


Create supporting schema 


To hold Orchestration variable and context properties. In this schema add fields corresponding to the context properties and variables which you want to use in map, here as I am demonstrating only one context property so only one field in the schema.




And mark the fields as Distinguished, so as to make it available in message assignment shape to assign values to it




 Create Orchestration



1. Declare messages and variable
  • Create three messages tied  to their respect respective Schema

  • XmlDoc variable of type System.Xml.XmlDocument

Why XmlDocument variable? The class "XmlDocument" is a super class for all XML messages, and hence it can hold any type of XML message and subsequently any type of orchestration message.

2. Add Receive shape to receive Incoming Order
3. Add construct shape with message assignment shape to construct the msgOrchestrationVariables, here we use LoadXml method to construct it.


Apart from Map, Message Assignment, using .Net, the fourth way to construct message is with the help of LoadXML method – which provides a way to load the instance of schema of the message which is to be created (with or without actual values)
4. Add construct shape to create msgSQLOrder, this construct shape uses transform shape to create destination message from incoming message and the supporting message which we created in previous step
5.Thus there are two source to this map, msgIncomingOrder and msgOrchestrationVariables and one destination – msgSQLOrder
Mapping is straight forward
6. Add send shape to send the destination message, final orchestration is as below

Deploy and Configure Application

Sign with strong name key, assign Application name,  Build the application and Deploy. After deploying configure the application in BizTalk Admin Console, Receive port with a receive location watching the file location (File Adapter) with xml as file mask.








Send Port with destination url set to the Order Table we created at start
We can directly import the binding which was added along with the schemas by the Add Generated Items wizard, this will create a preconfigured URI, Action and bindings. But here, I have created  static Sendport static  explicitly created and configured it manually

So how do I know what values to configure? It is the binding file which has all the information

 WCF-SQL adapter is dedicated adapter for SQL which has pre configured SQL Binding, but here WCF-Custom adapter is used so binding has to be explicitly selected, click on the drop down against the Bindingtype and select sqlBinding, all the properties will be populated with default values.

Testing


To test the application, two sample files generated from the Order schema, and dropped at the file location which is configured as Receive Location, and below is what appears in table 



Download Sample:


Related Post: