BizTalk Summit 2015 in London - Part 2 - Turning of the Tide?
In the previous article, BizTalk Summit 2015 in London - Part 1 - Microsoft’s Roadmap for Integration, we recapped how App Service is key in Microsoft’s roadmap and strategy for Integration and that Microsoft announced App Service will be available on-premises as part of the Microsoft Azure Pack. Microsoft also confirmed that the BizTalk brand is important and is here to stay.
Microsoft’s vision is to democratise integration (simplify so that developers not require specialist expert skills), be the iPaaS leader for enterprise, and build a rich ecosystem for the community and business partners.
This article shines light on various other jigsaw puzzle pieces that were presented at the BizTalk Summit 2015, to further help you understand the wider picture of where “Modern Integration” appears to be heading.
Enterprise Focus - Connecting with IBM Systems
In line with Microsoft’s vision for the enterprise and integrating with existing line of business systems, Paul Larsen, Principle Program Manager at Microsoft, presented the IBM MQ, DB2 and Informix Connectors. These connectors have been developed to allow Azure solutions to communicate with on-premises IBM systems.
An integration solution connecting to an IBM system could be deployed as a hybrid solution when using Azure in the cloud, or deployed completely on-premises solution using App Service on-premises with the Microsoft Azure Pack.
The MQ Connector currently only supports MQ version 8. In contrast, MQ version 8 is actually not supported by BizTalk Server 2013 R2. There was no information on whether version 8 would be supported in BizTalk Server 2016.
The DB2 Connector communicates to DB2 using the Distributed Relational Database Architecture (DRDA) database interoperability standard that DB2 supports. This connector also has support for custom SQL and the Open Data Protocol (OData). To achieve all of this, Microsoft have developed a new ADO.NET provider for DRDA - which also is utilised by the Informix Connector.
Democratisation Focus - The Durable Task Framework
Dan Rosanova, Senior Program Manager at Microsoft, presented a recently open-sourced library for durable, scalable, reliable, traceable and manageable .NET code-based workflows with eventual consistency.
The Durable Task Framework allows developers to write long-running persistent workflows in code using the async/await capabilities.
Interestingly, this framework originally was released as a preview in June 2013 - apparently without much fan-fare or wide reach in the community, because many people thought it was brand new.
The framework provides automatic persistence and check-pointing of program state, versioning of orchestrations and activities, error handling, compensation, automatic retries, asynchronous timers, and diagnostics.
Microsoft internally uses it to reliably orchestrate long-running provisioning, monitoring and management operations. The orchestrations can scale out horizontally by adding more worker machines.
The concepts of logic, state and the runtime are separated. State management currently happens in Azure Service Bus, optionally with an Azure Storage account. Your custom code is the orchestration logic.
Please refer to the Durable Task Framework Wiki for usage documentation.
One example use case for the use of this framework is the situation where distributed transactions or two-phase commit protocols typically would be required - and where the services being consumed do not support locking or transactions.
In the above scenario, it is common for a developer to create a BizTalk orchestration to robustly orchestrate the calling of those multiple services. Now however, there is another viable, reliable, durable and robust implementation option. Depending on the full scope of requirements, this scenario can be implemented in .NET with the Durable Task Framework and Service Bus - instead of within a BizTalk orchestration.
Imagine that part of your custom solution for a customer might be starting an orchestration task from within a Web App or an API App.
To summarise - robust, durable, traceable and manageable orchestration logic in pure .NET without the need for specialist, expert developer skills.
Democratisation Focus - Logic Apps
Stephen Siciliano, Senior Program Manager at Microsoft, presented an in-depth look at Logic Apps.
Stephen first demonstrated how to develop an application that archives Twitter tweets to a DropBox account.
The barrier of entry for developing Logic Apps is low, which means that specialised expert developer skills (such as those required for BizTalk development) are not required.
Some notable features of Logic Apps are that they are resilient against failure with an “at least once” guarantee, and they can directly call any REST endpoint without the need for an API App. There is also the ability to use parameters to separate configuration from the definitions (a good story for deployments to different environments), the support for large messages (less than 100mb), and support for binary blobs by externalising the state to storage or Base64 encoding it into the JSON message.
Enterprise Focus - Microsoft Azure BizTalk Services (MABS) 1.0 Features in App Service
Back to the enterprise vision, Stephen Siciliano also demonstrated an Enterprise Application Integration scenario with the usage of the BizTalk XML Validator, BizTalk Transform API App and BizTalk XPath Extractor in a Logic App.
Prashant Kumar, Senior Program Manager at Microsoft, discussed how the functionality of the Microsoft Azure BizTalk Services (MABS) 1.0 is available in App Service, including the XML Validator, Transform Service, and the Flat File Encoder etc.
Enterprise Focus - BizTalk B2B in App Service
Prashant Kumar also discussed and demonstrated some of the BizTalk B2B API Apps - TPM (for managing trading partners, agreements and artefacts), AS2, EDIFACT and X12.
Of note, there was discussion about how the features within the BizTalk platform are bit-by-bit being “broken out” into individual apps. One could almost describe that as a re-architecture.
Many more BizTalk features also now are becoming available in App Service, including: OAuth, JSON support, monetisation in the marketplace, pre-baked integration Logic App recipes, long running workflows and the BizTalk Rules API App.
Enterprise Focus - BizTalk Rules API App
The new BizTalk Rules API App can be used to help decouple business logic from application code. It retains many of the familiar concepts from the BizTalk Rules Engine (BRE) - such as Vocabulary, Policies and of course the Rules.
It was interesting and concerning to hear Microsoft talking in terms of enabling business users to make changes to business rules - for instance, on the fly, in production. It would be helpful if Microsoft addressed more real world concerns about a change workflow or lifecycle and the testing of business rule changes before they are deployed into production.
Enterprise Focus - API Management
Sameer Chabungbam, Principal Program Manager at Microsoft, presented how the API Apps are a powerful platform for building and managing API’s - something that is increasingly more important in the enterprise.
“Connector” API Apps let you build functionality to communicate with other systems - similar to the role of an adapter to another system.
Sameer demonstrated how to configure an API App with Swagger and optimise the API App for usage within a Logic App.
Migration of BizTalk Artefacts to App Service
Jon Fancey, Integration MVP, and Dan Probert, introduced a new company, The Migration Factory that in the near future, for a fee, aims to automate the migration of BizTalk artefacts to App Service for cloud deployment or on-premises hosting with the Azure Pack.
Due to technical complexities with some orchestration implementations, all solutions may not be 100% automated, but a very high percentage automated conversion rate is possible.
This is a rather a telling tale about the direction of Modern Integration implementations.
BizTalk Server Is Here to Stay
General consensus though is that BizTalk Server still has a niche in the market and will be around for many more years than some people can imagine. There are many on-premises systems that for various business reasons will never be deployed to the cloud.
BizTalk Server’s features are rich and powerful. As Michael Stephenson, Integration MVP coined it, BizTalk Server is the “Integration Swiss Army Knife” - it can do it all.
Steef-Jan Wiggers, Integration MVP, presented some strong cases for BizTalk Server playing a role in data enrichment and distribution, from deep integration with line of business systems such as SAP to consumer applications. Mission critical applications also are likely to remain on-premises and could be too difficult or not cost effective to redesign.
Further emphasising BizTalk Server’s relevance and interoperability in Modern Integration world, Steef-Jan also demonstrated BizTalk Server 2013 R2’s capability to communicate with REST endpoints and also convert between SOAP and REST payloads with JSON encoding and decoding.
Kent Weare, Integration MVP, also demonstrated how Azure API Management can be used to create a REST API endpoint that acts as a facade to an existing SOAP endpoint on an on-premises BizTalk Server.
Business Activity Monitoring (BAM) in Power BI
Todd Glad Nordahl, Technology Solution Professional at Microsoft, presented the Power BI Designer and Dashboard, and discussed how it can be used on top of BizTalk Server to connect to BizTalk’s Business Activity Monitoring (BAM) data and augment that with other data sources to help make business decisions.
Windows Communication Foundation (WCF)
There was little discussion about the future of WCF, which appears at odds with Modern Integration and the Web and API Apps. WCF seems to be fading into the background and becoming a Legacy Integration technology.
It would be interesting to see an official statement from Microsoft about the future of WCF.
Reflections
Reading between all these lines, the BizTalk platform is being re-architected for the Modern Integration world, into App Service, which can be used in the cloud or on-premises.
The deployment, versioning and dependency issues that plague many BizTalk Server solutions due to its dependency on the Global Assembly Cache (GAC) will finally be resolved by moving to the App Service architecture.
There however are still many unanswered questions and many unknowns.
- Within the world of Microservices and new container technologies, there will need to be a brand new set of guidelines and better practices to understand and apply.
- Of course, there may be a bunch of new challenges waiting to be discovered and solved, and avoiding deployment issues in a Microservice world is one of them.
- Logic Apps have tracking and archiving of content (does anyone have concerns about privacy with this?), and Azure API Management provides analytics. I am not sure how this will translate into equivalent and improved functionality of BAM.
All the pieces on the chess board are being lined up, and Microsoft now appears to have a solid architecture for Modern Integration. We now may be witnessing the start of the turning of the tide.
Architects and enterprises now will have the choice to use App Service functionality (in the cloud or on-premises) in addition to the option of implementing a solution using the equivalent functionality in BizTalk Server.