BizTalk Summit 2015 in London - Part 2 - Turning of the Tide?

In the previous article, BizTalk Summit 2015 in London - Part 1 - Microsoft’s Roadmap for Integration, we recapped how App Service is key in Microsoft’s roadmap and strategy for Integration and that Microsoft announced App Service will be available on-premises as part of the Microsoft Azure Pack. Microsoft also confirmed that the BizTalk brand is important and is here to stay.

Microsoft’s vision is to democratise integration (simplify so that developers not require specialist expert skills), be the iPaaS leader for enterprise, and build a rich ecosystem for the community and business partners.

This article shines light on various other jigsaw puzzle pieces that were presented at the BizTalk Summit 2015, to further help you understand the wider picture of where “Modern Integration” appears to be heading.

Enterprise Focus - Connecting with IBM Systems

In line with Microsoft’s vision for the enterprise and integrating with existing line of business systems, Paul Larsen, Principle Program Manager at Microsoft, presented the IBM MQ, DB2 and Informix Connectors. These connectors have been developed to allow Azure solutions to communicate with on-premises IBM systems.

An integration solution connecting to an IBM system could be deployed as a hybrid solution when using Azure in the cloud, or deployed completely on-premises solution using App Service on-premises with the Microsoft Azure Pack.

The MQ Connector currently only supports MQ version 8. In contrast, MQ version 8 is actually not supported by BizTalk Server 2013 R2. There was no information on whether version 8 would be supported in BizTalk Server 2016.

The DB2 Connector communicates to DB2 using the Distributed Relational Database Architecture (DRDA) database interoperability standard that DB2 supports. This connector also has support for custom SQL and the Open Data Protocol (OData). To achieve all of this, Microsoft have developed a new ADO.NET provider for DRDA - which also is utilised by the Informix Connector.

Democratisation Focus - The Durable Task Framework

Dan Rosanova, Senior Program Manager at Microsoft, presented a recently open-sourced library for durable, scalable, reliable, traceable and manageable .NET code-based workflows with eventual consistency.

The Durable Task Framework allows developers to write long-running persistent workflows in code using the async/await capabilities.

Interestingly, this framework originally was released as a preview in June 2013 - apparently without much fan-fare or wide reach in the community, because many people thought it was brand new.

The framework provides automatic persistence and check-pointing of program state, versioning of orchestrations and activities, error handling, compensation, automatic retries, asynchronous timers, and diagnostics.

Microsoft internally uses it to reliably orchestrate long-running provisioning, monitoring and management operations. The orchestrations can scale out horizontally by adding more worker machines.

The concepts of logic, state and the runtime are separated. State management currently happens in Azure Service Bus, optionally with an Azure Storage account. Your custom code is the orchestration logic.

Please refer to the Durable Task Framework Wiki for usage documentation.

One example use case for the use of this framework is the situation where distributed transactions or two-phase commit protocols typically would be required - and where the services being consumed do not support locking or transactions.

In the above scenario, it is common for a developer to create a BizTalk orchestration to robustly orchestrate the calling of those multiple services. Now however, there is another viable, reliable, durable and robust implementation option. Depending on the full scope of requirements, this scenario can be implemented in .NET with the Durable Task Framework and Service Bus - instead of within a BizTalk orchestration.

Imagine that part of your custom solution for a customer might be starting an orchestration task from within a Web App or an API App.

To summarise - robust, durable, traceable and manageable orchestration logic in pure .NET without the need for specialist, expert developer skills.

Democratisation Focus - Logic Apps

Stephen Siciliano, Senior Program Manager at Microsoft, presented an in-depth look at Logic Apps.

Stephen first demonstrated how to develop an application that archives Twitter tweets to a DropBox account.

The barrier of entry for developing Logic Apps is low, which means that specialised expert developer skills (such as those required for BizTalk development) are not required.

Some notable features of Logic Apps are that they are resilient against failure with an “at least once” guarantee, and they can directly call any REST endpoint without the need for an API App. There is also the ability to use parameters to separate configuration from the definitions (a good story for deployments to different environments), the support for large messages (less than 100mb), and support for binary blobs by externalising the state to storage or Base64 encoding it into the JSON message.

Enterprise Focus - Microsoft Azure BizTalk Services (MABS) 1.0 Features in App Service

Back to the enterprise vision, Stephen Siciliano also demonstrated an Enterprise Application Integration scenario with the usage of the BizTalk XML Validator, BizTalk Transform API App and BizTalk XPath Extractor in a Logic App.

Prashant Kumar, Senior Program Manager at Microsoft, discussed how the functionality of the Microsoft Azure BizTalk Services (MABS) 1.0 is available in App Service, including the XML Validator, Transform Service, and the Flat File Encoder etc.

Enterprise Focus - BizTalk B2B in App Service

Prashant Kumar also discussed and demonstrated some of the BizTalk B2B API Apps - TPM (for managing trading partners, agreements and artefacts), AS2, EDIFACT and X12.

Of note, there was discussion about how the features within the BizTalk platform are bit-by-bit being “broken out” into individual apps. One could almost describe that as a re-architecture.

Many more BizTalk features also now are becoming available in App Service, including: OAuth, JSON support, monetisation in the marketplace, pre-baked integration Logic App recipes, long running workflows and the BizTalk Rules API App.

Enterprise Focus - BizTalk Rules API App

The new BizTalk Rules API App can be used to help decouple business logic from application code. It retains many of the familiar concepts from the BizTalk Rules Engine (BRE) - such as Vocabulary, Policies and of course the Rules.

It was interesting and concerning to hear Microsoft talking in terms of enabling business users to make changes to business rules - for instance, on the fly, in production. It would be helpful if Microsoft addressed more real world concerns about a change workflow or lifecycle and the testing of business rule changes before they are deployed into production.

Enterprise Focus - API Management

Sameer Chabungbam, Principal Program Manager at Microsoft, presented how the API Apps are a powerful platform for building and managing API’s - something that is increasingly more important in the enterprise.

“Connector” API Apps let you build functionality to communicate with other systems - similar to the role of an adapter to another system.

Sameer demonstrated how to configure an API App with Swagger and optimise the API App for usage within a Logic App.

Migration of BizTalk Artefacts to App Service

Jon Fancey, Integration MVP, and Dan Probert, introduced a new company, The Migration Factory that in the near future, for a fee, aims to automate the migration of BizTalk artefacts to App Service for cloud deployment or on-premises hosting with the Azure Pack.

Due to technical complexities with some orchestration implementations, all solutions may not be 100% automated, but a very high percentage automated conversion rate is possible.

This is a rather a telling tale about the direction of Modern Integration implementations.

BizTalk Server Is Here to Stay

General consensus though is that BizTalk Server still has a niche in the market and will be around for many more years than some people can imagine. There are many on-premises systems that for various business reasons will never be deployed to the cloud.

BizTalk Server’s features are rich and powerful. As Michael Stephenson, Integration MVP coined it, BizTalk Server is the “Integration Swiss Army Knife” - it can do it all.

Steef-Jan Wiggers, Integration MVP, presented some strong cases for BizTalk Server playing a role in data enrichment and distribution, from deep integration with line of business systems such as SAP to consumer applications. Mission critical applications also are likely to remain on-premises and could be too difficult or not cost effective to redesign.

Further emphasising BizTalk Server’s relevance and interoperability in Modern Integration world, Steef-Jan also demonstrated BizTalk Server 2013 R2’s capability to communicate with REST endpoints and also convert between SOAP and REST payloads with JSON encoding and decoding.

Kent Weare, Integration MVP, also demonstrated how Azure API Management can be used to create a REST API endpoint that acts as a facade to an existing SOAP endpoint on an on-premises BizTalk Server.

Business Activity Monitoring (BAM) in Power BI

Todd Glad Nordahl, Technology Solution Professional at Microsoft, presented the Power BI Designer and Dashboard, and discussed how it can be used on top of BizTalk Server to connect to BizTalk’s Business Activity Monitoring (BAM) data and augment that with other data sources to help make business decisions.

Windows Communication Foundation (WCF)

There was little discussion about the future of WCF, which appears at odds with Modern Integration and the Web and API Apps. WCF seems to be fading into the background and becoming a Legacy Integration technology.

It would be interesting to see an official statement from Microsoft about the future of WCF.

Reflections

Reading between all these lines, the BizTalk platform is being re-architected for the Modern Integration world, into App Service, which can be used in the cloud or on-premises.

The deployment, versioning and dependency issues that plague many BizTalk Server solutions due to its dependency on the Global Assembly Cache (GAC) will finally be resolved by moving to the App Service architecture.

There however are still many unanswered questions and many unknowns.

  1. Within the world of Microservices and new container technologies, there will need to be a brand new set of guidelines and better practices to understand and apply.
  2. Of course, there may be a bunch of new challenges waiting to be discovered and solved, and avoiding deployment issues in a Microservice world is one of them.
  3. Logic Apps have tracking and archiving of content (does anyone have concerns about privacy with this?), and Azure API Management provides analytics. I am not sure how this will translate into equivalent and improved functionality of BAM.

All the pieces on the chess board are being lined up, and Microsoft now appears to have a solid architecture for Modern Integration. We now may be witnessing the start of the turning of the tide.

Architects and enterprises now will have the choice to use App Service functionality (in the cloud or on-premises) in addition to the option of implementing a solution using the equivalent functionality in BizTalk Server.

BizTalk Summit 2015 in London - Part 1 - Microsoft's Roadmap for Integration

I think everyone agrees that the BizTalk Summit 2015 in London was a smashing success. Saravana Kumar’s BizTalk360 team have once again done a fantastic job at uniting the world of Integration. This year saw 330 people from 20 countries, with all the speakers being either Microsoft employees or Integration MVP’s.

The first day primarily was filled with the presentations from Microsoft on their new App Service functionality, and in the second day the Integration MVP’s were allowed to shine.

This first article concentrates on the roadmap and strategy that Microsoft has for Integration.

Important On-Premises Announcements

Before we dive into the roadmap, there were two important announcements for the community in the area of on-premises integration.

Firstly, there will be a new major version of BizTalk Server released in 2016 to align with and provide support for the new releases of Windows platform products - such as the next version of Windows Server, SQL Server and Visual Studio.

Secondly, the App Service functionality will be available on-premises with the next release of the Windows Azure Pack. The release date however was not specified, but from a few discussions I have the impression that it will be months away, not weeks.

Microsoft’s Roadmap and Strategy for Integration

Josh Twist, Principal Program Manager on the Microsoft Azure team, originally was the keynote speaker but unfortunately was unable to attend. The audience nevertheless warmly received the keynote presentation by Karandeep Anand, Partner Director of Program Management at Microsoft.

Karandeep’s keynote articulately explained Microsoft’s roadmap and strategy for integration. He spoke of Microsoft’s journey to the cloud so far and the many learnings they have gained.

Learnings from Azure Websites

Azure Websites (now Web Apps) is by far, significantly the largest service currently used in Azure. The explanation, according to Karandeep, is the simple, low-complexity barrier of entry, with rich features and tooling, automatic load balancing, scaling and geo-redundancy.

The identified gaps with Websites however are a lack of integration with business logic, rules, triggers or workflow.

Learnings from BizTalk Services

There were many learnings from Microsoft’s BizTalk Services offering as well. Importantly for many customers, BizTalk as a brand name has been recognised officially by Microsoft as being important. Karandeep confirmed that the BizTalk name is here to stay. The BizTalk Services offering also validated various cloud design patterns, and hybrid connectivity was identified as critical and one of their differentiators in the market.

The identified gaps with BizTalk Services include the need for more out-of-the-box sources and destinations, pipeline templates, custom code support, long-running workflows and parallel execution.

All in all, Microsoft identified the need to significantly invest in this space to approach the same value and functionality as BizTalk Server.

Learnings from BizTalk Server

The learnings from BizTalk Server also were quite interesting. It was no surprise however when Karandeep mentioned that there is a high-complexity barrier of entry into the world of BizTalk. This unfortunately encourages a proliferation of the “hack zone” where developers hack together applications in an effort to develop a quick solution, but these applications or scripts don’t necessarily have the robustness, scalability or desired level of support and maintainability.

Microsoft’s Vision

Microsoft’s subsequent vision that addresses all of these learnings is three-fold.

  1. Firstly, Microsoft wants to fill the gap between the high-complexity barrier of entry of BizTalk Server and the low-complexity barrier of entry of Web Apps (Azure Websites). The idea is to democratise integration - by making it simple, easy, and approachable by the masses of developers, not just specialised experts.
  2. Next, Microsoft importantly will complement this ease of use with a heavy focus on the enterprise, aiming to be the Integration Platform-as-a-Service (iPaaS) leader, providing 24/7, robust, resilient services with all the solid functionality of BizTalk Server.
  3. Lastly, Microsoft realise that their offering should provide an extensible foundation that the community and partners can enrich, by creating a public marketplace that supports plugins and monetisation.

Microsoft’s Roadmap

Microsoft has a holistic roadmap that aims to:

  • Empower business employees with insights (analytics and statistics) to help make solid decisions,
  • Enable the transformation of businesses with the agility to develop solutions faster; and
  • Enable businesses to engage their customers by connecting with any device at Internet scale. Hence, the App Service cometh.

The App Service

App Service is the resulting integrated offering from Microsoft that enables the development of rich, engaging and intelligent applications that scale as your business grows.

For more information about App Service, please see http://azure.microsoft.com/en-us/services/app-service/.


Continued in Part 2 » BizTalk Summit 2015 in London - Part 2 - Turning of the Tide?