The "Reverse Waterfall" Software Development Methodology

   Submit to Reddit      
  

The facetious, arguably provocative and derogatory terminology "Reverse Waterfall" was coined to describe software development practices that begin the Implementation phase before sufficient requirements gathering, business analysis and design are performed. This referred to Implementation phase specifically consists of the developers performing coding and unit testing.

Please note that this terminology is not meant to be a strict reversal of all the sequential steps in the typical Waterfall software development methodology. Additionally, this terminology may be used describe any project's or team's software development practice regardless of the purported methodology being used by them - including but not limited to both Agile and Waterfall.

Taking a step back for a second, in traditional Waterfall methodologies, all possible requirements are [attempted to be] gathered at the start of project. This is then followed by phases of design, implementation and then testing. In Agile methodologies alternatively, when a team starts working on a specific story or task, just enough requirements are gathered, and analysis and design performed. Agile methodologies specifically welcome changing requirements, even late in the development cycle.

Regardless of the approach however, and regardless of changes to requirements at a later stage, in order for a working and valuable solution to be developed and delivered (incrementally or not), the following are prerequisites for coding:

  1. A minimum set of core business requirements must be known;
  2. A minimum amount of business analysis must have been performed to more finely specify required behaviour and functionality;
  3. A minimum amount of design (both architecture and component design) must have been performed.

If a team starts the Implementation phase without these prerequisites in place, coding the solution is like building a house without a blueprint and hoping it will remain standing.

The number of businesses that are willing to waste millions of dollars building straw houses quite frankly amazes me. I have seen this happen a number of times in the past when upper management sets a strategic goal and plucks an arbitrary date for software delivery out of the air without having appropriately considered and put in place the following:

  • clear core business requirements;
  • clear current and future (re-engineered) business processes;
  • business event triggers identified in the business processes;
  • an information architecture;
  • a solution architecture reviewed with a software development team that is an expert in the type of technical solution required, or the team that is likely to actually develop the solution;
  • estimations reviewed with a software development team that is an expert in the type of technical solution required, or the team that is likely to actually develop the solution; and
  • dependencies assessed and planned;

Software development is not just about coding and testing. When a business sets a strategic goal, a holistic approach needs to be taken. Part of the overall solution may involve the development of software, and the ability to design and develop working and valuable software is entirely dependent on the business first getting their business artefacts in order - starting with their business processes, business events and information architecture.

If these business artefacts are not in place at the start of a software development project, the project will likely stall and in a negative workplace, the software development team will be blamed for the failure of a project. How ironic it is that the business and upper management actually doomed the software development project to failure even before it started!

Reverse Waterfall is madness, but in my experience is all too common in the industry.

Have you had similar experiences? Leave a comment...

Behaviour Specification Writing Methodology - Part 2

   Submit to Reddit      
  

Please note, as of August 2015, this article is superceded by the The Behaviour Specification Handbook

In my previous article, Behaviour Specification Writing Methodology - Part 1, I discussed the prerequisites for writing Behaviour Specifications / scenarios. This article elaborates on my tips and suggestions for the next stage - actually writing the Behaviour Specifications.

The Behaviour Specification Writing Meeting

After all the prerequisites have been satisfied (i.e. core business requirements are known and the initial, low-level technical design has been drafted), it is now time to write the Behaviour Specifications. Schedule a Behaviour Specification writing meeting!

Attendees

The attendees of the Behaviour Specification writing meeting should include a holistic cross-section of the development team, including all the actual people who will be developing and testing the given functionality. I will come back to this point later.

The types of roles that should attend the meeting include:

  • Technical Lead/Architect;
  • Business Analysts;
  • Test Analysts; and
  • Developers.

Actual business representatives may of course also attend and can be especially useful if the requirements are not as clear as originally thought coming into this meeting. Assuming however that the requirements are crystal clear, the business representative would only attend predominantly for their own benefit and understanding, or for team building reasons.

Background Context and Requirements Overview

Please encourage the meeting attendees to ask questions at any time, and note all relevant feedback.

Start the Behaviour Specification writing meeting with some background context about the User Story. A walk-through of the business process leading into this User Story and other relevant high-level information about the business is always useful to set the scene.

It is also beneficial at this stage to now explain and walk the attendees through the requirements.

Technical Design Overview

Now have a technical representative walk the attendees through the low-level, draft technical design. Referencing some of the previously mentioned requirements, the technical representative should highlight the initially proposed systems and components and their relevant interactions.

This is a fantastic opportunity for the entire team (yes, even the Business Analysts) to understand the technical design and lingo, and initially challenge the technical design.

Checkpoint

At this stage, let the attendees ask any more relevant questions and note all relevant feedback.

As necessary, evolve the technical design on the spot to cater for any issue that has been identified.

If there are significant issues with the requirements or technical design, then consider whether it is prudent to postpone the rest of this meeting.

What's Happening In Our Minds?

Throughout all of the above overviews and discussions, each attendee has now learned valuable implicit knowledge. This is the reason why the attendees should be the actual people who will do the development and testing - because throughout this meeting they are being transformed into the experts of this business functionality!

As the team next proceeds to write the behaviours of each component, multiple team members from different roles and perspectives are mentally testing each behaviour against their understanding of the solution - fantastically optimizing the feedback loop and enabling the team to avoid mistakes and reduce costly, wastful effort!

The process of writing the Behaviour Specifications is an important time where the attendees actively engage their mind and further collaborate and communicate. The result of the attendees actively engaging in the process is the growth of their own tacit knowledge and expertise. Again, this is the reason why the attendees should be the developers and testers of this functionality, because they become the experts who can most effectively and efficiently perform the implementation work.

To Gherkin or Not To Gherkin?

The Gherkin language is the most widely used and recognised business-readable domain-specific language (DSL) for specifying behaviours. Behaviours are defined within "Feature" files. Each Behaviour Specification is a "Scenario", which has a description, and then is further defined as executable steps, with each step beginning with word "Given", "When", "Then", "And" or "But". Depending on the platform, there are different tools for parsing and executing the Gherkin. For instance, there is Cucumber for Ruby, SpecFlow for .NET, jBehave for Java, Behat for PHP...

The new kid on the street is Gauge. In Gauge, behaviours are defined in "Specification" files that follow the Markdown syntax. Each Behaviour Specification has a heading, description and then is further defined as executable steps.

The main difference between the syntaxes used in Gauge and Gherkin is that the syntax used in Gauge is completely free and open. There is no "Given", "When", "Then" restriction like in Gherkin, and this flexibility can be a powerful draw card for some people.

However, with so much flexibility and power, there is the need for great responsibility and consistency - especially in an enterprise. In a medium or larger business or enterprise, consistency in language and style is important. If such a business were to use Gauge, the first thing the business should do, in my opinion, is to create a style guide to promote consistency between teams. And congratulations - now the syntax is restricted - and that flexibility is tamed.

In the end, we may as well have just used Gherkin. Sure, there are other cross-language features in Guage that may pique your interest, but for my Behaviour Specification writing methodology, I am sticking with Gherkin. The recommendations presented here can apply to the Guage syntax if you so desire.

Behaviour Specification Writing

Now that the requirements are understood by the team and there is general agreement that sufficient requirements are known and the draft technical design is sound enough, the process of writing the Behaviour Specifications can begin.

This behaviour writing process is best broken down into two phases - indeed, often it is worth working through these phases in different meetings, primarily to allow the attendees to have a rest in between:

  1. Identify and write all the Behaviour Specification descriptions; and
  2. Write the steps that would cause the desired behaviour to manifest.

All members of a software development team, including developers, analysts and stakeholders should be able to understand the Behaviour Specification descriptions and steps - because they should be written in commonly accepted business terminology. On a daily basis, the development team normally should be communicating with the rest of the business using accepted business terminology. Similarly the behaviours should be written in the same accepted business style of language (as much as possible). One result of this approach is the minimisation of the costs to train new staff.

This approach also promotes more consistent communication within the development and support teams - and even to the wider business itself. An example of communicating with a wider part of a business is a project team (the wider part) that engages an internal software development team. The project team certainly could be interested in the documented Behaviour Specifications of a system that the internal software development team authored. Whether other wider parts of a business would be interested is debatable, and I think it may only apply to businesses with quite a high degree of software development maturity.

As an overarching style guideline, I highly recommend that all wording used in a Behaviour Specification should only need to change if there is a change to the core business requirement - and specifically the wording should not need to be changed if there is a change to the implementation.

Finding the right words to use when writing a Behaviour Specification can be thought of sometimes as an "art" due to the delicate balance between not using language that is too technical or specific, and at the other end of the scale, using language that is too generic that the reader cannot grasp the essence or actual purpose of the behaviour. Finding this balance can be a challenge, especially for novices. The best advice I can give is to copy the style of existing behaviours that your team agrees are well-written.

Tip: Take samples of existing well-written Behaviour Specifications into each behaviour writing meeting and refer to them as necessary! It is often helpful to review these in the meeting to refresh everyone's memory.

Behaviour Identification and Descriptions

Take each system or component that is represented in the low-level technical design, one by one, and answer the following question:

"Within the context, boundary and responsibility of just that component, what are the behaviours that component should perform in order to fulfil all the requirements?"

Generally, each Behaviour Specification should be focussed individually on a single purpose or business rule. By doing this, each behaviour becomes a building block of tested functionality. When writing the next behaviour, you now have a "line of trust" that everything so far is working as specified, and you do not need to re-specify the behaviour or functionality of those building blocks.

End To End Behaviours

In addition to specifying the behaviours for an individual subsystem/component, you should also specify a small number of behaviours of end to end functionality (remember the Test Pyramid!). I suggest these end to end behaviours would be specified in their own feature file, separate from the feature file for the each component, since these behaviours span multiple subsystems/components.

Start With "Should"

Describe each behaviour specification / scenario in a sentence, beginning with the word "Should".

"Should" at the start of a sentence helps to focus both the author and reader clearly on the actual behaviour. Behaviours are described with verbs. The next word after "Should" is typically a verb... so this helps everyone reading the scenario description to quickly understand the scenario and how it may differ from others.

The behaviour specifications are both living, executable documentation as well as being classified as test assets. The behaviours also are the specification that developers should use when they write code. The only code that should be written by developers is code to satisfy a behaviour specification. No behaviour specification - no code. Importantly, behaviour specifications are NOT part of a formal, legal requirements document.

Each behaviour specification / scenario description is similar to a description of a system's capability. A system's capability is not described in terms of "it must/shall/will do XYZ" - that is how a "requirement" might be worded. Behaviour specifications are determined after the business requirements have been identified and agreed upon by stakeholders.

The scenario description is a specification of a system's expected behaviour - how the system should behave, based on all current knowledge and understanding. Dan North, the originator of BDD, in Introducing BDD eloquently describes more about the usage of the word "should":

"A more subtle aspect of the word should becomes apparent when compared with the more formal alternatives of will or shall. Should implicitly allows you to challenge the premise of the test: "Should it? Really?" This makes it easier to decide whether a test is failing due to a bug you have introduced or simply because your previous assumptions about the system's behaviour are now incorrect."

Use Present Tense

The scenario description makes more logical sense when written in the present tense, because it is the describing the most up-to-date, current, at the present time, expected behaviour.

Avoid Generic Verbs

Avoid the word "handle" or any other verb that is generic and hides the actual behaviour.

For instance, let us pretend that someone wrote the scenario description: "Should handle the situation when the total amount is not equal to the sum of all item amounts".

The above scenario description has both positive and negative points. On the positive side, the "when" part is descriptive and is an easily understandable, differentiating condition for the scenario. On the negative side however, "handle" is not actually describing the behaviour that occurs under in that situation. What is actually meant by "handle"?

A better alternative is to be more specific about the behaviour. For instance, "Should error when the total amount is not equal to the sum of all item amounts". Using "error" as a verb instead of "handle" clearly describes the actual type of behaviour that should be performed.

Positive and Negative Behaviours

All good test suites cover both positive / happy path and negative / sad path scenarios. Behaviour specifications are the same.

In my experience, prefixing the scenario description (even before "Should") with "Positive - " or "Negative - " allows someone reading the behaviour specification or test report to quickly and clearly determine whether the behaviour is focussed on the happy or sad path.

Primary / Business and Secondary / System Behaviours

It can be useful for the purpose of general understanding to distinguish between two different types of behaviours:

  • Primary / Business Behaviours; and
  • Secondary / System Behaviours.

A Primary / Business-based Behaviour is a behaviour that has been defined as a direct result of trying to fulfil a core business requirement.

A Secondary / System-based Behaviour alternatively is a lower-level behaviour needed by a system in order to enable or support the functionality of a Primary / Business Behaviour. A Secondary Behaviour is in no way less important than a Primary Behaviour - in fact, from one perspective, one can think that a Secondary Behaviour is more important because a specific Primary Behaviour is dependent upon and cannot function correctly without the Secondary Behaviour.

Examples of Secondary / System Behaviours include:

  • Exception handling
  • Transaction concerns
  • Data storage and retrieval
  • Performance concerns
  • Auditing functionality

Unique Identifiers for Scenarios

When developers write new tests, modify existing tests and/or then need to run tests, it can be very useful to have a mechanism to easily identify and pick a specific scenario/test out of a list of many scenarios/tests. For instance, when developers are using their favourite IDE's Test Runner, if they want to find a specific scenario/test to run (even after using search or filtering on @tags), a unique identifier for each scenario can really help a human to visually find a specific scenario and select it more quickly.

I recommend including an incrementing number into each scenario description, with the number starting at one in each feature file.

In addition, prepending the feature name (or a shortened, unique version of it) helps to make a behaviour specification uniquely identifiable across multiple feature files.

Including all of the above information in each behaviour specification description makes life for all team members significantly easier when someone needs to reference a specific scenario in a communication, or navigate through a list of behaviours outside of the context of a feature file. For the small amount of discipline required, you will be cursing yourself and other team members if it isn't done and if you are the poor soul that now has to find a needle in a haystack. Just do it - it'll be worth it!

The template I find most useful for a scenario description is:

[FeatureName]-[Primary/Secondary]-01 - [Positive/Negative] - Should blah.

I acknowledge that at times this can become wordy, but in my experience the benefits of clarity with this approach have always trumped that small annoyance.

Feature Files

Zooming out from scenario descriptions within a feature file, let us now focus on other aspects related to the feature file.

File-System Folder Structure

I recommend grouping various feature files as appropriate into different file-system folders that follow a logical hierarchy. The top level of the hierarchy could be the name of an architectural capability or domain (a high-level grouping mechanism or category).

Subfolders could then be used for other logical groupings that make sense, and their names should include a verb (we are after all describing categories of behaviours). The subfolder / behaviour category name may describe the set of processes, actions, behaviour and activities that take place, and should be in business language - not implementation or system specific language.

Within the lowest-level subfolder are the feature files.

Below is an example folder structure:

Purchasing (top-level folder, also the name of a domain)
|
|---- ProcessingOrders (subfolder, a general category of behaviours)
        |
        |---- Shopping (subfolder, a general category of behaviours)
                |---- ManageShoppingCart.feature
                |---- CalculateShippingCosts.feature
                |---- ProcessPayments.feature
                |---- PlaceOrder.feature
        |
        |---- FulfillingOrders (subfolder, a general category of behaviours)
                |---- PickOrderItems.feature
                |---- ShipOrderItems.feature

The feature files for different subsystems typically will live in different code branches or projects within your source control repository. For consistency reasons, use the same folder structure for feature files across subsystems.

I also highly recommend not having whitespace in a folder or feature file name, as this can cause problems depending on the technology or development tools you are using.

Feature Name

At the top of a feature file is the feature name. I recommend making the folder hierarchy and grouping names part of the feature file's feature name value.

Each behaviour specification within a feature file is written from the context of a system, subsystem or component. Including the name of that subsystem or component in the description is also very using information, for the sake of clarity. Trickily, you may find that you have the exact same behaviour description and Gherkin in more than one component (for example, in both the User Interface and the back-end service layer), but the behaviour actually has a slightly different meaning in each because it must be understood and considered from within the context and perspective of that subsystem/component.

The template I find most useful for the name of a feature at the top of a feature file is:

Feature: [Hierarchy-LogicalGrouping]-[FileName]-[Subsystem/Component]

Here is an example feature name, for a User Interface component:

Feature: Purchasing-ProcessingOrders-Shopping-ManageShoppingCart-UI

And here is the same example feature, this time for the corresponding component in the back-end of a system:

Feature: Purchasing-ProcessingOrders-Shopping-ManageShoppingCart-ServiceLayer

In this example above, "Purchasing-ProcessingOrders-Shopping-ManageShoppingCart-ServiceLayer" is the "feature name" that you would include at the start of each behaviour specification description.


In my next article, I will discuss tips for writing the actual Given/When/Then steps of the Gherkin and about implementing the behaviours as automated tests.

Behaviour Specification Writing Methodology - Part 1

   Submit to Reddit      
  

Please note, as of August 2015, this article is superceded by the The Behaviour Specification Handbook

[Updated in April 2015]

Overview

As a long-time advocate of Behaviour-Driven Development (BDD), and being the technical lead and coach of BDD enterprise software development teams for more than 5 years, I have tried many different variations of the process of BDD and the writing and implementation of scenarios. The following is my suggested, prescriptive methodology for writing Behaviour Specifications (also known as Scenarios in the Gherkin language), including tips and suggestions that have repeatedly proven to me to be worthwhile practices.

Why BDD?

There are many articles that discuss the beginnings and benefits of BDD, so instead repeating similar information, here are some good starting materials:

Prerequisites

There are two important prerequisite inputs to a Behaviour Specification writing session - the requirements and an initial low-level technical design. I cannot stress enough how important these prerequisites are to hosting a successful Behaviour Specification writing session!

Requirement Analysis Complete

Before your team meets to determine the Behaviour Specifications (typically for a User Story, if your team is following an Agile methodology), ensure that the requirements analysis is as complete as possible. The nemesis of a Behaviour Specification writing session is unknown, unclear or incomplete requirements. Unclear requirements are the primary cause of elongated discussions, frustration and confusion in a Behaviour Specification writing meeting.

During the behaviour writing meeting, if missing requirements are identified, I recommend that you simply make a note of them for later and carry on - if possible. On the other hand, if there is a critical requirement missing, it is wiser to postpone the meeting until the relevant details of the requirement can be determined.

If a team ever walks away from a Behaviour Specification writing session exhausted or thinking that the process is difficult, I can almost guarantee that there was a lack of requirements analysis performed before the meeting. The requirements gathering, analysis and clarification is the difficult work that requires and causes the most discussion, time, effort and agreement. Requirements analysis is thus defined as a prerequisite that should remain a very separate process from behaviour writing.

I have witnessed many Behaviour Specification writing sessions turn into requirements analysis workshops. This in itself is not a bad thing - indeed the Behaviour Specification writing meeting has just succeeded in facilitating, identifying and initiating other important discussions. However, if this happens in your meetings, please be careful to clearly communicate to your team that the Behaviour Specifications have been postponed and the team instead is now performing more requirements analysis. If your boss thinks the team has been "doing Behaviour Specifications" for days with none of the expected outputs produced because all the effort has been requirements analysis, then your boss is not receiving accurate information for tracking and planning purposes. It is unfair to tarnish the name of and blame the process of BDD due to an unfulfilled prerequisite. Additionally, it is counter-productive for stakeholders to start questioning the process of BDD because they incorrectly heard that BDD takes a long time and is difficult - when the culprit really was requirements analysis.

Core Business Requirements

There are different types and ways of describing requirements. For the purpose of the Behaviour Specification writing meeting, it is most useful to have the "core business requirements" understood and documented before the behaviour writing meeting.

A "core business requirement" has a clear purpose and business value, and it explicitly does not contain a description or specification of the implementation details.

If a "requirement" includes implementation detail, then there is the awkward situation where non-technical business colleagues are effectively dictating the implementation design to the technical experts. The technical folk however may think of a better implementation option, if they were empowered and given the opportunity.

Sometimes the "core" requirements are already known, but many times the author of a requirement has innocently encoded implementation-specific details within the requirement. A common example is with User Interface requirements. The User Interface requirement may, for instance, specify the need for a new textbox on a screen. That however is not a "core business requirement" because both "textbox" and "screen" are implementation details.

It is more beneficial to take these implementation-specific requirements and convert them back to "core business requirements". The "5 Whys" principle can be used to find the "core" or root requirement. By iteratively asking "Why?" or "What is the real underlying business need for doing XYZ?" the "core business requirement" can be uncovered.

Below is a contrived example.

Let's assume that the documented requirement is: "Display a new textbox on screen XYZ so we can enter data for ABC".

Why?
"Because we want to enter data for ABC and screen XYZ has related stuff on it."

Why do you need the data for ABC?
"We need that data for ABC to calculate MNOP."

Why do you need to calculate MNOP?
"To correctly invoice the customer."

With this understanding we can now state that the "core business requirement" is: "To receive information about ABC so that the customer can be invoiced correctly".

It is this type of business language that I highly recommend be used in the Behaviour Specifications / scenarios - not implementation-specific language about a textbox!

I highly recommend that, as overarching style guideline, all wording that is used in a Behaviour Specification should only need to change if there is a change to the "core business requirement" - and should not need to be changed if there is a change to the technical implementation. If you follow this recommendation, your Behaviour Specifications will live and stand strong through the passage of time.

Draft Low-Level Technical Design Complete

After sufficient requirements have been gathered, the solution/technical architects, designers and gurus in the team should have met, discussed and sketched out an initial, draft version of a low-level technical design that identifies the likely systems and components that will be involved in the story.

During the Behaviour Specification writing meeting, the team will focus on defining the behaviours of each of the components that were identified in this low-level technical design.


In Behaviour Specification Writing Methodology - Part 2, I discuss how to successfully run a Behaviour Specification writing meeting.

How To Easily Call WCF Services Properly

   Submit to Reddit      
  

Please note: this article has been superceded by the documentation for the ChannelAdam WCF Library.

Background

In my previous article, How To Call WCF Services Properly, I researched and highlighted the typical problems with using WCF clients (such as memory and connection leaks), and concluded that the most correct solution for consuming WCF services would:

  • Perform the Close/Abort pattern without a race condition;
  • Handle the situation when the service operation throws exceptions;
  • Handle the situations when both the Close and Abort methods throw exceptions; and
  • Handle asynchronous exceptions such as the ThreadAbortException.

And below was my sample code for correctly using a WCF client, performing exception handling and correctly performing the Close/Abort pattern.

SampleServiceClient client = null;

try
{
    client = new SampleServiceClient();

    var response = client.SampleOperation(1234);

    // Do some business logic
}
catch (FaultException<MyCustomException>)
{
    // Do some business logic for this SOAP Fault Exception
}
catch (FaultException)
{
    // Do some business logic for this SOAP Fault Exception
}
catch (CommunicationException)
{
    // Catch this expected exception so it is not propagated further.
    // Perhaps write this exception out to log file for gathering statistics...
}
catch (TimeoutException)
{
    // Catch this expected exception so it is not propagated further.
    // Perhaps write this exception out to log file for gathering statistics...
}
catch (Exception)
{
    // An unexpected exception that we don't know how to handle.
    // Perhaps write this exception out to log file for support purposes...
    throw;
}
finally
{
    // This will:
    // - be executed if any exception was thrown above in the 'try' (including ThreadAbortException); and
    // - ensure that CloseOrAbortServiceChannel() itself will not be interrupted by a ThreadAbortException
    //   (since it is executing from within a 'finally' block)
    CloseOrAbortServiceChannel(client);

    // Unreference the client
    client = null;
}


private void CloseOrAbortServiceChannel(ICommunicationObject communicationObject)
{
    bool isClosed = false;

    if (communicationObject == null || communicationObject.State == CommunicationState.Closed)
    {
        return;
    }

    try 
    {
        if (communicationObject.State != CommunicationState.Faulted)
        {
            communicationObject.Close();
            isClosed = true;
        }
    }
    catch (CommunicationException)
    {
        // Catch this expected exception so it is not propagated further.
        // Perhaps write this exception out to log file for gathering statistics...
    }
    catch (TimeoutException)
    {
        // Catch this expected exception so it is not propagated further.
        // Perhaps write this exception out to log file for gathering statistics...
    }
    catch (Exception)
    {
        // An unexpected exception that we don't know how to handle.
        // Perhaps write this exception out to log file for support purposes...
        throw;
    }
    finally
    {
        // If State was Faulted or any exception occurred while doing the Close(), then do an Abort()
        if (!isClosed)
        {
            AbortServiceChannel(communicationObject);
        }
    }
}


private static void AbortServiceChannel(ICommunicationObject communicationObject)
{
    try
    {
        communicationObject.Abort();
    }
    catch (Exception)
    {
        // An unexpected exception that we don't know how to handle.
        // If we are in this situation:
        // - we should NOT retry the Abort() because it has already failed and there is nothing to suggest it could be successful next time
        // - the abort may have partially succeeded
        // - the actual service call may have been successful
        //
        // The only thing we can do is hope that the channel's resources have been released.
        // Do not rethrow this exception because the actual service operation call might have succeeded
        // and an exception closing the channel should not stop the client doing whatever it does next.
        //
        // Perhaps write this exception out to log file for gathering statistics and support purposes...
    }
}

Unfortunately, due to the way Microsoft designed WCF, if you want to use a WCF service client cleanly without connection or memory leaks, then above is the type of extensive code that must be written for every service operation call.

I then promised to share an even better way to achieve this - by using some programming tricks to provide a nice, clean API for working with WCF clients, that significantly reduces the amount of that code that developers have to write.

What Is The Easy, Clean and Robust Solution?

Well, here it is! Drum roll please...

After more than a month of developing late into the evening and early morning, allow me to introduce the ChannelAdam WCF Library - an open source (Apache License 2.0) library that allows you to forget about the complex subtleties of WCF clients, channels and communication states, and lets you just use your service client (very similar to how you probably are doing it now - but without the leaks!).

When you use the ChannelAdam WCF Library's ServiceConsumer, you know that you can keep calling service operations through the ServiceConsumer and it will 'just work' and not leak connections or memory - no matter what happens!

The ChannelAdam WCF Library's ServiceConsumer hides the inner complexity of service channels. For example, if a service channel / communication object / service client moves into a Fault state, the ServiceConsumer will correctly perform the Close/Abort pattern and dispose of the channel. The next time a request is made through the ServiceConsumer, a new channel is created under the covers to seamlessly process the request.

Where Do I Get The Library?

You can install the ChannelAdam.Wcf NuGet package from within Visual Studio, or find it on the NuGet.org gallery at
https://www.nuget.org/packages/ChannelAdam.Wcf.

If you are keen to see some of the programming tricks and how it works, or possibly contribute features to it, the source code for the ChannelAdam WCF Library is on CodePlex at https://channeladamwcf.codeplex.com.

Show Me The Basics! How Do I Use It?

Let's assume for the duration of this article that there is a web service named FakeService and in Visual Studio we have generated a service reference and interface to FakeService, and there is now an IFakeService interface and a FakeServiceClient class to use. That is all normal and basic Visual Studio functionality.

Let's also assume that the service has the following operation signature:

int AddIntegers(int first, int second);

Create a ServiceConsumer with the ServiceConsumerFactory

The first thing you need to understand is that you consume your service operations through the usage of the ChannelAdam.ServiceModel.ServiceConsumer class.

The second thing you need to understand is that the best way to create the ServiceConsumer is by using the Create method on the ChannelAdam.ServiceModel.ServiceConsumerFactory.

In order to use the ServiceConsumerFactory.Create method, simply pass in as a parameter a factory method for creating the service client.

For example:

IServiceConsumer<IFakeService> service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()));

The ServiceConsumerFactory.Create method returns a IServiceConsumer<TServiceInterface> type. In all further examples, I will use the var keyword instead - such as:

var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()));

Alternatively, you can pass in as a string parameter the endpoint configuration name from the service client section of your application's .config file. For example:

var service = ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService"));

Four Ways to Clean Up the ServiceConsumer

There are four ways in which the ServiceConsumer can be cleaned up:

  1. Explicitly with the Close method;
  2. Explicitly with the Dispose method;
  3. Implicitly via the Using statement; or
  4. Automatically by the Garbage Collector.

The Close Method

The ServiceConsumer and underlying channel can be closed immediately by calling the Close method on the ServiceConsumer instance.

service.Close();

If you want to, you still can immediately call another service operation via that same ServiceConsumer instance, as under the covers, a new service channel will be created seamlessly and automatically.

The Dispose Method

The ServiceConsumer and underlying channel can be disposed of immediately by calling the Dispose method on the ServiceConsumer instance.

service.Dispose();

As per standard IDisposable concepts, you should not attempt to reuse the ServiceConsumer after disposing it.

The Using Statement

Instead of calling the Close or Dispose methods directly, you can use the Using statement on the ServiceConsumer because it is a well behaved implementation of IDisposable (unlike Microsoft's service clients!).

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    ...
}

Garbage Collection

If you happen to accidently forget to wrap your ServiceConsumer instance in a Using statement or call the Close method, rest assured that the ServiceConsumer will still be cleaned up! When the ServiceConsumer instance goes out of scope, the .NET Garbage Collector will [eventually] kick in and call its destructor/finalise method, thus correctly cleaning up the service channel and preventing any leaks.

Since there is an indeterminate amount of time before the Garbage Collector kicks in, I would recommend the Using statement approach, as it minimises the number of open channels and other resources currently in use, over the lifetime of your application.

How To Call a Service Operation

There are two ways in which to call a service operation via the ServiceConsumer:

  1. Via the Operations property; or
  2. Via the Consume method.

The Operations Property

The Operations property exposes the interface of your service. It can be used just like the normal service client that you already would be familiar with.

Below is an example of how to call the AddIntegers operation via the Operations property.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    try
    {
        int actualValue = service.Operations.AddIntegers(1, 1); 
        Assert.AreEqual(2, actualValue);
    }
    ...
}

And below is an example of how to do some exception handling.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    try
    {
        int actualValue = service.Operations.AddIntegers(1, 1); 
        Assert.AreEqual(2, actualValue);    
    }
    catch (FaultException<MyCustomException> fe)
    {
        Console.WriteLine("Service operation threw a custom fault: " + fe.ToString());
    }
    catch (FaultException fe)
    {
        Console.WriteLine("Service operation threw a fault: " + fe.ToString());
    }
    catch (Exception ex)
    {
        Console.WriteLine("An unexpected error occurred while calling the service operation: " + ex.ToString());
    }
}

You may notice that there is now no need to catch a CommunicationException or TimeOutException and try to perform the Close/Abort pattern, as the underlying channel is dealt with automatically for you by the ServiceConsumer.

The Consume Method

As an alternative to the Operations property, you can call a service operation via the Consume method on the ServiceConsumer instance.

The Consume method returns an instance of IOperationResult<TReturnTypeOfYourOperation> which wraps the result of the operation. This interface has the following helpful properties:

  • TReturnTypeOfYourOperation Value - which gives you access to the return value from the operation;
  • bool HasException - whether there was an exception thrown by the service operation;
  • bool HasNoException - whether there was not an exception thrown by the service operation;
  • bool HasFaultException - whether an exception of type FaultException was thrown by the service operation;
  • bool HasFaultException<OfThisType> - whether an exception of FaultException<OfThisType> was thrown by the service operation; and
  • Exception Exception - the exception that was thrown by the service operation.

Below is an example of how to call the AddIntegers operation via the Consume method.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    IOperationResult<int> result = service.Consume(operation => operation.AddIntegers(1, 1));

    if (result.HasNoException)
    {
        Assert.AreEqual(2, result.Value);
    }
    ...
}

And below is an example of how to do some exception handling with IOperationResult.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    var result = service.Consume(operation => operation.AddIntegers(1, 1));

    if (result.HasNoException)
    {
        int actualValue = result.Value;
        Assert.AreEqual(2, actualValue);
    }
    else
    {
        if (result.HasFaultExceptionOfType<MyCustomException>())
        {
            Console.WriteLine("Service operation threw a custom fault: " + result.Exception.ToString());
        }
        else if (result.HasFaultException)
        {
            Console.WriteLine("Service operation threw a fault: " + result.Exception.ToString());
        }
        else
        {
            Console.WriteLine("An unexpected error occurred while calling the service operation: " + result.Exception.ToString());
        }
    }
}

More Advanced Usage - With IoC Containers

It is easy to use the ServiceConsumerFactory from Inversion of Control (IoC) containers. Below are examples with three popular IoC containers.

Let's assume that there is the following class and we want to perform constructor injection with it.

public class ConstructorInjectedController
{
    public IServiceConsumer<IFakeService> FakeService { get; set; }

    /// <summary>
    /// Initializes a new instance of the <see cref="ConstructorInjectedController"/> class.
    /// </summary>
    /// <param name="fakeService">The fake service.</param>
    public ConstructorInjectedController(IServiceConsumer<IFakeService> fakeService)
    {
        this.FakeService = fakeService;
    }
}

AutoFac

Here is how to use the ServiceConsumerFactory with AutoFac.

var builder = new Autofac.ContainerBuilder();
builder.Register(c => ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService")).InstancePerDependency();
builder.RegisterType<ConstructorInjectedController>();

var autofacContainer = builder.Build();

var controller = autofacContainer.Resolve<ConstructorInjectedController>();

SimpleInjector

Here is how to use the ServiceConsumerFactory with SimpleInjector.

var simpleInjectorContainer = new SimpleInjector.Container();
simpleInjectorContainer.Register(() => ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService"), SimpleInjector.Lifestyle.Transient);
simpleInjectorContainer.Verify();

var controller = simpleInjectorContainer.GetInstance<ConstructorInjectedController>();

Unity

And, here is how to use the ServiceConsumerFactory with Unity.

var unityContainer = new UnityContainer();

unityContainer
    .RegisterType<IServiceConsumer<IFakeService>>(
        new TransientLifetimeManager(),
        new InjectionFactory(c => ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService")))

    //
    // OR
    //
    //.RegisterType<IServiceConsumer<IFakeService>>(
    //    new InjectionFactory(c => ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))

    // 
    // OR more 'correctly'
    //
    //.RegisterType<IFakeService, FakeServiceClient>(new TransientLifetimeManager())
    //.RegisterType<IServiceConsumer<IFakeService>>(
    //    new InjectionFactory(c =>
    //        ServiceConsumerFactory.Create<IFakeService>(() => (ICommunicationObject)c.Resolve<IFakeService>())))
    ;

var controller = unityContainer.Resolve<ConstructorInjectedController>();

But Wait, There's More - Channel Close Trigger Strategies

When an exception occurs while calling the service operation, the ServiceConsumer follows a specific strategy to determine whether the exception and/or current channel state is severe enough to necessitate and trigger the closing/aborting of the service channel.

These "Service Channel Close Trigger Strategies" are an implementation of the interface IServiceChannelCloseTriggerStrategy. There is one method in this interface:

bool ShouldCloseChannel(ICommunicationObject channel, Exception exception);

If the method ShouldCloseChannel returns true, then the ServiceConsumer will perform the Close/Abort pattern.

Out of the box, the DefaultServiceChannelCloseTriggerStrategy is used. This strategy is very simple, effective and safe. It triggers a close to occur if the exception is anything but a FaultException, regardless of the state of the service channel. It is highly probable that the channel is no longer usable in the case where there was any other type of exception.

If you find that this strategy does not suit your need, you can create your own class that implements that interface and specify your class as one of the overloads of the Create method on the ServiceConsumerFactory. Alternatively, you can directly set the property ChannelCloseTriggerStrategy on the ServiceConsumer instance itself.

But Wait, There's More - Exception Behaviour Strategies

As you know, the ServiceConsumer automatically handles the clean up of service channels. If an exception occurs at any point throughout the usage of the channel, the ServiceConsumer will catch it and close the channel if necessary. In some cases, for example, if an exception occurs while attempting to close the channel, the ServiceConsumer does not want to propagate the exception back to the calling code, as it is not the caller's concern. In order to be a good citizen and prevent the ServiceConsumer from swallowing exceptions, the concept of the "Exception Behaviour Strategy" was born.

Whenever an exception occurs throughout the lifetime of the ServiceConsumer, a corresponding method on the configured Exception Behaviour Strategy class is invoked, thus allowing the exception to be logged and statistics recorded for an organisation's support purposes.

Each Exception Behaviour Strategy class implements the interface IServiceConsumerExceptionBehaviourStrategy. This interface contains the following methods:

  • PerformFaultExceptionBehaviour - the behaviour to perform when a fault exception occurs while the service operation is called;
  • PerformCommunicationExceptionBehaviour - the behaviour to perform when a communication exception occurs while the service operation is called;
  • PerformTimeoutExceptionBehaviour - the behaviour to perform when a timeout exception occurs while the service operation is called;
  • PerformUnexpectedExceptionBehaviour - the behaviour to perform when an unexpected exception occurs while the service operation is called;
  • PerformCloseCommunicationExceptionBehaviour - the behaviour to perform when a communication exception occurs during a close;
  • PerformCloseTimeoutExceptionBehaviour - the behaviour to perform when a timeout exception occurs during a close;
  • PerformCloseUnexpectedExceptionBehaviour - the behaviour to perform when an unexpected exception occurs during a close;
  • PerformAbortExceptionBehaviour - the behaviour to perform when an exception occurs during an abort; and
  • PerformDestructorExceptionBehaviour - the behaviour to perform when an exception occurs during a destructor/Finalize method.

The ChannelAdam.Wcf Library comes with the following Exception Behaviour Strategies:

  • NullServiceConsumerExceptionBehaviourStrategy - which does not write out any exception details anywhere;
  • StandardOutServiceConsumerExceptionBehaviourStrategy - which writes out all exceptions to the Standard Error stream; and
  • StandardErrorServiceConsumerExceptionBehaviourStrategy - which writes out all exceptions to the Standard Out stream.

By default, the ServiceConsumerFactory configures each ServiceConsumer with the StandardErrorServiceConsumerExceptionBehaviourStrategy.

There are three ways to change the Exception Behaviour Strategy that is assigned when you create a ServiceConsumer:

  1. Specify a different Exception Behaviour Strategy on one of the overloads of the Create method on the ServiceConsumerFactory;
  2. Directly set the property ChannelCloseTriggerStrategy on the ServiceConsumer instance itself; or
  3. Change the default - in some bootstrap code, set the static property ServiceConsumerFactory.DefaultExceptionBehaviourStrategy to your desired strategy. From then on, that strategy will be used by default.

As I expect will be the case, many organisations will want to use a logging library and write out the exceptions to a log file somewhere. Easy - just implement the interface and set it as the default in some bootstrap code.

But Wait, There's More - Retry Capability for Handling Transient Faults

When you call your web services, do you currently handle transient errors that may occur (such as the network briefly dropping out), and have some automatic retry logic?

Well, the ChannelAdam WCF Library has some built-in support for the Microsoft Practices Transient Fault Handling Library - but only when you use the Consume method and not when you use the Operations property.

Naive Retry Logic with the Operations Property

Below is an example naive implementation of retry logic (without the help of any library) while using the Operations property.

int retryCount = 1;
Exception lastException = null;

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()))
{
    while (retryCount > 0)
    {
        Console.WriteLine("#### Retry count: " + retryCount);

        try
        {
            int actual = service.Operations.AddIntegers(1, 1);

            Console.WriteLine("Actual: " + actual);
            Assert.AreEqual(2, actual);

            return;
        }
        catch (FaultException fe)
        {
            lastException = fe;
            Console.WriteLine("Service operation threw a fault: " + fe.ToString());
        }
        catch (Exception ex)
        {
            lastException = ex;
            Console.WriteLine("Technical error occurred while calling the service operation: " + ex.ToString());
        }

        retryCount--;
    }
}

Assert.Fail("Service operation was not successfully called");

Unfortunately that code is lengthy and error prone.

Retry Logic with Operations Property and the Microsoft Library

The code above can be improved upon by using the Microsoft Practices Transient Fault Handling Library, as below.

You will notice in this example, I am using the ChannelAdam.ServiceModel.SoapFaultWebServiceTransientErrorDetectionStrategy to determine if there was a transient error or not. If the given exception was a FaultException, this strategy says that was not a transient error and therefore do not retry. If on the other hand any other type of exception occurs, then this strategy will tell the Microsoft library to retry according to the retry policy.

Exception lastException;

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()))
{
    try
    {
        int actual = 0;

        var retryStrategy = new Incremental(5, TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(2));
        var retryPolicy = new RetryPolicy<SoapFaultWebServiceTransientErrorDetectionStrategy>(retryStrategy);

        retryPolicy.ExecuteAction(() =>
        {
            actual = service.Operations.AddIntegers(1, 1);
        });

        Console.WriteLine("Actual: " + actual);
        Assert.AreEqual(2, actual);

        return;
    }
    catch (FaultException fe)
    {
        lastException = fe;
        Console.WriteLine("Service operation threw a fault: " + fe.ToString());
    }
    catch (Exception ex)
    {
        lastException = ex;
        Console.WriteLine("Technical error occurred while calling the service operation: " + ex.ToString());
    }

    Assert.Fail("Service operation was not successfully called");

Retry Logic with the Consume Method

For me, both code examples above that use the Operations property is clunky - which is why I prefer using the Consume method!

When using the Consume method, there are three ways to set the retry policy:

  1. Setting the static property DefaultRetryPolicy on the ServiceConsumerFactory - which will apply to all created instances;
  2. As an overload to the Create method on the ServiceConsumerFactory - which will apply just to that ServiceConsumer instance to be created; or
  3. As one of the overloads on the Consume method itself.

For example, the overload on the Consume method:

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()))
{
    var retryStrategy = new Incremental(5, TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(2));
    var retryPolicy = new RetryPolicy<SoapFaultWebServiceTransientErrorDetectionStrategy>(retryStrategy);

    var result = service.Consume(operation => operation.AddIntegers(1, 1), retryPolicy);

    if (result.HasNoException)
    {
        Console.WriteLine("Actual: " + result.Value);
        Assert.AreEqual(2, result.Value);
    }
    else
    {
        if (result.HasFaultException)
        {
            Console.WriteLine("Service operation threw a fault: " + result.Exception.ToString());
        }
        else if (result.HasException)
        {
            Console.WriteLine("An unexpected error occurred while calling the service operation: " + result.Exception.ToString());
        }

        Assert.Fail("Service operation was not successfully called");
    }
}

Perhaps you too may agree that using the retry strategy with the Consume method is simpler and cleaner, in comparison with the Operations property.

What Else Does the ServiceConsumer Do?

If you want to know at a deeper level how the ServiceConsumer and its friends behave, perhaps you would like to read the Behaviour Specifications feature file.

Conclusion

The ChannelAdam WCF Library is designed to make your life as a developer easier when consuming WCF services, by abstracting away the complexities of service channel state transitions and the odd behaviour of the Microsoft WCF libraries that do not comply with even their own design guidelines.

In doing this, the ChannelAdam ServiceConsumer once and for all prevents memory and connection leaks, gives developers the ability to plug in their own strategies for handling different situations (such as logging exceptions), and provides a built-in mechanism to retry service operation calls when transient faults occur.

I hope you like it, and I welcome your feedback!