Behaviour Specification Writing Methodology - Part 1

   Submit to Reddit      
  

Please note, as of August 2015, this article is superceded by the The Behaviour Specification Handbook

[Updated in April 2015]

Overview

As a long-time advocate of Behaviour-Driven Development (BDD), and being the technical lead and coach of BDD enterprise software development teams for more than 5 years, I have tried many different variations of the process of BDD and the writing and implementation of scenarios. The following is my suggested, prescriptive methodology for writing Behaviour Specifications (also known as Scenarios in the Gherkin language), including tips and suggestions that have repeatedly proven to me to be worthwhile practices.

Why BDD?

There are many articles that discuss the beginnings and benefits of BDD, so instead repeating similar information, here are some good starting materials:

Prerequisites

There are two important prerequisite inputs to a Behaviour Specification writing session - the requirements and an initial low-level technical design. I cannot stress enough how important these prerequisites are to hosting a successful Behaviour Specification writing session!

Requirement Analysis Complete

Before your team meets to determine the Behaviour Specifications (typically for a User Story, if your team is following an Agile methodology), ensure that the requirements analysis is as complete as possible. The nemesis of a Behaviour Specification writing session is unknown, unclear or incomplete requirements. Unclear requirements are the primary cause of elongated discussions, frustration and confusion in a Behaviour Specification writing meeting.

During the behaviour writing meeting, if missing requirements are identified, I recommend that you simply make a note of them for later and carry on - if possible. On the other hand, if there is a critical requirement missing, it is wiser to postpone the meeting until the relevant details of the requirement can be determined.

If a team ever walks away from a Behaviour Specification writing session exhausted or thinking that the process is difficult, I can almost guarantee that there was a lack of requirements analysis performed before the meeting. The requirements gathering, analysis and clarification is the difficult work that requires and causes the most discussion, time, effort and agreement. Requirements analysis is thus defined as a prerequisite that should remain a very separate process from behaviour writing.

I have witnessed many Behaviour Specification writing sessions turn into requirements analysis workshops. This in itself is not a bad thing - indeed the Behaviour Specification writing meeting has just succeeded in facilitating, identifying and initiating other important discussions. However, if this happens in your meetings, please be careful to clearly communicate to your team that the Behaviour Specifications have been postponed and the team instead is now performing more requirements analysis. If your boss thinks the team has been "doing Behaviour Specifications" for days with none of the expected outputs produced because all the effort has been requirements analysis, then your boss is not receiving accurate information for tracking and planning purposes. It is unfair to tarnish the name of and blame the process of BDD due to an unfulfilled prerequisite. Additionally, it is counter-productive for stakeholders to start questioning the process of BDD because they incorrectly heard that BDD takes a long time and is difficult - when the culprit really was requirements analysis.

Core Business Requirements

There are different types and ways of describing requirements. For the purpose of the Behaviour Specification writing meeting, it is most useful to have the "core business requirements" understood and documented before the behaviour writing meeting.

A "core business requirement" has a clear purpose and business value, and it explicitly does not contain a description or specification of the implementation details.

If a "requirement" includes implementation detail, then there is the awkward situation where non-technical business colleagues are effectively dictating the implementation design to the technical experts. The technical folk however may think of a better implementation option, if they were empowered and given the opportunity.

Sometimes the "core" requirements are already known, but many times the author of a requirement has innocently encoded implementation-specific details within the requirement. A common example is with User Interface requirements. The User Interface requirement may, for instance, specify the need for a new textbox on a screen. That however is not a "core business requirement" because both "textbox" and "screen" are implementation details.

It is more beneficial to take these implementation-specific requirements and convert them back to "core business requirements". The "5 Whys" principle can be used to find the "core" or root requirement. By iteratively asking "Why?" or "What is the real underlying business need for doing XYZ?" the "core business requirement" can be uncovered.

Below is a contrived example.

Let's assume that the documented requirement is: "Display a new textbox on screen XYZ so we can enter data for ABC".

Why?
"Because we want to enter data for ABC and screen XYZ has related stuff on it."

Why do you need the data for ABC?
"We need that data for ABC to calculate MNOP."

Why do you need to calculate MNOP?
"To correctly invoice the customer."

With this understanding we can now state that the "core business requirement" is: "To receive information about ABC so that the customer can be invoiced correctly".

It is this type of business language that I highly recommend be used in the Behaviour Specifications / scenarios - not implementation-specific language about a textbox!

I highly recommend that, as overarching style guideline, all wording that is used in a Behaviour Specification should only need to change if there is a change to the "core business requirement" - and should not need to be changed if there is a change to the technical implementation. If you follow this recommendation, your Behaviour Specifications will live and stand strong through the passage of time.

Draft Low-Level Technical Design Complete

After sufficient requirements have been gathered, the solution/technical architects, designers and gurus in the team should have met, discussed and sketched out an initial, draft version of a low-level technical design that identifies the likely systems and components that will be involved in the story.

During the Behaviour Specification writing meeting, the team will focus on defining the behaviours of each of the components that were identified in this low-level technical design.


In Behaviour Specification Writing Methodology - Part 2, I discuss how to successfully run a Behaviour Specification writing meeting.

How To Easily Call WCF Services Properly

   Submit to Reddit      
  

Please note: this article has been superceded by the documentation for the ChannelAdam WCF Library.

Background

In my previous article, How To Call WCF Services Properly, I researched and highlighted the typical problems with using WCF clients (such as memory and connection leaks), and concluded that the most correct solution for consuming WCF services would:

  • Perform the Close/Abort pattern without a race condition;
  • Handle the situation when the service operation throws exceptions;
  • Handle the situations when both the Close and Abort methods throw exceptions; and
  • Handle asynchronous exceptions such as the ThreadAbortException.

And below was my sample code for correctly using a WCF client, performing exception handling and correctly performing the Close/Abort pattern.

SampleServiceClient client = null;

try
{
    client = new SampleServiceClient();

    var response = client.SampleOperation(1234);

    // Do some business logic
}
catch (FaultException<MyCustomException>)
{
    // Do some business logic for this SOAP Fault Exception
}
catch (FaultException)
{
    // Do some business logic for this SOAP Fault Exception
}
catch (CommunicationException)
{
    // Catch this expected exception so it is not propagated further.
    // Perhaps write this exception out to log file for gathering statistics...
}
catch (TimeoutException)
{
    // Catch this expected exception so it is not propagated further.
    // Perhaps write this exception out to log file for gathering statistics...
}
catch (Exception)
{
    // An unexpected exception that we don't know how to handle.
    // Perhaps write this exception out to log file for support purposes...
    throw;
}
finally
{
    // This will:
    // - be executed if any exception was thrown above in the 'try' (including ThreadAbortException); and
    // - ensure that CloseOrAbortServiceChannel() itself will not be interrupted by a ThreadAbortException
    //   (since it is executing from within a 'finally' block)
    CloseOrAbortServiceChannel(client);

    // Unreference the client
    client = null;
}


private void CloseOrAbortServiceChannel(ICommunicationObject communicationObject)
{
    bool isClosed = false;

    if (communicationObject == null || communicationObject.State == CommunicationState.Closed)
    {
        return;
    }

    try 
    {
        if (communicationObject.State != CommunicationState.Faulted)
        {
            communicationObject.Close();
            isClosed = true;
        }
    }
    catch (CommunicationException)
    {
        // Catch this expected exception so it is not propagated further.
        // Perhaps write this exception out to log file for gathering statistics...
    }
    catch (TimeoutException)
    {
        // Catch this expected exception so it is not propagated further.
        // Perhaps write this exception out to log file for gathering statistics...
    }
    catch (Exception)
    {
        // An unexpected exception that we don't know how to handle.
        // Perhaps write this exception out to log file for support purposes...
        throw;
    }
    finally
    {
        // If State was Faulted or any exception occurred while doing the Close(), then do an Abort()
        if (!isClosed)
        {
            AbortServiceChannel(communicationObject);
        }
    }
}


private static void AbortServiceChannel(ICommunicationObject communicationObject)
{
    try
    {
        communicationObject.Abort();
    }
    catch (Exception)
    {
        // An unexpected exception that we don't know how to handle.
        // If we are in this situation:
        // - we should NOT retry the Abort() because it has already failed and there is nothing to suggest it could be successful next time
        // - the abort may have partially succeeded
        // - the actual service call may have been successful
        //
        // The only thing we can do is hope that the channel's resources have been released.
        // Do not rethrow this exception because the actual service operation call might have succeeded
        // and an exception closing the channel should not stop the client doing whatever it does next.
        //
        // Perhaps write this exception out to log file for gathering statistics and support purposes...
    }
}

Unfortunately, due to the way Microsoft designed WCF, if you want to use a WCF service client cleanly without connection or memory leaks, then above is the type of extensive code that must be written for every service operation call.

I then promised to share an even better way to achieve this - by using some programming tricks to provide a nice, clean API for working with WCF clients, that significantly reduces the amount of that code that developers have to write.

What Is The Easy, Clean and Robust Solution?

Well, here it is! Drum roll please...

After more than a month of developing late into the evening and early morning, allow me to introduce the ChannelAdam WCF Library - an open source (Apache License 2.0) library that allows you to forget about the complex subtleties of WCF clients, channels and communication states, and lets you just use your service client (very similar to how you probably are doing it now - but without the leaks!).

When you use the ChannelAdam WCF Library's ServiceConsumer, you know that you can keep calling service operations through the ServiceConsumer and it will 'just work' and not leak connections or memory - no matter what happens!

The ChannelAdam WCF Library's ServiceConsumer hides the inner complexity of service channels. For example, if a service channel / communication object / service client moves into a Fault state, the ServiceConsumer will correctly perform the Close/Abort pattern and dispose of the channel. The next time a request is made through the ServiceConsumer, a new channel is created under the covers to seamlessly process the request.

Where Do I Get The Library?

You can install the ChannelAdam.Wcf NuGet package from within Visual Studio, or find it on the NuGet.org gallery at
https://www.nuget.org/packages/ChannelAdam.Wcf.

If you are keen to see some of the programming tricks and how it works, or possibly contribute features to it, the source code for the ChannelAdam WCF Library is on CodePlex at https://channeladamwcf.codeplex.com.

Show Me The Basics! How Do I Use It?

Let's assume for the duration of this article that there is a web service named FakeService and in Visual Studio we have generated a service reference and interface to FakeService, and there is now an IFakeService interface and a FakeServiceClient class to use. That is all normal and basic Visual Studio functionality.

Let's also assume that the service has the following operation signature:

int AddIntegers(int first, int second);

Create a ServiceConsumer with the ServiceConsumerFactory

The first thing you need to understand is that you consume your service operations through the usage of the ChannelAdam.ServiceModel.ServiceConsumer class.

The second thing you need to understand is that the best way to create the ServiceConsumer is by using the Create method on the ChannelAdam.ServiceModel.ServiceConsumerFactory.

In order to use the ServiceConsumerFactory.Create method, simply pass in as a parameter a factory method for creating the service client.

For example:

IServiceConsumer<IFakeService> service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()));

The ServiceConsumerFactory.Create method returns a IServiceConsumer<TServiceInterface> type. In all further examples, I will use the var keyword instead - such as:

var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()));

Alternatively, you can pass in as a string parameter the endpoint configuration name from the service client section of your application's .config file. For example:

var service = ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService"));

Four Ways to Clean Up the ServiceConsumer

There are four ways in which the ServiceConsumer can be cleaned up:

  1. Explicitly with the Close method;
  2. Explicitly with the Dispose method;
  3. Implicitly via the Using statement; or
  4. Automatically by the Garbage Collector.

The Close Method

The ServiceConsumer and underlying channel can be closed immediately by calling the Close method on the ServiceConsumer instance.

service.Close();

If you want to, you still can immediately call another service operation via that same ServiceConsumer instance, as under the covers, a new service channel will be created seamlessly and automatically.

The Dispose Method

The ServiceConsumer and underlying channel can be disposed of immediately by calling the Dispose method on the ServiceConsumer instance.

service.Dispose();

As per standard IDisposable concepts, you should not attempt to reuse the ServiceConsumer after disposing it.

The Using Statement

Instead of calling the Close or Dispose methods directly, you can use the Using statement on the ServiceConsumer because it is a well behaved implementation of IDisposable (unlike Microsoft's service clients!).

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    ...
}

Garbage Collection

If you happen to accidently forget to wrap your ServiceConsumer instance in a Using statement or call the Close method, rest assured that the ServiceConsumer will still be cleaned up! When the ServiceConsumer instance goes out of scope, the .NET Garbage Collector will [eventually] kick in and call its destructor/finalise method, thus correctly cleaning up the service channel and preventing any leaks.

Since there is an indeterminate amount of time before the Garbage Collector kicks in, I would recommend the Using statement approach, as it minimises the number of open channels and other resources currently in use, over the lifetime of your application.

How To Call a Service Operation

There are two ways in which to call a service operation via the ServiceConsumer:

  1. Via the Operations property; or
  2. Via the Consume method.

The Operations Property

The Operations property exposes the interface of your service. It can be used just like the normal service client that you already would be familiar with.

Below is an example of how to call the AddIntegers operation via the Operations property.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    try
    {
        int actualValue = service.Operations.AddIntegers(1, 1); 
        Assert.AreEqual(2, actualValue);
    }
    ...
}

And below is an example of how to do some exception handling.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    try
    {
        int actualValue = service.Operations.AddIntegers(1, 1); 
        Assert.AreEqual(2, actualValue);    
    }
    catch (FaultException<MyCustomException> fe)
    {
        Console.WriteLine("Service operation threw a custom fault: " + fe.ToString());
    }
    catch (FaultException fe)
    {
        Console.WriteLine("Service operation threw a fault: " + fe.ToString());
    }
    catch (Exception ex)
    {
        Console.WriteLine("An unexpected error occurred while calling the service operation: " + ex.ToString());
    }
}

You may notice that there is now no need to catch a CommunicationException or TimeOutException and try to perform the Close/Abort pattern, as the underlying channel is dealt with automatically for you by the ServiceConsumer.

The Consume Method

As an alternative to the Operations property, you can call a service operation via the Consume method on the ServiceConsumer instance.

The Consume method returns an instance of IOperationResult<TReturnTypeOfYourOperation> which wraps the result of the operation. This interface has the following helpful properties:

  • TReturnTypeOfYourOperation Value - which gives you access to the return value from the operation;
  • bool HasException - whether there was an exception thrown by the service operation;
  • bool HasNoException - whether there was not an exception thrown by the service operation;
  • bool HasFaultException - whether an exception of type FaultException was thrown by the service operation;
  • bool HasFaultException<OfThisType> - whether an exception of FaultException<OfThisType> was thrown by the service operation; and
  • Exception Exception - the exception that was thrown by the service operation.

Below is an example of how to call the AddIntegers operation via the Consume method.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    IOperationResult<int> result = service.Consume(operation => operation.AddIntegers(1, 1));

    if (result.HasNoException)
    {
        Assert.AreEqual(2, result.Value);
    }
    ...
}

And below is an example of how to do some exception handling with IOperationResult.

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))
{
    var result = service.Consume(operation => operation.AddIntegers(1, 1));

    if (result.HasNoException)
    {
        int actualValue = result.Value;
        Assert.AreEqual(2, actualValue);
    }
    else
    {
        if (result.HasFaultExceptionOfType<MyCustomException>())
        {
            Console.WriteLine("Service operation threw a custom fault: " + result.Exception.ToString());
        }
        else if (result.HasFaultException)
        {
            Console.WriteLine("Service operation threw a fault: " + result.Exception.ToString());
        }
        else
        {
            Console.WriteLine("An unexpected error occurred while calling the service operation: " + result.Exception.ToString());
        }
    }
}

More Advanced Usage - With IoC Containers

It is easy to use the ServiceConsumerFactory from Inversion of Control (IoC) containers. Below are examples with three popular IoC containers.

Let's assume that there is the following class and we want to perform constructor injection with it.

public class ConstructorInjectedController
{
    public IServiceConsumer<IFakeService> FakeService { get; set; }

    /// <summary>
    /// Initializes a new instance of the <see cref="ConstructorInjectedController"/> class.
    /// </summary>
    /// <param name="fakeService">The fake service.</param>
    public ConstructorInjectedController(IServiceConsumer<IFakeService> fakeService)
    {
        this.FakeService = fakeService;
    }
}

AutoFac

Here is how to use the ServiceConsumerFactory with AutoFac.

var builder = new Autofac.ContainerBuilder();
builder.Register(c => ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService")).InstancePerDependency();
builder.RegisterType<ConstructorInjectedController>();

var autofacContainer = builder.Build();

var controller = autofacContainer.Resolve<ConstructorInjectedController>();

SimpleInjector

Here is how to use the ServiceConsumerFactory with SimpleInjector.

var simpleInjectorContainer = new SimpleInjector.Container();
simpleInjectorContainer.Register(() => ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService"), SimpleInjector.Lifestyle.Transient);
simpleInjectorContainer.Verify();

var controller = simpleInjectorContainer.GetInstance<ConstructorInjectedController>();

Unity

And, here is how to use the ServiceConsumerFactory with Unity.

var unityContainer = new UnityContainer();

unityContainer
    .RegisterType<IServiceConsumer<IFakeService>>(
        new TransientLifetimeManager(),
        new InjectionFactory(c => ServiceConsumerFactory.Create<IFakeService>("BasicHttpBinding_IFakeService")))

    //
    // OR
    //
    //.RegisterType<IServiceConsumer<IFakeService>>(
    //    new InjectionFactory(c => ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient())))

    // 
    // OR more 'correctly'
    //
    //.RegisterType<IFakeService, FakeServiceClient>(new TransientLifetimeManager())
    //.RegisterType<IServiceConsumer<IFakeService>>(
    //    new InjectionFactory(c =>
    //        ServiceConsumerFactory.Create<IFakeService>(() => (ICommunicationObject)c.Resolve<IFakeService>())))
    ;

var controller = unityContainer.Resolve<ConstructorInjectedController>();

But Wait, There's More - Channel Close Trigger Strategies

When an exception occurs while calling the service operation, the ServiceConsumer follows a specific strategy to determine whether the exception and/or current channel state is severe enough to necessitate and trigger the closing/aborting of the service channel.

These "Service Channel Close Trigger Strategies" are an implementation of the interface IServiceChannelCloseTriggerStrategy. There is one method in this interface:

bool ShouldCloseChannel(ICommunicationObject channel, Exception exception);

If the method ShouldCloseChannel returns true, then the ServiceConsumer will perform the Close/Abort pattern.

Out of the box, the DefaultServiceChannelCloseTriggerStrategy is used. This strategy is very simple, effective and safe. It triggers a close to occur if the exception is anything but a FaultException, regardless of the state of the service channel. It is highly probable that the channel is no longer usable in the case where there was any other type of exception.

If you find that this strategy does not suit your need, you can create your own class that implements that interface and specify your class as one of the overloads of the Create method on the ServiceConsumerFactory. Alternatively, you can directly set the property ChannelCloseTriggerStrategy on the ServiceConsumer instance itself.

But Wait, There's More - Exception Behaviour Strategies

As you know, the ServiceConsumer automatically handles the clean up of service channels. If an exception occurs at any point throughout the usage of the channel, the ServiceConsumer will catch it and close the channel if necessary. In some cases, for example, if an exception occurs while attempting to close the channel, the ServiceConsumer does not want to propagate the exception back to the calling code, as it is not the caller's concern. In order to be a good citizen and prevent the ServiceConsumer from swallowing exceptions, the concept of the "Exception Behaviour Strategy" was born.

Whenever an exception occurs throughout the lifetime of the ServiceConsumer, a corresponding method on the configured Exception Behaviour Strategy class is invoked, thus allowing the exception to be logged and statistics recorded for an organisation's support purposes.

Each Exception Behaviour Strategy class implements the interface IServiceConsumerExceptionBehaviourStrategy. This interface contains the following methods:

  • PerformFaultExceptionBehaviour - the behaviour to perform when a fault exception occurs while the service operation is called;
  • PerformCommunicationExceptionBehaviour - the behaviour to perform when a communication exception occurs while the service operation is called;
  • PerformTimeoutExceptionBehaviour - the behaviour to perform when a timeout exception occurs while the service operation is called;
  • PerformUnexpectedExceptionBehaviour - the behaviour to perform when an unexpected exception occurs while the service operation is called;
  • PerformCloseCommunicationExceptionBehaviour - the behaviour to perform when a communication exception occurs during a close;
  • PerformCloseTimeoutExceptionBehaviour - the behaviour to perform when a timeout exception occurs during a close;
  • PerformCloseUnexpectedExceptionBehaviour - the behaviour to perform when an unexpected exception occurs during a close;
  • PerformAbortExceptionBehaviour - the behaviour to perform when an exception occurs during an abort; and
  • PerformDestructorExceptionBehaviour - the behaviour to perform when an exception occurs during a destructor/Finalize method.

The ChannelAdam.Wcf Library comes with the following Exception Behaviour Strategies:

  • NullServiceConsumerExceptionBehaviourStrategy - which does not write out any exception details anywhere;
  • StandardOutServiceConsumerExceptionBehaviourStrategy - which writes out all exceptions to the Standard Error stream; and
  • StandardErrorServiceConsumerExceptionBehaviourStrategy - which writes out all exceptions to the Standard Out stream.

By default, the ServiceConsumerFactory configures each ServiceConsumer with the StandardErrorServiceConsumerExceptionBehaviourStrategy.

There are three ways to change the Exception Behaviour Strategy that is assigned when you create a ServiceConsumer:

  1. Specify a different Exception Behaviour Strategy on one of the overloads of the Create method on the ServiceConsumerFactory;
  2. Directly set the property ChannelCloseTriggerStrategy on the ServiceConsumer instance itself; or
  3. Change the default - in some bootstrap code, set the static property ServiceConsumerFactory.DefaultExceptionBehaviourStrategy to your desired strategy. From then on, that strategy will be used by default.

As I expect will be the case, many organisations will want to use a logging library and write out the exceptions to a log file somewhere. Easy - just implement the interface and set it as the default in some bootstrap code.

But Wait, There's More - Retry Capability for Handling Transient Faults

When you call your web services, do you currently handle transient errors that may occur (such as the network briefly dropping out), and have some automatic retry logic?

Well, the ChannelAdam WCF Library has some built-in support for the Microsoft Practices Transient Fault Handling Library - but only when you use the Consume method and not when you use the Operations property.

Naive Retry Logic with the Operations Property

Below is an example naive implementation of retry logic (without the help of any library) while using the Operations property.

int retryCount = 1;
Exception lastException = null;

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()))
{
    while (retryCount > 0)
    {
        Console.WriteLine("#### Retry count: " + retryCount);

        try
        {
            int actual = service.Operations.AddIntegers(1, 1);

            Console.WriteLine("Actual: " + actual);
            Assert.AreEqual(2, actual);

            return;
        }
        catch (FaultException fe)
        {
            lastException = fe;
            Console.WriteLine("Service operation threw a fault: " + fe.ToString());
        }
        catch (Exception ex)
        {
            lastException = ex;
            Console.WriteLine("Technical error occurred while calling the service operation: " + ex.ToString());
        }

        retryCount--;
    }
}

Assert.Fail("Service operation was not successfully called");

Unfortunately that code is lengthy and error prone.

Retry Logic with Operations Property and the Microsoft Library

The code above can be improved upon by using the Microsoft Practices Transient Fault Handling Library, as below.

You will notice in this example, I am using the ChannelAdam.ServiceModel.SoapFaultWebServiceTransientErrorDetectionStrategy to determine if there was a transient error or not. If the given exception was a FaultException, this strategy says that was not a transient error and therefore do not retry. If on the other hand any other type of exception occurs, then this strategy will tell the Microsoft library to retry according to the retry policy.

Exception lastException;

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()))
{
    try
    {
        int actual = 0;

        var retryStrategy = new Incremental(5, TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(2));
        var retryPolicy = new RetryPolicy<SoapFaultWebServiceTransientErrorDetectionStrategy>(retryStrategy);

        retryPolicy.ExecuteAction(() =>
        {
            actual = service.Operations.AddIntegers(1, 1);
        });

        Console.WriteLine("Actual: " + actual);
        Assert.AreEqual(2, actual);

        return;
    }
    catch (FaultException fe)
    {
        lastException = fe;
        Console.WriteLine("Service operation threw a fault: " + fe.ToString());
    }
    catch (Exception ex)
    {
        lastException = ex;
        Console.WriteLine("Technical error occurred while calling the service operation: " + ex.ToString());
    }

    Assert.Fail("Service operation was not successfully called");

Retry Logic with the Consume Method

For me, both code examples above that use the Operations property is clunky - which is why I prefer using the Consume method!

When using the Consume method, there are three ways to set the retry policy:

  1. Setting the static property DefaultRetryPolicy on the ServiceConsumerFactory - which will apply to all created instances;
  2. As an overload to the Create method on the ServiceConsumerFactory - which will apply just to that ServiceConsumer instance to be created; or
  3. As one of the overloads on the Consume method itself.

For example, the overload on the Consume method:

using (var service = ServiceConsumerFactory.Create<IFakeService>(() => new FakeServiceClient()))
{
    var retryStrategy = new Incremental(5, TimeSpan.FromSeconds(2), TimeSpan.FromSeconds(2));
    var retryPolicy = new RetryPolicy<SoapFaultWebServiceTransientErrorDetectionStrategy>(retryStrategy);

    var result = service.Consume(operation => operation.AddIntegers(1, 1), retryPolicy);

    if (result.HasNoException)
    {
        Console.WriteLine("Actual: " + result.Value);
        Assert.AreEqual(2, result.Value);
    }
    else
    {
        if (result.HasFaultException)
        {
            Console.WriteLine("Service operation threw a fault: " + result.Exception.ToString());
        }
        else if (result.HasException)
        {
            Console.WriteLine("An unexpected error occurred while calling the service operation: " + result.Exception.ToString());
        }

        Assert.Fail("Service operation was not successfully called");
    }
}

Perhaps you too may agree that using the retry strategy with the Consume method is simpler and cleaner, in comparison with the Operations property.

What Else Does the ServiceConsumer Do?

If you want to know at a deeper level how the ServiceConsumer and its friends behave, perhaps you would like to read the Behaviour Specifications feature file.

Conclusion

The ChannelAdam WCF Library is designed to make your life as a developer easier when consuming WCF services, by abstracting away the complexities of service channel state transitions and the odd behaviour of the Microsoft WCF libraries that do not comply with even their own design guidelines.

In doing this, the ChannelAdam ServiceConsumer once and for all prevents memory and connection leaks, gives developers the ability to plug in their own strategies for handling different situations (such as logging exceptions), and provides a built-in mechanism to retry service operation calls when transient faults occur.

I hope you like it, and I welcome your feedback!

How To Call WCF Services Properly

   Submit to Reddit      
  

No matter how many client sites I visit, invariably I stumble across Windows Communication Foundation (WCF) client-side code that leaks memory and resources. This can result in an application crashing or connection limits being reached resulting in further service calls being rejected. Not only can the client-side code cause problems on the client machine, but if connections are not correctly closed on the client then that can cause the server to hit its maximum connection limit and reject service calls from other clients as well!

These types of issues are caused by developers performing incorrect error handling and incorrect disposal of the client and connection. Microsoft really is to blame for starting this mess, as they created an API that does not conform to typical practices - or even their own guidelines! For instance, Microsoft's guideline is never to throw an exception from a Dispose method - but the WCF client code does exactly that. At the end of the day however, it is the responsibility of individual developers to be aware of the issues and design restrictions and create software that works properly.

In many cases, WCF client-side code resides in small or medium-sized user applications that do not get used hard or long enough to exhibit a noticeable degradation in system resources and upset users. But as previously stated, depending on the number of users, it still may be causing issues on servers.

When it goes wrong - for instance, on a server - there can be a spectacular flurry of activity! It can be amazing to see how quickly IT departments can move when a server hosting core business services starts failing due to custom server applications leaking resources and using many Gigabytes of memory. Unfortunately it takes such an event for many businesses to pay attention to the need for higher-quality software development practices and testing.

In the past I have seen proud companies who develop and sell custom n-tier products "work around" known problems in their proprietary code by recommending to their clients that they recycle the IIS Application Pools frequently and run an excessive numbers of servers. I never really appreciated that band-aid attitude towards software development practices or clients.

This series of articles will show you how to fix the root cause - by correctly calling WCF services, handling errors and disposing of WCF clients.

Update 2014-10-01: The focus of this article is not about the overall design of WCF. Other parts of WCF are designed very well - especially with regards to its extensible nature. This article is a narrow focus on the design decisions that cause the common issues I see too often on so many client sites.

The Problems

The "Using" Statement and Understanding States

One of the problems that Microsoft caused was by not following their own guideline of not throwing exceptions in the Dispose method. The using statement is an excellent and common pattern for automatically disposing IDisposable objects at the end of the code block. Avoiding Problems with the Using Statement describes how even though the WCF client is disposable, developers should NOT use the using statement with them.

The C# "using" statement results in a call to Dispose(). This is the same as Close(), which may throw exceptions when a network error occurs. Because the call to Dispose() happens implicitly at the closing brace of the "using" block, this source of exceptions is likely to go unnoticed both by people writing the code and reading the code. This represents a potential source of application errors.

Microsoft demonstrates in that article how to clean up "correctly" when an exception occurs.

try
{
    ...
    client.Close();
}
catch (CommunicationException e)
{
    ...
    client.Abort();
}
catch (TimeoutException e)
{
    ...
    client.Abort();
}
catch (Exception e)
{
    ...
    client.Abort();
    throw;
}

It is important to understand that if the client is in a State of Faulted, then the only action that should be taken by client code is the Abort method. As documented in Expected Exceptions the TimeoutException, CommunicationException and any derived class of CommunicationException are 'expected' exceptions from the WCF client.

If an expected exception occurs, the client may or may not be usable afterwards. To determine if the client is still usable, check that the State property is CommunicationState.Opened. If it is still opened, then it is still usable. Otherwise you should abort the client and release all references to it.

Caution: You may observe that clients that have a session are often no longer usable after an exception, and clients that do not have a session are often still usable after an exception. However, neither of these is guaranteed, so if you want to try to continue using the client after an exception your application should check the State property to verify the client is still opened.

Code that calls a client communication method must catch the TimeoutException and CommunicationException.

However, all this talk of checking for the State property is cautioned by Accessing Services Using a WCF Client:

Checking the value of the ICommunicationObject.State property is a race condition and is not recommended to determine whether to reuse or close a channel.

If you were to check the State property in order to determine whether to Abort or Close, depending on your approach there could be a race condition. For instance the following code could result in a race condition:

if (this.State == CommunicationState.Faulted) 
{ 
    this.Abort();
}
else 
{
    this.Close();
}

The race condition could occur because when the State property is checked to see if it is Faulted it might be Opened at that point in time. However, by the short time that Close method is reached, the State might now be Faulted, and when Close is called an exception would be thrown. This code alone is not sufficient.

The Communication State Enumeration documentation explains the meaning of each possible State:

The Closed state is equivalent to being disposed and the configuration of the object can still be inspected.

The Faulted state is used to indicate that the object has transitioned to a state where it can no longer be used. There are two primary scenarios where this can happen:

  • If the Open method fails for any reason, the object transitions to the faulted state.
  • If a session-based channel detects an error that it cannot recover from, it transitions to the faulted state. This can happen for instance if there is a protocol error (that is, it receives a protocol message at an invalid time) or if the remote endpoint aborts the session.

An object in the Faulted state is not closed and may be holding resources. The Abort method should be used to close an object that has faulted. If Close is called on an object in the Faulted state, a CommunicationObjectFaultedException is thrown because the object cannot be gracefully closed.

The article Understanding State Changes describes how the State property can transition to different states. This article expands upon and somewhat contradicts the previous by indicating that if the object is in the Faulted state then the Close method will call Abort for you and return.

The Close() method can be called at any state. It tries to close the object normally. If an error is encountered, it terminates the object. The method does nothing if the current state is Closing or Closed. Otherwise it sets the state to Closing. If the original state was Created, Opening or Faulted, it calls Abort().

I'm confused - how about you?

To remedy my confusion, I went to the source of truth - the code. The reference source for CommunicationObject shows us that the Close method handles being called from any State value, and that it will perform an Abort if required. In addition, if the State was Faulted, the Close method actually will throw a CommunicationObjectFaultedException.

...
               switch (originalState)
                {
                    case CommunicationState.Created:
                    case CommunicationState.Opening:
                    case CommunicationState.Faulted:
                        this.Abort();
                        if (originalState == CommunicationState.Faulted)
                        {
                            throw TraceUtility.ThrowHelperError(this.CreateFaultedException(), Guid.Empty, this);
                        }
                        break;
...

The article Understanding State Changes also explicitly states that the Abort method can throw exceptions.

The Abort() method does nothing if the current state is Closed or if the object has been terminated before (for example, possibly by having Abort() executing on another thread). Otherwise it sets the state to Closing and calls OnClosing() (which raises the Closing event), OnAbort(), and OnClosed() in that order (does not call OnClose because the object is being terminated, not closed). OnClosed() sets the state to Closed and raises the Closed event. If any of these throw an exception, it is re-thrown to the caller of Abort.

No sample code from Microsoft or anywhere else that I have seen handles the situation where the Abort method throws an exception.

Stack Overflow has an interesting thread about how to work around the using block issue and perform the close/abort pattern. Their top-voted solution for the close/abort pattern, to fix the race condition is:

bool success = false; 
try 
{
    if (State != CommunicationState.Faulted) 
    {
        Close(); 
        success = true;
    }
}
finally 
{ 
    if (!success) 
    {
        Abort();
    }
}

In this code, if the State is already Faulted or the execution of the Close method throws an exception (which is implicitly caught and ignored), then finally the Abort method will be called. That's pretty good. But as we now know, the Abort method can throw exceptions, and that code does not handle it.

Exception Catching Order

The article Sending and Receiving Faults shows us that we need to catch the exceptions in a specific order - especially in relation to the SOAP-based FaultException.

Because FaultException derives from FaultException, and FaultException derives from CommunicationException, it is important to catch these exceptions in the proper order. If, for example, you have a try/catch block in which you first catch CommunicationException, all specified and unspecified SOAP faults are handled there; any subsequent catch blocks to handle a custom FaultException exception are never invoked.

Remember that one operation can return any number of specified faults. Each fault is a unique type and must be handled separately.

Closing the channel can throw exceptions if the connection cannot be cleanly closed or is already closed, even if all the operations returned properly.

Typically, client object channels are closed in one of the following ways:

  • When the WCF client object is recycled.
  • When the client application calls ClientBase.Close.
  • When the client application calls ICommunicationObject.Close.
  • When the client application calls an operation that is a terminating operation for a session.

In all cases, closing the channel instructs the channel to begin closing any underlying channels that may be sending messages to support complex functionality at the application level. For example, when a contract requires sessions a binding attempts to establish a session by exchanging messages with the service channel until a session is established. When the channel is closed, the underlying session channel notifies the service that the session is terminated. In this case, if the channel has already aborted, closed, or is otherwise unusable (for example, when a network cable is unplugged), the client channel cannot inform the service channel that the session is terminated and an exception can result.

Abort the Channel If Necessary

Because closing the channel can also throw exceptions, then, it is recommended that in addition to catching fault exceptions in the correct order, it is important to abort the channel that was used in making the call in the catch block. If the fault conveys error information specific to an operation and it remains possible that others can use it, there is no need to abort the channel (although these cases are rare). In all other cases, it is recommended that you abort the channel. For a sample that demonstrates all of these points, see Expected Exceptions.

And here is the sample code from that article:

using System;
using System.ServiceModel;
using System.ServiceModel.Channels;
using Microsoft.WCF.Documentation;

public class Client
{
  public static void Main()
  {
    SampleServiceClient wcfClient = new SampleServiceClient();

    try
    {
      wcfClient.SampleMethod("hello");

      wcfClient.Close();
    }
    catch (TimeoutException timeProblem)
    {
      wcfClient.Abort();
    }
    catch (FaultException<MyCustomFault> myCustomFault)
    {
      wcfClient.Abort();
    }
    catch (FaultException<MyOtherCustomFault> myOtherCustomFault)
    {
      wcfClient.Abort();
    }
    catch (FaultException unknownFault)
    {
      wcfClient.Abort();
    }
    catch (CommunicationException commProblem)
    {
      wcfClient.Abort();
    }
  }
}

Note the following about this sample code:

  • The client is not closed or aborted if there is an unexpected exception; and
  • There is no concern about catching exceptions from the Abort method.

Other Exceptions

There is one more type of exception that never seems to be mentioned in sample code or in any literature I have seen related to WCF, and that is the ThreadAbortException.

When this exception is raised, the runtime executes all the finally blocks before ending the thread. Because the thread can do an unbounded computation in the finally blocks or call Thread.ResetAbort to cancel the abort, there is no guarantee that the thread will ever end.

The ThreadAbortException is a special exception that can occur asynchronously. If the WCF client is called from within a thread, and if the thread is aborted, then it might be prudent to clean up the client before the thread finishes.

The top-voted solution from Stack Overflow does partially and elegantly handle this situation, as well as the other asynchronous exceptions such as OutOfMemoryException and StackOverflowException.

Oh My!

So does all that sound complicated enough? No wonder so many don't get it right...

How To Do It Correctly?

In my opinion, the most correct solution would:

  • Perform the Close/Abort pattern without a race condition
  • Handle the situation when the service operation throws exceptions
  • Handle the situations when both the Close and Abort methods throw exceptions
  • Handle asynchronous exceptions such as the ThreadAbortException

Below is my proposed solution for correctly using a WCF client.

SampleServiceClient client = null;

try
{
    client = new SampleServiceClient();

    var response = client.SampleOperation(1234);

    // Do some business logic
}
catch (FaultException<MyCustomException>)
{
    // Do some business logic for this SOAP Fault Exception
}
catch (FaultException)
{
    // Do some business logic for this SOAP Fault Exception
}
catch (CommunicationException)
{
    // Catch this expected exception so it is not propagated further.
    // Perhaps write this exception out to log file for gathering statistics...
}
catch (TimeoutException)
{
    // Catch this expected exception so it is not propagated further.
    // Perhaps write this exception out to log file for gathering statistics...
}
catch (Exception)
{
    // An unexpected exception that we don't know how to handle.
    // Perhaps write this exception out to log file for support purposes...
    throw;
}
finally
{
    // This will:
    // - be executed if any exception was thrown above in the 'try' (including ThreadAbortException); and
    // - ensure that CloseOrAbortServiceChannel() itself will not be interrupted by a ThreadAbortException
    //   (since it is executing from within a 'finally' block)
    CloseOrAbortServiceChannel(client);

    // Unreference the client
    client = null;
}



private void CloseOrAbortServiceChannel(ICommunicationObject communicationObject)
{
    bool isClosed = false;

    if (communicationObject == null || communicationObject.State == CommunicationState.Closed)
    {
        return;
    }

    try 
    {
        if (communicationObject.State != CommunicationState.Faulted)
        {
            communicationObject.Close();
            isClosed = true;
        }
    }
    catch (CommunicationException)
    {
        // Catch this expected exception so it is not propagated further.
        // Perhaps write this exception out to log file for gathering statistics...
    }
    catch (TimeoutException)
    {
        // Catch this expected exception so it is not propagated further.
        // Perhaps write this exception out to log file for gathering statistics...
    }
    catch (Exception)
    {
        // An unexpected exception that we don't know how to handle.
        // Perhaps write this exception out to log file for support purposes...
        throw;
    }
    finally
    {
        // If State was Faulted or any exception occurred while doing the Close(), then do an Abort()
        if (!isClosed)
        {
            AbortServiceChannel(communicationObject);
        }
    }
}

private static void AbortServiceChannel(ICommunicationObject communicationObject)
{
    try
    {
        communicationObject.Abort();
    }
    catch (Exception)
    {
        // An unexpected exception that we don't know how to handle.
        // If we are in this situation:
        // - we should NOT retry the Abort() because it has already failed and there is nothing to suggest it could be successful next time
        // - the abort may have partially succeeded
        // - the actual service call may have been successful
        //
        // The only thing we can do is hope that the channel's resources have been released.
        // Do not rethrow this exception because the actual service operation call might have succeeded
        // and an exception closing the channel should not stop the client doing whatever it does next.
        //
        // Perhaps write this exception out to log file for gathering statistics and support purposes...
    }
}

Well, that's quite depressing, isn't it! Imagine that you have an application that calls many services. If you were to tell me that I should duplicate all that code every time I want to make a service operation call, as a developer I won't be happy.

Unfortunately that is exactly the situation that Microsoft has forced upon developers.

You could take some short-cuts and not do all the exception handling, but no doubt on the day that one of those perhaps rare exceptions happen (and it will!), you will be glad that you handled those edge cases.

Please Tell Me There Is a Better Way!

In my next article, I will show you how to use some programming tricks to significantly reduce the amount of code that developers have to write and provide a nice, clean API for working with WCF clients.

Thoughts on the Naming of SOA Services and Operations

   Submit to Reddit      
  

Recently I was having some discussions on the naming of Service-Oriented Architecture (SOA) services and their operations. There were some differing opinions on the length of the names and concerns about the duplication of some words in both the service name and operation names.

Naming Principles

As a general set of principles, I prefer to use names that are: intuitive, understandable, more readable in typical English, discoverable and allow for extensibility.

Microsoft has some useful naming conventions and guidelines for .NET that are reasonably mature and generally can be applied to SOA service names.

Service Names

As an example, let us assume that in a domain we have products/stock/inventory that is acquired and sold. We could say that there should be a service for the management of inventory items.

We therefore could propose an "Inventory Management" service to record the addition and removal of stock. A valid alternative name also could be "Inventory" service. "Inventory" as a service name obviously is shorter than "Inventory Management". So is that better?

One could at this point perhaps flip a coin and naively settle on either option. But I wouldn't recommend that...

Do those service names comply with the above-stated principles? Maybe - it depends on the domain.

Let me now give you some more possible information about the domain. When inventory is sold or moved to another department, there are typically corresponding accounting entries that need to be recorded. We therefore could propose an "Inventory Movement" service to encapsulate this.

So now in this domain, we optionally could have the following:

  1. "Inventory" and "Inventory Movement" services, or
  2. "Inventory Management" and "Inventory Movement" services.

For me, seeing the juxtaposition of both service names in option (1) makes me feel uneasy, highlights ambiguity and causes me confusion. Since there is "Inventory Movement", what is the role of "Inventory"?

Additionally, "Inventory" sounds very much like a data entity that one might perform CRUD operations on - as opposed to a Business Service.

Option (2) however is more intuitive, less ambiguous and causes me less confusion. "Inventory Management" intuitively sounds like it could be related to the management of inventory items (whatever that may be...). Overall for me, this makes more common sense than option (1).

So now, if you agree, we can see that the more specific service name of "Inventory Management" (even though it is longer and slightly more annoying to repeatedly type) better allows for business changes within a domain. In other words, more specific service names allow for future extensibility of the domain with less ambiguity. Having said this, we do need to remain vigilant and be careful that we are not overtly specific with our service names such that we make our services to narrowly focused...

Ok, let's now look at the operation names.

Operation Names

As an example, let us assume that we have an "Order Management" service. In this service we need to have operations for adding new orders to the system. We might therefore have an operation named "Add Order". Alternatively, we could have an operation just named "Add". "Add" is obviously shorter than "Add Order", and does not repeat the word "Order" - which you know from the service name. So is that better?

One could at this point perhaps flip a coin and naively settle on either option. But I wouldn't recommend that...

Do those operation names comply with the above-stated principles? Maybe - it depends on the service.

Let me now give you some more possible information about the service. Existing orders may need to be adjusted. In fact, there might be a need to add new order items to an existing order. We therefore could propose another operation named "Add Order Item" or "Add Item".

So now in this service, we optionally could have the following:

  1. "Add" and "Add Item" operations, or
  2. "Add Order" and "Add Order Item" operations.

Again, for me, seeing the juxtaposition of both operations names in option (1) makes me feel uneasy, highlights ambiguity and causes me confusion. Since there is "Add Item", what does the operation "Add" actually add?

Option (2) however is more intuitive, less ambiguous and causes me less confusion. Overall for me, this makes more common sense than option (1), and provides extensibility to allow for other operations in the future with less ambiguity.

Summary

I recommend erring on the safe side and selecting names that remain intuitive when taken out of the context of its grouping/category, and be carefully explicit with names - because you don't know how the business is going to change and how that could affect the domain. This is especially true if your team is operating in an agile fashion and developing and evolving the services and operations as you go.

The length of names and duplication of words in operation and service names should be less of a concern than using names that are intuitive and allow for change with less ambiguity. If you are not explicit enough, then you increase the risk of future services or operations making your existing names ambiguous. Alternatively, if you are too explicit, you could narrow the scope too much and reduce extensibility in other ways.