Table of Contents

Overview

The following information relates to Service Fabric SDK version 6.

Attaching the Debugger

If a service is running as a domain user, the debugger in Visual Studio (running as Administrator) will not be able to break into the CreateServiceInstanceListeners() method on your StatelessService or CreateServiceReplicaListeners() method on your StatefulService UNLESS the domain user is also a member of the local Administrators group.

Windows Principal / Identity

When a Service Fabric service is running:

• System.Threading.Thread.CurrentPrincipal.Identity is an instance of System.Security.Principal.GenericIdentity - with a blank Name!
• System.Security.Principal.WindowsIdentity.GetCurrent() returns an instance of System.Security.Principal.WindowsIdentity with correct details of the account the service is running under.

Target .NET Version

• Service Fabric 6 SDK (and the 2.8 NuGet packages) still require the full .NET Framework (on Windows).
• Recommendation: When you create a new service in your fabric application, at least target .NET Framework 4.7.1.
• Support for running on .NET Core 2.0 (instead of the full .NET Framework) is coming soon - with the 3.0 Service Fabric NuGet libraries.
• See my .NET Notebook

Naming & Max Path Limitations

• Windows Server 2012 R2 has a max path limitation of 260 characters.
• Windows Server 2016 has a Group Policy setting allowing that limitation to be removed to support long paths - at Computer Configuration > Administrative Templates > System > Filesystem > Enable NTFS long paths
• The default Service Fabric data root (FabricDataRoot specified in the cluster configuration) is D:\ProgramData\SF
• Service Fabric deploys service binaries to:
  • [FabricDataRoot]\[NodeName]\Fabric\work\Applications\[ApplicationTypeName]_App??\[ServiceName]Pkg.Code.[Version]\
  • e.g. D:\ProgramData\SF\myserver-vm0\Fabric\work\Applications\MyCompany.MyDomain.MySubDomain.FabricAppType_App23\MyAwesomeServicePkg.Code.1.0.10\MyCompany.MyDomain.MySubDomain.MyAwesomeService.Logic.dll.config
  • The example above is 206 characters and is well within the max path limit.
• Naming Recommendation:
  • Ensure that the package name (service name + “Pkg”) does not contain the full namespace - as that could easily cause the max path limit to be exceeded
  • e.g. set the package name to “MyAwesomeServicePkg” instead of “MyCompany.MyDomain.MySubDomain.MyAwesomeServicePkg”
  • In the ServiceManifest.xml, set the package name in the XML attribute //ServiceManifest@Name - to e.g. “MyAwesomeServicePkg”
  • In the ApplicationManifest.xml, set the same package name in the XML attribute //ServiceManifestImport/ServiceManifestRef@ServiceManifestName

How to Create a Service Hosted with Kestrel

NuGet Packages To Install

When you first create a Kestrel service (and of course you selected .NET 4.7.1 or later), perform the following NuGet library installations in order so that a bunch of .NET Standard 1.x System libraries are not installed:

Install:

• Microsoft.AspNetCore.Server.Kestrel (version > 2.0)
• Microsoft.AspNetCore.Server.Kestrel.Https

Update:

• Microsoft.NETCore.Platforms

Install:

• Microsoft.ServiceFabric.AspNetCore.Kestrel (which depends on Microsoft.AspNetCore.Server.Kestrel version >1.0)

Configure Kestrel Options for HTTPS & Dynamic Ports

In order to configure HTTPS, you will need to use the KestrelServerOptions.Listen() method. However, KestrelServerOptions.Listen() requires an IPEndpoint or IPAddress and Port to be specified.

Choose a Service Fabric Reverse Proxy Compatible IP Address

Note that:

• IPAddress.Loopback will cause the IP address “127.0.0.1” to be resolved by the Service Fabric Reverse Proxy
• IPAddress.Any will cause the IP address “0.0.0.0” to be resolved by the Service Fabric Reverse Proxy.

In both of these cases, and depending on the number of instances of a given service, a multi-server cluster will experience delays and even failures when an API call is performed via the Service Fabric Reverse Proxy.

IPAddress.IPv6Any is the only IP Address that is compatible with the Service Fabric Reverse Proxy. This will cause the server’s fully qualified domain name to be registered correctly and resolved by the Service Fabric Reverse Proxy.

Choose a Dynamic Port

As per the Kestrel documentation, port number 0 will cause a port to be allocated dynamically.

Example Kestrel Configuration

 1/// <summary>
 2/// Optional override to create listeners (e.g., TCP, HTTP) for this service replica to handle client or user requests.
 3/// </summary>
 4/// <returns>A collection of listeners.</returns>
 5protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
 6{
 7    return new ServiceInstanceListener[]
 8    {
 9        new ServiceInstanceListener(serviceContext =>
10            new KestrelCommunicationListener(serviceContext, (url, listener) =>
11                new WebHostBuilder()
12                    .UseKestrel(ConfigureKestrelServerOptions)
13                    .ConfigureServices(services => services.AddSingleton<StatelessServiceContext>(serviceContext))
14                    .UseContentRoot(Directory.GetCurrentDirectory())
15                    .UseServiceFabricIntegration(listener, ServiceFabricIntegrationOptions.UseReverseProxyIntegration | ServiceFabricIntegrationOptions.UseUniqueServiceUrl)
16                    .UseStartup<Startup>()
17                    .UseUrls(url)
18                    .Build()
19            ))
20    };
21}
22
23private void ConfigureKestrelServerOptions(KestrelServerOptions options)
24{
25    options.AddServerHeader = false;
26    // options.Limits.
27    // options.ApplicationServices
28
29    IPAddress serviceFabricReverseProxyCompatibleIPAddress = IPAddress.IPv6Any;
30    const int dynamicPort = 0;
31
32    options.Listen(serviceFabricReverseProxyCompatibleIPAddress, dynamicPort);
33    
34    // FOR HTTPS, DO THIS INSTEAD:
35    //options.Listen(
36    //    serviceFabricReverseProxyCompatibleIPAddress,
37    //    dynamicPort,
38    //    listenOptions => listenOptions.UseHttps(new HttpsConnectionAdapterOptions
39    //    {
40    //        CheckCertificateRevocation = true,
41    //        ClientCertificateMode = ClientCertificateMode.NoCertificate,
42    //        //ClientCertificateValidation = myAdditionalClientCertificateValidationCallback
43    //        //ServerCertificate = TODO get an X509Certificate2
44    //        SslProtocols = System.Security.Authentication.SslProtocols.Tls12
45    //    })
46    //);
47}

Configuration & Deployment

• When deployment is performed, Service Fabric will update and reformat your ApplicationManifest.xml and other configuration files. For instance, attributes with default values will be removed.

• Deployment completely fails when a <SecurityAccessPolicy> is defined within <Policies><SecurityAccessPolicies>:

   1<Principals>
   2    <Users>
   3        <User Name="ServiceAccount" AccountType="DomainUser" AccountName="mydomain\myaccount" Password="encrypted=" PasswordEncrypted="true" />
   4    </Users>
   5</Principals>
   6<Policies>
   7    <SecurityAccessPolicies>
   8        <SecurityAccessPolicy ResourceRef="ServiceEndpoint" PrincipalRef="ServiceAccount" GrantRights="Full" />
   9    </SecurityAccessPolicies>
  10</Policies>
  

  The deployment error is:

  12>Upload to Image Store succeeded
  22>Registering application type...
  32>Register-ServiceFabricApplicationType : Object reference not set to an instance of an object.
  42>At C:\Program Files\Microsoft SDKs\Service
  52>Fabric\Tools\PSModule\ServiceFabricSDK\Publish-NewServiceFabricApplication.ps1:251 char:9
  62>+         Register-ServiceFabricApplicationType -ApplicationPathInImageStore $appl ...
  72>+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  82>    + CategoryInfo          : InvalidOperation: (Microsoft.Servi...usterConnection:ClusterConnection) [Register-ServiceFabricApplicationType], FabricException
  92>    + FullyQualifiedErrorId : RegisterApplicationTypeErrorId,Microsoft.ServiceFabric.Powershell.RegisterApplicationType
  

RunAs Policy, HTTP/S URL Reservation & Elevated Privilege Requirement

This section is only relevant when needing to reserve a URL for HTTP or HTTPS - such as when writing WCF services within Service Fabric (using the WcfCommunicationListener). That requirement is described in this WCF documentation due to WCF’s usage of Http.sys.

The information below pertains to running Service Fabric services under a domain user account - using the various types of RunAs policy configuration options in the ApplicationManifest.xml file.

The user that attempts to reserve the URL requires elevated privileges - such as being a local Administrator. URL reservation can be achieved in a few ways:

• Manually login in as an administrator and run netsh as per this WCF documentation - but that would require static ports to be defined (an approach that is not really suitable in a modern, distributed services orchestration platform such as Service Fabric);
• Actually grant the RunAs user membership of the local Administrator group (which is not a good idea because the service will be running as a local administrator and not abiding by the Principle of Least Privilege);
• Use <DefaultRunaAsPolicy> instead of an explicit RunAsPolicy specified in <ServiceManifestImport>;
• When using an explicit RunAsPolicy in in <ServiceManifestImport>, specify a <SecurityAccessPolicy> as per this documentation.

The following scenarios are true for a domain user account that is NOT a member of the local Administrators group.

(a) THIS WORKS -> configuring a <DefaultRunaAsPolicy> with a domain user (without a sibling SecurityAccessPolicy):

1<Principals>
2    <Users>
3        <User Name="ServiceAccount" AccountType="DomainUser" AccountName="mydomain\myaccount" Password="encrypted=" PasswordEncrypted="true" />
4    </Users>
5</Principals>
6<Policies>
7    <DefaultRunAsPolicy UserRef="ServiceAccount" />
8</Policies>

(b) THIS DOES NOT WORK -> configuring a RunAsPolicy in the <ServiceManifestImport> section (without a sibling SecurityAccessPolicy):

1<ServiceManifestImport>
2    <ServiceManifestRef ServiceManifestName="Stateful1Pkg" ServiceManifestVersion="1.0.0" />
3    <ConfigOverrides />
4    <Policies>
5        <RunAsPolicy CodePackageRef="Code" EntryPointType="Main" UserRef="ServiceAccount" />
6    </Policies>
7</ServiceManifestImport>

Regardless of whether EntryPointType is set to All or Main, the following AddressAccessDeniedException exception occurs:

 1Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
 2Replica had multiple failures during open on _Node_0. API call: IStatefulServiceReplica.ChangeRole(P); Error = System.ServiceModel.AddressAccessDeniedException (-2146233087)
 3HTTP could not register URL http://+:30004/. Your process does not have access rights to this namespace (see http://go.microsoft.com/fwlink/?LinkId=70353 for details).
 4   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
 5   at Microsoft.ServiceFabric.Services.Communication.Wcf.Runtime.WcfCommunicationListener`1.b__10_0(IAsyncResult ar)
 6   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
 7--- End of stack trace from previous location where exception was thrown ---
 8   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
 9   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
10   at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceReplicaAdapter.d__26.MoveNext()
11--- End of stack trace from previous location where exception was thrown ---
12   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
13   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
14   at Microsoft.ServiceFabric.Services.Runtime.StatefulServiceReplicaAdapter.d__18.MoveNext()
15For more information see: http://aka.ms/sfhealth

When this occurs, Service Fabric reports the service as unhealthy and repeatedly calls the CreateServiceInstanceListeners() method on your StatelessService or CreateServiceReplicaListeners() method on your StatefulService - hoping that one day it will work.

(c) THIS WORKS -> configuring a RunAsPolicy in the <ServiceManifestImport> section - with a sibling SecurityAccessPolicy as per this documentation:

1<ServiceManifestImport>
2    <ServiceManifestRef ServiceManifestName="Stateful1Pkg" ServiceManifestVersion="1.0.0" />
3    <ConfigOverrides />
4    <Policies>
5        <RunAsPolicy CodePackageRef="Code" UserRef="ServiceAccount" />
6        <SecurityAccessPolicy ResourceRef="ServiceEndpoint" PrincipalRef="ServiceAccount" GrantRights="Full" />
7    </Policies>
8</ServiceManifestImport>

Who Reserved the URL & Who is the Service Running As?

The command netsh http show urlacl can be run to show the current HTTP/S URL reservations.

(a) When running without an explicit RunAs policy in the ApplicationManifest.xml (which means the Network Service local account will be used by default), the Network Service account reserves the HTTP URLs:

 1Reserved URL            : http://+:30012/
 2    User: NT AUTHORITY\NETWORK SERVICE
 3        Listen: Yes
 4        Delegate: No
 5        SDDL: D:(A;;GX;;;NS)
 6
 7Reserved URL            : http://localhost:30012/
 8    User: NT AUTHORITY\NETWORK SERVICE
 9        Listen: Yes
10        Delegate: No
11        SDDL: D:(A;;GX;;;NS)

and:

• User name from process in Task Manager: “NETWORK SERVICE”
• System.Threading.Thread.CurrentPrincipal.Identity: "”
• System.Security.Principal.WindowsIdentity.GetCurrent(): “NT AUTHORITY\NETWORK SERVICE”

(b) When using a <DefaultRunaAsPolicy> with a domain user:

1<Principals>
2    <Users>
3        <User Name="ServiceAccount" AccountType="DomainUser" AccountName="mydomain\myaccount" Password="encrypted=" PasswordEncrypted="true" />
4    </Users>
5</Principals>
6<Policies>
7    <DefaultRunAsPolicy UserRef="ServiceAccount" />
8</Policies>

as expected, the domain user account reserves the HTTP URLs:

 1Reserved URL            : http://+:30035/
 2    User: mydomain\myaccount
 3        Listen: Yes
 4        Delegate: No
 5        SDDL: D:(A;;GX;;;S-1-5-21-2197669547-3912936102-3481227320-25699)
 6
 7Reserved URL            : http://localhost:30035/
 8    User: mydomain\myaccount
 9        Listen: Yes
10        Delegate: No
11        SDDL: D:(A;;GX;;;S-1-5-21-2197669547-3912936102-3481227320-25699)

and:

• User name from process in Task Manager: “myaccount”
• System.Threading.Thread.CurrentPrincipal.Identity: "”
• System.Security.Principal.WindowsIdentity.GetCurrent(): “mydomain\myaccount”

(c) When configuring a RunAsPolicy in the <ServiceManifestImport> section - with a sibling SecurityAccessPolicy:

1<ServiceManifestImport>
2    <ServiceManifestRef ServiceManifestName="Stateful1Pkg" ServiceManifestVersion="1.0.0" />
3    <ConfigOverrides />
4    <Policies>
5        <RunAsPolicy CodePackageRef="Code" UserRef="ServiceAccount" />
6        <SecurityAccessPolicy ResourceRef="ServiceEndpoint" PrincipalRef="ServiceAccount" GrantRights="Full" />
7    </Policies>
8</ServiceManifestImport>

as expected, the domain user account reserves the HTTP URLs:

 1Reserved URL            : http://+:30018/
 2    User: mydomain\myaccount
 3        Listen: Yes
 4        Delegate: No
 5        SDDL: D:(A;;GX;;;S-1-5-21-2197669547-3912936102-3481227320-25699)
 6
 7Reserved URL            : http://localhost:30018/
 8    User: mydomain\myaccount
 9        Listen: Yes
10        Delegate: No
11        SDDL: D:(A;;GX;;;S-1-5-21-2197669547-3912936102-3481227320-25699)

and:

• User name from process in Task Manager: “myaccount”
• System.Threading.Thread.CurrentPrincipal.Identity: "”
• System.Security.Principal.WindowsIdentity.GetCurrent(): “mydomain\myaccount”

(d) When the RunAs policy is specified in <ServiceManifestImport> (AND the domain user account is a member of the local Administrators group):

1<ServiceManifestImport>
2    <ServiceManifestRef ServiceManifestName="Stateful1Pkg" ServiceManifestVersion="1.0.0" />
3    <ConfigOverrides />
4    <Policies>
5        <RunAsPolicy CodePackageRef="Code" EntryPointType="All" UserRef="ServiceAccount" />
6    </Policies>
7</ServiceManifestImport>

However, the Network Service account reserves the HTTP URLs:

 1Reserved URL            : http://+:30037/
 2    User: NT AUTHORITY\NETWORK SERVICE
 3        Listen: Yes
 4        Delegate: No
 5        SDDL: D:(A;;GX;;;NS)
 6
 7Reserved URL            : http://localhost:30037/
 8    User: NT AUTHORITY\NETWORK SERVICE
 9        Listen: Yes
10        Delegate: No
11        SDDL: D:(A;;GX;;;NS)

even though the identity is still the domain account:

• User name from process in Task Manager: “myaccount”
• System.Threading.Thread.CurrentPrincipal.Identity: "”
• System.Security.Principal.WindowsIdentity.GetCurrent(): “mydomain\myaccount”

Endpoint Configuration & HTTPS URL Reservation

• Endpoints must be setup via configuration in the ServiceManifest.xml.
• I haven’t discovered how to configure an endpoint in code - especially one that uses a dynamic port and has Service Fabric perform the URL reservation. Wish I knew how…
• Specifying the Protocol attribute in the <Endpoint> in the ServiceManifest.xml file is important - otherwise the URL reservation does not work.
• The endpoint configuration (such as the Protocol attribute) can be overridden in the <ResourceOverrides> section of the ApplicationManifest.xml.
• Even if you retrieve and change the endpoint configuration from/in the ServiceContext, URL reservation does not take place on your changed values.

Troubleshooting

• If a service does not start, there could be multiple reasons:

  • The listener is failing to initialise:
    • RunAs issues with a service account - incorrect credentials, group membership, disabled account
    • HTTPS URL reservation is failing because the endpoints are not configured correctly
      • Issue with the endpoint configuration - see above regarding RunAsPolicy, DefaultRunAsPolicy, SecurityAccessPolicy
      • EndpointBindingPolicy for binding a certificate as part of the HTTPS URL reservation
  • Max path issue (see above)
  • Replicator failures:
    • Do not add your own custom parameters into the <ReplicatorConfig> section - if you do, your services will not start - a replica error
    • Replicator security configuration - issues with the certificate not being found or being invalid

• Diagnosis techniques:

  • RDP to the node and attempt to execute the service executable - to determine if it fails due to a max path error
  • Turn on <ConsoleRedirection> in your ServiceManifest.xml and inspect the log files - if they are created
    • On a local developer cluster, the logs are in C:\SfDevCluster\Data\_App\[Node.x]\[ApplicationType]\log
    • On a default installation in Azure, the logs are in D:\SvcFab\_App\[ApplicationType]\log
  • Inspect the Windows Event Log
    • The Application log
    • The Service Fabric Admin and Ops logs
  • Use PerfView to monitor Event Tracing for Windows (ETW) events
  • Use Microsoft Message Analyzer

Bookmarks

• Service Fabric
• Service Fabric Issues on Github
• Specifying Resources in a Service Manifest
• Configure Stateful Reliable Services
• Understanding Service Fabric Partitions
• Build and deploy a multi-container application in Azure Service Fabric

The Service Fabric Notebook has been updated to include critical information about configuring Kestrel listener IP addresses so they will be correctly registered and resolved by the Service Fabric Reverse Proxy.

See Service Fabric Notebook for more information.

I’ve been intending to do this for a while - announcing that the ChannelAdam Software Development Notebooks are now live!

Notebooks are a growing collection of learnings, study notes, and bookmarks. News and blog articles are so static and point-in-time, whereas Notebooks contain content that is dynamic and will grow and change over time.

To start with, here is the start of my first Notebook on Microsoft Service Fabric.