Coordinated Shutdown

There's an ActorSystem extension called CoordinatedShutdown that will stop certain Akka.NET actors / services and execute tasks in a programmable order during shutdown.

The default phases and their orderings are defined in the default HOCON configuration as akka.coordinated-shutdown.phases, and they are defined below:

    # do other work.
    received-message-size-limit = unlimited

    # Enable fine grained logging of what goes on inside the implementation.
    # Be aware that this may log more than once per message sent to the actors
    # of the tcp implementation.
    trace-logging = off

    # Fully qualified config path which holds the dispatcher configuration
    # to be used for running the select() calls in the selectors
    selector-dispatcher = "akka.io.pinned-dispatcher"

    # Fully qualified config path which holds the dispatcher configuration
    # for the read/write worker actors
    worker-dispatcher = "akka.actor.internal-dispatcher"

    # Fully qualified config path which holds the dispatcher configuration
    # for the selector management actors
    management-dispatcher = "akka.actor.internal-dispatcher"
  }

  dns {
    # Fully qualified config path which holds the dispatcher configuration
    # for the manager and resolver router actors.
    # For actual router configuration see akka.actor.deployment./IO-DNS/*
    dispatcher = "akka.actor.internal-dispatcher"

    # Name of the subconfig at path akka.io.dns, see inet-address below
    resolver = "inet-address"

    inet-address {
      # Must implement akka.io.DnsProvider
      provider-object = "Akka.IO.InetAddressDnsProvider"

      # These TTLs are set to default java 6 values
      positive-ttl = 30s
      negative-ttl = 10s

      # How often to sweep out expired cache entries.
      # Note that this interval has nothing to do with TTLs
      cache-cleanup-interval = 120s

      # Use IPv6 for host resolution. Set to false if
      # akka.io.tcp.outgoing-socket-force-ipv4 = true
      use-ipv6 = true
    }
  }
}
# CoordinatedShutdown is an extension that will perform registered
# tasks in the order that is defined by the phases. It is started
# by calling CoordinatedShutdown(system).run(). This can be triggered
# by different things, for example:
# - JVM shutdown hook will by default run CoordinatedShutdown
# - Cluster node will automatically run CoordinatedShutdown when it
#   sees itself as Exiting
# - A management console or other application specific command can
#   run CoordinatedShutdown
coordinated-shutdown {
  # The timeout that will be used for a phase if not specified with
  # 'timeout' in the phase
  default-phase-timeout = 5 s

  # Terminate the ActorSystem in the last phase actor-system-terminate.
  terminate-actor-system = on

  # Exit the CLR(Environment.Exit(0)) in the last phase actor-system-terminate
  # if this is set to 'on'. It is done after termination of the
  # ActorSystem if terminate-actor-system=on, otherwise it is done
  # immediately when the last phase is reached.
  exit-clr = off

  # Run the coordinated shutdown when the CLR process exits, e.g.
  # via kill SIGTERM signal (SIGINT ctrl-c doesn't work).

Custom Phases

As an end-user, you can register tasks to execute during any of these shutdown phases and add additional phases if you wish.

More phases can be added to an application by overriding the HOCON of an existing phase to include additional members in its phase.depends-on property. Here's an example where an additional phase might be executing before shutting down the cluster, for instance:

akka.coordinated-shutdown.phases.before-cluster-shutdown.depends-on = [service-stop, my-phase]
my-phase{
    timeout = 10s
    recover = on
}

For each phase, the following properties can be set:

1. depends-on - specifies which other phases have to run successfully first before this phase can begin;
2. timeout - specifies how long the tasks in this phase are allowed to run before timing out;
3. recover - if set to off, if any of the tasks in this phase throw an exception then the CoordinatedShutdown will be aborted and no further phases will be run. If set to on then any thrown errors are suppressed and the system will continue to execute the CoordinatedShutdown.

The default phases are defined in a linear order, but in practice the phases are ordered into a directed acyclic graph (DAG) using Topological sort.

Registering Tasks to a Phase

For instance, if you're using Akka.Cluster it's commonplace to register application-specific cleanup tasks during the cluster-leave and cluster-exiting phases. Here's an example:

var coordShutdown = CoordinatedShutdown.Get(myActorSystem);
coordShutdown.AddTask(CoordinatedShutdown.PhaseClusterLeave, "cleanup-my-api", () =>
{
    return _myCustomSocketApi.CloseAsync().ContinueWith(tr => Done.Instance);
});

Each shutdown task added to a phase must specify a function that returns a value of type Task<Done>. This function will be invoked once the CoordinatedShutdown.Run() command is executed and the Task<Done> returned by the function will be completed in parallel with all other tasks executing in the given phase. Those tasks may complete in any possible order; CoordinatedShutdown doesn't attempt to order the execution of tasks within a single phase. Once all of those tasks have completed OR if the phases timeout has expired, the next phase will begin.

Tasks should be registered as early as possible, preferably at system startup, in order to ensure that all registered tasks are run. If tasks are added after the CoordinatedShutdown have begun its run, it's possible that the newly registered tasks will not be executed.

Running CoordinatedShutdown

There are a few different ways to start the CoordinatedShutdown process.

If you wish to execute the CoordinatedShutdown yourself, you can simply call CoordinatedShutdown.Run(CoordinatedShutdown.Reason), which takes a CoordinatedShutdown.Reason argument will return a Task<Done>.

var actorSystem = ActorSystem.Create("MySystem");

// shutdown with reason "CLR exit" - meaning the process was being terminated
// task completes once node has left cluster and terminated the ActorSystem
Task shutdownTask = CoordinatedShutdown.Get(actorSystem)
    .Run(CoordinatedShutdown.ClrExitReason.Instance);
await shutdownTask;

// shutdown reason gets cached here.
// The`Reason` type can be subclassed with custom properties if needed
CoordinatedShutdown.Get(actorSystem).ShutdownReason.Should()
    .Be(CoordinatedShutdown.ClrExitReason.Instance);

It's safe to call this method multiple times as the shutdown process will only be run once and will return the same completion task each time. The Task<Done> will complete once all phases have run successfully, or a phase with recover = off failed.

Note

It's possible to subclass the CoordinatedShutdown.Reason type and pass in a custom implementation which includes custom properties and data. This data is accessible inside the shutdown phases themselves via the CoordinatedShutdown.ShutdownReason property.

Automatic ActorSystem and Process Termination

By default, when the final phase of the CoordinatedShutdown executes the calling ActorSystem will be terminated. This behavior can be changed by setting the following HOCON value in your configuration:

akka.coordinated-shutdown.terminate-actor-system = off

If this setting is disabled (it is enabled b default), the ActorSystem will not be terminated as the final phase of the CoordinatedShutdown phases.

CoordinatedShutdown phases, by default, are also executed when the ActorSystem is terminated. You can change this behavior by disabling this HOCON value in your configuration:

akka.coordinated-shutdown.run-by-actor-system-terminate = off

Note

It is illegal to have run-by-actor-system-terminate enabled and have terminate-actor-system disabled. Having these configuration combination will raise a ConfigurationException during ActorSystem startup.

The CLR process will still be running, even when the ActorSystem is terminated by the CoordinatedShutdown. If you'd like to automatically terminate the process running your ActorSystem, you can set the following HOCON value in your configuration:

akka.coordinated-shutdown.exit-clr = on

If this setting is enabled (it's disabled by default), you'll be able to shutdown the current running process automatically via an Environment.Exit(0) call made during the final phase of the CoordinatedShutdown.

CoordinatedShutdown and Akka.Cluster

If you're using Akka.Cluster, the CoordinatedShutdown will automatically register tasks for completing the following:

1. Gracefully leaving the cluster;
2. Gracefully handing over / terminating ClusterSingleton and Cluster.Sharding instances; and
3. Terminating the Cluster system itself.

By default, this graceful leave action will by triggered whenever the CoordinatedShutdown.Run(Reason) method is called. Conversely, calling Cluster.Leave on a cluster member will also cause the CoordinatedShutdown to run and will terminate the ActorSystem once the node has left the cluster.

CoordinatedShutdown.Run() will also be executed if a node is removed via Cluster.Down (non-graceful exit), but this can be disabled by changing the following Akka.Cluster HOCON setting:

akka.cluster.run-coordinated-shutdown-when-down = off

Invoking CoordinatedShutdown.Run() on Process Exit

By default CoordinatedShutdown.Run() will be called whenever the current process attempts to exit (using the AppDomain.ProcessExit event hook) and this will give the ActorSystem and the underlying clustering tools an opportunity to cleanup gracefully before the process finishes exiting.

If you wish to disable this behavior, you can pass in the following HOCON configuration value:

akka.coordinated-shutdown.run-by-clr-shutdown-hook = off