How to handle a stale akka.net remote iactorref

Question

I am using remote actors in a akka.net in a windows service that stays up long term. I obtain an IActorRef for the remote actor using ActorSelection and I keep this IActorRef alive for an extended period of time in the service. The IActorRef points to an actor system running in another windows service. I understand that a restart of the remote actor will not invalidate the remote actor ref. However, it is conceivable that the remote windows service might get restarted at some point and the IActorRef in the calling windows service would become invalid.

What is the best practice for handling this? A naive approach would be to use ActorSelection to obtain a new IActorRef every time I want to make a call to the remote actor. This would obviously be inefficient.

Another approach might be to simply wrap every call I make on that IActorRef in some kind of error handling envelope that traps exceptions and obtains a new IActorRef using actorselection and retries? Or the envelope might make a test call before every actual call to see if the remote actor is still alive and if not get a new actor ref.

Any better way?

Answer 1

Default option for detecting dead actors is to Watch them (see documentation). When one actor watches another, it will receive Terminated message, once watched actor becomes dead or unreachable.

Answer 2

Watching for a Terminated message will alert a system that a remote actor has died, but there is a question of how exactly to respond to a terminated remote actor. Assuming that an actor obtains an IActorRef to a remote actor via its constructor, how would the actor obtain a new IActorRef to the remote actor when it becomes alive again. One way would be to have the actor fail and delegate to parent actor which would then obtain a new IActorRef to the remote actor via actor selection. The problem with this, however, is that the original actor selection for the remote actor might have taken place in non-actor code in a composition root where dependency injection normally would occur. I suppose you could get around this by passing a actor selection factory delegate around which could be used to reconstruct the remote IActorRef. An alternative that I came up with is to create a wrapper class that implements IActorRef called FaultTolerantActorRef.

This class takes the path of the remote (or local) actor in the constructor and periodically does an actor selection to get a refresh IActorRef to the remote actor. This way if for some reason the remote actor dies calls on the FaultTolerantActorRef will end up in dead letters while the remote actor is dead. However, when the remote actor eventually comes on line again, calls to the FaultTolerantActorRef will eventually reach the newly revived remote actor without having to take any explicit action on the part of the calling local actor.

There is an Invalidate method which will force the FaultTolerantActorRef to do a fresh actor selection on the next call. This could presumebly be called by an actor in response to a Terminated message from the remote actor. Even without calling Invalidate, a fresh actor selection will occur based on the refresh interval passed to the constructor.

using Akka.Actor;
using System;
using Akka.Util;
using System.Threading;

namespace JA.AkkaCore
{
    public class FaultTolerantActorRef : IActorRef
    {
        public IActorRef ActorRef
        {
            get
            {
                if (!_valid || DateTime.Now.Ticks > Interlocked.Read(ref _nextRefreshTime))
                    RefreshActorRef();
                return _actorRef;
            }

        }

        public ActorPath Path
        {
            get
            {
                return ActorRef.Path;
            }
        }

        object _lock = new object();
        IActorRef _actorRef;
        volatile bool _valid;
        string _path;
        IActorRefFactory _actorSystem;
        private TimeSpan _requestTimeout;
        private TimeSpan _refreshInterval;
        //private DateTime _nextRefreshTime = DateTime.MinValue;
        private long _nextRefreshTime = DateTime.MinValue.Ticks;

        public FaultTolerantActorRef(IActorRefFactory actorSystem, IActorRef actorRef,
            TimeSpan refreshInterval = default(TimeSpan), TimeSpan requestTimeout = default(TimeSpan))
            : this(actorSystem, actorRef.Path.ToString(), refreshInterval, requestTimeout)
        {
            _actorRef = actorRef;
            _valid = true;
        }
        public FaultTolerantActorRef(IActorRefFactory actorSystem, string actorPath,
            TimeSpan refreshInterval = default(TimeSpan), TimeSpan requestTimeout = default(TimeSpan))
        {
            if (refreshInterval == default(TimeSpan))
                _refreshInterval = TimeSpan.FromSeconds(60);
            else
                _refreshInterval = refreshInterval;
            if (requestTimeout == default(TimeSpan))
                _requestTimeout = TimeSpan.FromSeconds(60);
            else
                _requestTimeout = requestTimeout;
            _actorSystem = actorSystem;
            _valid = false;
            _path = actorPath;
        }
        private void RefreshActorRef()
        {
            lock(_lock)
            {
                if (!_valid || DateTime.Now.Ticks > _nextRefreshTime)
                {
                    _actorRef = _actorSystem.ActorSelectionOne(_path, _requestTimeout);
                    Interlocked.Exchange(ref _nextRefreshTime,DateTime.Now.Ticks + _refreshInterval.Ticks);
                    _valid = true;
                }
            }
        }

        public void Invalidate()
        {
            _valid = false;
        }

        public void Tell(object message, IActorRef sender)
        {
            ActorRef.Tell(message, sender);
        }

        public bool Equals(IActorRef other)
        {
            return ActorRef.Equals(other);
        }

        public int CompareTo(IActorRef other)
        {
            return ActorRef.CompareTo(other);
        }

        public ISurrogate ToSurrogate(ActorSystem system)
        {
            return ActorRef.ToSurrogate(system);
        }

        public int CompareTo(object obj)
        {
            return ActorRef.CompareTo(obj);
        }
    }
}