Serialization issue with BCL exceptions in mixed .NET Core & .NET Framework cluster

[BrettJaner]

Is it advised to run a mixed .NET Core & .NET Framework Akka.Net cluster? Are there known limitations? Currently, I'm having an issue with the SystemMessageSerializer. I have a two node cluster, one instance running .NET 8 and the other running .NET Framework 4.8.1. The parent actor is defined in the .NET 8 instance and the children are defined in the .NET Framework instance. When a child instance throws a BCL exception like InvalidOperationException and then the parent going to restart the child actor, I get an AssociationError. From what it looks like this is because the InvalidOperationException is defined in a different assembly depending on what framework we are targeting, either mscorlib or System.Private.CoreLib.

[15:58:21 ERR] AssociationError [akka.tcp://my-akka-system@localhost:53027] -> akka.tcp://my-akka-system@localhost:53013: Error [Value cannot be null.
Parameter name: type] [ at System.Runtime.Serialization.SerializationInfo..ctor(Type type, IFormatterConverter converter, Boolean requireSameTokenInPartialTrust)
at Akka.Remote.Serialization.ExceptionSupport.ExceptionFromProtoNet(ExceptionData proto)
at Akka.Remote.Serialization.SystemMessageSerializer.RecreateFromProto(Byte[] bytes)
at Akka.Serialization.Serialization.Deserialize(Byte[] bytes, Int32 serializerId, String manifest)
at Akka.Remote.MessageSerializer.Deserialize(ExtendedActorSystem system, Payload messageProtocol)
at Akka.Remote.DefaultMessageDispatcher.Dispatch(IInternalActorRef recipient, Address recipientAddress, Payload message, IActorRef senderOption)
at Akka.Remote.EndpointReader.b__16_0(Message msg)
at Akka.Util.Internal.Extensions.ForEach[T](IEnumerable1 source, Action1 action)
at Akka.Remote.EndpointReader.b__11_0(InboundPayload inbound)
at lambda_method(Closure , Object , Action1 , Action1 , Action1 ) at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture4.Handle(T value)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.UntypedActor.Receive(Object message)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)]

Any advice on how to go about a solution? Is this a bug? Should I go down the road of writing a custom SystemMessageSerializer? Should I expect other Akka.Net serializers to have the same issue? Any other "gotchyas" I should be aware of running a mixed framework cluster?

[Aaronontheweb]

Working between .NET Framework and .NET Core / .NET 5-8 is workable but it's a bit hacky if you're relying on reflection-based serialization to carry the day there. If you migrate to schema-based serialization this gets a lot easier - guide on how to do that here: https://www.youtube.com/watch?v=oRll1Mzoyl4

As for this specific error, this looks like a supervision error coming from a remotely deployed actor - is that what happened here? A remotely deployed actor died and the Exception couldn't be deserialized?

[BrettJaner]

Working between .NET Framework and .NET Core / .NET 5-8 is workable but it's a bit hacky if you're relying on reflection-based serialization to carry the day there. If you migrate to schema-based serialization this gets a lot easier - guide on how to do that here: https://www.youtube.com/watch?v=oRll1Mzoyl4

Thanks for the vid, I'll take a look.

As for this specific error, this looks like a supervision error coming from a remotely deployed actor - is that what happened here? A remotely deployed actor died and the Exception couldn't be deserialized?

What I believe I'm seeing is that the remotely deployed child is throwing an InvalidOperationException (throwing this exception on purpose for demo purposes). The InvalidOperationException seems to serialize fine back to the parent where I believe the supervision strategy takes over. I'm using the default supervision strategy here, so we are just going to restart the child. Now the InvalidOperationException gets serialized back to the child because it needs to be passed along as the "Cause" of the restart. Here is where my issue is. Another exception (the issue exception) is thrown during the deserialization of the InvalidOperationException in the process where the child is being restarted. If you'd like a small repo app I could whip ya one up.

[Aaronontheweb]

Yeah if you could send us a repro that would be great because this issue seems a lot simpler than it looks: in one of the two platforms we're trying to recreate the Exception but it's missing all of the necessary information to do so.

Edit: worth nothing, this problem might not be fixable - we switched away from using reflection to deserialize exceptions in newer versions of Akka.NET because it's Exception serialization using things like BinaryFormatter is a totally banned practice now (security.) If any of your .NET Framework functionality once depended on that, it's not coming back.

[BrettJaner]

Here's a link to a repo with the repro: https://github.com/BrettJaner/akka-exception-serialization-issue. Running instructions are in the README. I appreciate you taking a look, so thanks in advance!

[BrettJaner]

Did a bit more digging on the issue and I'm starting to better understand the problem. Akka may not be using the binary formatter any more to serialize exceptions, but it is still using the type name qualified with the assembly name to deserialize at the destination. The issue is that this assembly name can change from version to version of .NET. Some of these assembly move changes are automatically handled by Type Forwarding, especially in a .NET Core application, but no so much in .NET Framework. The .NET Framework doesn't have knowledge of the System.Private.CoreLib library. After doing some research on the web, I was able to come up with a solution by making a modification to one of Akka's internal extension methods. The change involves getting the assembly name of the type by using the TypeForwardedFromAttribute if one exists on the type. The assembly identified by the TypeForwardedFromAttribute should be considered the owner of that type as recommended by this post: dotnet/runtime #47113

namespace Akka.Util
{
        /// <summary>
        /// INTERNAL API
        /// Utility to be used by implementers to create a manifest from the type.
        /// The manifest is used to look up the type on deserialization.
        /// </summary>
        /// <param name="type">TBD</param>
        /// <returns>Returns the type qualified name including namespace and assembly, but not assembly version.</returns>
        [InternalApi]
        public static string TypeQualifiedName(this Type type)
        {
            if (ShortenedTypeNames.TryGetValue(type, out var shortened))
            {
                return shortened;
            }

            // respect type forwarded from attribute since types can move assemblies between .net versions
            var typeForwardedFromAttribute = type.GetCustomAttribute<TypeForwardedFromAttribute>();

            if (typeForwardedFromAttribute != null)
            {
                var assemblyName = cleanAssemblyVersionRegex.Replace(typeForwardedFromAttribute.AssemblyFullName, string.Empty);

                shortened = $"{type.FullName}, {assemblyName}";
            }
            else
            {
                shortened = cleanAssemblyVersionRegex.Replace(type.AssemblyQualifiedName, string.Empty);
            }

            ShortenedTypeNames.TryAdd(type, shortened);
            
            return shortened;
        }
    }
}

I'm going to look at next on how I can get this code into my application, maybe by extending or reimplementing the serializers. However, it'd probably be easier on me if this change would be viable for the core Akka library. Thoughts?

[BrettJaner]

Closing this discussion and opening issue #7215