TestActor behaves unpredictably - looks like a static variable coupled with thread-race issue

[brah-mcdude]

This issue has a bit of a history in discord: https://discord.com/channels/974500337396375553/974500337954222133/1245403250841096325

Here is a copy of the discussion:

brah.mcdude — Yesterday at 6:47 PM
hi all.
i am running tests using Akka.Hosting.TestKit. I have a test that mostly passes. But randomly fails. I don't know why.
Here is the code:

public class Spec(ITestOutputHelper output) : TestKit(output, LogLevel.Debug)
{
    private IActorRef? myActor;

    [Theory]
    [InlineData(1)]
    [InlineData(2)]
    [InlineData(3)]
    [InlineData(4)]
    [InlineData(5)]
    [InlineData(6)]
    [InlineData(7)]
    public async Task Spec1(int _)
    {
        myActor.Tell(new { });
        await ExpectMsgAsync<object>();
    }

    protected override void ConfigureAkka(AkkaConfigurationBuilder builder, IServiceProvider provider)
    {
        builder.WithActors((system, registry, resolver) =>
        {
            var props = resolver.Props<MyActor>();
            myActor = system.ActorOf(props, nameof(MyActor));
            registry.Register<MyActor>(myActor);
        });
    }

    public class MyActor : UntypedActor
    {
        protected override void OnReceive(object message)
        {
            Sender.Tell(new { });
        }
    }
}

Here is the log of the failure:
Failed: Timeout 00:00:03 while waiting for a message of type System.Object
brah.mcdude — Yesterday at 7:51 PM
ok. found out the cause of the issue. for some reason, we need a probe:

        var probe = CreateTestProbe();
        myActor.Tell(new { }, probe);
        await probe.ExpectMsgAsync<object>();

Should i report it as a bug?

Aaronontheweb — Yesterday at 7:53 PM
I think I know what your issue is
#237

GitHub
Akka.Hosting.TestKit: TestActor is no longer the implicit sender ...
Version Information Version of Akka.NET? 1.0.3 Which Akka.NET Modules? Akka.Hosting.TestKit Describe the bug foreach (var i in Enumerable.Range(0, 2)) { var update1 = Dsl.Update(CounterKey, GCounte...
Akka.Hosting.TestKit: TestActor is no longer the implicit sender ...

a very annoying bug with the TestKit
in Akka.Hosting

brah.mcdude — Today at 1:08 AM
unlikely. my tests mostly pass. 80% of the time. it can't be something like "wrong sender". as that would always fail.

brah.mcdude — Today at 1:58 PM
my intuition is that there are static variables. maybe the test-actor is static. and that only multiple tests running asynchronously are causing the issue.

brah.mcdude — Today at 3:18 PM
i'll open a bug to discuss this.

Aaronontheweb — Today at 4:40 PM
that actor context bug is a thread static variable
and the conditions under which you can lose that context can be racy
so I suspect that's what it is - have run into it myself using the Akka.Hosting.TestKit before

[brah-mcdude]

here is another example of an implicit use of the test-actor that gives unpredictable results:

/// this code gives unpredictable results: 
        var status = await actor.Ask<Status>(GetStatus.Instance);

/// until this issue is resolved, the code above should be replaced with this code: 
        probe = CreateTestProbe();
        actor.Tell(GetStatus.Instance, probe);
        var status = await probe.ExpectMsgAsync<Status>();

[Aaronontheweb]

I'm attempting to reproduce this today and it looks like I no longer can - everything seems to be working ok even with "run until failure" using v1.5.30.1

[Aaronontheweb]

This makes me wonder if this was an xUnit issue possibly

[Aaronontheweb]

Yes, it looks like this was an issue that was fixed in xUnit 2.8.0, which we upgraded to in June as part of #463

xUnit release notes here: https://xunit.net/releases/v2/2.8.0

[Aaronontheweb]

Welp, I take that back - was able to reproduce the failure with

   [Fact]
    public async Task ShouldAllowDuplicatePauseCommands()
    {
        // arrange
        var cohortId = TestCohortId;
        var cohortActor = await ActorRegistry.GetAsync<SubscriptionsActorRegistryKeys.CohortActorKey>();
        var pauseCmd = new CohortSendingCommands.PauseCohortSends(cohortId, DateTime.UtcNow);
        
        // act
        cohortActor.Tell(pauseCmd);
        
        
        // assert
        var resp1 = await ExpectMsgAsync<CommandResponse>();
        
        cohortActor.Tell(pauseCmd);
        var resp2 = await ExpectMsgAsync<CommandResponse>();
        
        Assert.Equal(CommandResponseCode.Success, resp1.Code);
        Assert.Equal(CommandResponseCode.NoOp, resp2.Code);
    }

The second Tell dead letters - implicit TestActor ref is lost here