Assertion failed: (r == 0), function rwlock_wrlock, file tinycthread_extra.c, line 157

[okomen-foreflight]

Hello, sometimes I get the error in the title
Assertion failed: (r == 0), function rwlock_wrlock, file tinycthread_extra.c, line 157

It tanks the entire application?

Do you have any idea why it happens or how to debug it?

It may also be related to another error I get very rarely: error opening store at location.
I have set a StateDir like so:
StateDir = Path.Combine(Directory.GetCurrentDirectory(), "RocksDbState"),

It happens randomly. Do I have to give it some permissions or similar?

This is my toplogy description:

Topologies: Sub-topology: 0 for global store (will not generate tasks) Source: KSTREAM-SOURCE-0000000001 (topics: [fleet-flightId-metadata-positions]) --> KTABLE-SOURCE-0000000002 Processor: KTABLE-SOURCE-0000000002 (stores: [fleet-flightId-metadata-positions-STATE-STORE-0000000000]) --> none <-- KSTREAM-SOURCE-0000000001 Sub-topology: 5 Source: UpdateEventProcessor-request-source (topics: [UpdateEventProcessor-request]) --> UpdateEventProcessor Processor: UpdateEventProcessor (stores: []) --> none <-- UpdateEventProcessor-request-source Sub-topology: 4 Source: KSTREAM-SOURCE-0000000024 (topics: [ProcessorAggregatorGroup-repartition]) --> PreviousUpdatesAggregated Processor: PreviousUpdatesAggregated (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000021]) --> KTABLE-TOSTREAM-0000000025 <-- KSTREAM-SOURCE-0000000024 Processor: KTABLE-TOSTREAM-0000000025 (stores: []) --> UpdateEventProcessor-request-sink <-- PreviousUpdatesAggregated Sink: UpdateEventProcessor-request-sink (topic: UpdateEventProcessor-request) <-- KTABLE-TOSTREAM-0000000025 Sub-topology: 3 Source: KSTREAM-SOURCE-0000000016 (topics: [TransformationAggregatorGroup-repartition]) --> Transform Processor: Transform (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000013]) --> KTABLE-TOSTREAM-0000000017 <-- KSTREAM-SOURCE-0000000016 Processor: KTABLE-TOSTREAM-0000000017 (stores: []) --> TransformedStream <-- Transform Processor: TransformedStream (stores: []) --> Enrich <-- KTABLE-TOSTREAM-0000000017 Processor: Enrich (stores: []) --> KSTREAM-FILTER-0000000023 <-- TransformedStream Processor: KSTREAM-FILTER-0000000023 (stores: []) --> KSTREAM-SINK-0000000022 <-- Enrich Sink: KSTREAM-SINK-0000000022 (topic: ProcessorAggregatorGroup-repartition) <-- KSTREAM-FILTER-0000000023 Sub-topology: 2 Source: KSTREAM-SOURCE-0000000006 (topics: [fleet-tail-positions]) --> Filter Processor: Filter (stores: []) --> GenerateFlightId <-- KSTREAM-SOURCE-0000000006 Processor: GenerateFlightId (stores: [KSTREAM-AGGREGATE-STATE-STORE-0000000009]) --> KTABLE-TOSTREAM-0000000010 <-- Filter Processor: KTABLE-TOSTREAM-0000000010 (stores: []) --> GeneratedFlightIdStream <-- GenerateFlightId Processor: GeneratedFlightIdStream (stores: []) --> KSTREAM-FILTER-0000000015 <-- KTABLE-TOSTREAM-0000000010 Processor: KSTREAM-FILTER-0000000015 (stores: []) --> KSTREAM-SINK-0000000014 <-- GeneratedFlightIdStream Sink: KSTREAM-SINK-0000000014 (topic: TransformationAggregatorGroup-repartition) <-- KSTREAM-FILTER-0000000015 Sub-topology: 1 Source: KSTREAM-SOURCE-0000000003 (topics: [fleet-flightId-updated-metadata]) --> KSTREAM-LEFTJOIN-0000000004 Processor: KSTREAM-LEFTJOIN-0000000004 (stores: []) --> KSTREAM-SINK-0000000005 <-- KSTREAM-SOURCE-0000000003 Sink: KSTREAM-SINK-0000000005 (topic: fleet-flightId-metadata-positions) <-- KSTREAM-LEFTJOIN-0000000004

[LGouellec]

Hey @okomen-foreflight ,

StateDir is basically where the local state stores are persisted, so the service must be able to read/write in this directory.

Do you have a stacktrace error when the Assertion failed: (r == 0), function rwlock_wrlock, file tinycthread_extra.c, line 157 issue happens ?

You can configure your application to have more troubleshoot informations :

var config = new StreamConfig();
config.Debug = "all";
config.Logger = LoggerFactory.Create((b) => {
   b.SetMinimumLevel(LogLevel.Debug);
   b.AddConsole();
   //...
});

[okomen-foreflight]

Hmmm it's hard to reproduce, but good to know about the LoggerFactory. If I encounter it again I will reopen this discussion and provide you with as much information as I can gather from the logs!
Thanks for the logger configuration tip!