Questions about logging

[ncupertino]

I have an application where this is logging: Processed 58 total records, ran 0 punctuators and committed 2 total tasks since the last update

Can you explain to me what ran and committed 2 mean?

[LGouellec]

Hi @ncupertino ,

Punctuator is a scheduler, you can access it if you use the Processor API (low level api Process(..) and Transform(..))

The Task is assigned to a single partition, so committed 2 total tasks means that the application has committed two partitions since the last commit.

[ncupertino]

Forgive me, I still don't understand.

How can I access via the API? Can you provide me with an example?

I have another question. Can the thread read more than one partition? In my scenario I have a consumer hosted on a k8s with 3 active PODS. As you explained, do the 3 PODS read the 3 partitions randomly?

[LGouellec]

Processor API Example : https://github.com/LGouellec/kafka-streams-dotnet/blob/develop/test/Streamiz.Kafka.Net.Tests/Processors/KStreamProcessorAPITests.cs

Thread can read more than one partition for sure. If you scale your app with 1 instance, and 1 thread per instance, let's suppose your source topic has 4 partitions. You have 1 thread = 4 partitions, 2 threads = 2 partitions each ..., 4 threads = 1 partition each

3 PODS with a source topic with 3 partitions = 1 pod per partition yes.

[ncupertino]

Now I'm confused lol.

You said that my log was committed to 2 partitions. Can you explain this log better to me? I will post an image here.

In this image, it means, for example, I processed 29 records and of these 29 records I committed 1 time.
In the second line I processed 28 records and committed them to 2 partitions, that is, was my single thread able to read from 2 partitions?

Am I correct in my analysis?

Could you please suggest what I could put more information in the log>

[LGouellec]

was my single thread able to read from 2 partitions?

Not necessary, it means you have committed twice during the last update (We have an update every minutes), and by default the application commit every 30 seconds.