natchez integration and lambda init

[skirsdeda]

There's a problem when using tracing in combo with resources that are cached (in init or setup part of execution), which is even included in one of the examples: https://github.com/typelevel/feral/blob/main/examples/src/main/scala/feral/examples/KinesisLambda.scala#L52-L53.

The problem:

1. Actual root span cannot be used in init because it will never be closed. This is even worse if you reuse Trace[IO] instance inside request handler code, because all spans there would become children of this never-ending root span. And some natchez backend implementations (like XRay) only send root span together with all children when root span is finished. So you get no recorded spans as a result.
2. If you use fake root span to solve the first problem (smth like this: Empty Span natchez#566 (comment)), then spans produced during init (like DB connection pool initialization with skunk) are separate from spans produced in the first request (handler code) of that lambda instance.

This means that TracedHandler implementation is useless if you need to have Trace[IO] during init. In case of skunk, you would want to see SQL query spans inside request spans and for that you need to reuse same instance of Trace[IO] for all requests.

How it should work:

It should be possible to build a single Trace[IO] instance for the lifetime of lambda instance. Spans produced during initialization would get recorded as children of root span for the first ever request to that lambda instance.

I hope I managed to describe this clearly 😅. Would be great to work this out and get that example fixed so that it would show how to use skunk and tracing with feral.

[armanbilge]

Thanks for writing this up! Indeed, there are still some things to work out with Tracing :)

If I understand correctly, I think you've run into a version of problem (2) as described in typelevel/otel4s#11 (comment). See also bayou for some exploratory work at improving the situation.

In the short term, instead of using IO-based tracing, perhaps Kleisli based tracing would work here?

[armanbilge]

@skirsdeda did you have any more luck with this? Any ideas or feedback you have would be very valuable, thanks!

[skirsdeda]

@armanbilge I am working on it. I think I am somewhere around half way through at this point. Should have smth to show next week.