Add support to generate OTEL logs in generate script

[shantanualsi]

Desciption:

This PR adds support to generate OTEL logs from the generate script. It creates another service loki-ingester-otel.
It defines another logger alongside AppLogger that pushes logs to the OTEL collector.

Few notes:

• Dockerfile is modified that starts otel collector which is required to ingest logs
• Loki address in otel-config is configurable as per the environment used. Can be extracted as env-var later
• Adds slog as it provides a bridge for otel provider
• Some refactor to move files to appropriate directories
• There are a lot of improvements that can be made in the script related to structured metadata and labels. Can be made in subsequent PRs.

Fixes: #224

[gtk-grafana]

@shantanualsi Thanks for taking a look at this, getting better OTEL logs in the generator is something we've wanted for a while now: #224

But I can't seem to get this to work locally.

When running yarn generate-logs: I get

2024/12/02 09:26:48 rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.logs.v1.LogsService

We might want a copy of the start.sh script that is generated by the Dockerfile available to call in the generate-logs script available in the repo?

Also yarn server and yarn server:ci appear broken as well, generator container is filled with logs like the following:

2024-12-02T15:30:11.506Z	info	internal/retry_sender.go:126	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "logs", "name": "otlphttp/logs", "error": "failed to make an HTTP request: Post \"http://localhost:3100/otlp/v1/logs\": dial tcp [::1]:3100: connect: connection refused", "interval": "38.571661375s"}

Is there some additional config I need to set?

[shantanualsi]

Oh hey @gtk-grafana, sorry coming back after quite a while here!
I'll check this again. I wasn't aware about yarn generate-logs at all. I always used the dockerfile or directly run the go app.

That said, I think we'll have to resort to always running this through dockerfile because pushing otel logs needs a otel collector running somewhere. I'm open to suggestions though.

[shantanualsi]

messed up the git commits 🤦 . I'll fix the branch and reopen the PR in a while