Add log forwarding Pebble layer to sparkd

[rgildein]

Add log forwarding layer yaml file, which is used in sparkd.sh and it
use environment variables to configure it.
The env used are:

• LOKI_URL - url, which can be either Loki itself or Grafana-agent
• FLAVOUR - executor | driver to distinguish between the executor and the driver
• SPARK_APPLICATION_ID - spark application id
• SPARK_USER - user running these Pods / Jobs
• HOSTNAME - Pod name / hostname

Add integration tests.

How I tested it:

$ rockcraft pack
charmed-spark_3.4.2_amd64.rock 
$ rockcraft.skopeo --insecure-policy copy oci-archive:charmed-spark_3.4.2_amd64.rock docker-daemon:my-charmed-spark:3.4.2 --dest-tls-verify=false
Getting image source signatures
Copying blob 6414378b6477 done   | 
Copying blob b2aef2bd7872 done   | 
Copying blob b73c860d028d done   | 
Copying blob 786708121d29 done   | 
Copying blob 96a2d5fe6170 done   | 
Copying config 4ea9dc90c4 done   | 
Writing manifest to image destination 
$ docker images
REPOSITORY         TAG       IMAGE ID       CREATED       SIZE
<none>             <none>    d932ff815a6c   4 hours ago   1.21GB
<none>             <none>    cc4074381dc8   4 weeks ago   1.21GB
my-charmed-spark   3.4.2     4ea9dc90c4d4   4 weeks ago   1.21GB
$ docker build -t my-charmed-spark:3.4.2 --build-arg BASE_IMAGE="my-charmed-spark:3.4.2" .
[+] Building 0.1s (5/5) FINISHED                                                    docker:default
 => [internal] load build definition from Dockerfile                                          0.0s
 => => transferring dockerfile: 209B                                                          0.0s
 => [internal] load .dockerignore                                                             0.0s
 => => transferring context: 2B                                                               0.0s
 => [internal] load metadata for docker.io/library/my-charmed-spark:3.4.2                     0.0s
 => [1/1] FROM docker.io/library/my-charmed-spark:3.4.2                                       0.0s
 => exporting to image                                                                        0.0s
 => => exporting layers                                                                       0.0s
 => => writing image sha256:43967a182b5d1db6c347972fece1d8db876b1b2d79344ed1ee680875b433d616  0.0s
 => => naming to docker.io/library/my-charmed-spark:3.4.2                                     0.0s
$ docker save my-charmed-spark:3.4.2 -o my-charmed-spark.img
$ microk8s ctr images import --base-name ghcr.io/canonical/my-charmed-spark my-charmed-spark.img
unpacking docker.io/library/my-charmed-spark:3.4.2 (sha256:0cbcb7a400d59e48928983ddf659b576f61a3903c9d900e4752d43b12ae3c2f6)...done

$ cat spark-defaults.conf 
spark.kubernetes.container.image=docker.io/library/my-charmed-spark:3.4.2 
# using grafana-agent-k8s deployed only for this purpose
$ spark-client.pyspark --username spark --namespace spark --properties-file ./spark-defaults.conf -v --conf spark.executorEnv.LOKI_URL=http://grafana-agent-k8s-0.grafana-agent-k8s-endpoints.spark-o11y.svc.cluster.local:3500/loki/api/v1/push
# followed example from https://canonical.com/data/docs/spark/k8s/t-spark-shell to see some logs
...
>>> spark.sparkContext.parallelize(lines.splitlines(), 2).map(count_vowels).reduce(add)

$ k -n spark get pods
NAME                                   READY   STATUS    RESTARTS   AGE
pysparkshell-fb00bc9289e61c8a-exec-1   1/1     Running   0          2s
pysparkshell-fb00bc9289e61c8a-exec-2   1/1     Running   0          2s
$ k -n spark logs pysparkshell-fb00bc9289e61c8a-exec-1 | grep -A 15 "Configuring log-forwarding to Loki."
2024-10-14T07:20:14.451Z [sparkd] Configuring log-forwarding to Loki.
2024-10-14T07:20:14.454Z [sparkd] log-targets:
2024-10-14T07:20:14.454Z [sparkd]   grafana-agent-k8s:
2024-10-14T07:20:14.454Z [sparkd]     override: replace
2024-10-14T07:20:14.454Z [sparkd]     type: loki
2024-10-14T07:20:14.454Z [sparkd]     location: http://grafana-agent-k8s-0.grafana-agent-k8s-endpoints.spark-o11y.svc.cluster.local:3500/loki/api/v1/push
2024-10-14T07:20:14.454Z [sparkd]     services: [all]
2024-10-14T07:20:14.454Z [sparkd]     labels:
2024-10-14T07:20:14.454Z [sparkd]       product: charmed-spark
2024-10-14T07:20:14.454Z [sparkd]       app: spark
2024-10-14T07:20:14.454Z [sparkd]       app_id: spark-201e7bde966b447b99445ae77dd098bd
2024-10-14T07:20:14.454Z [sparkd]       user: rgildein
2024-10-14T07:20:14.454Z [sparkd]       pod: pysparkshell-fb00bc9289e61c8a-exec-1
2024-10-14T07:20:14.457Z [pebble] POST /v1/layers 280.628µs 200
2024-10-14T07:20:14.457Z [sparkd] Layer "logging" added successfully from "/tmp/rendered_log_layer.yaml"

spark-client.spark-submit \
                  --username spark --namespace spark \
                  --conf spark.kubernetes.driver.request.cores=100m \
                  --conf spark.kubernetes.executor.request.cores=100m \
                  --conf spark.kubernetes.container.image=docker.io/library/my-charmed-spark:3.4.2 \
                  --conf spark.executorEnv.LOKI_URL='http://grafana-agent-k8s-0.grafana-agent-k8s-endpoints.spark-o11y.svc.cluster.local:3500/loki/api/v1/push' \
                  --conf spark.kubernetes.driverEnv.LOKI_URL='http://grafana-agent-k8s-0.grafana-agent-k8s-endpoints.spark-o11y.svc.cluster.local:3500/loki/api/v1/push' \
                  --class org.apache.spark.examples.SparkPi \
                  local:///opt/spark/examples/jars/spark-examples_2.12-3.4.2.jar 1000

[deusebio]

The changes look fine. I only have some comments about the name of the labels. I would like to keep the labelling as similar as possible to the one that we have in the metrics.

@welpaolo @theoctober19th could you please check (and have a thought) that the label names are consistent with the one we push?

[deusebio]

todo This should be spark_job_id to be consistent with the label in the metrics, see here

[theoctober19th]

+1

[deusebio]

todo this should be role I believe

[welpaolo]

Looks good to me after you address the comment made by Enrico about the names of the labels

[theoctober19th]

Good to merge with the change in the labels.

[deusebio]

LGTM

[welpaolo]

LGTM