extraMetaRules for executionEvent

[sethjones348]

Utilizing the following extraMetaRule, I was successfully able to pass the sparkContextId from the databricks spark context to the spline server within the executionPlan results:

{
  "executionPlan": {
    "extra": {
      "sparkContextId": {
        "$js": "session.conf().get('spark.databricks.sparkContextId')"
      }
    }
  }
}

I would really like to have that extra field in the GET /execution-events response. However, when I update that rule to the following, I am not seeing the extra property in the items within the execution-events response.

{
  "executionEvent": {
    "extra": {
      "sparkContextId": {
        "$js": "session.conf().get('spark.databricks.sparkContextId')"
      }
    }
  }
}

Here is what my response looks like for GET /execution-events:

{
    "items": [
        {
            "executionEventId": "018afa63-8ca1-5d6e-a18b-4598984f2c09:lxkgz5sc",
            "executionPlanId": "018afa63-8ca1-5d6e-a18b-4598984f2c09",
            "frameworkName": "spark 3.3.2",
            "applicationName": "Databricks Shell",
            "applicationId": "app-20240618133426-0000",
            "timestamp": 1718719100940,
            "durationNs": 203862151,
            "dataSourceName": "...",
            "dataSourceUri": "...",
            "dataSourceType": "delta",
            "append": false
        },
        ...
      ]
      ...
}

Is there something I need to configure to see the extra property in the GET /execution-events response? Any help is much appreciated!

Thanks!

[wajda]

Assuming you are using Spline REST server, the extra field should be stored in the database, but unfortunately there is currently no API to get one over the REST. The Consumer REST API is quite specific for the Spline UI purposes, and the extra event metadata isn't shown on the UI.
It's easy to add that feature thought. It needs to be added here and here.
Create an issue and a pull-request and we'll merge it.

[sethjones348]

Thanks for the reply! With that in mind I am planning to get the PR open against the spline repo ASAP. If I am able to get the PR open today or tomorrow, approximately how long do you think it would be before it would be released and available? Assuming the code looks good?

[sethjones348]

@wajda Here is the PR: AbsaOSS/spline#1339

Let me know if you have any concerns with the changes I've made!

[wajda]

Thanks for the reply! With that in mind I am planning to get the PR open against the spline repo ASAP. If I am able to get the PR open today or tomorrow, approximately how long do you think it would be before it would be released and available? Assuming the code looks good?

Let me review the current state. It's been a while since the last commit in Spline, the team has to switch to another project, so Spline isn't actively maintained at the moment. From what I remember the develop branch was more or less ready the last time I worked on it, but it was missing a few important thinks. Let me review it and I'll let you know.

[wajda]

Please expect 1.0.0 Spline server release this or next week the latest.