feat: add resource-scoped jobs plugin for Databricks Lakeflow Jobs

[keugenek]

Summary

Resource-scoped jobs plugin following the files plugin pattern. Jobs are configured as named resources discovered from environment variables at startup.

Design

• Resource-scoped: Only configured jobs are accessible — this is not an open SDK wrapper
• Env-var discovery: Jobs are discovered from DATABRICKS_JOB_<KEY> env vars (e.g. DATABRICKS_JOB_ETL=123)
• Single-job shorthand: DATABRICKS_JOB_ID maps to the "default" key
• Manifest declares job resources with CAN_MANAGE_RUN permission
• Works with databricks apps init --features jobs

API

// Trigger a configured job
const { run_id } = await appkit.jobs("etl").runNow();

// Trigger and wait for completion
const run = await appkit.jobs("etl").runNowAndWait();

// OBO access
await appkit.jobs("etl").asUser(req).runNow();

// List recent runs
const runs = await appkit.jobs("etl").listRuns({ limit: 10 });

// Single-job shorthand
await appkit.jobs("default").runNow();

Files changed

• plugins/jobs/manifest.json — declares job resource with CAN_MANAGE_RUN permission
• plugins/jobs/types.ts — JobAPI, JobHandle, JobsExport, IJobsConfig types
• plugins/jobs/plugin.ts — JobsPlugin with discoverJobs(), getResourceRequirements(), resource-scoped createJobAPI()
• plugins/jobs/index.ts — barrel exports
• connectors/jobs/client.ts — listRuns now respects limit parameter
• plugins/jobs/tests/plugin.test.ts — 32 tests covering discovery, resource requirements, exports, OBO, multi-job, and auto-fill

Documentation safety checklist

• Examples use least-privilege permissions
• Sensitive values are obfuscated
• No insecure patterns introduced

[atilafassina]

Found also 2 potential type mismatches with protobuf

1. @bufbuild/protobuf toJson() defaults to camelCase. The useProtoFieldName: true option is needed for snake_case as documented.

2. Partial<MessageShape<T>> allows setting properties to undefined explicitly, whereas PartialMessage<T> from protobuf properly handles nested partial messages and enum defaults.

[pkosiec]

@keugenek Took a quick high-level look and I think we should take a step back and think about the user flow for this plugin. Right now it is a light wrapper over the JS SDK client and effectively re-exports the API. I don't see any significant benefits to use plugin over the SDK :thinking_face:

For all the current plugins, a user can do databricks apps init and select a plugin and interactively all the required resources (or pass them with a flag), and then we put the selection in databricks.yml and .env for seamless local development and zero-config deployment.

I'd assume it should work similarly with the jobs plugin - which means, it would need to have a job-oriented API (multiple jobs can be supported too, but in the interactive selection, we'd support one - similarly as in the model serving API).

So I think we need to agree on the API first, and then we need to have template changes, and test the existing apps init flow.

Let me know if that makes sense! Feel free to dismiss my review if I'd be OOO, but I'd really love to rework the plugin to be more user-oriented. Thank you!

Dismissing after the last commit

[pkosiec]

Thanks very much @keugenek! I didn't check the PR details but the high-level changes are exactly what I was looking for. Dismissed my review before going offline to avoid blocking merge 👍

Thanks again for making those changes!

[atilafassina]

Reopening from upstream branch to fix CI secrets access (fork PRs can't access JFrog token)