feat: Add esbuild plugin for auto instrumentation (https://github.com/open-telemetry/opentelemetry-js/issues/4174)

[drewcorlin1]

Which problem is this PR solving?

Providing an esbuild plugin for auto instrumentation open-telemetry/opentelemetry-js#4174

Short description of the changes

This change adds a proof of concept for an esbuid plugin to auto instrument node JS apps. The approach at a high level is:

1. Configure the packages + versions we care about intercepting + adding the instrumentation to (metapackages/esbuild-plugin-node/src/config), this leverages the getModuleDefinitions() methods on instrumentation classes.
2. For matching packages, load the existing source, wrapping it in an IIFE and then getting the exported contents of that module
3. Instantiating the relevant plugin (ie FastifyInstrumentation) and passing the existing module exports to the relevant InstrumentationNodeModuleDefinition's patch function (this gets a little more tricky for things like the AWS SDK that has many packages going on, but largely follows the same flow as the existing auto instrumentation).

There are a number of things worth noting

• This approach ignores built in modules (ie instrumenting fs or fs/promises). The current assumption is that you will use existing auto-instrumentation from this repo for the built-in functions since they are not subject to the same issues with bundlers and the require()-intercepting based approach of patching third party modules (the problem is outlined in more detail in Support instrumentation of modules via bundler plugins opentelemetry-js#4174).
• Configuration for the plugins is limited, notably in that functions will not be passed to the underlying plugins.

I'm hoping to discuss any concerns with the points noted above or anything else before expanding support to all modules in this repo. Let me know if I can provide any additional information!

The README and some descriptions are slightly misleading until these points are discussed.

Example

The plugin would be used as follows (assuming my app is in src/server.ts).

import { openTelemetryPlugin } from '@opentelemetry/esbuild-plugin-node';
import { build } from 'esbuild';

build({
  entryPoints: ['src/server.ts'],
  bundle: true,
  outfile: 'dist/server.js',
  target: 'node20',
  platform: 'node',
  sourcemap: true,
  plugins: [
    openTelemetryPlugin({
      externalModules: ['#node-web-compat'], // This package also causes jest issues
      pathPrefixesToIgnore: ['~/'], // We configure this to alias `src/` via `tsconfig#compilerOptions.paths`
      instrumentationConfig: {
        '@opentelemetry/instrumentation-aws-sdk': {
          suppressInternalInstrumentation: true
        },
      },
    }),
  ],
})

[drewcorlin1]

Alternatively, maybe we could just not allow passing functions to the configs (like logHook) and only allow simple options (strings, booleans, numbers, etc) since this would likely just trip people up.

[drewcorlin1]

I removed config parameters that are functions, still need to update types accordingly

[drewcorlin1]

@blumamir @haddasbronfman could you take a look when you get a chance/let me know if I should be tagging someone else? Thanks!

[drewcorlin1]

@pichlermarc apologies for the random tag, but I'm not sure who else to tag to get a review/discussion started on this. Could you help me out with that?

[pichlermarc]

Hi @drewcorlin1, thanks for reaching out. 🙂
This does look like a really cool feature, I have not had the time yet to look into this in detail, a few notes though:

• keep in mind that with any new component like this we'll also need people to act as code owners for it, for that please add the new package to .github/component_owners.yml
  • ideally we'd need two or more people to maintain this new package, usually that's the author, as well as another person that's familiar with the tooling and the new plugin.
• I think before accepting this new component we should discuss this with the other SIG members as I expect adding this to entail significant additional maintenance effort.
  • I will add this to the agenda for our next SIG meeting (Jan 04, 2024 Jan 03, 2024) to discuss it there. Feel free to join if you can. You can find the link to the agenda here https://github.com/open-telemetry/community#implementation-sigs (under JavaScript: SDK), the link to the meeting is in the linked Google Doc 🙂

[drewcorlin1]

@pichlermarc thank you! I'm happy to join the SIG meeting. Is it Jan 3 or Jan 4 though, that link says meetings are on Wednesdays and this link says the Javascript SIG is on the 3rd?

[trentm]

Is it Jan 3 or Jan 4 though

Jan 3rd is Wednesday, no?

[drewcorlin1]

Is it Jan 3 or Jan 4 though

Jan 3rd is Wednesday, no?

Yep, just that his message said the 4th. I'll plan on coming to the Javascript SIG meeting on the 3rd

[trentm]

Yep, just that his message said the 4th

Ah, duh. I misread. Yah, it'll be the 3rd. Or if not, you and I can hang out for a few minutes then come back the next day. :)

[pichlermarc]

@drewcorlin1 Ah, yes 4th was a typo - it's January 3rd; sorry about the confusion.

[drewcorlin1]

@trentm @pichlermarc it seems like there is clear demand from others for this feature. I know there are these open PRs, but since they were opened a few months ago there's been more movement. Can we get some support in trying to get this feature merged?

• feat: support 'otel:bundle:load' diagnostics_channel message for instrumentation in bundles opentelemetry-js#4818
• feat(esbuild-plugin): esbuild plugin to prepare bundled modules for instrumentation #2295

[github-actions]

This package does not have an assigned component owner and is considered unmaintained. As such this package is in feature-freeze and this PR will be closed with 14 days unless a new owner or a sponsor (a member of @open-telemetry/javascript-approvers) for the feature is found. It is the responsibility of the author to find a sponsor for this feature.
Are you familiar with this package? Consider becoming a component owner.