Task dependencies?

[alecmev]

How do I automatically chain tasks/commands based on dependencies? I want to replicate Turbo's dependent workspace task dependencies. The docs don't really delve into that.

[paularmstrong]

This feature should not be necessary when using source dependencies.

Generally speaking, each workspace should be able to validate and build independent of its dependencies needing to build and validate.

The only area this should matter is ensuring packages are published when necessary, which the one change version and one change publish commands handle internally.

[alecmev]

I do like the "Build step? What build step?" aspect of source dependencies, and tried them in the past, but some bike sheds aside, we're just dealing with more than TypeScript. Examples:

1. We have our own ESLint configuration, as a TypeScript package. ESLint knows nothing of TypeScript. Could use ts-node, but we purposefully moved away from it because it kept breaking even with minor releases of Node.js, along with having a bunch of other annoyances. This creates a dependency, @acme/eslint-config needs to be built before running ESLint.

2. We have @acme/app-1 and @acme/app-2, both of which depend on @acme/asset-1 and @acme/asset-2, both of which depend on @acme/asset-builder. Even if we create a command that prepares all the prerequisites for the apps, and reuse it between the apps, we still end up building things twice. And that command doesn't feel trivial.

3. We have @acme/electron-app that depends on @acme/next-js-app. I suppose this could be expressed as a special sequential task, but what if we keep adding relationships like this? More tasks?

4. As I mentioned in Caching? #669, we're also interested in code generation. Ignoring caching, it feels a bit daunting to have to write a custom command/plugin for every case where we might need code generation (GraphQL, Prisma, etc.). I don't even know where to begin, what would such a command/task look like, that builds @acme/generated-code before its dependants try to import it? And updates it when its dependencies or sources change?

5. We might need to create WebAssembly modules or native Node.js addons in the future, certainly requiring a dedicated build step, even if still authored in TypeScript.

The above is about building, but I'm also very much interested in watching, and it feels like even if all of this is solved for building, then the watching story will need a separate more complicated solution.

I understand that not having Turbo-/Wireit-like pipelines is a conscious design choice, but I was thinking there would be an alternative story, maybe code-based instead of being declarative.

I like oneRepo, but my intuition tells me that we probably need something else. Do you agree, or are the "monster commands" I'm imagining not going to materialize? 😄 Just in case, it's not your responsibility to evaluate oneRepo for us, of course. I'm asking this more for others, so that when they read this they can see if this applies to them or not.

Regardless, thank you for being so helpful these last few days!

[paularmstrong]

You have options… speaking frankly, without knowing the details, this setup sounds like a monster and I don't think I'd enjoy maintaining it :)

1. How fast is this? I would create a build step for the package and auto-run it on post-merge so that it's always up to date for developers.

   Assuming your Workspace name is lint-config. Create a onerepo.config.ts in that Workspace's root:

   export const config {
     tasks: {
       'post-merge': { parallel: ['$0 ws lint-config build'] }
     },
   };

   Alternatively… since ESLint is not directly compatible with using TypeScript… don't use typescript 🤷‍♂️ I generally try to avoid doing things in TypeScript where it's not first-party supported.

2. How does source dependencies not avoid this? Seems like there may be something in your setup being done unnecessarily.

3. I would put this directly in the electron app's build/run custom commands. eg, create a step within those commands to run the appropriate build/run commands.

4. Like no. 3, I would do these only when necessary. You can create top-level commands to do this if all of your Workspaces are written the same way with the same requirements (which IMO, they should be 😉). Alternatively, it almost sounds like that generated code should be checked into the repo, if it's depended on so heavily that you can't run anything without it. I'd probably add a pre-commit task for this.

   For example, oneRepo's docs force-generate on pre-commit. They don't actually need to and could be done at build time, but they're done ahead particularly because it actually is helpful to review what is generated during code review.

   Specifically for graphql – again, I would require this be a pre-commit hook and commit the code. You can mark it as read-only (eg in vscode and by using file.writeSafe and verify the signature in CI)

   Moving the generation time to a specific point when you know it needs to be done alleviates potential issues with varying environments, enables code review, prevents re-running across every environment, and prevents other developers of needing to know anything of the dependency.

5. This is too hypothetical and out of my scope to comment on

Generally speaking, I've never needed, and also completely avoided, the dependent command workflow chains across Workspaces. They become a spiderweb of internal and specialized knowledge that become difficult to change as requirements change. They also require DSLs or configurations with other specialized knowledge (sidebar: honestly oneRepo's tasks is even far too much in this direction and I'd love an alternative).

oneRepo puts an emphasis on writing commands and particularly because of the complex nature of the development lifecycle, they're fully testable. I've seen test suites on dev lifecycle tasks save hours and days worth of potentially breakages.

---

One thing that I've done in the past is to create root-level start and build commands. These can automate all of this for you and avoid anyone creating a new Workspace the need to know whether they need to include pre/parallel/post steps in their lifecycles at all.

Let's say we have the following repository:

├── apps
│   ├── foo     # depends on graphql
│   ├── bar
│   └── qux     # depends on graphql
└── modules
    └── graphql # needs to be generated

We have two apps (foo, qux) that depend on the graphql workspace, while another (bar) does not. We've already declared the dependency in package.json, so maybe we can avoid needing to do any other configuration for these apps (and any future apps).

Let's consider a build command as the minimal case:

export const command = 'build';

export const description = 'Build any workspace across the monorepo';

type Argv = { workspace: string };

export const builder: Builder<Argv> = (yargs) => 
  yargs.usage(`$0 ${command}`)
    .option('workspace', { type: ', alias: 'w', description: 'Workspace name to build', demandOption: true });

export const handler: Handler<Argv> = async (argv, { graph }) => {
  const { workspace: wsName } = argv;
  const workspace = graph.getByName(wsName);
  const graphql = graph.getByName('graphql');

  // If the workspace in question requires `graphql`, we know it needs to be generated first
  if (graph.dependencies(workspace).includes(graphql)) {
    await run({
      name: 'Generate GraphQL code',
      // nb: argv[0] is always `node`, argv[1] is the entrypoint to the oneRepo CLI that was 
      // used to invoke the current command. This ensures that the correct reference is always used.
      cmd: process.argv[1], 
      args: ['ws', 'graphql', 'build']
    });
  }

  // Check for workspace-local build command and run that, then eject
  // See NOTE
  if (await file.exists(ws.resolve('commands/build.ts')) {
    await run({
      name: `Build ${wsName}`
      cmd: process.argv[1],
      args: ['build']
      opts: { cwd: ws.location }
    });
    return;  
  }

  // Check for a `vite` config and build with that
  if (await file.exists(ws.resolve('vite.config.ts')) {
    await run({
      name: `Build ${wsName} with Vite`
      cmd: 'vite',
      args: ['build']
      opts: { cwd: ws.location }
    });
    return;
  }

  // Check other requirements and build depending on how they are meant to be
};

This enables us to always be able to run the same command and trace its flow for every workspace without needing to jump from one setup to another. But also still allows us to completely eject out of the majority case if we have specific needs, while still being able to build from the same command.

one build -w foo

 ┌ Generate GraphQL code
 └ ✔ 27ms
 ┌ Build foo with Vite
 └ ✔ 34ms

one build -w bar

 ┌ Build bar with Electron
 └ ✔ 34ms

This could be refactored to use builders.withWorkspaces in the build argument dependencies and then loop over every Workspace. We could go even further and use builders.withAffected to enable auto-building all workspaces that have been modified or will be affected by current changes.

Note

There's an experimental/internal way currently to run command handlers in the same process as the current command instead of subprocessing. This enables better log output and reduces resource usage, but it needs fleshing out a little more before it is made reusable and exposed in the API, particularly around how log output is captured and buffered: #674

Used in the tasks handler:

onerepo/modules/onerepo/src/core/tasks/tasks.ts

Lines 286 to 302 in 9534837

	fn = async () => {
	const step = logger.createStep(name, { writePrefixes: false });
	const subLogger = bufferSubLogger(step);
	const { yargs } = await setup({
	require: jiti,
	root: graph.root.location,
	config,
	yargs: createYargs([...args, ...passthrough], undefined, jiti),
	corePlugins,
	logger: subLogger.logger,
	});
	await yargs.parse();
	await subLogger.end();
	await step.end();
	return ['', ''];
	};