design: A unified optimizer interface

[mgree]

We have talked off and on about unifying the interface to the optimizer. This design doc discusses why that would be useful and what it would look like.

Markdown rendering of

• the PR version of the design doc.
• the merged version of the design doc.

Motivation

• This PR refactors existing code.

Tips for reviewer

Conspicuously absent: the actual proposed interface. I think this is best discussed at the next on-site.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered.
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
• This PR includes the following user-facing behavior changes:

[aalexandrov]

Adding a comment to put some extra items that I would like to discuss and nail in this doc:

1. What kinds of context to we have (mutable vs read-only)?
2. Who initializes and owns them (Coordinator, DataflowBuilder, ...)?

For example, we want to communicate notices the optimizer back to the caller (the Coordinator). For me, the ideal way to do this is to say that:

1. The Coordinator initializes a piece of mutable context of type Vec<OptimizerNotice> that it can pass down.
2. The various optimizer stages then populate this context.
3. The Coordinator than interprets the context (e.g., by forwarding any notices to the user or by saving them on the side so we can expose them in the web console).

P.S.: Sadly, even if we had this the fact that HIR ⇒ MIR lowering and local MIR ⇒ MIR optimization happen before we enter the Coordinator leaves our hands a bit tied.

[aalexandrov]

@mjibson / @ggevay / @mgree I've pushed a new version of the design doc along with an MVP PR in #22115.

If you agree with the approach and the proposed API I will start forking off separate PRs for each statement type along the lines suggested in the doc and demonstrated in #22115.

[aalexandrov]

BTW this is the case even now because "execute SELECT" statements produce a dataflow with an exported IndexDesc that we don't create in the "explain SELECT".

materialize/src/adapter/src/coord/sequencer/inner.rs

Lines 2319 to 2326 in 10ae7a9

	dataflow.export_index(
	index_id,
	IndexDesc {
	on_id: view_id,
	key: key.clone(),
	},
	typ.clone(),
	);

It's not something that currently affects the EXPLAIN output, but still...

[maddyblue]

So well written and explained. This looks great! Excited to see this happen.

[teskje]

LGTM!

[ggevay]

Looks great!

[mgree]

Thanks for all this work and extra detail!