Frontend-Backend split

[ed255]

This PR builds on top of #243 which now has become outdated. I haven't removed the original PR yet because it can be useful for reviewing as it contains a much smaller diff.

The main change in this PR from the previous one is moving all the code into different crates:

• frontend name = "halo2_frontend": the library to write circuits, calculate witness and MockProver
• backend name = "halo2_backend": the library that implements the proof system
• middleware name = "halo2_middleware": the traits and structs used between frontend and backend
• common name = "halo2_common": shared parts between frontend and backend
• halo2_proofs name = "halo2_proofs": lightweight layer + re-exports to make it easy to update a halo2 dependency to this new version, without any API change

Ideally the common crate shouldn't exist; but there are many parts of the halo2 implementation that share structs in the frontend and backend. For example, the Polynomial type is used to calculate commitments in the backend, but also to hold the witness assignment during witness generation in the frontend. The plan is to move as many things as possible from common to either frontend or backend, ideally reaching a point where common disappears. But that takes some effort so I would request to leave common in this PR and iterate it further in other PRs.

Big changes that are not just moving code around in this PR:

• Introduce a new permutation::keygen::Assembly called AssemblyFront in common/src/plonk/permutation.rs to be used by the frontend. Related to [Preview - historical] Frontend-Backend split #243 (comment) . This type now just holds a vector of pairs of cells that need to be copied, without any further processing. All the mappings and cycles are calculated by the backend. This also required a change in the MockProver, as it now has direct access to pairs of copy cells, simplifying the error checking.
• Added "Privacy Scaling Explorations team" to the authors of the crates in all Cargo.toml. Should we also add some entry to include the community?
• Temporarily disabled thread-safe-region permutation Assembly in backend/src/plonk/permutation/keygen.rs. I plan to bring this back later. See [post frontend-backend split] Bring back thread-safe-region permutation Assembly #258
• Added ChallengeMid in middleware/src/circuit.rs and moved original Challenge to common/src/plonk/circuit.rs. This is because Challenge has a public API which returns Expression is is not middleware; so ChallengeMid is a lightweight version of Challenge that can be converted to Challenge to be used in the frontend/backend.
• Added ColumnMid in middleware/src/circuit.rs which follows the same reasoning as ChallengeMid, to not depend on Expression in middleware.

Note: I'm not sure what's the best way to review this PR, but I certainly think that looking at the entire diff is not the best approach! The majority of the changes are just moving things around, the actual changes can be reviewed in #243

Missing things

• Remove unnecessary dependencies from each Cargo.toml
• Remove the crate halo2_proofs_rm. This is just a copy of halo2 before the crate split, which I'm using for reference while I'm working on this.
• Uncomment all the allow(unused_code) and such. I added these lines to make progress faster; once everything builds and all tests pass, I'll work on that
  • Remove unnecessary imports
• Bring back pub(crate) where necessary. Since some types are now imported from a different crate, they have become pub from pub(crate). To make my life easier I made a lot of stuff just pub while moving things around. Once types and functions are settled, I need to add pub(crate) where necessary. [post frontend-backend split] Review pub types / fields #259
• Configure the lints like in the original halo2_proofs for the public facing crates:

#![cfg_attr(docsrs, feature(doc_cfg))]
// The actual lints we want to disable.
#![allow(clippy::op_ref, clippy::many_single_char_names)]
#![deny(rustdoc::broken_intra_doc_links)]
#![deny(missing_debug_implementations)]
#![deny(missing_docs)]
#![deny(unsafe_code)]

• Add getters (and setters) where necessary. I made many fields pub to make progress faster while I was figuring out where to put things; but the end goal is that user facing types don't expose internals if possible (following the original design of halo2) [post frontend-backend split] Review pub types / fields #259
• After all reviewing the getters, setters and bringing back pub(crate) where appropriate, write the documentation for the new public functions/types that don't have it. [post frontend-backend split] Review pub types / fields #259
• Fix the tests (in particular, many use imports will be broken)
• Fix the benchmarks (I haven't just checked them)
• Fix the code in the docs (the imports will most likely be broken)
• Pass clippy checks
• Move poly to backend [post frontend-backend split] Move poly to backend #257
• Bring back thread-safe-region permutation Assembly [post frontend-backend split] Bring back thread-safe-region permutation Assembly #258
• Review exposed types/functions in halo2_proofs [post frontend-backend split] Review exposed types/functions in halo2_proofs #260

[CPerezz]

The tmp removal of the thread-safe-region makes sense.
We should anyways fill an issue to re-address it after this is merged.

[ed255]

The tmp removal of the thread-safe-region makes sense. We should anyways fill an issue to re-address it after this is merged.

I created issues for all the tasks that need to be done after this is merged, I hope I didn't forget anything! This is the issue for the thread-safe-region: #258

[CPerezz]

The tmp removal of the thread-safe-region makes sense. We should anyways fill an issue to re-address it after this is merged.

I created issues for all the tasks that need to be done after this is merged, I hope I didn't forget anything! This is the issue for the thread-safe-region: #258

Thanks for putting all of these together!!!

[adria0]

@ed255 Why we need middleware plonk module if is not used by CompiledCircuit? ( AFAIU the main goal of middleware is to hold a CompiledCircuit )

[ed255]

@ed255 Why we need middleware plonk module if is not used by CompiledCircuit? ( AFAIU the main goal of middleware is to hold a CompiledCircuit )

The plonk::Assigned type is used to encode the witness that the frontend passes to the backend via the backend method:

halo2/halo2_backend/src/plonk/prover.rs

Lines 258 to 267 in 25666a4

	/// Commit the `witness` at `phase` and return the challenges after `phase`.
	#[allow(clippy::type_complexity)]
	pub fn commit_phase(
	&mut self,
	phase: u8,
	witness: Vec<Vec<Option<Vec<Assigned<Scheme::Scalar>>>>>,
	) -> Result<HashMap<usize, Scheme::Scalar>, Error>
	where
	Scheme::Scalar: WithSmallOrderMulGroup<3> + FromUniformBytes<64>,
	{

The reason for having Vec<Assigned<Scheme::Scalar>> instead of Vec<Scheme::Scalar> is that currently the halo2 frontend sometimes generates divisions or inverted field elements as witness; and it's more efficient to do the inversion in a batch; and that's currently being done in the backend. It could also be done in the frontend, and then we would be able to remove the Assigned from the middleware. Perhaps that would be better!

Update: I've removed the Assigned from the prover API here 137b46e and removed Assigned from middleware here 7162953

[CPerezz]

LGTM!!

Awesome work mate!!! <3

[adria0]

Amazing job, @ed255 !