Zeromorph and HyperKZG improvement (Arecibo backports)

[huitseeker]

What's in this PR?

This implements the Zeromorph PCS as a Nova PCS, along with :

• some performance improvements in the HyperKZG PCS (see comparative benchmarks below ¹),
• a mutualisation of the commitment scheme, which is common to Zeromorph and HyperKZG,

This is the work of a team at Lurk Lab, including @adr1anh, @emmorais, @huitseeker, @mpenciak, @storojs72 and @tchataigner.

Absolute Benchmarks

Complete bench output at https://gist.github.com/huitseeker/8401e4be3ef1430277d3fee6cd84efc4

Comparative benchmark

To gauge the impact of this PR on HyperKZG (a component, but not the major point of this PR), we can ask: how does this PR (at e34f0aa) compare to main (at 55a55dd) or main before #299 and #300 (at eab053b)?

group                                                                           main                     main_before_299_300                    zeromorp_and_kzg_pr
-----                                                                           ----                     -------------------                    -------------------
PCS-Proving 10/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/10            7.7ms                                   8.2ms                                  7.5ms
PCS-Proving 10/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/10                                                                                           7.6ms
PCS-Proving 12/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/12           18.5ms                                  22.5ms                                 18.4ms
PCS-Proving 12/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/12                                                                                          17.9ms
PCS-Proving 14/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/14           52.6ms                                  57.6ms                                 50.5ms
PCS-Proving 14/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/14                                                                                          44.6ms
PCS-Proving 16/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/16          167.7ms                                 181.2ms                                162.3ms
PCS-Proving 16/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/16                                                                                         131.8ms
PCS-Proving 18/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/18          580.6ms                                 575.3ms                                495.3ms
PCS-Proving 18/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/18                                                                                         402.8ms
PCS-Proving 20/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/20         1790.0ms                                    2.0s                               1725.2ms
PCS-Proving 20/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/20                                                                                        1330.8ms
PCS-Verifying 10/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/10       1963.3µs                                   2.3ms                               1458.2µs
PCS-Verifying 10/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/10                                                                                      1513.6µs
PCS-Verifying 12/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/12          2.2ms                                   2.4ms                               1732.4µs
PCS-Verifying 12/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/12                                                                                      1629.7µs
PCS-Verifying 14/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/14          2.1ms                                   2.4ms                               1690.0µs
PCS-Verifying 14/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/14                                                                                      1607.4µs
PCS-Verifying 16/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/16          2.1ms                                   2.4ms                               1687.9µs
PCS-Verifying 16/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/16                                                                                      1696.3µs
PCS-Verifying 18/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/18          2.1ms                                   2.5ms                               1751.8µs
PCS-Verifying 18/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/18                                                                                      1738.2µs
PCS-Verifying 20/hyperkzg::EvaluationEngine<Bn256, Bn256EngineKZG>/20          2.1ms                                   2.5ms                               1775.7µs
PCS-Verifying 20/non_hiding_zeromorph::ZMPCS<Bn256, Bn256EngineZM>/20                                                                                      1783.8µs

Provenance

This is a port of several Arecibo PRs (click to unfold)

• feat: implement Zeromorph lurk-lang/arecibo#145
• Add MLKZG support (Nova forward port) lurk-lang/arecibo#172
• chore: Remove redundant absorptions in MLKZG implementation lurk-lang/arecibo#206
• Fix mlkzg::EvaluationArgument fields lurk-lang/arecibo#207
• Test vectors & benchmark for PCS lurk-lang/arecibo#222
• Refactor 'kzg_verify_batch' closure in MLKZG verification lurk-lang/arecibo#231
• Parallel computation of Pi polynomials lurk-lang/arecibo#233
• Optimising proving by avoiding recomputing final Pi polynomial in MLKZG lurk-lang/arecibo#242

Footnotes

1. PRs Rename mlkzg and remove redundant transcript absorbs #299 and Optimizations to KZG commitment scheme #300 also include work initially developed in Arecibo as part of this effort. ↩

[srinathsetty]

Hi @adr1anh, @emmorais, @huitseeker, @mpenciak, @storojs72 and @tchataigner,

Thanks for the PR! For simplicity, could we please unfold this PR into several PRs?

(1) Remaining improvements to HyperKZG (i.e., those not covered by #299 and #300) + PCS benchmark. From my quick skim at this PR, this appears to be improvements to KZG verifier to compute a multipairing instead of two separate pairings. Hopefully, we can merge this quickly as it should be a self-contained and localized change.

(2) Refactoring (univariate) KZG out of HyperKZG
(3) ZeroMorph on top of (univariate) KZG.

On items (2) and (3), I'm not yet sure of the exact shape of (2) or (3) because ZeroMorph and HyperKZG have different SRS requirements. Also, it is not yet clear whether it makes sense to maintain both ZeroMorph and HyperKZG because of the following reasons:

• They have the same commitment costs;
• ZM has slightly better proving costs as shown in your benchmarks (though HyperKZG hasn't been fully optimized, so this point may change in the near future), but it is unclear how much of that translates to improved proving times in ppsnark as PCS evaluation is invoked only once whereas commitment is invoked an order of magnitude more times; and
• It seems like ZM inherently needs a bigger SRS.

In the absence of ZM, it may actually be simpler to have HyperKZG in the current form (i.e., we may not need item (2)), for easier auditability, simpler code, etc.

[huitseeker]

On items (2) and (3), I'm not yet sure of the exact shape of (2) or (3) because ZeroMorph and HyperKZG have different SRS requirements. Also, it is not yet clear whether it makes sense to maintain both ZeroMorph and HyperKZG because of the following reasons:
(...)
It seems like ZM inherently needs a bigger SRS.

To provide more clarity on the SRS: the PR already demonstrates how to use the existing Nova traits to overcome this conceptual difficulty.

The KZG CommitmentTrait defines how to commit using the KZG commitment scheme, which is exactly identical (in fact, the same code path) between ZM & HyperKZG. That CommitmentTrait uses a CommitmentKey that contains a vector of G1 points, and a vector of G2 points, and the commitment scheme imposes no particular constraint on the length of that vector of G2 points, as they are not needed for commitments.

However, ZM and HyperKZG do not implement the EvaluationEngineTrait in the same way. It is there, in the implementation of setup(ck: &CommitmentKey<E>), that the length of the vector of G2 points is inspected, and, in the case of ZM, fails if the amount of bases supported in the SRS is unsuitable for the ZM scheme. As you know, this setup is called as part of the setup of the CompressedSNARK, which is where the user-facing failure would occur if the SRS characteristics were found to be insufficient.

In other words, ZM does not carry an incremental maintenance cost beyond the specifics of its EvaluationEngine implementation (~800 LoC, including 300 LoC for tests).

[srinathsetty]

Thanks for the clarifications @huitseeker! It makes sense that Item#3 is not significantly more complex.

If ZM's benefits aren't significant (needs more examination!), then we also can avoid the (conceptual) complexity introduced from factoring out (univariate) KZG (item#2), which unlike the current hyperkzg.rs, spreads the code over multiple modules and some abstractions may be unnecessary if there's only going to be one KZG-based scheme. To give more time to understand these trade-offs, it may still make it easy to unfold item#1. What do you think?

[huitseeker]

I've done a pass to minimize disturbance to the Nova module organization, there no longer is a non_hiding_kzg file.

Item (1) as asked above seems to consist of:

• this PR,
• minus
  • the non_hiding_zeromorph file,
  • the declaration of the Bn256EngineZM in provider/mod.rs lines 67-79,
  • the two lines mentioning Bn256EngineZM in the benches (one for imports, one in asset definition),
  • the couple of lines where Bn256EngineZM has been added alongside other options in generic tests for the Compressed SNARK in src/lib.rs.

[srinathsetty]

Thanks @huitseeker! For (1), It think it may be easy to create a new lean PR with just the following changes:

• benches/pcs.rs
• modifications to src/provider/hyperkzg.rs with the improved pairing check (but without univariate KZG factored out):

let pairing_inputs = [
        (&(-L).to_affine(), &E::G2Prepared::from(vk.h)),
        (&R.to_affine(), &E::G2Prepared::from(vk.beta_h)),
      ];

let pairing_result = E::multi_miller_loop(&pairing_inputs).final_exponentiation();
pairing_result.is_identity().into()

[srinathsetty]

Is this still being worked on? I ask because I worry about divergence with main, making it harder to merge this down the line.