Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce GPBFT message compression #786

Closed
masih opened this issue Dec 11, 2024 · 1 comment · Fixed by #793
Closed

Introduce GPBFT message compression #786

masih opened this issue Dec 11, 2024 · 1 comment · Fixed by #793
Assignees
@masih
Milestone
M3: Mainnet Activ...

Comments

@masih
Copy link
Member

masih commented Dec 11, 2024

Based on the experiment performed as part of passive testing network filecoin/46 compressed GPBFT messages are on average 60% smaller. This makes compression of messages a worthwhile exercise considering the small footprint of compression on resource consumption. See: #757 (comment)

Ideally, the compression introduced as part of this work should be backward compatible:

  • the implementation should continue to be able to interpret uncompressed messages in order to facilitate as much participation as possible since not all nodes will be upgraded to the latest.
  • Eventually we can remove this backward compatibility entirely and only accept compressed messages (similar to the removal of dynamic manifest).
@github-project-automation github-project-automation bot moved this to Todo in F3 Dec 11, 2024
@masih masih self-assigned this Dec 11, 2024
@masih
Copy link
Member Author

masih commented Dec 11, 2024

On further discussion, backward compatibility might not be worth it if enough nodes with power don't upgrade. Opting to make it optional in manifest as on or off thing.

@masih masih moved this from Todo to In progress in F3 Dec 11, 2024
masih added a commit that referenced this issue Dec 12, 2024
@masih
Implement GPBFT message compression using zstd
ea3ad13 
Add the ability to compress GPBFT messages controllable via manifest.
Implement benchmarks to compare vanilla CBOR and ZSTD encoding.

Basic local run:
```
BenchmarkCborEncoding-12    	   47173	     25491 ns/op	  135409 B/op	      87 allocs/op
BenchmarkCborDecoding-12    	   64550	     18078 ns/op	   61728 B/op	     209 allocs/op
BenchmarkZstdEncoding-12    	   29061	     41489 ns/op	  193455 B/op	      88 allocs/op
BenchmarkZstdDecoding-12    	   66172	     17924 ns/op	  176517 B/op	     211 allocs/op
```

Fixes #786
@masih masih mentioned this issue Dec 12, 2024
Merged
masih added a commit that referenced this issue Dec 13, 2024
@masih
Implement GPBFT message compression using zstd
6093105 
Add the ability to compress GPBFT messages controllable via manifest.
Implement benchmarks to compare vanilla CBOR and ZSTD encoding.

Basic local run:
```
BenchmarkCborEncoding-12    	   47173	     25491 ns/op	  135409 B/op	      87 allocs/op
BenchmarkCborDecoding-12    	   64550	     18078 ns/op	   61728 B/op	     209 allocs/op
BenchmarkZstdEncoding-12    	   29061	     41489 ns/op	  193455 B/op	      88 allocs/op
BenchmarkZstdDecoding-12    	   66172	     17924 ns/op	  176517 B/op	     211 allocs/op
```

Fixes #786
masih added a commit that referenced this issue Dec 13, 2024
@masih
Implement GPBFT message compression using zstd
1578660 
Add the ability to compress GPBFT messages controllable via manifest.
Implement benchmarks to compare vanilla CBOR and ZSTD encoding.

Basic local run:
```
BenchmarkCborEncoding-12    	   47173	     25491 ns/op	  135409 B/op	      87 allocs/op
BenchmarkCborDecoding-12    	   64550	     18078 ns/op	   61728 B/op	     209 allocs/op
BenchmarkZstdEncoding-12    	   29061	     41489 ns/op	  193455 B/op	      88 allocs/op
BenchmarkZstdDecoding-12    	   66172	     17924 ns/op	  176517 B/op	     211 allocs/op
```

Fixes #786
github-merge-queue bot pushed a commit that referenced this issue Dec 13, 2024
@masih
Implement GPBFT message compression using zstd (#793)
b04ede1 
Add the ability to compress GPBFT messages controllable via manifest.
Implement benchmarks to compare vanilla CBOR and ZSTD encoding.

Basic local run:
```
BenchmarkCborEncoding-12    	   47173	     25491 ns/op	  135409 B/op	      87 allocs/op
BenchmarkCborDecoding-12    	   64550	     18078 ns/op	   61728 B/op	     209 allocs/op
BenchmarkZstdEncoding-12    	   29061	     41489 ns/op	  193455 B/op	      88 allocs/op
BenchmarkZstdDecoding-12    	   66172	     17924 ns/op	  176517 B/op	     211 allocs/op
```

Fixes #786
@github-project-automation github-project-automation bot moved this from In progress to Done in F3 Dec 13, 2024
@BigLep BigLep mentioned this issue Dec 16, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@masih masih

Labels
None yet
Projects
F3
Status: Done
Milestone
M3: Mainnet Activation
Development

Successfully merging a pull request may close this issue.

Implement GPBFT message compression using zstd filecoin-project/go-f3
1 participant
@masih