feat: use zstd compressed cars as blockstores #3149

aatifsyed · 2023-07-07T15:28:54Z

Summary of changes

Changes introduced in this pull request:

new forest snapshot compress ... compresses snapshots with many zstd frames.
forest snapshot validate ... now supports compressed snapshots prepared by the above.

Change checklist

I have performed a self-review of my own code,
I have made corresponding changes to the documentation,
I have added tests that prove my fix is effective or that my feature works (if possible),
I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

CLAassistant · 2023-07-11T11:14:04Z

All committers have signed the CLA.

CLAassistant · 2023-07-11T11:14:04Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

src/utils/mod.rs

…t-manyframe

LesnyRumcajs

LGTM

LesnyRumcajs · 2023-07-13T08:39:04Z

src/cli/subcommands/snapshot_cmd.rs

+                assert_eq!(
+                    unsigned_varint::io::read_usize(&[0xFD, 0x2F, 0xB5, 0x28][..]).unwrap(),
+                    6141,
+                );


Does it belong in the runtime? It seems like something we could check at compile time, though this read_usize probably is not const. Perhaps it would belong more in tests if anywhere?

The purpose of the assertion is to illustrate why we can't easily tell whether an archive is compressed or not.

LesnyRumcajs · 2023-07-13T08:40:50Z

src/cli/subcommands/snapshot_cmd.rs

It would be great to add some integration tests for this subcommand.

Yeah it would be great to add some bin tests under tests/

src/car_backed_blockstore.rs

hanabi1224 · 2023-07-13T11:36:13Z

.config/forest.dic

 BLS
 callee
 CAR/SM
+CARv1/SM


nit: I would just backquote terms like CARv1 to bypass spell checker

lemmih

Excellent PoC. Love that compressed files are faster than uncompressed.

aatifsyed force-pushed the aatifsyed/compressed-random-access-car branch 2 times, most recently from 0a8f2f3 to 3bdd327 Compare July 10, 2023 12:59

aatifsyed force-pushed the aatifsyed/compressed-random-access-car branch 4 times, most recently from 71c3069 to 3348fec Compare July 12, 2023 12:18

aatifsyed requested review from lemmih, elmattic and LesnyRumcajs and removed request for lemmih, LesnyRumcajs and elmattic July 12, 2023 12:18

aatifsyed changed the title ~~Aatifsyed/compressed random access car~~ feat: use zstd compressed cars are blockstores Jul 12, 2023

aatifsyed force-pushed the aatifsyed/compressed-random-access-car branch from 123eb3c to 4f01f5d Compare July 12, 2023 13:19

aatifsyed marked this pull request as ready for review July 12, 2023 13:19

aatifsyed requested review from LesnyRumcajs and hanabi1224 July 12, 2023 13:21

lemmih self-requested a review July 12, 2023 13:21

aatifsyed changed the title ~~feat: use zstd compressed cars are blockstores~~ feat: use zstd compressed cars as blockstores Jul 12, 2023

aatifsyed mentioned this pull request Jul 12, 2023

update doc comment for buffer size in car backed blockstore #3101

Closed

aatifsyed commented Jul 12, 2023

View reviewed changes

src/utils/mod.rs Outdated Show resolved Hide resolved

aatifsyed added 7 commits July 13, 2023 03:03

doc: comment buffer size for CarBackedBlockstore indexing

23a77cb

wip: compressed_random_access_car

2a7c44d

feat: compress each varint frame

fc3a7d0

feat: custom compression level

97e9424

feat: aggregate varint frames in zstd frames

069845c

fix: bound the compressed zstd frame size, not the uncompressed size

345664d

feat: count zstd frames

642f2ca

aatifsyed and others added 19 commits July 13, 2023 03:03

feat: zstd manyframing

2692080

wip: add benchmarks

4c5651d

wip: benchmarkable with hyperfine

fb94379

wip: fixup error types, max frame size

e4e6821

feat: uncompress each zstd frame

1d420ae

feat: uncompressed block store has read cache

444dda5

docs: reorganise and document

2aaf8b8

chore: remove unused deps and benches

ce3c5f6

run: rm benches/car-backed-blockstore.rs test-snapshots/chain4.car.zs…

7b662ce

…t-manyframe

chore: fixup tests

ed64334

fix: unrename car_backed_blockstore.rs

2cd2a91

feat: forest-cli snapshot compress

7f4e04d

feat: can validate maybe compressed blockstore

819cdde

run: rm src/bin/car-utils.rs

12b0a47

doc: when users should recompress

3117e2f

fix: remove DynBlockstore

0df07d4

fix: rebase and chores

50a2fc9

chore: spellcheck

ab858b6

docs: wording

758e582

LesnyRumcajs approved these changes Jul 13, 2023

View reviewed changes

aatifsyed commented Jul 13, 2023

View reviewed changes

src/car_backed_blockstore.rs Outdated Show resolved Hide resolved

aatifsyed force-pushed the aatifsyed/compressed-random-access-car branch 2 times, most recently from 18151a0 to 758e582 Compare July 13, 2023 09:53

hanabi1224 reviewed Jul 13, 2023

View reviewed changes

.config/forest.dic

BLS

callee

CAR/SM

CARv1/SM

Copy link

Contributor

hanabi1224 Jul 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I would just backquote terms like CARv1 to bypass spell checker

lemmih approved these changes Jul 14, 2023

View reviewed changes

lemmih added this pull request to the merge queue Jul 14, 2023

Merged via the queue into main with commit 63684b7 Jul 14, 2023

lemmih deleted the aatifsyed/compressed-random-access-car branch July 14, 2023 06:49

aatifsyed mentioned this pull request Jul 18, 2023

Forest CAR roadmap #3222

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: use zstd compressed cars as blockstores #3149

feat: use zstd compressed cars as blockstores #3149

Uh oh!

aatifsyed commented Jul 7, 2023 •

edited

Loading

Uh oh!

CLAassistant commented Jul 11, 2023 •

edited

Loading

Uh oh!

CLAassistant commented Jul 11, 2023

Uh oh!

Uh oh!

LesnyRumcajs left a comment

Uh oh!

LesnyRumcajs Jul 13, 2023

Uh oh!

lemmih Jul 13, 2023

Uh oh!

LesnyRumcajs Jul 13, 2023

Uh oh!

hanabi1224 Jul 13, 2023

Uh oh!

Uh oh!

hanabi1224 Jul 13, 2023

Uh oh!

lemmih left a comment

Uh oh!

Uh oh!

feat: use zstd compressed cars as blockstores #3149

feat: use zstd compressed cars as blockstores #3149

Uh oh!

Conversation

aatifsyed commented Jul 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Other information and links

Change checklist

Uh oh!

CLAassistant commented Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Jul 11, 2023

Uh oh!

Uh oh!

LesnyRumcajs left a comment

Choose a reason for hiding this comment

Uh oh!

LesnyRumcajs Jul 13, 2023

Choose a reason for hiding this comment

Uh oh!

lemmih Jul 13, 2023

Choose a reason for hiding this comment

Uh oh!

LesnyRumcajs Jul 13, 2023

Choose a reason for hiding this comment

Uh oh!

hanabi1224 Jul 13, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hanabi1224 Jul 13, 2023

Choose a reason for hiding this comment

Uh oh!

lemmih left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aatifsyed commented Jul 7, 2023 •

edited

Loading

CLAassistant commented Jul 11, 2023 •

edited

Loading