Implementing pack-gen

[GaurangTandon]

Hi, as far I understand, the current repository doesn't implement pack generation. (The third issue #3 is still open). So, before pack-send, we obviously need to generate packs first.

So, to begin implemention, I start by understanding the original C code, starting with pack-objects.c? I have read the Git Internals part of the ProGit book, but it felt surface level in this context.

So, as apparent from the README, I should likely first look at these three tasks, starting with the first:

• create new pack
• create 'thin' pack
• Add support for zlib-ng for 2.5x compression performance and 20% faster decompression

Ideally, this functionality would be implemented in git-odb/src/pack/data/encode.rs.

Is this overall understanding correct?

[Byron]

Yes, that's a decent starting point. The ZLIB upgrade is certainly orthogonal, and in order to warm up with the codebase that one could even be done first.

Furthermore I recommend to divide create new pack into:

• create base objects (full objects entirely stored in the back without nothing more than zlib compression)
• create 'ref' objects, which are exactly what's used in 'thin' packs. It's nothing more than a reference to another object by SHA1 and trivial.
• finally, one could try to create deltified objects, this algorithm is where the dark magic happens and would certainly be the crown jewel of this crate if done well. This is where packs become packs with a 60x compression ratio (or more).

Please let me know if there are any questions along the way and I will try my best to answer them timely. I also offer a video call to get you started more quickly - for that just reach out to me on keybase.io or via email.

The best of success!

[Byron]

By the way, this discussion may be helpful for you as I keep posting new insights there regularly. The tasks file was updated with a new train of work which involves pack generation via implementing the server side of a pull.

[GaurangTandon]

Hey @Byron I was really hoping to work on this issue. Initially I dug around a bit, and studied some resources to understanding git pack generation properly, and was looking into the codebase. But since then I am sorry I have been rerally busy, and will remain so till next month end :( (I wasn't anticipating this change)

If you were hoping to get this issue done earlier than the next month, feel free to go ahead and start implementing it. If not, then please wait for me as I join forces next month :)

[Byron]

Thanks for the update :) - I do hope to get started earlier, but as you will see from tasks.md there is enough work across multiple layers to not step on each others toes. In any case, I would be happy to help you get started whenever you are ready to join.

[Byron]

Please note that (a lot has happened and) I have begun with the first steps of pack generation, here is the tracking ticket.

If you think you can join with these efforts, please feel free to rebase this PR, otherwise it's probably ok to close it.

[GaurangTandon]

Yeah, I should not keep you waiting any longer. I really wanted to contribute at least something to this project, as git and rust are two of the favorite technologies to me. I can assure you I had tried my best to carve out time for this, but I am sorry I was not able to commit anything :(

Wish you good luck!

[Byron]

There is no need to apologize for anything, sometimes plans don't pan out.

For the next time, you could have a look at other ways to contribute, many of which are small and can certainly be completed in one sitting or two.

Good luck to you too!