Skip to content

Implementing pack-gen #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

GaurangTandon
Copy link

@GaurangTandon GaurangTandon commented Feb 10, 2021

Hi, as far I understand, the current repository doesn't implement pack generation. (The third issue #3 is still open). So, before pack-send, we obviously need to generate packs first.

So, to begin implemention, I start by understanding the original C code, starting with pack-objects.c? I have read the Git Internals part of the ProGit book, but it felt surface level in this context.

So, as apparent from the README, I should likely first look at these three tasks, starting with the first:

  • create new pack
  • create 'thin' pack
  • Add support for zlib-ng for 2.5x compression performance and 20% faster decompression

Ideally, this functionality would be implemented in git-odb/src/pack/data/encode.rs.

Is this overall understanding correct?

@Byron
Copy link
Member

Byron commented Feb 11, 2021

Yes, that's a decent starting point. The ZLIB upgrade is certainly orthogonal, and in order to warm up with the codebase that one could even be done first.

Furthermore I recommend to divide create new pack into:

  • create base objects (full objects entirely stored in the back without nothing more than zlib compression)
  • create 'ref' objects, which are exactly what's used in 'thin' packs. It's nothing more than a reference to another object by SHA1 and trivial.
  • finally, one could try to create deltified objects, this algorithm is where the dark magic happens and would certainly be the crown jewel of this crate if done well. This is where packs become packs with a 60x compression ratio (or more).

Please let me know if there are any questions along the way and I will try my best to answer them timely. I also offer a video call to get you started more quickly - for that just reach out to me on keybase.io or via email.

The best of success!

@Byron Byron force-pushed the main branch 2 times, most recently from f02f48f to f05037c Compare March 15, 2021 05:13
@Byron
Copy link
Member

Byron commented Mar 21, 2021

By the way, this discussion may be helpful for you as I keep posting new insights there regularly. The tasks file was updated with a new train of work which involves pack generation via implementing the server side of a pull.

@GaurangTandon
Copy link
Author

Hey @Byron I was really hoping to work on this issue. Initially I dug around a bit, and studied some resources to understanding git pack generation properly, and was looking into the codebase. But since then I am sorry I have been rerally busy, and will remain so till next month end :( (I wasn't anticipating this change)

If you were hoping to get this issue done earlier than the next month, feel free to go ahead and start implementing it. If not, then please wait for me as I join forces next month :)

@Byron
Copy link
Member

Byron commented Mar 21, 2021

Thanks for the update :) - I do hope to get started earlier, but as you will see from tasks.md there is enough work across multiple layers to not step on each others toes. In any case, I would be happy to help you get started whenever you are ready to join.

@Byron
Copy link
Member

Byron commented May 8, 2021

Please note that (a lot has happened and) I have begun with the first steps of pack generation, here is the tracking ticket.

If you think you can join with these efforts, please feel free to rebase this PR, otherwise it's probably ok to close it.

@GaurangTandon
Copy link
Author

Yeah, I should not keep you waiting any longer. I really wanted to contribute at least something to this project, as git and rust are two of the favorite technologies to me. I can assure you I had tried my best to carve out time for this, but I am sorry I was not able to commit anything :(

Wish you good luck!

@Byron
Copy link
Member

Byron commented May 8, 2021

There is no need to apologize for anything, sometimes plans don't pan out.

For the next time, you could have a look at other ways to contribute, many of which are small and can certainly be completed in one sitting or two.

Good luck to you too!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants