-
Notifications
You must be signed in to change notification settings - Fork 13.4k
replace the system allocator in executables #18915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,111 @@ | ||
#include <stddef.h> | ||
|
||
void *je_malloc(size_t size); | ||
void *je_calloc(size_t num, size_t size); | ||
int je_posix_memalign(void **memptr, size_t alignment, size_t size); | ||
void *je_aligned_alloc(size_t alignment, size_t size); | ||
void *je_realloc(void *ptr, size_t size); | ||
void je_free(void *ptr); | ||
|
||
void *je_mallocx(size_t size, int flags); | ||
void *je_rallocx(void *ptr, size_t size, int flags); | ||
size_t je_xallocx(void *ptr, size_t size, size_t extra, int flags); | ||
size_t je_sallocx(const void *ptr, int flags); | ||
void je_dallocx(void *ptr, int flags); | ||
void je_sdallocx(void *ptr, size_t size, int flags); | ||
size_t je_nallocx(size_t size, int flags); | ||
|
||
int je_mallctl(const char *name, void *oldp, size_t *oldlenp, void *newp, size_t newlen); | ||
int je_mallctlnametomib(const char *name, size_t *mibp, size_t *miblenp); | ||
int je_mallctlbymib(const size_t *mib, size_t miblen, void *oldp, size_t *oldlenp, void *newp, | ||
size_t newlen); | ||
void je_malloc_stats_print(void (*write_cb)(void *, const char *), void *je_cbopaque, | ||
const char *opts); | ||
size_t je_malloc_usable_size(const void *ptr); | ||
|
||
void *je_memalign(size_t alignment, size_t size); | ||
#if !defined(__ANDROID__) | ||
void *je_valloc(size_t size); | ||
#endif | ||
|
||
void *malloc(size_t size) { | ||
return je_malloc(size); | ||
} | ||
|
||
void *calloc(size_t num, size_t size) { | ||
return je_calloc(num, size); | ||
} | ||
|
||
int posix_memalign(void **memptr, size_t alignment, size_t size) { | ||
return je_posix_memalign(memptr, alignment, size); | ||
} | ||
|
||
void *aligned_alloc(size_t alignment, size_t size) { | ||
return je_aligned_alloc(alignment, size); | ||
} | ||
|
||
void *realloc(void *ptr, size_t size) { | ||
return je_realloc(ptr, size); | ||
} | ||
|
||
void free(void *ptr) { | ||
je_free(ptr); | ||
} | ||
|
||
void *mallocx(size_t size, int flags) { | ||
return je_mallocx(size, flags); | ||
} | ||
|
||
void *rallocx(void *ptr, size_t size, int flags) { | ||
return je_rallocx(ptr, size, flags); | ||
} | ||
|
||
size_t xallocx(void *ptr, size_t size, size_t extra, int flags) { | ||
return je_xallocx(ptr, size, extra, flags); | ||
} | ||
|
||
size_t sallocx(const void *ptr, int flags) { | ||
return je_sallocx(ptr, flags); | ||
} | ||
|
||
void dallocx(void *ptr, int flags) { | ||
je_dallocx(ptr, flags); | ||
} | ||
|
||
void sdallocx(void *ptr, size_t size, int flags) { | ||
je_sdallocx(ptr, size, flags); | ||
} | ||
|
||
size_t nallocx(size_t size, int flags) { | ||
return je_nallocx(size, flags); | ||
} | ||
|
||
int mallctl(const char *name, void *oldp, size_t *oldlenp, void *newp, size_t newlen) { | ||
return je_mallctl(name, oldp, oldlenp, newp, newlen); | ||
} | ||
|
||
int mallctlnametomib(const char *name, size_t *mibp, size_t *miblenp) { | ||
return je_mallctlnametomib(name, mibp, miblenp); | ||
} | ||
|
||
int mallctlbymib(const size_t *mib, size_t miblen, void *oldp, size_t *oldlenp, void *newp, | ||
size_t newlen) { | ||
return je_mallctlbymib(mib, miblen, oldp, oldlenp, newp, newlen); | ||
} | ||
|
||
void malloc_stats_print(void (*write_cb)(void *, const char *), void *je_cbopaque, | ||
const char *opts) { | ||
return je_malloc_stats_print(write_cb, je_cbopaque, opts); | ||
} | ||
|
||
size_t malloc_usable_size(const void *ptr) { | ||
return je_malloc_usable_size(ptr); | ||
} | ||
|
||
void *memalign(size_t alignment, size_t size) { | ||
return je_memalign(alignment, size); | ||
} | ||
|
||
void *valloc(size_t size) { | ||
return je_valloc(size); | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How come this reexports a number of jemalloc symbols without the
je_
prefix? I would expect the standard libc weak symbols to be exposed, but the jemalloc symbols aren't able to be overridden, right?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
None of the public symbols defined by jemalloc are weak symbols. I'm exporting these to address the demand that
mallocx
be usable as it is in vanilla jemalloc with no prefix.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you elaborate on this "demand" a little more? This is basically one of the possible shims rustc can inject, and the purpose is to override the system malloc/free, and I am unaware of the desire to export jemalloc-specific symbols as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's necessary for Rust's jemalloc to satisfy the needs of third party code calling into jemalloc. That was the primary argument against the last pull request...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If Rust doesn't do this, then third party code using jemalloc cannot be used. C libraries don't usually have versions in the symbol names, so you can't just have multiple copies living side-by-side without problems.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was under the impression that this "third party code" was primarily code in other processes that Rust itself was linked into. Either via a staticlib, dylib, or dlopen()'d dylib. Within a Rust executable itself (which this PR is focused on), however, I don't think that this would help too much. Libraries should be written knowing that the allocator is not their decision, and should plan appropriately (not relying on an upstream definition of jemalloc). Native code linked into an executable cannot rely on the existence of these symbols as the compiler is the one choosing whether to link in jemalloc or not, not the code itself.
Note that I'm just at this from the perspective of having this shim be as small as possible. I'd rather stick to well-known standardized apis like malloc than duplicate the nonstandard apis of jemalloc. If these were to fall out of sync with the jemalloc definitions, then I imagine badness could ensue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If that code depends on jemalloc, then it will need to be using Rust's jemalloc.
The only argument against the previous one was that it would break code relying on mixing
mallocx
andfree
. The previous pull request was simpler and didn't have the added overhead of these wrapper functions. I'll just reopen it in favour of this one if that dubious argument has been abandoned.They are not "duplicated" in any way. It is manually removing the prefix because you rejected my pull request doing this the easy and low-overhead way by using the default configuration.
It's a stable API. There was a long deprecation period for the old experimental API before the shift to this one.