runtime: mallocs cause "base outside usable address space" panic when running on iOS 14

[rayvbr]

What version of Go are you using (go version)?

$ go version
go version go1.16.5 darwin/amd64 (also reproducible on 1.15 and 1.14)

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

ios/arm64

What did you do?

Create a simple Go package, doing a few medium sized allocs (creating a few byte slices of several tens of MB will do, as long as it forces the Go runtime to ask the OS for more memory), compile it with gomobile to an iOS .framework file, use the resulting framework in an iOS project and compile and run for iOS 14.

If the iOS project doesn't do a lot of memory allocs itself, everything works fine.
If the iOS project does use reserve a lot of memory (either for CPU or GPU) before initialising the Go framework, the Go runtime crashes with the following panic.

runtime: memory allocated by OS [0x2ac000000, 0x2b0000000) not in usable address space: base outside usable address space
fatal error: memory reservation exceeds address space limit

Note that the exact amount of memory needed seems to be device dependent, although in all cases the panic happens way before the typical OOM point for the device in question. Issues was reproduced using iPhone 12 (allocating 1.1GB of RAM in the iOS project is sufficient to make it crash), iPhone 12 Pro and iPhone SE 2020. Note that everything works fine when compiling with pre-iOS 14 versions of XCode.

Although I'm way out of my league here, the addresses indicated in the panic seem to indicate that iOS no longer has a 33-bit memory address limit, as assumed by the Go runtime. Perhaps related to iPhone 12 Pro having 6GB of RAM, which would mean the 4GB limit assumed by the Go runtime would no longer be sufficient?

[toothrot]

/cc @mknyszek @hyangah

[mknyszek]

Although I'm way out of my league here, the addresses indicated in the panic seem to indicate that iOS no longer has a 33-bit memory address limit, as assumed by the Go runtime. Perhaps related to iPhone 12 Pro having 6GB of RAM, which would mean the 4GB limit assumed by the Go runtime would no longer be sufficient?

This might be the case. To be honest, I couldn't find any documentation about this. This was the best I could infer from experiments and the experiments of others I found. Basically, the assumption I was operating under is that each process has 4 GiB of address space, starting from the 4 GiB offset (the bottom 4 GiB are reserved).

The amount of RAM on the phone theoretically shouldn't matter, because I think (but cannot say for sure) iOS keeps each process's reserved address space somewhat limited, IIUC. The aforementioned experiments revealed you could barely make a 2 GiB mapping (PROT_NONE!) without iOS saying we were out of memory. But, I think you're right that something changed in iOS 14.

One solution here is to just run new experiments and increase the size of the address space that we assume. I'll try to think of something better, but until there's actually any documentation on this, I think the address space structure we have is maybe not a great fit for a platform that doesn't document the size of its available address space. Sigh.

[rayvbr]

Thanks @mknyszek.

i did some searching on changes in iOS 14, and it seems Apple introduced something called Extended Virtual Address Space. Details are scarce, but it sounds very related to our problem above.

[rayvbr]

Maybe this will be of use: https://github.com/apple/darwin-xnu/blob/a1babec6b135d1f35b2590a1990af3c5c5393479/bsd/kern/kern_exec.c#L2928

vm_map_set_jumbo can be found here: https://opensource.apple.com/source/xnu/xnu-4570.41.2/osfmk/vm/vm_map.c.auto.html

[rayvbr]

Update: I can reproduce with an empty iOS project, and having the Go library allocate several GB of data on its own. Instead of the alloc succeeding, or throwing an OOM error, it throws:

runtime: memory allocated by OS [0x2ac000000, 0x2b0000000) not in usable address space: base outside usable address space
fatal error: memory reservation exceeds address space limit

[rayvbr]

@mknyszek Given that Go 1.13 doesn't suffer from the problem, I was able to trace the issue back to this single-line commit 198f045

I've confirmed that when I change the 33 into 39, everything works fine.

That said, I don't understand this code well enough to determine what side-effects that might have. What are the potential risks of me doing so? Is the base outside usable address space error just an extra safeguard against something that should never happen in practice because the OS would never assign a memory address outside of addressable space? Or is me changing it to 39 a serious risk that could result in runtime panics?

[mknyszek]

Increasing that number is exactly what I was talking about earlier. The risk with increasing that number naively, though, is that it

1. causes a larger virtual mapping to be created for a data structure in the runtime, and
2. slows down access to that data structure slightly (not enough to matter, I think).

This may or may not cause errors in older iOS versions, but I think the mapping for 39 bits should still be relatively small. I should calculate that. The reason why you don't see an issue with Go 1.13 is that I added that in Go 1.14.

My question, however, is: is 39 the right number? What are the actual virtual address space limits when you have the Extended Virtual Addressing entitlement? I can't actually find any documentation on this.

[rayvbr]

I double checked the extended virtual addressing entitlement, but we actually have it turned off. So my theory is that something changed in the way iOS does memory addressing to allow for that feature to be possible, but those changes are also active when the entitlement is not used.

I'll do some more searching to find the 'right' number. The highest memory address I've seen so far is 0x2b8000000.

But I'm afraid changing it to 39 has serious issues. I've had the following crash on Swift side at least once, since testing with 39:

2021-06-25 16:14:36.961698+0200 flat[43222:11907111] Uncaught exception: NSInvalidArgumentException: *** NSAllocateMemoryPages(18446744071865635812) failed
(
    0   CoreFoundation                      0x000000019bbfa5c8 58500388-BF36-397C-84CF-17315A3445B6 + 1217992
    1   libobjc.A.dylib                     0x00000001b06797a8 objc_exception_throw + 60
    2   Foundation                          0x000000019cf81224 NSZoneMalloc + 0
    3   Foundation                          0x000000019ce53f14 63D26DEE-A1FB-34B0-9ADA-CC52E7F1D60C + 57108
    4   Foundation                          0x000000019ce569b8 63D26DEE-A1FB-34B0-9ADA-CC52E7F1D60C + 68024

[mknyszek]

That failure suggests to me (but I'm not 100% sure) that Swift's malloc is having trouble getting virtual address space. If that's the case, I think 39 is too high.

Thanks for looking into this! Does 35 work for you? I noticed your original reply (which got sent to my email) had 35 instead of 39.

[rayvbr]

Yes, indeed. I set it to 35 before, but then I noticed it was not high enough; the issue still appeared, albeit at higher memory levels. I'll try a few more values until I find the 'right' one. The main issue is that it is difficult to say for sure when a value is correct. The Swift error above for example only seems to appear when the circumstances are just right and is not easy to reproduce.

[rayvbr]

Seems like iOS 15 will complicate things even further: https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_kernel_increased-memory-limit?changes=latest_beta