darwin/arm64 build

[dansimau]

I didn't see an existing issue, so I thought I'd open an issue to track building an arm64 (Apple Silicon) binary for macOS.

After migrating to a new Mac, I have seen at least one issue using Terraform almost straight away:

module.certs_us-east-1["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m27s elapsed]
module.certs["wildcard"].aws_acm_certificate_validation.validation: Still creating... [34m57s elapsed]
module.certs_us-east-1["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m37s elapsed]
module.certs["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m7s elapsed]

assertion failed [abi_info.kind == AbiKind::TranslatedCode]: emulated forward to an arm pc that isn't in translated code. arm_pc=0x1053794d8 abi_kind=6 emulation_interval=[0x105464e60,0x105464e74) instruction_interval=[0x105464e4c, 0x105464e74) x86_rip=0x106148c
(ThreadContextRegisterState.cpp:677 move_to_instruction_boundary)


^C

(CTRL-C did not kill it; when I killed the provider processes they became zombies; I had to SIGKILL the main terraform process.)

Current Terraform Version

Terraform v0.14.2

References

• runtime: tracking bug for ARM-based macOS and GOOS/GOARCH values golang/go#38485 (will at least require Go build support for darwin/arm64)

[apparentlymart]

Hi @dansimau! Thanks for raising this.

There are some things that must happen before we could produce official Terraform releases on arm64 macOS, or before such releases would be practically useful:

• The Go toolchain we use must have production-quality support for that platform. Based on available information, it seems that Go 1.15 (our current version) has only early, experimental support, and that full support is expected in Go 1.16 which is not due until February.

• We will need to acquire hardware with the new CPU in order to verify that all of the platform features we depend on are available in Go's new port to the platform.

• At least the most commonly-used Terraform providers must have at least one release that supports this platform, so that Terraform will be able to install and run those plugins. (A Terraform executable without any provider plugins is not useful.)

  The provider development teams are also constrained by the availability of a suitable Go toolchain and hardware, so they will probably also only be able to do make experimental efforts until Go 1.16 is released, in which case there will be some lag after the Go 1.16 release before sufficient providers are available.

• Our policy for Go language upgrades is that we only accept patch-level upgrades in minor releases, and so the earliest we would adopt Go 1.16 is in the forthcoming v0.15 release. Given that the dependencies here include releases of several other pieces of software, it seems that we should not assume that everything lands in the correct order for macOS arm64 support to necessarily be included in v0.15, but it seems like that's the earliest release where we could support it.

Given that Apple's strategy for gradual ecosystem migration is emulation, I feel optimistic that problems in the current version of the emulation layer will be gradually addressed as the Apple team becomes aware of the problems, so hopefully future updates to the arm64 version of macOS will address the problems that are currently causing the assertion failure you saw. We'll monitor Go releases to see if there are any minor release updates to Go 1.15 that might be issued to improve compatibility with Apple's emulator when running x86_64 binaries, in which case we may be able to include them in v0.14.x minor releases.

[dansimau]

Makes sense.

After using it a bit more, I have concluded that Terraform is basically unusable under emulation on macOS arm64 (Big Sur 11.01).

Here are a sample of some more errors I have been getting:

$ terraform import module.auth.aws_security_group.presignup-lambda sg-XXX
module.auth.aws_security_group.presignup-lambda: Importing from ID "sg-XXX"...
module.auth.aws_security_group.presignup-lambda: Import prepared!
  Prepared aws_security_group for import
module.auth.aws_security_group.presignup-lambda: Refreshing state... [id=sg-XXX]

^C
^C^C^C
^C
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

(Eventually I killed the provider processes).

$ terraform import module.auth.aws_iam_role.presignup-lambda cognito-presignup-lambda

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to retrieve schema from provider "registry.terraform.io/hashicorp/aws":
rpc error: code = Unavailable desc = connection error: desc = "transport:
authentication handshake failed: context deadline exceeded"

(Is the second time I have had to reinit).

$ terraform apply

Error: timeout while waiting for plugin to start


assertion failed [abi_info.kind == AbiKind::TranslatedCode]: emulated forward to an arm pc that isn't in translated code. arm_pc=0x104d204d8 abi_kind=6 emulation_interval=[0x104d77908,0x104d7791c) instruction_interval=[0x104d778f4, 0x104d7791c) x86_rip=0x4044049
(ThreadContextRegisterState.cpp:677 move_to_instruction_boundary)

At the moment it's roughly a coin toss as to whether a command will work or not.

[danieldreier]

@dansimau as it happens, I just got a an M1 Macbook, so I'm able to reproduce this. I did a state import of a security group, and it worked perfectly for me. However, I have my shell set to run under emulation by default, and when I use the default shell, I (on one run, haven't been able to reproduce since then) see the same behavior as you where it hangs waiting for the plugin to start.

Can you try running a shell under rosetta with a command like arch -x86_64 bash and then re-run these commands to see if the issue you're seeing still happens running in an emulated shell?

[dansimau]

@danieldreier I was already running an x86_64 bash (v5), because I was using /usr/local/bin/bash from Homebrew that was copied over after running Migration Assistant.

[dansimau]

Just FWIW, another symptom:

I randomly opened Activity Monitor and found this provider process using 100% CPU. Not even sure where it came from because I stopped running Terraform locally (unless the vscode language server runs it?)

Anyway, I will update to macOS 11.1 and see if there is any change (there is at least one report of Rosetta bug fixes in 11.1).

Side note, I had a lot of issues with other (non-Terraform) Go binaries compiled for darwin/x86_64 (e.g. my VSCode environment ground to a halt). I compiled Go HEAD for arm64, trashed my $GOHOME and reinstalled/recompiled everything. Now that environment is mostly working and stable (which is to be expected) with the notable exception of anything using cgo.

[tuannvm]

@dansimau

Anyway, I will update to macOS 11.1 and see if there is any change (there is at least one report of Rosetta bug fixes in 11.1).

I had same problem when kubectl & terraform x86 went crazy and ate up a lot of CPU. But upgrading to macOS 11.1 fixed that.

[dansimau]

Upgraded to 11.1 and it didn't change anything. The first time I ran Terraform it hung with the same symptoms (provider at ~100% CPU).

Here's a sample of the process:
https://gist.github.com/dansimau/b424531d9ad69acd1dcc08f2713341e0

[steveetm]

For v0.12 users, terraform plan -out was crashing me as well but since 11.1 it is fixed.