CONFIG_THUMB2_KERNEL fails to boot when linked w/ LLD #773
Comments
nickdesaulniers
added
[BUG] Untriaged
|
maybe not a bug, via https://nickdesaulniers.github.io/blog/2018/10/24/booting-a-custom-linux-kernel-in-qemu-and-debugging-it-with-gdb/, I can see the serial driver get loaded w/ BFD but not with LLD |
|
Also, BFD linked kernel loaded the initrd just fine. |
|
It looks like anything marked with __init isn't being called - the "Freeing unused kernel memory" stanza is completely missing and always comes before ramdisk init launch, and the ramdisk is never extracted (populate_rootfs is __init and run through __define_initcall which sticks it in .init). .. could lld be dropping initcall sections? It's also rather curious that it seems to have lost the /init in "Run /init as init process" and replaced it with a NULL or something:
|
|
I think I found what happens. Some __init functions do get called, up until futex.c runs some asm to check a runtime feature. The issue is that when |
|
I wonder if this is a similar issue to #282 (https://lore.kernel.org/r/[email protected]/) (I'm suspecting there may be a bug in |
|
No, that's not the issue. I manually disabled the exception table sorting, there's no difference. The exception table is fine. The issue happens because the code in the exception table handlers isn't correct. This seems to happen for all handlers created through AFAICT, that address should be filled by |
|
BAD (ld.lld): GOOD (ld.bfd): |
|
Can you provide a dump of the Looks like just a bunch of standard |
|
Attached: kernel_futex_extable.txt
Don't think so. It returns the same handler for the same instruction that triggered the exception, in both cases. bfd: |
|
I still suspect that sorting is failing. It might be interesting to print insn/fixup pairs of the __ex_table BEFORE runtime sorting, and compare+validate against bfd. |
Probably not. I get a similar behavior, with the fixups pointing to other functions in kernel, by dropping Also, while |
|
The more time I look at this, the more I'm convinced this is related to #325. |
|
One existing problem in LLD that I know about is that strictly it should not interwork or perform the BL to BLX transition when the symbol has type STT_NOTYPE. LLD at the moment does not have the symbol type available to it when it decides whether to do a BL or BLX so it goes on the bottom bit regardless of the Symbol type. I've put an example below to illustrate the problem: $ llvm-mc --triple=armv7a-linux-gnueabihf --arm-add-build-attributes inter.s -filetype=obj -o inter.o
$ ld.lld inter.o -o inter.axf
$ llvm-objdump -d inter.axf .syntax unified
.section .arm_target, "ax", %progbits
.balign 4
.arm
.type bar, %function
bar:
bx lr
.section .thumb_target, "ax", %progbits
.balign 4
.thumb
.type foo, %function
foo:
bx lr
.section .arm_caller, "ax", %progbits
.balign 4
.arm
bl .arm_target // expect bl STT_SECTION (bit 0 clear anyway)
bl .thumb_target // expect bl STT_SECTION bit 0 clear anyway)
bl foo // expect blx STT_FUNC bit 0 clear
bl bar // expect bl STT_FUNC bit 0 set
.section .thumb_caller, "ax", %progbits
.thumb
.balign 4
bl .arm_target // expect bl STT_SECTION (bit 0 clear, so LLD does blx)
bl .thumb_target // expect bl STT_SECTION (bit 0 clear, so LLD does blx)
bl foo // expect bl STT_FUNC bit 0 set
bl bar // expect blx STT_FUNC bit 0 cleargives me: Whereas GNU ld correctly gives me: If you have branches to non STT_FUNC symbols then there this a reasonable chance of LLD doing interworking when you don't want it to as most non STT_FUNC symbols will look like ARM to it. This is fixable in LLD by threading the symbol through to the relocations. If there is a concrete use case from an important program then I think I've got a good argument that the patch is necessary. |
|
I spoke more with @smithp35 today (cc @MaskRay ) online.
I suspect that may be what's happening. In particular, via this comment, this code reproduced: #define __futex_atomic_ex_table(err_reg) \
"3:\n" \
" .pushsection __ex_table,\"a\"\n" \
" .align 3\n" \
" .long 1b, 4f, 2b, 4f\n" \
" .popsection\n" \
" .pushsection .text.fixup,\"ax\"\n" \
" .align 2\n" \
"4: mov %0, " err_reg "\n" \
" b 3b\n" \
" .popsection"@smithp35 suspects the Also, playing with I spent some time playing with gcc vs clang, bfd vs lld, and it seems that this is reproducible when using lld, regardless of compiler. So to reproduce: $ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make LD=ld.lld -j71 defconfig
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make LD=ld.lld -j71 menuconfig
# enable CONFIG_THUMB2_KERNEL=y and save
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make LD=ld.lld -j71
# llvm-objdump has issues w/ thumb vs arm:
# https://llvm.org/pr44626
$ arm-linux-gnueabihf-objdump -drj .text.fixup kernel/futex.o
kernel/futex.o: file format elf32-littlearm
Disassembly of section .text.fixup:
00000000 <.text.fixup>:
0: 4604 mov r4, r0
2: f000 b862 b.w c8 <.LC4+0x60>
2: R_ARM_THM_JUMP24 .text
6: bf00 nop
8: 4610 mov r0, r2
a: f002 beec b.w 2ddc <do_futex+0x530>
a: R_ARM_THM_JUMP24 .text
e: bf00 nop
10: 4610 mov r0, r2
12: f002 bfae b.w 2f60 <do_futex+0x6b4>
12: R_ARM_THM_JUMP24 .text
16: bf00 nop
18: 4610 mov r0, r2
1a: f002 bfd2 b.w 2fa8 <do_futex+0x6fc>
1a: R_ARM_THM_JUMP24 .text
1e: bf00 nop
20: 4610 mov r0, r2
22: f002 bff6 b.w 2ff0 <do_futex+0x744>
22: R_ARM_THM_JUMP24 .text
26: bf00 nop
28: 4610 mov r0, r2
2a: f003 b819 b.w 3036 <do_futex+0x78a>
2a: R_ARM_THM_JUMP24 .text
2e: bf00 nop
So there's 6 pair's of "fixups" (each followed by a nop to realign to 2^2 B). To check the symbol type: $ llvm-readelf -S kernel/futex.oTODO: finish comment |
$ llvm-readelf -S kernel/futex.o
...
[10] __ex_table PROGBITS 00000000 0038d0 000060 00 A 0 0 8
[11] .rel__ex_table REL 00000000 043768 0000c0 08 I 45 10 4
[12] .text.fixup PROGBITS 00000000 003930 000030 00 AX 0 0 4
[13] .rel.text.fixup REL 00000000 043828 000030 08 I 45 12 4
...Shows the interesting section id/numbers (ie. 10, 11, 12, and 13). To see the symbol type: $ llvm-readelf -s kernel/futex.o
...
Num: Value Size Type Bind Vis Ndx Name
...
14: 00000000 0 SECTION LOCAL DEFAULT 10 __ex_table
15: 00000000 0 NOTYPE LOCAL DEFAULT 10 $d
16: 00000000 0 SECTION LOCAL DEFAULT 12 .text.fixup
17: 00000000 0 NOTYPE LOCAL DEFAULT 12 $tAnd $ llvm-readobj --section-symbols -s kernel/futex.o
...
Section {
Index: 12
Name: .text.fixup (111)
Type: SHT_PROGBITS (0x1)
Flags [ (0x6)
SHF_ALLOC (0x2)
SHF_EXECINSTR (0x4)
]
Address: 0x0
Offset: 0x3930
Size: 48
Link: 0
Info: 0
AddressAlignment: 4
EntrySize: 0
Symbols [
Symbol {
Name: .text.fixup (0)
Value: 0x0
Size: 0
Binding: Local (0x0)
Type: Section (0x3)
Other: 0
Section: .text.fixup (0xC)
}
Symbol {
Name: $t (9)
Value: 0x0
Size: 0
Binding: Local (0x0)
Type: None (0x0)
Other: 0
Section: .text.fixup (0xC)
}
]
}Notice Now when we repro @ihalip 's setup from above first with bfd, then lld: BFDrebuild: # Also enabled CONFIG_DEBUG_INFO for debug symbols.
$ ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j71run in qemu: $ qemu-system-arm -kernel arch/arm/boot/zImage -nographic -m 2048 \
--append "console=ttyAMA0 root=/dev/ram0" -machine virt \
-initrd /android0/continuous-integration/images/arm/rootfs.cpio \
-s -Sattach GDB with a script of @ihalip 's commands # fixup.gdb
target remote :1234
hbreak fixup_exception
continue
n
p fixup
x/i fixup->insn
x/3i fixup->fixup$ /android0/aosp/prebuilts/gdb/linux-x86/bin/gdb vmlinux -x fixup.gdb
...
Reading symbols from vmlinux...
0x40000000 in ?? ()
Hardware assisted breakpoint 1 at 0xc0311450: file arch/arm/mm/extable.c, line 12.
Breakpoint 1, fixup_exception (regs=0xee0bbe50) at arch/arm/mm/extable.c:12
12 fixup = search_exception_tables(instruction_pointer(regs));
13 if (fixup) {
$1 = (const struct exception_table_entry *) 0xc112d288
0xc0397572 <cmpxchg_futex_value_locked+78>: ldrex r7, [r1]
0xc039a928: mov r4, r0
0xc039a92a: b.w 0xc0397588 <cmpxchg_futex_value_locked+100>
0xc039a92e: nopThen compare this to the disassembly. Looks like
$ arm-linux-gnueabihf-objdump -d vmlinux
...
c0397524 <cmpxchg_futex_value_locked>:
...
c0397572: e851 7f00 ldrex r7, [r1]
...
c0397588: 4620 mov r0, r4
...
c039a828 <__se_sys_futex_time32>:
...
c039a928: 4604 mov r4, r0
c039a92a: f7fc be2d b.w c0397588 <cmpxchg_futex_value_locked+0x64>
c039a92e: bf00 nop
c039a930: 4610 mov r0, r2
c039a932: f7ff bcb3 b.w c039a29c <do_futex+0x530>
c039a936: bf00 nop
c039a938: 4610 mov r0, r2
c039a93a: f7ff bd71 b.w c039a420 <do_futex+0x6b4>
c039a93e: bf00 nop
c039a940: 4610 mov r0, r2
c039a942: f7ff bd91 b.w c039a468 <do_futex+0x6fc>
c039a946: bf00 nop
c039a948: 4610 mov r0, r2
c039a94a: f7ff bdb1 b.w c039a4b0 <do_futex+0x744>
c039a94e: bf00 nop
c039a950: 4610 mov r0, r2
c039a952: f7ff bdd0 b.w c039a4f6 <do_futex+0x78a>
c039a956: bf00 nopOh look, our 6 "fixups" are in some random label because |
LLDrebuild: run in qemu (same as above), run in gdb (with same fixup.gdb script) to get: eek. now we have:
So again our fixup object is found and has valid pointers, but the |
Specifically, it seems that symbols with Just to try to reproduce the failure more, I've compiled $ arm-linux-gnueabihf-gcc futex.s -o futex.o -Wa,-mimplicit-it=always -c
$ llvm-readelf -S futex.o
...
[ 8] .text.fixup PROGBITS 00000000 0000e0 000008 00 AX 0 0 4
...
# ok, so .text.fixup is section id/number 8
$ llvm-readelf -s futex.o
...
12: 00000000 0 SECTION LOCAL DEFAULT 8 .text.fixup
13: 00000000 0 NOTYPE LOCAL DEFAULT 8 $t
# and symbols for the section and $t were the only symbols in section id/number 8comparing the disassemblies of futex.lld.o: IIUC, the lld produced object code jumps to a thunk ( ** EDIT ** Err, the lld produced object jumps past the end of the thunk. Not sure how that works, as I cant see anything in the disassembly at that address. |
|
https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gold/arm.cc#l11897 gold can make decisions based on Threading symbol through sounds reasonable. I created https://reviews.llvm.org/D73250 and https://reviews.llvm.org/D73254 |
|
Thanks for the investigation, and thanks for the patches above. I'll do my best to look at those today, at a glance they look like they are enough to make the necessary changes to do interworking in a strict ABI way. Will get a patch as soon as I can (sorry running a bit behind at the moment). |
|
I've submitted https://reviews.llvm.org/D73474 which is independent of https://reviews.llvm.org/D73250 https://reviews.llvm.org/D73254 . This is likely sufficient to fix the B.w problem described above. Once D73254 lands then I'll work on a patch to fix BL and BLX. |
|
That patch allows me to boot successfully! |
nickdesaulniers
added
[BUG] llvm
…mbols ELF for the ARM architecture requires linkers to provide interworking for symbols that are of type STT_FUNC. Interworking for other symbols must be encoded directly in the object file. LLD was always providing interworking, regardless of the symbol type, this breaks some programs that have branches from Thumb state targeting STT_NOTYPE symbols that have bit 0 clear, but they are in fact internal labels in a Thumb function. LLD treats these symbols as ARM and inserts a transition to Arm. This fixes the problem for in range branches, R_ARM_JUMP24, R_ARM_THM_JUMP24 and R_ARM_THM_JUMP19. This is expected to be the vast majority of problem cases as branching to an internal label close to the function. There is at least one follow up patch required. - R_ARM_CALL and R_ARM_THM_CALL may do interworking via BL/BLX substitution. In theory range-extension thunks can be altered to not change state when the symbol type is not STT_FUNC. I will need to check with ld.bfd to see if this is the case in practice. Fixes (part of) ClangBuiltLinux/linux#773 Differential Revision: https://reviews.llvm.org/D73474
|
Fixed in llvm/llvm-project@4f38ab2 Closing for now as we can now boot thumb2. It sounds like there may be more work to do for arm callers calling into thumb. |
nickdesaulniers
added
[FIXED][LLVM] 11
…mbols ELF for the ARM architecture requires linkers to provide interworking for symbols that are of type STT_FUNC. Interworking for other symbols must be encoded directly in the object file. LLD was always providing interworking, regardless of the symbol type, this breaks some programs that have branches from Thumb state targeting STT_NOTYPE symbols that have bit 0 clear, but they are in fact internal labels in a Thumb function. LLD treats these symbols as ARM and inserts a transition to Arm. This fixes the problem for in range branches, R_ARM_JUMP24, R_ARM_THM_JUMP24 and R_ARM_THM_JUMP19. This is expected to be the vast majority of problem cases as branching to an internal label close to the function. There is at least one follow up patch required. - R_ARM_CALL and R_ARM_THM_CALL may do interworking via BL/BLX substitution. In theory range-extension thunks can be altered to not change state when the symbol type is not STT_FUNC. I will need to check with ld.bfd to see if this is the case in practice. Fixes (part of) ClangBuiltLinux/linux#773 Differential Revision: https://reviews.llvm.org/D73474 (cherry picked from commit 4f38ab2)
…TT_FUNC symbols ELF for the ARM architecture requires linkers to provide interworking for symbols that are of type STT_FUNC. Interworking for other symbols must be encoded directly in the object file. LLD was always providing interworking, regardless of the symbol type, this breaks some programs that have branches from Thumb state targeting STT_NOTYPE symbols that have bit 0 clear, but they are in fact internal labels in a Thumb function. LLD treats these symbols as ARM and inserts a transition to Arm. This fixes the problem for in range branches, R_ARM_JUMP24, R_ARM_THM_JUMP24 and R_ARM_THM_JUMP19. This is expected to be the vast majority of problem cases as branching to an internal label close to the function. There is at least one follow up patch required. - R_ARM_CALL and R_ARM_THM_CALL may do interworking via BL/BLX substitution. In theory range-extension thunks can be altered to not change state when the symbol type is not STT_FUNC. I will need to check with ld.bfd to see if this is the case in practice. Fixes (part of) ClangBuiltLinux/linux#773 Differential Revision: https://reviews.llvm.org/D73474 (cherry picked from commit 4f38ab2) Bug: 145916209 Change-Id: I6c9bf5576fd45109f56c918b303c470db7474b95 Signed-off-by: Nick Desaulniers <[email protected]>
ELF for the ARM architecture requires linkers to provide interworking for symbols that are of type STT_FUNC. Interworking for other symbols must be encoded directly in the object file. LLD was always providing interworking, regardless of the symbol type, this breaks some programs that have branches from Thumb state targeting STT_NOTYPE symbols that have bit 0 clear, but they are in fact internal labels in a Thumb function. LLD treats these symbols as ARM and inserts a transition to Arm. This fixes the problem for in range branches, R_ARM_JUMP24, R_ARM_THM_JUMP24 and R_ARM_THM_JUMP19. This is expected to be the vast majority of problem cases as branching to an internal label close to the function. There is at least one follow up patch required. - R_ARM_CALL and R_ARM_THM_CALL may do interworking via BL/BLX substitution. In theory range-extension thunks can be altered to not change state when the symbol type is not STT_FUNC. I will need to check with ld.bfd to see if this is the case in practice. Fixes (part of) ClangBuiltLinux/linux#773 Differential Revision: https://reviews.llvm.org/D73474 Change-Id: I4282ac7f4a347926f95636d43fc404bf15589995 Signed-off-by: mydongistiny <[email protected]>
…mbols ELF for the ARM architecture requires linkers to provide interworking for symbols that are of type STT_FUNC. Interworking for other symbols must be encoded directly in the object file. LLD was always providing interworking, regardless of the symbol type, this breaks some programs that have branches from Thumb state targeting STT_NOTYPE symbols that have bit 0 clear, but they are in fact internal labels in a Thumb function. LLD treats these symbols as ARM and inserts a transition to Arm. This fixes the problem for in range branches, R_ARM_JUMP24, R_ARM_THM_JUMP24 and R_ARM_THM_JUMP19. This is expected to be the vast majority of problem cases as branching to an internal label close to the function. There is at least one follow up patch required. - R_ARM_CALL and R_ARM_THM_CALL may do interworking via BL/BLX substitution. In theory range-extension thunks can be altered to not change state when the symbol type is not STT_FUNC. I will need to check with ld.bfd to see if this is the case in practice. Fixes (part of) ClangBuiltLinux/linux#773 Differential Revision: https://reviews.llvm.org/D73474
…mbols ELF for the ARM architecture requires linkers to provide interworking for symbols that are of type STT_FUNC. Interworking for other symbols must be encoded directly in the object file. LLD was always providing interworking, regardless of the symbol type, this breaks some programs that have branches from Thumb state targeting STT_NOTYPE symbols that have bit 0 clear, but they are in fact internal labels in a Thumb function. LLD treats these symbols as ARM and inserts a transition to Arm. This fixes the problem for in range branches, R_ARM_JUMP24, R_ARM_THM_JUMP24 and R_ARM_THM_JUMP19. This is expected to be the vast majority of problem cases as branching to an internal label close to the function. There is at least one follow up patch required. - R_ARM_CALL and R_ARM_THM_CALL may do interworking via BL/BLX substitution. In theory range-extension thunks can be altered to not change state when the symbol type is not STT_FUNC. I will need to check with ld.bfd to see if this is the case in practice. Fixes (part of) ClangBuiltLinux/linux#773 Differential Revision: https://reviews.llvm.org/D73474 (cherry picked from commit 992b5ff)
MTK asked me about CONFIG_THUMB2_KERNEL support for ARCH=arm. Looks like everything in mainline is fine and boots when linked with BFD. With LLD, I get no output from QEMU. Tested on mainline, defconfig+CONFIG_THUMB2_KERNEL.
The text was updated successfully, but these errors were encountered: