CONFIG_SECCOMP panic with LLD

[nathanchance]

Forked off from ClangBuiltLinux/continuous-integration#81

With my fork of the Pi kernel, I can successfully build, boot, and run a kernel compiled with Clang on both 4.14 and 4.20. Additionally, thanks to the top two patches, I can link the kernel successfully with lld. However, it panics before getting to userspace. I have pulled a log, which I am happy to triage as needed but honestly, I don't know where to start.

[nickdesaulniers]

The oops' all look the same

[    2.401795] PC is at 0x80008274
[    2.401812] LR is at do_seccomp+0x138/0x73c

so ARM always has a dedicated register for its parent stack frame, "the link register," LR. So LR is do_seccomp+0x138 meaning if you disassemble the object file from the source file that defines do_seccomp (use grep or ag to find which translation unit it comes from), then look at the instruction at 0x138 bytes from the start of do_seccomp label, it's probably making a call into another function (on ARM, a call instruction is bl "branch and link" which I think of as call+set LR). So do_seccomp is jumping to 0x80008274. Looks like the kernels unwinder wasn't able to tell you what that is.

I don't know what seccomp is, and haven't looked, but maybe LLD is broken with seccomp? Maybe if seccomp can be disabled, you might be able to boot, then we can go from there.

[nathanchance]

Thank you very much for that analysis! Disabling CONFIG_SECCOMP allows it to boot successfully and everything appears to be functioning normally (been running for about five minutes without issues). I'll re-enable CONFIG_SECCOMP and poke my head around the disassembly and see what it looks like.

[nathanchance]

I wasn't able to fully dive into my analysis tonight since I didn't have all the time I was hoping for but I uploaded the full disassembly of kernel/seccomp.o and a new stacktrace in case someone wants to glance them over.

https://gist.github.com/75872ee94ec8cbea55e279ab32f4d8e2
https://gist.github.com/7af41fcbb2f804a5f65bbedb81a1462a

It did occur to me that @kees is the seccomp maintainer, he may have some insight into this (although, there's no guarantee that this is isolated to seccomp, it could be a global issue that is just happening with seccomp in this config).

[nickdesaulniers]

Yes @kees can help then. Next question, is this isolated to arm32? I assume it's maybe not in any defconfig, so we don't have CI coverage of it?

[nickdesaulniers]

Thanks for the disassembly. Because it's the disassembly of an object file that hasn't been linked yet, the addresses are not fully resolved yet, so the addresses from the stack trace don't quite line up with the disassembled object file. That's ok.

[    2.377552] LR is at do_seccomp+0x1a8/0x768

(note, this is different from the first trace, took me a few minutes to figure that out, screws up the below math)

000004f8 <do_seccomp>:

0x1a8 + 0x4f8 == 0x6a0

     6a0:	e3500000 	cmp	r0, #0

Now another special thing is that the LR points to the next instruction after the child stack frame returns, we need to subtract 4 bytes (1 word on arm32).

0x6a0 - 0x1 == 0x69c

     69c:	ebfffffe 	bl	0 <arm_copy_from_user>

aha, there's a branch and link instruction (bl) which would have set the LR to 0x6a0. Looks like arm_copy_from_user is called twice from do_seccomp, why does this second one fail? Why is it special? do_seccomp might not immediately call arm_copy_from_user, but through inlining it may be the case.

Next idea to try; can you find the two invocations of arm_copy_from_user (or maybe copy_from_user) in do_seccomp? If you comment out one invocation and maybe return early, or jump to some error handler, or put some dummy value in the variable it's trying to read into, does booting progress? (We're trying to isolate the problematic source)

[nickdesaulniers]

Also, I'll bet that arm_copy_from_user is implemented in assembly, which may be why it doesn't show up next to the PC in the unwind (I'm not sure how that's fixed, honestly).

[nathanchance]

Next question, is this isolated to arm32? I assume it's maybe not in any defconfig, so we don't have CI coverage of it?

CONFIG_SECCOMP=y is in multi_v7_defconfig and arm64 defconfig so I guess real hardware is exploiting this in a way QEMU isn't. I can't say if this is isolated to just arm32 because I don't have an arm64 board.

Also, I'll bet that arm_copy_from_user is implemented in assembly, which may be why it doesn't show up next to the PC in the unwind (I'm not sure how that's fixed, honestly).

Correct: https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/arch/arm/lib/copy_from_user.S

aha, there's a branch and link instruction (bl) which would have set the LR to 0x6a0. Looks like arm_copy_from_user is called twice from do_seccomp, why does this second one fail? Why is it special? do_seccomp might not immediately call arm_copy_from_user, but through inlining it may be the case.

Yeah that's interesting... I see three instances of copy_from_user (which are turned into arm_copy_from_user) and none of them are directly in do_seccomp so inlining must have happened.

https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/kernel/seccomp.c#L422
https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/kernel/seccomp.c#L428
https://github.com/nathanchance/pi-kernel/blob/rpi-4.14.y/kernel/seccomp.c#L909

Since CONFIG_COMPAT isn't relevant for a 32-bit kernel, it must be the last two.

Next idea to try; can you find the two invocations of arm_copy_from_user (or maybe copy_from_user) in do_seccomp? If you comment out one invocation and maybe return early, or jump to some error handler, or put some dummy value in the variable it's trying to read into, does booting progress? (We're trying to isolate the problematic source)

The problematic one appears to be in in seccomp_prepare_user_filter, as I can successfully boot with this diff (although my modules didn't load so I wasn't able to verify the version string successfully):

diff --git a/kernel/seccomp.c b/kernel/seccomp.c
index 075e344a87c3..6e243d9886dc 100644
--- a/kernel/seccomp.c
+++ b/kernel/seccomp.c
@@ -425,9 +425,11 @@ seccomp_prepare_user_filter(const char __user *user_filter)
                fprog.filter = compat_ptr(fprog32.filter);
        } else /* falls through to the if below. */
 #endif
+#if 0
        if (copy_from_user(&fprog, user_filter, sizeof(fprog)))
                goto out;
        filter = seccomp_prepare_filter(&fprog);
+#endif
 out:
        return filter;
 }

[nickdesaulniers]

CONFIG_SECCOMP=y is in multi_v7_defconfig and arm64 defconfig so I guess real hardware is exploiting this in a way QEMU isn't.

That would be bad, and peculiar if something like that was the linker's fault. But the Pi's config is not the same as our defconfigs (and we should verify booting the pi kernel in qemu).

CONFIG_MMU changes the implementation of raw_copy_from_user. Is that set for our QEMU tested defconfigs, but not for the pi, or vice-versa? (I feel like that would have blown up in other ways, and it's not clear to my why this lone call site is problematic).

[nathanchance]

and we should verify booting the pi kernel in qemu

Yeah this doesn't really seem to easy (at least with their config). Doing some research, it seems like everyone is using this repo and its build script uses arch/arm/configs/versatile_defconfig as a base, not arch/arm/configs/bcm2709_defconfig (because it's only concerned with running RPI userspace, not its kernel).

I will try to do more research on this tomorrow and I'll double check all of the results I've posted so far since I agree that it's super bizarre that JUST changing the linker is resulting in this issue...

CONFIG_MMU changes the implementation of raw_copy_from_user. Is that set for our QEMU tested defconfigs, but not for the pi, or vice-versa? (I feel like that would have blown up in other ways, and it's not clear to my why this lone call site is problematic).

It is default y on both arm and arm64 so I assume QEMU should be using it.

[kees]

Does this fail under qemu too? Have you compared the disassemble between ld.bfd and ld.lld?

There is nothing "special" about this copy_from_user() that I can see.

It looks like the Oops is at address 8000000d -- which seems like a strange address: it's the kernel base plus 14, which is pretty small... I would assume there is a dereference from some structure address but the address is very wrong. Perhaps the linker failed to apply the right offset, and used only the kernel base offset?

[kees]

Can you do an objdump of vmlinux, instead, and include the copy_from_user() code? That should give us a view into exactly what the linked result looks like, and what it's trying to do at that address.

[arndb]

Yeah this doesn't really seem to easy (at least with their config).

You should be able to run 'make kvm_guest.config' to enable the additional options you need for any kernel inside of qemu (kvm or emulated, using "-m virt" instead of -m versatile). In addition you will want 
CONFIG_DEBUG_INFO=y, CONFIG_DEBUG_INFO=y and possibly CONFIG_DEBUG_INFO=n, to allow you attach gdb to the qemu instance and use the provided tools from the kernel.

[nathanchance]

I will debug this best I can over the weekend or early next week, thank you both for the tips!

[nathanchance]

Can you do an objdump of vmlinux, instead, and include the copy_from_user() code? That should give us a view into exactly what the linked result looks like, and what it's trying to do at that address.

Sorry for taking so long to get back to this :(

Here are the various snippets, along with the full dump here:

804c0124 <__copy_from_user_memcpy>:
804c0124:       e92d4ff0        push    {r4, r5, r6, r7, r8, r9, sl, fp, lr}
804c0128:       e24dd004        sub     sp, sp, #4
804c012c:       e1a0600d        mov     r6, sp
804c0130:       e1a08000        mov     r8, r0
804c0134:       e7cc601f        bfc     r6, #0, #13
804c0138:       e1a04002        mov     r4, r2
804c013c:       e1a05001        mov     r5, r1
804c0140:       e5960008        ldr     r0, [r6, #8]
804c0144:       e3500000        cmp     r0, #0
804c0148:       0a000066        beq     804c02e8 <__copy_from_user_memcpy+0x1c4>
804c014c:       e5969004        ldr     r9, [r6, #4]
804c0150:       e3590000        cmp     r9, #0
804c0154:       1a000003        bne     804c0168 <__copy_from_user_memcpy+0x44>
804c0158:       e596000c        ldr     r0, [r6, #12]
804c015c:       e5900338        ldr     r0, [r0, #824]  ; 0x338
804c0160:       e280003c        add     r0, r0, #60     ; 0x3c
804c0164:       eb0d49b8        bl      8081284c <down_read>
804c0168:       e3540000        cmp     r4, #0
804c016c:       e58d6000        str     r6, [sp]
804c0170:       0a000051        beq     804c02bc <__copy_from_user_memcpy+0x198>
804c0174:       e7e86655        ubfx    r6, r5, #12, #9
804c0178:       e1a0baa5        lsr     fp, r5, #21
804c017c:       e1a0000d        mov     r0, sp
804c0180:       e3a0100c        mov     r1, #12
804c0184:       e7cc0011        bfi     r0, r1, #0, #13
804c0188:       e5900000        ldr     r0, [r0]
804c018c:       e5900338        ldr     r0, [r0, #824]  ; 0x338
804c0190:       e5900024        ldr     r0, [r0, #36]   ; 0x24
804c0194:       e790a18b        ldr     sl, [r0, fp, lsl #3]
804c0198:       e35a0000        cmp     sl, #0
804c019c:       0a000019        beq     804c0208 <__copy_from_user_memcpy+0xe4>
804c01a0:       e21a0002        ands    r0, sl, #2
804c01a4:       1a000017        bne     804c0208 <__copy_from_user_memcpy+0xe4>
804c01a8:       e3060804        movw    r0, #26628      ; 0x6804
804c01ac:       e3001e3c        movw    r1, #3644       ; 0xe3c
804c01b0:       e34800d0        movt    r0, #32976      ; 0x80d0
804c01b4:       e34810e1        movt    r1, #32993      ; 0x80e1
804c01b8:       e5900000        ldr     r0, [r0]
804c01bc:       e5911000        ldr     r1, [r1]
804c01c0:       e060062a        rsb     r0, r0, sl, lsr #12
804c01c4:       e0800180        add     r0, r0, r0, lsl #3
804c01c8:       e0810100        add     r0, r1, r0, lsl #2
804c01cc:       e280701c        add     r7, r0, #28
804c01d0:       e1a00007        mov     r0, r7
804c01d4:       eb0d541c        bl      8081524c <_raw_spin_lock>
804c01d8:       e7cba01f        bfc     sl, #0, #12
804c01dc:       e24a0481        sub     r0, sl, #-2130706432    ; 0x81000000
804c01e0:       e7900106        ldr     r0, [r0, r6, lsl #2]
804c01e4:       e2000003        and     r0, r0, #3
804c01e8:       e3500003        cmp     r0, #3
804c01ec:       0a00001f        beq     804c0270 <__copy_from_user_memcpy+0x14c>
804c01f0:       f57ff05b        dmb     ish
804c01f4:       e1d700b0        ldrh    r0, [r7]
804c01f8:       e2800001        add     r0, r0, #1
804c01fc:       e1c700b0        strh    r0, [r7]
804c0200:       f57ff04a        dsb     ishst
804c0204:       e320f004        sev
804c0208:       e3590000        cmp     r9, #0
804c020c:       1a000004        bne     804c0224 <__copy_from_user_memcpy+0x100>
804c0210:       e59d0000        ldr     r0, [sp]
804c0214:       e590000c        ldr     r0, [r0, #12]
804c0218:       e5900338        ldr     r0, [r0, #824]  ; 0x338
804c021c:       e280003c        add     r0, r0, #60     ; 0x3c
804c0220:       ebf2c44b        bl      80171354 <up_read>
804c0224:       e59d0000        ldr     r0, [sp]
804c0228:       e5900008        ldr     r0, [r0, #8]
804c022c:       e2401001        sub     r1, r0, #1
804c0230:       e1a00005        mov     r0, r5
804c0234:       eb0cdbe3        bl      807f71c8 <__get_user_1>
804c0238:       e1801009        orr     r1, r0, r9
804c023c:       e3500000        cmp     r0, #0
804c0240:       13000006        movwne  r0, #6
804c0244:       e3510000        cmp     r1, #0
804c0248:       1a000005        bne     804c0264 <__copy_from_user_memcpy+0x140>
804c024c:       e59d0000        ldr     r0, [sp]
804c0250:       e590000c        ldr     r0, [r0, #12]
804c0254:       e5900338        ldr     r0, [r0, #824]  ; 0x338
804c0258:       e280003c        add     r0, r0, #60     ; 0x3c
804c025c:       eb0d497a        bl      8081284c <down_read>
804c0260:       e3a00000        mov     r0, #0
804c0264:       e3500000        cmp     r0, #0
804c0268:       0affffc3        beq     804c017c <__copy_from_user_memcpy+0x58>
804c026c:       ea00001a        b       804c02dc <__copy_from_user_memcpy+0x1b8>
804c0270:       e1a00005        mov     r0, r5
804c0274:       e1a01005        mov     r1, r5
804c0278:       e7df061f        bfc     r0, #12, #20
804c027c:       e2606a01        rsb     r6, r0, #4096   ; 0x1000
804c0280:       e1a00008        mov     r0, r8
804c0284:       e1560004        cmp     r6, r4
804c0288:       81a06004        movhi   r6, r4
804c028c:       e1a02006        mov     r2, r6
804c0290:       eb0cde6a        bl      807f7c40 <memcpy>
804c0294:       f57ff05b        dmb     ish
804c0298:       e0544006        subs    r4, r4, r6
804c029c:       e0855006        add     r5, r5, r6
804c02a0:       e1d700b0        ldrh    r0, [r7]
804c02a4:       e0888006        add     r8, r8, r6
804c02a8:       e2800001        add     r0, r0, #1
804c02ac:       e1c700b0        strh    r0, [r7]
804c02b0:       f57ff04a        dsb     ishst
804c02b4:       e320f004        sev     
804c02b8:       1affffad        bne     804c0174 <__copy_from_user_memcpy+0x50>
804c02bc:       e3a04000        mov     r4, #0
804c02c0:       e3590000        cmp     r9, #0
804c02c4:       1a000004        bne     804c02dc <__copy_from_user_memcpy+0x1b8>
804c02c8:       e59d0000        ldr     r0, [sp]
804c02cc:       e590000c        ldr     r0, [r0, #12]
804c02d0:       e5900338        ldr     r0, [r0, #824]  ; 0x338
804c02d4:       e280003c        add     r0, r0, #60     ; 0x3c
804c02d8:       ebf2c41d        bl      80171354 <up_read>
804c02dc:       e1a00004        mov     r0, r4
804c02e0:       e28dd004        add     sp, sp, #4
804c02e4:       e8bd8ff0        pop     {r4, r5, r6, r7, r8, r9, sl, fp, pc}
804c02e8:       e1a00008        mov     r0, r8
804c02ec:       e1a01005        mov     r1, r5
804c02f0:       e1a02004        mov     r2, r4
804c02f4:       eb0cde51        bl      807f7c40 <memcpy>
804c02f8:       e3a04000        mov     r4, #0
804c02fc:       eafffff6        b       804c02dc <__copy_from_user_memcpy+0x1b8>
...
804c0310 <arm_copy_from_user>:
804c0310:       e352003f        cmp     r2, #63 ; 0x3f
804c0314:       8a000000        bhi     804c031c <arm_copy_from_user+0xc>
804c0318:       ea0cd643        b       807f5c2c <__copy_from_user_std>
804c031c:       eaffff80        b       804c0124 <__copy_from_user_memcpy>
...
807f5c2c <__copy_from_user_std>:
807f5c2c:       e1a036ad        lsr     r3, sp, #13
807f5c30:       e1a03683        lsl     r3, r3, #13
807f5c34:       e5933008        ldr     r3, [r3, #8]
807f5c38:       e091c002        adds    ip, r1, r2
807f5c3c:       e2433001        sub     r3, r3, #1
807f5c40:       315c0003        cmpcc   ip, r3
807f5c44:       23a01000        movcs   r1, #0
807f5c48:       e320f014        csdb    
807f5c4c:       e3a03000        mov     r3, #0
807f5c50:       e92d401d        push    {r0, r2, r3, r4, lr}
807f5c54:       e2522004        subs    r2, r2, #4
807f5c58:       ba000032        blt     807f5d28 <__copy_from_user_std+0xfc>
807f5c5c:       e210c003        ands    ip, r0, #3
807f5c60:       f5d1f000        pld     [r1]
807f5c64:       1a000038        bne     807f5d4c <__copy_from_user_std+0x120>
807f5c68:       e211c003        ands    ip, r1, #3
807f5c6c:       1a000042        bne     807f5d7c <__copy_from_user_std+0x150>
807f5c70:       e252201c        subs    r2, r2, #28
807f5c74:       e92d01e0        push    {r5, r6, r7, r8}
807f5c78:       ba000013        blt     807f5ccc <__copy_from_user_std+0xa0>
807f5c7c:       f5d1f000        pld     [r1]
807f5c80:       e2522060        subs    r2, r2, #96     ; 0x60
807f5c84:       f5d1f01c        pld     [r1, #28]
807f5c88:       ba000002        blt     807f5c98 <__copy_from_user_std+0x6c>
807f5c8c:       f5d1f03c        pld     [r1, #60]       ; 0x3c
807f5c90:       f5d1f05c        pld     [r1, #92]       ; 0x5c
807f5c94:       f5d1f07c        pld     [r1, #124]      ; 0x7c
807f5c98:       e4913004        ldr     r3, [r1], #4
807f5c9c:       e4914004        ldr     r4, [r1], #4    
807f5ca0:       e4915004        ldr     r5, [r1], #4    
807f5ca4:       e4916004        ldr     r6, [r1], #4    
807f5ca8:       e4917004        ldr     r7, [r1], #4    
807f5cac:       e4918004        ldr     r8, [r1], #4
807f5cb0:       e491c004        ldr     ip, [r1], #4
807f5cb4:       e491e004        ldr     lr, [r1], #4
807f5cb8:       e2522020        subs    r2, r2, #32
807f5cbc:       e8a051f8        stmia   r0!, {r3, r4, r5, r6, r7, r8, ip, lr}
807f5cc0:       aafffff3        bge     807f5c94 <__copy_from_user_std+0x68>
807f5cc4:       e3720060        cmn     r2, #96 ; 0x60
807f5cc8:       aafffff2        bge     807f5c98 <__copy_from_user_std+0x6c>
807f5ccc:       e212c01c        ands    ip, r2, #28
807f5cd0:       e26cc020        rsb     ip, ip, #32
807f5cd4:       108ff00c        addne   pc, pc, ip
807f5cd8:       ea000011        b       807f5d24 <__copy_from_user_std+0xf8>
807f5cdc:       e320f000        nop     {0}
807f5ce0:       e4913004        ldr     r3, [r1], #4
807f5ce4:       e4914004        ldr     r4, [r1], #4
807f5ce8:       e4915004        ldr     r5, [r1], #4
807f5cec:       e4916004        ldr     r6, [r1], #4
807f5cf0:       e4917004        ldr     r7, [r1], #4
807f5cf4:       e4918004        ldr     r8, [r1], #4
807f5cf8:       e491e004        ldr     lr, [r1], #4
807f5cfc:       e08ff00c        add     pc, pc, ip
807f5d00:       e320f000        nop     {0}
807f5d04:       e320f000        nop     {0}
807f5d08:       e4803004        str     r3, [r0], #4
807f5d0c:       e4804004        str     r4, [r0], #4
807f5d10:       e4805004        str     r5, [r0], #4
807f5d14:       e4806004        str     r6, [r0], #4
807f5d18:       e4807004        str     r7, [r0], #4
807f5d1c:       e4808004        str     r8, [r0], #4
807f5d20:       e480e004        str     lr, [r0], #4
807f5d24:       e8bd01e0        pop     {r5, r6, r7, r8}
807f5d28:       e1b02f82        lsls    r2, r2, #31
807f5d2c:       14d13001        ldrbne  r3, [r1], #1
807f5d30:       24d14001        ldrbcs  r4, [r1], #1
807f5d34:       24d1c001        ldrbcs  ip, [r1], #1
807f5d38:       14c03001        strbne  r3, [r0], #1
807f5d3c:       24c04001        strbcs  r4, [r0], #1
807f5d40:       24c0c001        strbcs  ip, [r0], #1
807f5d44:       e28dd008        add     sp, sp, #8
807f5d48:       e8bd8011        pop     {r0, r4, pc}
807f5d4c:       e26cc004        rsb     ip, ip, #4
807f5d50:       e35c0002        cmp     ip, #2
807f5d54:       c4d13001        ldrbgt  r3, [r1], #1
807f5d58:       a4d14001        ldrbge  r4, [r1], #1
807f5d5c:       e4d1e001        ldrb    lr, [r1], #1
807f5d60:       c4c03001        strbgt  r3, [r0], #1
807f5d64:       a4c04001        strbge  r4, [r0], #1
807f5d68:       e052200c        subs    r2, r2, ip
807f5d6c:       e4c0e001        strb    lr, [r0], #1
807f5d70:       baffffec        blt     807f5d28 <__copy_from_user_std+0xfc>
807f5d74:       e211c003        ands    ip, r1, #3
807f5d78:       0affffbc        beq     807f5c70 <__copy_from_user_std+0x44>
807f5d7c:       e3c11003        bic     r1, r1, #3
807f5d80:       e35c0002        cmp     ip, #2
807f5d84:       e491e004        ldr     lr, [r1], #4
807f5d88:       0a000032        beq     807f5e58 <__copy_from_user_std+0x22c>
807f5d8c:       ca000063        bgt     807f5f20 <__copy_from_user_std+0x2f4>
807f5d90:       e252201c        subs    r2, r2, #28
807f5d94:       ba000025        blt     807f5e30 <__copy_from_user_std+0x204>
807f5d98:       e92d03e0        push    {r5, r6, r7, r8, r9}
807f5d9c:       f5d1f000        pld     [r1]
807f5da0:       e2522060        subs    r2, r2, #96     ; 0x60
807f5da4:       f5d1f01c        pld     [r1, #28]
807f5da8:       ba000002        blt     807f5db8 <__copy_from_user_std+0x18c>
807f5dac:       f5d1f03c        pld     [r1, #60]       ; 0x3c
807f5db0:       f5d1f05c        pld     [r1, #92]       ; 0x5c
807f5db4:       f5d1f07c        pld     [r1, #124]      ; 0x7c
807f5db8:       e4914004        ldr     r4, [r1], #4
807f5dbc:       e4915004        ldr     r5, [r1], #4
807f5dc0:       e4916004        ldr     r6, [r1], #4
807f5dc4:       e4917004        ldr     r7, [r1], #4
807f5dc8:       e1a0342e        lsr     r3, lr, #8
807f5dcc:       e2522020        subs    r2, r2, #32
807f5dd0:       e4918004        ldr     r8, [r1], #4
807f5dd4:       e4919004        ldr     r9, [r1], #4
807f5dd8:       e491c004        ldr     ip, [r1], #4
807f5ddc:       e491e004        ldr     lr, [r1], #4
807f5de0:       e1833c04        orr     r3, r3, r4, lsl #24
807f5de4:       e1a04424        lsr     r4, r4, #8
807f5de8:       e1844c05        orr     r4, r4, r5, lsl #24
807f5dec:       e1a05425        lsr     r5, r5, #8
807f5df0:       e1855c06        orr     r5, r5, r6, lsl #24
807f5df4:       e1a06426        lsr     r6, r6, #8
807f5df8:       e1866c07        orr     r6, r6, r7, lsl #24
807f5dfc:       e1a07427        lsr     r7, r7, #8
807f5e00:       e1877c08        orr     r7, r7, r8, lsl #24
807f5e04:       e1a08428        lsr     r8, r8, #8
807f5e08:       e1888c09        orr     r8, r8, r9, lsl #24
807f5e0c:       e1a09429        lsr     r9, r9, #8
807f5e10:       e1899c0c        orr     r9, r9, ip, lsl #24
807f5e14:       e1a0c42c        lsr     ip, ip, #8
807f5e18:       e18ccc0e        orr     ip, ip, lr, lsl #24
807f5e1c:       e8a013f8        stmia   r0!, {r3, r4, r5, r6, r7, r8, r9, ip}
807f5e20:       aaffffe3        bge     807f5db4 <__copy_from_user_std+0x188>
807f5e24:       e3720060        cmn     r2, #96 ; 0x60
807f5e28:       aaffffe2        bge     807f5db8 <__copy_from_user_std+0x18c>
807f5e2c:       e8bd03e0        pop     {r5, r6, r7, r8, r9}
807f5e30:       e212c01c        ands    ip, r2, #28
807f5e34:       0a000005        beq     807f5e50 <__copy_from_user_std+0x224>
807f5e38:       e1a0342e        lsr     r3, lr, #8
807f5e3c:       e491e004        ldr     lr, [r1], #4
807f5e40:       e25cc004        subs    ip, ip, #4
807f5e44:       e1833c0e        orr     r3, r3, lr, lsl #24
807f5e48:       e4803004        str     r3, [r0], #4
807f5e4c:       cafffff9        bgt     807f5e38 <__copy_from_user_std+0x20c>
807f5e50:       e2411003        sub     r1, r1, #3
807f5e54:       eaffffb3        b       807f5d28 <__copy_from_user_std+0xfc>
807f5e58:       e252201c        subs    r2, r2, #28
807f5e5c:       ba000025        blt     807f5ef8 <__copy_from_user_std+0x2cc>
807f5e60:       e92d03e0        push    {r5, r6, r7, r8, r9}
807f5e64:       f5d1f000        pld     [r1]
807f5e68:       e2522060        subs    r2, r2, #96     ; 0x60
807f5e6c:       f5d1f01c        pld     [r1, #28]
807f5e70:       ba000002        blt     807f5e80 <__copy_from_user_std+0x254>
807f5e74:       f5d1f03c        pld     [r1, #60]       ; 0x3c
807f5e78:       f5d1f05c        pld     [r1, #92]       ; 0x5c
807f5e7c:       f5d1f07c        pld     [r1, #124]      ; 0x7c
807f5e80:       e4914004        ldr     r4, [r1], #4
807f5e84:       e4915004        ldr     r5, [r1], #4
807f5e88:       e4916004        ldr     r6, [r1], #4
807f5e8c:       e4917004        ldr     r7, [r1], #4
807f5e90:       e1a0382e        lsr     r3, lr, #16
807f5e94:       e2522020        subs    r2, r2, #32
807f5e98:       e4918004        ldr     r8, [r1], #4
807f5e9c:       e4919004        ldr     r9, [r1], #4
807f5ea0:       e491c004        ldr     ip, [r1], #4
807f5ea4:       e491e004        ldr     lr, [r1], #4
807f5ea8:       e1833804        orr     r3, r3, r4, lsl #16
807f5eac:       e1a04824        lsr     r4, r4, #16
807f5eb0:       e1844805        orr     r4, r4, r5, lsl #16
807f5eb4:       e1a05825        lsr     r5, r5, #16
807f5eb8:       e1855806        orr     r5, r5, r6, lsl #16
807f5ebc:       e1a06826        lsr     r6, r6, #16
807f5ec0:       e1866807        orr     r6, r6, r7, lsl #16
807f5ec4:       e1a07827        lsr     r7, r7, #16
807f5ec8:       e1877808        orr     r7, r7, r8, lsl #16
807f5ecc:       e1a08828        lsr     r8, r8, #16
807f5ed0:       e1888809        orr     r8, r8, r9, lsl #16
807f5ed4:       e1a09829        lsr     r9, r9, #16
807f5ed8:       e189980c        orr     r9, r9, ip, lsl #16
807f5edc:       e1a0c82c        lsr     ip, ip, #16
807f5ee0:       e18cc80e        orr     ip, ip, lr, lsl #16
807f5ee4:       e8a013f8        stmia   r0!, {r3, r4, r5, r6, r7, r8, r9, ip}
807f5ee8:       aaffffe3        bge     807f5e7c <__copy_from_user_std+0x250>
807f5eec:       e3720060        cmn     r2, #96 ; 0x60
807f5ef0:       aaffffe2        bge     807f5e80 <__copy_from_user_std+0x254>
807f5ef4:       e8bd03e0        pop     {r5, r6, r7, r8, r9}
807f5ef8:       e212c01c        ands    ip, r2, #28
807f5efc:       0a000005        beq     807f5f18 <__copy_from_user_std+0x2ec>
807f5f00:       e1a0382e        lsr     r3, lr, #16
807f5f04:       e491e004        ldr     lr, [r1], #4
807f5f08:       e25cc004        subs    ip, ip, #4
807f5f0c:       e183380e        orr     r3, r3, lr, lsl #16
807f5f10:       e4803004        str     r3, [r0], #4
807f5f14:       cafffff9        bgt     807f5f00 <__copy_from_user_std+0x2d4>
807f5f18:       e2411002        sub     r1, r1, #2
807f5f1c:       eaffff81        b       807f5d28 <__copy_from_user_std+0xfc>
807f5f20:       e252201c        subs    r2, r2, #28
807f5f24:       ba000025        blt     807f5fc0 <__copy_from_user_std+0x394>
807f5f28:       e92d03e0        push    {r5, r6, r7, r8, r9}
807f5f2c:       f5d1f000        pld     [r1]
807f5f30:       e2522060        subs    r2, r2, #96     ; 0x60
807f5f34:       f5d1f01c        pld     [r1, #28]
807f5f38:       ba000002        blt     807f5f48 <__copy_from_user_std+0x31c>
807f5f3c:       f5d1f03c        pld     [r1, #60]       ; 0x3c
807f5f40:       f5d1f05c        pld     [r1, #92]       ; 0x5c
807f5f44:       f5d1f07c        pld     [r1, #124]      ; 0x7c
807f5f48:       e4914004        ldr     r4, [r1], #4
807f5f4c:       e4915004        ldr     r5, [r1], #4
807f5f50:       e4916004        ldr     r6, [r1], #4
807f5f54:       e4917004        ldr     r7, [r1], #4
807f5f58:       e1a03c2e        lsr     r3, lr, #24
807f5f5c:       e2522020        subs    r2, r2, #32
807f5f60:       e4918004        ldr     r8, [r1], #4
807f5f64:       e4919004        ldr     r9, [r1], #4
807f5f68:       e491c004        ldr     ip, [r1], #4
807f5f6c:       e491e004        ldr     lr, [r1], #4
807f5f70:       e1833404        orr     r3, r3, r4, lsl #8
807f5f74:       e1a04c24        lsr     r4, r4, #24
807f5f78:       e1844405        orr     r4, r4, r5, lsl #8
807f5f7c:       e1a05c25        lsr     r5, r5, #24
807f5f80:       e1855406        orr     r5, r5, r6, lsl #8
807f5f84:       e1a06c26        lsr     r6, r6, #24
807f5f88:       e1866407        orr     r6, r6, r7, lsl #8
807f5f8c:       e1a07c27        lsr     r7, r7, #24
807f5f90:       e1877408        orr     r7, r7, r8, lsl #8
807f5f94:       e1a08c28        lsr     r8, r8, #24
807f5f98:       e1888409        orr     r8, r8, r9, lsl #8
807f5f9c:       e1a09c29        lsr     r9, r9, #24
807f5fa0:       e189940c        orr     r9, r9, ip, lsl #8
807f5fa4:       e1a0cc2c        lsr     ip, ip, #24
807f5fa8:       e18cc40e        orr     ip, ip, lr, lsl #8
807f5fac:       e8a013f8        stmia   r0!, {r3, r4, r5, r6, r7, r8, r9, ip}
807f5fb0:       aaffffe3        bge     807f5f44 <__copy_from_user_std+0x318>
807f5fb4:       e3720060        cmn     r2, #96 ; 0x60
807f5fb8:       aaffffe2        bge     807f5f48 <__copy_from_user_std+0x31c>
807f5fbc:       e8bd03e0        pop     {r5, r6, r7, r8, r9}
807f5fc0:       e212c01c        ands    ip, r2, #28
807f5fc4:       0a000005        beq     807f5fe0 <__copy_from_user_std+0x3b4>
807f5fc8:       e1a03c2e        lsr     r3, lr, #24
807f5fcc:       e491e004        ldr     lr, [r1], #4
807f5fd0:       e25cc004        subs    ip, ip, #4
807f5fd4:       e183340e        orr     r3, r3, lr, lsl #8
807f5fd8:       e4803004        str     r3, [r0], #4
807f5fdc:       cafffff9        bgt     807f5fc8 <__copy_from_user_std+0x39c>
807f5fe0:       e2411001        sub     r1, r1, #1
807f5fe4:       eaffff4f        b       807f5d28 <__copy_from_user_std+0xfc>
807f5fe8:       d4d4d4d4        ldrble  sp, [r4], #1236 ; 0x4d4
807f5fec:       d4d4d4d4        ldrble  sp, [r4], #1236 ; 0x4d4
807f5ff0:       d4d4d4d4        ldrble  sp, [r4], #1236 ; 0x4d4
807f5ff4:       d4d4d4d4        ldrble  sp, [r4], #1236 ; 0x4d4
807f5ff8:       d4d4d4d4        ldrble  sp, [r4], #1236 ; 0x4d4
807f5ffc:       d4d4d4d4        ldrble  sp, [r4], #1236 ; 0x4d4

Getting the Pi kernel to boot in QEMU is still a WIP... A multi_v7_defconfig kernel boots when linked with LLD just fine: https://travis-ci.com/nathanchance/continuous-integration/jobs/162884028

[nickdesaulniers]

If the only difference is the linker, than the disassembly without relocations shouldn't show us any differences. The only way I know to print relocations is via [llvm-]objdump's -r flag. Either LLD is creating a bad relocation, or dropping a section it should not be.

Also, from the problematic section I wonder if user_filter is a C style string that could be printk'd or is just a bag of bytes? If it's a C string, maybe its value is a clue. It's possible that something else went wrong in the kernel, which then gave userspace a bad value, which passed it back into the kernel, at which point we then observe a failure (from a cascade of failures).

[nathanchance]

I will have to try and pull out my Pi and see if I can reproduce this with the latest tip of tree source.

[nickdesaulniers]

Looks like CrOS is now hitting this issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1020633#c39

[nickdesaulniers]

@m-gupta reports the diff in https://bugs.chromium.org/p/chromium/issues/detail?id=1020633#c47 works. RFT