Packager script broken on Arch Linux

[skrat]

So, adding NO_LIBC to aws_lambda_package_target(${PROJECT_NAME}) solved my issues. I looked into packager and found that if NO_LIBC is not set, then PKG_LD remains empty, and generates bootstrap file like this:

#!/bin/bash
set -euo pipefail
export AWS_EXECUTION_ENV=lambda-cpp
exec $LAMBDA_TASK_ROOT/lib/ --library-path $LAMBDA_TASK_ROOT/lib $LAMBDA_TASK_ROOT/bin/hello ${_HANDLER}

Which of course fails. Looks pretty similar to #45 , this is on Arch Linux

[skrat]

Furthermore, when I run package_libc_via_pacman, I get

[skrat@apex build]$ pacman --files --list --quiet glibc | sed -E '/\.so$|\.so\.[0-9]+$/!d'
warning: database file for 'core' does not exist (use '-Fy' to download)
warning: database file for 'extra' does not exist (use '-Fy' to download)
warning: database file for 'community' does not exist (use '-Fy' to download)
warning: database file for 'multilib' does not exist (use '-Fy' to download)
warning: database file for 'repo-ck' does not exist (use '-Fy' to download)
error: package 'glibc' was not found

So I run sudo pacman -Fy, then try again:

[skrat@apex build]$ pacman --files --list --quiet glibc | sed -E '/\.so$|\.so\.[0-9]+$/!d'
usr/lib/audit/sotruss-lib.so
usr/lib/gconv/ANSI_X3.110.so
usr/lib/gconv/ARMSCII-8.so
usr/lib/gconv/ASMO_449.so
usr/lib/gconv/BIG5.so
usr/lib/gconv/BIG5HKSCS.so
usr/lib/gconv/BRF.so
usr/lib/gconv/CP10007.so
usr/lib/gconv/CP1125.so
...

But still no cigar, so I looked at the packager:

list=$(ldd "$PKG_BIN_PATH" | awk '{print $(NF-1)}')
libc_libs=()
libc_libs+=($(package_libc_via_dpkg))
libc_libs+=($(package_libc_via_rpm))
libc_libs+=($(package_libc_via_pacman))

Should this really be union? Shouldn't it be one or the other? I have all 3 on my system because I package for those paltforms. Below is fixed version of package_libc_via_pacman, yours doesn't work, there's no ID_LIKE in /etc/os-release. Also, why do you look into /etc/os-release just for pacman and only do type check for dpkg and rpm?

function package_libc_via_pacman {
    if [[ $(sed '/^ID/!d;s/ID=//' < /etc/os-release) == "arch" ]]; then
        if type pacman > /dev/null 2>&1; then
            pacman --files --list --quiet glibc | sed -E '/\.so$|\.so\.[0-9]+$/!d' | sed 's/^usr/\/usr/'
        fi
    fi
}

But... still no cigar, now the hasElement is filtering out the loader. I don't want to say that the packager script is utterly broken, but it's very hard to follow what's going on there.

[marcomagdy]

adding NO_LIBC to aws_lambda_package_target(${PROJECT_NAME}) solved my issues.

If I recall correctly, ArchLinux uses GNU libc, so I strongly recommend against using NO_LIBC. NO_LIBC is meant to be used only on AmazonLinux 2017.03 since it is guaranteed to be compatible with Lambda's environment.
If you're using ArchLinux with musl libc, then you should be fine and better off by specifying NO_LIBC actually.

hasElement is filtering out the loader.

That was fixed in v0.2.2, which version are you using?

Should this really be union? Shouldn't it be one or the other? I have all 3 on my system because I package for those platforms.

It should not be a union. It should be just one. If you have all 3 on your system, only one of them should have GNU libc files.

I don't want to say that the packager script is utterly broken

No. It's not. It works perfectly fine on Debian, Ubuntu, Red Hat, Amazon Linux and Alpine. I don't have continuous integration for ArchLinux (I will add it after your feedback)

[skrat]

I'm using git HEAD

[skrat]

It appears to kick in when the binary is linked against /usr/lib64, in that case, the set intersection that the list, libc_lib and hasElement is doing is failing, because pacman only reports files in /usr/lib

 ➜ ~/W/r/h/build pacman --files --list --quiet glibc | sed -E '/\.so$|\.so\.[0-9]+$/!d' | sed 's/^usr/\/usr/' | grep ld
/usr/lib/ld-2.29.so
/usr/lib/ld-linux-x86-64.so.2
}

Whereas cmake/gcc linked the binary to /usr/lib64:

 ➜ ~/W/r/h/build ldd hello | awk '{print $(NF-1)}' | grep ld
/usr/lib64/ld-linux-x86-64.so.2

All other libraries that it links to are in /usr/lib

UPDATE So on Arch (maybe elsewhere) /usr/lib64 is a symlink to /usr/lib

[marcomagdy]

Quick update, I updated the script (in a branch) to handle the case of linking to the loader's symlink instead of the loader itself.

Also, I changed the test for Arch Linux. Instead of looking for the pattern ID_LIKE, I just grep for "Arch Linux" in /etc/os-release. ID_LIKE used to exist in the Arch Linux version I tested with back in November 2018 (latest from Docker at the time). That seems to not be the case with the latest version though.

I ran into a snag with pacman and I'm about to take a few days off, so that's the bad news. The output from pacman's database is not absolute file paths, instead they're like usr/lib/.. which makes it tricky to script properly. I will work on this as soon as I get back. Sorry for the inconvenience.

Also, it seems that Arch Linux ships with both glibc and musl libc. So, it might be worth looking into how to build your code on Arch using musl.

[namarrgon]

Using "pacman -Flq", i.e. querying the remote file-list, is not an ideal way since one risks to run into desync issues between the remote state and the current state of the system. "pacman -Qlq glibc" will ensure that the file-list is current and it will return absolute paths as a bonus.

[marcomagdy]

@namarrgon awesome. I’ll give that a try

[skrat]

@namarrgon @marcomagdy I tried the arch branch, and I can confirm that it currently is broken, however, after patching packager with pacman -Qlq glibc, all is well, the packager finds the loader and everything else. Without the patch, it just fails on the relative file paths.

[skrat]

@namarrgon Would it be possible to merge those changes along with the updated pacman invocation? Thank you

[marcomagdy]

It's ready to merge, pending review.