[Bug]: Unable to get Python pandas to work

[mvgijssel]

Current Behavior (bug)

I'm unable to run a test suite which includes the PyPI library pandas. I get the following stack trace trying to execute the test suite using Bazel:

    import pandas._libs.window.aggregations as window_aggregations
E   ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory

Now adding in stdenv.cc.cc.lib as mentioned here (#1596) and here (#1276) results in Python and Devbox itself to stop working due to missing GLIBC_2.*:

python3: /nix/store/qyd4c78dl0kbpbc3lc2klp5bzy6585g7-glibc-2.33-108/lib/libc.so.6: version `GLIBC_2.35' not found (required by /project/.devbox/nix/profile/default/lib/libgcc_s.so.1)
python3: /nix/store/qyd4c78dl0kbpbc3lc2klp5bzy6585g7-glibc-2.33-108/lib/libc.so.6: version `GLIBC_2.34' not found (required by /project/.devbox/nix/profile/default/lib/libgcc_s.so.1)

and devbox

devcontainer@accb8d220bc6:/project$ devbox add vim
2023/11/21 12:22:24 findProjectDir: path is
2023/11/21 12:22:24 finding devbox.json in dir: /project
2023/11/21 12:22:24 findProjectDir: path is
2023/11/21 12:22:24 finding devbox.json in dir: /project
Error: exit status 1

2023/11/21 12:22:24 Command stderr: nix: /nix/store/0ckxcm0bnsh64a4vi40d5wjs96i014nl-glibc-2.37-8/lib/libc.so.6: version `GLIBC_2.38' not found (required by /project/.devbox/nix/profile/default/lib/libstdc++.so.6)

2023/11/21 12:22:24
ExecutionID:cb862a02fed74ffdaf8c730630c435ba
<nil>
2023/11/21 12:22:24 findProjectDir: path is
2023/11/21 12:22:24 finding devbox.json in dir: /project

Expected Behavior (fix)
I expect running Python libraries within a Devbox shell to work

Additional context

devcontainer@accb8d220bc6:/project$ devbox version -v
2023/11/21 12:23:45 findProjectDir: path is
2023/11/21 12:23:45 finding devbox.json in dir: /project
2023/11/21 12:23:45 findProjectDir: path is
2023/11/21 12:23:45 finding devbox.json in dir: /project
Version:     0.8.2
Platform:    linux_arm64
Commit:      33ea788cb4dbf6958208669c9e4df9c1e858ad2c
Commit Time: 2023-11-15T00:23:52Z
Go Version:  go1.21.3
Launcher:    0.2.0
2023/11/21 12:23:45 findProjectDir: path is
2023/11/21 12:23:45 finding devbox.json in dir: /project

Using the following devbox file:

{
  "packages": {
    "docker": "20.10.26",
    "docker-compose": "2.23.0",
    "bazelisk": "1.18.0",
    "python": "3.9.10",
    "nodejs": "16.12.0",
    "postgresql_13": "13.6",
    "black": "22.12.0",
    "nbstripout": "0.6.1",
    "pre-commit": "3.3.3",
    "sqlfluff": "1.4.5",
    "nodePackages.prettier": "2.8.8",
    "buildifier": "5.1.0",
    "go-task": "3.24.0",
    "bazel-remote": "2.3.9",
    "awscli2": "2.13.3",
    "git": "2.42.0",
    "moreutils": "0.67"
  },
  "shell": {
    "init_hook": ["alias bazel='bazelisk'"],
    "scripts": {
      "test": ["echo \"Error: no test specified\" && exit 1"]
    }
  }
}

and the following Dockerfile

# NOTE: not pinning on sha here so we are pulling the right image for the right arch
FROM ubuntu:mantic-20231011

ARG TARGETARCH

# Setup right locale
ENV LC_ALL="en_US.UTF-8" 
ENV LANG="en_US.UTF-8" 
ENV LANGUAGE="en_US.UTF-8"
RUN apt-get update \
    && apt-get install -y locales locales-all \
    && locale-gen en_US.UTF-8 \
    && update-locale LANG=en_US.UTF-8 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Setup the Europe/Amsterdam timezone
ENV TZ=Europe/Amsterdam
RUN apt-get update \
    && apt-get install -y tzdata \
    && ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Setup non-root user for the container
# with the ability to use passwordless sudo
RUN apt-get update \
    && apt-get install -y sudo \
    && userdel --remove ubuntu \
    && useradd -u 1000 -m devcontainer -s /bin/bash \
    && usermod -a -G sudo devcontainer \
    && echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /project
RUN chown devcontainer:devcontainer /project

# Install nix and devbox installer dependencies
RUN apt-get update \
    && apt-get install -y ca-certificates curl xz-utils \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

USER devcontainer

RUN curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install linux --init none --no-confirm
RUN sudo chown -R devcontainer:devcontainer /nix

RUN curl -fsSL https://get.jetpack.io/devbox -o /tmp/install-devbox.sh && \
    chmod +x /tmp/install-devbox.sh && \
    /tmp/install-devbox.sh -f && \
    rm -f /tmp/install-devbox.sh

COPY --chown=devcontainer devbox.json devbox.lock ./

RUN devbox run -- echo "Installed Packages."

RUN devbox shellenv --init-hook >> ~/.profile

SHELL ["/bin/bash", "--login",  "-c"]

# Ensure bazel is globally enabled (not sure why the alias does not work inside devbox.json)
RUN sudo ln -s $(which bazelisk) /usr/local/bin/bazel

# The py_wheel rule requires a global python and does not work with the devbox one.
RUN sudo ln -s /project/.devbox/nix/profile/default/bin/python /usr/bin/python
RUN sudo ln -s /project/.devbox/nix/profile/default/bin/python /usr/bin/python3

COPY --chown=devcontainer . .
RUN mkdir -p /home/devcontainer/bazel