Skip to content

[Bug]: ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory #1276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tubo opened this issue Jul 15, 2023 · 34 comments
Labels
bug Something isn't working linker Related to linking (ld, ld.so or dyld) python Support for Python, pip, Poetry, etc.

Comments

@Tubo
Copy link

Tubo commented Jul 15, 2023

Current Behavior (bug)
When using Python packages via requirements.txt, this time the package is ortools.

This bug was related to #710 and #760.
LD_LIBRARY_PATH is /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib

Content of LD_LIBRARY_PATH is:

libecpg.so       libecpg_compat.so       libpgcommon.a        libpgport_shlib.a     libpython3.so  python3.11
libecpg.so.6     libecpg_compat.so.3     libpgcommon_shlib.a  libpython3.11.so      pgxs           
libecpg.so.6.14  libecpg_compat.so.3.14  libpgport.a          libpython3.11.so.1.0  pkgconfig   

Error messages:

============================================================ test session starts =============================================================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
django: settings: radscheduler.settings (from env)
rootdir: /home/tubo/Sync/Projects/radscheduler/radscheduler
configfile: pytest.ini
plugins: django-4.5.2
collected 0 items / 1 error                                                                                                                  

=================================================================== ERRORS ===================================================================
_______________________________________________________ ERROR collecting core/tests.py _______________________________________________________
ImportError while importing test module '/home/tubo/Sync/Projects/radscheduler/radscheduler/core/tests.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/xlpbkwgf4nrnr56a3r0k0r32jxlxyf59-python3-3.11.4/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
radscheduler/core/tests.py:2: in <module>
    from .core import model
radscheduler/core/core.py:1: in <module>
    from ortools.sat.python import cp_model
../../../Projects/radscheduler/.devbox/virtenv/python311Packages.pip/.venv/lib/python3.11/site-packages/ortools/sat/python/cp_model.py:56: in <module>
    from ortools.sat.python import swig_helper
E   ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
========================================================== short test summary info ===========================================================
ERROR radscheduler/core/tests.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================== 1 error in 0.15s ==============================================================

Expected Behavior (fix)
Running pytest should not have this error.

Additional context

OS: KDE Neon (Ubuntu 22.04)
Devbox version: 0.5.5 to 0.5.7
devbox.json:

{
  "packages": [
    "python@latest",
    "python311Packages.pip@latest",
    "openssl@latest",
    "postgresql@latest",
    "python311Packages.numpy@latest"
  ],
  "env": {
    "DJANGO_SETTINGS_MODULE": "radscheduler.settings",
    "PGPORT": "5434"
  },
  "shell": {
    "init_hook": [
      ". $VENV_DIR/bin/activate",
      "pip install -r requirements.txt --use-pep517"
    ],
    "scripts": {
      "create_db": [
        "echo \"Creating DB\"",
        "dropdb --if-exists rs_db",
        "createdb rs_db",
        "psql rs_db -c \"CREATE USER rs_user WITH PASSWORD 'secretpassword';\"",
        "python radscheduler/manage.py makemigrations",
        "python radscheduler/manage.py migrate"
      ],
      "makemigrations": "radscheduler/manage.py makemigrations",
      "migrate": "radscheduler/manage.py migrate",
      "server": "radscheduler/manage.py runserver",
      "test": "pytest radscheduler"
    }
  }
}

requirements.txt:

django==4.2.2
psycopg2==2.9.5
sqlparse==0.4.4
asgiref==3.6.0
pytest==7.4.0
pytest-django==4.5.2
ortools==9.6.2534

@Tubo Tubo added the bug Something isn't working label Jul 15, 2023
@Lagoja
Copy link
Contributor

Lagoja commented Jul 18, 2023

Hey @Tubo, I'm working on recreating this. Do you have an example of the test you're trying to run with pytest that triggers this issue?

@Lagoja
Copy link
Contributor

Lagoja commented Jul 18, 2023

One thing that might work is running devbox add stdenv.cc.cc.lib, which should add the libstdc++ libraries to your dev environment.

@Tubo
Copy link
Author

Tubo commented Jul 19, 2023

Hey @Tubo, I'm working on recreating this. Do you have an example of the test you're trying to run with pytest that triggers this issue?

It's dead simple:

from django.test import TestCase

def test_pytest():
    assert False

@Tubo
Copy link
Author

Tubo commented Jul 19, 2023

One thing that might work is running devbox add stdenv.cc.cc.lib, which should add the libstdc++ libraries to your dev environment.

Nah didn't work unfortunately:

~/P/radscheduler ❯❯❯ devbox add stdenv.cc.cc.lib

Installing package: stdenv.cc.cc.lib.

[1/1] stdenv.cc.cc.lib
[1/1] stdenv.cc.cc.lib: Success
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/nd8dpdgpjkh3g3vvcjx9qdw7mjzz0l90-nix-2.13.3/lib/libnixstore.so)
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/nd8dpdgpjkh3g3vvcjx9qdw7mjzz0l90-nix-2.13.3/lib/libnixutil.so)
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/wy40dp0iwqs9hch6c6yqxriizzgpgm3z-aws-sdk-cpp-1.9.294/lib/libaws-cpp-sdk-transfer.so)
nix: /home/tubo/Sync/Projects/radscheduler/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /nix/store/wy40dp0iwqs9hch6c6yqxriizzgpgm3z-aws-sdk-cpp-1.9.294/lib/libaws-cpp-sdk-core.so)

Error: There was an error installing nix packages
source: error running "nix profile list": exit status 1

Error: There was an internal error. Run with DEVBOX_DEBUG=1 for a detailed error message, and consider reporting it at https://github.com/jetpack-io/devbox/issues

Debug information:

[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:72: Command stderr: 
[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/boxcli/midcobra/debug.go:74: 
ExecutionID:c0c228a182d34f178f2cddfc74900523
error running "nix profile list": exit status 1
go.jetpack.io/devbox/internal/nix.ProfileList
        go.jetpack.io/devbox/internal/nix/profiles.go:47
go.jetpack.io/devbox/internal/nix/nixprofile.ProfileListItems
        go.jetpack.io/devbox/internal/nix/nixprofile/profile.go:28
go.jetpack.io/devbox/internal/impl.(*Devbox).extraPackagesInProfile
        go.jetpack.io/devbox/internal/impl/packages.go:419
go.jetpack.io/devbox/internal/impl.(*Devbox).tidyProfile
        go.jetpack.io/devbox/internal/impl/packages.go:357
go.jetpack.io/devbox/internal/impl.(*Devbox).syncPackagesToProfile
        go.jetpack.io/devbox/internal/impl/packages.go:249
go.jetpack.io/devbox/internal/impl.(*Devbox).ensurePackagesAreInstalled
        go.jetpack.io/devbox/internal/impl/packages.go:207
go.jetpack.io/devbox/internal/impl.(*Devbox).Add
        go.jetpack.io/devbox/internal/impl/packages.go:97
go.jetpack.io/devbox/internal/boxcli.addCmdFunc
        go.jetpack.io/devbox/internal/boxcli/add.go:68
go.jetpack.io/devbox/internal/boxcli.addCmd.func1
        go.jetpack.io/devbox/internal/boxcli/add.go:42
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/[email protected]/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/[email protected]/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/[email protected]/command.go:968
go.jetpack.io/devbox/internal/boxcli/midcobra.(*midcobraExecutable).Execute
        go.jetpack.io/devbox/internal/boxcli/midcobra/midcobra.go:58
go.jetpack.io/devbox/internal/boxcli.Execute
        go.jetpack.io/devbox/internal/boxcli/root.go:102
go.jetpack.io/devbox/internal/boxcli.Main
        go.jetpack.io/devbox/internal/boxcli/root.go:123
main.main
        go.jetpack.io/devbox/cmd/devbox/main.go:11
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1598
[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/impl/dir.go:22: findProjectDir: path is 
[DEBUG] 2023/07/19 23:22:28 go.jetpack.io/devbox/internal/impl/dir.go:66: finding devbox.json in dir: /home/tubo/Projects/radscheduler

@Lagoja
Copy link
Contributor

Lagoja commented Jul 19, 2023

Gotcha that's finding the libstdc file now, but it's not the right version. Let me do a little more debugging from there

@aacebedo
Copy link

Got the same problem. I cannot import python modules if I do not add the stdlib. But when I do that, other packages such as libsrvg are not working anymore. Unsetting LD_LIBRARY_PATH in the init hook resolve the problem for packages but breaks the import.

@savil
Copy link
Collaborator

savil commented Aug 25, 2023

@aacebedo we're digging into this class of errors. Could you please share some files so we can repro it: devbox.json, devbox.lock, requirements.txt, and possibly some command to repro your specific error? While a good fix may take some time, we can try to advise you on how to get unblocked.

Feel free to pull it into a new issue to keep this one scoped to Tubo's scenario?

@aacebedo
Copy link

Thanks @savil for the answer.

here is my devbox.yaml

{
  "packages": [
    "python310@latest",
    "python310Packages.pip@latest",
    "[email protected]",
    "[email protected]",
    "[email protected]",
    "xorg.xorgserver@latest",
    "[email protected]",
    "[email protected]",
    "stdenv.cc.cc.lib",
    "[email protected]"
  ],
  "shell": {
    "init_hook": [
      ". $VENV_DIR/bin/activate",
      "pip install -q -r requirements.txt"
    ]
  },
  "nixpkgs": {
    "commit": "a16f7eb56e88c8985fcc6eb81dabd6cade4e425a"
  }
}

and the requirements file:

Sphinx==5.3.0
sphinx_rtd_theme==1.1.1
sphinxcontrib-plantuml==0.25
sphinxcontrib-svg2pdfconverter==1.2.1
sphinxcontrib-drawio==0.0.16
sphinxcontrib-bibtex==2.5.0
Pillow==9.3.0
matplotlib==3.7.2

If you try it you will see that python import matplotlib is working but not the srvg-convert binary. If you add
unset LD_LIBRARY_PATH to the init hook, srvg-convert works but not the python import matplotlib with the same error as OP.

@savil
Copy link
Collaborator

savil commented Aug 25, 2023

Thanks @aacebedo. I'm not quite able to repro because the drawio nix package appears to be broken, so it won't install (for my intel mac). I can try devbox generate dockerfile and running it in the debian container, assuming the linux package is not broken. What OS are you on?

(I have to pause for a few hours for some personal work, but can look at it a bit later today)

@aacebedo
Copy link

aacebedo commented Aug 25, 2023

Thanks ! I am using it on NixOS.

@savil
Copy link
Collaborator

savil commented Aug 25, 2023

@aacebedo in a debian-container I can get a devbox shell working.

I can import matplotlib:

(.venv) (devbox) devbox@0616836e2023:/code$ cat main.py
import matplotlib

def main():
    print("hello world")

main()
(.venv) (devbox) devbox@0616836e2023:/code$ python main.py
hello world

but running rsvg-convert seems to complain...

/nix/store/0ccn204v5kds9i08x7plf4x644l6wm87-librsvg-2.52.5/bin/rsvg-convert
/nix/store/0ccn204v5kds9i08x7plf4x644l6wm87-librsvg-2.52.5/bin/rsvg-convert: /nix/store/4s21k8k7p1mfik0b33r2spq5hq7774k1-glibc-2.33-108/lib/libc.so.6: version `GLIBC_2.35' not found (required by /code/.devbox/nix/profile/default/lib/libgcc_s.so.1)
/nix/store/0ccn204v5kds9i08x7plf4x644l6wm87-librsvg-2.52.5/bin/rsvg-convert: /nix/store/4s21k8k7p1mfik0b33r2spq5hq7774k1-glibc-2.33-108/lib/libc.so.6: version `GLIBC_2.34' not found (required by /code/.devbox/nix/profile/default/lib/libgcc_s.so.1)

Is this the error you saw?

@savil
Copy link
Collaborator

savil commented Aug 25, 2023

For some reason rsvg is not in PATH for me...

@aacebedo
Copy link

It is exactly this issue. You can also try convert that exhibit a similar issue.

@aacebedo
Copy link

you can try adding unset LD_LIBRARY_PATH to the init hook

@savil
Copy link
Collaborator

savil commented Aug 25, 2023

@aacebedo I think changing the devbox.lock to use this nixpkgs commit hash for stdenv.cc.cc.lib should unblock you:

     "stdenv.cc.cc.lib": {
      "resolved": "github:NixOS/nixpkgs/c6fd903606866634312e40cceb2caee8c0c9243f#stdenv.cc.cc.lib",
      "source": "nixpkg"
    },

We're investigating more principled fixes.

@aacebedo
Copy link

aacebedo commented Sep 9, 2023

Hi,
I tried the modification but without success.
convert and rsvg-convert still does not start with libc.so.6 errors

@savil
Copy link
Collaborator

savil commented Sep 13, 2023

@aacebedo I was discussing a similar issue with a user on Discord (is that you as well?).

In that case, something like this helped:

# main.py
import matplotlib

def main():
    print("hello world")

main()

# requirements.txt
matplotlib==3.7.2

# devbox.json
{
  "packages": [
    "python310@latest",
    "python310Packages.pip@latest",
    "drawio@latest",
    "librsvg@latest",
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib"
  ],
  "shell": {
    "init_hook": [
      "export DEVBOX_DEBUG=1",
      ". $VENV_DIR/bin/activate",
      "pip install -q -r requirements.txt"
    ]
  },
  "nixpkgs": {
    "commit": "73e1976309fc789706b9f306407e9e7622a57d25"
  }
}


# devbox.lock
{
  "lockfile_version": "1",
  "packages": {
    "drawio@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#drawio",
      "source": "devbox-search",
      "version": "21.6.8"
    },
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib": {},
    "librsvg@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#librsvg",
      "source": "devbox-search",
      "version": "2.56.3"
    },
    "python310@latest": {
      "last_modified": "2023-02-24T18:08:35Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/806075be2bdde71895359ed18cb530c4d323e6f6#python3",
      "source": "devbox-search",
      "version": "3.10.9"
    },
    "python310Packages.pip@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#python310Packages.pip",
      "source": "devbox-search",
      "version": "23.2.1"
    }
  }

The reasoning is:

  • The error seen was: Original error was: /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libc.so.6: version 'GLIBC_2.36' not found (required by /tmp/test/.devbox/nix/profile/default/lib/libstdc++.so.6)
  • What's happening is that that the stdenv.cc.cc.lib is expecting a GLIBC 2.36, but the code executed by the python binary (either directly, or indirectly via the matplotlib -> numpy dependencies) is linked against GLIBC 2.35.
  • I searched https://www.nixhub.io/packages/glibc and version 2.35-224 is from nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#glibc
  • So, to try fixing it I replaced stdenv.cc.cc.lib with github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib

@aacebedo
Copy link

@aacebedo I was discussing a similar issue with a user on Discord (is that you as well?).

In that case, something like this helped:

# main.py
import matplotlib

def main():
    print("hello world")

main()

# requirements.txt
matplotlib==3.7.2

# devbox.json
{
  "packages": [
    "python310@latest",
    "python310Packages.pip@latest",
    "drawio@latest",
    "librsvg@latest",
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib"
  ],
  "shell": {
    "init_hook": [
      "export DEVBOX_DEBUG=1",
      ". $VENV_DIR/bin/activate",
      "pip install -q -r requirements.txt"
    ]
  },
  "nixpkgs": {
    "commit": "73e1976309fc789706b9f306407e9e7622a57d25"
  }
}


# devbox.lock
{
  "lockfile_version": "1",
  "packages": {
    "drawio@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#drawio",
      "source": "devbox-search",
      "version": "21.6.8"
    },
    "github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib": {},
    "librsvg@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#librsvg",
      "source": "devbox-search",
      "version": "2.56.3"
    },
    "python310@latest": {
      "last_modified": "2023-02-24T18:08:35Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/806075be2bdde71895359ed18cb530c4d323e6f6#python3",
      "source": "devbox-search",
      "version": "3.10.9"
    },
    "python310Packages.pip@latest": {
      "last_modified": "2023-09-10T10:53:27Z",
      "plugin_version": "0.0.1",
      "resolved": "github:NixOS/nixpkgs/78058d810644f5ed276804ce7ea9e82d92bee293#python310Packages.pip",
      "source": "devbox-search",
      "version": "23.2.1"
    }
  }

The reasoning is:

  • The error seen was: Original error was: /nix/store/lqz6hmd86viw83f9qll2ip87jhb7p1ah-glibc-2.35-224/lib/libc.so.6: version 'GLIBC_2.36' not found (required by /tmp/test/.devbox/nix/profile/default/lib/libstdc++.so.6)
  • What's happening is that that the stdenv.cc.cc.lib is expecting a GLIBC 2.36, but the code executed by the python binary (either directly, or indirectly via the matplotlib -> numpy dependencies) is linked against GLIBC 2.35.
  • I searched https://www.nixhub.io/packages/glibc and version 2.35-224 is from nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#glibc
  • So, to try fixing it I replaced stdenv.cc.cc.lib with github:nixos/nixpkgs/6b70761ea8c896aff8994eb367d9526686501860#stdenv.cc.cc.lib

Yes it is me. I thought I could find some help from the community on it. Still I am facing other libc errors on other software installed with devbox so I try to investigate this with the clue about the libc version change you gave me.

@savil
Copy link
Collaborator

savil commented Oct 6, 2023

@aacebedo we've been researching a fix for your issue. In this repository https://github.com/savil/devbox-issue-1276, I have a devbox.json and some local flake.nix files that illustrate an approach to solving this problem. The README explains a bit about what its doing.

Could you please try it out in your project, and let us know if this approach works?

If it does, we can build this as a feature in Devbox. Ideally, it should work with minimal user work i.e. no need to manually write your own flake.nix.

@aacebedo
Copy link

aacebedo commented Nov 1, 2023

@aacebedo we've been researching a fix for your issue. In this repository https://github.com/savil/devbox-issue-1276, I have a devbox.json and some local flake.nix files that illustrate an approach to solving this problem. The README explains a bit about what its doing.

Could you please try it out in your project, and let us know if this approach works?

If it does, we can build this as a feature in Devbox. Ideally, it should work with minimal user work i.e. no need to manually write your own flake.nix.

Thanks for the work !
Tried it and convert is still exhibiting the issue :(

@savil
Copy link
Collaborator

savil commented Nov 1, 2023

@aacebedo That's a bummer. Sorry to hear that. A couple of requests:

Request 1: repro code
I thought we were running the same code as you, so am surprised it is not working. It would help if you could share a github repository with the project code you are trying to run, which exhibits this issue. That would help us debug the exact code you're running.

Request 2: try Devbox 0.8.0-dev's glibc_patch feature
In the upcoming Devbox release (release notes), @gcurtis implemented this approach (with some details modified) as a product feature.

  • You can use this version of devbox by setting export DEVBOX_USE_VERSION=0.8.0-dev.
  • You'll need to modify packages in devbox.json to a be json map. And set patch_glibc: true for python and librsvg. Refer to the PR Summary of as an example.
  • You'll also need to initialize the python virtualenv as shown in the virtenv script in that PR's example.

It would be great if you could try that out, and let us know if it works or not for you. Thanks!

@aacebedo
Copy link

aacebedo commented Nov 1, 2023

@aacebedo That's a bummer. Sorry to hear that. A couple of requests:

Request 1: repro code I thought we were running the same code as you, so am surprised it is not working. It would help if you could share a github repository with the project code you are trying to run, which exhibits this issue. That would help us debug the exact code you're running.

Request 2: try Devbox 0.8.0-dev's glibc_patch feature In the upcoming Devbox release (release notes), @gcurtis implemented this approach (with some details modified) as a product feature.

  • You can use this version of devbox by setting export DEVBOX_USE_VERSION=0.8.0-dev.
  • You'll need to modify packages in devbox.json to a be json map. And set patch_glibc: true for python and librsvg. Refer to the PR Summary of as an example.
  • You'll also need to initialize the python virtualenv as shown in the virtenv script in that PR's example.

It would be great if you could try that out, and let us know if it works or not for you. Thanks!

I cloned the devbox-issue-1276 repo and ran devbox shell in it. Then I ran convert and saw the issue :(
I am running devbox 0.7.1 but I'll give a try to the upcoming release as you adviced.

@savil
Copy link
Collaborator

savil commented Nov 1, 2023

I cloned the devbox-issue-1276 repo and ran devbox shell in it. Then I ran convert and saw the issue :(

@aacebedo 😱 oh wow. That is odd. Its been a while, but IIRC I'd been running the Dockerfile in that repo to test it out on a debian-slim image (since I have a Mac; this issue affects linux). I'll have to retry my steps in case something changed so that it is no longer working.

@savil
Copy link
Collaborator

savil commented Nov 6, 2023

@aacebedo sorry for the delay. After trying out the steps, I think I understand the problem. In my prototype, I've patched python and librsvg using the local flake. Those do work.

You are trying convert that came from the imagemagick nix package. You'll need to patch that via a local flake (as I illustrated with python or librsvg). We don't patch all packages automatically to be conservative, as I explain at the end of the README of https://github.com/savil/devbox-issue-1276, and until we get some validating feedback from real world projects.

Or better, I'd suggest using Devbox 0.8.0-dev and modifying the devbox.json to specify patch_glibc:true for whichever packages are hitting the libc errors. Let us know if that works, and thanks for pursuing this with us!

@aacebedo
Copy link

aacebedo commented Nov 6, 2023

@savil thanks a lot to take a look to this !

Ok I'll try the 0.8.0-dev to check this out

@aacebedo
Copy link

aacebedo commented Nov 6, 2023

@aacebedo sorry for the delay. After trying out the steps, I think I understand the problem. In my prototype, I've patched python and librsvg using the local flake. Those do work.

You are trying convert that came from the imagemagick nix package. You'll need to patch that via a local flake (as I illustrated with python or librsvg). We don't patch all packages automatically to be conservative, as I explain at the end of the README of https://github.com/savil/devbox-issue-1276, and until we get some validating feedback from real world projects.

Or better, I'd suggest using Devbox 0.8.0-dev and modifying the devbox.json to specify patch_glibc:true for whichever packages are hitting the libc errors. Let us know if that works, and thanks for pursuing this with us!

Ok, so I checked out the v0.8.0-dev, built it (go build ./cmd/devbox), ran a sudo nix-collect-garbage -d.
I exported DEVBOX_USE_VERSION=0.8.0-dev and finally did a devbox shell with the following file:

{
  "packages": {
    "imagemagick": {
      "version": "6.9.11-60",
      "patch_glibc": true
    },
    "librsvg": {
      "version": "2.52.5"
    }
  }
}

and convert still does not work :(

convert: /nix/store/3wa1xwnfv8ada1za1r8m4vmsiz1jifqq-glibc-2.32-35/lib/libc.so.6: version `GLIBC_2.33' not found (required by /tmp/devbox-issue-1276/.devbox/nix/profile/default/lib/librsvg-2.so.2)
convert: /nix/store/3wa1xwnfv8ada1za1r8m4vmsiz1jifqq-glibc-2.32-35/lib/libc.so.6: version `GLIBC_2.33' not found (required by /nix/store/nw2gv9l8lzi4scbddbg1vfvivzcpvag3-harfbuzz-3.2.0/lib/libharfbuzz.so.0)

@savil
Copy link
Collaborator

savil commented Nov 6, 2023

@aacebedo indeed, I think something is not working correctly. I can repro the error you are seeing. We're looking into it...

@savil
Copy link
Collaborator

savil commented Nov 6, 2023

@aacebedo thank you ! @gcurtis and I just looked at this. We added patch_glibc: true to librsvg.

This exposed a couple of issues:

  1. A bug in the current implementation. The patch_glibc was only working for one package. The fix for that is straightforward.
  2. We still do run into an error executing convert. The error has to do with a libpango not finding a g_memdup2 symbol. The root cause of that appears to be that there's a version mismatch with a different library called glib (https://gitlab.gnome.org/GNOME/glib/). We need to investigate a good fix for this, but sadly it'll take a bit of time :(. Will circle back when we have some more news.

@aacebedo
Copy link

aacebedo commented Nov 6, 2023

@aacebedo thank you ! @gcurtis and I just looked at this. We added patch_glibc: true to librsvg.

This exposed a couple of issues:

  1. A bug in the current implementation. The patch_glibc was only working for one package. The fix for that is straightforward.
  2. We still do run into an error executing convert. The error has to do with a libpango not finding a g_memdup2 symbol. The root cause of that appears to be that there's a version mismatch with a different library called glib (https://gitlab.gnome.org/GNOME/glib/). We need to investigate a good fix for this, but sadly it'll take a bit of time :(. Will circle back when we have some more news.

Thanks to you both looking into this !
I'll continue to watch the issue and test fixes if it can help to solve this issue.

@sandangel
Copy link

I have the same issue. I was using poetry command after installing it with devbox add poetry.
Before devbox, when using nix flake, I had to wrap poetry with this:

          poetry = pkgs.symlinkJoin {
            name = "poetry";
            paths = [ pkgs.poetry ];
            buildInputs = [ pkgs.makeWrapper ];
            postBuild = ''
              wrapProgram $out/bin/poetry --set LD_LIBRARY_PATH ${lib.makeLibraryPath [ pksgs.stdenv.cc.cc.lib ]}
            '';
          };
          ciDeps = with pkgs; [
            cmake
          ] ++ [ poetry ];

@gcurtis gcurtis added python Support for Python, pip, Poetry, etc. linker Related to linking (ld, ld.so or dyld) labels Jan 30, 2024
@augustebaum
Copy link

augustebaum commented May 14, 2024

Same problem here. Unsetting LD_LIBRARY_PATH does not solve the issue; neither does devbox add stdenv.cc.cc.lib (or any combination of the two).
I also tried managing my Python dependencies by running devbox add [email protected] and following the instructions (installing packages with pip instead of poetry).

EDIT: Solved by setting LD_LIBRARY_PATH to <my-project-path>/.devbox/nix/profile/default/lib/.

EDIT: Not quite solved: My script now works, but:

  1. python3 -c "import numpy" fails with:
(outerbounds-examples-py3.11) ❯ p3 -c 'import numpy'
Traceback (most recent call last):
  File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libz.so.1: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/__init__.py", line 130, in <module>
    from numpy.__config__ import show as show_config
  File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy.core._multiarray_umath import (
  File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.11 from "/home/auguste/Desktop/outerbounds-examples/.venv/bin/python3"
  * The NumPy version is: "1.26.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libz.so.1: cannot open shared object file: No such file or directory


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/auguste/Desktop/outerbounds-examples/.venv/lib/python3.11/site-packages/numpy/__init__.py", line 135, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

although I think it might've failed even before the fix.

  1. All nix commands now fail with
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by nix)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by nix)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixexpr.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixexpr.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixmain.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixmain.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixfetchers.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixfetchers.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixstore.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixstore.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixutil.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixutil.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixcmd.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.31' not found (required by /nix/store/v7gdmdc9yhx3ngh840xx5kr46840s73b-nix-2.18.2/lib/libnixcmd.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/dgc42gd76af2syp7fk5d9mv2abqrggkg-aws-sdk-cpp-1.11.207/lib/libaws-cpp-sdk-transfer.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/dgc42gd76af2syp7fk5d9mv2abqrggkg-aws-sdk-cpp-1.11.207/lib/libaws-cpp-sdk-s3.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/dgc42gd76af2syp7fk5d9mv2abqrggkg-aws-sdk-cpp-1.11.207/lib/libaws-cpp-sdk-core.so)
nix: /home/auguste/Desktop/outerbounds-examples/.devbox/nix/profile/default/lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /nix/store/a88l22177h3mb4ifiy734mr71c40pfxp-aws-crt-cpp-0.24.7/lib/libaws-crt-cpp.so)

EDIT: Importing numpy does not fail when done in IPython (installed through devbox). Uh, what?

@augustebaum
Copy link

I ended up using a shell.nix instead, using nixpkgs unstable rather than 23.11, and that seemed to work alright. I'll try devbox again when I have the time.

@dwt
Copy link

dwt commented Oct 2, 2024

This problem looks very similar to a problem I worked around recently. Maybe it is the same one here?

To See the fix, uncomment the line $DYLD_FALLBACK_LIBRARY_PATH.

@Tubo
Copy link
Author

Tubo commented Oct 4, 2024

Resolved in latest devbox.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working linker Related to linking (ld, ld.so or dyld) python Support for Python, pip, Poetry, etc.
Development

No branches or pull requests

8 participants