DllNotFoundException on Mono Linux

[xoofx]

Hi there!

While trying to run git-rocket-filter on Linux, I'm getting a DllNotFoundException exception if I don't run the exe from the same folder where LibGit2Sharp.dll is.

Enabling MONO_LOG_LEVEL = debug, I'm getting weird resolves (verbose log below). It adds a little prefix lib to the NativeBinaries folder. So in the end, it gets something like /home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so while it should be /home/xoofx/GitRocketFilter/NativeBinaries/linux/amd64/libgit2-4d6362b.so it cannot resolve the assembly correctly. It is working when we are in the same folder because in this case, dllmap doesn't mess-up by adding lib prefix...
Is this a known issue? Did Mono change their dllmap stuff recently?

Platform: Ubuntu 14.0+ x64
Mono version: 3.2.8+dfsg-4ubuntu1.1

Mono: DllImport attempting to load: 'NativeBinaries/linux/amd64/libgit2-4d6362b.so'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'NativeBinaries/linux/amd64/libgit2-4d6362b.so': 'NativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport unable to load library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport attempting to load: 'NativeBinaries/linux/amd64/libgit2-4d6362b.so'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'NativeBinaries/linux/amd64/libgit2-4d6362b.so': 'NativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport unable to load library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport attempting to load: 'NativeBinaries/linux/amd64/libgit2-4d6362b.so'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'NativeBinaries/linux/amd64/libgit2-4d6362b.so': 'NativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport unable to load library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport attempting to load: 'NativeBinaries/linux/amd64/libgit2-4d6362b.so'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'NativeBinaries/linux/amd64/libgit2-4d6362b.so': 'NativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport unable to load library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport attempting to load: 'NativeBinaries/linux/amd64/libgit2-4d6362b.so'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'NativeBinaries/linux/amd64/libgit2-4d6362b.so': 'NativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so': '/home/xoofx/GitRocketFilter/libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport error loading library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so': 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.
Mono: DllImport unable to load library 'libNativeBinaries/linux/amd64/libgit2-4d6362b.so: cannot open shared object file: No such file or directory'.

[bording]

While trying to run git-rocket-filter on Linux, I'm getting a DllNotFoundException exception if I don't run the exe from the same folder where LibGit2Sharp.dll is.

I'm not clear on what scenario you're describing here. Are you saying your exe and LibGit2Sharp.dll aren't located in the same directory?

[xoofx]

Oh, no, my directory is a standard one, like:

• git-rocket-filter.exe
• LibGit2Sharp.dll
• LibGit2Sharp.dll.config
• NativeBinaries/

In fact, looking at Mono code mono/loader.c:1532, It looks like they didn't really expect sub folders, so they prefix by lib if it does not start by lib, which is fine for a dynamic library file name, but if the dllmap.target starts by a directory, it doesn't work if the directory does not start by "lib".

A workaround is to rename the NativeBinaries folder by something like libsnative (just tested and this is working) Do you want me to PR on NativeBinaries?

[bording]

I'm still trying to figure out what scenario you're running into where this is a problem, because this isn't something I've run into.

Can you point me at some code and a set or steps for me to see what you're talking about?

[xoofx]

I'm still trying to figure out what scenario you're running into where this is a problem, because this isn't something I've run into.
Can you point me at some code and a set or steps for me to see what you're talking about?

Sure, that's pretty easy to reproduce. Anykind of exe application using LibGit2Sharp on Linux won't work if it is not run from the folder they are installed.

You can try by downloading previous v1.1 version of GitRocketFilter (binaries). Unzip to a folder, go to a differnt folder and try to run mono ../GitRocketFilter/git-rocket-filter.exe and you should get a DllNotFoundException.

[bording]

Ok, now I see what you're talking about, and yeah it does seem to have a problem there. 😦

Looking at what it's doing, to me it feels more like a problem with mono's loader code and how it interacts with the dllmap entries. If the entry specifies a path, it should be trying to use it directly as specified first, before falling back into the "let's add 'lib' everywhere and see if that works" code path.

The workaround does appear to be pretty simple: we'll need a second directory for the *nix libraries, called libNativeBinaries, and then everything works, without needing to make any changes to the actual dllmap config file.

@Therzok Before we go and work around this problem, does this seem like something that would be worth going upstream and filing as an actual mono bug? The behavior seems wrong to me, especially based on the docs I used to set up the dllmap stuff initially.

[ethomson]

I don't like the approach of a libNativeBinaries directory.

It's adding a lib prefix because libraries are generally just foo.dll on windows but libfoo.so everywhere else. Naming the directory with a lib prefix seems like kind of a hack.

How did this work before?

[bording]

Naming the directory with a lib prefix seems like kind of a hack.

It's definitely a hack, but it's a hack that's needed because of what mono is doing. Everything works as expected if your current directory matches the directory where the dllmap config file is, because it will pass the relative path in a way that works.

The more I think about it, the more I think this feels like a mono bug. When there is a dllmap config file, it should use those values directly first, which it isn't doing.

[bording]

Oh, and this is all related to this PR:

libgit2/libgit2sharp.nativebinaries#13

The desire was to make the nativebinaries nuget package include the binaries for all platforms all the time, so the dllmap config file was added to make sure everything worked.

[xoofx]

While I agree that It should be fixed in Mono, in the meantime it should also be workaround-fixed in LibGit2Sharp. As of now, LibGit2Sharp does not work out of the box on Linux and prefixing by lib is an easy and quickly releasable fix.

[Therzok]

Yes, best file a bug against the mono runtime.

[bording]

I'm going to look into filing a mono runtime bug on this behavior, which I'm hoping to have some time to do this weekend.

In the meantime, we should figure out the best way to deal with this. @nulltoken Have you had a chance to look over this at all? I know you're still likely getting caught up after being gone.

[nulltoken]

@xoofx Good catch!

In the meantime, we should figure out the best way to deal with this.

@bording I agree with @ethomson that adding a lib prefix to workaround a Mono loader doesn't feel right. However, I can't think of another workaround. Given that there are chances (should this be considered as a bug by the Mono project) that it will only be fixed in the 4.x branch, we might have to jump through that hoop (unless someone can come up with a nicer approach). Would we go down this road, maybe would it be a good time to rename the amd64 folder to x64?

Thoughts?

[bording]

@nulltoken I still haven't had time to look into filling an issue on this, but like you said, even if a change was made, it's pretty much guaranteed to not make it's way back to older versions.

So, assuming we want to keep the linux/osx binaries in the nested folder structure that libgit2/libgit2sharp.nativebinaries#13 introduced (and there are certainly good reasons to do so) renaming the folder to play nice with mono is the simplest way forward.

The change I'm thinking of would make the output folders look like this:

• NativeBinaries
  • amd64
  • x86
• libNativeBinaries
  • linux
    • amd64
  • osx

So you'd end up with two folders. The Windows stuff would stay where it is, so all of the code around being able to change that path in Windows should still work. It would just be the *nix binaries that would get moved, and changing that path from code isn't allowed anyway,

Would we go down this road, maybe would it be a good time to rename the amd64 folder tox64

That almost seems like a separate issue here. I think amd64 makes sense on the linux side of things since that's how the architecture is named in that world. It would be more appropriate to change it for the Windows sub-folder, but that would require changes to the LibGit2Sharp code as well.

Speaking of LibGit2Sharp code changes, a much more radical way of going about solving all of these path issues would be to switch from using DllImport to access libgit2 and call LoadLibrary/dlopen and GetProcAddress/dlsym directly instead. I know that's way too big of a change to actually consider, and would introduce another set of challenges, but it would have some nice benefits! We'd be able to control the exact path to the native binaries for all platforms, and we'd be able to adjust that path at runtime on all platforms too!