Dependency on installed public sublibrary without --dependency flag #7270

fgaz · 2021-02-03T15:52:52Z

Depending on a globally installed public sublibrary is problematic unless --dependency=pkgname:sublibname=installed-id is specified.

The bug (?) does not get triggered when that flag is used, for example:

when manually running Setup configure --dependency
when running cabal-install on source packages

Here are two occasions in which it can manifest (b depends on a:foo):

In nix, which I think does not use --dependency and instead relies on Cabal's basic solver, I get this error:

build log

building '/nix/store/djsi6bxhgwgqlvymi3rf5078567jry3w-b-1.drv'...
setupCompilerEnvironmentPhase
Build with /nix/store/2lkbmrf9iw3v4dnf0yixhldvff5397xl-ghc-8.10.3.
unpacking sources
unpacking source archive /nix/store/0h1n08ri9fph049mx0rxxx8qnl4pz408-b
source root is b
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/build/setup-package.conf.d -j4 +RTS -A64M -RTS -threaded -rtsopts
[1 of 1] Compiling Main             ( /nix/store/4mdp8nhyfddh7bllbi7xszz7k9955n79-Setup.hs, /build/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/pl31hr7cnig3nfqak49nkgiivnphddm9-b-1 --libdir=$prefix/lib/$compiler --libsubdir=$abi/$libname --with-gcc=gcc --package-db=/build/package.conf.d --ghc-options=-j4 +RTS -A64M -RTS --disable-split-objs --enable-library-profiling --profiling-detail=exported-functions --disable-profiling --enable-shared --disable-coverage --enable-static --disable-executable-dynamic --enable-tests --disable-benchmarks --enable-library-vanilla --disable-library-for-ghci --ghc-option=-split-sections --extra-lib-dirs=/nix/store/8v0snmd618lmk6w3xzapf4ygac1n8adx-ncurses-6.2/lib --extra-lib-dirs=/nix/store/8b4r4lmhybjxhxx33z0si8gifhi8hfmn-libffi-3.3/lib --extra-lib-dirs=/nix/store/smhc5diparm6im6dp3iqbz460hwrlv8s-gmp-6.2.1/lib
Using Parsec parser
Configuring b-1...
Warning: Packages using 'cabal-version: >= 1.10' must specify the
'default-language' field for each component (e.g. Haskell98 or Haskell2010).
If a component uses different languages in different modules then list the
other ones in the 'other-languages' field.
Dependency a:{foo} -any: using a-1
Dependency base -any: using base-4.14.1.0
Source component graph: component lib:bar
CallStack (from HasCallStack):
  dieNoWrap, called at libraries/Cabal/Cabal/Distribution/Utils/LogProgress.hs:61:9 in Cabal-3.2.1.0:Distribution.Utils.LogProgress
Error: Dependency on unbuildable library 'foo' from a

builder for '/nix/store/djsi6bxhgwgqlvymi3rf5078567jry3w-b-1.drv' failed with exit code 1
error: build of '/nix/store/djsi6bxhgwgqlvymi3rf5078567jry3w-b-1.drv' failed

in cabal-install: when depending on a globally installed package (for which the .cabal is not available)

build log

> cabal configure -v3
File monitor 'config' changed: first run
Project settings changed, reconfiguring...
creating /home/build/p/b/dist-newstyle
creating /home/build/p/b/dist-newstyle/cache
this build was affected by the following (project) config files:
File monitor 'improved-plan' changed: first run
File monitor 'elaborated-plan' changed: first run
File monitor 'compiler' changed: first run
Compiler settings changed, reconfiguring...
Searching for ghc in path.
Found ghc at /opt/ghc/bin/ghc
/opt/ghc/bin/ghc --numeric-version
/opt/ghc/bin/ghc is version 8.10.1
looking for tool ghc-pkg near compiler in /opt/ghc/bin
candidate locations:
["/opt/ghc/bin/ghc-pkg","/opt/ghc/8.10.1/bin/ghc-pkg-ghc-8.10.1","/opt/ghc/8.10.1/bin/ghc-pkg-8.10.1","/opt/ghc/8.10.1/bin/ghc-pkg"]
found ghc-pkg in /opt/ghc/bin/ghc-pkg
/opt/ghc/bin/ghc-pkg --version
/opt/ghc/bin/ghc-pkg is version 8.10.1
/opt/ghc/bin/ghc --supported-languages
/opt/ghc/bin/ghc --info
File monitor 'solver-plan' changed: first run
/opt/ghc/bin/ghc --print-global-package-db
Reading installed packages...
/opt/ghc/bin/ghc-pkg dump --global -v0
/opt/ghc/bin/ghc --print-libdir
Reading available packages of hackage.haskell.org...
Using most recent state (could not read timestamp file)
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.RemoteRepo {remoteRepoName = "hackage.haskell.org",
remoteRepoURI = http://hackage.haskell.org/, remoteRepoSecure = Just True,
remoteRepoRootKeys =
["fe331502606802feac15e514d9b9ea83fee8b6ffef71335479a2e68d84adc6b0","1ea9ba32c526d1cc91ab5e5bd364ec5e9e8cb67179a471872f6e26f0ae773d42","2c6c3627bd6c982990239487f1abd02e08a02e6cf16edb105a8012d444d870c3","0a5c7ea47cd1b15f01f5f51a33adda7e655bc0f0b0615baa8e271f4c3351e21d","51f0161b906011b52c6613376b1ae937670da69322113a246a09f807c62f6921"],
remoteRepoKeyThreshold = 3, remoteRepoShouldTryHttps = True}
index-state(hackage.haskell.org) =
Searching for pkg-config in path.
Found pkg-config at /usr/bin/pkg-config
/usr/bin/pkg-config --version
/usr/bin/pkg-config is version 0.29
/usr/bin/pkg-config --variable pc_path pkg-config
Searching for pkg-config in path.
Found pkg-config at /usr/bin/pkg-config
/usr/bin/pkg-config --version
/usr/bin/pkg-config is version 0.29
/usr/bin/pkg-config --list-all
/usr/bin/pkg-config --modversion systemd formw menuw menu 'ncurses++' panelw panel ncurses tinfo tic ncursesw form 'ncurses++w' udev
Resolving dependencies...
targets: b
constraints: 
  any.base installed (non-upgradeable package)
  any.ghc-prim installed (non-upgradeable package)
  any.integer-gmp installed (non-upgradeable package)
  any.template-haskell installed (non-upgradeable package)
  setup.Cabal >=3.2 (minimum version of Cabal used by Setup.hs)
  setup.Cabal <3.4 (maximum version of Cabal used by Setup.hs)
  b.b ==1 (user target)
preferences: 
  b [TestStanzas,BenchStanzas]
strategy: PreferLatestForSelected
reorder goals: False
count conflicts: True
fine grained conflicts: True
minimize conflict set: False
independent goals: False
avoid reinstalls: False
shadow packages: False
strong flags: False
allow boot library installs: False
only constrained packages: OnlyConstrainedNone
max backjumps: 4000
[__0] trying: b-1 (user goal)
[__1] trying: base-4.14.0.0/installed-4.14.0.0 (dependency of b)
[__2] trying: rts-1.0/installedrts (dependency of base)
[__3] trying: integer-gmp-1.0.3.0/installed-1.0.3.0 (dependency of base)
[__4] trying: ghc-prim-0.6.1/installed-0.6.1 (dependency of base)
[__5] next goal: a (dependency of b)
[__5] trying: a-1/installed-7i1EAAn7OKY8zea1XCf99I
[__6] done
Elaborating the install plan...
File monitor 'source-hashes' changed: first run
Component graph for b-1: component lib:bar
CallStack (from HasCallStack):
  dieNoWrap, called at ./Distribution/Utils/LogProgress.hs:61:9 in Cabal-3.2.0.0-DWUJBcdbHvD7uNp9v26EzF:Distribution.Utils.LogProgress
Error:
    Dependency on unbuildable library 'foo' from a
    In the stanza 'library bar'
    In the inplace package 'b-1'

It's the same error!
Unfortunately due to the failure it isn't possible to extract the plan.json, but by looking at the output it would seem that either cabal-install or Cabal decided to depend on the main a library (7i1EAAn7OKY8zea1XCf99I is the id of the main library)

In fact, the same error happens when manually invoking Setup.hs configure without the --dependency flag or with an invalid one (either by changing a:foo to a, or by changing the ipid):

Error: Dependency on unbuildable library 'foo' from a

The above was on cabal 3.2. On master:

Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: b-1 (user goal)
[__1] next goal: a (dependency of b)
[__1] rejecting: a-1/installed-7i1EAAn7OKY8zea1XCf99I (does not contain
library 'foo', which is required by b)
[__1] fail (backjumping, conflict set: a, b)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: b, a

So even the solver tries to use the main library

Questions:

why isn't cabal-install passing --dependency flags for installed packages (or passing wrong ones)?
why isn't the basic Cabal solver solving the right library? Is it a bug that it doesn't or should downstream tools use --dependency?

The text was updated successfully, but these errors were encountered:

phadej · 2021-02-03T16:17:30Z

It's sad that reproducing this needs messing with global-db.

Whatever this is, it does look like a bug.

Compare to running ./Setup configure on package with unsatisfiable constraints (i.e. no dependencies in package-db provided).

Configuring base64-bytestring-type-1.0.1...
Setup.hs: Encountered missing dependencies:
QuickCheck >=2.11.3 && <2.15,
aeson >=1.2.3.0 && <1.6,
base-compat >=0.9.3 && <0.12,
base64-bytestring >=1.0.0.1 && <1.3,
hashable >=1.2.6.1 && <1.4

So Distributions.Simple.Configure seems to proceed somewhere where invariants&assumptions do not hold.

fgaz · 2021-02-03T16:27:24Z

By the way that was on cabal 3.2, I updated the OP with the behavior of @grayjay's solver patch

fgaz · 2021-02-03T16:30:50Z

It's sad that reproducing this needs messing with global-db.

Yeah. That --package-db flag should really be implemented (#5773), so that it's easier to provide prebuilt packages to cabal from many sources

fgaz · 2021-09-04T15:27:22Z

@gbaz I think this is only a bug in Cabal's solver, not cabal-install's, since it can be reproduced with the Setup.hs interface too

edit: nevermind, i even wrote it in the op:

So even the [cabal-install] solver tries to use the main library

grayjay · 2022-02-21T07:51:55Z

I found that one reason for the issue in cabal-install is that multilibs aren't fully supported by the solver. I think that that part of the issue is covered by #6039, so I commented there and removed the solver label from this issue.

Fixes haskell#7270

fgaz · 2022-04-08T09:08:43Z

After yesterday's discussion with @grayjay, I investigated this further and found that the information that Cabal needs is all there: the package-name and lib-name ipi fields should be enough for Cabal to select the correct installed package, since --as far as I know-- Cabal cannot deal with multiple package instances for a single (package name, version, library name) tuple unless --dependency is used.

So I dug into the code and found a subtle misuse of lookupDependency. lookupDependency was used to pick any instance of (package name, latest version) from the package db in case --dependency is missing, but the library name was not considered at all! #8089 swaps lookupDependency with lookupInternalDependency and should finally close this ticket.

cabal-install though does care about package instances, and it will always use --dependency when calling Cabal, skipping the problematic code path entirely, so #6039 is a different issue.

Fixes haskell#7270

Same for "visibility is experimental feature" With haskell#7270 fixed, it's time to allow hackage uploads Closes haskell#6801

Fixes haskell#7270

Same for "visibility is experimental feature" With haskell#7270 fixed, it's time to allow hackage uploads. PWTExperimental is not used anymore, but for now I'm keeping it in case we add other experimental features in the future, so we don't keep removing and adding it (pushing CPP on users). Since it isn't used though there is no way to run cabal-testsuite/PackageTests/DuplicateExperimental/cabal.project so I'm just removing that test. Closes haskell#6801

Same for "visibility is experimental feature" With haskell#7270 fixed, it's time to allow hackage uploads. PWTExperimental is not used anymore, but for now I'm keeping it in case we add other experimental features in the future, so we don't keep removing and adding it (pushing CPP on users). Since it isn't used though there is no way to run cabal-testsuite/PackageTests/DuplicateExperimental so I'm just removing that test. Closes haskell#6801

When Cabal is passed --exact-configuration, it knows from --dependency flags the ids of all dependencies, including sublibraries. When this flag does not exists though, Cabal tries to locate them by looking them up in the package databases. lookupDependency was used for this, but it only looks up main (unnamed) libraries (LMainLibName), so Cabal was failing to locate installed public sublibraries. Using lookupInternalDependency and passing the library name fixes this. Fixes haskell#7270

Same for "visibility is experimental feature" With haskell#7270 fixed, it's time to allow hackage uploads. PWTExperimental is not used anymore, but for now I'm keeping it in case we add other experimental features in the future, so we don't keep removing and adding it (pushing CPP on users). Since it isn't used though there is no way to run cabal-testsuite/PackageTests/DuplicateExperimental so I'm just removing that test. Closes haskell#6801

fgaz mentioned this issue Feb 3, 2021

Post-multilibs checklist #5660

Open

31 tasks

fgaz added Cabal: cmd/configure Cabal: other type: bug labels Feb 3, 2021

fgaz mentioned this issue Jul 1, 2021

Suppress warnings for verbosity #7470

Merged

3 tasks

fgaz added priority: high 🔥 attention: pr-welcome labels Aug 19, 2021

fgaz mentioned this issue Aug 22, 2021

Cannot reference internal libraries with cabal-version: 3.4 #7565

Open

gbaz added the cabal-install: solver label Aug 22, 2021

grayjay mentioned this issue Sep 4, 2021

Make the cabal-install solver multilibs-aware #6039

Open

fgaz removed the cabal-install: solver label Sep 13, 2021

fgaz mentioned this issue Sep 13, 2021

Remove "colon specifier is experimental feature (issue #5660)" for 3.8 #6801

Closed

Mikolaj added the 3.8 label Sep 14, 2021

fgaz mentioned this issue Sep 22, 2021

Add package-db flag for v2 commands #7676

Merged

7 tasks

sevanspowell mentioned this issue Oct 14, 2021

Provide Cardano.Api.Tx generators cardano-foundation/cardano-wallet#2975

Closed

1 task

kczulko mentioned this issue Nov 24, 2021

Add support for 'experimental' colon syntax (public sublibraries) tweag/gazelle_cabal#6

Merged

fgaz added the cabal-install: solver label Dec 18, 2021

grayjay removed the cabal-install: solver label Feb 21, 2022

Kleidukos removed the 3.8 label Mar 12, 2022

fgaz added a commit to fgaz/cabal that referenced this issue Apr 8, 2022

Look up sublibraries also with nonexact deps

d33b3f6

Fixes haskell#7270

fgaz mentioned this issue Apr 8, 2022

Look up sublibraries also with nonexact deps #8089

Merged

8 tasks

fgaz added a commit to fgaz/cabal that referenced this issue Apr 14, 2022

Look up sublibraries also with nonexact deps

3238e87

Fixes haskell#7270

fgaz added a commit to fgaz/cabal that referenced this issue Apr 14, 2022

Add test for haskell#7270

769dbc1

fgaz added a commit to fgaz/cabal that referenced this issue Apr 15, 2022

Add test for haskell#7270

f39ef8b

fgaz added a commit to fgaz/cabal that referenced this issue Apr 25, 2022

Look up sublibraries also with nonexact deps

46cd0c1

Fixes haskell#7270

fgaz added a commit to fgaz/cabal that referenced this issue Apr 25, 2022

Add test for haskell#7270

e905400

fgaz added a commit to fgaz/cabal that referenced this issue May 2, 2022

Look up sublibraries also with nonexact deps

ca29bc6

Fixes haskell#7270

fgaz added a commit to fgaz/cabal that referenced this issue May 2, 2022

Add test for haskell#7270

6adea7a

fgaz added a commit to fgaz/cabal that referenced this issue May 2, 2022

Remove "colon specifier is experimental feature"

0cdca20

Same for "visibility is experimental feature" With haskell#7270 fixed, it's time to allow hackage uploads Closes haskell#6801

fgaz added a commit to fgaz/cabal that referenced this issue May 2, 2022

fixup! Add test for haskell#7270

c5606a5

fgaz added a commit to fgaz/cabal that referenced this issue May 4, 2022

Look up sublibraries also with nonexact deps

4eb3489

Fixes haskell#7270

fgaz added a commit to fgaz/cabal that referenced this issue May 4, 2022

Add test for haskell#7270

ec55d3a

fgaz added a commit to fgaz/cabal that referenced this issue May 5, 2022

Add test for haskell#7270

6abd70a

andreabedini closed this as completed in #8089 May 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Dependency on installed public sublibrary without --dependency flag #7270

Dependency on installed public sublibrary without --dependency flag #7270

fgaz commented Feb 3, 2021 •

edited

Loading

phadej commented Feb 3, 2021 •

edited

Loading

Uh oh!

fgaz commented Feb 3, 2021

Uh oh!

fgaz commented Feb 3, 2021 •

edited

Loading

Uh oh!

fgaz commented Sep 4, 2021 •

edited

Loading

Uh oh!

grayjay commented Feb 21, 2022

Uh oh!

fgaz commented Apr 8, 2022

Uh oh!

Dependency on installed public sublibrary without --dependency flag #7270

Dependency on installed public sublibrary without --dependency flag #7270

Comments

fgaz commented Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

phadej commented Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fgaz commented Feb 3, 2021

Uh oh!

fgaz commented Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fgaz commented Sep 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grayjay commented Feb 21, 2022

Uh oh!

fgaz commented Apr 8, 2022

Uh oh!

fgaz commented Feb 3, 2021 •

edited

Loading

phadej commented Feb 3, 2021 •

edited

Loading

fgaz commented Feb 3, 2021 •

edited

Loading

fgaz commented Sep 4, 2021 •

edited

Loading