cabal-3.2.0.0 fails to build/install several packages - assert fails

[mouse07410]

Describe the bug
Currently-released cabal-3.2.0.0 aborts at build/install of several packages, e.g., pandoc, upon failing assertion.

To Reproduce

$ cabal install pandoc
Warning: Parsing the index cache failed (Data.Binary.Get.runGet at position
16: Non-matching structured hashes: 0210baca00000000ec34895e00000000;
expected: a257ca064dfb5e0cb74f74e64a975b9e). Trying to regenerate the index
cache...
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - HTTP-4000.3.14 (lib) (requires build)
 - http-client-0.6.4.1 (lib) (requires build)
 - socks-0.6.1 (lib) (requires build)
 - tls-1.5.4 (lib) (requires build)
 - connection-0.3.1 (lib:connection) (requires build)
 - http-client-tls-0.3.5.3 (lib) (requires build)
 - pandoc-2.9.2.1 (lib) (requires build)
Assertion failed
CallStack (from HasCallStack):
  assert, called at ./Distribution/Client/ProjectPlanning.hs:242:5 in main:Distribution.Client.ProjectPlanning

Expected behavior
Successful build/installation, as with cabal-3.0.0.0.

System information

• MacOS Catalina 10.15.4, Xcode-11.4.
• cabal-3.2.0.0, ghc-8.8.3

Additional context
Cabal configured to build dynamic libraries and executables, profiling is disabled, documentation enabled.

Cabal installed via $ cabal install cabal-install --overwrite-policy=always

On a probably-unrelated note, it looks like Cabal doesn't take command-line arguments --enable-static and --enable-executable-static: when I specify those, it still seems to follow what's in ~/.cabal/config, which says "dynamic".

[hvr]

@mouse07410 You installed cabal-install from the Git repo, didn't you? I.e. you were inside the Git repo when you invoked cabal install cabal-install --overwrite-policy=always?

Does this also occur with the executable from https://downloads.haskell.org/~cabal/cabal-install-3.2.0.0/cabal-install-3.2.0.0-x86_64-apple-darwin17.7.0.tar.xz ?

[mouse07410]

You installed cabal-install from the Git repo, didn't you?

Initially - yes, but when I found that it was officially released, I installed it from the Hackage by doing $ cd ; cabal install cabal-install --overwrite-policy-=always. So, no - I was not inside the Git repo during my last couple of re-installations. And without that --overwrite... it fails to create a symlink, so the failure to pass that flag couldn't pass unnoticed.

Does this also occur with the executable from https://downloads.haskell.org/~cabal/cabal-install-3.2.0.0/cabal-install-3.2.0.0-x86_64-apple-darwin17.7.0.tar.xz ?

Hmm, I've no idea - never downloaded individual binaries manually before. Always used either ghcup tool, or "upgrade in place" (aka - rebuilding locally)... It appears that downloaded cabal-install binaries work fine, and do not exhibit this problem. Only the locally-rebuilt ones do.

[phadej]

I cannot reproduce this on Linux.
My suggestion would be to wipe ~/.cabal/store/ghc-8.8.3/ and try again.

I'm quite sure that some of these assertions trigger when store gets inconsistent (for whatever reasons).

[mouse07410]

I'm quite sure that some of these assertions trigger when store gets inconsistent (for whatever reasons).

Could switching between Cabal-3.3.0.0 build from Git repo, Cabal-3.2.0.0 built from Git repo, Cabal-3.0.0.0 distributed via ghcup, and Cabal-3.2.0.0 built from Hackage contribute/cause this?

If so, does it mean that once I "stabilize" on built-from-Hackage Cabal-3.2.0.0, it should go away? Or you still recommend blowing away ~/.cabal/store/ghc-8.8.3?

[phadej]

Could switching between Cabal-3.3.0.0 build from Git repo, Cabal-3.2.0.0 built from Git repo, Cabal-3.0.0.0 distributed via ghcup, and Cabal-3.2.0.0 built from Hackage contribute/cause this?

No, that shouldn't happen. More likely cancelling a build (or failing build) left things in inconsistent state.

[mouse07410]

More likely cancelling a build (or failing build) left things in inconsistent state...

Darn... Lord knows I've had more than a few of those.

Do you still advise wiping out ~/.cabal/store/ghc-8.8.3? Or wait and see if I get that problem again, and blow it then?

[phadej]

If it doesn't happen consistently, and you cannot reproduce it right away, there's nothing to do.

[hvr]

Always used either ghcup tool, or "upgrade in place" (aka - rebuilding locally)...

@mouse07410 fwiw, that's the very same binary ghcup would download too

[mouse07410]

I'm getting that error consistently now with cabal-install that I build from the source.

I wiped ~/.cabal/store/ghc-8.8.3. I built cabal-3.3.0.0 from the current Cabal master - succeeded.

Then I tried to build HIE. Failed as shown above.

Then I tried to install floskell (cd ~/src; cabal install floskell). Failed:

$ cabal install floskell --overwrite-policy=always
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - StateVar-1.2 (lib) (requires build)
 - base-compat-0.11.1 (lib) (requires build)
 - base-orphans-0.8.2 (lib) (requires build)
 - cabal-doctest-1.0.8 (lib) (requires build)
 - colour-2.3.5 (lib) (requires build)
 - uuid-types-1.0.3 (lib) (requires build)
 - dlist-0.8.0.8 (lib) (requires build)
 - data-default-class-0.1.2.0 (lib:data-default-class) (requires build)
Assertion failed
CallStack (from HasCallStack):
  assert, called at ./Distribution/Client/ProjectPlanning.hs:243:5 in main:Distribution.Client.ProjectPlanning

I don't know where to find more detailed logs - ~/.cabal/logs/ghc-8.8.3 did not help.

Then I tried Cabal-3.2.0.0 installed by ghcup, and it worked flawlessly.

The only difference between our environments that I'm aware of - on my machine both Cabal and Stack are configured to build shared libraries and dynamic executables.

I wouldn't venture into building my "own" Cabal, but I need the fix provided by #6623.

[mouse07410]

And somehow I seem unable to force static build?

$ time cabal install exe:cabal --enable-static --disable-shared
Wrote tarball sdist to
/Users/uri/src/cabal/dist-newstyle/sdist/Cabal-3.3.0.0.tar.gz
Wrote tarball sdist to
/Users/uri/src/cabal/dist-newstyle/sdist/cabal-testsuite-3.tar.gz
Wrote tarball sdist to
/Users/uri/src/cabal/dist-newstyle/sdist/cabal-install-3.3.0.0.tar.gz
Wrote tarball sdist to
/Users/uri/src/cabal/dist-newstyle/sdist/solver-benchmarks-3.tar.gz
Wrote tarball sdist to
/Users/uri/src/cabal/dist-newstyle/sdist/Cabal-quickcheck-3.3.0.0.tar.gz
Wrote tarball sdist to
/Users/uri/src/cabal/dist-newstyle/sdist/cabal-benchmarks-3.tar.gz
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - cabal-install-3.3.0.0 (exe:cabal) (requires build)
Starting     cabal-install-3.3.0.0 (exe:cabal)
Building     cabal-install-3.3.0.0 (exe:cabal)
Installing   cabal-install-3.3.0.0 (exe:cabal)
Completed    cabal-install-3.3.0.0 (exe:cabal)
Symlinking 'cabal'

real	3m41.965s
user	3m21.837s
sys	0m15.473s

$ otool -L ~/.cabal/bin/cabal
/Users/uri/.cabal/bin/cabal:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
	@rpath/libHSsync-2.2.2-ca63dbfc-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libHSstm-2.5.0.0-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libHSrslv-0.1.2.0-daf83fbe-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libHShshbl-1.3.0.0-27da760f-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libHShckg-scrty-0.6.0.1-36decdf1-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libHSzlb-0.6.2.1-c632b378-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libHStr-0.5.1.1-86143c27-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	. . . . .
@rpath/libHSghc-prim-0.5.3-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
	@rpath/libHSrts_thr-ghc8.8.3.dylib (compatibility version 0.0.0, current version 0.0.0)
$ otool -L ~/.cabal/bin/cabal | wc -l
  35
$

[mouse07410]

Wiping out ~/.cabal/store/ghc-8.8.3 does not help. Done it multiple times now, with zero results:

• cabal-install built from source keeps failing on that assertion, pretty much regardless of what package I ask it to build/install;
• cabal-install downloaded via ghcup works fine.

[phadej]

I tried to reproduce this in docker container

% docker run --rm -ti phadej/ghc:8.8.3-bionic bash

root@abcdef:/# cd

We have cabal-install-3.2 there:

root@abcdef:~# which cabal
/opt/cabal/3.2/bin/cabal

root@abcdef:~# cabal --version
cabal-install version 3.2.0.0
compiled using version 3.2.0.0 of the Cabal library 

Let's update index and build pandoc

root@abcdef:~# cabal update
Config file path source is default config file.
Config file /root/.cabal/config not found.
Writing default configuration to /root/.cabal/config
Downloading the latest package list from hackage.haskell.org

root@abcdef:~# cabal install pandoc
...
Symlinking 'pandoc'

root@abcdef:~# pandoc -v
pandoc 2.9.2.1
Compiled with pandoc-types 1.20, texmath 0.12.0.1, skylighting 0.8.3.4
Default user data directory: /root/.local/share/pandoc or /root/.pandoc
Copyright (C) 2006-2020 John MacFarlane
Web:  https://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

It works. As in your case.

Let's remove it and build cabal from master

root@abcdef:~# rm $(which pandoc)

root@abcdef:~# rm -rf ~/.cabal/store/

root@abcdef:~# git clone https://github.com/haskell/cabal.git
Cloning into 'cabal'...
...

root@abcdef:~# cd cabal/

I use current master, specific commit to be precise:

root@abcdef:~/cabal# git checkout 79649a5f3f8fd616f169b9a8606f597865f99aab
Note: checking out '79649a5f3f8fd616f169b9a8606f597865f99aab'.

...

HEAD is now at 79649a5f3 Merge pull request #6684 from phadej/issue-6610

Then installing cabal

root@abcdef:~/cabal# cabal install cabal
...
Starting     cabal-install-3.3.0.0 (exe:cabal)
Building     cabal-install-3.3.0.0 (exe:cabal)
Installing   cabal-install-3.3.0.0 (exe:cabal)
Completed    cabal-install-3.3.0.0 (exe:cabal)
Symlinking 'cabal'

We need to remember to rehash to use just built version:

root@abcdef:~/cabal# hash -r

root@abcdef:~/cabal# which cabal
/root/.cabal/bin/cabal

root@abcdef:~/cabal# cabal --version
cabal-install version 3.3.0.0
compiled using version 3.3.0.0 of the Cabal library 

Let's rebuild pandoc again:

root@abcdef:~/cabal# cd 

The index cache is format is different, as is expected due recent changes,
and also output of Symlinking... is different, it's changed recently.

root@abcdef:~# cabal install pandoc
Warning: Parsing the index cache failed (Data.Binary.Get.runGet at position
16: Non-matching structured hashes: a257ca064dfb5e0cb74f74e64a975b9e;
expected: 3cde863fbd21d2968a50ad918c73d775). Trying to regenerate the index
cache...
...

Starting     pandoc-2.9.2.1 (exe:pandoc)
Building     pandoc-2.9.2.1 (exe:pandoc)
Installing   pandoc-2.9.2.1 (exe:pandoc)
Completed    pandoc-2.9.2.1 (exe:pandoc)
Symlinking 'pandoc' to '/root/.cabal/bin/pandoc'

But it all works. There are some missing bits. It would be very helpful,
if you could make a reproducer using e.g. the docker image here.

Otherwise we'd simply need ot wait until we replace asserts in ProjectPlanning with type guarantees, which could take a while :(