cabal test sometimes builds with the wrong flags. How to control its flags?

[rrnewton]

We typically run Jenkins testing jobs with large matrices of configuration options.

We have been assuming that the following series of commands would guarantee that A and B only get built once, and only with -fmyflag

cabal install ./A ./B --enable-tests -fmyflag
(cd A; cabal test)
(cd B; cabal test)

(BTW, I don't know if there is any such "algebra" of command sequences that carry guarantees! But it would help to have "cabal laws", because I currently have no idea what the "semantics" of cabal test is.)

Unfortunately, it seems that cabal test is sometimes causing a recompile of the packages. When it does that the package seems like it gets built without the right flags.

• Why not pass -fmyflag to cabal test? Well, it doesn't support flag arguments!
• Why not do everything from one cabal install line with --run-tests. Two reasons:
  • Sometimes we need extra logic in the script to determine if a particular test suite can run
  • We want to get the testing output in the logs. We really like
    --show-details=always, but as far as I can tell we cannot get that through cabal install.

So install doesn't take arguments meant for test and test doesn't take arguments meant for cabal. But these different modes call eachother! Fixing our problem could be accomplished either by fixing these argument limitations or guaranteeing that cabal test won't reinstall.

This is tangentially related to tickets: #1647, #2058, #2075, #2176

[23Skidoo]

What if instead of cd A; cabal test you do cd A; cabal configure -fmyflag --enable-tests; cabal test?

[rrnewton]

That seems to work! Well, rather, it fails properly ;-) -- the failures occur exactly where they should in the matrix.

I hadn't tried this before because I thought rerunning cabal configure would necissitate recompiling, but it looks like it doesn't, so that is a good solution for us.

[23Skidoo]

No, configure never recompiles anything except maybe Setup.hs.

[rrnewton]

Any idea why the extra configure does the trick when the packages were already configured by the install step?

[23Skidoo]

Can I take a look at your build bot script?

[rrnewton]

Sure, here is the line of code where we added the extra cabal configure.

And the links earlier in this thread go to the Jenkins job for running that script. (It also should be possible to run ./.jenkins_script.sh locally, as a validate script, with a fresh checkout of that lvars repo.)

[23Skidoo]

Looks like this is caused by my fix to #1281. When you run cabal install ./A, it uses a non-standard dist directory (dist/dist-$hash), but cabal test uses dist, thinks that the package wasn't configured and proceeds to build it for the second time. So in this scenario cabal test should always result in rebuilding, with or without an extra configure. Maybe it does after all?

A proper fix would be to stop using dist/dist-$hash unconditionally in cabal install and only do that when installing add-source deps.

[rrnewton]

Well, it looks like it does "build" again, but that build is a noop (i.e. it uses the existing .o/.hi files...):

Configuring lvish-2.0...
+ cabal-1.20 test --show-details=streaming '--test-options=-j1 --jxml=test-results.xml --jxml-nested'
Building lvish-2.0...
Preprocessing library lvish-2.0...

Control/LVish/SchedIdempotentInternal.hs:27:0:
     warning: #warning "Compiling with Chase-Lev work-stealing deque"
In-place registering lvish-2.0...
Preprocessing test suite 'test-lvish' for lvish-2.0...
Linking dist/build/test-lvish/test-lvish ...
Running 1 test suites...

[23Skidoo]

Which is strange, since if my analysis is correct this shouldn't happen. When I try to reproduce this on my machine, cabal test performs a full rebuild.

[rrnewton]

Oh wait a sec... it may well have done a full rebuild on mine too. If that version of cabal is doing a parallel build by default it wouldn't show the full output right? But it clearly ran the preprocessor because of that warning, so I assume it had to do a full compile then. So we are consistent then.

Anyway, I don't mind if I have to build twice to test... as long as it works ;-).

[23Skidoo]

Yes, I considered that, but if it was doing a parallel build, it wouldn't have printed "Preprocessing"/"Linking" messages.