hie-bios can't load tests and executables in subdirs

[maksbotan]

Hi,

I've noticed this strange behavior on projects with top-level cabal.project and *.cabal files in subdirs, like https://github.com/haskell-servant/servant.

First of all, versions. I use hie and hie-bios used by it is:

hie-bios version 0.5.0 compiled by GHC 8.8.3

Relevant parts of hie.yaml (generated with implicit-hie, but that should not matter):

cradle:
  cabal:
    - path: "servant-server/src"
      component: "lib:servant-server"

    - path: "servant-server/example/greet.hs"
      component: "servant-server:exe:greet"

    - path: "servant-server/test"
      component: "servant-server:test:spec"

Now, what happens if I try to load some file from the test suite:

$ hie-bios check servant-server/test/Servant/ServerSpec.hs 
hie-bios: can't find file: test/Spec.hs

So, for some reason something drops servant-test/ prefix from the path, although it's present in hie.yaml.

From hie logs:

2020-06-13 00:00:47.056703 [ThreadId 15] - New cradle: /Users/maksbotan/temp/servant/servant-server/test/Servant/ServerSpec.hs
2020-06-13 00:00:47.056912 [ThreadId 15] - Found "/Users/maksbotan/temp/servant/hie.yaml" for "/Users/maksbotan/temp/servant/servant-server/test/Servant/ServerSpec.hs"
2020-06-13 00:00:47.057219 [ThreadId 15] - Module "/Users/maksbotan/temp/servant/servant-server/test/Servant/ServerSpec.hs" is loaded by Cradle: Cradle {cradleRootDir = "/Users/mak
sbotan/temp/servant", cradleOptsProg = CradleAction: Cabal}
2020-06-13 00:00:47.057403 [ThreadId 15] - Found cradle: Cradle {cradleRootDir = "/Users/maksbotan/temp/servant", cradleOptsProg = CradleAction: Cabal}
2020-06-13 00:00:47.057557 [ThreadId 5] - <--2--{"jsonrpc":"2.0","params":{"value":{"kind":"end"},"token":464},"method":"$/progress"}
2020-06-13 00:00:47.05763 [ThreadId 5] - <--2--{"jsonrpc":"2.0","params":{"token":465},"method":"window/workDoneProgress/create","id":465}
2020-06-13 00:00:47.057698 [ThreadId 5] - <--2--{"jsonrpc":"2.0","params":{"value":{"kind":"begin","cancellable":false,"title":"Initializing Cabal project","percentage":0},"token":
465},"method":"$/progress"}
2020-06-13 00:00:47.058043 [ThreadId 4] - ---> {"jsonrpc":"2.0","id":465,"result":null}
2020-06-13 00:00:47.058105 [ThreadId 4] - haskell-lsp:Got reply message:"{\"jsonrpc\":\"2.0\",\"id\":465,\"result\":null}"
2020-06-13 00:00:47.058202 [ThreadId 10] - ****** reactor: got message number:1920
2020-06-13 00:00:47.058247 [ThreadId 10] - reactor:got RspFromClient:ResponseMessage {_jsonrpc = "2.0", _id = IdRspInt 465, _result = Just Null, _error = Nothing}
2020-06-13 00:00:47.058282 [ThreadId 10] - ****** reactor: top of loop
2020-06-13 00:00:48.826147 [ThreadId 13886] - Resolving dependencies...
2020-06-13 00:00:49.502951 [ThreadId 13886] - Build profile: -w ghc-8.8.3 -O0
2020-06-13 00:00:49.503037 [ThreadId 13886] - In order, the following will be built (use -v for more details):
2020-06-13 00:00:49.503088 [ThreadId 13886] -  - servant-0.17 (lib) (configuration changed)
2020-06-13 00:00:49.503132 [ThreadId 13886] -  - servant-server-0.17 (lib) (configuration changed)
2020-06-13 00:00:49.503171 [ThreadId 13886] -  - servant-server-0.17 (test:spec) (configuration changed)
2020-06-13 00:00:49.519729 [ThreadId 13886] - Configuring library for servant-0.17..
2020-06-13 00:00:51.352339 [ThreadId 13886] - Preprocessing library for servant-0.17..
2020-06-13 00:00:51.365653 [ThreadId 13886] - Building library for servant-0.17..
2020-06-13 00:00:52.562677 [ThreadId 13886] - Configuring library for servant-server-0.17..
2020-06-13 00:00:54.507245 [ThreadId 13886] - Preprocessing library for servant-server-0.17..
2020-06-13 00:00:54.517199 [ThreadId 13886] - Building library for servant-server-0.17..
2020-06-13 00:00:55.965902 [ThreadId 13886] - Configuring test suite 'spec' for servant-server-0.17..
2020-06-13 00:00:57.200188 [ThreadId 13884] - Warning: The package has an extraneous version range for a dependency on an
2020-06-13 00:00:57.200304 [ThreadId 13884] - internal library: servant-server -any && ==0.17, servant-server -any &&
2020-06-13 00:00:57.200355 [ThreadId 13884] - ==0.17. This version range includes the current package but isn't needed as
2020-06-13 00:00:57.200393 [ThreadId 13884] - the current package's library will always be used.
2020-06-13 00:00:58.096714 [ThreadId 13886] - Preprocessing test suite 'spec' for servant-server-0.17..
2020-06-13 00:00:58.23108 [ThreadId 13881] - Source error on cradle initialisation: can't find file: test/Spec.hs

hie-bios debug:

$ hie-bios debug servant-server/test/Servant/ServerSpec.hs 
Root directory:      /Users/maksbotan/temp/servant
Component directory: /Users/maksbotan/temp/servant/servant-server
GHC options:         -fbuilding-cabal-package -O0 -outputdir /Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/spec-tmp -odir /Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/spec-tmp -hidir /Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/spec-tmp -stubdir /Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/spec-tmp -i -i/Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/spec-tmp -itest -i/Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/autogen -i/Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/global-autogen -I/Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/autogen -I/Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/global-autogen -I/Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/spec-tmp -optP-include -optP/Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/build/spec/autogen/cabal_macros.h -hide-all-packages -Wmissing-home-modules -no-user-package-db -package-db /Users/maksbotan/.cabal/store/ghc-8.8.3/package.db -package-db /Users/maksbotan/temp/servant/dist-newstyle/packagedb/ghc-8.8.3 -package-db /Users/maksbotan/temp/servant/dist-newstyle/build/x86_64-osx/ghc-8.8.3/servant-server-0.17/t/spec/noopt/package.conf.inplace -package-id QuickCheck-2.13.2-IFlQsVhepoXHrDkLASCySh -package-id aeson-1.4.7.1-CYBfyzHWco0JrS94obsAvi -package-id base-4.13.0.0 -package-id base-compat-0.11.1-4oueEGUxmwHC6tuHqQ6hE2 -package-id base64-bytestring-1.0.0.3-G2dYBQRtOn8Kxdm0ndZRY -package-id bytestring-0.10.10.0 -package-id directory-1.3.6.0 -package-id hspec-2.7.1-aWprQ9Cwpe3oyEKitw2xL -package-id hspec-wai-0.10.1-LnpMCbFbKnxFyLrTCjtYOD -package-id http-types-0.12.3-B9EkuT6UGia2QJglhdFRHH -package-id mtl-2.2.2 -package-id resourcet-1.2.4-6K2EEC3PWvYL6IkanWK5wW -package-id safe-0.3.19-LxDJURZu1921pHCEJqjvRE -package-id servant-0.17-inplace -package-id servant-server-0.17-inplace -package-id should-not-typecheck-2.1.0-Aj1oA7w1Cu9GKPJbdEbgtJ -package-id string-conversions-0.4.0.1-KKjGH1i6WmbIm0l1zjJpaj -package-id temporary-1.3-6rtkIO88VZm9X1XintkbH8 -package-id text-1.2.4.0 -package-id transformers-0.5.6.2 -package-id transformers-compat-0.6.5-4FrvPCFBsgnJ9PLf29P7hZ -package-id wai-3.2.2.1-FkX8kCk55MR1uSuQhWVyx2 -package-id wai-extra-3.0.29.1-F5qpFPbsyBgB3suZR2dFQg -XHaskell2010 test/Spec.hs Servant.ArbitraryMonadServerSpec Servant.Server.ErrorSpec Servant.Server.Internal.ContextSpec Servant.Server.Internal.RoutingApplicationSpec Servant.Server.RouterSpec Servant.Server.StaticFilesSpec Servant.Server.StreamingSpec Servant.Server.UsingContextSpec Servant.Server.UsingContextSpec.TestCombinators Servant.HoistSpec Servant.ServerSpec -Wall -hide-all-packages
System libraries:    /nix/store/qdxkswf4vg9b2ylmg819ff0n8gl8yqip-ghc-shell-for-packages-ghc-8.8.3-env/lib/ghc-8.8.3
Config Location:     /Users/maksbotan/temp/servant/hie.yaml
Cradle:              Cradle {cradleRootDir = "/Users/maksbotan/temp/servant", cradleOptsProg = CradleAction: Cabal}
Dependencies:        cabal.project cabal.project.local

This is annoying, since once I open some of test's files in my editor, hie just starts to consume CPU by restarting in a loop and failing with the same error.

[fendor]

This option: -itest is wrong. It should be an absolute path.

[fendor]

I did some more investigation.
Firstly, my first hunch that the option -itest is wrong was incorrect. It is correct, the path is made absolute later in the pipeline which cant be seen here (namely in initSession).
Secondly, $ hie-bios check servant-server/test/Servant/ServerSpec.hs fails due to the same reason cabal repl servant-server/test/Spec.hs fails (a bug in cabal 3.0.0.0 haskell/cabal#6622 that has been fixed in upcoming versions).
Neither of these issues caused the compilation error you observed afaict. Also, I could only reproduce half of the problem. An error was shown when you opened test/Spec.hs but the rest of the test-suite was working fine. The problem of test/Spec.hs seems to be related to hspec-discover (related issue #187).
I was able to load the project with no errors after globally installing hspec-discover.

[maksbotan]

You can clearly see this in the command line:

... -XHaskell2010 test/Spec.hs Servant.ArbitraryMonadServerSpec Servant.Server.ErrorSpec ...

I.e., cabal for some reason adds test/Spec.hs as a file, not module name.

BTW, hspec-discover is in $PATH for me (due to nix), so that issue is unrelated.

Have you seen my PR #202? I can confirm that with this patch (rebased on top of 0.5.0) hie works fine with servant.

[maksbotan]

And my hie-bios and hie are built with Cabal-3.2.0.0, as far as I can tell. And that version is in $PATH as well.

[maksbotan]

Ah, haskell/cabal#6622 has not yet made it to any release, maybe it's the reason I'm getting this error.

So, in other words, we have to wait for Cabal release?

[fendor]

You can clearly see this in the command line:

Yes, but the flags are post-processed later on. Unfortunately, they do represent the exact flags that are used by the ghc session anymore.

are built with Cabal-3.2.0.0

I think the fix is going to be shipped with the next minor version cabal-3.2.0.1 release.

So, in other words, we have to wait for Cabal release?

No, it should work. Actually I think I got it wrong again. It should be unrelated to the cabal bug.

I.e., cabal for some reason adds test/Spec.hs as a file, not module name.

That is normal for the main module.

[maksbotan]

By the way, this succeeds:

$ cabal new-repl servant-server/test/Servant/ServerSpec.hs 

That is normal for the main module.

Indeed, but it's broken for packages in subdirs :) That's why I added preprocessing for this specific case in #202.

[fendor]

By the way, this succeeds:

Yeah, the error is thrown by the GHC API.

The fix looks good in general, I am playing around with different ways.

[maksbotan]

Okay!

I'm not saying I think my fix is great, I just want to attract your attention to the root of the problem :)

[fendor]

I'm not saying I think my fix is great, I just want to attract your attention to the root of the problem :)

I think I have understood it now :) Adding a test and pushing my version of the fix for a review soon.

[maksbotan]

Nice to hear!