[RFC] Build Drivers for cabal

[Kleidukos]

Summary

The proposal is about the creation of a new stanza in the cabal file format: build-drivers. This stanza takes a list of executables that run before the build of the Cabal components, in order to provide any required generated Haskell code, foreign C libraries, etc.
The roles of building foreign packages are distributed as such:

• Dependencies: Left to the build system with which the build driver interacts
• Calling other build systems: This is what the build driver does
• Linking: This part is supported by Cabal & GHC when it comes to C-style interfaces, and provides a wide enough coverage of FFI candidates.

Background

Historically, this function has been occupied by Setup.hs and the Custom build-type, using user hooks like "pre-build". As we are now in a development phase of Cabal where we aim to get rid of Setup.hs, I would like to offer a clear and straightforward migration path for our users on this usage of the user hooks.

Rationale

• This helps us get rid of Setup.hs in a clear and predictable way
• We can remove some less-used logic from cabal-install when it comes to custom preprocessor, and don't have adopt new ones should they emerge (WASM, JS, etc)

Example

Here is an example of a Cabal file that uses the future cargo-driver, to build Cargo packages with the Rust compiler:

library
    hs-source-dirs:     src
    default-language:   Haskell2010
    ghc-options:        -Wall
    build-depends:
        base >=4.10.0.0 && <5,
        bytestring >=0.11.0.0
    data-files:
      native/rust-library
    extra-bundled-libraries:    rustLibrary -- the rust library's C interface that we are building with the cargo driver
    build-drivers:
      cargo-driver -- cargo-driver then parses a TOML file that holds its necessary configuration

# Configuration of the cargo build driver
# Every one of those options is over-ridable by environment variables
# But this is in the cargo-driver logic, not in Cabal

# rustLib.a will then be put in the cabal build directory so that Cabal can statically link to it.
extra-bundled-library = "rustLib.a"

[[cargo-projects]]
  path = "native/rust-library"
  mode = "release" # or "debug"
  default-features = true
  features = ["feature1", "feature2"]

Challenges

Since there won't be any information passing between Cabal and the build drivers, some preliminary data should written on file in order to make this all work. Especially, the precise location of the build directory that needs to contain the .a is something to have.

Community feedback

Having taken the time to survey members of the community beforehand, I took their feedback into account and have reported their questions:

These are real questions I've received, if you have more that I haven't answered here, please ask them to me.

Q: Cabal is going to have to learn how to configure rust/zig/D/C++/meson/bazel projects?

A: No. The logic is offloaded to build drivers (like cargo-driver) that get the necessary information from a custom stanza

---

Q: Cabal already knows how to do it, it's called Setup.hs

A: Setup.hs' unrestricted code execution proves to be a liability. The ability here is to provide a clear and official path
to allow the transition from this Setup.hs use-case. One build driver can be analysed for n projects, but n Setup.hs files
must be analysed for n projects.

---

Q: This cargo-driver of yours, who's going to be in charge?

A: A healthy mix of stakeholders and community members, the idea being that some central coordination with the cabal team is needed and we can't have this left to the good will of the community altogether, because we want to present a coherent and reliable "Rust integration" story. The Haskell Foundation can have a stake in it, for instance. But anybody can make a driver, it's not a walled-garden. cargo-driver will be maintained in a more “official” capacity due to its ecosystem importance and significance.

---

Q: This is too heavy/rigid for me, why can't we just let the user figure it out?

A: The goal is to offer a clear and straightforward path to native code integration without having to retain the knowledge of those systems in Cabal. At the same time, more flexible tools are an open door to unregulated code execution.

---

Q: This is the wrong approach, you should put that knowledge in something higher-level like Bazel

A: This doesn't prevent you from using Bazel but forcing people to learn Bazel in order to do what other languages provide for free is asking too much for too few benefits.

---

Q: You're bloating Cabal, external dependencies shouldn't be necessary to use external dependencies

A: We think that the weight of having a build driver as a compile-time dependency to handle other build dependencies is an acceptable compromise.

---

Q: Realistically, what system only knows how to do cabal build? Can't you run a command before cabal build that takes care of everything?

A: Not every system is able to run arbitrary command when compiling a Haskell package. More specifically each time a package X depends on a package Y, cabal will build the dependency with a standard cabal build. Build platforms that support cabal shouldn't have to adapt to something too flexible when it comes to building cabal packages.

[angerman]

extra-bundled-library = "rustLib.a"

This should probably be build-product, output, or something? I see the idea of aligning extra-bundled-libraries from cabal with the extra-bundled-library here. I find it confusing though as we are not bundling anything with the driver?

We could in principle also bundle shared libraries, they don't necessarily have to be static, there's just more work involved to get that right.

A minor nit: rustLibrary and rustLib.a don't line up. We currently have a convention that we have lib<LANG><name>[-<version>].<ext>. Where <LANG> takes values of HS and C right now. name is base, ffi, ... so we end up with libHSbase.a, libCffi.a, ..., should we encourage this convention? libRSmylib.a?

[NorfairKing]

Soft -1 because the proposal does not discuss how to integrate this with things that use cabal like:

• stack
• nixpkgs
• cabal2nix
• Haskell.nix

[gbaz]

@angerman -- note the proposal isn't for the syntax of that toml file, which is just given as an example. it is just for the build-drivers field in cabal files.

@NorfairKing this is an extension to cabal files that Setup.hs will be taught about. Since afaik all those systems make use of Setup.hs, I believe no special integration with them is necessary, and your comment doesn't apply.

[NorfairKing]

I believe no special integration with them is necessary, and your comment doesn't apply.

+1 then :D

[angerman]

@gbaz I understand. The libraries should still match up for readability. I've in fact been discussing this with @Kleidukos, and mentioned my disliking of the initial x- embedding of the Toml file in the cabal file. I'm raising the naming topic here only, to clarify the embedded example. I very much like the fact that this proposal comes with an example usecase to demonstrate how the full thing could look

[Kleidukos]

Hi, thank you for your feedback.

@NorfairKing Yes I expect the integration with other tools to be transparent. :)

@angerman

should we encourage this convention? libRSmylib.a?

It was my understanding that Rust produced C-compatible libraries, so in my opinion we can call the archive "libCmylib.a".

I find it confusing though as we are not bundling anything with the driver?

Yes you are right it is confusing, I wanted to nod at the feature in Cabal but it doesn't have to be called like the Cabal stanza.

We could in principle also bundle shared libraries, they don't necessarily have to be static, there's just more work involved to get that right.

Could you expand a bit about it?

[fgaz]

To clarify, this is basically an isolated preConf hook except that its arguments are passed as files, right?

Why is build-drivers better than / avoids the prioblems of a custom setup?

Setup.hs' unrestricted code execution proves to be a liability

Isn't a build driver the same?

One build driver can be analysed for n projects, but n Setup.hs files must be analysed for n projects.

This is not necessarily true. A custom setup could simply consist of setup-depends: some-driver in the .cabal file + main = defaultMainFromSomeDriver in Setup.hs. How does this proposal differ from this kind of setup?

---

A remark about the hypothetical cargo driver:

Cabal the library is supposed to just take a set of inputs and produce a set of outputs, and as you write should not manage dependencies. I think this should include Cabal's subprocesses. Cargo can download and manage its own dependencies, and this can be a huge pain for distributions, especially since the cargo call would be deep in the process tree and harder to control. For example, in nixpkgs, cargo packages are built by first vendoring all dependencies, hashing them, then telling cargo where to find them. If cargo is called by another tool, there is no way to do the last part without writing a cargo wrapper. It is usually better to let the "topmost" build tool / package manager (nix in my example) orchestrate all the others, and avoid "build tool -> package manager" chains.

So I do at least partially agree with

This is the wrong approach, you should put that knowledge in something higher-level like Bazel

By the way, in "This doesn't prevent you from using Bazel but forcing people to learn Bazel in order to do what other languages provide for free is asking too much for too few benefits." what do you mean by "what other languages provide for free"?

[gbaz]

My understanding is that its a preconf hook in essence. The arguments aren't passed directly, so they can be in files, or they could not exist -- its outside of the scope. The problem with custom setups is they have access to all sorts of stuff and can configure or do anything -- they can reconfigure which modules are in the package or do lots of other stuff that affects later portions of the build. So custom setups require a fully different build path and produce packages whose basic metadata can't be statically known. This just runs some code without being able to affect other phases or feed anything back into cabal -- so its much more restricted.

(btw, note the range of issues that we have with custom setups being just generally flakier and having corner cases compared to build-type: Simple -- https://github.com/haskell/cabal/issues?q=is%3Aissue+is%3Aopen+label%3A%22Cabal%3A+stanza%2Fcustom-setup%22%2C%22Cabal%3A+custom%22+)

also see the motivation in the "What is needed to remove Custom build type" project: https://github.com/haskell/cabal/projects)

[fgaz]

Another advantage of this proposal is that build-drivers is local to a component, so it does not prevent per-component builds, does it?

[Kleidukos]

@fgaz Elixir has something called Rustler: https://hexdocs.pm/rustler/0.18.0/basics.html / https://github.com/rusterlium/rustler

It has enabled to simplify the integration of projects like html5ever: https://hex.pm/packages/html5ever

Another advantage of this proposal is that build-drivers is local to a component, so it does not prevent per-component builds, does it?

Yes it is a per-component basis. :)

[fgaz]

@fgaz Elixir has something called Rustler: https://hexdocs.pm/rustler/0.18.0/basics.html / https://github.com/rusterlium/rustler

It has enabled to simplify the integration of projects like html5ever: https://hex.pm/packages/html5ever

Yes, I'm not saying it isn't possible. Such a driver would work ok when used through cabal-install in an unrestricted system. But Cabal is meant to be used as basis for other package managers as well, where restrictions may actually be enforced.

For example, from the rustler docs:

:cargo - Specify how to envoke the rust compiler. Options are:

:system (default) - use cargo from the system (must be in $PATH)
{:system, } - use cargo from the system with the given channel. Specified as a string, passed directly to cargo (e.g. "+nightly").
{:rustup, } - use rustup to specify which channel to use. Available options include: :stable, :beta, :nightly, or a string which specifies a specific version (e.g. "1.39.0").
{:bin, "/path/to/binary"} - provide a specific path to cargo.

This kind of option (especially the last two) would have to be patched out to integrate a package that uses rustler in most distributions, where the compiler is provided externally.

As an analogy, calling cargo to install (transitive) rust dependencies is like automatically calling apt to install pkgconfig-depends entries.

But this is only relevant to the hypothetical cargo driver and not to this RFC, since wrapping a package manager can be done with current custom setups anyway.

[angerman]

I'm confused, this is not for installing system-library equivalents, right? This is in the case where you build a hybrid haskell/rust application. Similar to how we have hybrid haskell/c applications today.

If this is intended as a general "my application needs foreign library xyz; let's just pull that in", I'm going to be against that; that's what we have package management systems for, apt, rpm, ports, nix, homebrew, fink, ...

If this is to enable hybrid applications in which parts are written in rust and others in haskell, I think this is a sane approach. We can't go the C route, unless we want to turn cabal into a rust solver and builder; that would be unadvisable IMO.

extra-bundled-libraries is explicitly for the case where you want to bundle something with the package that you can't get from the system, but that is built outside of cabal itself, yet should be part of the package cabal puts into the package db.

If such packages break distributions trying to package these up, they ought to figure out how to do this or work around it; arguing against this from a perspective of "but others won't be able to", is artificially limiting. I did mention that the rustup and absolute path in the rustler solution is nothing that should be permitted.

The utility of this driver to me is to use external tools (rustc, cargo, ...) either on PATH or via environment variables, and encapsulate/abstract the build logic for a foreign library away from cabal, while still being able to tie the result back into the cabal package.

Why are custom setups terrible to deal with? Because they are effectively blackboxes doing arbitrary logic. If this is a driver hook, I can decide to not run it and figure out what to do about it or not. If we have a proliferation of custom setups, and we don't want to run custom setups (for various legit reasons), we are worse off. This adds declarative logic, whereas setup.hs is just an arbitrary blackbox. In the former case, I can whitelist build-drivers I'm ok with, and prohibit anything else just from the cabal description.

[Kleidukos]

@fgaz

But this is only relevant to the hypothetical cargo driver and not to this RFC, since wrapping a package manager can be done with current custom setups anyway.

Yes, this is a discussion we can (and should) concern ourselves with at a later time. :)