Improve local development workflow #861

staticfloat · 2020-07-28T17:16:40Z

This adds two new capabilities:

--deploy=local; for those who really just want to test out their new
JLLs locally, this makes it a cinch. Just tack --deploy=local onto
the end of your build_tarballs.jl invocation, and you'll get the
artifacts installed and the JLL packages built within your
~/.julia/dev directory. (Note that you will still need to add that
directory into any environment you want to use it from)
override directories for dev'ed out JLL packages. This allows a
user to override the binaries within a JLL easily and cheaply. At
compile time, the JLL package will look for an override directory in
its project root, and if it finds it, it will attempt to load binaries
from there instead of from an Artifact. To aid in setting this up, JLLs
now have a dev_jll() method that will perform the checkout and copy
over the normal artifact's contents to the override directory,
allowing for the user to easily see what must be overwritten in order to
test out a local build of something.

This adds two new capabilities: * `--deploy=local`; for those who really just want to test out their new JLLs locally, this makes it a cinch. Just tack `--deploy=local` onto the end of your `build_tarballs.jl` invocation, and you'll get the artifacts installed and the JLL packages built within your `~/.julia/dev` directory. (Note that you will still need to add that directory into any environment you want to use it from) * `override` directories for `dev`'ed out JLL packages. This allows a user to override the binaries within a JLL easily and cheaply. At compile time, the JLL package will look for an `override` directory in its project root, and if it finds it, it will attempt to load binaries from there instead of from an Artifact. To aid in setting this up, JLLs now have a `dev_jll()` method that will perform the checkout and copy over the normal artifact's contents to the `override` directory, allowing for the user to easily see what must be overwritten in order to test out a local build of something.

giordano · 2020-07-28T17:20:16Z

src/AutoBuild.jl

        \"\"\"
            is_available()

        Return whether the artifact is available for the current platform.
        \"\"\"
-        is_available() = false
+        is_available() = wrapper_available


Maybe call the function wrapper_available? I wasn't 100% happy with is_available

I don't care about the name; I just care about the compiler warnings about precompilation being broken when we redefine a top-level method. :P

If you want a different name, you can pick whatever you want. :)

fingolfin

Thank you!!! That's super helpful!

I left a bunch of nitpicky comments, trying to read from the perspective of a dumb user who tries to make sense of it all (i.e., mine). I hope this doesn't come across as complaining, it is intended as a constructive feedback.

fingolfin · 2020-07-29T16:35:25Z

docs/src/building.md

+developer you may want to easily use custom binaries.  In both cases, the new `dev` and
+`overrides` directory support make it easy to get complete control over the local JLL package
+state.


The way this is phrased (esp. the use of "the new ... support") makes it well suited for a release announcement, but IMHO somewhat less so for a permanent entry in the manual. I don't claim to know the perfect way to do this, but in an attempt to be constructive and not just destructive, here is an attempt for an alternative phrasing

Suggested change

developer you may want to easily use custom binaries. In both cases, the new `dev` and

`overrides` directory support make it easy to get complete control over the local JLL package

state.

developer you may want to easily use custom binaries. To support these use cases, we provide

means to make it easy to get complete control over the local JLL package state, which are

described in the following subsections.

fingolfin · 2020-07-29T16:35:53Z

docs/src/building.md

+## Overriding a prebuilt JLL package's binaries
+
+After running `pkg> dev LibFoo_jll`, a local JLL package will be checked out to your depot's
+`dev` directory, (on most installations this is `~/.julia/dev`) and by default it will make


"it" = the JLL?

Suggested change

`dev` directory, (on most installations this is `~/.julia/dev`) and by default it will make

`dev` directory (on most installations this is `~/.julia/dev`), and by default the JLL will make

fingolfin · 2020-07-29T16:42:38Z

docs/src/building.md

+After running `pkg> dev LibFoo_jll`, a local JLL package will be checked out to your depot's
+`dev` directory, (on most installations this is `~/.julia/dev`) and by default it will make
+use of binaries within your depot's `artifacts` directory.  However, if an `override`
+directory is present within that package directory, it will look within that directory for


Again, "it".

Also, when reading this I was unsure what "that package directory" refers to -- presumably the one in ~/.julia/dev? I didn't see mention of another, but "that" sounds (to me, as a non-native speaker, so this may very well be entirely be my own fault!) as if there was some alternate dir to consider? (And perhaps there is and I am just oblivious to it). Perhaps this could be made clearer; I am not quite sure how so; but for me, the following already is a bit clearer.

Suggested change

directory is present within that package directory, it will look within that directory for

directory is present within its package directory, the JLL will look within that directory for

fingolfin · 2020-07-29T16:45:58Z

docs/src/building.md

+
+### Auto-populating `override` directory
+
+To ease creation of an `override` directory, JLL packages now contain a `dev_jll()` function,


I am a bit torn of the use of "now" here, for the reason I already outlined above: how will this read to somebody in one year? One way to go about this wold be to insert a date in here, e.g. "since August 2020". But that doesn't seem great, also it depends a bit on when the patch is merged and Yggdrasil updated to use it?

So I guess instead I would suggest this: drop "now" here, and instead insert a comment somewhere along the lines of: "Note that older JLLs which were generated before this feature was introduced may lack a dev_jll
function."

Suggested change

To ease creation of an `override` directory, JLL packages now contain a `dev_jll()` function,

To ease creation of an `override` directory, JLL packages contain a `dev_jll()` function,

fingolfin · 2020-07-29T16:46:22Z

docs/src/building.md

+that will ensure that a `~/.julia/dev/<jll name>` package is `dev`'ed out, and will copy the
+normal artifact contents into the appropriate `override` directory.  This will result in no
+functional difference from simply using the artifact directory, but provides a template of
+files that can be replaced by custom-built binaries.


Suggested change

files that can be replaced by custom-built binaries.

files that can be replaced by custom-built binaries.

Note that JLLs which were generated before this feature was introduced to

BinaryBuilder may lack a `dev_jll` function.

fingolfin · 2020-07-29T16:47:24Z

docs/src/building.md

+## Building a custom JLL package locally
+
+When building a new version of a JLL package, if `--deploy` is passed to `build_tarballs.jl`
+then a JLL package will be deployed to a GitHub repository.  (Read the documentation given


Suggested change

then a JLL package will be deployed to a GitHub repository. (Read the documentation given

then the newly-built JLL package will be deployed to a GitHub repository. (Read the documentation given

fingolfin · 2020-07-29T16:47:33Z

docs/src/building.md

+
+When building a new version of a JLL package, if `--deploy` is passed to `build_tarballs.jl`
+then a JLL package will be deployed to a GitHub repository.  (Read the documentation given
+by passing `--help` to a `build_tarballs.jl` script for more on `--deploy` options)  If


Suggested change

by passing `--help` to a `build_tarballs.jl` script for more on `--deploy` options) If

by passing `--help` to a `build_tarballs.jl` script for more on `--deploy` options). If

fingolfin · 2020-07-29T16:48:20Z

docs/src/building.md

+by passing `--help` to a `build_tarballs.jl` script for more on `--deploy` options)  If
+`--deploy=local` is passed, the JLL package will still be built in the `~/.julia/dev/`
+directory, but it will not be uploaded anywhere.  This is useful for local testing and
+validation that the built artifacts are working with your package.


Suggested change

validation that the built artifacts are working with your package.

for validating that the built artifacts are working with your package.

fingolfin · 2020-07-29T16:49:18Z

docs/src/jll.md

+their own binaries, the user may use the `dev_jll()` method inside of JLL packages
+to check out a mutable copy of the package to their `~/.julia/dev` directory.  An
+`override` directory will be created within the package directory, providing a
+convenient location for the user to copy in their own libraries over the typical


Suggested change

convenient location for the user to copy in their own libraries over the typical

convenient location for the user to copy in their own files over the typical

I'd also consider changing "binaries" at the start of this paragraph to "files", as in principle one might also want to tweak, say, a header or config file, right? Wrong?

fingolfin · 2020-07-29T16:50:02Z

docs/src/jll.md

+In the event that a user wishes to override the binaries within a JLL package with
+their own binaries, the user may use the `dev_jll()` method inside of JLL packages


Suggested change

In the event that a user wishes to override the binaries within a JLL package with

their own binaries, the user may use the `dev_jll()` method inside of JLL packages

In the event that a user wishes to override the binaries within a JLL package with

their own binaries, the user may use the `dev_jll()` method provided by that JLL package

fingolfin · 2020-08-02T21:10:08Z

I think this resolves #521

giordano reviewed Jul 28, 2020

View reviewed changes

simeonschaub mentioned this pull request Jul 28, 2020

Ruby 2.7.1 JuliaPackaging/Yggdrasil#1339

Merged

fingolfin reviewed Jul 29, 2020

View reviewed changes

mortenpi mentioned this pull request Jul 30, 2020

Allow specifying the dev directory #866

Open

fingolfin mentioned this pull request Aug 2, 2020

local build of foo_jll package #521

Closed

Tweak docs

b779e8b

staticfloat merged commit 7d4bdce into master Aug 4, 2020

staticfloat deleted the sf/jll_overrides branch August 4, 2020 21:16

fingolfin mentioned this pull request Aug 5, 2020

build_tarballs: fix call to get_meta_json #874

Merged

giordano mentioned this pull request Aug 6, 2020

Add SCIP JuliaPackaging/Yggdrasil#1388

Merged

liuwa666 mentioned this pull request Oct 5, 2023

Improve local development workflow by staticfloat · Pull Request #861 · JuliaPackaging/BinaryBuilder.jl liuwa666/webclipper#137

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improve local development workflow #861

Improve local development workflow #861

Uh oh!

staticfloat commented Jul 28, 2020

Uh oh!

giordano Jul 28, 2020

Uh oh!

staticfloat Jul 28, 2020

Uh oh!

fingolfin left a comment

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin Jul 29, 2020

Uh oh!

fingolfin commented Aug 2, 2020

Uh oh!

Uh oh!

-developer you may want to easily use custom binaries.  In both cases, the new `dev` and
-`overrides` directory support make it easy to get complete control over the local JLL package
-state.
+developer you may want to easily use custom binaries. To support these use cases, we provide
+means to make it easy to get complete control over the local JLL package state, which are
+described in the following subsections.

	`dev` directory, (on most installations this is `~/.julia/dev`) and by default it will make
	`dev` directory (on most installations this is `~/.julia/dev`), and by default the JLL will make

	directory is present within that package directory, it will look within that directory for
	directory is present within its package directory, the JLL will look within that directory for


		### Auto-populating `override` directory

		To ease creation of an `override` directory, JLL packages now contain a `dev_jll()` function,

	then a JLL package will be deployed to a GitHub repository. (Read the documentation given
	then the newly-built JLL package will be deployed to a GitHub repository. (Read the documentation given

	by passing `--help` to a `build_tarballs.jl` script for more on `--deploy` options) If
	by passing `--help` to a `build_tarballs.jl` script for more on `--deploy` options). If

	validation that the built artifacts are working with your package.
	for validating that the built artifacts are working with your package.

	convenient location for the user to copy in their own libraries over the typical
	convenient location for the user to copy in their own files over the typical

		In the event that a user wishes to override the binaries within a JLL package with
		their own binaries, the user may use the `dev_jll()` method inside of JLL packages

Improve local development workflow #861

Improve local development workflow #861

Uh oh!

Conversation

staticfloat commented Jul 28, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fingolfin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fingolfin commented Aug 2, 2020

Uh oh!

Uh oh!