"unresolved extern crate" when importing rustc crates from a locally built toolchain

[RalfJung]

To reproduce, first locally build a rustc (./x.py build compiler --stage 2), then check out Miri, configure it for RA as shown below, and open it in vscode. Now open the src/lib.rs file. There are errors for extern crate rustc_middle; and all the other rustc crate imports: "unresolved extern crate".

A ./miri check works fine, so the crates do exist, but RA somehow fails to find them.

rust-analyzer version: rust-analyzer version: 0.3.1148-standalone (2b472f6 2022-07-31)

rustc version: locally built rustc master

relevant settings:

// Place your settings in this file to overwrite default and user settings.
{
    "rust-analyzer.rustc.source": "discover",
    "rust-analyzer.linkedProjects": [
        "./Cargo.toml",
        "./cargo-miri/Cargo.toml"
    ],
    "rust-analyzer.checkOnSave.overrideCommand": [
        "./miri",
        "check",
        "--message-format=json"
    ],
    "rust-analyzer.buildScripts.overrideCommand": [
        "./miri",
        "check",
        "--message-format=json",
    ],
    "rust-analyzer.rustfmt.extraArgs": [
        "+nightly"
    ],
}

[bjorn3]

"rust-analyzer.rustc.source": "discover" doesn't work for linked toolchains as the rust-src component is absent. You need to manually point it to the root Cargo.toml of your rust checkout.

[RalfJung]

I switch between linked toolchains and downloaded toolchains all the time, so that is not a solution.

Why can't RA auto-discover linked toolchains? Miri for example auto-discovers the rust-src location for both linked and installed toolchains just fine.

[Yvan-xy]

I have the same issue.

[Veykril]

I'm not sure how this all works, so I'm just gonna lay out what r-a does right now for discovering. First it checks if RUST_SRC_PATH is set, and uses that if it is. Otherwise r-a invokes rustc --print sysroot with the containing directory of the workspace toml as the working directory, then checks whether {sysroot}/lib/rustlib/src/rust/library exists to use that as the path. What is miri doing differently here, what should we maybe do differently here?

[RalfJung]

Yeah {sysroot}/lib/rustlib/src/rust/library works for normal toolchains but not for linked toolchains. I can't find where Miri even has that logic, but for a linked sysroot, the folder is $(rustc --print sysroot)/../../../library.

[RalfJung]

Oh, Miri doesn't have that logic any more since $(rustc --print sysroot)/lib/rustlib/src is a symlink so these things work even without doing something special on the Miri side.

Could R-A be confused by the symlink?

[Veykril]

Ah wait, the rustc that is being invoked there is not the installed toolchain one right? (well in your case miri but whatever). I imagine that might be the problem then? I'll add some logging for the sysroot stuff that way you should be able to tell what might be going on here (I don't feel like building rustc myself tbh)

Edit: Actually setting RA_LOG=project_model::sysroot=debug should already log the commands that are invoked at least, maybe that shows some helpful info?

[RalfJung]

The override in the Miri folder is set to the linked toolchain. So rustc --print sysroot in the Miri folder should print the right folder.

The rustc build will take a bit before I can provide logs.

[Veykril]

How is the override specified? It might not work for r-a here depending on how you set the override and where

Edit: Looks like a directory override if I see this right? That should work fine then hmm
fwiw this is how we fetch the executable path for rustc and the like

rust-analyzer/crates/toolchain/src/lib.rs

Lines 28 to 56 in 3392573

	fn get_path_for_executable(executable_name: &'static str) -> PathBuf {
	// The current implementation checks three places for an executable to use:
	// 1) Appropriate environment variable (erroring if this is set but not a usable executable)
	// example: for cargo, this checks $CARGO environment variable; for rustc, $RUSTC; etc
	// 2) `<executable_name>`
	// example: for cargo, this tries just `cargo`, which will succeed if `cargo` is on the $PATH
	// 3) `~/.cargo/bin/<executable_name>`
	// example: for cargo, this tries ~/.cargo/bin/cargo
	// It seems that this is a reasonable place to try for cargo, rustc, and rustup
	let env_var = executable_name.to_ascii_uppercase();
	if let Some(path) = env::var_os(&env_var) {
	return path.into();
	}
	if lookup_in_path(executable_name) {
	return executable_name.into();
	}
	if let Some(mut path) = home::home_dir() {
	path.push(".cargo");
	path.push("bin");
	path.push(executable_name);
	if let Some(path) = probe(path) {
	return path;
	}
	}
	executable_name.into()
	}

[RalfJung]

How is the override specified? It might not work for r-a here depending on how you set the override and where

rustup override set

[Veykril]

Added some logging to emit the sysroot and rustc paths being used #13441

[RalfJung]

The RA_LOG you suggested prints this several times

[DEBUG project_model::sysroot] Discovering sysroot for /home/r/src/rust/miri
[DEBUG project_model::sysroot] Discovering sysroot by "rustc" "--print" "sysroot"
[DEBUG project_model::sysroot] Checking sysroot: /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/src/rust/library
[DEBUG project_model::sysroot] Discovering rustc source for /home/r/src/rust/miri/./Cargo.toml
[DEBUG project_model::sysroot] Discovering sysroot by "rustc" "--print" "sysroot"
[DEBUG project_model::sysroot] Checking for rustc source code: /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml
[DEBUG project_model::sysroot] Discovering sysroot for /home/r/src/rust/miri/./cargo-miri
[DEBUG project_model::sysroot] Discovering sysroot by "rustc" "--print" "sysroot"
[DEBUG project_model::sysroot] Checking sysroot: /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/src/rust/library
[DEBUG project_model::sysroot] Discovering rustc source for /home/r/src/rust/miri/./cargo-miri/Cargo.toml
[DEBUG project_model::sysroot] Discovering sysroot by "rustc" "--print" "sysroot"
[DEBUG project_model::sysroot] Checking for rustc source code: /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml

Some of the paths it mentions indeed exist (/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/src/rust/library) but others do not (/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml).

[RalfJung]

So looks like we have to extend whatever code sets up the symlink in lib/rustlib/src/rust to also set up the same symlink at lib/rustlib/rustc-src/rust?

[Veykril]

Ye I would guess so? This is what r-a is doing fwiw

rust-analyzer/crates/project-model/src/sysroot.rs

Lines 219 to 238 in 82ac6f7

	fn get_rustc_src(sysroot_path: &AbsPath) -> Option<ManifestPath> {
	let rustc_src = sysroot_path.join("lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml");
	let rustc_src = ManifestPath::try_from(rustc_src).ok()?;
	tracing::debug!("checking for rustc source code: {}", rustc_src.display());
	if fs::metadata(&rustc_src).is_ok() {
	Some(rustc_src)
	} else {
	None
	}
	}
	fn get_rust_src(sysroot_path: &AbsPath) -> Option<AbsPathBuf> {
	let rust_src = sysroot_path.join("lib/rustlib/src/rust/library");
	tracing::debug!("checking sysroot library: {}", rust_src.display());
	if fs::metadata(&rust_src).is_ok() {
	Some(rust_src)
	} else {
	None
	}
	}

Just simply append the path to the sysroot, and I don't think there is much more we can do from our side? Though it's odd, how does rust do this then ...

[RalfJung]

Indeed I can confirm that rust-lang/rust#103347 fixes this problem. :)