Windows jobs on Jenkins hang on sbt startup since sbt 1.3 upgrade

[SethTisue]

both 2.12.x and 2.13.x are affected, so we should fix on 2.12.x and merge the fix forwards.

• 2.12: [backport] Upgrade 2.12.x build to SBT 1.3.7 [ci: last-only] scala#8660, https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/
• 2.13: move to sbt 1.3, fix build on JDK 13+ scala#8525, https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.13.x-integrate-windows/

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/1642/console

++ SBT='java -Dfile.encoding=UTF-8 -server -XX:+AggressiveOpts -XX:+UseParNewGC -Xmx2G -Xss1M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=128M -Dpartest.threads=4 -Duser.home=y:/jenkins -Djava.io.tmpdir=y:/jenkins/tmp -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native -Dsbt.ivy.home=y:\jenkins\workspace\scala-2.12.x-integrate-windows/.ivy2 -jar C:\sbt\sbt-launch.jar -Dsbt.override.build.repos=true -Dsbt.repository.config=y:\jenkins\workspace\scala-2.12.x-integrate-windows/scripts/sbt-repositories-config'
++ java -Dfile.encoding=UTF-8 -server -XX:+AggressiveOpts -XX:+UseParNewGC -Xmx2G -Xss1M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=128M -Dpartest.threads=4 -Duser.home=y:/jenkins -Djava.io.tmpdir=y:/jenkins/tmp -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native '-Dsbt.ivy.home=y:\jenkins\workspace\scala-2.12.x-integrate-windows/.ivy2' -jar 'C:\sbt\sbt-launch.jar' -Dsbt.override.build.repos=true '-Dsbt.repository.config=y:\jenkins\workspace\scala-2.12.x-integrate-windows/scripts/sbt-repositories-config' -warn setupPublishCore generateBuildCharacterPropertiesFile publishLocal
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? Build timed out (after 200 minutes). Marking the build as aborted.
Build was aborted

as originally reported a couple weeks ago here, but we merged the upgrades anyway, which is fine. this has now worked its way to the top of my priorities

the jobs run scripts/jobs/integrate/windows — one place to start would be to add -debug to the sbt invocation (on a branch) to get more info about where it's getting stuck

[SethTisue]

😱

[SethTisue]

running with -debug isn't helping: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/1643/console

the command line invocation is java ... -jar 'C:\sbt\sbt-launch.jar' ..., one possibility is that the sbt launch jar is simply too old. perhaps we should try upgrading that before considering other hypotheses

doc on how to ssh to the Windows worker is at https://github.com/scala/scala-jenkins-infra/blob/master/doc/client-setup.md#windows-setup , it's old but probably still works

once I'm in I could try replacing the launch jar; I could also run jstack to see where sbt is getting stuck

[SethTisue]

@soronpo notes that this might be sbt/sbt#5128 /cc @eed3si9n

[OlegYch]

someone should really fix that lib to not use powershell, or remove it altogether

[SethTisue]

I haven't had any luck sshing to the Windows worker — I suspect something changed the last time Adriaan redid stuff (chef->ansible)

I see over on the sbt ticket that setting -Dsbt.coursier.home might be a workaround, trying that next

(the things I'm trying are commits to https://github.com/SethTisue/scala/commits/sd-666)

[SethTisue]

okay, now at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-windows/1644/console we see:

[error] Caused by: java.io.IOException: Cannot run program "powershell.exe": CreateProcess error=2, The system cannot find the file specified
[error] 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
[error] 	at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.Util.runCommands(Util.java:176)
[error] 	at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.Util.getWinDirs(Util.java:147)
[error] 	at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.Util.getWinDirs(Util.java:120)
[error] 	at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.ProjectDirectories.fromPath(ProjectDirectories.java:221)
[error] 	at lmcoursier.internal.shaded.coursier.cache.shaded.io.github.soc.directories.ProjectDirectories.from(ProjectDirectories.java:274)
[error] 	at lmcoursier.internal.shaded.coursier.paths.CoursierPaths.coursierDirectories(CoursierPaths.java:82)
[error] 	at lmcoursier.internal.shaded.coursier.paths.CoursierPaths.computeCacheDirectory(CoursierPaths.java:44)
[error] 	at lmcoursier.internal.shaded.coursier.paths.CoursierPaths.cacheDirectory(CoursierPaths.java:70)
[error] 	at lmcoursier.internal.shaded.coursier.paths.CachePath.defaultCacheDirectory(CachePath.java:93)
[error] 	at lmcoursier.internal.shaded.coursier.cache.CacheDefaults$.location$lzycompute(CacheDefaults.scala:16)
[error] 	at lmcoursier.internal.shaded.coursier.cache.CacheDefaults$.location(CacheDefaults.scala:16)
[error] 	at lmcoursier.internal.shaded.coursier.cache.FileCache$.apply(FileCache.scala:1088)
[error] 	at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:134)
[error] 	at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60)

this appears to be dirs-dev/directories-jvm#29 (to which sbt/sbt#5386 is a reference)

I think the next step here is probably for me to get ssh access to the Windows worker, otherwise the troubleshooting process (pushing changes to scripts/jobs/integrate/windows) is just too cumbersome

[eed3si9n]

I'd be curious to know if powershell can be invoked from ssh as well.

Also setting COURSIER_HOME environmental variable might be a workaround?

[SethTisue]

Also setting COURSIER_HOME environmental variable might be a workaround?

I did that (well, -Dsbt.coursier.home, not COURSIER_HOME, but it's the same, right?)

Before that change, it was hanging. After, we get Cannot run program "powershell.exe"

[som-snytt]

It's Number 666 because it's the Powers of Hell.

[eed3si9n]

https://github.com/coursier/sbt-coursier/blob/460a79b96d80d4e2d4fe315cea1a5a8587221357/modules/lm-coursier/src/main/scala/lmcoursier/CoursierDependencyResolution.scala#L134-L140

    val cache0 = coursier.cache.FileCache()
      .withLocation(cache)
      .withCachePolicies(cachePolicies)
      .withTtl(ttl)
      .withChecksums(checksums)
      .withCredentials(conf.credentials.map(ToCoursier.credentials))
      .withFollowHttpToHttpsRedirections(conf.followHttpToHttpsRedirections.getOrElse(true))

It seems like FileCache() or FileCache$.apply is where it's blowing up so there may be a difference between -Dsbt.coursier.home and COURSIER_HOME.

[alexarchambault]

I confirm that the COURSIER_HOME env var should currently work better than the sbt.coursier.home Java property. That needs to be fixed in coursier. (The problem is that calling FileCache() evaluates some default values, and one of them runs that powershell command at the end, all of that before users have a chance to call withLocation.)