scalac 2.13.1 "incompatible type in overriding" subclassing Hadoop FileSystem that javac accepts

[srowen]

Hi all, long time listener, first time poster. I have an odd scalac 2.13.1 issue that's affecting Apache Spark's transition to 2.13 (https://issues.apache.org/jira/browse/SPARK-25075 -> https://issues.apache.org/jira/browse/SPARK-30132)

It occurs in scalac 2.13.1 but not 2.12.10. This occurs in Java 8. I didn't see an obvious open issue here, or easy answer from the gitter channel, but #11575 looks possibly related.

Consider this simple subclass of a Hadoop LocalFileSystem class:

object foo extends org.apache.hadoop.fs.LocalFileSystem {}

Compiled with scalac 2.13.1, I get:

$ scalac -cp ~/.m2/repository/org/apache/hadoop/hadoop-common/3.2.0/hadoop-common-3.2.0.jar foo.scala 
foo.scala:1: error: incompatible type in overriding
def appendFile(x$1: org.apache.hadoop.fs.Path): org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] forSome { type S <: org.apache.hadoop.fs.FSDataOutputStream; type B <: org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] } (defined in class FileSystem)
  with def appendFile(x$1: org.apache.hadoop.fs.Path): org.apache.hadoop.fs.FSDataOutputStreamBuilder (defined in class FilterFileSystem);
 found   : (x$1: org.apache.hadoop.fs.Path)org.apache.hadoop.fs.FSDataOutputStreamBuilder
 required: (x$1: org.apache.hadoop.fs.Path)org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] forSome { type S <: org.apache.hadoop.fs.FSDataOutputStream; type B <: org.apache.hadoop.fs.FSDataOutputStreamBuilder[S,B] }
object foo extends org.apache.hadoop.fs.LocalFileSystem {}
       ^

This says that the compiled class hierarchy for this Hadoop class is invalid, though naturally, javac has accepted and compiled it. The definition in FileSystem.java in Hadoop 3.2.0:

  public FSDataOutputStreamBuilder appendFile(Path path) {
    return new FileSystemDataOutputStreamBuilder(this, path).append();
  }

The subtype FilterFileSystem.java has:

  public FSDataOutputStreamBuilder appendFile(Path path) {
    return new FileSystemDataOutputStreamBuilder(this, path).append();
  }

The signature is evidently identical. Now, FSDataOutputStreamBuilder does take two generic types, not specified here in the Hadoop source code. I kind of understand the error, that you can't say that the generic types in the overload are consistent.

But is this something that scalac should reject?

[Jasper-M]

Looks like it might be related to #11770

[joshlemer]

Minimized example:

Example.java

public class Example {
    public static abstract class X<A extends java.lang.Object, B extends X<A, B>> {}

    public static abstract class Y {
        public X appendFile(String s) { return null; }
    }

    public static class Z extends Y {
        @Override
        public X appendFile(String s) { return null; }
    }
}

Main.scala

object foo extends Example.Z {}

Error:

Error:(1, 8) incompatible type in overriding
def appendFile(s: String): Example.X[A,B] forSome { type A; type B <: Example.X[A,B] } (defined in class Y)
  with def appendFile(s: String): Example.X (defined in class Z);
 found   : (s: String)Example.X
 required: (s: String)Example.X[A,B] forSome { type A; type B <: Example.X[A,B] }
object foo extends Example.Z {}

[Jasper-M]

Probably also #11584

[smarter]

scala/scala#8643 should fix this, you can try it by adding a resolver to your build and setting the scalaVersion:

resolvers += "pr" at "https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/"
scalaVersion := "2.13.2-bin-ba53545-SNAPSHOT"