Support user-defined types when generating schema

[gschulze]

Currently, the schema function in ApiV1.kt does not handle user-defined types correctly. If a field has a type with a UDT registration, the corresponding UserDefinedType should be used as schema.

I've drafted an implementation at https://github.com/gschulze/kotlin-spark-api. However, I'm not sure whether this is the right place to do it.

[asm0dey]

Thank you very much!

Could I please ask you to create minimalistic example with UDT for me to be able to test it?

[gschulze]

Hi, sorry for the late reply. I suggest waiting until both Support for Spark 2.4.x and 3.x has landed. I am happy to work on the issue once the structure has been finalized.

[asm0dey]

You'll be disappointed by new structure, but I don't have any idea on how to improve it yet :(

[gschulze]

I suggest using Maven profiles whenever possible to target different Spark/Scala versions. I'll try to come up with something and open a separate issue.

[asm0dey]

I thought about it, but there are some troubles: most important parts (scala code and kotlin code creating encoders and so on, not extensions) are different. Also, that's not possible (IIRC) to release from different profiles at same time.

[gschulze]

I filed a new ticket for building for specific environments, in particular Scala 2.11: #59

[asm0dey]

@gschulze with a couple of issues we already have working (and better!) implementation for both Spark 2.4 and Spark 3.0, so we can continue discussion on topic