Infer 'mixed' types as strings when using Arrow serialization

[rshkv]

Background

Follow-up for #679. When serializing Pandas dataframes, Arrow assumes Python 2 string columns as "binary". When we encounter such a column, we try to infer if it's actually a "string" column (in a user-friendly sense). If it is, we override Arrow's assessment and use "string" as type in the Spark schema.

Problem

When inferring whether "binary" columns are in fact "string", we rely on asking Pandas what it thinks the column is. Pandas is more friendly of Python 2, and thinks that Python 2 string columns are actually "string".

But we encounter an edge case when a column contains values of mixed type, even if all types could be treated as strings. E.g., if a column contains both unicode and binary values, Pandas infers the column as "mixed". We currently don't infer a column as string if Pandas deemed it of "mixed" type.

In [1]: pandas_df = pd.DataFrame(data={"mixed_strings": [u"unicode", b"binary"]})

In [2]: spark.createDataFrame(pandas_df).printSchema()
root
 |-- mixed_strings: binary (nullable = true) # oops

You can also see this behaviour in the failing test that I added (CircleCI).

What changes were proposed in this pull request?

We are now more generous when inferring whether columns should be "string" type. When Pandas infers a column as "mixed" type, we do an additional check: We read the first value in the column. If that value is a string, we consider the whole column a string.

For this check, we're using the six package, which will correctly determine that e.g. a unicode value is of string type, but not e.g. a bytearray.

Alternative

We could simplify this by using Pandas' is_string_dtype function which takes a column and infers if it's of "string" type. But the function is very lenient (pandas-dev/pandas#15585). E.g. it will consider a column bytearrays a string (which would be an example of a "true binary" column even under Python 2). It will also consider a column of float tuplies a string (effectively anything that is an array of something).

How was this patch tested?

Existing test and new added tests. Commit history shows that tests were failing before changes were made.

[rshkv]

cc @helenyugithub