Support types other than String and Int for partition columns #1154
+132
−57
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
timsaucer
reviewed
Jun 17, 2025
python/datafusion/context.py
Outdated
| @@ -535,7 +535,7 @@ def register_listing_table( | |||
| self, | |||
| name: str, | |||
| path: str | pathlib.Path, | |||
| table_partition_cols: list[tuple[str, str]] | None = None, | |||
| table_partition_cols: list[tuple[str, pa.DataType]] | None = None, | |||
There was a problem hiding this comment.
To avoid a breaking change, we could make the type hint
table_partition_cols: list[tuple[str, pa.DataType]] | list[tuple[str, str]] | None = None
And then in the python code just below we can do type checking of the table_partition_cols and coerce to pyarrow data type. Alternatively we could overload the method signature and mark the old one as deprecated for a couple of releases. Either way we could avoid a breaking change without giving users the opportunity to upgrade at their schedule.
There was a problem hiding this comment.
That's fair, I opted for overloading and raising the deprecation warning
timsaucer
approved these changes
Jun 19, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1155
Rationale for this change
Following datafusion PR4221, this PR exposes the functionality in the python bindings
What changes are included in this PR?
Change of the
table_partition_colsarguments inPySessionContext.register_listing_table,PySessionContext.register_parquet,PySessionContext.register_avro,PySessionContext.register_json,PySessionContext.read_parquet,PySessionContext.read_avro,PySessionContext.read_jsonAre there any user-facing changes?
Yes, the signature of the functions above is changed