Skip to content

[SPARK-35599][PYTHON] Adjust check_exact parameter for older pd.testing #32772

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from

Conversation

xinrong-meng
Copy link
Member

@xinrong-meng xinrong-meng commented Jun 3, 2021

What changes were proposed in this pull request?

Adjust the check_exact parameter for non-numeric columns to ensure pandas-on-Spark tests passed with all pandas versions.

Why are the changes needed?

pd.testing utils are utilized in pandas-on-Spark tests.
Due to pandas-dev/pandas#35446, check_exact=True for non-numeric columns doesn't work for older pd.testing utils, e.g. assert_series_equal. We wanted to adjust that to ensure pandas-on-Spark tests pass for all pandas versions.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing unit tests.

Keyword: SPARK-35337

@SparkQA
Copy link

SparkQA commented Jun 3, 2021

Test build #139298 has finished for PR 32772 at commit 787b3ac.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43821/

@SparkQA
Copy link

SparkQA commented Jun 3, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43821/

@xinrong-meng xinrong-meng changed the title [WIP][SPARK-35599][PYTHON] Adjust check_exact parameter for older pd.testing [SPARK-35599][PYTHON] Adjust check_exact parameter for older pd.testing Jun 4, 2021
@xinrong-meng xinrong-meng marked this pull request as ready for review June 4, 2021 18:30
@SparkQA
Copy link

SparkQA commented Jun 5, 2021

Test build #139358 has finished for PR 32772 at commit d3f44f0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 5, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43880/

@SparkQA
Copy link

SparkQA commented Jun 5, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43880/

Copy link
Member

@ueshin ueshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@xinrong-meng
Copy link
Member Author

Doc builds failed. I will rebase and retest.

@SparkQA
Copy link

SparkQA commented Jun 5, 2021

Test build #139362 has finished for PR 32772 at commit 59ad0a7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 5, 2021

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43884/

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants