Skip to content

Use the leetcode assessment of test correctness #149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

dpkatz
Copy link
Contributor

@dpkatz dpkatz commented Apr 1, 2025

The code currently decides whether or not the answer is the expected answer by comparing the code_answer and expected_code_answer output blocks in the response to see if they're the same. This can be an issue for problems where the output is a list but the problem specifies that the order doesn't matter. For instance, for problem 49 (Group Anagrams) the problem states that "You can return the answer in any order." However, running my solution to that problem gave:

FAIL:

Code Answer:
[["eat","tea","ate"],["tan","nat"],["bat"]]
[[""]]
[["a"]]

Expected Code Answer:
[["bat"],["nat","tan"],["ate","eat","tea"]]
[[""]]
[["a"]]

in the *leetcode-result-49* buffer. The problem here is that the we're not respecting the order-insensitivity of the answer the problem specifies.

Looking at the result data structure, we see that there is also a correct_answer field which is the Leetcode assessment of whether the answer was correct or not. For example, in the case shown above the "correct_answer" field is set to "true".

This change uses the "correct_answer" field to decide whether the test passed or failed, rather than comparing the output blocks directly.

Resolves #148

The code currently decides whether or not the answer is the expected
answer by comparing the `code_answer` and `expected_code_answer`
output blocks in the response to see if they're the same.  This can be
an issue for problems where the output is a list but the problem
specifies that the order doesn't matter.  For instance, for problem
49 (Group Anagrams) the problem states that "You can return the answer
in **any order**."  However, running my solution to that problem gave:

    FAIL:

    Code Answer:
    [["eat","tea","ate"],["tan","nat"],["bat"]]
    [[""]]
    [["a"]]

    Expected Code Answer:
    [["bat"],["nat","tan"],["ate","eat","tea"]]
    [[""]]
    [["a"]]

in the `*leetcode-result-49*` buffer.  The problem here is that the
we're not respecting the order-insensitivity of the answer the problem
specifies.

Looking at the result data structure, we see that there is also a
`correct_answer` field which is the Leetcode assessment of whether the
answer was correct or not.  For example, in the case shown above the
"correct_answer" field is set to "true".

This change uses the "correct_answer" field to decide whether the test
passed or failed, rather than comparing the output blocks directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorrect marking of passing test as failing in cases of order insensitivity
1 participant