Description
The problem:
The code for deciding whether or not the test cases pass or fail is based on directly comparing the code_answer and expected_code_answer output blocks in the response to see if they're the same. This can be an issue for problems where the output is a list but the problem specifies that the order of the output doesn't matter. For instance, for problem 49 (Group Anagrams) the problem states that "[y]ou can return the answer in any order." However, running my solution to that problem gave:
FAIL:
Code Answer:
[["eat","tea","ate"],["tan","nat"],["bat"]]
[[""]]
[["a"]]
Expected Code Answer:
[["bat"],["nat","tan"],["ate","eat","tea"]]
[[""]]
[["a"]]
in the *leetcode-result-49* buffer. Since the only difference between the two blocks is the ordering of the elements of the lists, it looks like we're not respecting the order-insensitivity the problem specifies when assessing whether the tests passed or not.
Possible solution:
It looks like the result data structure coming back from Leetcode has a correct_answer field which is the Leetcode assessment of whether the answer was correct or not. For example, in the case shown above the correct_answer field is set to true.
It seems to me that we could use the Leetcode assessment of correctness directly - rather than doing it by hand - to decide whether to emit FAIL: or PASS: in the test buffer.
I have created a PR with that change (linked).