Enable a mode in the parser in which it inspects alternative token choices to mutate test cases #1340
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@swift-ci Please test |
bnbarham
approved these changes
Feb 14, 2023
Sources/SwiftParser/TokenSpec.swift
Outdated
| /// matches this `TokenSpec`. | ||
| /// | ||
| /// IMPORTANT: Should only be used when generating tokens during the | ||
| /// modifiation of test cases. This should never be used in the parser itself. |
There was a problem hiding this comment.
Suggested change
| /// modifiation of test cases. This should never be used in the parser itself. | |
| /// modification of test cases. This should never be used in the parser itself. |
Comment on lines
560
to
562
| // FIXME: Currently, tests are failing when we enable test case mutation. | ||
| // Once all of those issues are fixed, this should become | ||
| // let enableTestCaseMutation = ProcessInfo.processInfo.environment["SKIP_LONG_TESTS"] != "1" |
There was a problem hiding this comment.
Out of interest, how much longer does this take to run?
There was a problem hiding this comment.
Oh, it’s quite significant. It increases test time from 39s to 250s if you are not skipping long tests (SKIP_LONG_TESTS=1)
There was a problem hiding this comment.
How many mutations happen? Given it's mutating one at a time (concurrently, but each one just has the one change), That's maybe not as bad as I thought it would be 😅. Wonder if it's worth batching some together.
There was a problem hiding this comment.
How many mutations happen?
I didn’t measure but I imagine there are on average 3 to 5 different choices for each token. I thought about batching them together, but you really need to be clever so one doesn’t shadow the other. And trying to be clever is usually a bad idea in my experience.
ahoppen
force-pushed
the
ahoppen/source-alteration
branch
from
78fe006 to
cf26631
Compare
ahoppen
added a commit
to ahoppen/swift-syntax
that referenced
this pull request
Apr 14, 2023
These parser failures were found by swiftlang#1340. - There was one round-trip failure introduced by swiftlang#1464 - A few cases were just consume tokens as the wrong child - The list of expected token kinds of `DeclModifierSynax` didn’t contain one modifier that was actually parsed
ahoppen
added a commit
to ahoppen/swift
that referenced
this pull request
Apr 14, 2023
ahoppen
force-pushed
the
ahoppen/source-alteration
branch
from
cf26631 to
f3e13ef
Compare
ahoppen
added a commit
to ahoppen/swift-syntax
that referenced
this pull request
Apr 14, 2023
These parser failures were found by swiftlang#1340. - There was one round-trip failure introduced by swiftlang#1464 - A few cases were just consume tokens as the wrong child - The list of expected token kinds of `DeclModifierSynax` didn’t contain one modifier that was actually parsed
…oices to mutate test cases The basic idea is that the parser records which `TokenSpec`s it checked for at individual offsets in the source. We can then use that information to generate new, interesting test cases by replacing a token by the one of the `TokenSpec` we checked for. This technique has found 11 bugs in the parser and I’m expecting it to find quite a few more once we assert that tokens have one of the expected kinds. Gathering of that information is hidden behind a conditional compilation flag because just performing the check of whether we want to record alternative token choices inflicts a 6% performance regression, which doesn’t provide any value except when we are running SwiftParserTest.
ahoppen
force-pushed
the
ahoppen/source-alteration
branch
from
f3e13ef to
b32317c
Compare
|
@swift-ci Please test |
ahoppen
added a commit
to ahoppen/swift-syntax
that referenced
this pull request
Apr 18, 2023
Enable a mode in the parser in which it inspects alternative token choices to mutate test cases
ahoppen
added a commit
to ahoppen/swift-syntax
that referenced
this pull request
Apr 18, 2023
Enable a mode in the parser in which it inspects alternative token choices to mutate test cases
meg-gupta
pushed a commit
to meg-gupta/swift
that referenced
this pull request
Apr 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The basic idea is that the parser records which
TokenSpecs it checked for at individual offsets in the source. We can then use that information to generate new, interesting test cases by replacing a token by the one of theTokenSpecwe checked for. This technique has found 7 bugs in the parser and I’m expecting it to find quite a few more once we assert that tokens have one of the expected kinds. I’m fixing the bugs this technique found in #1341.Gathering of that information is hidden behind a conditional compilation flag because just performing the check of whether we want to record alternative token choices inflicts a 6% performance regression, which doesn’t provide any value except when we are running SwiftParserTest.