Skip to content

Parse optionals and repeats without regexes #1826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 15, 2025

Conversation

traviscross
Copy link
Contributor

@traviscross traviscross commented May 15, 2025

Rather than parsing optionals and repeats fully in the recursive descent style, we were using regular expressions to do part of the matching and parsing. That's fine for what it is, but as we think about extending the grammar language surrounding repeats further, it might be more straightforward for this to be parsed in the more usual way. So let's do that. Doing this also results in better and more targeted errors when parsing malformed syntax.

We had been supporting a space between an expression and the optional and repeat sigils ?, *, and + (but not between an expression and the {a..b} ranged repeat syntax). In making this change, we drop this support and adjust the affected productions. We were only using this in a handful of places, and the clarity of the productions seem the same or better by removing these spaces.

We verified that, setting aside the removal of these spaces, the rendered output of the Reference is byte identical before and after this change.

cc @ehuss

@traviscross
Copy link
Contributor Author

The motivation for this refactoring is to next add repeats with separators.

@traviscross traviscross marked this pull request as ready for review May 15, 2025 13:33
@rustbot rustbot added the S-waiting-on-review Status: The marked PR is awaiting review from a maintainer label May 15, 2025
@traviscross traviscross force-pushed the TC/parse-repeats-without-regex branch from 747ca2f to 7e7e973 Compare May 15, 2025 13:36
Rather than parsing optionals and repeats fully in the recursive
descent style, we were using regular expressions to do part of the
matching and parsing.  That's fine for what it is, but as we think
about extending the grammar language surrounding repeats further, it
might be more straightforward for this to be parsed in the more usual
way.  So let's do that.  Doing this also results in better and more
targeted errors when parsing malformed syntax.

We had been supporting a space between an expression and the optional
and repeat sigils `?`, `*`, and `+` (but not between an expression and
the `{a..b}` ranged repeat syntax).  In making this change, we drop
this support and adjust the affected productions.  We were only using
this in a handful of places, and the clarity of the productions seem
the same or better by removing these spaces.

We verified that, setting aside the removal of these spaces, the
rendered output of the Reference is byte identical before and after
this change.
@traviscross traviscross force-pushed the TC/parse-repeats-without-regex branch from 7e7e973 to 3570070 Compare May 15, 2025 14:57
Copy link
Contributor

@ehuss ehuss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ehuss ehuss added this pull request to the merge queue May 15, 2025
Merged via the queue into master with commit c703c8d May 15, 2025
5 checks passed
Zalathar added a commit to Zalathar/rust that referenced this pull request May 20, 2025
Update books

## rust-lang/book

4 commits in d33916341d480caede1d0ae57cbeae23aab23e88..230c68bc1e08f5f3228384a28cc228c81dfbd10d
2025-05-19 14:25:14 UTC to 2025-05-08 21:28:56 UTC

- Chapter 6 from tech review (rust-lang/book#4370)
- Chapter 5 from tech review (rust-lang/book#4359)
- Chapter 4 from tech review (rust-lang/book#4358)
- Chapter 3 from tech review (rust-lang/book#4353)

## rust-lang/reference

12 commits in 387392674d74656f7cb437c05a96f0c52ea8e601..acd0231ebc74849f6a8907b5e646ce86721aad76
2025-05-19 15:41:22 UTC to 2025-05-06 21:36:01 UTC

- Add doc for avx512 target features (rust-lang/reference#1778)
- Parse grammar without regexes (rust-lang/reference#1827)
- Parse optionals and repeats without regexes (rust-lang/reference#1826)
- Fix grammar for `RangePatternBound` regarding literals (rust-lang/reference#1825)
- Fix grammar for `LiteralPattern` regarding `-` (rust-lang/reference#1824)
- Doc: Add the LoongArch stabilized target features (rust-lang/reference#1707)
- Fix naked em-dash (rust-lang/reference#1820)
- Add missing attribute for statement macros (rust-lang/reference#1819)
- Make linked rules are clicked, highlight the color (rust-lang/reference#1817)
- Use the reference grammar for inline assembly (rust-lang/reference#1807)
- Fix typo in introduction (rust-lang/reference#1810)
- Add an example admonition (rust-lang/reference#1812)

## rust-lang/rust-by-example

2 commits in 8a8918c698534547fa8a1a693cb3e7277f0bfb2f..c9d151f9147c4808c77f0375ba3fa5d54443cb9e
2025-05-13 17:49:05 UTC to 2025-05-13 17:48:43 UTC

- fix(docs): standardize on `no_run` attribute for documentation examples (rust-lang/rust-by-example#1929)
- Fix typo in Japanese translation (rust-lang/rust-by-example#1928)
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request May 20, 2025
Rollup merge of rust-lang#141259 - rustbot:docs-update, r=ehuss

Update books

## rust-lang/book

4 commits in d33916341d480caede1d0ae57cbeae23aab23e88..230c68bc1e08f5f3228384a28cc228c81dfbd10d
2025-05-19 14:25:14 UTC to 2025-05-08 21:28:56 UTC

- Chapter 6 from tech review (rust-lang/book#4370)
- Chapter 5 from tech review (rust-lang/book#4359)
- Chapter 4 from tech review (rust-lang/book#4358)
- Chapter 3 from tech review (rust-lang/book#4353)

## rust-lang/reference

12 commits in 387392674d74656f7cb437c05a96f0c52ea8e601..acd0231ebc74849f6a8907b5e646ce86721aad76
2025-05-19 15:41:22 UTC to 2025-05-06 21:36:01 UTC

- Add doc for avx512 target features (rust-lang/reference#1778)
- Parse grammar without regexes (rust-lang/reference#1827)
- Parse optionals and repeats without regexes (rust-lang/reference#1826)
- Fix grammar for `RangePatternBound` regarding literals (rust-lang/reference#1825)
- Fix grammar for `LiteralPattern` regarding `-` (rust-lang/reference#1824)
- Doc: Add the LoongArch stabilized target features (rust-lang/reference#1707)
- Fix naked em-dash (rust-lang/reference#1820)
- Add missing attribute for statement macros (rust-lang/reference#1819)
- Make linked rules are clicked, highlight the color (rust-lang/reference#1817)
- Use the reference grammar for inline assembly (rust-lang/reference#1807)
- Fix typo in introduction (rust-lang/reference#1810)
- Add an example admonition (rust-lang/reference#1812)

## rust-lang/rust-by-example

2 commits in 8a8918c698534547fa8a1a693cb3e7277f0bfb2f..c9d151f9147c4808c77f0375ba3fa5d54443cb9e
2025-05-13 17:49:05 UTC to 2025-05-13 17:48:43 UTC

- fix(docs): standardize on `no_run` attribute for documentation examples (rust-lang/rust-by-example#1929)
- Fix typo in Japanese translation (rust-lang/rust-by-example#1928)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-review Status: The marked PR is awaiting review from a maintainer
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants