Merge pull request #153 from oyvindln/master

 Add benchmarks of regex in release mode

Author: Mark-Simulacrum

collector/README.md

@@ -25,7 +25,10 @@ Follow the standard instructions above, but run like this:
 
 $RUSTC should point to the compiled rustc in stage1 or stage2, i.e.
 `./build/x86_64-unknown-linux-gnu/stage2/bin/rustc`. Stage 2 is likely a little better for
-benchmarking purposes, but not critical.
+benchmarking purposes, but not critical. The rustc binary has to be in a subdirectory of the
+`rustc-perf` root directory for the benchmark run to succeed.
+
+$DATE is a date specified in the [`RFC3339 format`](https://www.ietf.org/rfc/rfc3339.txt).
 
 Once this is done, you can run the site (`./target/release/site $RUSTC_TIMING`) and use the
 comparison page to compare the before/after runs.
@@ -51,4 +54,3 @@ x86_64-unknown-linux-gnu is supported, but the system should trivially expand to
 Will only work for commits that have builds on s3://rust-lang-ci/rustc-builds: these merged after
 rust-lang/rust#38748 (bors sha: 927c55d86b0be44337f37cf5b0a76fb8ba86e06c). Additionally, try builds
 can also be tested, but the process is currently manual.
-

collector/benchmarks/README.md

@@ -10,7 +10,7 @@ many times in succession, applying a different patch each time.
 
 Each benchmark contains a makefile. This makefile defines various
 standard targets. The basic procedure for running a benchmark is defined
-in `process.sh` and looks like this:
+in `../src/bin/rustc-perf-collector/execute.rs` and looks like this:
 
 1. Invoke `make patches` to get a list of patches. This target should
    output a series of patches, or else output nothing. If it outputs nothing,
@@ -19,17 +19,17 @@ in `process.sh` and looks like this:
    benchmarks, it would output a series of patch names like `@000-base` or
    `@010-foo`. By convention these begin with an `@` and a number that increments
    by 10 (000, 010, 020) --- this leaves room for inserting steps later. They
-   then have a brief description. 
+   then have a brief description.
 2. To run the benchmark, you then invoke `make all$PATCH` for each patch.
    So, for non-incremental tests, this is just `make all`. For incremental tests,
    it would be something like `make all@000-base`, followed by `make all@010-foo`, etc.
    - Each stage should run cargo to build the target crate. The basic
      invocation should be something like this, where `CARGO_OPTS` and
-     `CARGO_RUSTC_OPTS` are variables set by `process.sh`:
+     `CARGO_RUSTC_OPTS` are variables set by `rustc-perf-collector`:
      - `cargo rustc -p target_crate $CARGO_OPTS -- $CARGO_RUSTC_OPTS`
    - These `CARGO_RUSTC_OPTS` will include `-Z time-passes`, causing the compiler to
-     emit timing information. This information is saved (by `process.sh`) into an intermediate
-     file. It can later be scraped.
+     emit timing information. This information is saved (by `rustc-perf-collector`) into an
+     intermediate file. It can later be scraped.
 3. After executing all the `make all@...` targets, `make touch` is used to reset the state
    to the initial state. We can then repeat step 2 as many times as desired.
 4. Finally, `make clean` is used to restore everything and remove any temporary data.
@@ -42,4 +42,3 @@ Local runs comparing two different compilers can be performed with
 > `./compare.py <rustc1> <rustc2>`
 
 This is useful when evaluating compile-time optimizations.
-

collector/benchmarks/regex-opt-0.1.80/.gitignore

@@ -0,0 +1,6 @@
+target
+Cargo.lock
+bench-log
+.*.swp
+wiki
+tags

collector/benchmarks/regex-opt-0.1.80/.travis.yml

@@ -0,0 +1,53 @@
+language: rust
+rust:
+  - 1.3.0
+  - stable
+  - beta
+  - nightly
+sudo: false
+script:
+  - cargo build --verbose
+  - if [ "$TRAVIS_RUST_VERSION" = "nightly" ]; then
+      cargo build --verbose --manifest-path=regex-debug/Cargo.toml;
+      RUSTFLAGS="-C target-feature=+ssse3" cargo test --verbose --features 'simd-accel pattern';
+    else
+      travis_wait cargo test --verbose;
+    fi
+  - ./run-shootout-test
+  - cargo doc --verbose
+  - cargo test --verbose --manifest-path=regex-syntax/Cargo.toml
+  - cargo doc --verbose --manifest-path=regex-syntax/Cargo.toml
+  - if [ "$TRAVIS_RUST_VERSION" = "nightly" ]; then
+      (cd regex-capi && cargo build --verbose);
+      (cd regex-capi/ctest && ./compile && LD_LIBRARY_PATH=../target/debug ./test);
+      (cd regex-capi/examples && ./compile && LD_LIBRARY_PATH=../target/release ./iter);
+      (cd bench && travis_wait ./run rust);
+      (cd bench && travis_wait ./run rust-bytes --no-run);
+      (cd bench && travis_wait ./run pcre1 --no-run);
+      (cd bench && travis_wait ./run onig --no-run);
+      travis_wait cargo test --verbose --manifest-path=regex_macros/Cargo.toml;
+    fi
+addons:
+  apt:
+    packages:
+      - libcurl4-openssl-dev
+      - libelf-dev
+      - libdw-dev
+      - binutils-dev
+after_success: |
+  [ $TRAVIS_BRANCH = master ] &&
+  [ $TRAVIS_PULL_REQUEST = false ] &&
+  [ $TRAVIS_RUST_VERSION = nightly ] &&
+  echo '<meta http-equiv=refresh content=0;url=regex/index.html>' > target/doc/index.html &&
+  pip install ghp-import --user $USER &&
+  $HOME/.local/bin/ghp-import -n target/doc &&
+  git push -qf https://${TOKEN}@github.com/${TRAVIS_REPO_SLUG}.git gh-pages &&
+  wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
+    tar xzf master.tar.gz && mkdir kcov-master/build && cd kcov-master/build && cmake .. && make && make install DESTDIR=../tmp && cd ../.. &&
+  PATH="./kcov-master/tmp/usr/local/bin:$PATH" ./run-kcov --coveralls-id $TRAVIS_JOB_ID
+env:
+  global:
+    secure: VvIrYRW/a8FmBA61hn1bDrqWwR92tANOT6PCeLYd9A9ViQrN07PE6uGsnd9iwr8itck10Ctl1mThZYUkK8BDFEmlvSxOFJ/Po5eRe6A1CYuFF40zizJ+3NllVkN20kwoQDe0kxwZVDGO9Yi1icHUrbRPWDfS+1tfobO/UT8Dlng=
+notifications:
+  email:
+    on_success: never

collector/benchmarks/regex-opt-0.1.80/030-compile_one.diff

@@ -0,0 +1,13 @@
+diff --git a/regex-0.1.80/src/compile.rs b/regex-0.1.80/src/compile.rs
+index 9db743f..ef1948e 100644
+--- a/regex-0.1.80/src/compile.rs
++++ b/regex-0.1.80/src/compile.rs
+@@ -137,6 +137,8 @@ impl Compiler {
+     }
+ 
+     fn compile_one(mut self, expr: &Expr) -> result::Result<Program, Error> {
++        {} // @030
++
+         // If we're compiling a forward DFA and we aren't anchored, then
+         // add a `.*?` before the first capture group.
+         // Other matching engines handle this by baking the logic into the

collector/benchmarks/regex-opt-0.1.80/040-is_valid_cap_letter.diff

@@ -0,0 +1,12 @@
+diff --git a/regex-0.1.80/src/expand.rs b/regex-0.1.80/src/expand.rs
+index 9bea703..3b6ae94 100644
+--- a/regex-0.1.80/src/expand.rs
++++ b/regex-0.1.80/src/expand.rs
+@@ -84,6 +84,7 @@ fn find_cap_ref(mut replacement: &[u8]) -> Option<CaptureRef> {
+ }
+ 
+ fn is_valid_cap_letter(b: &u8) -> bool {
++    { }
+     match *b {
+         b'0' ... b'9' | b'a' ... b'z' | b'A' ... b'Z' | b'_' => true,
+         _ => false,

collector/benchmarks/regex-opt-0.1.80/050-expand.diff

@@ -0,0 +1,12 @@
+diff --git a/regex-0.1.80/src/expand.rs b/regex-0.1.80/src/expand.rs
+index 9bea703..a28b82d 100644
+--- a/regex-0.1.80/src/expand.rs
++++ b/regex-0.1.80/src/expand.rs
+@@ -5,6 +5,7 @@ use memchr::memchr;
+ use bytes::Captures;
+ 
+ pub fn expand(caps: &Captures, mut replacement: &[u8], dst: &mut Vec<u8>) {
++    { }
+     while !replacement.is_empty() {
+         match memchr(b'$', replacement) {
+             None => break,

collector/benchmarks/regex-opt-0.1.80/060-Compiler-new.diff

@@ -0,0 +1,12 @@
+diff --git a/regex-0.1.80/src/compile.rs b/regex-0.1.80/src/compile.rs
+index 9db743f..fb812ae 100644
+--- a/regex-0.1.80/src/compile.rs
++++ b/regex-0.1.80/src/compile.rs
+@@ -54,6 +54,7 @@ impl Compiler {
+     ///
+     /// Various options can be set before calling `compile` on an expression.
+     pub fn new() -> Self {
++        {}
+         Compiler {
+             insts: vec![],
+             compiled: Program::new(),

collector/benchmarks/regex-opt-0.1.80/060-reverse.diff

@@ -0,0 +1,12 @@
+diff --git a/regex-0.1.80/src/compile.rs b/regex-0.1.80/src/compile.rs
+index 9db743f..4e56c2d 100644
+--- a/regex-0.1.80/src/compile.rs
++++ b/regex-0.1.80/src/compile.rs
+@@ -114,6 +114,7 @@ impl Compiler {
+     /// When set, the machine returned is suitable for matching text in
+     /// reverse. In particular, all concatenations are flipped.
+     pub fn reverse(mut self, yes: bool) -> Self {
++        {}
+         self.compiled.is_reverse = yes;
+         self
+     }

collector/benchmarks/regex-opt-0.1.80/070-BYTE_FREQUENCIES.diff

@@ -0,0 +1,13 @@
+diff --git a/regex-0.1.80/src/freqs.rs b/regex-0.1.80/src/freqs.rs
+index 92bafc1..6eb5799 100644
+--- a/regex-0.1.80/src/freqs.rs
++++ b/regex-0.1.80/src/freqs.rs
+@@ -12,7 +12,7 @@
+ // edit directly
+ 
+ pub const BYTE_FREQUENCIES: [u8; 256] = [
+-     55, // '\x00'
++     54+1, // '\x00'
+      52, // '\x01'
+      51, // '\x02'
+      50, // '\x03'

collector/benchmarks/regex-opt-0.1.80/080-SparseSet.diff

@@ -0,0 +1,20 @@
+diff --git a/regex-0.1.80/src/sparse.rs b/regex-0.1.80/src/sparse.rs
+index 34c05e7..ef5188e 100644
+--- a/regex-0.1.80/src/sparse.rs
++++ b/regex-0.1.80/src/sparse.rs
+@@ -16,13 +16,13 @@ pub struct SparseSet {
+     /// Dense contains the instruction pointers in the order in which they
+     /// were inserted. Accessing elements >= self.size is illegal.
+     dense: Vec<usize>,
++    /// The number of elements in the set.
++    size: usize,
+     /// Sparse maps instruction pointers to their location in dense.
+     ///
+     /// An instruction pointer is in the set if and only if
+     /// sparse[ip] < size && ip == dense[sparse[ip]].
+     sparse: Vec<usize>,
+-    /// The number of elements in the set.
+-    size: usize,
+ }
+ 
+ impl SparseSet {

collector/benchmarks/regex-opt-0.1.80/090-Job.diff

@@ -0,0 +1,14 @@
+diff --git a/regex-0.1.80/src/backtrack.rs b/regex-0.1.80/src/backtrack.rs
+index 3c06254..4b72fd4 100644
+--- a/regex-0.1.80/src/backtrack.rs
++++ b/regex-0.1.80/src/backtrack.rs
+@@ -82,8 +82,8 @@ impl Cache {
+ /// stack to do it.
+ #[derive(Clone, Copy, Debug)]
+ enum Job {
+-    Inst { ip: InstPtr, at: InputAt },
+     SaveRestore { slot: usize, old_pos: Option<usize> },
++    Inst { ip: InstPtr, at: InputAt },
+ }
+ 
+ impl<'a, 'm, 'r, 's, I: Input> Bounded<'a, 'm, 'r, 's, I> {

collector/benchmarks/regex-opt-0.1.80/CHANGELOG.md

@@ -0,0 +1,128 @@
+0.1.80
+======
+* [PR #292](https://github.com/rust-lang-nursery/regex/pull/292):
+  Fixes bug #291, which was introduced by PR #290.
+
+0.1.79
+======
+* Require regex-syntax 0.3.8.
+
+0.1.78
+======
+* [PR #290](https://github.com/rust-lang-nursery/regex/pull/290):
+  Fixes bug #289, which caused some regexes with a certain combination
+  of literals to match incorrectly.
+
+0.1.77
+======
+* [PR #281](https://github.com/rust-lang-nursery/regex/pull/281):
+  Fixes bug #280 by disabling all literal optimizations when a pattern
+  is partially anchored.
+
+0.1.76
+======
+* Tweak criteria for using the Teddy literal matcher.
+
+0.1.75
+======
+* [PR #275](https://github.com/rust-lang-nursery/regex/pull/275):
+  Improves match verification performance in the Teddy SIMD searcher.
+* [PR #278](https://github.com/rust-lang-nursery/regex/pull/278):
+  Replaces slow substring loop in the Teddy SIMD searcher with Aho-Corasick.
+* Implemented DoubleEndedIterator on regex set match iterators.
+
+0.1.74
+======
+* Release regex-syntax 0.3.5 with a minor bug fix.
+* Fix bug #272.
+* Fix bug #277.
+* [PR #270](https://github.com/rust-lang-nursery/regex/pull/270):
+  Fixes bugs #264, #268 and an unreported where the DFA cache size could be
+  drastically under estimated in some cases (leading to high unexpected memory
+  usage).
+
+0.1.73
+======
+* Release `regex-syntax 0.3.4`.
+* Bump `regex-syntax` dependency version for `regex` to `0.3.4`.
+
+0.1.72
+======
+* [PR #262](https://github.com/rust-lang-nursery/regex/pull/262):
+  Fixes a number of small bugs caught by fuzz testing (AFL).
+
+0.1.71
+======
+* [PR #236](https://github.com/rust-lang-nursery/regex/pull/236):
+  Fix a bug in how suffix literals were extracted, which could lead
+  to invalid match behavior in some cases.
+
+0.1.70
+======
+* [PR #231](https://github.com/rust-lang-nursery/regex/pull/231):
+  Add SIMD accelerated multiple pattern search.
+* [PR #228](https://github.com/rust-lang-nursery/regex/pull/228):
+  Reintroduce the reverse suffix literal optimization.
+* [PR #226](https://github.com/rust-lang-nursery/regex/pull/226):
+  Implements NFA state compression in the lazy DFA.
+* [PR #223](https://github.com/rust-lang-nursery/regex/pull/223):
+  A fully anchored RegexSet can now short-circuit.
+
+0.1.69
+======
+* [PR #216](https://github.com/rust-lang-nursery/regex/pull/216):
+  Tweak the threshold for running backtracking.
+* [PR #217](https://github.com/rust-lang-nursery/regex/pull/217):
+  Add upper limit (from the DFA) to capture search (for the NFA).
+* [PR #218](https://github.com/rust-lang-nursery/regex/pull/218):
+  Add rure, a C API.
+
+0.1.68
+======
+* [PR #210](https://github.com/rust-lang-nursery/regex/pull/210):
+  Fixed a performance bug in `bytes::Regex::replace` where `extend` was used
+  instead of `extend_from_slice`.
+* [PR #211](https://github.com/rust-lang-nursery/regex/pull/211):
+  Fixed a bug in the handling of word boundaries in the DFA.
+* [PR #213](https://github.com/rust-lang-nursery/regex/pull/213):
+  Added RE2 and Tcl to the benchmark harness. Also added a CLI utility from
+  running regexes using any of the following regex engines: PCRE1, PCRE2,
+  Oniguruma, RE2, Tcl and of course Rust's own regexes.
+
+0.1.67
+======
+* [PR #201](https://github.com/rust-lang-nursery/regex/pull/201):
+  Fix undefined behavior in the `regex!` compiler plugin macro.
+* [PR #205](https://github.com/rust-lang-nursery/regex/pull/205):
+  More improvements to DFA performance. Competitive with RE2. See PR for
+  benchmarks.
+* [PR #209](https://github.com/rust-lang-nursery/regex/pull/209):
+  Release 0.1.66 was semver incompatible since it required a newer version
+  of Rust than previous releases. This PR fixes that. (And `0.1.66` was
+  yanked.)
+
+0.1.66
+======
+* Speculative support for Unicode word boundaries was added to the DFA. This
+  should remove the last common case that disqualified use of the DFA.
+* An optimization that scanned for suffix literals and then matched the regular
+  expression in reverse was removed because it had worst case quadratic time
+  complexity. It was replaced with a more limited optimization where, given any
+  regex of the form `re$`, it will be matched in reverse from the end of the
+  haystack.
+* [PR #202](https://github.com/rust-lang-nursery/regex/pull/202):
+  The inner loop of the DFA was heavily optimized to improve cache locality
+  and reduce the overall number of instructions run on each iteration. This
+  represents the first use of `unsafe` in `regex` (to elide bounds checks).
+* [PR #200](https://github.com/rust-lang-nursery/regex/pull/200):
+  Use of the `mempool` crate (which used thread local storage) was replaced
+  with a faster version of a similar API in @Amanieu's `thread_local` crate.
+  It should reduce contention when using a regex from multiple threads
+  simultaneously.
+* PCRE2 JIT benchmarks were added. A benchmark comparison can be found
+  [here](https://gist.github.com/anonymous/14683c01993e91689f7206a18675901b).
+  (Includes a comparison with PCRE1's JIT and Oniguruma.)
+* A bug where word boundaries weren't being matched correctly in the DFA was
+  fixed. This only affected use of `bytes::Regex`.
+* [#160](https://github.com/rust-lang-nursery/regex/issues/160):
+  `Captures` now has a `Debug` impl.