annotate relnotes/6.3 @ 50338:81c7d04f4722 stable

match: match explicit file using a set

The matcher as all the logic to do quick comparison against explicit patterns,
however the pattern matcher was shadowing the code using that set and used the
compiled regex pattern in all cases, which is quite slow. We restore the usage
of the set based matching to boost performance.

Building the regexp is still consuming a large amount of time (actually, the
majority of the time), which is still silly. Maybe using re2 would help that,
but this is a quest for another adventure.

Another path to improve this is to have a pattern type dedicated to match the
exact path to a file only (not a directory). This pattern could use the set
matching only and be skipped in the regex all together.


Benchmarks
==========

In the following benchmark we are comparing the `hg cat` and `hg files` run
time when matching against all files in the repository. They are
run:
 - without the rust extensions
 - with the standard python engine (so without re2)


Performance improvement in this series
--------------------------------------

###### hg files ###############################################################
### mercurial-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:    0.230092 seconds
prev-changeset:    0.230069 seconds
this-changeset:    0.211425 seconds    (-8.36%)
### mercurial-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset:    0.234235 seconds
prev-changeset:    0.231165 seconds    (-1.38%)
this-changeset:    0.212300 seconds    (-9.43%)

### pypy-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:    0.613567 seconds
prev-changeset:    0.616799 seconds
this-changeset:    0.510852 seconds   (-16.82%)
### pypy-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset:    0.801880 seconds
prev-changeset:    0.616393 seconds   (-23.22%)
this-changeset:    0.511903 seconds   (-36.23%)

### netbeans-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:   21.541828 seconds
prev-changeset:   21.586773 seconds
this-changeset:   13.648347 seconds   (-36.76%)
### netbeans-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset:  172.759857 seconds
prev-changeset:   21.908197 seconds   (-87.32%)
this-changeset:   13.945110 seconds   (-91.93%)

### mozilla-central-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:   62.474221 seconds
prev-changeset:   61.279490 seconds    (-1.22%)
this-changeset:   29.529469 seconds   (-52.40%)
### mozilla-central-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 1364.180218 seconds
prev-changeset:   62.473549 seconds   (-95.40%)
this-changeset:   30.625249 seconds   (-97.75%)

###### hg cat #################################################################
### mercurial-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:    0.764407 seconds
prev-changeset:    0.763883 seconds
this-changeset:    0.737326 seconds    (-3.68%)
### mercurial-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset:    0.768924 seconds
prev-changeset:    0.765848 seconds
this-changeset:    0.174d0b seconds    (-4.44%)

### pypy-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:    2.065220 seconds
prev-changeset:    2.070498 seconds
this-changeset:    1.939482 seconds    (-6.08%)
### pypy-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset:    2.276388 seconds
prev-changeset:    2.069197 seconds    (-9.15%)
this-changeset:    1.931746 seconds   (-15.19%)

### netbeans-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:   40.967983 seconds
prev-changeset:   41.392423 seconds
this-changeset:   32.181681 seconds   (-22.20%)
### netbeans-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset:  216.388709 seconds
prev-changeset:   41.648689 seconds   (-80.88%)
this-changeset:   32.580817 seconds   (-85.04%)

### mozilla-central-2018-08-01-zstd-sparse-revlog ### sorted
base-changeset:  105.228510 seconds
prev-changeset:  103.315670 seconds    (-1.23%)
this-changeset:   69.416118 seconds   (-33.64%)
### mozilla-central-2018-08-01-zstd-sparse-revlog ### shuffled
base-changeset: 1448.722784 seconds
prev-changeset:  104.369358 seconds   (-92.80%)
this-changeset:   70.554789 seconds   (-95.13%)

Different way to list the same data with this revision
------------------------------------------------------

###### hg files ###############################################################
### mercurial-2018-08-01-zstd-sparse-revlog
root:            0.119182 seconds
glob:            0.120697 seconds    (+1.27%)
sorted:          0.211425 seconds   (+77.40%)
shuffled:        0.212300 seconds   (+78.13%)
### pypy-2018-08-01-zstd-sparse-revlog
root:            0.121986 seconds
glob:            0.124822 seconds    (+2.32%)
sorted:          0.510852 seconds  (+318.78%)
shuffled:        0.511903 seconds  (+319.64%)
### netbeans-2018-08-01-zstd-sparse-revlog
root:            0.173984 seconds
glob:            0.227203 seconds   (+30.59%)
sorted:         13.648347 seconds (+7744.59%)
shuffled:       13.945110 seconds (+7915.16%)
### mozilla-central-2018-08-01-zstd-sparse-revlog
root:            0.366463 seconds
glob:            0.491030 seconds   (+33.99%)
sorted:         29.529469 seconds (+7957.96%)
shuffled:       30.625249 seconds (+8256.97%)

###### hg cat #################################################################
### mercurial-2018-08-01-zstd-sparse-revlog
glob:            0.647471 seconds
root:            0.643120 seconds
shuffled:        0.174d0b seconds   (+13.92%)
sorted:          0.737326 seconds   (+13.88%)
### mozilla-central-2018-08-01-zstd-sparse-revlog
glob:           40.596983 seconds
root:           40.129136 seconds
shuffled:       70.554789 seconds   (+73.79%)
sorted:         69.416118 seconds   (+70.99%)
### netbeans-2018-08-01-zstd-sparse-revlog
glob:           18.777924 seconds
root:           18.613905 seconds
shuffled:       32.580817 seconds   (+73.51%)
sorted:         32.181681 seconds   (+71.38%)
### pypy-2018-08-01-zstd-sparse-revlog
glob:            1.555319 seconds
root:            1.536534 seconds
shuffled:        1.931746 seconds   (+24.20%)
sorted:          1.939482 seconds   (+24.70%)

= Mercurial 6.3.3 =

 * tests: filter out PEP 657 error locations in tracebacks (issue6780)
 * tests: optional PEP 657 error location in test-extension.t (issue6781)
 * tests: optional PEP 657 error location in test-lfs-serve-access.t (issue6782)
 * histedit: byteify the help for the multifold action
 * sparse: fix a py2 based usage of `map()`
 * convert: stop passing str to the dateutil API in darcs
 * convert: turn the last str regex into bytes in cvs.py (issue6789)
 * convert: change socket mode from b'r+' to 'rwb' in cvs.py (issue6789)
 * convert: replace repr() by pycompat.byterepr() in cvsps.py (issue6789)
 * tests: os module is frozen in Python 3.11 (issue6786)
 * hgweb: unbyteify the 100-continue check
 * resourceutil: start using importlib.resources.files() when possible
 * revset: the `random` sort should not depend on sys.maxsize (issue6770)
 * tests: make sure pygments can detect python script without extension
 * convert: brz 3.3.0 moved NoSuchFile exception to breezy.transport
 * tests: pygments 2.14+ highlight whitespace in python code
 * hghave: make different has_pyoxidizer functions have different names
 * hghave: refactor checks for pygments versions using checkvers()
 * rust-narrow: fix loop that never loops
 * scmutil: make checknewlabel() allow "_" in otherwise numeric names (issue6737)
 * bundlerepo: enforce the requirements declared by the underlying repository
 * setup: make the version computation process more resistant
 * fix: add more information to the debug output
 * tag: disallow tagging the working directory
 * dirstate: handle missing backup file on restoration
 * dirstate-v2: complain early on docket name collision
 * rust: upgrade minimum `rayon` dependency
 * setup: support building from an ongoing merge
 * rust: add debug log about skipping dirstate update
 * rust-dirstate: trace append/no append to help debugging
 * dirstate-v2: don't mmap the data file when on NFS
 * run-tests: make it possible to nest conditionals
 * dirstate: add some debug output when writing the dirstate
 * testing: introduce util function to synchronize concurrent commands on files
 * dirstate: add a way to test races happening during status
 * dirstate: use more than a bool to control append behavior
 * dirstate-v2: add devel config option to control write behavior
 * rhg: fix race when a fixup file is deleted on disk
 * rhg: fix race when an ambiguous file is deleted on disk
 * dirstate: deal with read-race for pure python code
 * dirstate: deal with read-race for python code using rust object
 * dirstate: deal with read-race for pure rust code path (rhg)
 * dirstate: set identity whenever we read the dirstate's v2 docket
 * rust-dirstate-v2: don't write dirstate if data file has changed
 * rhg: remember the inode of .hg/dirstate

= Mercurial 6.3.2 =

 * [ecfc84b956a8] tests: expect the message from 1baf0fffd82f in test-hghave.t (issue6762)
 * [5c095119bff4] tests: add the missing space to test-hghave.t (issue6762)
 * [2c346c1c75ec] tests: use an all too familiar executable in test-run-tests.t (issue6661)
 * [13c0e3b4fd35] tests: use `test -f` instead of `ls` to see if a file is present (issue6662)
 * [8ced4ca30ea1] bisect: correct message about aborting an in-progress bisect (issue6527)
 * filemerge: fix crash when using filesets in [partial-merge-tools]
 * help: fix a py3 error interpolating Set into b'%s'
 * match: make the FLAG_RE pattern a raw string
 * python-compat: adapt to Python 3.11 BC breakage with `random.sample`
 * rust-status: fix thread count ceiling
 * hg: show the correct message when cloning an LFS repo with extension disabled
 * extensions: process disabled external paths when `hgext` package is in-memory
 * emitrevision: consider ancestors revision to emit as available base
 * make: add a target for building pyoxidizer tests on macOS
 * run-tests: support --pyoxidized on macOS
 * packaging: add dependencies to the PyOxidizer build on macOS
 * Miscellaneous test fixes

= Mercurial 6.3.1 =

 * memory-usage: fix `hg log --follow --rev R F` space complexity (dcb2581e33be)
 * Improve portability and robustness of test harness
 * hg-core: relax dependencies pinning
 * matcher: fix issues regex flag contained in pattern (issue6759)
 * matcher: do not prepend '.*' to pattern using ^ after flags
 * packaging: refresh dependency hashes (issue6750)
 * cffi: fix a bytes vs str issue on macOS when listing directories
 * commit: properly consider file include and exclude options when closing branch
 * dirstate-v2: do not put the dirstate data file in a transaction

= Mercurial 6.3 =

== New Features ==

 * testlib: add `--raw-sha1` option to `f`
 * rhg: add `config.rhg` helptext
 * config: add alias from `hg help rhg` to `hg help rust`
 * rhg: add a config option to fall back immediately
 * bundle: introduce a --exact option
 * perf-bundle: add a new command to benchmark bundle creation time
 * perf-bundle: accept --rev arguments
 * perf-bundle: accept --type argument
 * perf-unbundle: add a perf command to time the unbundle operation
 * perf: introduce a benchmark for delta-find
 * contrib: add support for rhel9
 * phase-shelve: Implement a 'shelve.store' experimental config
 * debug-delta-find: introduce a quiet mode
 * sort-revset: introduce a `random` variant
 * phase: introduce a dedicated requirement for the `archived` phase
 * rebase: add boolean config item rebase.store-source
 * rhg: make [rhg status -v] work when it needs no extra output
 * rhg: support "!" syntax for disabling extensions
 * rhg: add debugrhgsparse command to help figure out bugs in rhg
 * rhg: add sparse support
 * rhg-status: add support for narrow clones
 * templates: add filter to reverse list
 * contrib: add pull_logger extension
 * revset: handle wdir() in `roots()`
 * revset: handle wdir() in `sort(..., -topo)`
 * rhg: support tweakdefaults
 * rhg: parallellize computation of [unsure_is_modified]

== Default Format Change ==

These changes affect newly created repositories (or new clones) done with
Mercurial 6.3.

== New Experimental Features ==

== Bug Fixes ==

 * shelve: demonstrate that the state is different across platforms (issue6735)
 * shelve: in test for trailing whitespace, strip commit (issue6735)
 * shelve: remove strip and rely on prior state (issue6735)
 * tests: fix http-bad-server expected errors for python 3.10 (issue6643)
 * status: let `--no-copies` override `ui.statuscopies`
 * releasenotes: use re.MULTILINE mode when checking admonitions
 * rhg: fallback to slow path on invalid patterns in hgignore
 * Fix a bunch of leftover str/bytes issues from Python 3 migration
 * keepalive: ensure `close_all()` actually closes all cached connections
 * lfs: fix blob corruption when tranferring with workers on posix
 * lfs: avoid closing connections when the worker doesn't fork
 * dirstate-v2: update constant that wasn't kept in sync
 * dirstate-v2: fix edge case where entries aren't sorted
 * upgrade: no longer keep all revlogs in memory at any point
 * rust-status: save new dircache even if just invalidated
 * dirstate-v2: hash the source of the ignore patterns as well
 * rhg: fallback when encountering ellipsis revisions
 * shelve: handle empty parents and nodestoremove in shelvedstate (issue6748)
 * profile: prevent a crash when line number is unknown
 * tags-fnode-cache: do not repeatedly open the filelog in a loop
 * tags-fnode-cache: skip building a changectx in getfnode
 * rust: create wrapper struct to reduce `regex` contention issues

== Backwards Compatibility Changes ==

 * chg worker processes will now correctly load per-repository configuration
   when given a both a relative `--repository` path and an alternate working
   directory via `--cwd`. A side-effect of this change is that these workers
   will now return an error if hg cannot find the current working directory,
   even when a different directory is specified via `--cwd`.
 * phase: rename the requirement for internal-phase from `internal-phase` to `use-internal-phase` (see 74fb1842f8b962cf03d7cd5b841dbcf2ae065587)

== Internal API Changes ==

== Miscellaneous ==

 * sslutil: use proper attribute to select python 3.7+
 * typing: suppress a few pyi-errors with more recent pytype
 * ci: bump pytype to 2022.03.29
 * bundlespec: add documentation about existing option
 * subrepo: avoid opening console window for non-native subrepos on Windows
 * setup: unconditionally enable the `long-paths-support` option on Windows
 * setup: use the full executable manifest from `python.exe`
 * tests: work around libmagic bug in svn subrepo tests
 * packagelib: use python3 by default
 * Improve `hg bisect` performance
 * perf: properly process formatter option in perf::unbundle
 * compare-disco: miscellaneous display improvements
 * fsmonitor: better compatibility with newer Pythons
 * revlog: finer computation of "issnapshot"
 * rhg: don't fallback if `strip` or `rebase` are activated
 * perf: make perf::bundle compatible before 61ba04693d65
 * perf: make perf::bundle compatible down to 5.2
 * perf-unbundle: improve compatibility
 * run-tests: display the time it took to install Mercurial
 * mergetools: don't let meld open all changed files on startup
 * dirstate-v2: skip evaluation of hgignore regex on cached directories