hgdemandimport: apply lazy module loading to sys.meta_path finders
Python's `sys.meta_path` finders are the primary objects whose job it
is to find a module at import time. When `import` is called, Python
iterates objects in this list and calls `o.find_spec(...)` to find
a `ModuleSpec` (or None if the module couldn't be found by that
finder). If no meta path finder can find a module, import fails.
One of the default meta path finders is `PathFinder`. Its job is to
import modules from the filesystem and is probably the most important
importer. This finder looks at `sys.path` and `sys.path_hooks` to do
its job.
The `ModuleSpec` returned by `MetaPathImporter.find_spec()` has a
`loader` attribute, which defines the concrete module loader to use.
`sys.path_hooks` is a hook point for teaching `PathFinder` to
instantiate custom loader types.
Previously, we injected a custom `sys.path_hook` that told `PathFinder`
to wrap the default loaders with a loader that creates a module object
that is lazy.
This approach worked. But its main limitation was that it only applied
to the `PathFinder` meta path importer. There are other meta path
importers that are registered. And in the case of PyOxidizer loading
modules from memory, `PathFinder` doesn't come into play since
PyOxidizer's own meta path importer was handling all imports.
This commit changes our approach to lazy module loading by proxying
all meta path importers. Specifically, we overload the `find_spec()`
method to swap in a wrapped loader on the `ModuleSpec` before it
is returned. The end result of this is all meta path importers should
be lazy.
As much as I would have loved to utilize .__class__ manipulation to
achieve this, some meta path importers are implemented in C/Rust
in such a way that they cannot be monkeypatched. This is why we
use __getattribute__ to define a proxy.
Also, this change could theoretically open us up to regressions in
meta path importers whose loader is creating module objects which
can't be monkeypatched. But I'm not aware of any of these in the
wild. So I think we'll be safe.
According to hyperfine, this change yields a decent startup time win of
5-6ms:
```
Benchmark #1: ~/.pyenv/versions/3.6.10/bin/python ./hg version
Time (mean ± σ): 86.8 ms ± 0.5 ms [User: 78.0 ms, System: 8.7 ms]
Range (min … max): 86.0 ms … 89.1 ms 50 runs
Time (mean ± σ): 81.1 ms ± 2.7 ms [User: 74.5 ms, System: 6.5 ms]
Range (min … max): 77.8 ms … 90.5 ms 50 runs
Benchmark #2: ~/.pyenv/versions/3.7.6/bin/python ./hg version
Time (mean ± σ): 78.9 ms ± 0.6 ms [User: 70.2 ms, System: 8.7 ms]
Range (min … max): 78.1 ms … 81.2 ms 50 runs
Time (mean ± σ): 73.4 ms ± 0.6 ms [User: 65.3 ms, System: 8.0 ms]
Range (min … max): 72.4 ms … 75.7 ms 50 runs
Benchmark #3: ~/.pyenv/versions/3.8.1/bin/python ./hg version
Time (mean ± σ): 78.1 ms ± 0.6 ms [User: 70.2 ms, System: 7.9 ms]
Range (min … max): 77.4 ms … 80.9 ms 50 runs
Time (mean ± σ): 72.1 ms ± 0.4 ms [User: 64.4 ms, System: 7.6 ms]
Range (min … max): 71.4 ms … 74.1 ms 50 runs
```
Differential Revision: https://phab.mercurial-scm.org/D7954
hgdemandimport: disable on Python 3.5
The demand importer functionality isn't working at all on Python 3.5.
I'm not sure what's wrong.
Since it isn't working, let's disable it completely.
```
$ HGRCPATH= hyperfine -w 1 -r 50 -- "~/.pyenv/versions/3.5.9/bin/python ./hg version" \
"HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version"
Benchmark #1: ~/.pyenv/versions/3.5.9/bin/python ./hg version
Time (mean ± σ): 163.7 ms ± 2.2 ms [User: 148.5 ms, System: 15.7 ms]
Range (min … max): 161.0 ms … 170.2 ms 50 runs
Benchmark #2: HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version
Time (mean ± σ): 164.3 ms ± 1.4 ms [User: 148.2 ms, System: 16.6 ms]
Range (min … max): 161.4 ms … 169.8 ms 50 runs
```
Differential Revision: https://phab.mercurial-scm.org/D7953
py3: suppress unraisable exceptions in test-worker.t
Python 3.8 calls sys.unraisablehook when an unraisable
exception is encountered. The default behavior is to print a
warning.
test-worker.t was triggering this hook due to a race between
a newly forked process exiting and that process's
_os.register_at_fork handlers running. I was seeing the
stdlib's random module in the stack re-seeding itself. Although
there could be other after-fork handlers in the mix.
This commit defines sys.unraisablehook to effectively no-op.
This suppresses the warning and makes test output on Python 3.8
consistent with prior versions. test-worker.t now passes on
Python 3.8.
Differential Revision: https://phab.mercurial-scm.org/D7949
rust: add a README
In particular to explain how to build any of the rust. It's neither
obvious, nor easy to find out, nor easy to determine if you did it
right without some documentation.
Differential Revision: https://phab.mercurial-scm.org/D7952
rust: move hgcli's README out of the way
My understanding is that it's not meant to be used in the current
form.
Differential Revision: https://phab.mercurial-scm.org/D7951
verify: avoid spurious integrity warnings in verbose mode (
issue6172)
The issue seems to revolve around renames in filtered commits, and only occurred
in verbose mode. The problem occurs in the `# check renames` stage, around line
577. Without using the unfiltered repo, this test would have printed:
$ hg verify -v
repository uses revlog format 1
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
foo@25: checking rename of
71ec0570c325: filtered revision '25'
foobar@26: checking rename of
1b549296015b: filtered revision '26'
checked 28 changesets with 16 changes to 11 files
2 integrity errors encountered!
(first damaged changeset appears to be 25)
[1]
Differential Revision: https://phab.mercurial-scm.org/D7950
py3: glob over exception in test-check-py3-compat.t
Python 3.6+ raise ModuleNotFoundError and older versions raise
ImportError. Glob over the exception differences.
For whatever reason, we were already doing this for one failure.
But not all occurrences of ModuleNotFoundError were changed.
Who knows.
This test should now pass on all Python versions (although I didn't
check Windows).
Differential Revision: https://phab.mercurial-scm.org/D7939
py3: string normalization and I/O tweaks in test-lfs.t
The print was inserting b'' on Python 3. In addition, since we
weren't writing to the ui instance (which isn't readily available
in this function), output order could get mixed up.
We add some pycompat casts and a stdout flush to make the test
happy on all Python versions.
Differential Revision: https://phab.mercurial-scm.org/D7938
help: minor copy editing to the `config.format` section
Differential Revision: https://phab.mercurial-scm.org/D7936
changectx: mark parent of changesets as non filtered
If a node is not filtered, its parents cannot be filtered.
Differential Revision: https://phab.mercurial-scm.org/D7502
changectx: use unfiltered changelog to walk ancestors in annotate
Since we are only walking ancestors, it is safe to use an unfiltered repository.
(Because if the original rev is not filtered, none of its ancestors will be).
Differential Revision: https://phab.mercurial-scm.org/D7501
localrepo: also fast past the parents of working copies parents
There are descent odds that they will be needed too. So we also cache and
fastpath them.
Differential Revision: https://phab.mercurial-scm.org/D7498
localrepo: recognize trivial request for '.'
Same logic as for `null`, this is a command request and skipping the revset
logic can avoid triggering the changelog filtering logic.
Differential Revision: https://phab.mercurial-scm.org/D7495
localrepo: fastpath access to "."
"." is just an alias for `p1(wdir())`, let us handle it that way.
Differential Revision: https://phab.mercurial-scm.org/D7494
localrepo: also fastpath access to working copy parents when possible
If the filter level guarantee that the working copy parents will be visible, we
allow fast path access to them. With this change multiple commands can now run
without triggering filtering.
After using the quick access mechanism introduced, the whole series results in
pretty good performance gain:
```
All benchmarks:
before after ratio
[
8e095512] [
36b2f659]
- 711±0.8ms 60.7±0.2ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 712±0.8ms 61.6±0.2ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 690±1ms 93.5±0.3ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 688±1ms 93.8±0.3ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 714±1ms 60.7±0.8ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 713±1ms 60.9±0.3ms 0.09 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 689±1ms 93.7±0.2ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 687±2ms 92.8±0.2ms 0.14 simple_command.read.diff.empty.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 799±2ms 98.1±0.6ms 0.12 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 800±0.8ms 100.0±0.4ms 0.12 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 711±0.9ms 111±0.2ms 0.16 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 711±1ms 112±0.3ms 0.16 simple_command.read.export.bare.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 760±1ms 59.8±0.1ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 763±2ms 62.2±0.3ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 689±1ms 93.1±0.3ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 688±1ms 94.3±0.3ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 1) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 763±1ms 60.1±0.2ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 763±1ms 62.1±0.4ms 0.08 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py2.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
- 689±0.8ms 93.2±0.2ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYc-HGWITHRUSTEXTcpython]
- 687±0.9ms 94.1±0.3ms 0.14 simple_command.read.status.wc_clean.default.time_bench('mercurial-filtered-2019-11-22', 'zstd', 'default', True, True, True, True, True, 2) [citrea/virtualenv-py3.7-pyyaml-HGMODULEPOLICYrust+c-HGWITHRUSTEXTcpython]
```
Differential Revision: https://phab.mercurial-scm.org/D7492
examples: refer to nightly rustfmt in Windows-compatible way
Thanks to Jun Wu for the tip. I found that the new form also gave
better error messages when the nightly rustfmt wasn't installed (it
told me which command to run instead of just saying "error: not a
file: <some path>").
Differential Revision: https://phab.mercurial-scm.org/D7911