packaging: ship only a single binary Debian package We merge the mercurial and mercurial-common binary packages into a single mercurial package. This is essentially to ease installation (and upgrade) using a simple "dpkg -i" command. This also simplifies debian/rules by removing arch (in)dependent cleanups during installation. We have the mercurial binary Breaks: and Replaces: mercurial-common so that the latter will be removed upon upgrade. Also note the change from "override_dh_install" to "override_dh_auto_install" in debian/rules: this is because we do not want "make install" to be run automatically as we need the --install-layout=deb of "setup.py install" (otherwise, files would end up in $DESTDIR/usr/local).

packaging: avoid running bare "make install" in debian/rules We change the "override_dh_install" target to "override_dh_auto_install" in debian/rules (see dh_auto_install(1) for details). This is because we do not want "make install" to be run automatically as we need the --install-layout=deb of "setup.py install". Otherwise, files end up in $DESTDIR/usr/local. At the moment, this is not a problem since files installed in debian/tmp (the default location) are not packed into binary packages (because there are debian/mercurial and debian/mercurial-common directories). This is cleaner to avoid running make more than needed.

packaging: distinguish clean and build steps from install in Debian

packaging: also move Debian .buildinfo file in output directory

packaging: upgrade packages distributed with Windows installers We like to use the latest versions of things. I added pywin32-ctypes to the explicit list of packages so running on !Windows will pull in the dependency.

automation: install Python 2.7.17, 3.7.5, and PyPy 7.2.0 These were all recently released and we should use them in automation.

contrib: install Python 2.7.17 and 3.7.5 in Windows environment These are the latest Python versions and we should use them.

packaging: use /usr/bin/python3 shebang for scripts in Debian "hg" script is already correct because it is handled by setup.py but "hg-ssh" will be rewritten by dh_python into "/usr/bin/python" which is not wanted as we target Python 3. By passing --shebang=/usr/bin/python3 to dh_python3, we force shebangs to be set with this value.

packaging: fix path where .deb files are looked for In builddeb script, the path where .deb files are looked for should simply be $OUTPUTDIR since the previous instruction moves those files there. This fixes "make deb".

packaging: let Debian package depends on sensible-utils This fixes the following Lintian error: E: mercurial-common: missing-depends-on-sensible-utils etc/mercurial/hgrc.d/default-tools.rc

packaging: upgrade Debian packaging to build with Python 3 Also drop the explicit "Depends: python" as debhelper will add it.

py3: decode bytes before logging in run-tests.py Avoids messages like "Found prerequisite b'diff' at b'/usr/bin/diff'" under Python 3.

Added signature for changeset 59338f956109

Added tag 5.2rc0 for changeset 59338f956109

merge default into stable for 5.2 release

grep: update docs to reflect new --all-files default These docs no longer even mention the old weird behaviour that was poorly understood and underutilised. I think this undocumentation effectively deprecates it.

grep: enable all-files by default (BC) This patch deprecates the `--all-files` flag and make the all-files behaviour as default. In test-grep.t, I removed '--all-files' from every command where it was used, to reflect that all-files behaviour is default and there is no change even after the removal. And other changes in test files are because of changed behaviour. Differential Revision: https://phab.mercurial-scm.org/D7000 .. bc:: `hg grep` now searches working copy file contents by default. We recognize this is a significant change from past behavior, but surveys of large bodies of users indicated nobody used (and almost nobody understood) the previous no-flags behavior of `hg grep`. The new behavior aligns with the behavior most users expected (including hg's maintainers), which also happens to be the behavior of `git grep`. Given that the old behavior was confusing to the point of being unusable, we were comfortable changing this behavior.

lfs: dedent documentation section about .hglfs file The leading ".hglfs::" is interpreted as a macro in generated man page and, as it is unknown, the whole section does not render. Also, having the section marked as preformatted (::) makes it render verbatim in HTML, which is not desired as the text contains formatting markers. So we just dedent the section and remove the ".hglfs::" line. The example file is still indented and rendered preformatted.

copies: simplify chain loop This is a mostly gratuitous change to make the code more direct. Differential Revision: https://phab.mercurial-scm.org/D7131

sidedatacopies: only fetch information once for merge Before this change, merge would result in reading the data from revlog twice. With this change, we keep the information in memory until we encounter the other parent. When looking at pypy, I see about 1/3 of the changesets with copy information being merge. Not doing duplicated fetch for them provide a significant speedup. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 0.767042 comb 0.760000 user 0.750000 sys 0.010000 (median of 11) after: ! wall 0.671162 comb 0.670000 user 0.650000 sys 0.020000 (median of 13) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.170169 comb 1.170000 user 1.130000 sys 0.040000 (median of 10) after: ! wall 1.030596 comb 1.040000 user 1.010000 sys 0.030000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.209846 comb 0.200000 user 0.200000 sys 0.000000 (median of 46) after: ! wall 0.170981 comb 0.170000 user 0.170000 sys 0.000000 (median of 56) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013248 comb 0.010000 user 0.010000 sys 0.000000 (median of 223) after: ! wall 0.013295 comb 0.020000 user 0.020000 sys 0.000000 (median of 222) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001672 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001666 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8010) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8007) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.168599 comb 0.160000 user 0.160000 sys 0.000000 (median of 58) after: ! wall 0.133316 comb 0.140000 user 0.140000 sys 0.000000 (median of 73) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.036052 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) after: ! wall 0.032558 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7127

copies: move from a copy on branchpoint to a copy on write approach Before this changes, any branch points results in a copy of the dictionary containing the copy information. This can be very costly for branchy history with few rename information. Instead, we take a "copy on write" approach. Copying the input data only when we are about to update them. In practice we where already doing the copying in half of these case (because `_chain` makes a copy), so we don't add a significant cost here even in the linear case. However the speed up in branchy case is very significant. Here are some timing on the pypy repository. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 1.399863 comb 1.400000 user 1.370000 sys 0.030000 (median of 10) after: ! wall 0.766453 comb 0.770000 user 0.750000 sys 0.020000 (median of 11) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.876748 comb 1.890000 user 1.870000 sys 0.020000 (median of 10) after: ! wall 1.167223 comb 1.170000 user 1.150000 sys 0.020000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.242457 comb 0.240000 user 0.240000 sys 0.000000 (median of 39) after: ! wall 0.211476 comb 0.210000 user 0.210000 sys 0.000000 (median of 45) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013193 comb 0.020000 user 0.020000 sys 0.000000 (median of 224) after: ! wall 0.013290 comb 0.010000 user 0.010000 sys 0.000000 (median of 222) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001673 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001677 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 8023) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 7997) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.201898 comb 0.210000 user 0.200000 sys 0.010000 (median of 48) after: ! wall 0.167415 comb 0.170000 user 0.160000 sys 0.010000 (median of 58) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.036820 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) after: ! wall 0.035797 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) The extra cost in the linear case can be reclaimed later with some extra logic. Differential Revision: https://phab.mercurial-scm.org/D7124

copies: compute the exact set of revision to walk This change make the code clearer by removing the revision queue. It comes without very noticeable performance impact. However the simpler code will be easier to update in later changesets. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 1.430082 comb 1.430000 user 1.390000 sys 0.040000 (median of 10) after: ! wall 1.405192 comb 1.410000 user 1.390000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 1.971366 comb 1.970000 user 1.950000 sys 0.020000 (median of 10) after: ! wall 1.892541 comb 1.890000 user 1.870000 sys 0.020000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.252594 comb 0.250000 user 0.240000 sys 0.010000 (median of 38) after: ! wall 0.240075 comb 0.240000 user 0.240000 sys 0.000000 (median of 40) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013100 comb 0.010000 user 0.010000 sys 0.000000 (median of 226) after: ! wall 0.013247 comb 0.010000 user 0.010000 sys 0.000000 (median of 223) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.001633 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001670 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000078 comb 0.000000 user 0.000000 sys 0.000000 (median of 11984) after: ! wall 0.000119 comb 0.000000 user 0.000000 sys 0.000000 (median of 7982) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.207093 comb 0.210000 user 0.210000 sys 0.000000 (median of 47) after: ! wall 0.201551 comb 0.200000 user 0.200000 sys 0.000000 (median of 48) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.038462 comb 0.040000 user 0.040000 sys 0.000000 (median of 100) after: ! wall 0.036578 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7076

copies: add an explicit test using multiple roots We explicitly add a test checking for copies of connected revision but involving a different roots. This make sure the copies coming purely from the new roots does not interfere with the merge. Differential Revision: https://phab.mercurial-scm.org/D7075

sidedatacopies: fast path data fetching if revision has no sidedata When using the side data mode, we know their won't be any copy information sidedata. Skipping revision restoration give an important speed boost. In the future, there will be other user of sidedata, reducing the efficiency of this. We should consider adding a dedicated flag in revlog V2 to preserve this optimisation. The current situation is good enough for now. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 2.401569 comb 2.400000 user 2.390000 sys 0.010000 (median of 10) after: ! wall 1.429294 comb 1.430000 user 1.410000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 3.519140 comb 3.520000 user 3.470000 sys 0.050000 (median of 10) after: ! wall 1.963332 comb 1.960000 user 1.960000 sys 0.000000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.593880 comb 0.600000 user 0.590000 sys 0.010000 (median of 15) after: ! wall 0.251679 comb 0.250000 user 0.250000 sys 0.000000 (median of 38) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013414 comb 0.020000 user 0.020000 sys 0.000000 (median of 220) after: ! wall 0.013222 comb 0.020000 user 0.020000 sys 0.000000 (median of 223) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.002711 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) after: ! wall 0.001631 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000077 comb 0.000000 user 0.000000 sys 0.000000 (median of 12208) after: ! wall 0.000078 comb 0.000000 user 0.000000 sys 0.000000 (median of 12012) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.410067 comb 0.410000 user 0.410000 sys 0.000000 (median of 23) after: ! wall 0.207786 comb 0.200000 user 0.200000 sys 0.000000 (median of 46) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.097004 comb 0.090000 user 0.090000 sys 0.000000 (median of 100) after: ! wall 0.038495 comb 0.030000 user 0.030000 sys 0.000000 (median of 100) Differential Revision: https://phab.mercurial-scm.org/D7074

largefiles: use context manager for setting "lfstatus" on subrepos too It's safe since we now reset to the previous value. Differential Revision: https://phab.mercurial-scm.org/D7138

largefiles: reset "lfstatus" attribute to previous value in "finally" We were resetting it to False, which means nesting of these overrides would be incorrect. Differential Revision: https://phab.mercurial-scm.org/D7137

largefiles: add context manager for setting/clearing "lfstatus" attribute Differential Revision: https://phab.mercurial-scm.org/D7136

py3: adjust expected traceback in test-hook.t In Python 3, traceback.format_exception() displays the chain of exceptions so we get extra results from our grep. Also, ModuleNotFoundError is raised instead of ImportError from Python 3.6.

tests: use non-reverse grep in traceback in test-hook.t This will prepare for updating test output for Python 3.

py3: add Python 3 exception output to test-hook.t This is similar to, e.g., 3e9c6cef949b.

py3: ajust abort message in test-hook.t

rust-cpython: prepare for writing tests that require libpython What I wanted is to disable the "cpython/extension-module<ver>" feature while building tests executable, but that seems not doable. Instead, this patch adds new features dedicated for tests. The make rule is extracted so that we can easily run cargo tests. Added a minimal test of ref-sharing as an example. More tests will follow.

rust-cpython: make inner functions and structs of ref_sharing private Most of these methods were public because they had to be accessible from macro-generated functions. Some "unsafe" can be removed since we can guarantee the data consistency across non-public operations.

rust-cpython: keep Python<'a> token in PyRefMut This just clarifies that the GIL is obtained while PyRefMut is dereferenced, so there's no need of extra acquire_gil() to drop the reference.

rust-cpython: require GIL to borrow immutable reference from PySharedRefCell Since the inner value may be leaked, we probably need GIL to guarantee that there's no data race. inner(py).borrow() is replaced with inner_shared(py).borrow(), which basically means any PySharedRefCell data should be accessed through PySharedRef wrapper.

rust-cpython: make PyLeakedRef operations relatively safe This patch encapsulates the access to the leaked reference to make most leaked-ref operations safe. The only exception is leaked_ref.map(). I couldn't figure out how to allow arbitrary map operation safely over an unsafe static reference. See the docstring and inline comment for details. Now leak_immutable() can be safely implemented as the PyLeakedRef owns its inner data.

rust-cpython: put leaked reference in PyLeakedRef The next patch will make PyLeakedRef manage the lifetime of the underlying object. leak_handle.data.take() will be removed soon.

run-tests: make code coverage work on Python 3 This code path was obviously not tested on Python 3 because it blew up in several places due to str/bytes mismatch. For internal code, we normalize paths to bytes. For code calling into `coverage`, we normalize paths to str, which is what `coverage` seems to expect. After this, `run-tests.py -H` works on Python 3! Differential Revision: https://phab.mercurial-scm.org/D7133

py3: don't index into bytes in phabricator's _tokenize() `phabread`ing a stack using `hg phabread :D1234` under py3 will currently die with a KeyError because it will index into `b':D1234'` and return `58` instead of `b':'` as a token. Differential Revision: https://phab.mercurial-scm.org/D7129

rust-dirstate-status: use fast-path even with fsmonitor and sparse extensions When I initially ran the tests on my series, there were test failures related to those extensions. Now that the initial series has landed, I felt like going back to those issues because people with performance issues will often want to use fsmonitor and sparse. Either because of flaky tests or because the series has changed so much, I can't seem to reproduce these issues... let's widen the scope of the fast-path. Differential Revision: https://phab.mercurial-scm.org/D7128

rust-dirstatemap: remove additional lookups in dirstatemap We're copying this shortcut from the Python implementation, pretty standard for this codebase. Differential Revision: https://phab.mercurial-scm.org/D7117

tests: avoid chmod on windows in hgrc tests Follow up on changeset d201a637c971 introducing this test, which fails on Windows.

py3: fix sorting of obsolete markers during push This fixes similar errors as in 01e8eefd9434: TypeError: '<' not supported between instances of 'NoneType' and 'tuple'

exchange: extract a function to sort obsolete markers Follows up on changeset 01e8eefd9434, several other occurrences of sorted() need to be fixed. The _sortedmarkers() handles sorting obsmarkers with a None value as last element on Python 3.

py3: encode underlying error message during parse error of %include

convert: convert os.devnull to bytes before trying to join it with other bytes Together with the previous commit relating to emailparser this gets test-convert-tla.t passing under py3. Differential Revision: https://phab.mercurial-scm.org/D7062

rust-dirstate-status: add call to rust-fast path for `dirstate.status` The reasoning behind this patch is explained in the first patch of the series. Differential Revision: https://phab.mercurial-scm.org/D7060

rust-dirstate-status: rust-cpython bindings for `dirstate.status` The ref-sharing mechanism has improved, but its ergonomics still left a bit to be desired, as expected. Differential Revision: https://phab.mercurial-scm.org/D7059

rust-refsharing: add missing lifetime parameter in ref_sharing Differential Revision: https://phab.mercurial-scm.org/D7110

rust-dirstate-status: add first Rust implementation of `dirstate.status` Note: This patch also added the rayon crate as a Cargo dependency. It will help us immensely in making Rust code parallel and easy to maintain. It is a stable, well-known, and supported crate maintained by people on the Rust team. The current `dirstate.status` method has grown over the years through bug reports and new features to the point where it got too big and too complex. This series does not yet improve the logic, but adds a Rust fast-path to speed up certain cases. Tested on mozilla-try-2019-02-18 with zstd compression: - `hg diff` on an empty working copy: - c: 1.64(+-)0.04s - rust+c before this change: 2.84(+-)0.1s - rust+c: 849(+-)40ms - `hg commit` when creating a file: - c: 5.960s - rust+c before this change: 5.828s - rust+c: 4.668s - `hg commit` when updating a file: - c: 4.866s - rust+c before this change: 4.371s - rust+c: 3.855s - `hg status -mard` - c: 1.82(+-)0.04s - rust+c before this change: 2.64(+-)0.1s - rust+c: 896(+-)30ms The numbers are clear: the current Rust `dirstatemap` implementation is super slow, its performance needs to be addressed. This will be done in a future series, immediately after this one, with the goal of getting Rust to be at least to the speed of the Python + C implementation in all cases before the 5.2 freeze. At worse, we gate dirstatemap to only be used in those cases. Cases where the fast-path is not executed: - for commands that need ignore support (`status`, for example) - if subrepos are found (should not be hard to add, but winter is coming) - any other matcher than an `alwaysmatcher`, like patterns, etc. - with extensions like `sparse` and `fsmonitor` The next step after this is to rethink the logic to be closer to Jane Street's Valentin Gatien-Baron's Rust fast-path which does a lot less work when possible. Differential Revision: https://phab.mercurial-scm.org/D7058

share: unmark --relative as EXPERIMENTAL Differential Revision: https://phab.mercurial-scm.org/D7001

rust-cpython: removed now useless py_set() conversion In rust-cpython 0.3.0, HashSets implement the appropriate ToPythonObject, we can therefore get rid of this hacky conversion. There still remains an inefficiency in `MissingAncestors.bases()`: we have to clone, because `to_py_object()` requires full ownership. However: - the only use case outside of unit tests used to be from `setdiscovery.partialdiscovery` which is now fully implemented in Rust. - it's not worse than what `py_set()` used to do Differential Revision: https://phab.mercurial-scm.org/D7120

tests: stabilize test-eol-update on Windows These appeared in the recent eol fix series. I'm not sure about the identity mismatch warning- it seems like a general timing issue, because it happened on the first run but not when run with `--loop`. The buildbot also flagged this. Differential Revision: https://phab.mercurial-scm.org/D7112

tests: opt into python3 when testing virtualenv installation Otherwise it complains that Mercurial requires '~=2.7'. Since the existing linux py3 test doesn't flag this, I'm assuming that virtualenv simply isn't installed on that system. Differential Revision: https://phab.mercurial-scm.org/D7111

grep: put --diff in the first line of usage Differential Revision: https://phab.mercurial-scm.org/D7003

ui: option to preserve the progress bar Some interface like the fast-import format of git are meant for pipe processing. It can be still desirable to have progress bars in this case, but clearing the bar after each output operation is not desirable. Differential Revision: https://phab.mercurial-scm.org/D6843

phabricator: update hgmatcher to cope with the new data format The new conduit format can't be matched by the existing matcher since it shifts all the data into an urlencoded string of JSON, the order of which isn't stable between runs. Instead detect JSON values of params and load them into python dicts, which python will then naturally deep-equal compare. Differential Revision: https://phab.mercurial-scm.org/D7055

phabricator: treat non-utf-8 text files as binary as phabricator requires Phabricator can't cope with text files that are not UTF-8, so requires them to be submitted as binary files instead. This has the unfortunate effect of making them practically unreviewable in Phabricator since it will only display the separate versions of the file in other views, not a diff. `phabread`ing such submissions are similar, since it will just output the binary patch, but `hg import` copes with it fine and `hg diff` afterwards will show the actual changes. It is still a marked improvement over trying to submit them as text, which just leads to corruption (Phabricator will either output ? or HTML entities for non-UTF-8 characters, depending on context). Running decode on the whole file like this seems slightly unfortunate, but I'm not aware of a better way. Needs to be done to p1() version as well to detect conversions to UTF-8. Differential Revision: https://phab.mercurial-scm.org/D7054

phabricator: switch to the creatediff endpoint This lets the extension submit binary files, as well as set branch info so that it is exposed in the Phabricator interface. Differential Revision: https://phab.mercurial-scm.org/D7053

phabricator: change conduit data format to match arcanist The existing form isn't handled well by Phabricator for certain data types. Differential Revision: https://phab.mercurial-scm.org/D7052

phabricator: remove tests and all recordings The next commit is going to change the format of conduit API requests so none of the VCR recordings will match and all the tests will fail. Differential Revision: https://phab.mercurial-scm.org/D7051

phabricator: convert phabhunk and phabchange keys to bytes when finalising I thought I had included this code already but I'd missed it out. One of the disadvantages of attrs is that all the keys are unicode strings on py3, but we need them to be byte strings. Differential Revision: https://phab.mercurial-scm.org/D7066

phabricator: add addadded function This is the most complicated part of the new code, and is responsible for adding all added files, whether brand new, copied, or moved. This also includes creating the phabchanges for the original files that have been moved or copied from, since they might need their awayPaths and type updating if multiple copies are involved. Differential Revision: https://phab.mercurial-scm.org/D7050

sidedatacopies: directly fetch copies information from sidedata When using the sidedata mode, we don't need a complicated and expensive `context` object. Instead we directly fetch copies information from the sidedata (through a changelogrevision object). More optimisations coming. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 filelog: ! wall 3.679613 comb 3.680000 user 3.580000 sys 0.100000 (median of 3) base: ! wall 8.884369 comb 8.880000 user 8.850000 sys 0.030000 (median of 3) before: ! wall 4.681985 comb 4.680000 user 4.640000 sys 0.040000 (median of 3) after: ! wall 3.955894 comb 3.950000 user 3.940000 sys 0.010000 (median of 3) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 filelog: ! wall 0.003357 comb 0.010000 user 0.010000 sys 0.000000 (median of 781) base: ! wall 12.398524 comb 12.400000 user 12.330000 sys 0.070000 (median of 3) before: ! wall 6.459592 comb 6.470000 user 6.390000 sys 0.080000 (median of 3) after: ! wall 5.505774 comb 5.500000 user 5.410000 sys 0.090000 (median of 3) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 filelog: ! wall 2.754687 comb 2.760000 user 2.650000 sys 0.110000 (median of 4) base: ! wall 1.423166 comb 1.420000 user 1.400000 sys 0.020000 (median of 8) before: ! wall 0.961048 comb 0.960000 user 0.940000 sys 0.020000 (median of 11) after: ! wall 0.882950 comb 0.880000 user 0.880000 sys 0.000000 (median of 11) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f filelog: ! wall 1.552293 comb 1.550000 user 1.510000 sys 0.040000 (median of 6 base: ! wall 0.022662 comb 0.020000 user 0.020000 sys 0.000000 (median of 128) before: ! wall 0.021649 comb 0.020000 user 0.020000 sys 0.000000 (median of 135) after: ! wall 0.020951 comb 0.020000 user 0.020000 sys 0.000000 (median of 141) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 filelog: ! wall 1.500983 comb 1.500000 user 1.420000 sys 0.080000 (median of 7) base: ! wall 0.006956 comb 0.010000 user 0.010000 sys 0.000000 (median of 392) before: ! wall 0.004022 comb 0.000000 user 0.000000 sys 0.000000 (median of 735) after: ! wall 0.003988 comb 0.000000 user 0.000000 sys 0.000000 (median of 736) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e filelog: ! wall 0.011745 comb 0.020000 user 0.020000 sys 0.000000 (median of 250) base: ! wall 0.000156 comb 0.000000 user 0.000000 sys 0.000000 (median of 17180) before: ! wall 0.000118 comb 0.000000 user 0.000000 sys 0.000000 (median of 19170) after: ! wall 0.000097 comb 0.000000 user 0.000000 sys 0.000000 (median of 27276) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe filelog: ! wall 3.228230 comb 3.230000 user 3.110000 sys 0.120000 (median of 4) base: ! wall 0.997640 comb 1.000000 user 0.980000 sys 0.020000 (median of 10) before: ! wall 0.679500 comb 0.680000 user 0.680000 sys 0.000000 (median of 15) after: ! wall 0.596779 comb 0.600000 user 0.600000 sys 0.000000 (median of 17) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d filelog: ! wall 1.052501 comb 1.060000 user 1.040000 sys 0.020000 (median of 10 base: ! wall 0.214519 comb 0.220000 user 0.220000 sys 0.000000 (median of 45) before: ! wall 0.149675 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) after: ! wall 0.130786 comb 0.130000 user 0.130000 sys 0.000000 (median of 75) Differential Revision: https://phab.mercurial-scm.org/D7072

copies: use an unfiltered repository for the changeset centric algorithm Since the algorithm work form heads to ancestors, we don't need to check filtering for anything but the two entries. Using an unfiltered version is noticeably more efficient. Some timing on the pypy repo: revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 2.717861 comb 2.720000 user 2.700000 sys 0.020000 (median of 10) after: ! wall 2.582204 comb 2.580000 user 2.560000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 4.003146 comb 4.010000 user 3.970000 sys 0.040000 (median of 10) after: ! wall 3.814613 comb 3.810000 user 3.760000 sys 0.050000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.704204 comb 0.700000 user 0.700000 sys 0.000000 (median of 13) after: ! wall 0.657387 comb 0.650000 user 0.640000 sys 0.010000 (best of 14) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013493 comb 0.020000 user 0.020000 sys 0.000000 (median of 219) after: ! wall 0.013523 comb 0.020000 user 0.020000 sys 0.000000 (median of 218) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.003017 comb 0.000000 user 0.000000 sys 0.000000 (median of 985) after: ! wall 0.002876 comb 0.000000 user 0.000000 sys 0.000000 (median of 1000) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000073 comb 0.000000 user 0.000000 sys 0.000000 (median of 12672) after: ! wall 0.000082 comb 0.000000 user 0.000000 sys 0.000000 (median of 11456) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.478061 comb 0.470000 user 0.470000 sys 0.000000 (median of 19) after: ! wall 0.452420 comb 0.450000 user 0.450000 sys 0.000000 (median of 21) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.116015 comb 0.110000 user 0.110000 sys 0.000000 (median of 84) after: ! wall 0.109153 comb 0.100000 user 0.100000 sys 0.000000 (median of 90) Differential Revision: https://phab.mercurial-scm.org/D7123

copies: extract data extraction into a `revinfo` function The function is build once at the beginning of the algorithm and used fetch appropriate information for each revision. This abstracts some implementation details from the main algorithm and will help us to access the data more efficiently in future changesets. Differential Revision: https://phab.mercurial-scm.org/D7070

copies: avoid instancing more changectx to access parent revisions We just need to know the revision numbers of the parents, creating full context is needlessly expensive. This provide a small, but noticeable performance boost. revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 before: ! wall 2.885636 comb 2.900000 user 2.870000 sys 0.030000 (median of 10) after: ! wall 2.702270 comb 2.710000 user 2.690000 sys 0.020000 (median of 10) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 before: ! wall 4.298271 comb 4.290000 user 4.240000 sys 0.050000 (median of 10) after: ! wall 3.976610 comb 3.970000 user 3.920000 sys 0.050000 (median of 10) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 before: ! wall 0.773397 comb 0.770000 user 0.770000 sys 0.000000 (median of 11) after: ! wall 0.701634 comb 0.700000 user 0.700000 sys 0.000000 (median of 13) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f before: ! wall 0.013585 comb 0.010000 user 0.010000 sys 0.000000 (median of 217) after: ! wall 0.013550 comb 0.010000 user 0.010000 sys 0.000000 (median of 218) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 before: ! wall 0.003202 comb 0.000000 user 0.000000 sys 0.000000 (median of 929) after: ! wall 0.002993 comb 0.010000 user 0.010000 sys 0.000000 (median of 992) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e before: ! wall 0.000077 comb 0.000000 user 0.000000 sys 0.000000 (median of 12060) after: ! wall 0.000072 comb 0.000000 user 0.000000 sys 0.000000 (median of 12804) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe before: ! wall 0.510614 comb 0.500000 user 0.500000 sys 0.000000 (median of 18) after: ! wall 0.473681 comb 0.470000 user 0.470000 sys 0.000000 (median of 20) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d before: ! wall 0.126552 comb 0.130000 user 0.130000 sys 0.000000 (median of 77) after: ! wall 0.115240 comb 0.110000 user 0.110000 sys 0.000000 (median of 85) Differential Revision: https://phab.mercurial-scm.org/D7122

copies: get copies information directly from _copies Performance measurement does not show any significant performance movement. This is not surprising since p1copies() code is `self._copies[0]`. Differential Revision: https://phab.mercurial-scm.org/D7121

copies: simplify the handling of merges Instead of stacking copies for both parent on the head, we move copies outside of the heap into a dedicated dictionary. The two side of merge can we merged sooner, making the algorithm simpler. This simplicity reflect in the heap structure and speed up the execution for copies involving a large amount of merges. Here are timing for perfpathcopies of multiple revision pairs. - filelog: timing using filelog (with the introrev condition dropped) - base: this series base - before: the parent of this changeset - after: this changeset revision: large amount; added files: large amount; rename small amount; c3b14617fbd7 9ba6ab77fd29 filelog: ! wall 3.679613 comb 3.680000 user 3.580000 sys 0.100000 (median of 3) base: ! wall 8.884369 comb 8.880000 user 8.850000 sys 0.030000 (median of 3) before: ! wall 8.443747 comb 8.420000 user 8.410000 sys 0.010000 (median of 3) after: ! wall 4.697917 comb 4.690000 user 4.660000 sys 0.030000 (median of 3) revision: large amount; added files: small amount; rename small amount; c3b14617fbd7 f650a9b140d2 filelog: ! wall 0.003357 comb 0.010000 user 0.010000 sys 0.000000 (median of 781) base: ! wall 12.398524 comb 12.400000 user 12.330000 sys 0.070000 (median of 3) before: ! wall 10.852593 comb 10.850000 user 10.800000 sys 0.050000 (median of 3) after: ! wall 6.750832 comb 6.750000 user 6.640000 sys 0.110000 (median of 3) revision: large amount; added files: large amount; rename large amount; 08ea3258278e d9fa043f30c0 filelog: ! wall 2.754687 comb 2.760000 user 2.650000 sys 0.110000 (median of 4) base: ! wall 1.423166 comb 1.420000 user 1.400000 sys 0.020000 (median of 8) before: ! wall 1.068041 comb 1.060000 user 1.050000 sys 0.010000 (median of 10) after: ! wall 1.045916 comb 1.050000 user 1.040000 sys 0.010000 (median of 10) revision: small amount; added files: large amount; rename large amount; df6f7a526b60 a83dc6a2d56f filelog: ! wall 1.552293 comb 1.550000 user 1.510000 sys 0.040000 (median of 6 base: ! wall 0.022662 comb 0.020000 user 0.020000 sys 0.000000 (median of 128) before: ! wall 0.021111 comb 0.020000 user 0.020000 sys 0.000000 (median of 139) after: ! wall 0.021577 comb 0.020000 user 0.020000 sys 0.000000 (median of 138) revision: small amount; added files: large amount; rename small amount; 4aa4e1f8e19a 169138063d63 filelog: ! wall 1.500983 comb 1.500000 user 1.420000 sys 0.080000 (median of 7) base: ! wall 0.006956 comb 0.010000 user 0.010000 sys 0.000000 (median of 392) before: ! wall 0.004356 comb 0.010000 user 0.010000 sys 0.000000 (median of 675) after: ! wall 0.004329 comb 0.000000 user 0.000000 sys 0.000000 (median of 682) revision: small amount; added files: small amount; rename small amount; 4bc173b045a6 964879152e2e filelog: ! wall 0.011745 comb 0.020000 user 0.020000 sys 0.000000 (median of 250) base: ! wall 0.000156 comb 0.000000 user 0.000000 sys 0.000000 (median of 17180) before: ! wall 0.000100 comb 0.000000 user 0.000000 sys 0.000000 (median of 26912) after: ! wall 0.000105 comb 0.000000 user 0.000000 sys 0.000000 (median of 25689) revision: medium amount; added files: large amount; rename medium amount; c95f1ced15f2 2c68e87c3efe filelog: ! wall 3.228230 comb 3.230000 user 3.110000 sys 0.120000 (median of 4) base: ! wall 0.997640 comb 1.000000 user 0.980000 sys 0.020000 (median of 10) before: ! wall 0.778291 comb 0.780000 user 0.780000 sys 0.000000 (median of 13) after: ! wall 0.706594 comb 0.710000 user 0.710000 sys 0.000000 (median of 15) revision: medium amount; added files: medium amount; rename small amount; d343da0c55a8 d7746d32bf9d filelog: ! wall 1.052501 comb 1.060000 user 1.040000 sys 0.020000 (median of 10 base: ! wall 0.214519 comb 0.220000 user 0.220000 sys 0.000000 (median of 45) before: ! wall 0.160804 comb 0.160000 user 0.160000 sys 0.000000 (median of 62) after: ! wall 0.163736 comb 0.160000 user 0.160000 sys 0.000000 (median of 60) Differential Revision: https://phab.mercurial-scm.org/D7069

rust-utils: introduce a debug util to print the python stack trace Differential Revision: https://phab.mercurial-scm.org/D7057

rust-cross-platform: remove `unimplemented!` to get compile-time errors We should get a compile-time error that there is missing functionality for the targeted platform instead of the program breaking at runtime. Differential Revision: https://phab.mercurial-scm.org/D7107

widening: pass in matchers instead of patterns This matches how it's done for the non-ellipsis case. The oldmatch is not used yet, but it should be used eventually when widening no longer resends manifests and files the client already has. Differential Revision: https://phab.mercurial-scm.org/D7099

widening: trust user to give full "known" set The new narrow_widen wire protocol command is supposed to return data for exactly the revisions that the client requested (in order for it to be strip-free). So we should not add ancestors the client didn't ask for. Differential Revision: https://phab.mercurial-scm.org/D7098

widening: remove "depth" argument since it's always None Differential Revision: https://phab.mercurial-scm.org/D7097

widening: remove unused "heads" argument Differential Revision: https://phab.mercurial-scm.org/D7096

widening: remove pointless code for second changegroup The caller passes in common==heads, so we will never find any nodes to visit in this code and there will therefore never be a second changegroup emitted. Differential Revision: https://phab.mercurial-scm.org/D7095

widening: remove always-true condition in widening code The function is now specifically about widening, so we don't need to check if we're widening. Differential Revision: https://phab.mercurial-scm.org/D7094

narrow: drop server support for widening using the getbundle command (BC) The client still supports widening using the getbundle, which we (Google) still depend on a for a little while more (we've started the migration to the new narrow_widen command, but we're not done yet). Differential Revision: https://phab.mercurial-scm.org/D7093

widening: duplicate generateellipsesbundle2() for widening The widening and the non-widening code are quite different. It will be clearer to have them as sepearate functions. To start with, I've just copied it exactly, so it's clearer over the next few patches how they're different. The new function should gradually become more similar to bundle2.widen_bundle(), and should perhaps eventually be merged with that function. However, I've left it in narrowbundle2.py for now since it still depends on constants like _KILLNODESIGNAL there. Differential Revision: https://phab.mercurial-scm.org/D7092

tests: show graph log before and after widening in more cases I'm about to make some changes to which revisions get sent during widening. Some more tests will make it clearer what changes. I've also switched to graph log so we easily notice if the graph shape changes. Differential Revision: https://phab.mercurial-scm.org/D7091

tracing: add a couple of trace points on obsolete and repoview I was seeing weird stalls in some fast commands, and these trace events help explain them. Differential Revision: https://phab.mercurial-scm.org/D6550

dirstate: add some traces on listdir calls I'm starting to spend some time trying to make `hg status` do less work, and knowing what's happening in here and what's expensive is an important start. Differential Revision: https://phab.mercurial-scm.org/D7109

cleanup: hgdemandimport.tracing accepts strings, not bytes This does mean that the resulting traces will have some b'' goo in them on Python 3, but I think that's a worthwile price to pay since it doesn't break any tests. Differential Revision: https://phab.mercurial-scm.org/D7108

tests: fix formatting test after move of black config Differential Revision: https://phab.mercurial-scm.org/D7102

rust-dependencies: pin the dependencies of hg-core to prevent breakage A new version of the `rand` crate broke two discovery tests. We should upgrade our dependencies periodically manually. Differential Revision: https://phab.mercurial-scm.org/D7115

automation: install Python 3.8.0 Python 3.8 has been released. Let's install it instead of the RC / development version. Differential Revision: https://phab.mercurial-scm.org/D7114

automation: schedule an EC2Launch run on next boot Without this, launching EC2 instances constructed from the AMI won't go through the normal EC2 instance launch machinery. This missing machinery does important things like set up network routes to use the instance metadata service and process any UserData. Since EC2Launch now runs on subsequent boots and UserData is processed, we needed to make setting of UserData conditional on bootstrapping mode. Differential Revision: https://phab.mercurial-scm.org/D7113

dirs: give formatting oversight to clang-format Differential Revision: https://phab.mercurial-scm.org/D7104

dirs: tag a struct as not being formattable Differential Revision: https://phab.mercurial-scm.org/D7103

sidedatacopies: only store an entry if it has values This will make for a shorter storage and help use to write faster code in simple case. This change already provided a speed boost for copy tracing. For example here is combined time of running copies tracing on a 6989 pairs of revision on the pypy repos: before: 771s after: 631s - 18% This also has a very positive impact on changelog size. For example here are the number for the `00changelog.d` file of pypy. before: 30449712 Bytes after: 24973718 Bytes - 18% This give an overall quite acceptable overhead for storing copies into the changelog: filelog-only: 23370586 sidedata: 24973718 Bytes + 7% Differential Revision: https://phab.mercurial-scm.org/D7068

sidedatacopies: only read from copies when in this mode If we know we expect data from sidedata, we read them from sidedata and nothing else. This avoid looking into extra for revision without sidedata entries. Such revision will be introduced in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D7067

dirs: remove mutable string optimization at all As far as I can see, the optimization trick has been dead since 42e89b87ca79 "dirs: speed up by storing number of direct children per dir". After 42e89b87ca79, the key variable is cleared to NULL at each iteration.

py3: prevent comparison with None in curses histedit (issue6196)

fix: match patterns relative to root I was surprised fixer patterns (used to determine which fixers to run) are applies to the parent directory, not the repo root directory. Danny Hooper (the author of the extension) seemed to agree that it's better to apply them to the repo root, so that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D7101

tests: add test showing that fixer patterns are currently relative to $PWD Differential Revision: https://phab.mercurial-scm.org/D7100

black: also ignore grey.py Differential Revision: https://phab.mercurial-scm.org/D7090

black: use multiline exclude definition Our excludes list is pretty complicated, so let's give ourselves some readability. Black treats multiline exclude definitions as verbse regular expressions, so we can split this up and make it easier to consume. Differential Revision: https://phab.mercurial-scm.org/D7089

black: move remaining config knobs into toml file Differential Revision: https://phab.mercurial-scm.org/D7088

black: rename pyproject.toml to black.toml Black won't read this automatically (you'll have to specify --config), but having a pyproject.toml *at all* puts pip in PEP 517/518 mode which breaks us for obscure reasons I don't understand. Rather than waste a ton of time fighting with pip, let's just do this. Differential Revision: https://phab.mercurial-scm.org/D7087

fix: remove a never-true check for unset pattern in Fixer.affects() We don't create an instance of a Fixer if the pattern is None, so we don't need to check that in affects(). Differential Revision: https://phab.mercurial-scm.org/D7086

fix: warn when a fixer doesn't have a configured command It seems we currently produce an empty command line and then decide to not run it, but it seems better to skip that part too. Differential Revision: https://phab.mercurial-scm.org/D7085

fix: make Fixer initialization more explicit for clarity I found it quite confusing that Fixer accessed fields that seemed like they didn't exist. Differential Revision: https://phab.mercurial-scm.org/D7084

fix: don't pass in default value when looking up config The defaults are now registered correctly so we don't need to pass in the default value when we look up a config value. Differential Revision: https://phab.mercurial-scm.org/D7083

fix: fix registration of config item defaults Before this patch, because of the "(:<name>)?", all registered patterns would match and the default value would not be the one we thought we had registered (maybe it just took the default value for the first match?). This didn't matter because we didn't care about the default value; we used our own, intended default value in getfixers() anyway. We also have to look up each config item individually in order to not get developer warnings. Differential Revision: https://phab.mercurial-scm.org/D7082

automation: capture additional exception when formatting I started encountering this for some reason. Differential Revision: https://phab.mercurial-scm.org/D7078

contrib: update to latest Windows package versions Let's keep modern. Differential Revision: https://phab.mercurial-scm.org/D7077

rust-cpython: drop direct dependency on python(27|3)_sys We no longer use it.

rust-cpython: leverage upstreamed py_capsule_fn!() macro

rust-cpython: bump cpython crates to 0.3 Unblocks py_capsule_fn!().

rust-cpython: turn inline comments into non-doc comments

rust-cpython: fix signature of make_dirstate_tuple() Fortunately, the layout of PyObject {} is compatible with a C pointer, but we shouldn't rely on that. This also fixes the handling of NULL pointer.

rust-cpython: mark capsule function as unsafe

rust-cpython: add wrapper around decapsule_make_dirstate_tuple() There are a couple of safety issues. First, the returned function pointer must be unsafe. Second, its return value must be a raw pointer (i.e. python27_sys::PyObject), not a cpython::PyObject. The wrapper function will address these issues.

eol: don't fallback to use .hgeol from tip (BC) If no .hgeol were found in the current working directory, eol would fallback to use the one in tip. That could in some cases give very confusing or wrong behaviour when it applied wrong filters. It might be convenient to have plain 'clone' immediately apply 'native' encoding patterns in the cloned repo. But it is wrong to assume that this revision is tip, and even more wrong to also apply it when not cloning - for example when updating between history revisions. The encoding should always match the content of the current .hgeol . It should never use anything else.

eol: tweak test-eol-clone.t with better descriptions and logging Expose impact of changes coming next ...

eol: fix update - don't use and apply removed .hgeol patterns 'hg up -C' to revisions with different .hgeol patterns could leave dirty changes in the working directory. That could make deployment of new .hgeol filters tricky: they would "occasionally" apply also in branches where they shouldn't. Fixed by dropping all old patterns before applying new ones.

eol: cache needs update, also if it has same timestamp as the source Ignoring same timestamp could (in theory?) cause changes to not be detected. It might happen quite often that the cache is populated right after .hgeol has been updated and they thus have the same time stamp second. But we want correctness, and if it populates the cache so fast, then it can also not be a big problem to run it again next time when the timestamp has moved on.

eol: update isbinary filter to work without compat wrapper

localrepo: fix variable binding in handling of old filters The lambda was referencing oldfn in outer scope without binding the current value. If oldfn function were reassigned before use, wrong filters could be used. Fixed by having oldfn as named parameter default value of the lambda.

localrepo: debug log of filter name when filtering through a function

eol: test-eol-update.t coverage around update --clean using filters ... badly This will reveal problems and track their fixes.

copies: drop the findlimit logic We don't use the limit anymore so we should stop computing that limit. I did not bother measuring the potential performance gain. I am assuming that not running any code will be faster that doing some computation and not using the result.

pathcopies: give up any optimization based on `introrev` Between 8a0136f69027 and d98fb3f42f33, we sped up the search for the introduction revision during path copies. However, further checking show that finding the introduction revision is still expensive and that we are better off without it. So we simply drop it and only rely on the linkrev optimisation. I ran `perfpathcopies` on 6989 pair of revision in the pypy repository (`hg perfhelper-pathcopies`. The result is massively in favor of dropping this condition. The result of the copy tracing are unchanged. Attempt to use a smaller changes preserving linkrev usage were unsuccessful, it can return wrong result. The following changesets broke test-mv-cp-st-diff.t - if not f.isintroducedafter(limit): + if limit >= 0 and f.linkrev() < limit: return None Here are various numbers (before this changeset/after this changesets) source destination before after saved-time ratio worth cases e66f24650daf 695dfb0f493b 1.062843 1.246369 -0.183526 1.172675 c979853a3b6a 8d60fe293e79 1.036985 1.196414 -0.159429 1.153743 22349fa2fc33 fbb1c9fd86c0 0.879926 1.038682 -0.158756 1.180420 682b98f3e672 a4878080a536 0.909952 1.063801 -0.153849 1.169074 5adabc9b9848 920958a93997 0.993622 1.147452 -0.153830 1.154817 worse 1% dbfbfcf077e9 aea8f2fd3593 1.016595 1.082999 -0.066404 1.065320 worse 5% c95f1ced15f2 7d29d5e39734 0.453694 0.471156 -0.017462 1.038488 worse 10% 3e144ed1d5b7 2aef0e942480 0.035140 0.037535 -0.002395 1.068156 worse 25% 321fc60db035 801748ba582a 0.009267 0.009325 -0.000058 1.006259 median 2088ce763fc2 e6991321d78b 0.000665 0.000651 0.000014 0.978947 best 25% 915631a97de6 385b31354be6 0.040743 0.040363 0.000380 0.990673 best 10% ad495c36a765 19c10384d3e7 0.431658 0.411490 0.020168 0.953278 best 5% d13ae7d283ae 813c99f810ac 1.141404 1.075346 0.066058 0.942126 best 1% 81593cb4a496 99ae11866969 1.833297 0.063823 1.769474 0.034813 best cases c3b14617fbd7 743a0fcaa4eb 1101.811740 2.735970 1099.075770 0.002483 c3b14617fbd7 9ba6ab77fd29 1116.753953 2.800729 1113.953224 0.002508 058b99d6e81f 57e249b7a3ea 1246.128485 3.042762 1243.085723 0.002442 9a8c361aab49 0354a250d371 1253.111894 3.085796 1250.026098 0.002463 442dbbc53c68 3ec1002a818c 1261.786294 3.138607 1258.647687 0.002487 As one can see, the average case is not really impacted. However, the worth case we get after this changeset are much better than the one we had before it. We have 30 pairs where improvements are above 10 minutes. This reflect in the combined time for all pairs before: 26256s after: 1300s (-95%) If we remove these pathological 30 cases, we still see a significant improvements: before: 1631s after: 1245s (-24%)

perf: introduce a `--contains` flag to the `perfdirstate` command The new flag benchmark a large amount of `filepath in dirstate` call. This will be useful to compare the Python and Rust implementation of the dirstatemap.

perf: introduce a `--iteration` to `perfdirstate` This flag benchmark an iteration over all the file in the dirstate. This will be useful to compare the Python and the Rust implementation of the dirstate.

tests: open file in binary mode in test-upgrade-repo.t Otherwise the file sizes printed in the output diverge from other platforms. Differential Revision: https://phab.mercurial-scm.org/D7081

hghave: use a native string to invoke the `black` command Windows can't create a process based on bytes. Apparently only the output test in `matchoutput` wants bytes. Differential Revision: https://phab.mercurial-scm.org/D7080

py3: convert cwd to native string when running `fix` This prevents a bunch of failures on Windows. Differential Revision: https://phab.mercurial-scm.org/D7079

format: format commands.py, which recently regressed # skip-blame just reformatting Differential Revision: https://phab.mercurial-scm.org/D7064

py3: fix test-bad-extension expectations for py35 ModuleNotFoundError was introduced in 3.6, so the tests need to expect to see ImportError with 3.5. Differential Revision: https://phab.mercurial-scm.org/D7063

phabricator: add addremoved and addmodified functions These are relatively simple so add them together. As the name indicates, these add phabchange objects for removed and modified files to the phabdiff. Differential Revision: https://phab.mercurial-scm.org/D7049

phabricator: add makebinary and addoldbinary functions These populate the phabchange with the data for a binary file, much as maketext does for text files. Differential Revision: https://phab.mercurial-scm.org/D7048

phabricator: add the uploadfile function This is needed to be able to submit binary files such as images in commits (and also non-UTF-8 text files). One of the primary disadvantages of the current use of createrawdiff is that Phabricator simply drops any binary diffs included in the patch, but not the files, so if such a diff is then committed it corrupts any binary files in it. Differential Revision: https://phab.mercurial-scm.org/D7047

phabricator: add the uploadchunks function Differential Revision: https://phab.mercurial-scm.org/D7046

phabricator: add the maketext function This add the diff data for a text file to a phabchange. Differential Revision: https://phab.mercurial-scm.org/D7045

phabricator: add the phabdiff data structure This holds all the data about a commit, and is passed to the differential.creatediff API. Differential Revision: https://phab.mercurial-scm.org/D7044

phabricator: add the phabchange data structure These store data about individual files in a commit. Differential Revision: https://phab.mercurial-scm.org/D7043

phabricator: add the phabhunk data structure These store the actual diff data (for UTF-8 text files anyway) and are equivalent to hunks in a patch file. Differential Revision: https://phab.mercurial-scm.org/D7042

phabricator: add the DiffChangeType and DiffFileType constants These are used in Phabricator change objects. There are more values but so far as I can tell we don't need them. Differential Revision: https://phab.mercurial-scm.org/D7041

help: adding a help category to narrow and remotefilelog commands Differential Revision: https://phab.mercurial-scm.org/D6998

rust-cpython: drop self.borrow_mut() in favor of PySharedRef wrapper

rust-cpython: drop self.leak_immutable() in favor of PySharedRef wrapper

rust-cpython: add safe wrapper representing shared data borrowed from PyObject PySharedRef is a tempoary wrapper around PySharedRefCell. It provides safe functions for each shared data. $shared_accessor implements a safe method to construct PySharedRefCell. This allows us to add more than once PySharedRefCell to a Python object.

rust-cpython: move $leaked struct out of macro It wasn't easy to hack the $leaked struct since errors in macro would generate lots of compile errors. Let's make it a plain struct so we can easily extend it. PyLeakedRef keeps a more generic PyObject instead of the $name struct since it no longer has to call any specific methods implemented by the $name class. $leaked parameter in py_shared_iterator!() is kept for future change.

rust-cpython: store leaked reference to PySharedState in $leaked struct I want to move it out of the macro, and allow multiple sharable objects per PyObject.

rust-cpython: mark PySharedState as Sync so &'PySharedState can be Send The goal is to store &'static PySharedState in $leaked struct, which allows us to move the $leaked struct out of the macro. Currently, it depends on inner.$data_member(py), which can't be generalized. PySharedState is Sync because any mutation or read operation is synchronized by the Python GIL, py: Python<'a>, which should guarantee that &'PySharedState can be sent to another thread.

rust-cpython: move borrow_mut() to PySharedRefCell PySharedRefCell() will host almost all py_shared public functions. This change is the first step. borrow_mut() can be safely implemented since PySharedRefCell knows its inner object is managed by its own py_shared_state.

rust-cpython: move py_shared_state to PySharedRefCell object The goal of this series is to encapsulate more "py_shared" thingy and reduce the size of the macro, which is hard to debug. Since py_shared_state manages the borrowing state of the object owned by PySharedRefCell, this change makes more sense. If a PyObject has more than one data to be leaked into Python world, each PySharedState should incref the parent PyObject, and keep track of the corresponding borrowing state.

py3: decode username and password before SMTP login smtplib.SMTP.login() requires str on Python 3. For 'password', we only need to decode when value comes from config as getpass() returns the correct type already.

py3: use socket.makefile() instead of dropped smtplib.SSLFakeFile The latter (undocumented internal) class got removed in Python 3.3. Use socket.makefile() as suggested: https://docs.python.org/3/whatsnew/3.3.html#porting-python-code

py3: call SMTP.docmd() with an str Otherwise, this always returns (502, '5.5.2 Error: command not recognized').

py3: call SMTP.has_extn() with an str Passing a bytes on Python 3 always returns False, thus starttls is not properly detected.

py3: fix sorting of obsolete markers when building bundle Last item of marker tuple (parents) is either None or tuple. Comparison thus fails on Python 3 with: TypeError: '<' not supported between instances of 'tuple' and 'NoneType' Fixing this by coercing None to the empty tuple when sorting markers in exchange._getbundleobsmarkerpart().

sidedata: rename the configuration option to `exp-use-side-data` We don't want setup configured to use the final version that would end using the experimental one while using and older version. Differential Revision: https://phab.mercurial-scm.org/D7040

py3: drop ui.flush() during interactive patch filtering With previous changeset fixing line buffering on stdout, this is no longer needed.

py3: keep stdout as defined by pycompat in procutil According to https://docs.python.org/3/library/functions.html#open, it's not possible to use 1 as buffering argument value in binary mode. This is probably why line buffering does not work well in python3. On the other hand, sys.stdout.buffer appears to be line-buffered already on python3. So by not replacing it, there should be no behavior change. This fixes buffering issue in "hg email" (confirmation prompt shown before information to be confirmed).

notify: cast hash to bytes This is needed to avoid a str/bytes mismatch when interpolating a line or 2 later. Differential Revision: https://phab.mercurial-scm.org/D7021

push: support config option to require revs be specified when running push Differential Revision: https://phab.mercurial-scm.org/D6989

py3-discovery: using plain str in stats dict rust-cpython converts automatically from Rust strings to the appropriate `str` for the target Python version. Insisting on discovery stats dict keys to be bytes hence breaks the process (this is spotted by test-setdiscovery.t). Now that byteify-strings has been run on the entire codebase, and the import transformer is not there any more, the simplest fix is to make the keys plain str again. Another possible fix would be to forcefully convert to bytes in rust-cpython code, but that feels less natural, and would probably have to be reverted down the road. Differential Revision: https://phab.mercurial-scm.org/D7039

perf: fix `perfhelper-mergecopies` report of #changesets Same as `perfhelper-pathcopies`. The previous computation `<base>::<target>` was wrong, what we actually need is `::<target> - ::<base>`. This is now fixed.

perf: fix `perfhelper-pathcopies` report of #changesets The previous computation `<base>::<target>` was wrong, what we actually need is `::<target> - ::<base>`. This is now fixed

py3: use integer division in curseschunkselector.printstring() This fixes a crash when scrolling in curses UI when refresh() is called when a float value (namely 'self.firstlineofpadtoprint', taking its value indirectly from 'self.linesprintedtopadsofar').

crecord: drop duplicated set of firstlineofpadtoprint attribute The attribute is already set a couple of lines above, in curseschunkselector.__init__().

patchbomb: use mail.Generator alias for py2/py3 compat

py3: use email.generator.BytesGenerator in patch.split() This fixes test-import.t on python3. We add Generator alias in mail module to handle python2/python3 compatibility.

py3: only flush before prompting during interactive patch filtering Follows up on c9093ae8d6c4. It's enough to flush just before each prompt.

py3: add a missing b'' prefix in test extension for chg Differential Revision: https://phab.mercurial-scm.org/D7038

fuzz: new target to fuzz jsonescapeu8fast This code just feels complicated enough we should go ahead and give it a dedicated fuzzer: we've found bugs in similar things before. Differential Revision: https://phab.mercurial-scm.org/D7034

fuzz: new fuzzer for fncache-related functions Not all of these are strictly fncache-related, but they all have th same signature and similar-enough behavior that we may as well fuzz them together. No obvious bugs for once, but these felt like they were just complicated enough to cover. Differential Revision: https://phab.mercurial-scm.org/D7033

fuzz: exercise a little more revlog code Differential Revision: https://phab.mercurial-scm.org/D7032

fuzz: new fuzzer for dirs.c This found a six-year-old bug immediately, and then I put it through a few CPU-days of time before sending it. Differential Revision: https://phab.mercurial-scm.org/D7031

dirs: fix trivial over-read of input data This code, introduced in 8c0a7eeda06d, was intentionally over-reading an input string to avoid getting a shared string object for a one-byte input. Unfortunately with an empty input (like in the case of a fuzzer getting started) this was a trivial over-read and triggered an AddressSanitizer failure. I went out of my way to make sure the code still does the copy-avoidance tricks. I don't think this change will cost us much performance since the one-character strings should be cached aggressively anyway. Differential Revision: https://phab.mercurial-scm.org/D7030

sidedatacopies: deal with upgrading and downgrading to that format This is quite useful to test this on real life data. Differential Revision: https://phab.mercurial-scm.org/D6955

sidedatacopies: move various copies related function to the copies modules We will need to access these logic form the copies module. So we move them from their higher level module to the lower level `copies` module. We cannot use them from their top level module as it would create cycles. Differential Revision: https://phab.mercurial-scm.org/D6954

sidedatacopies: read rename information from sidedata Repository using the new format now use changeset centric algorithm and read the copies information from the changelog sidedata. Differential Revision: https://phab.mercurial-scm.org/D6953

changelog: make copies related function return None or a valid value With the previous code, existing but empty value were not "decoded", leading to the method returning one of `None`, some valid value (`list` or `dict`) or `b''`. On a general basis, not explicitly checking for None is a source of bugs. Having a clean return types will help the side-data copies code in future changesets. Differential Revision: https://phab.mercurial-scm.org/D7037

test: fix zstd related output in pure tests Differential Revision: https://phab.mercurial-scm.org/D7036

sidedatacopies: get and store sidedata in the changelogrevision object The object provide a simple way to access changelog entry, we need it to also bear the sidedata value. Since the sidedata are retrieved at the same time as the revision, we can do that without extra cost. Differential Revision: https://phab.mercurial-scm.org/D6951

sidedatacopies: write copies information in sidedata when applicable If the format of the repository indicate it stores copies information into changeset's sidedata, then we actually write that information into sidedata at commit time. It will be put to use in later changesets. Currently, we store all field unconditionally, but that is likely to change in the future for the sake of efficiency. Differential Revision: https://phab.mercurial-scm.org/D6950

sidedatacopies: introduce a sidedata testcase for test-copies-in-changeset.t For now it is equivalent to the filelog case, but introducing this early helps make the comings changesets clearer. Differential Revision: https://phab.mercurial-scm.org/D6949

sidedatacopies: introduce a sidedata testcase for test-copies-unrelated.t For now it is equivalent to the filelog case, but introducing this early helps make the comings changesets clearer. Differential Revision: https://phab.mercurial-scm.org/D6948

sidedatacopies: introduce a sidedata testcase for test-copies.t For now it is equivalent to the filelog case, but introducing this early helps make the comings changesets clearer. Differential Revision: https://phab.mercurial-scm.org/D6947

sidedatacopies: teach upgrade about the new requirement The `debugformat` and `debugupgraderepo` command now detect the requirement. (upgrade to and from are not currently possible). Differential Revision: https://phab.mercurial-scm.org/D6946

sidedatacopies: add a new requirement for storing copies into sidedata The end goal is to have changesets centric sidedata information stored into changelog sidedata. This make it possible to use the changeset based copy tracing algorithm on any repository without affecting hashes. The actual implementation is coming. The feature is marked as experimental (do not use in production) until we stabilise details about the format. Differential Revision: https://phab.mercurial-scm.org/D6945

debugsidedata: small doc improvement The command has a mode to actually show the sidedata, but it wasn't documented. Differential Revision: https://phab.mercurial-scm.org/D6944

upgrade: allow for `sidedata` removal If the side-date feature is removed, we need to drop them. Differential Revision: https://phab.mercurial-scm.org/D6943

upgrade: allow upgrade to repository using sidedata Repository can now be migrated to support sidedata. More requirements and migration will be needed to actual side-data usage. This is a step in that direction. To test the feature, we leverage the test extension. It make sure the `update` part of the side-data migration actually works. Differential Revision: https://phab.mercurial-scm.org/D6942

revlog: add a way to control sidedata changes during revlog.clone We introduce a new argument to pass a callable dealing with the side data logic. Since the side data are usually associated to higher level logic, the revlog code itself is unlikely to know that to do itself. As a result the higher level upgrade code will be responsible to decide what needs to changes. The lower level revlog.clone just get simple instructions to apply. Differential Revision: https://phab.mercurial-scm.org/D6941

sidedata: use only changegroup3 if sidedata is in use This is necessary to exchange the flags over the wire (and within bundles). Differential Revision: https://phab.mercurial-scm.org/D6940

sidedata: apply basic but tight security around exchange We don't currently have code to deal with exchange between repository using sidedata and repository not using sidedata. Until we implement such code (eg: dropping side data when pushing to a non-sidedata repo) we prevent the two kind of repo to speak to each other. This is somewhere similar to what 'treemanifest' does. Note that sidedata exchange is broken unless one use changegroup v3 anyway. See next changeset for details. Differential Revision: https://phab.mercurial-scm.org/D6939

perf: use `setup` function in `perfdirstatewrite` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstatewrite`

perf: use `setup` function in `perfdirfoldmap` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirfoldmap`

perf: use `setup` function in `perfdirstatefoldmap` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstatefoldmap`

perf: use `setup` function in `perfdirstatedirs` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstatedirs`

perf: use `setup` function in `perfdirstate` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstate`

perf: document `perfstatus`

py3: flush ui after each message in interactive patch filtering Otherwise, actions from ui.write() are buffered and displayed at end of interactive session.

py3: decode prompt string before calling rawinput Calling input() (aka pycompat.rawinput() on python3) with a byte string displays a byte string in the console. E.g. in interactive commit, we get: b"examine changes to '<file>'?\n(enter ? for help) [Ynesfdaq?]". Similarly, "hg email" prompts are messed up. We thus decode the prompt string before running rawinput().

cleanup: join string literals that are already on one line Thanks to Kyle for noticing this and for providing the regular expression to run on the codebase. This patch has been reviewed by the test suite and they approved of it. # skip-blame: fallout from mass reformatting Differential Revision: https://phab.mercurial-scm.org/D7028

debugcommands: add a few more writenoi18n() Differential Revision: https://phab.mercurial-scm.org/D7027

py3: delete b'' prefix from safehasattr arguments Differential Revision: https://phab.mercurial-scm.org/D7029

destutil: provide hint on rebase+merge for how to specify destination/rev Without a destination specified, rebase and merge attempt to identify a good candidate; this fails if there's too many heads, the heads have a bookmark, or other reasons. Complicating the issue, users may have specified -t, thinking it means target, and are confused when the error message says that they need to specify an explicit rev or explicit destination - they feel like they already have. Differential Revision: https://phab.mercurial-scm.org/D7024

hghave: document format for version feature checks as <name><vers>, no dots I had a bit of a difficult time when attempting to fix a use of `py-38` when trying to figure out what the correct way of specifying the feature was. By having the strings 'py-3*' and 'py3.8' in the hghave.checkvers docstring, I would have had a much easier time of it instead of having to trace exactly what was happening here. Differential Revision: https://phab.mercurial-scm.org/D7023

check-commit: allow foo_bar naming in functions nameswithallthewordssmashedtogetherarehardtoread. especiallyifenglishisnotyourprimarylanguage. Let's align with the rest of the programming universe and allow_the_use_of_underscores_in_names. We took a hand poll at the 5.2 sprint regarding this change and all but 1 person supported it. The person who didn't expressed concerns around excessive API breakage if we mass renamed things. But we're not planning to mass rename things for the sake of renaming, so all should be well. Differential Revision: https://phab.mercurial-scm.org/D2010

infinitepush: mark extension as likely to be deleted .. bc:: The infinitepush extension is believed to be unused, and will be deleted at the end of 2020 unless users contact mercurial-devel@mercurial-scm.org. Differential Revision: https://phab.mercurial-scm.org/D7022

tests: use range() in generate-churning-module.py This is a test-only script. Performance on Python 2 for creating a full list instead of a generator doesn't matter. With this change, test-check-pyflakes.t passes on Python 3! Differential Revision: https://phab.mercurial-scm.org/D7019

tests: allow warning about file The code is protected by a block that makes it Python 2 only, so use of file is acceptable. Differential Revision: https://phab.mercurial-scm.org/D7018

lsprof: remove __main__ functionality I'm pretty sure nobody uses this. I noticed it because Python 3 linting is complaining about execfile. Differential Revision: https://phab.mercurial-scm.org/D7017

tests: use proper Python 3.8 feature Fix typo introduced in 830eacef67f8. Differential Revision: https://phab.mercurial-scm.org/D7016

py3: finish porting iteritems() to pycompat and remove source transformer This commit finishes porting .iteritems() to pycompat.iteritems() for the mercurial package. The translation of .iteritems() to .items() was the last conversion performed by the source transformer. With the porting to pycompat complete, we no longer have a need for the source transformer. So the source transformer has been removed. Good riddance! The code base is now compatible with Python 2 and Python 3. For the record, as the person who introduced the source transformer, it brings me joy to delete it. It accomplished its goal to facilitate a port to Python 3 without overly burdening people on some painful low-level differences between Python 2 and 3. It is unfortunate we still have to wallpaper over many differences with the pycompat shim. But it is what it is. Differential Revision: https://phab.mercurial-scm.org/D7015

py3: define and use pycompat.iteritems() for hgext/ .iteritems() -> .items() is the last source transform being performed. But it is also the most widely used. This commit adds a pycompat.iteritems symbol and imports it in place of .iteritems() for usage in hgext/. I chose to stop at just hgext/ because the patch will be large and it is an easy boundary to stop at since we can disable source transformation on a per-package basis. There are places where the type does implement items() and we could call items() directly. However, this would require critical thought and I thought it would be easier to just blindly change the code. We know which call sites need to be audited in the future because they have "pycompat.iteritems." With this change, we no longer perform source transformation on hgext! Differential Revision: https://phab.mercurial-scm.org/D7014

py3: define and use pycompat.itervalues() .itervalues() only exists on Python 2. Python 3's equivalent is .values(). But we don't want to blindly use .values() everywhere because on Python 2, it will create a list, which will have performance implications. This commit introduces pycompat.itervalues() which will call the appropriate method on the passed object. We update all callers of obj.itervalues() to pycompat.itervalues(obj) instead. With this commit, the only source tranforming remaining is for iteritems(). Victory is near... Differential Revision: https://phab.mercurial-scm.org/D7013

py3: stop normalizing 2nd argument of *attr() to unicode Now that we don't byteify strings, we can stop normalizing the 2nd string argument to getattr() and remove explicit overrides we were using in the code base. We no longer use some helper functions in the source transformer, so we remove those as well. Differential Revision: https://phab.mercurial-scm.org/D7012

log: populate keywords if specified in custom -Tjson(...) or -Tcbor(...) To make things simple, early return for ui.quiet is disabled if the formatter is templated and provides some datahint().

formatter: map -Tjson(...) and -Tcbor(...) to templater Even though custom JSON output could be generated by using --config template.json="{dict(...)|json}" ..., doing that is tedious because of the trailing comma handling. This patch introduces special syntax for JSON/CBOR formats. -Tjson(...) is translated to template as if function-style template definition were supported: [templates] json(...) = "{dict(...)|json}" json(...):docheader = "[\n " json(...):docfooter = "\n]\n" json(...):separator = ",\n "

formatter: parse name of built-in formatter templates in standard way This slightly makes it easier to add "-Tjson(...)" handling, which should be enabled only if the template specifier doesn't look like a literal template. In other words, it should be handled after "if '{' in tmpl". This makes "log -Tpickle" and "log -Tdebug" abort, which I think is better than just printing "picklepicklepickle...".

formatter: pass in template spec to templateformatter as argument Prepare for the next patch, which will unify handling of the formatter names and the template names.

templater: add public parseexpr() function to parse "-Tjson(...)" Extracted _addparseerrorhint() to show nicer hint on error.

import: read X-Mercurial-Node email header to determine nodeid This would be useful to import and obsolete patches sent using 'hg email --plain', using evolve's --obsolete option of 'hg import'. If email body contains Mercurial patch header ('# HG changeset patch' block), nodeid parsed from X-Mercurial-Node header will still be overridden by respective value found in body.

patchbomb: use procutil.shellquote() instead of shlex to escape email address This basically backs out 2cc453284d5c, and inserts procutil.shellquote() instead. I don't care about Windows compatibility here, but shlex.quote() can't handle byte strings on Python 3.

formatting: introduce a `test-check-format-black.t` that enforce formatting This should prevent use to drift away from the expect format.

formatting: run black on all file again Apparently, since the blackgnarok, we divergence from the expected formatting. Formatted using:: grey.py -S $(hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**" - hgext/fsmonitor/pywatchman/** - contrib/grey.py') # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions

formatting: make black --quiet in the example `hg fix` config I do like cake, but I cannot have so many of them.

formatting: remove the data-ogre from the config example Without he final `-`, black ignore the stdin and just looks at file provided as command line argument. Since `hg fix` feeds the file content through stdin and does not pass file argument, this meant black happily exited successful (all files passed as argument were formatted) without any output. Fix picked this "no output" as the new file content, deleting all previous content. I appreciate the fact this effectively removed all buggy code in any files processing that way, but this also ate all my data. The example config is now fixed in that regards.

py3: stop normalizing .encode()/.decode() arguments to unicode Now that we don't byte transform string literals, we no longer need this transform. While we're here, we also drop some superfluous u'' prefix in existing callers. Differential Revision: https://phab.mercurial-scm.org/D7011

py3: manually import pycompat.delattr where it is needed And with this change, we no longer need the auto-inserted import statement in the source transformer, so it has been removed! Differential Revision: https://phab.mercurial-scm.org/D7010

py3: manually import getattr where it is needed The march continues. Differential Revision: https://phab.mercurial-scm.org/D7009

py3: stop injecting pycompat.hasattr into modules I only found a single user of this pattern, probably because we use util.hasattr everywhere. Differential Revision: https://phab.mercurial-scm.org/D7008

py3: manually import pycompat.setattr where it is needed Continuing to eliminate the implicit import of symbols in the Python 3 source transformer so we can eliminate it. Differential Revision: https://phab.mercurial-scm.org/D7007

py3: stop implicitly importing unicode We should be pycompat.unicode everywhere. It turns out we were doing this everywhere except for one place in templatefilters! Differential Revision: https://phab.mercurial-scm.org/D7006

py3: manually import pycompat.open into files that need it We want to eliminate the source transformer. Currently it inserts a `from mercurial.pycompat import ...` at the top of files to alias some builtins. This commit replaces the implicit import of `open` with an explicit import on files that need it and changes the source transformer to no longer import `open`. As part of this, we needed to store an explicit local for `open` in the Python 2 code path in `pycompat` so the import works. (Builtins that are automatically in scope cannot be imported.) Differential Revision: https://phab.mercurial-scm.org/D7005

import-checker: allow symbol imports from mercurial.pycompat Currently, the source transformer inserts `from mercurial.pycompat import delattr, getattr, hasattr, setattr, open, unicode` to the top of every file. As part of getting rid of the source transformer, we'll need to have source code call these wrappers directly. Rather than rewrite all call sites to call pycompat.*, I think it makes sense to import needed symbols via explicit imports. That requires loosening the import checker to allow this. Differential Revision: https://phab.mercurial-scm.org/D7004

contrib: add a fork of black (as "grey") that includes my changes This is black with https://github.com/psf/black/pull/826 applied as of today. The current git hash of black master is d9e71a75ccfefa3d9156a64c03313a0d4ad981e5, and the hash of my commit is dc1add6e94e212eff37bb3619e1422fb3c6d8dc8. In order to use this, you need to install `black` (from github master) and `typed-ast` using pip, preferably into python3, and then you can run `grey.py` with that Python and you'll have my patched version of black, which is how we've been formatting the codebase. Once my PR is merged, I'll follow up by removing this fork and updating instructions in the example config. # no-check-commit bad style Differential Revision: https://phab.mercurial-scm.org/D7002

rust-cpython: change license of ref_sharing.rs to MIT Since we plan to upstream this feature, it's better to continue further refactoring under the same license as rust-cpython crate. According to the file history, copyright holders are: - Raphaël Gomès <rgomes@octobus.net> - Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> - Yuya Nishihara <yuya@tcha.org>

contrib: fix check-code to be able to detect missing _() with bytestrings Differential Revision: https://phab.mercurial-scm.org/D6997

cleanup: mark some ui.(status|note|warn|write) calls as not needing i18n These used to be marked with no-op parens, but black removes those now and this is more explicit. # skip-blame: fallout from mass reformatting Differential Revision: https://phab.mercurial-scm.org/D6996

ui: define (write|status|warn|note)noi18n aliases We currently use `write(('...'))` to suppress check-code warnings about not using translated strings. However, when we run black, it will strip the `((...))`. In order to placate black, we'll need to use a different mechanism to pass untranslatable strings. This commit introduces a `writenoi18n` alias (and friends) to `write` for that purpose. Differential Revision: https://phab.mercurial-scm.org/D6994

mangler: stop rewriting string constants to be bytes literals We've rewritten everything, so we no longer require this step. Differential Revision: https://phab.mercurial-scm.org/D6973

formatting: byteify all mercurial/ and hgext/ string literals Done with python3.7 contrib/byteify-strings.py -i $(hg files 'set:mercurial/**.py - mercurial/thirdparty/** + hgext/**.py - hgext/fsmonitor/pywatchman/** - mercurial/__init__.py') black -l 80 -t py33 -S $(hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**" - hgext/fsmonitor/pywatchman/**') # skip-blame mass-reformatting only Differential Revision: https://phab.mercurial-scm.org/D6972

formatting: blacken the codebase This is using my patch to black (https://github.com/psf/black/pull/826) so we don't un-wrap collection literals. Done with: hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**"' | xargs black -S # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D6971

style: run a patched black on a subset of mercurial This applied black to the 20 smallest files in mercurial/: ls -S1 mercurial/*.py | tail -n20 | xargs black --skip-string-normalization Note that a few files failed to format, presumably due to a bug in my patch. The intent is to be able to compare results to D5064 with https://github.com/python/black/pull/826 applied to black. I skipped string normalization on this patch for clarity - in reality I think we'd want one pass without string normalization, followed by another to normalize strings (which is basically replacing ' with " globally.) # skip-blame mass-reformatting only Differential Revision: https://phab.mercurial-scm.org/D6342

tests: conditionalize output for Python 3 It appears that the random.randint() behavior is different between Python 2 and Python 3. So make the test conditional on that. This makes the test pass on Python 3.7 (and presumably other Python 3 versions). Differential Revision: https://phab.mercurial-scm.org/D6964

hg: remove HGUNICODEPEDANTRY and RTUNICODEPEDANTRY This was added in 73e4a02e6d23 and 89822d7a9d5f as a preliminary way to better support Python 3. With the Python 3 port nearly done and better procedures in place, we shouldn't need it. Differential Revision: https://phab.mercurial-scm.org/D6991

convert: use pycompat.fsencode() This avoids a `TypeError: Can't mix strings and bytes in path components`. Differential Revision: https://phab.mercurial-scm.org/D6965

tests: make tarball output conditional on Python version Python 3.8 changed the default tar file archive format: https://docs.python.org/3.8/whatsnew/3.8.html#tarfile. This commit teaches our tests about the new behavior. Differential Revision: https://phab.mercurial-scm.org/D6986

tests: conditionalize test-run-tests.t for Python 3.8 It looks like Python 3.8 changed the pretty XML formatting slightly to strip some whitespace. Let's teach our tests about that. Differential Revision: https://phab.mercurial-scm.org/D6988

hgweb: use importlib.reload() if available reload() was nuked in Python 3. We need to use importlib.reload() instead. But pyflakes isn't smart enough to detect our conditional usage, so we allow this error. Differential Revision: https://phab.mercurial-scm.org/D6992

tests: allow xrange warning from perf.py The use of xrange in this file is acceptable. We need to make the output optional - and not conditional on the Python version - because `pyflakes` could be executed by any Python version and pyflakes behaves differently depending on the Python version. Differential Revision: https://phab.mercurial-scm.org/D6990

debugsidedata: fix verbose mode on python3 We need to be gently with python3 for it to display this binary data. Differential Revision: https://phab.mercurial-scm.org/D6995

import: add debug messages when parsing data from patch header

examples: include a sample of how to use black with fix It's commented out for now since my patch hasn't landed, but we can uncomment it when that lands. Differential Revision: https://phab.mercurial-scm.org/D6974