narrow: add option for automatically removing unused includes It's been a somewhat common request among our users to have Mercurial automatically pick includes to remove. This patch adds an option for that: `hg tracked --auto-remove-includes`. I'm not sure if this is the right name and semantics for it. Perhaps the feature should also add excludes of large subdirectories even if other files in the include are needed? Narrow clones are experimental, so we can change the name and/or semantics later if necessary. Differential Revision: https://phab.mercurial-scm.org/D6848

narrow: don't hexify paths and double-hexify known nodes on wire (BC) It isn't obvious, but wireprototypes.encodelist() is meant only for binary nodeids. So when we used it for encoding hex nodeids and paths, the encoded result was surprising and hard to read. This patch changes the encoding to make the list of paths a comma-separated list and the list of common nodes to be a encodelist()-encoded list of binary nodeids (so the result is just singly-hexified nodeids). This is clearly a breaking change, but the feature is experimental and we're not aware of anyone running a server using this command yet. Differential Revision: https://phab.mercurial-scm.org/D6851

remotefilelog: replace repack lock to solve race condition 2c74337e6483 reduced the probability of race-conditions when starting background repack and prefetch and we saw the difference in our CI instance with all failures disappearing except one where one call to waitonrepack seems to returns too early. I'm not sure what exactly goes wrong but I realized that while the prefetch operation uses a standard Mercurial lock, the repack operation is using a custom lock based on `fcntl.flock` on available platforms. As `extutil.flock` fallback on traditional Mercurial locks on other platforms and the tests are stable on my laptop, our CI environment and GCC112, I'm sending this patch to standardize the behavior across environments. Differential Revision: https://phab.mercurial-scm.org/D6844

perf: add a --stats argument to perfhelper-pathcopies The arguments will display some statisting about the distribution of the value we measure.

perf: add a --stats argument to perfhelper-mergecopies The arguments will display some statistics about the distribution of the value we measure.

archive: add XZ support if built with Python 3

rust-cpython: add sanity check to PySharedState::decrease_leak_count() If decrease_leak_count() were called unnecessarily, there must be a serious bug. It's better to not silently ignore such cases.

tests: stabilize test-fix.t on Windows `pwd` prints /tmp/... style paths, not C:\... needed for $TESTTMP to be substituted. In the final test, for whatever reason, Windows was missing EOL in the files and printing: [wdir] changedlines: printf: warning: ignoring excess arguments, starting with 'printf' even though it was trying to run: printf "Line ranges:\n"; printf "2 through 2\n"; I tried wrapping both :command and :linerange in `sh -c "..."`, and while that fixed the missing EOL, it missed the "2 through 2" output. Differential Revision: https://phab.mercurial-scm.org/D6852

zstandard: vendor python-zstandard 0.12 The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. test-repo-compengines.t was updated to reflect a change in behavior of the zstd library. The project contains a vendored copy of zstandard 1.4.3. The old version was 1.3.8. This should result in some minor performance wins. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D6858

uncommit: enable support for adding a note This comes from the evolve extension's version of uncommit. The logic was already in place, and appears to be the last of the trivial things that can be enabled. Should these note options (including on amend) be marked advanced to keep the help text clutter level down? Differential Revision: https://phab.mercurial-scm.org/D6857

amend: enable support for using the secret phase This comes from the evolve extension's version of amend. The logic was already in place, and appears to be the last of the trivial things that can be enabled. Differential Revision: https://phab.mercurial-scm.org/D6856

amend: enable support for closing the branch This comes from the evolve extension's version of amend. The logic was already in place. Differential Revision: https://phab.mercurial-scm.org/D6855

amend: prevent '\n' in the note string This comes from the evolve function. I'm not sure why this check was missing in core, since it was present when the length check was added to evolve. I didn't flag this as BC because 530b7361e3a9 mentioned this argument wasn't added to the release notes due to no display capability, and that hasn't changed AFAIK. Differential Revision: https://phab.mercurial-scm.org/D6854

amend: add option to update to the current user This is also from the evolve extension's version of amend. A side effect of this refactoring is for uncommit to support `rewrite.update-timestamp`. Differential Revision: https://phab.mercurial-scm.org/D6853

bundle2: fix an off-by-one in debug message of number of parts Differential Revision: https://phab.mercurial-scm.org/D6850

tests: move a config write to top of file since it applies to all tests I'm about to add another test that depends on this config. Differential Revision: https://phab.mercurial-scm.org/D6849

idirstate: group private methods and attrs that are in the interface This makes it a little more obvious at a glance what work is left. Fortunately there's not a whole lot left. I suspect the ignore logic is going to be the tricky bit. Differential Revision: https://phab.mercurial-scm.org/D6839

idirstate: remove now non-public _map attribute Differential Revision: https://phab.mercurial-scm.org/D6838

interfaces: introduce an interface for dirstate implementations As usual with adding interface definitions, this describes the way things are, not the way we'd like things to be. There are some clear problems in the interface right now (eg ._map leaks in a few places), but I have plans to clean those up. There are also many missing docstrings, but again, we'll make a second pass to clean that up. Differential Revision: https://phab.mercurial-scm.org/D6836

cleanup: fix leakage of dirstate._map to client code We already had proper accessors for most of the behavior of dirstate._map that callers cared about exposed in the actual dirstate class as public methods. Sigh. There are two remaining privacy violations in the codebase after this change: 1) In the perf extension, which I suspect has to stick around because it's really testing the dirstate implementation directly 2) In largefiles, where we deal with standins and mutating status. Looking at this, I _strongly_ suspect a formal dirstate interface would allow this to work via composition instead of inheritance and monkeypatching. Fortunately, such wins are a part of my motivation for this work. I anticipate we'll come back to this in due time. Differential Revision: https://phab.mercurial-scm.org/D6837

exchange: convert bookmark nodes from hex to bin ASAP Differential Revision: https://phab.mercurial-scm.org/D6831

exchange: avoid unnecessary conversion of bookmark nodes to hex (API) Differential Revision: https://phab.mercurial-scm.org/D6830

highlight: fix encoding issues to enable Py3 compatibility This commit fixes various encoding issues with the `highlight` extension to enable compatibility with Python 3. Python `.encode()` and `.decode()` requires the target encoding to be passed as a `str`, so the value of `mercurial.encoding.encoding` must be converted before passing to the function. Pygments also assumes the `str` type for values it works with, so we must perform conversions before and after receiving values from its APIs. After applying this patch, `test-highlight.t` passes under Python 3. We add it to `python3-whitelist` as well. Tested with Pygments 2.4.2. Differential Revision: https://phab.mercurial-scm.org/D6832

hgweb: add a `message` attribute to `hgweb.common.ErrorResponse` This fixes a Python 3 bug where hgweb assumes an Exception subclass will have a `.message` attribute after running `Exception.__init__`.[1] The Python 3 way to get this info would be `e.args[0]`, but adding a new named attribute is more ergonomic in my view. [1] https://www.mercurial-scm.org/repo/hg-committed/file/6ccf539aec71/mercurial/hgweb/hgwebdir_mod.py#l459 Differential Revision: https://phab.mercurial-scm.org/D6840

uncommit: make -D/--date and -U/--user mutually exclusive This is how amend and graft work (but not MQ). I'm not sure why this didn't work for me when I first tried it. Differential Revision: https://phab.mercurial-scm.org/D6842

uncommit: drop the hyphen from --current-user and --current-date I didn't pay enough attention to these long forms- graft, amend and MQ already use the old style naming. It's probably more important to be consistent than modern. The hypenated style came from evolve. Yuya mentioned this naming discrepancy in 4145fd3569c3, but it didn't attract any discussion[1]. There's also a bit of inconsistency in that the default parameter for `currentdate` is `False` for graft, and `None` for the rest. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-January/126767.html Differential Revision: https://phab.mercurial-scm.org/D6841

hgweb: fix websub regex flag syntax on Python 3 The `websub` config section for hgweb is broken under Python 3 when using regex flags syntax (ie the optional `i` in the example from `hg help config.websub`: patternname = s/SEARCH_REGEX/REPLACE_EXPRESSION/[i] Flags are pulled out of the specified byte-string using a regular expression, and uppercased. The flags are then iterated over and passed to the `re` module using `re.__dict__[item]`, to get the object attribute of the same name from the `re` module. So on Python 2 if the `il` flags are passed, this transition looks like: `'il'` -> `'IL'` -> `'I'` -> `re.__dict__['I']` -> `re.I` However on Python 3, these are bytes objects. When we iterate over a bytes object in Python 3, instead of getting the individual characters in the string as string objects of length one, we get the integer \ value corresponding to that byte. So the same transition looks like: `b'il'` -> `b'IL'` -> `73` -> `re.__dict__[73]` -> `KeyError` This commit fixes the type mismatch by converting the bytes to a system string before iterating over each element to pass to `re`. The transition will now look like: `b'il'` -> `u'IL'` -> `u'I'` -> `re.__dict__[u'I']` -> `re.I` In addition we expand `test-websub.t` to cover the regex flag case (for both the `websub` section and `interhg`). Differential Revision: https://phab.mercurial-scm.org/D6788

merge with stable

relnotes: we now require `sh` to support $(command) syntax to run test suite For example, Solaris before version 11 had /bin/sh pointing to the old Bourne Shell (which doesn't support $(command) syntax). Differential Revision: https://phab.mercurial-scm.org/D6833

doc: fix up confusing doc comment Differential Revision: https://phab.mercurial-scm.org/D6829

strip: fix bug with treemanifests and unordered linkrevs This is the treemanifest version of f45f7390c1c5 (strip: calculate list of extra nodes to save and pass it to changegroupsubset, 2008-01-19). Differential Revision: https://phab.mercurial-scm.org/D6795

repair: extract a helper for generating all manifest revlogs We'll need to walk the manifest revlogs also to figure out which manifests have linkrevs out of order (for fixing the bug shown in the previous patch). By the way, perhaps it would be more efficient in many cases to find only the relevant directory manifest revlogs based on the files instead of walking the entire store, but that can be changed later. Differential Revision: https://phab.mercurial-scm.org/D6794

tests: show broken strip with treemanifests and unordered linkrevs This is the treemanifest version of issue764. Differential Revision: https://phab.mercurial-scm.org/D6793

tests: split out manifest case from test-strip-cross.t The manifest case was added on after the other cases, in d67cfe0d4714 (test-strip-cross: test handling of linkrev crosses in the manifest, 2008-01-20). I think it's easier to read and modify if it's separated. Differential Revision: https://phab.mercurial-scm.org/D6792

tests: don't log manifest-file in test-strip-cross.t I'm confident that the file is there only to help produce a certain manifest log; there is nothing special about the file's filelog itself. (And there already is a `hg debugindex --manifest` call higher up in the file that shows the crossed linkrevs.) Differential Revision: https://phab.mercurial-scm.org/D6791

tests: use positive revision numbers in test-strip-cross.t It took me a long time to realize that '-r -1' was pulling revision -1 because I didn't even notice the minus sign until I tried changing the revision number. The order of arguments doesn't matter, so I changed from decreasing order to increasing while at it. Differential Revision: https://phab.mercurial-scm.org/D6790

automation: implement "publish-windows-artifacts" command The new command and associated functionality can be used to automate the publishing of Windows release artifacts. It supports uploading wheels to PyPI (using twine) and copying the artifacts to mercurial-scm.org and updating the latest.dat file to advertise them via the website. I ran `automation.py publish-windows-artifacts 5.1.1` and it appeared to "just work." But the real test will be to do this on the next release... Differential Revision: https://phab.mercurial-scm.org/D6786

automation: upgrade to latest packages in requirements.txt Let's stay modern. Differential Revision: https://phab.mercurial-scm.org/D6785

localrepo: push manifestlog and changelog construction code into store This feels substantially more appropriate, as the store is actually the layer with knowledge of how to handle this storage. I didn't move the caching decorators for now because that's going to require some more involved work, and this unblocks my current experimentation. Differential Revision: https://phab.mercurial-scm.org/D6732

notify: add option for deterministic message-id generation Copied from email by durin42: > Pierre-Yves asked offline why I asked a new option for this and why it > is not the default. Message-Id is supposed to be unique world-wide and > while it is desirable to have a deterministic mechanism for them for > creating follow-up emails, it needs organisational control for ensuring > the uniqueness. Differential Revision: https://phab.mercurial-scm.org/D6824

uncommit: add options to update to the current user or current date These are also from the evolve extension's version of uncommit. I tried adding validation that both forms of user or date can't be specified at the same time, but that fails because these show up in `opts` with a None value whether or not the option was given on the command line. Presumably that means the conditional in `resolvecommitoptions` could be simplified. But this is how both evolve and MQ handle it. Differential Revision: https://phab.mercurial-scm.org/D6828

uncommit: add support to modify the commit message and date Currently, the evolve extension's version of this command supports it. Differential Revision: https://phab.mercurial-scm.org/D6827

run-tests: add a dedicated 'isoptional' function This is clearer than repeated manual call to to 'endswith'. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: remove the artificial indentation

run-tests: extract a `process_out_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. To avoid a huge diff, the code is kept at the same indentation. We'll re-indent in the next changesets. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: extract a `process_cmd_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. (This is a gratuitous cleanup that I made while investigating a bug).

changegroup: move message about added changes to transaction summary Before that, applying multiple changegroups in the same transaction issued the message multiple time. This result in a confusing output: adding changesets adding manifests adding file changes added 32768 changesets with 60829 changes to 2668 files adding changesets adding manifests adding file changes added 8192 changesets with 16885 changes to 1553 files adding changesets adding manifests adding file changes added 1020 changesets with 1799 changes to 536 files adding changesets adding manifests ... Instead, we now only issue the message once at the end of the transaction, summing up all added changesets, changes and files. The line is identical, but happens sightly later in the output. There are other suboptimal behavior around issue multiple changegroup (eg: progress bar). We'll cover them later. This impact of lot of test as one would expect, but a two pass check show they are just the order change we expected. To deal with "under the hood" bundle application by internal code, we had to take a slightly hacky move. We could clean that up with a more official way to enter "under the hood" section, however I want to keep this series simple to get it landed. This kind of change have a very high bit rot rate since it impact a lot of test output.

sshserver: flush stream after command dispatch I am not sure why this is not working as expected, but without this client might not see some important output. Without this patch moving some output at transaction closing time makes it disapear for ssh client in various sitaution.

narrow: rely on setting `quiet` mode instead of `pushbuffer` The `quiet` approach is what `shelve` uses and give the same result. This will help us to add less code in future changesets.

transaction: issue "new obsmarkers" message at the end of the transaction Instead of making bundle2 code responsible for this, it seems better to have it handled and the transaction level. First, it means the message will be more consistently printed. Second it means we won't spam the message over and over if the data arrive in multiple piece. Third, we are planning to move other similar message at the same level (for the same reason) so having them all at the same location will help us to control the order they are displayed.

debugobsolete: also issue the "new obsmarkers" messsage We are going to improve the way this message is issued in the core codebase. This will make it appears for `hg debugobsolete` too. Since this seems like a good idea, we make the output change in a previous changesets to clarify the next changeset.

split: use literal syntax to build a set of one element

rust-cpython: leverage py_shared_iterator::from_inner() where appropriate

rust-cpython: remove Option<_> from interface of py_shared_iterator It's the implementation detail of the py_shared_iterator that the leaked reference is kept in Option<_> so that it can be dropped early.

rust-cpython: rename py_shared_iterator_impl to py_shared_iterator It's a public interface now.

rust-cpython: replace dyn Iterator<..> of mapping with concrete type See the previous commit for why. The docstring is moved accordingly.

rust-cpython: replace dyn Iterator<..> of sequence with concrete type We wouldn't care the cost of the dynamic dispatch, but I feel a concrete type helps understanding error messages.

rust-dirstate: provide CopyMapIter and StateMapIter types They will be used in the declaration of Python iterator types.

rust-dirstate: specify concrete return type of DirsMultiset::iter() This allows us to put a returned iterator in a struct. We could implement DirsMultisetIter(hash_map::Keys<..>) struct to hide the implementation detail, but I think type alias is good enough for us.

discovery: replace "heads" by "changesets" in a output note (BC) When using `hg push --rev X`, the subset considered by discovery is only `::X`. In addition, `X` can be any local revisions not just local heads. As a result the message "all local heads known locally" can be misleading. We replace it with the more accurate "all local changesets known remotely". The message appears when in verbose not, so this is stricly speaking a BC breakage. I am not sure this would be a real issue in practice...

py3: drop incorrect fsencode(findexe(...)) in procutil I recently added the bad call, thinking that findexe() was a standard library function returning a string result, but it's actually our own function returning bytes. Thanks to Yuya for noticing. Differential Revision: https://phab.mercurial-scm.org/D6826

flagprocessors: small code update to clarify parameters 'raw' is really a third mode, not a small variant. Differential Revision: https://phab.mercurial-scm.org/D6807

flagprocessors: deprecate _processflags People should use the specialized version instead now. Differential Revision: https://phab.mercurial-scm.org/D6806

simplestorerepo: stop using `_processflags` directly We now use the specialized versions. Differential Revision: https://phab.mercurial-scm.org/D6805

revlog: stop using `_processflags` directly We now use the specialized versions. Differential Revision: https://phab.mercurial-scm.org/D6804

flagprocessors: use _processflagsraw in easy cases When there are no ambiguity regarding the `raw` value, we can simply use the new method. Differential Revision: https://phab.mercurial-scm.org/D6803

flagprocessors: use _processflagsread in simple cases When there are no ambiguity regarding the `raw` value, we can simply use the new method. Differential Revision: https://phab.mercurial-scm.org/D6802

flagprocessors: use _processflagswrite for write operation There are no ambiguity for 'write' operation so it is simple to replace. Differential Revision: https://phab.mercurial-scm.org/D6801

flagprocessors: introduce specialized functions This make the call site clearer and the open the way to more diverse return types. For now, the same old code is still in use under the hood. Differential Revision: https://phab.mercurial-scm.org/D6800

flagutil: use it in simplestorerepo This remove the other code duplication of the `_processflags` code. To be honest, this code looks very dead since it is not run by either developer nor buildbot. However, we update the code to make the task of reviving this less daunting. Differential Revision: https://phab.mercurial-scm.org/D6799

flagutil: make the error class used by the mixin configurable One of the code duplication use a different error class. So let's make it possible to do so. Differential Revision: https://phab.mercurial-scm.org/D6798

flagutil: use the new mixin use in remotefilelog This remove one of the code duplication. Hooray \o/. Differential Revision: https://phab.mercurial-scm.org/D6797

flagutil: introduce a flagprocessorsmixin class To avoid code duplication, we will provide a simple "ready to use" mixin that carry the appropriate logic. First we use it in standard revlog, we'll remove code duplication in later changesets. Differential Revision: https://phab.mercurial-scm.org/D6796

check-code: allow command substitution with $(command) Both `command` and $(command) are specified by POSIX. The latter nests better. I don't see why we shouldn't allow both (or only the latter). Differential Revision: https://phab.mercurial-scm.org/D6789

run-tests: rename `lcmd` variable to `line_cmd` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: rename `lout` variable to `out_line` This is clearer and more in line with some other variable names. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: clarify "l" variable as "out_rawline" More explicit variable name never hurt. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: use symbolic constant instead of arbitrary number line matching (This is a gratuitous cleanup that I made while investigating a bug).

rustfilepatterns: shorter code for concatenating slices Differential Revision: https://phab.mercurial-scm.org/D6765

match: simplify the regexps created for glob patterns For legibility of the resulting regexes, although it may help with performance as well. Differential Revision: https://phab.mercurial-scm.org/D6764

rustfilepatterns: refactor the pattern of removing a prefix from a &[u8] Differential Revision: https://phab.mercurial-scm.org/D6766

tests: show the pattern generated for a relative glob Differential Revision: https://phab.mercurial-scm.org/D6763

copies: remove existing copy info from the changeset on amend (BC) When amending a changeset with copy information in the changeset and the new changeset doesn't have any copy information (or similar for "filesadded" and "filesremoved"), we shouldn't keep it. A drawback of this is that we now unconditionally remove these four entries from the extras, breaking any extensions that happened to write entries with the same names (which seems very unlikely). I think I'd heard that there was list of blacklisted keys that would be removed from the extras when a commit is rewritten, but I couldn't find that. It would make sense to add the keys mentioned above there instead of the custom filtering I've added in this patch. Differential Revision: https://phab.mercurial-scm.org/D6752

tests: show invalid copies when turning off copies-in-changeset If you turn on copies in changesets and write a commit with a copy, then turn it off and amend the commit while undoing the copy, the invalid copy information will remain. The read path doesn't crash in invalid copy data, but it seems better to not produce the invalid data. Differential Revision: https://phab.mercurial-scm.org/D6751

context: filter out invalid copies from workingctx.p[12]copies() workingctx normally gets its lists of modified, added, removed files etc. based on the dirstate status. Its constructor also accepts a "changes" argument to override the status from the dirstate. This is used for partial commits. If a "changed" argument was passed, we should clearly also filter out copies to those paths, which I had previously missed. This patch adds that filtering and fixes the bugs demonstrated in the previous patch. Differential Revision: https://phab.mercurial-scm.org/D6750

tests: demonstrate crash when committing subset of copies to changeset When writing copy metadata to the changeset and not committing all copies in the dirstate, we get a ProgrammingError. This commit adds two tests showing how to trigger this bug. Differential Revision: https://phab.mercurial-scm.org/D6749

bdiff-torture: fix pyflakes warning reporting undefined name 'inst' Looks like I got a latest version of pyflakes somehow or it's running on py3 and it spotted this. Differential Revision: https://phab.mercurial-scm.org/D6757

split: handle partial commit of renames when doing split or record (issue5723) When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This was because the logic for handling partial application of the patches relies on knowing what files are "new with modifications" and it doesn't treat "rename destination" as "new". There was a complicating issue, however. We're relying on the patch header specifying the copy from/to information, which works as long as the 'copy from' file is there. In the case of renames, however, the 'rename from' file is *not* there, so we need to add it back. Differential Revision: https://phab.mercurial-scm.org/D6768

split: handle partial commit of copies when doing split or record When using split or record, using either interface (text or curses), selecting portions of the file to be committed/recorded did not work; the entire file was treated as having been selected. This appears to be because the logic for handling partial application of the patches relies on knowing what files are "new with modifications", and it doesn't treat "copy destination" as "new". Handling renames correctly is more difficult and will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D6767

py3: use pycompat.sysargv[0] for instead of fsencode(sys.argv[0]) Yuya noted in a recent review that fsencode(sys.argv[0]) could be incorrect on Windows. Differential Revision: https://phab.mercurial-scm.org/D6782

httppeer: use context manager when reading temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6784

httppeer: use context manager when writing temporary bundle to send Differential Revision: https://phab.mercurial-scm.org/D6783

rust-cpython: mark unsafe functions as such It wasn't trivial to fix leak_immutable() to be safe since we have to allow immutable operations (e.g. iter()) on the leaked reference. So let's mark it unsafe for now. Callers must take care of the returned object to guarantee the memory safety. I'll revisit this later. I think $leaked<T: 'static> could have a function that converts itself into $leaked<U: 'static> with a given FnOnce(&T) -> &U, where T is $inner_struct, and U is $iterator_type for example.

rust-cpython: pair leaked reference with its manager object Still leak_immutable() is unsafe since leak_handle must live longer than the leaked_ref.

rust-cpython: introduce restricted variant of RefCell This should catch invalid borrow_mut() calls. Still the ref-sharing interface is unsafe.

rust-cpython: fix unsafe inner(py).borrow_mut() calls Since self.inner is managed by PySharedState, it must not be borrowed mutably through the RefCell interface. Otherwise, the underlying object could be mutated while a reference is leaked to Python world.