highlight: fix encoding issues to enable Py3 compatibility This commit fixes various encoding issues with the `highlight` extension to enable compatibility with Python 3. Python `.encode()` and `.decode()` requires the target encoding to be passed as a `str`, so the value of `mercurial.encoding.encoding` must be converted before passing to the function. Pygments also assumes the `str` type for values it works with, so we must perform conversions before and after receiving values from its APIs. After applying this patch, `test-highlight.t` passes under Python 3. We add it to `python3-whitelist` as well. Tested with Pygments 2.4.2. Differential Revision: https://phab.mercurial-scm.org/D6832

hgweb: add a `message` attribute to `hgweb.common.ErrorResponse` This fixes a Python 3 bug where hgweb assumes an Exception subclass will have a `.message` attribute after running `Exception.__init__`.[1] The Python 3 way to get this info would be `e.args[0]`, but adding a new named attribute is more ergonomic in my view. [1] https://www.mercurial-scm.org/repo/hg-committed/file/6ccf539aec71/mercurial/hgweb/hgwebdir_mod.py#l459 Differential Revision: https://phab.mercurial-scm.org/D6840

uncommit: make -D/--date and -U/--user mutually exclusive This is how amend and graft work (but not MQ). I'm not sure why this didn't work for me when I first tried it. Differential Revision: https://phab.mercurial-scm.org/D6842

uncommit: drop the hyphen from --current-user and --current-date I didn't pay enough attention to these long forms- graft, amend and MQ already use the old style naming. It's probably more important to be consistent than modern. The hypenated style came from evolve. Yuya mentioned this naming discrepancy in 4145fd3569c3, but it didn't attract any discussion[1]. There's also a bit of inconsistency in that the default parameter for `currentdate` is `False` for graft, and `None` for the rest. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-January/126767.html Differential Revision: https://phab.mercurial-scm.org/D6841

hgweb: fix websub regex flag syntax on Python 3 The `websub` config section for hgweb is broken under Python 3 when using regex flags syntax (ie the optional `i` in the example from `hg help config.websub`: patternname = s/SEARCH_REGEX/REPLACE_EXPRESSION/[i] Flags are pulled out of the specified byte-string using a regular expression, and uppercased. The flags are then iterated over and passed to the `re` module using `re.__dict__[item]`, to get the object attribute of the same name from the `re` module. So on Python 2 if the `il` flags are passed, this transition looks like: `'il'` -> `'IL'` -> `'I'` -> `re.__dict__['I']` -> `re.I` However on Python 3, these are bytes objects. When we iterate over a bytes object in Python 3, instead of getting the individual characters in the string as string objects of length one, we get the integer \ value corresponding to that byte. So the same transition looks like: `b'il'` -> `b'IL'` -> `73` -> `re.__dict__[73]` -> `KeyError` This commit fixes the type mismatch by converting the bytes to a system string before iterating over each element to pass to `re`. The transition will now look like: `b'il'` -> `u'IL'` -> `u'I'` -> `re.__dict__[u'I']` -> `re.I` In addition we expand `test-websub.t` to cover the regex flag case (for both the `websub` section and `interhg`). Differential Revision: https://phab.mercurial-scm.org/D6788

merge with stable

phabricator: don't abort if property writing fails during amending Currently if one of the writediffproperty calls fails due to network issues during the amending of commit messages to include the Diff. Rev. line, the transaction is aborted and rolled back. This means that the associations between the commits and the DREVs are lost for any already amended commits because the removal of the local tags isn't covered by the rollback. Differential Revision: https://phab.mercurial-scm.org/D6835

relnotes: we now require `sh` to support $(command) syntax to run test suite For example, Solaris before version 11 had /bin/sh pointing to the old Bourne Shell (which doesn't support $(command) syntax). Differential Revision: https://phab.mercurial-scm.org/D6833

merge: respect parents order when using `graft` on a merge The previous code did not record the index of the replaced parent. It was always using the "graft" destination as `p1`. This could switch parents order in some situation (eg: some of the evolve evolving merge case). Recording and using the information fixes the issue in evolve. We are not aware of core commands calling graft in that fashion, so we could not build a simple test case for it using core commands.

doc: fix up confusing doc comment Differential Revision: https://phab.mercurial-scm.org/D6829

strip: fix bug with treemanifests and unordered linkrevs This is the treemanifest version of f45f7390c1c5 (strip: calculate list of extra nodes to save and pass it to changegroupsubset, 2008-01-19). Differential Revision: https://phab.mercurial-scm.org/D6795

repair: extract a helper for generating all manifest revlogs We'll need to walk the manifest revlogs also to figure out which manifests have linkrevs out of order (for fixing the bug shown in the previous patch). By the way, perhaps it would be more efficient in many cases to find only the relevant directory manifest revlogs based on the files instead of walking the entire store, but that can be changed later. Differential Revision: https://phab.mercurial-scm.org/D6794

tests: show broken strip with treemanifests and unordered linkrevs This is the treemanifest version of issue764. Differential Revision: https://phab.mercurial-scm.org/D6793

tests: split out manifest case from test-strip-cross.t The manifest case was added on after the other cases, in d67cfe0d4714 (test-strip-cross: test handling of linkrev crosses in the manifest, 2008-01-20). I think it's easier to read and modify if it's separated. Differential Revision: https://phab.mercurial-scm.org/D6792

tests: don't log manifest-file in test-strip-cross.t I'm confident that the file is there only to help produce a certain manifest log; there is nothing special about the file's filelog itself. (And there already is a `hg debugindex --manifest` call higher up in the file that shows the crossed linkrevs.) Differential Revision: https://phab.mercurial-scm.org/D6791

tests: use positive revision numbers in test-strip-cross.t It took me a long time to realize that '-r -1' was pulling revision -1 because I didn't even notice the minus sign until I tried changing the revision number. The order of arguments doesn't matter, so I changed from decreasing order to increasing while at it. Differential Revision: https://phab.mercurial-scm.org/D6790

automation: implement "publish-windows-artifacts" command The new command and associated functionality can be used to automate the publishing of Windows release artifacts. It supports uploading wheels to PyPI (using twine) and copying the artifacts to mercurial-scm.org and updating the latest.dat file to advertise them via the website. I ran `automation.py publish-windows-artifacts 5.1.1` and it appeared to "just work." But the real test will be to do this on the next release... Differential Revision: https://phab.mercurial-scm.org/D6786

automation: upgrade to latest packages in requirements.txt Let's stay modern. Differential Revision: https://phab.mercurial-scm.org/D6785

localrepo: push manifestlog and changelog construction code into store This feels substantially more appropriate, as the store is actually the layer with knowledge of how to handle this storage. I didn't move the caching decorators for now because that's going to require some more involved work, and this unblocks my current experimentation. Differential Revision: https://phab.mercurial-scm.org/D6732

notify: add option for deterministic message-id generation Copied from email by durin42: > Pierre-Yves asked offline why I asked a new option for this and why it > is not the default. Message-Id is supposed to be unique world-wide and > while it is desirable to have a deterministic mechanism for them for > creating follow-up emails, it needs organisational control for ensuring > the uniqueness. Differential Revision: https://phab.mercurial-scm.org/D6824

uncommit: add options to update to the current user or current date These are also from the evolve extension's version of uncommit. I tried adding validation that both forms of user or date can't be specified at the same time, but that fails because these show up in `opts` with a None value whether or not the option was given on the command line. Presumably that means the conditional in `resolvecommitoptions` could be simplified. But this is how both evolve and MQ handle it. Differential Revision: https://phab.mercurial-scm.org/D6828

uncommit: add support to modify the commit message and date Currently, the evolve extension's version of this command supports it. Differential Revision: https://phab.mercurial-scm.org/D6827

run-tests: add a dedicated 'isoptional' function This is clearer than repeated manual call to to 'endswith'. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: remove the artificial indentation

run-tests: extract a `process_out_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. To avoid a huge diff, the code is kept at the same indentation. We'll re-indent in the next changesets. (This is a gratuitous cleanup that I made while investigating a bug).

run-tests: extract a `process_cmd_line` from the main function The main function doing line comparison is quite complex. Slicing it in smaller piece should clarify it. (This is a gratuitous cleanup that I made while investigating a bug).

changegroup: move message about added changes to transaction summary Before that, applying multiple changegroups in the same transaction issued the message multiple time. This result in a confusing output: adding changesets adding manifests adding file changes added 32768 changesets with 60829 changes to 2668 files adding changesets adding manifests adding file changes added 8192 changesets with 16885 changes to 1553 files adding changesets adding manifests adding file changes added 1020 changesets with 1799 changes to 536 files adding changesets adding manifests ... Instead, we now only issue the message once at the end of the transaction, summing up all added changesets, changes and files. The line is identical, but happens sightly later in the output. There are other suboptimal behavior around issue multiple changegroup (eg: progress bar). We'll cover them later. This impact of lot of test as one would expect, but a two pass check show they are just the order change we expected. To deal with "under the hood" bundle application by internal code, we had to take a slightly hacky move. We could clean that up with a more official way to enter "under the hood" section, however I want to keep this series simple to get it landed. This kind of change have a very high bit rot rate since it impact a lot of test output.

sshserver: flush stream after command dispatch I am not sure why this is not working as expected, but without this client might not see some important output. Without this patch moving some output at transaction closing time makes it disapear for ssh client in various sitaution.