phabricator: add the uploadfile function This is needed to be able to submit binary files such as images in commits (and also non-UTF-8 text files). One of the primary disadvantages of the current use of createrawdiff is that Phabricator simply drops any binary diffs included in the patch, but not the files, so if such a diff is then committed it corrupts any binary files in it. Differential Revision: https://phab.mercurial-scm.org/D7047

phabricator: add the uploadchunks function Differential Revision: https://phab.mercurial-scm.org/D7046

phabricator: add the maketext function This add the diff data for a text file to a phabchange. Differential Revision: https://phab.mercurial-scm.org/D7045

phabricator: add the phabdiff data structure This holds all the data about a commit, and is passed to the differential.creatediff API. Differential Revision: https://phab.mercurial-scm.org/D7044

phabricator: add the phabchange data structure These store data about individual files in a commit. Differential Revision: https://phab.mercurial-scm.org/D7043

phabricator: add the phabhunk data structure These store the actual diff data (for UTF-8 text files anyway) and are equivalent to hunks in a patch file. Differential Revision: https://phab.mercurial-scm.org/D7042

phabricator: add the DiffChangeType and DiffFileType constants These are used in Phabricator change objects. There are more values but so far as I can tell we don't need them. Differential Revision: https://phab.mercurial-scm.org/D7041

help: adding a help category to narrow and remotefilelog commands Differential Revision: https://phab.mercurial-scm.org/D6998

rust-cpython: drop self.borrow_mut() in favor of PySharedRef wrapper

rust-cpython: drop self.leak_immutable() in favor of PySharedRef wrapper

rust-cpython: add safe wrapper representing shared data borrowed from PyObject PySharedRef is a tempoary wrapper around PySharedRefCell. It provides safe functions for each shared data. $shared_accessor implements a safe method to construct PySharedRefCell. This allows us to add more than once PySharedRefCell to a Python object.

rust-cpython: move $leaked struct out of macro It wasn't easy to hack the $leaked struct since errors in macro would generate lots of compile errors. Let's make it a plain struct so we can easily extend it. PyLeakedRef keeps a more generic PyObject instead of the $name struct since it no longer has to call any specific methods implemented by the $name class. $leaked parameter in py_shared_iterator!() is kept for future change.

rust-cpython: store leaked reference to PySharedState in $leaked struct I want to move it out of the macro, and allow multiple sharable objects per PyObject.

rust-cpython: mark PySharedState as Sync so &'PySharedState can be Send The goal is to store &'static PySharedState in $leaked struct, which allows us to move the $leaked struct out of the macro. Currently, it depends on inner.$data_member(py), which can't be generalized. PySharedState is Sync because any mutation or read operation is synchronized by the Python GIL, py: Python<'a>, which should guarantee that &'PySharedState can be sent to another thread.

rust-cpython: move borrow_mut() to PySharedRefCell PySharedRefCell() will host almost all py_shared public functions. This change is the first step. borrow_mut() can be safely implemented since PySharedRefCell knows its inner object is managed by its own py_shared_state.

rust-cpython: move py_shared_state to PySharedRefCell object The goal of this series is to encapsulate more "py_shared" thingy and reduce the size of the macro, which is hard to debug. Since py_shared_state manages the borrowing state of the object owned by PySharedRefCell, this change makes more sense. If a PyObject has more than one data to be leaked into Python world, each PySharedState should incref the parent PyObject, and keep track of the corresponding borrowing state.

py3: decode username and password before SMTP login smtplib.SMTP.login() requires str on Python 3. For 'password', we only need to decode when value comes from config as getpass() returns the correct type already.

py3: use socket.makefile() instead of dropped smtplib.SSLFakeFile The latter (undocumented internal) class got removed in Python 3.3. Use socket.makefile() as suggested: https://docs.python.org/3/whatsnew/3.3.html#porting-python-code

py3: call SMTP.docmd() with an str Otherwise, this always returns (502, '5.5.2 Error: command not recognized').

py3: call SMTP.has_extn() with an str Passing a bytes on Python 3 always returns False, thus starttls is not properly detected.

py3: fix sorting of obsolete markers when building bundle Last item of marker tuple (parents) is either None or tuple. Comparison thus fails on Python 3 with: TypeError: '<' not supported between instances of 'tuple' and 'NoneType' Fixing this by coercing None to the empty tuple when sorting markers in exchange._getbundleobsmarkerpart().

sidedata: rename the configuration option to `exp-use-side-data` We don't want setup configured to use the final version that would end using the experimental one while using and older version. Differential Revision: https://phab.mercurial-scm.org/D7040

py3: drop ui.flush() during interactive patch filtering With previous changeset fixing line buffering on stdout, this is no longer needed.

py3: keep stdout as defined by pycompat in procutil According to https://docs.python.org/3/library/functions.html#open, it's not possible to use 1 as buffering argument value in binary mode. This is probably why line buffering does not work well in python3. On the other hand, sys.stdout.buffer appears to be line-buffered already on python3. So by not replacing it, there should be no behavior change. This fixes buffering issue in "hg email" (confirmation prompt shown before information to be confirmed).

notify: cast hash to bytes This is needed to avoid a str/bytes mismatch when interpolating a line or 2 later. Differential Revision: https://phab.mercurial-scm.org/D7021

push: support config option to require revs be specified when running push Differential Revision: https://phab.mercurial-scm.org/D6989

py3-discovery: using plain str in stats dict rust-cpython converts automatically from Rust strings to the appropriate `str` for the target Python version. Insisting on discovery stats dict keys to be bytes hence breaks the process (this is spotted by test-setdiscovery.t). Now that byteify-strings has been run on the entire codebase, and the import transformer is not there any more, the simplest fix is to make the keys plain str again. Another possible fix would be to forcefully convert to bytes in rust-cpython code, but that feels less natural, and would probably have to be reverted down the road. Differential Revision: https://phab.mercurial-scm.org/D7039

perf: fix `perfhelper-mergecopies` report of #changesets Same as `perfhelper-pathcopies`. The previous computation `<base>::<target>` was wrong, what we actually need is `::<target> - ::<base>`. This is now fixed.

perf: fix `perfhelper-pathcopies` report of #changesets The previous computation `<base>::<target>` was wrong, what we actually need is `::<target> - ::<base>`. This is now fixed

py3: use integer division in curseschunkselector.printstring() This fixes a crash when scrolling in curses UI when refresh() is called when a float value (namely 'self.firstlineofpadtoprint', taking its value indirectly from 'self.linesprintedtopadsofar').

crecord: drop duplicated set of firstlineofpadtoprint attribute The attribute is already set a couple of lines above, in curseschunkselector.__init__().

patchbomb: use mail.Generator alias for py2/py3 compat

py3: use email.generator.BytesGenerator in patch.split() This fixes test-import.t on python3. We add Generator alias in mail module to handle python2/python3 compatibility.

py3: only flush before prompting during interactive patch filtering Follows up on c9093ae8d6c4. It's enough to flush just before each prompt.

py3: add a missing b'' prefix in test extension for chg Differential Revision: https://phab.mercurial-scm.org/D7038

fuzz: new target to fuzz jsonescapeu8fast This code just feels complicated enough we should go ahead and give it a dedicated fuzzer: we've found bugs in similar things before. Differential Revision: https://phab.mercurial-scm.org/D7034

fuzz: new fuzzer for fncache-related functions Not all of these are strictly fncache-related, but they all have th same signature and similar-enough behavior that we may as well fuzz them together. No obvious bugs for once, but these felt like they were just complicated enough to cover. Differential Revision: https://phab.mercurial-scm.org/D7033

fuzz: exercise a little more revlog code Differential Revision: https://phab.mercurial-scm.org/D7032

fuzz: new fuzzer for dirs.c This found a six-year-old bug immediately, and then I put it through a few CPU-days of time before sending it. Differential Revision: https://phab.mercurial-scm.org/D7031

dirs: fix trivial over-read of input data This code, introduced in 8c0a7eeda06d, was intentionally over-reading an input string to avoid getting a shared string object for a one-byte input. Unfortunately with an empty input (like in the case of a fuzzer getting started) this was a trivial over-read and triggered an AddressSanitizer failure. I went out of my way to make sure the code still does the copy-avoidance tricks. I don't think this change will cost us much performance since the one-character strings should be cached aggressively anyway. Differential Revision: https://phab.mercurial-scm.org/D7030

sidedatacopies: deal with upgrading and downgrading to that format This is quite useful to test this on real life data. Differential Revision: https://phab.mercurial-scm.org/D6955

sidedatacopies: move various copies related function to the copies modules We will need to access these logic form the copies module. So we move them from their higher level module to the lower level `copies` module. We cannot use them from their top level module as it would create cycles. Differential Revision: https://phab.mercurial-scm.org/D6954

sidedatacopies: read rename information from sidedata Repository using the new format now use changeset centric algorithm and read the copies information from the changelog sidedata. Differential Revision: https://phab.mercurial-scm.org/D6953

changelog: make copies related function return None or a valid value With the previous code, existing but empty value were not "decoded", leading to the method returning one of `None`, some valid value (`list` or `dict`) or `b''`. On a general basis, not explicitly checking for None is a source of bugs. Having a clean return types will help the side-data copies code in future changesets. Differential Revision: https://phab.mercurial-scm.org/D7037

test: fix zstd related output in pure tests Differential Revision: https://phab.mercurial-scm.org/D7036

sidedatacopies: get and store sidedata in the changelogrevision object The object provide a simple way to access changelog entry, we need it to also bear the sidedata value. Since the sidedata are retrieved at the same time as the revision, we can do that without extra cost. Differential Revision: https://phab.mercurial-scm.org/D6951

sidedatacopies: write copies information in sidedata when applicable If the format of the repository indicate it stores copies information into changeset's sidedata, then we actually write that information into sidedata at commit time. It will be put to use in later changesets. Currently, we store all field unconditionally, but that is likely to change in the future for the sake of efficiency. Differential Revision: https://phab.mercurial-scm.org/D6950

sidedatacopies: introduce a sidedata testcase for test-copies-in-changeset.t For now it is equivalent to the filelog case, but introducing this early helps make the comings changesets clearer. Differential Revision: https://phab.mercurial-scm.org/D6949

sidedatacopies: introduce a sidedata testcase for test-copies-unrelated.t For now it is equivalent to the filelog case, but introducing this early helps make the comings changesets clearer. Differential Revision: https://phab.mercurial-scm.org/D6948

sidedatacopies: introduce a sidedata testcase for test-copies.t For now it is equivalent to the filelog case, but introducing this early helps make the comings changesets clearer. Differential Revision: https://phab.mercurial-scm.org/D6947

sidedatacopies: teach upgrade about the new requirement The `debugformat` and `debugupgraderepo` command now detect the requirement. (upgrade to and from are not currently possible). Differential Revision: https://phab.mercurial-scm.org/D6946

sidedatacopies: add a new requirement for storing copies into sidedata The end goal is to have changesets centric sidedata information stored into changelog sidedata. This make it possible to use the changeset based copy tracing algorithm on any repository without affecting hashes. The actual implementation is coming. The feature is marked as experimental (do not use in production) until we stabilise details about the format. Differential Revision: https://phab.mercurial-scm.org/D6945

debugsidedata: small doc improvement The command has a mode to actually show the sidedata, but it wasn't documented. Differential Revision: https://phab.mercurial-scm.org/D6944

upgrade: allow for `sidedata` removal If the side-date feature is removed, we need to drop them. Differential Revision: https://phab.mercurial-scm.org/D6943

upgrade: allow upgrade to repository using sidedata Repository can now be migrated to support sidedata. More requirements and migration will be needed to actual side-data usage. This is a step in that direction. To test the feature, we leverage the test extension. It make sure the `update` part of the side-data migration actually works. Differential Revision: https://phab.mercurial-scm.org/D6942

revlog: add a way to control sidedata changes during revlog.clone We introduce a new argument to pass a callable dealing with the side data logic. Since the side data are usually associated to higher level logic, the revlog code itself is unlikely to know that to do itself. As a result the higher level upgrade code will be responsible to decide what needs to changes. The lower level revlog.clone just get simple instructions to apply. Differential Revision: https://phab.mercurial-scm.org/D6941

sidedata: use only changegroup3 if sidedata is in use This is necessary to exchange the flags over the wire (and within bundles). Differential Revision: https://phab.mercurial-scm.org/D6940

sidedata: apply basic but tight security around exchange We don't currently have code to deal with exchange between repository using sidedata and repository not using sidedata. Until we implement such code (eg: dropping side data when pushing to a non-sidedata repo) we prevent the two kind of repo to speak to each other. This is somewhere similar to what 'treemanifest' does. Note that sidedata exchange is broken unless one use changegroup v3 anyway. See next changeset for details. Differential Revision: https://phab.mercurial-scm.org/D6939

perf: use `setup` function in `perfdirstatewrite` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstatewrite`

perf: use `setup` function in `perfdirfoldmap` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirfoldmap`

perf: use `setup` function in `perfdirstatefoldmap` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstatefoldmap`

perf: use `setup` function in `perfdirstatedirs` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstatedirs`

perf: use `setup` function in `perfdirstate` The command seems to pre-date the introduction of the `setup` support in timer. We move the line that is obviously about benchmark setup in such `setup` function.

perf: document `perfdirstate`

perf: document `perfstatus`

py3: flush ui after each message in interactive patch filtering Otherwise, actions from ui.write() are buffered and displayed at end of interactive session.

py3: decode prompt string before calling rawinput Calling input() (aka pycompat.rawinput() on python3) with a byte string displays a byte string in the console. E.g. in interactive commit, we get: b"examine changes to '<file>'?\n(enter ? for help) [Ynesfdaq?]". Similarly, "hg email" prompts are messed up. We thus decode the prompt string before running rawinput().

cleanup: join string literals that are already on one line Thanks to Kyle for noticing this and for providing the regular expression to run on the codebase. This patch has been reviewed by the test suite and they approved of it. # skip-blame: fallout from mass reformatting Differential Revision: https://phab.mercurial-scm.org/D7028

debugcommands: add a few more writenoi18n() Differential Revision: https://phab.mercurial-scm.org/D7027

py3: delete b'' prefix from safehasattr arguments Differential Revision: https://phab.mercurial-scm.org/D7029

destutil: provide hint on rebase+merge for how to specify destination/rev Without a destination specified, rebase and merge attempt to identify a good candidate; this fails if there's too many heads, the heads have a bookmark, or other reasons. Complicating the issue, users may have specified -t, thinking it means target, and are confused when the error message says that they need to specify an explicit rev or explicit destination - they feel like they already have. Differential Revision: https://phab.mercurial-scm.org/D7024

hghave: document format for version feature checks as <name><vers>, no dots I had a bit of a difficult time when attempting to fix a use of `py-38` when trying to figure out what the correct way of specifying the feature was. By having the strings 'py-3*' and 'py3.8' in the hghave.checkvers docstring, I would have had a much easier time of it instead of having to trace exactly what was happening here. Differential Revision: https://phab.mercurial-scm.org/D7023

check-commit: allow foo_bar naming in functions nameswithallthewordssmashedtogetherarehardtoread. especiallyifenglishisnotyourprimarylanguage. Let's align with the rest of the programming universe and allow_the_use_of_underscores_in_names. We took a hand poll at the 5.2 sprint regarding this change and all but 1 person supported it. The person who didn't expressed concerns around excessive API breakage if we mass renamed things. But we're not planning to mass rename things for the sake of renaming, so all should be well. Differential Revision: https://phab.mercurial-scm.org/D2010

infinitepush: mark extension as likely to be deleted .. bc:: The infinitepush extension is believed to be unused, and will be deleted at the end of 2020 unless users contact mercurial-devel@mercurial-scm.org. Differential Revision: https://phab.mercurial-scm.org/D7022

tests: use range() in generate-churning-module.py This is a test-only script. Performance on Python 2 for creating a full list instead of a generator doesn't matter. With this change, test-check-pyflakes.t passes on Python 3! Differential Revision: https://phab.mercurial-scm.org/D7019

tests: allow warning about file The code is protected by a block that makes it Python 2 only, so use of file is acceptable. Differential Revision: https://phab.mercurial-scm.org/D7018

lsprof: remove __main__ functionality I'm pretty sure nobody uses this. I noticed it because Python 3 linting is complaining about execfile. Differential Revision: https://phab.mercurial-scm.org/D7017

tests: use proper Python 3.8 feature Fix typo introduced in 830eacef67f8. Differential Revision: https://phab.mercurial-scm.org/D7016

py3: finish porting iteritems() to pycompat and remove source transformer This commit finishes porting .iteritems() to pycompat.iteritems() for the mercurial package. The translation of .iteritems() to .items() was the last conversion performed by the source transformer. With the porting to pycompat complete, we no longer have a need for the source transformer. So the source transformer has been removed. Good riddance! The code base is now compatible with Python 2 and Python 3. For the record, as the person who introduced the source transformer, it brings me joy to delete it. It accomplished its goal to facilitate a port to Python 3 without overly burdening people on some painful low-level differences between Python 2 and 3. It is unfortunate we still have to wallpaper over many differences with the pycompat shim. But it is what it is. Differential Revision: https://phab.mercurial-scm.org/D7015

py3: define and use pycompat.iteritems() for hgext/ .iteritems() -> .items() is the last source transform being performed. But it is also the most widely used. This commit adds a pycompat.iteritems symbol and imports it in place of .iteritems() for usage in hgext/. I chose to stop at just hgext/ because the patch will be large and it is an easy boundary to stop at since we can disable source transformation on a per-package basis. There are places where the type does implement items() and we could call items() directly. However, this would require critical thought and I thought it would be easier to just blindly change the code. We know which call sites need to be audited in the future because they have "pycompat.iteritems." With this change, we no longer perform source transformation on hgext! Differential Revision: https://phab.mercurial-scm.org/D7014

py3: define and use pycompat.itervalues() .itervalues() only exists on Python 2. Python 3's equivalent is .values(). But we don't want to blindly use .values() everywhere because on Python 2, it will create a list, which will have performance implications. This commit introduces pycompat.itervalues() which will call the appropriate method on the passed object. We update all callers of obj.itervalues() to pycompat.itervalues(obj) instead. With this commit, the only source tranforming remaining is for iteritems(). Victory is near... Differential Revision: https://phab.mercurial-scm.org/D7013

py3: stop normalizing 2nd argument of *attr() to unicode Now that we don't byteify strings, we can stop normalizing the 2nd string argument to getattr() and remove explicit overrides we were using in the code base. We no longer use some helper functions in the source transformer, so we remove those as well. Differential Revision: https://phab.mercurial-scm.org/D7012

log: populate keywords if specified in custom -Tjson(...) or -Tcbor(...) To make things simple, early return for ui.quiet is disabled if the formatter is templated and provides some datahint().

formatter: map -Tjson(...) and -Tcbor(...) to templater Even though custom JSON output could be generated by using --config template.json="{dict(...)|json}" ..., doing that is tedious because of the trailing comma handling. This patch introduces special syntax for JSON/CBOR formats. -Tjson(...) is translated to template as if function-style template definition were supported: [templates] json(...) = "{dict(...)|json}" json(...):docheader = "[\n " json(...):docfooter = "\n]\n" json(...):separator = ",\n "

formatter: parse name of built-in formatter templates in standard way This slightly makes it easier to add "-Tjson(...)" handling, which should be enabled only if the template specifier doesn't look like a literal template. In other words, it should be handled after "if '{' in tmpl". This makes "log -Tpickle" and "log -Tdebug" abort, which I think is better than just printing "picklepicklepickle...".

formatter: pass in template spec to templateformatter as argument Prepare for the next patch, which will unify handling of the formatter names and the template names.

templater: add public parseexpr() function to parse "-Tjson(...)" Extracted _addparseerrorhint() to show nicer hint on error.

import: read X-Mercurial-Node email header to determine nodeid This would be useful to import and obsolete patches sent using 'hg email --plain', using evolve's --obsolete option of 'hg import'. If email body contains Mercurial patch header ('# HG changeset patch' block), nodeid parsed from X-Mercurial-Node header will still be overridden by respective value found in body.

patchbomb: use procutil.shellquote() instead of shlex to escape email address This basically backs out 2cc453284d5c, and inserts procutil.shellquote() instead. I don't care about Windows compatibility here, but shlex.quote() can't handle byte strings on Python 3.

formatting: introduce a `test-check-format-black.t` that enforce formatting This should prevent use to drift away from the expect format.

formatting: run black on all file again Apparently, since the blackgnarok, we divergence from the expected formatting. Formatted using:: grey.py -S $(hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**" - hgext/fsmonitor/pywatchman/** - contrib/grey.py') # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions

formatting: make black --quiet in the example `hg fix` config I do like cake, but I cannot have so many of them.

formatting: remove the data-ogre from the config example Without he final `-`, black ignore the stdin and just looks at file provided as command line argument. Since `hg fix` feeds the file content through stdin and does not pass file argument, this meant black happily exited successful (all files passed as argument were formatted) without any output. Fix picked this "no output" as the new file content, deleting all previous content. I appreciate the fact this effectively removed all buggy code in any files processing that way, but this also ate all my data. The example config is now fixed in that regards.

py3: stop normalizing .encode()/.decode() arguments to unicode Now that we don't byte transform string literals, we no longer need this transform. While we're here, we also drop some superfluous u'' prefix in existing callers. Differential Revision: https://phab.mercurial-scm.org/D7011

py3: manually import pycompat.delattr where it is needed And with this change, we no longer need the auto-inserted import statement in the source transformer, so it has been removed! Differential Revision: https://phab.mercurial-scm.org/D7010

py3: manually import getattr where it is needed The march continues. Differential Revision: https://phab.mercurial-scm.org/D7009

py3: stop injecting pycompat.hasattr into modules I only found a single user of this pattern, probably because we use util.hasattr everywhere. Differential Revision: https://phab.mercurial-scm.org/D7008

py3: manually import pycompat.setattr where it is needed Continuing to eliminate the implicit import of symbols in the Python 3 source transformer so we can eliminate it. Differential Revision: https://phab.mercurial-scm.org/D7007

py3: stop implicitly importing unicode We should be pycompat.unicode everywhere. It turns out we were doing this everywhere except for one place in templatefilters! Differential Revision: https://phab.mercurial-scm.org/D7006

py3: manually import pycompat.open into files that need it We want to eliminate the source transformer. Currently it inserts a `from mercurial.pycompat import ...` at the top of files to alias some builtins. This commit replaces the implicit import of `open` with an explicit import on files that need it and changes the source transformer to no longer import `open`. As part of this, we needed to store an explicit local for `open` in the Python 2 code path in `pycompat` so the import works. (Builtins that are automatically in scope cannot be imported.) Differential Revision: https://phab.mercurial-scm.org/D7005

import-checker: allow symbol imports from mercurial.pycompat Currently, the source transformer inserts `from mercurial.pycompat import delattr, getattr, hasattr, setattr, open, unicode` to the top of every file. As part of getting rid of the source transformer, we'll need to have source code call these wrappers directly. Rather than rewrite all call sites to call pycompat.*, I think it makes sense to import needed symbols via explicit imports. That requires loosening the import checker to allow this. Differential Revision: https://phab.mercurial-scm.org/D7004

contrib: add a fork of black (as "grey") that includes my changes This is black with https://github.com/psf/black/pull/826 applied as of today. The current git hash of black master is d9e71a75ccfefa3d9156a64c03313a0d4ad981e5, and the hash of my commit is dc1add6e94e212eff37bb3619e1422fb3c6d8dc8. In order to use this, you need to install `black` (from github master) and `typed-ast` using pip, preferably into python3, and then you can run `grey.py` with that Python and you'll have my patched version of black, which is how we've been formatting the codebase. Once my PR is merged, I'll follow up by removing this fork and updating instructions in the example config. # no-check-commit bad style Differential Revision: https://phab.mercurial-scm.org/D7002

rust-cpython: change license of ref_sharing.rs to MIT Since we plan to upstream this feature, it's better to continue further refactoring under the same license as rust-cpython crate. According to the file history, copyright holders are: - Raphaël Gomès <rgomes@octobus.net> - Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> - Yuya Nishihara <yuya@tcha.org>

contrib: fix check-code to be able to detect missing _() with bytestrings Differential Revision: https://phab.mercurial-scm.org/D6997

cleanup: mark some ui.(status|note|warn|write) calls as not needing i18n These used to be marked with no-op parens, but black removes those now and this is more explicit. # skip-blame: fallout from mass reformatting Differential Revision: https://phab.mercurial-scm.org/D6996

ui: define (write|status|warn|note)noi18n aliases We currently use `write(('...'))` to suppress check-code warnings about not using translated strings. However, when we run black, it will strip the `((...))`. In order to placate black, we'll need to use a different mechanism to pass untranslatable strings. This commit introduces a `writenoi18n` alias (and friends) to `write` for that purpose. Differential Revision: https://phab.mercurial-scm.org/D6994

mangler: stop rewriting string constants to be bytes literals We've rewritten everything, so we no longer require this step. Differential Revision: https://phab.mercurial-scm.org/D6973

formatting: byteify all mercurial/ and hgext/ string literals Done with python3.7 contrib/byteify-strings.py -i $(hg files 'set:mercurial/**.py - mercurial/thirdparty/** + hgext/**.py - hgext/fsmonitor/pywatchman/** - mercurial/__init__.py') black -l 80 -t py33 -S $(hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**" - hgext/fsmonitor/pywatchman/**') # skip-blame mass-reformatting only Differential Revision: https://phab.mercurial-scm.org/D6972

formatting: blacken the codebase This is using my patch to black (https://github.com/psf/black/pull/826) so we don't un-wrap collection literals. Done with: hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**"' | xargs black -S # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D6971

style: run a patched black on a subset of mercurial This applied black to the 20 smallest files in mercurial/: ls -S1 mercurial/*.py | tail -n20 | xargs black --skip-string-normalization Note that a few files failed to format, presumably due to a bug in my patch. The intent is to be able to compare results to D5064 with https://github.com/python/black/pull/826 applied to black. I skipped string normalization on this patch for clarity - in reality I think we'd want one pass without string normalization, followed by another to normalize strings (which is basically replacing ' with " globally.) # skip-blame mass-reformatting only Differential Revision: https://phab.mercurial-scm.org/D6342

tests: conditionalize output for Python 3 It appears that the random.randint() behavior is different between Python 2 and Python 3. So make the test conditional on that. This makes the test pass on Python 3.7 (and presumably other Python 3 versions). Differential Revision: https://phab.mercurial-scm.org/D6964

hg: remove HGUNICODEPEDANTRY and RTUNICODEPEDANTRY This was added in 73e4a02e6d23 and 89822d7a9d5f as a preliminary way to better support Python 3. With the Python 3 port nearly done and better procedures in place, we shouldn't need it. Differential Revision: https://phab.mercurial-scm.org/D6991

convert: use pycompat.fsencode() This avoids a `TypeError: Can't mix strings and bytes in path components`. Differential Revision: https://phab.mercurial-scm.org/D6965

tests: make tarball output conditional on Python version Python 3.8 changed the default tar file archive format: https://docs.python.org/3.8/whatsnew/3.8.html#tarfile. This commit teaches our tests about the new behavior. Differential Revision: https://phab.mercurial-scm.org/D6986

tests: conditionalize test-run-tests.t for Python 3.8 It looks like Python 3.8 changed the pretty XML formatting slightly to strip some whitespace. Let's teach our tests about that. Differential Revision: https://phab.mercurial-scm.org/D6988

hgweb: use importlib.reload() if available reload() was nuked in Python 3. We need to use importlib.reload() instead. But pyflakes isn't smart enough to detect our conditional usage, so we allow this error. Differential Revision: https://phab.mercurial-scm.org/D6992

tests: allow xrange warning from perf.py The use of xrange in this file is acceptable. We need to make the output optional - and not conditional on the Python version - because `pyflakes` could be executed by any Python version and pyflakes behaves differently depending on the Python version. Differential Revision: https://phab.mercurial-scm.org/D6990

debugsidedata: fix verbose mode on python3 We need to be gently with python3 for it to display this binary data. Differential Revision: https://phab.mercurial-scm.org/D6995

import: add debug messages when parsing data from patch header

examples: include a sample of how to use black with fix It's commented out for now since my patch hasn't landed, but we can uncomment it when that lands. Differential Revision: https://phab.mercurial-scm.org/D6974

black: add a pyproject.toml that contains our black settings Differential Revision: https://phab.mercurial-scm.org/D6993

contrib: stop whining about whitespace around = This was conflicting with black, so dump it. Differential Revision: https://phab.mercurial-scm.org/D6979

contrib: black manages whitespace after Python keywords now Differential Revision: https://phab.mercurial-scm.org/D6978

contrib: allow trailing commas in lists black manages this for us, so we don't need to worry about this anymore. Differential Revision: https://phab.mercurial-scm.org/D6977

contrib: remove check-code line length limit Both Python and C code are managed by formatters now, so relax this. Differential Revision: https://phab.mercurial-scm.org/D6976

changegroup: use positive logic for treemanifest changegroup3 logic We are about to add more cases, in that context, it is simpler to avoid double negative. En addition, We document the situation for the next soul. Differential Revision: https://phab.mercurial-scm.org/D6938

fix: add :enabled sub-config for fixer tools This allows distributing opt-in fixer tool configurations in .hgrc files. This may be useful for adding default configs in core, or for orgranizations that want to provide configs to their users. Tools are still enabled by default because it would be confusing to add a config and find that it has no effect until you add enabled=true. Differential Revision: https://phab.mercurial-scm.org/D6975

automation: add a command to submit to a Try server The CI code for running the Try Server requires more thorough review. Let's add just the client-side bits for submitting to Try so others can start using it. Differential Revision: https://phab.mercurial-scm.org/D6983

automation: improve documentation for credentials management Differential Revision: https://phab.mercurial-scm.org/D6982

automation: switch to us-west-2 by default This is where the AWS account I'm running lives. Differential Revision: https://phab.mercurial-scm.org/D6981

automation: upgrade various packages Ran `pip-compile -U` to upgrade to latest versions. Differential Revision: https://phab.mercurial-scm.org/D6980

statprof: drop python #! and exec bit we don't use For some reason this wasn't flagged until I ran a reformatter on this file. I'm deeply confused by this, but since we don't execute this drop the support for it. Differential Revision: https://phab.mercurial-scm.org/D6985

cvsps: switch a file open to a with statement For some reason this only showed up in check-code after running black. Puzzling. Differential Revision: https://phab.mercurial-scm.org/D6984

templatefuncs: account for user's diffopts in diff() (BC)

transaction: re-wrap line to avoid a black bug Differential Revision: https://phab.mercurial-scm.org/D6970

discovery: re-wrap expression to avoid a black bug Differential Revision: https://phab.mercurial-scm.org/D6969

sidedata: adjust string for python3 This was pointed out by Gregory Szorc: Python 3 will require a r'' because of source transformation. Differential Revision: https://phab.mercurial-scm.org/D6968

py3: fix phabricator's use of json.loads() for py3.5 Missed this in c340a8ac7ef3 since `loads()` takes bytes from 3.6 onwards. Differential Revision: https://phab.mercurial-scm.org/D6961

run-tests: make coverage work out of tree Currently coverage fails when run on an out-of-tree extension since run-tests.py tries to load sitecustomize.py from self._testdir, which is the dir for the extension's tests. Differential Revision: https://phab.mercurial-scm.org/D6960

unfinishedstate: suggested `hg update .` (including `.`) to complete update `hg update` can update to a different and undesired commit. For users who have commands.update.requiredest=yes, it's even an error to run just `hg update. Differential Revision: https://phab.mercurial-scm.org/D6956

setup: build extensions in parallel by default The build_ext distutils command in Python 3.5+ has a "parallel" option that controls whether to build extensions in parallel. It is disabled by default (None) and can be set to an integer value for number of cores or True to indicate use all available CPU cores. This commit changes our build_ext command override to set "parallel" to True unless a value has been provided by the caller. On my machine, this makes `python setup.py build_ext` 1-4s faster. It is worth noting that at this time, each individual source file constituting the extension is still built serially. For Mercurial, this means that we can't build faster than the slowest-to-build extension, which is the zstd extension by a long shot. This means that setup.py is still not very efficient at utilizing multiple cores. But we're better than before. Differential Revision: https://phab.mercurial-scm.org/D6923 # no-check-commit because of foo_bar naming

merge with stable

sidedata: check that the sidedata safely roundtrip We actually use the data we stored during the test. It is a way to make sure their content is not corrupted. Differential Revision: https://phab.mercurial-scm.org/D6898

sidedata: add a debugsidedata command The command provide basic capabilities to inspect revision side data. Differential Revision: https://phab.mercurial-scm.org/D6897

sidedata: test we can successfully write sidedata For now we just write them and rejoice on the lack of crashes. Differential Revision: https://phab.mercurial-scm.org/D6896

sidedata: make sure we don't use the flag if there are not sidedata Ensuring this at this level seems safer than relying on caller doing the right thing. G: changed mercurial/revlog.py Differential Revision: https://phab.mercurial-scm.org/D6895

revlog: add the appropriate flag is sidedata are passed to `addrevision` If we need to store sidedata, we need the flag to be set and the associated processing to be called. Differential Revision: https://phab.mercurial-scm.org/D6894

sidedata: register the flag processors if the repository allows for it Differential Revision: https://phab.mercurial-scm.org/D6893

sidedata: add a function for _processflagsraw usage Entry with side data cannot directly validate their rawtext Differential Revision: https://phab.mercurial-scm.org/D6892

sidedata: add a function to write sidedata into a raw text Differential Revision: https://phab.mercurial-scm.org/D6891

sidedata: add a function to read sidedata from revlog raw text This implement the "reading" part of a `sidedata` flag processor. Differential Revision: https://phab.mercurial-scm.org/D6890

sidedata: add a new module with basic documentation For now the storage strategy is very simple. We can augment it in the future if needed. Code to actually support what is described will be introduced in later changesets. Differential Revision: https://phab.mercurial-scm.org/D6889

sidedata: add a new revision flag constant for side data Small step first. Differential Revision: https://phab.mercurial-scm.org/D6888

upgrade: detect the side-data format variants Note that for now we cannot upgrade/downgrade to it. Differential Revision: https://phab.mercurial-scm.org/D6887

sidedata: introduce a new requirement to protect the feature The feature will be usable only within repository supported by version of Mercurial with the support. Since the support is not fully implemented yet, the requirement has a temporary, experimental name. Differential Revision: https://phab.mercurial-scm.org/D6886

Added signature for changeset 181e52f2b62f

Added tag 5.1.2 for changeset 181e52f2b62f

copies: move file input processsing early If we are to store the same kind of data outside of extra, we need to explicitly prepare them before that. On the long run, other storage (eg: sidedata) might use a different encoding to store this information, since the constraint from extra does not apply to it. Differential Revision: https://phab.mercurial-scm.org/D6937

copies: prepare changelog for more copies storage mode We want to store copies information in sidedata, the logic will be quite similar to the one used for extra, but sightly different (not in extra, no hash impact). We start with small refactoring to make our work easier and the coming changesets cleaners. Differential Revision: https://phab.mercurial-scm.org/D6936

vfs: give all vfs an options attribute by default Multiple code path assume vfs have an options attribute, including the vfs module itself. So for consistency we explicitly add one to all vfs. This will prevent various crash in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D6935

revlog: explicitly set revlogv0 in vfs options Relying on having an attribute or not for something so fundamental seems too fragile to me. (And indeed I had issue with that later in this series). So we explicitly record the fact the repository use revlog-v0. Differential Revision: https://phab.mercurial-scm.org/D6934

context: clarify the various mode in the _copies property cache The previous code was compact but a bit dense. The new proposed code deal with each mode separately, there are some duplicated lines, but the meaning of each mode stand out. One of the benefit it to make it simpler to add further mode in the future. Differential Revision: https://phab.mercurial-scm.org/D6933

context: clarify the various mode in the filesremoved method The previous code was compact but a bit dense. The new proposed code deal with each mode separately, there are some duplicated lines, but the meaning of each mode stand out. One of the benefit it to make it simpler to add further mode in the future. Differential Revision: https://phab.mercurial-scm.org/D6932

context: clarify the various mode in the filesadded method The previous code was compact but a bit dense. The new proposed code deal with each mode separately, there are some duplicated lines, but the meaning of each mode stand out. One of the benefit it to make it simpler to add further mode in the future. Differential Revision: https://phab.mercurial-scm.org/D6931

copies: expand the logic of usechangesetcentricalgo Using intermediate variable is clearer and will make is simple to expand the logic. Differential Revision: https://phab.mercurial-scm.org/D6930

patchbomb: protect email addresses from shell When patchbomb sends email via a sendmail-like program it invokes this using procutil.popen which passes the string to a shell to be parsed. To protect any special characters in the email addresses on the command line from being interpretered by the shell they must be quoted.

automation: support and use Debian Buster by default While Debian Buster (Debian 10) was released in July, the AWS AMIs were not published until mid September. This commit teaches the automation system to create AMIs for Debian Buster. Since Debian Buster is the new stable Debian release, we make it the default distribution for automation. Differential Revision: https://phab.mercurial-scm.org/D6917

automation: use latest AMIs This cuts down on the overhead to create a new AMI, as we don't have to install as many package upgrades. Differential Revision: https://phab.mercurial-scm.org/D6916

automation: increase size of Linux AMI build volume I ran out of space attempting to build a few AMIs. Let's give the environment an extra 2 GB to utilize. Differential Revision: https://phab.mercurial-scm.org/D6915

automation: install awscli and python3-boto3 packages This provides the `aws` command-line utility as well as a Python library for interacting with AWS. It is useful to have these packages installed in AMIs so you can easily interact with other AWS services. Differential Revision: https://phab.mercurial-scm.org/D6914

automation: upgrade packages in Linux environment Let's stay modern. Differential Revision: https://phab.mercurial-scm.org/D6913

automation: remove Ubuntu 18.10 It reached end of life on July 18, 2019. I don't think we should care about supporting it. Differential Revision: https://phab.mercurial-scm.org/D6912

automation: always install docker-ce Docker has published packages for Ubuntu Disco and we no longer need this one-off. Differential Revision: https://phab.mercurial-scm.org/D6911

automation: use LSB_RELEASE instead of DEBIAN_VERSION This should be more robust since I believe the minor version can change mid release. Differential Revision: https://phab.mercurial-scm.org/D6910

tests: use silenttestrunner in test-simplemerge.py The time monkeypatching doesn't appear to work reliably in Python 3, possibly due to unittest using a different time function. This makes the test intermittent due to the execution time not always being `0.00s`. We have our own wrapper around unittest for more deterministic test output. So let's use it. As a bonus, all test output disappeared, so we can remove the .out file! Differential Revision: https://phab.mercurial-scm.org/D6921

contrib: use pycompat.xrange in bdiff-torture.py For Python 3 compatibility. Differential Revision: https://phab.mercurial-scm.org/D6920

interfaces: use triple quotes for Attribute value Otherwise the source transformer converts it to bytes and test-check-interfaces.py complains on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6919

tests: suppress output from write() Otherwise it is printed and the test fails with tons of output on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6918

tests: look for additional pip warning variant The wording has been tweaked in new versions of pip. Differential Revision: https://phab.mercurial-scm.org/D6909

tests: use test harness `hg` in test-hghave.t This prevents the sub test harness from rebuilding Mercurial. It shaves ~30s from the test wall time on my machine. Differential Revision: https://phab.mercurial-scm.org/D6908

synthrepo: use pycompat.xrange This silences some test-check-pyflakes.t warnings on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6907

perf: fix perfhelper-pathcopies without --stats Some conditional were missing.

upgrade: also register copied `.d` files to fncache Oops, we forgot to register data file when blanky copying revlog. Now that we are actually copying these `.d` files, we should also register them. Differential Revision: https://phab.mercurial-scm.org/D6905

upgrade: fix handling of `.d` data file The test did not used any `.d` file. So of course, their handling was broken (for vfs encoding reason again). We fix the issue and update the test. Note that some fncache issue remains with `.d` file. It will be taken care of in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D6904

upgrade: also copy data file Differential Revision: https://phab.mercurial-scm.org/D6903

upgrade: create the correct destination directory for copies revlogs The encoding in vfs mess up with the directory we are trying to create manually. We could duplicate the advanced logic some more, but that seems ill fated. Instead, we let the vfs deal with directory creation. We update the test to contains a name affected by encoding. Differential Revision: https://phab.mercurial-scm.org/D6902

upgrade: fix DELTAREUSEFULLADD implementation in revlog.clone If we do a full addition, we need to start from the full text. Differential Revision: https://phab.mercurial-scm.org/D6901

upgrade: document DELTAREUSEFULLADD in revlog.clone The documentation of the `debugupgraderepo` command was updated when `re-delta-fulladd` was introduced. However, the docstring of the `revlog.clone` method was not. Differential Revision: https://phab.mercurial-scm.org/D6900