discovery: re-wrap expression to avoid a black bug Differential Revision: https://phab.mercurial-scm.org/D6969

sidedata: adjust string for python3 This was pointed out by Gregory Szorc: Python 3 will require a r'' because of source transformation. Differential Revision: https://phab.mercurial-scm.org/D6968

py3: fix phabricator's use of json.loads() for py3.5 Missed this in c340a8ac7ef3 since `loads()` takes bytes from 3.6 onwards. Differential Revision: https://phab.mercurial-scm.org/D6961

run-tests: make coverage work out of tree Currently coverage fails when run on an out-of-tree extension since run-tests.py tries to load sitecustomize.py from self._testdir, which is the dir for the extension's tests. Differential Revision: https://phab.mercurial-scm.org/D6960

unfinishedstate: suggested `hg update .` (including `.`) to complete update `hg update` can update to a different and undesired commit. For users who have commands.update.requiredest=yes, it's even an error to run just `hg update. Differential Revision: https://phab.mercurial-scm.org/D6956

setup: build extensions in parallel by default The build_ext distutils command in Python 3.5+ has a "parallel" option that controls whether to build extensions in parallel. It is disabled by default (None) and can be set to an integer value for number of cores or True to indicate use all available CPU cores. This commit changes our build_ext command override to set "parallel" to True unless a value has been provided by the caller. On my machine, this makes `python setup.py build_ext` 1-4s faster. It is worth noting that at this time, each individual source file constituting the extension is still built serially. For Mercurial, this means that we can't build faster than the slowest-to-build extension, which is the zstd extension by a long shot. This means that setup.py is still not very efficient at utilizing multiple cores. But we're better than before. Differential Revision: https://phab.mercurial-scm.org/D6923 # no-check-commit because of foo_bar naming

merge with stable

sidedata: check that the sidedata safely roundtrip We actually use the data we stored during the test. It is a way to make sure their content is not corrupted. Differential Revision: https://phab.mercurial-scm.org/D6898

sidedata: add a debugsidedata command The command provide basic capabilities to inspect revision side data. Differential Revision: https://phab.mercurial-scm.org/D6897

sidedata: test we can successfully write sidedata For now we just write them and rejoice on the lack of crashes. Differential Revision: https://phab.mercurial-scm.org/D6896

sidedata: make sure we don't use the flag if there are not sidedata Ensuring this at this level seems safer than relying on caller doing the right thing. G: changed mercurial/revlog.py Differential Revision: https://phab.mercurial-scm.org/D6895

revlog: add the appropriate flag is sidedata are passed to `addrevision` If we need to store sidedata, we need the flag to be set and the associated processing to be called. Differential Revision: https://phab.mercurial-scm.org/D6894

sidedata: register the flag processors if the repository allows for it Differential Revision: https://phab.mercurial-scm.org/D6893

sidedata: add a function for _processflagsraw usage Entry with side data cannot directly validate their rawtext Differential Revision: https://phab.mercurial-scm.org/D6892

sidedata: add a function to write sidedata into a raw text Differential Revision: https://phab.mercurial-scm.org/D6891

sidedata: add a function to read sidedata from revlog raw text This implement the "reading" part of a `sidedata` flag processor. Differential Revision: https://phab.mercurial-scm.org/D6890

sidedata: add a new module with basic documentation For now the storage strategy is very simple. We can augment it in the future if needed. Code to actually support what is described will be introduced in later changesets. Differential Revision: https://phab.mercurial-scm.org/D6889

sidedata: add a new revision flag constant for side data Small step first. Differential Revision: https://phab.mercurial-scm.org/D6888

upgrade: detect the side-data format variants Note that for now we cannot upgrade/downgrade to it. Differential Revision: https://phab.mercurial-scm.org/D6887

sidedata: introduce a new requirement to protect the feature The feature will be usable only within repository supported by version of Mercurial with the support. Since the support is not fully implemented yet, the requirement has a temporary, experimental name. Differential Revision: https://phab.mercurial-scm.org/D6886

copies: move file input processsing early If we are to store the same kind of data outside of extra, we need to explicitly prepare them before that. On the long run, other storage (eg: sidedata) might use a different encoding to store this information, since the constraint from extra does not apply to it. Differential Revision: https://phab.mercurial-scm.org/D6937

copies: prepare changelog for more copies storage mode We want to store copies information in sidedata, the logic will be quite similar to the one used for extra, but sightly different (not in extra, no hash impact). We start with small refactoring to make our work easier and the coming changesets cleaners. Differential Revision: https://phab.mercurial-scm.org/D6936

vfs: give all vfs an options attribute by default Multiple code path assume vfs have an options attribute, including the vfs module itself. So for consistency we explicitly add one to all vfs. This will prevent various crash in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D6935

revlog: explicitly set revlogv0 in vfs options Relying on having an attribute or not for something so fundamental seems too fragile to me. (And indeed I had issue with that later in this series). So we explicitly record the fact the repository use revlog-v0. Differential Revision: https://phab.mercurial-scm.org/D6934

context: clarify the various mode in the _copies property cache The previous code was compact but a bit dense. The new proposed code deal with each mode separately, there are some duplicated lines, but the meaning of each mode stand out. One of the benefit it to make it simpler to add further mode in the future. Differential Revision: https://phab.mercurial-scm.org/D6933

context: clarify the various mode in the filesremoved method The previous code was compact but a bit dense. The new proposed code deal with each mode separately, there are some duplicated lines, but the meaning of each mode stand out. One of the benefit it to make it simpler to add further mode in the future. Differential Revision: https://phab.mercurial-scm.org/D6932

context: clarify the various mode in the filesadded method The previous code was compact but a bit dense. The new proposed code deal with each mode separately, there are some duplicated lines, but the meaning of each mode stand out. One of the benefit it to make it simpler to add further mode in the future. Differential Revision: https://phab.mercurial-scm.org/D6931

copies: expand the logic of usechangesetcentricalgo Using intermediate variable is clearer and will make is simple to expand the logic. Differential Revision: https://phab.mercurial-scm.org/D6930

patchbomb: protect email addresses from shell When patchbomb sends email via a sendmail-like program it invokes this using procutil.popen which passes the string to a shell to be parsed. To protect any special characters in the email addresses on the command line from being interpretered by the shell they must be quoted.

automation: support and use Debian Buster by default While Debian Buster (Debian 10) was released in July, the AWS AMIs were not published until mid September. This commit teaches the automation system to create AMIs for Debian Buster. Since Debian Buster is the new stable Debian release, we make it the default distribution for automation. Differential Revision: https://phab.mercurial-scm.org/D6917

automation: use latest AMIs This cuts down on the overhead to create a new AMI, as we don't have to install as many package upgrades. Differential Revision: https://phab.mercurial-scm.org/D6916

automation: increase size of Linux AMI build volume I ran out of space attempting to build a few AMIs. Let's give the environment an extra 2 GB to utilize. Differential Revision: https://phab.mercurial-scm.org/D6915

automation: install awscli and python3-boto3 packages This provides the `aws` command-line utility as well as a Python library for interacting with AWS. It is useful to have these packages installed in AMIs so you can easily interact with other AWS services. Differential Revision: https://phab.mercurial-scm.org/D6914

automation: upgrade packages in Linux environment Let's stay modern. Differential Revision: https://phab.mercurial-scm.org/D6913

automation: remove Ubuntu 18.10 It reached end of life on July 18, 2019. I don't think we should care about supporting it. Differential Revision: https://phab.mercurial-scm.org/D6912

automation: always install docker-ce Docker has published packages for Ubuntu Disco and we no longer need this one-off. Differential Revision: https://phab.mercurial-scm.org/D6911

automation: use LSB_RELEASE instead of DEBIAN_VERSION This should be more robust since I believe the minor version can change mid release. Differential Revision: https://phab.mercurial-scm.org/D6910

tests: use silenttestrunner in test-simplemerge.py The time monkeypatching doesn't appear to work reliably in Python 3, possibly due to unittest using a different time function. This makes the test intermittent due to the execution time not always being `0.00s`. We have our own wrapper around unittest for more deterministic test output. So let's use it. As a bonus, all test output disappeared, so we can remove the .out file! Differential Revision: https://phab.mercurial-scm.org/D6921

contrib: use pycompat.xrange in bdiff-torture.py For Python 3 compatibility. Differential Revision: https://phab.mercurial-scm.org/D6920

interfaces: use triple quotes for Attribute value Otherwise the source transformer converts it to bytes and test-check-interfaces.py complains on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6919

tests: suppress output from write() Otherwise it is printed and the test fails with tons of output on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6918

tests: look for additional pip warning variant The wording has been tweaked in new versions of pip. Differential Revision: https://phab.mercurial-scm.org/D6909

tests: use test harness `hg` in test-hghave.t This prevents the sub test harness from rebuilding Mercurial. It shaves ~30s from the test wall time on my machine. Differential Revision: https://phab.mercurial-scm.org/D6908

synthrepo: use pycompat.xrange This silences some test-check-pyflakes.t warnings on Python 3. Differential Revision: https://phab.mercurial-scm.org/D6907

perf: fix perfhelper-pathcopies without --stats Some conditional were missing.

upgrade: also register copied `.d` files to fncache Oops, we forgot to register data file when blanky copying revlog. Now that we are actually copying these `.d` files, we should also register them. Differential Revision: https://phab.mercurial-scm.org/D6905

upgrade: fix handling of `.d` data file The test did not used any `.d` file. So of course, their handling was broken (for vfs encoding reason again). We fix the issue and update the test. Note that some fncache issue remains with `.d` file. It will be taken care of in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D6904

upgrade: also copy data file Differential Revision: https://phab.mercurial-scm.org/D6903

upgrade: create the correct destination directory for copies revlogs The encoding in vfs mess up with the directory we are trying to create manually. We could duplicate the advanced logic some more, but that seems ill fated. Instead, we let the vfs deal with directory creation. We update the test to contains a name affected by encoding. Differential Revision: https://phab.mercurial-scm.org/D6902

upgrade: fix DELTAREUSEFULLADD implementation in revlog.clone If we do a full addition, we need to start from the full text. Differential Revision: https://phab.mercurial-scm.org/D6901

upgrade: document DELTAREUSEFULLADD in revlog.clone The documentation of the `debugupgraderepo` command was updated when `re-delta-fulladd` was introduced. However, the docstring of the `revlog.clone` method was not. Differential Revision: https://phab.mercurial-scm.org/D6900

upgrade: move most of revlog.clone method into a _clone method The content of the clone method now focus on parameters validation and processing. The `_clone` method focus on the actual cloning logic. Splitting the method out save some indentation and clarify each method code since it a focussed on one goal. Differential Revision: https://phab.mercurial-scm.org/D6899

flagprocessors: remove flagprocessorsmixin It became an empty shell. Differential Revision: https://phab.mercurial-scm.org/D6823

flagprocessors: move _flagserrorclass attribute on revlog & co This is a small duplication, and the last bit we need to get rid of the mixin. Honestly, I am not fan of that class attribute and it mostly exist to accomodate The simple-storage whose usage of flag processors is dumbious and that is currently dead code anyway. However I don't want to be pulled into futher unrelated cleaning so it is a small price to pay. Differential Revision: https://phab.mercurial-scm.org/D6822

flagprocessors: directly duplicate the deprecated layer back into revlog The code duplication benign and will get removed in a couple of month anyway. Differential Revision: https://phab.mercurial-scm.org/D6821

flagprocessors: make `processflagsraw` a module level function One more steps toward removing the mixin. Differential Revision: https://phab.mercurial-scm.org/D6820

flagprocessors: make `processflagsread` a module level function One more steps toward removing the mixin. Differential Revision: https://phab.mercurial-scm.org/D6819

flagprocessors: make `processflagswrite` a module level function One more step towards removing the mixin. Differential Revision: https://phab.mercurial-scm.org/D6818

flagprocessors: make `_processflagsfunc` a module level function This is the first step toward removing the flag processing mixin. Differential Revision: https://phab.mercurial-scm.org/D6817

flagprocessors: writetransform function take side data as parameter (API) If we want some flag processors to be able to store sidedata it needs to be actually fed that data. Differential Revision: https://phab.mercurial-scm.org/D6816

flagprocessors: add a `sidedata` parameters to _processflagswrite To read sidedata using flagprocessors, we need flag processors to store them. So we pass this information to the flag processing layer. Differential Revision: https://phab.mercurial-scm.org/D6815

revlog: add a `sidedata` parameters to addrevision If we want to eventually store sidedata we need to be able to pass them along. Differential Revision: https://phab.mercurial-scm.org/D6814

flagprocessors: have the read transform function return side data (API) This makes it possible for flag processors to -read- flag data. Differential Revision: https://phab.mercurial-scm.org/D6813

flagprocessors: return flagdata in the main processing function This function input and return are becoming stranger and stranger bnut I don't have a good plan to make is saner without problematic code duplication, so it will be this way to now. Differential Revision: https://phab.mercurial-scm.org/D6812

flagprocessors: return sidedata map in `_processflagsread` Right now, flag processors does not return sidedata, by they will. So, we prepare the caller to receive it. Differential Revision: https://phab.mercurial-scm.org/D6811

revlog: use the new sidedata map return in the sidedata method So far things, seems logical. Differential Revision: https://phab.mercurial-scm.org/D6810

revlog: return sidedata map from `_revisiondata` Nothing extra any side data yet. However, it will happens in the future. So we better prepare the callers of the `_revisiondata` to deal with it. Differential Revision: https://phab.mercurial-scm.org/D6809

revlog: introduce a `sidedata` method The method give access to extra information related to the revision. Such data will not be part of the hash be strongly related to the revision. Having them stored at the revlog level helps the storage consistency story and simplify various things. Example of data we could store there: - copy tracing related informations - graph structure related information (useful for discovery) - unresolved conflict data The full implementation will be introduced gradually in the coming changesets. Differential Revision: https://phab.mercurial-scm.org/D6808

update: clarify calculateupdate() call sites by specifying argument names merge.calculateupdate() takes a lot of parameters and I get confused all the time which is which. See also b14fdf1fb615 (update: clarify update() call sites by specifying argument names, 2017-02-09). Differential Revision: https://phab.mercurial-scm.org/D6883

bookmarks: remove changectx() method from bmstore (API) All the callsites of this method have access to the repo, and I'd rather not have to duplicate this across alternative bmstore implementations. Besides, it feels like a bit of a layering violation. .. api:: `mercurial.bookmarks.bmstore` no longer has a convenience method for looking up changectx instances from a bookmark name. Use `repo[repo.bookmarks[name]]` intead of `repo.bookmarks.changectx(name)`. Differential Revision: https://phab.mercurial-scm.org/D6884

histedit: sniff-test for untracked file conflicts before prompting for rules This bug is as old as histedit, which is more than 10 years! I'm a little sad about the extra calculations here that we're just going to throw out, but I don't see any better way to look for untracked file conflicts and I want the bug fixed. Differential Revision: https://phab.mercurial-scm.org/D6882

histedit: demonstrate breakage when `update` to a revision breaks I'm honestly impressed that nobody has hit this in the over a decade that histedit has existed, but here we are. Differential Revision: https://phab.mercurial-scm.org/D6881

rebase: track new nodes when --keep is set When --keep is passed with rebase, the new nodes created are not accessible from templates. This change enables accessing the newly-created nodes from nodechanges, just as if --keep was not set. Differential Revision: https://phab.mercurial-scm.org/D6880

uncommit: fix typo in help text Differential Revision: https://phab.mercurial-scm.org/D6874

phabricator: use exthelper to register commands, config, and templates Differential Revision: https://phab.mercurial-scm.org/D6875

merge: check argument value with if/raise instead of an assert This shouldn't make any difference for legal code, but it'll prevent the assertion from being optimized out if someone decides to do -O on our code. Differential Revision: https://phab.mercurial-scm.org/D6879

hg: have `updatetotally` more thoroughly check updatecheck argument (API) .. api:: `mercurial.hg.updatetotally` is now more thorough about checking its `updatecheck` keyword argument. Previously invalid values would have used the configured default updatecheck method, but now will raise ValueError. Differential Revision: https://phab.mercurial-scm.org/D6878

merge: replace magic strings with NAMED_CONSTANTS (API) .. api:: `mercurial.hg.update*` and `mercurial.merge.update` now expect a value from a set of NAMED_CONSTANTS (`merge.UPDATECHECK_*` constants) rather than a collection of magic strings. As of now, the values are the same, but code should be prepared for these values to change in the future. Differential Revision: https://phab.mercurial-scm.org/D6877

singlehead: introduce special handling of closed heads Until now, the experimental option `single-head-per-branch` was also refusing closed heads. The logic is now ignoring them by default and a suboption have been added to refuse them too `single-head-per-branch:account-closed-heads`.

testlib: allow more argument to mkcommit This is simple and handy. See next changesets for usage.

singlehead: fix a small typo in a test comment The sentence is now correct.

examples: add sample fix integration for `rustfmt` Differential Revision: https://phab.mercurial-scm.org/D6873

contrib: add new examples area and start it out with a config for `hg fix` This is the configuration contributors should use for our C/C++ code. I expect to expand this file as we get more automated formatter oversight. Differential Revision: https://phab.mercurial-scm.org/D6872

tests: recognize DNS timeouts as well Differential Revision: https://phab.mercurial-scm.org/D6870

transaction: detect an attempt to truncate-to-extend on playback, raise error On some networked filesystems, writes can have delayed finalization/confirmation and write races can occur such that a remote modification will "win" and modifications will be lost. There is no functionality for providing this feedback to userspace programs (in fact, there's not even functionality for providing this information to the Linux kernel...), so these programs may see the files suddenly change. We've noticed that there have been cases where Mercurial has detected something has gone wrong and attempts to abort (rolling back the transaction), which is good. However, when rolling back the transaction, for the append-only files, we attempt to "truncate" the file back to the size it was in before the hg transaction started, but end up *extending* it. This may be harmless, but if this happens to the 00changelog.i file, we get a bunch of nulls on the end of the file and this causes hg to become *really* confused. :) If we detect that some modification of the file outside of this Mercurial process has caused the file to be smaller than the size we are attempting to truncate to, let's just exit and stop trying to clean up the repository - continuing will likely just cause more damage. Differential Revision: https://phab.mercurial-scm.org/D6867

osutil: tolerate Py_GetArgcArgv not being set up properly Differential Revision: https://phab.mercurial-scm.org/D6866

osutil: allow disabling setprocname via a define passed to the compiler In some situations, we run a custom python launcher that appears to not set up Py_GetArgcArgv correctly. We then proceed to promptly crash when we attempt to dereference NULL. Being able to completely disable setprocname is beneficial in these situations, since we won't even attempt to use it, even if the case that causes the crash is fixed. Right now, if I compile osutil.so with -DSETPROCNAME_USE_NONE, the compilation fails on python3 due to SETPROCNAME_USE_NONE redefinition. I could possibly work around that, but it's likely helpful to have a way of disabling this completely without it being brittle (i.e. if python3 ever gains the ability to perform this operation). Differential Revision: https://phab.mercurial-scm.org/D6865

stack: use repo.revs() instead of revsetlang.formatspec() + scmutil.revrange() Using scmutil.revrange() it's possible to use multiple revsets at the same time, but we're not using that functionality in stack. I thought maybe that function could be used to make stack definition customizable (by combining various parts into one set), but scmutil.revrange() gives the union of all provided revsets, which is not very useful in stack's case (we want "and" between parts, not "or").

merge with stable

rust-hgpath: replace all paths and filenames with HgPath/HgPathBuf Differential Revision: https://phab.mercurial-scm.org/D6774

rust-hgpath: add HgPath and HgPathBuf structs to encapsulate handling of paths This change is a direct consequence of this discussion on the mailing list: https://www.mercurial-scm.org/pipermail/mercurial-devel/2019-August/133574.html The implementations of `HgPath` and `HgPathBuf` are, for the most part, taken directly from `OsStr` and `OsString` respectively from the standard library. What this change does *not* yet do is implement the Windows MBCS to WTF8 conversion, but it lays the basis for a very flexible interface for paths. Differential Revision: https://phab.mercurial-scm.org/D6773

wireprototypes: clarify documentation of getbundle argument types It seems like it was a mix of what the Python code would see and what was sent over the wire. I've tried to clarify both the type seen in Python and how it's transmitted. Differential Revision: https://phab.mercurial-scm.org/D6871

merge with stable

py3: don't double-convert "opts" to bytes The "opts" are already converted to bytes at the beginning of the function. Doing it twice results in a crash, which makes test-uncommit.t fail. The extra call was added recently, in ff1ff2aae132 (uncommit: add support to modify the commit message and date, 2019-09-07). test-uncommit.t passes again after this patch. Differential Revision: https://phab.mercurial-scm.org/D6864

py3: byte-prefix sanitisation regexes in phabricator.py So it doesn't die with "TypeError: cannot use a string pattern on a bytes-like object". Differential Revision: https://phab.mercurial-scm.org/D6863

py3: pass a bytestring into querydrev instead of a string that'll TypeError This was a regression I introduced in c19d259fd6ad. When the string gets to the memoryview in _tokenize under py3 it'll die. Differential Revision: https://phab.mercurial-scm.org/D6869