commandserver: remove redundant "if True" block

commandserver: extract handler of new socket connection This prepares for adding an IPC socket.

commandserver: pass around option to hook repo instance creation This is necessary to wrap a repo instance so the master process will be notified on repo.close().

py3: stop subscripting socket.error In 3.3 and later, this is now an alias for OSError. I hacked up the server code enough that I was able to trigger the exception handler in server.py from test-http-bundle1.t. Other instances of this either subscript through the `args` member, or reference the errno or strerror attributes. Note that on Windows, the errno value seems to reflect the Winsock error, so the various tests for EPIPE seem like they would always fail. But that seems to be the case in py2 as well.

test: update test-logtoprocess.t to not requires less The test previously assumed that a working pager was present in the test environment. Update it to use the fakepager instead. Differential Revision: https://phab.mercurial-scm.org/D5399

py3: quote several instances of $PYTHON for Windows Python3 lives in Program Files by default. The last remaining unquoted instance is in test-hghave.t. I can't figure out how to quote that, as it either complains: 'c:\\Program' is not recognized as an internal or external command, operable program or batch file. or $TESTTMP.sh: line 22: "C:/Program Files/Python37/python.exe": $ENOENT$

py3: adapt test-largefiles-wireproto.t to python3

py3: byteify test-push-race.t

py3: convert filename to bytes in test-hardlinks.t Windows was complaining about passing str to CreateFileA() in win32.py.

narrow: keep bookmarks temporarily stripped for as long as commits are The narrow extension also has support for shallowness and for inserting older commits on pull. It works by temporarily stripping newer commits, adding the older commits, then re-applying the stripped bundle. The regular Mercurial server uses that when you widen, although it shouldn't be necessary there. Our Google-internal server does it when the user requests an older commit. Our Google-internal tests fail since 7caf632e30c3 (filecache: unimplement __set__() and __delete__() (API), 2018-10-20). I haven't quite understood the problem, but it's related to the way we temporarily hide bookmarks while the commits they point to are stripped. When a transaction is started, Mercurial tries to read various things from the repo for the transaction summary. That leads to computation of hidden commits, which leads to an attempt to find commits pinned by bookmarks. This is the reason we temporarily hide the bookmarks. I think the aforementioned commit makes the restored bookmarks visible earlier than before (which seems like an improvement), so we can no longer incorrectly rely on the repo._bookmarks field being cached too long (IIUC). This patch makes it so we restore the temporarily hidden bookmarks only after the temporary bundle has been re-applied. It also adapts the code to update the repo.__bookmarks field using the pattern described in the aforementioned commit instead of writing directly to the fiels. I have spent many hours trying to understand what was going on here, but I still don't know if this can also happen without our custom server. So this patch unfortunately does not add any tests; I have only been able to test the fix using our Google-internal tests. Differential Revision: https://phab.mercurial-scm.org/D5398

narrow: drop obsolete support for old Mercurial Not needed since narrow is shipped with core. Differential Revision: https://phab.mercurial-scm.org/D5397

tests: test that narrow preserves bookmarks on widen When widening inserts older commits in the changelog, we have to preserve bookmarks so they are not removed by the call to repair.strip() we temporarily do. We didn't have any tests for that before. Differential Revision: https://phab.mercurial-scm.org/D5396

rebase: clarify that commits that become empty are skipped Our message just said that it created no changes, but we didn't explicitly say that that meant that the the commit was not rebased. It also wasn't clear why it created no changes, so this patch also clarifies that that was because the changes were already in the destination. Differential Revision: https://phab.mercurial-scm.org/D5395

rebase: format commit in usual way when explaining that it became empty We use the _ctxdesc() for other similar messages, so it seems we should use it here too. Differential Revision: https://phab.mercurial-scm.org/D5394

rust: propagate error of index_get_parents() properly Before, rustla_contains() would return 0 on error, and the exception would be cleared or noticed somewhere else. We need to propagate the error from AncestorsIterator up to the FFI surface.

rust: look up HgRevlogIndex_GetParents() from symbol table And removes the unused index_get_parents_checked() function. I expect the Index struct will be turned into a pyobject type, though I haven't written any PoC-level patches yet.

revlog: add public CPython function to get parent revisions Since this is a public function, it validates the input revision, and supports nullrev. index_get_parents_checked() will be replaced by this function.

revlog: rename indexType to HgRevlogIndex_Type as it's a global symbol It follows the CPython naming convention, but uses Hg instead of Py.

revlog: export symbol of indexType The idea is to wrap the index object with rust-cpython. I haven't tried it, but it should be doable. We'll probably need a better interface than raw function pointers to do more in Rust.

setup: fix path of rust source dependencies Appears that they were screwed up while moving the source files around. Since rust-direct-ffi sources are automatically added to the depends list, we only need to list hg-core sources.

rebase: remove now-unnecessary arguments to _abort() Since _abort() is now an instance method, we don't need to pass state that's available on "self". Differential Revision: https://phab.mercurial-scm.org/D5393

rebase: move abort() onto rebaseruntime The function depends on a lot of the state of rebaseruntime, so it makes sense for it to be an instance method. This will let us remove many of the arguments to the method. That will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D5392

cleanupnodes: trust caller when "moves" is not None If "moves" (indicating how to move bookmarks) is None, we fill it out based on "replacements" (indicating which obsmarkers to add). If "moves" is not None, we would still add items based on "replacements". This makes it impossible to pass "moves={}" and not move bookmarks, which surprised me. The only caller that currently passes a value for "moves" was the rebase extension and there we were already adding bookmark moves corresponding to obsmarker additions, so it should not be impacted. Differential Revision: https://phab.mercurial-scm.org/D5391

shelve: change transaction description from "commit" to "shelve" "commit" was probably a copy&paste mistake. Differential Revision: https://phab.mercurial-scm.org/D5390

shelve: drop unnecessary backup of dirstate for phase-based case Regular shelve has a hack using an uncommitted transaction that's then aborted at the end of the operation. It preserves the dirstate across the abort, however, by saving a backup copy of it. Phase-based shelve instead commits the transaction, so the hack shouldn't be necessary there. Differential Revision: https://phab.mercurial-scm.org/D5389

tests: split test-shelve.t in two test-shelve.t dominated run time for all shelve tests. Before: # Ran 9 tests, 1 skipped, 0 failed. real 0m43.568s user 2m15.822s sys 0m40.857s After: # Ran 11 tests, 1 skipped, 0 failed. real 0m24.574s user 2m21.354s sys 0m40.435s Differential Revision: https://phab.mercurial-scm.org/D5388

remotefilelog: rely on progress helper for keeping track of position The progress helper class keeps track of its current position, so we don't need a "count" variable for that. Differential Revision: https://phab.mercurial-scm.org/D5386

remotefilelog: reduce use of "count" container We had already given the initial value of "count[0]" a name, so just use that when applicable. Differential Revision: https://phab.mercurial-scm.org/D5385

remotefilelog: replace a "a=[expr]; b=a[0]" by "b=expr; a = [b]" Simpler is better. Differential Revision: https://phab.mercurial-scm.org/D5384

remotefilelog: remove an unnecessary update of "count" container The "count" container is never used after this point, so there is no need to update it. Differential Revision: https://phab.mercurial-scm.org/D5383

remotefilelog: avoid temporarily using "count" variable as synonym for "total" The "count" variable is generally used for updating progress, but early in fileserverclient.request(), its used to mean the total count. We already have another "total" variable for that, so it seems much clearer to use that. Differential Revision: https://phab.mercurial-scm.org/D5382

remotefilelog: use progress helper in fileserverclient Differential Revision: https://phab.mercurial-scm.org/D5381

remotefilelog: use progress helper in shallowrepo Differential Revision: https://phab.mercurial-scm.org/D5380

remotefilelog: use progress helper in shallowbundle Differential Revision: https://phab.mercurial-scm.org/D5379

remotefilelog: use progress helper in repack Differential Revision: https://phab.mercurial-scm.org/D5378

remotefilelog: use progress helper in remotefilelogserver Differential Revision: https://phab.mercurial-scm.org/D5377

remotefilelog: use progress helper in basestore Differential Revision: https://phab.mercurial-scm.org/D5376

remotefilelog: use progress helper when analyzing repos Differential Revision: https://phab.mercurial-scm.org/D5375

fastannotate: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5374

debugbuildannotatecache: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5373

upgrade: clarify "aggressivemergedelta" handling We rename "aggressivemergedelta" argument to "forceaggressivemergedelta". The previous argument naming could infer an absolute control on the behavior. However, the code show we respect the config option if set.

upgrade: test that fncache is valid after repository upgrade The behavior was correct. Now it is correct and tested.

tests: update `rm` invocation for new location of checklink This should fix the test on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5387

bookflow: new extension for bookmark-based branching This extension should be helpful for feature branches - based workflows. At my company we first considered branches, but weren't sure about creating a lot of permanent objects. We tried bookmarks, but found some scenarios to be difficult to control. The main problem, was the active bookmark being moved on update. Disabling that made everything a lot more predictable. Bookmarks move on commit, and updating means switching between them. The extension also implements a few minor features to better guide the workflow: - hg bookmark NAME can be ambiguous (create or move), unlike hg branch. The extension requires -rev to move. - require an active bookmark on commit. - some bookmarks can be protected (like @), useful for teams, that require code reviews. - block creation of new branches. The initial implementation requires no changes in the core, but it does rely on some implementation details. I thought it may be useful to discuss the functionality first, and then focus making the code more robust. Differential Revision: https://phab.mercurial-scm.org/D4312

rust: make clean takes care of rust/target This isn't the prettiest way of doing it, but it doesn't require looking up cargo, or wondering whether that should be part of setup.py clean. Differential Revision: https://phab.mercurial-scm.org/D5369

rust: rename local variables in AncestorsIterator::next It was confusing to have p1 and parents.1 ; (p1, p2) is clearer. Differential Revision: https://phab.mercurial-scm.org/D5365

tests: stabilize test-inherit-mode.t on FreeBSD and macOS again This is the same fix as 90e26ef4cbb1, just repeated on the new file location. Differential Revision: https://phab.mercurial-scm.org/D5371

tests: drop redundant "#if execbit" from test-rebase-inmemory.t The whole file is already guarded by "#require execbit". Differential Revision: https://phab.mercurial-scm.org/D5366

narrow: don't resurrect old commits when narrowing (don't strip obsmarkers) If you have an old obsolescence-chain of commits that has been pruned and you narrow your repo so that some of those commits get stripped (because they affected the removed paths), then we would currently resurrect the commit that came before (along the obsmarker chain) the last stripped commit. That happens by the usual rules for obsmarker-stripping. However, it's quite surprising when it happens when you narrow your repo. This patch makes narrowing not strip obsmarkers. Differential Revision: https://phab.mercurial-scm.org/D5364

commandserver: turn server debug messages into logs They were ui.debug() just because commandserver.log() was noop if no client connected.

commandserver: add config knob for various logging options The default rotating options are copied from the blackbox extension.

commandserver: expand log path for convenience This allows us to set the log path relative to $XDG_RUNTIME_DIR, for instance. [cmdserver] log = $XDG_RUNTIME_DIR/chg/server.log

commandserver: switch logging facility to ui.log() interface The "pager subcommand" message is removed since ui isn't accessible there. I think that's okay as cmdtable[cmd]() will call attachio() and some debug message will be printed.

commandserver: install logger to record server events through canonical API The global commandserver.log() will be replaced with this.

commandserver: enable logging when server process started This allows us to keep track of server events before client connects to the server. Tests will be added later. Currently there's no log() call to check if things are working well.

test-commandserver: change way of triggering early crash Future patches will move the logging facility out of the server class, so cmdserver.log can't be (ab)used for this purpose. Instead, let's hook the factory function to raise exception.

loggingutil: add basic logger backends These classes will be used in command server. They are similar to the blackboxlogger, but it can't be factored out since the blackbox is so tightly coupled with a repo object.

merge with stable

rust: peek_mut optim for lazy ancestors This is one of the two optimizations that are also present in the Python code: replacing pairs of pop/push on the BinaryHeap by single updates, hence having it under the hood maintain its consistency (sift) only once. On Mozilla central, the measured gain (see details below) is around 7%. Creating the PeekMut object by calling peek_mut() right away instead of peek() first is less efficient (gain is only 4%, stats not included). Our interpretation is that its creation has a cost which is vasted in the cases where it ends by droping the value (Peekmut::pop() just does self.heap.pop() anyway). On the other hand, the immutable peek() is very fast: it's just taking a reference in the underlying vector. The Python version still has another optimization: if parent(current) == current-1, then the heap doesn't need to maintain its consistency, since we already know that it's bigger than all the others in the heap. Rust's BinaryHeap doesn't allow us to mutate its biggest element with no housekeeping, but we tried it anyway, with a copy of the BinaryHeap implementation with a dedicaded added method: it's not worth the technical debt in our opinion (we measured only a further 1.6% improvement). One possible explanation would be that the sift is really fast anyway in that case, whereas it's not in the case of Python, because it's at least partly done in slow Python code. Still it's possible that replacing BinaryHeap by something more dedicated to discrete ordered types could be faster. Measurements on mozilla-central: Three runs of 'hg perfancestors' on the parent changeset: Moyenne des médianes: 0.100587 ! wall 0.100062 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.135804 comb 0.130000 user 0.130000 sys 0.000000 (max of 98) ! wall 0.102864 comb 0.102755 user 0.099286 sys 0.003469 (avg of 98) ! wall 0.101486 comb 0.110000 user 0.110000 sys 0.000000 (median of 98) ! wall 0.096804 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.132235 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.100258 comb 0.100300 user 0.096000 sys 0.004300 (avg of 100) ! wall 0.098384 comb 0.100000 user 0.100000 sys 0.000000 (median of 100) ! wall 0.099925 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.133518 comb 0.140000 user 0.130000 sys 0.010000 (max of 98) ! wall 0.102381 comb 0.102449 user 0.098265 sys 0.004184 (avg of 98) ! wall 0.101891 comb 0.090000 user 0.090000 sys 0.000000 (median of 98) Mean of the medians: 0.100587 On the present changeset: ! wall 0.091344 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.122728 comb 0.120000 user 0.110000 sys 0.010000 (max of 100) ! wall 0.093268 comb 0.093300 user 0.089300 sys 0.004000 (avg of 100) ! wall 0.092567 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.093294 comb 0.080000 user 0.080000 sys 0.000000 (best of 100) ! wall 0.144887 comb 0.150000 user 0.140000 sys 0.010000 (max of 100) ! wall 0.097708 comb 0.097700 user 0.093400 sys 0.004300 (avg of 100) ! wall 0.094980 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.091262 comb 0.090000 user 0.080000 sys 0.010000 (best of 100) ! wall 0.123772 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.093188 comb 0.093200 user 0.089300 sys 0.003900 (avg of 100) ! wall 0.092364 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) Mean of the medians is 0.0933 Differential Revision: https://phab.mercurial-scm.org/D5358

fuzz: grep away HAVE_GETC_UNLOCKED in pyconfig.h to avoid msan badness Per discussion with Greg Smith and the patches on https://bugs.python.org/issue35214. This, combined with the previous patch, fixes msan builds on oss-fuzz. Differential Revision: https://phab.mercurial-scm.org/D5363

fuzz: more correctly specify CFLAGS and LDFLAGS when building Python Gets us closer to a working msan build alongside our asan build. Differential Revision: https://phab.mercurial-scm.org/D5362

tests: stabilize test-blackbox.t on Windows I didn't look into why the error is more detailed, but that seems like it's a good thing (other than for recording tests).

tests: stabilize for recent wcache changes This goes with 47e3f554df35::d5622dfe4ba3. I'm not sure if it was really expected that there would be no wcache directory if neither execbit nor symlink is supported.

extdiff: avoid double backslashes in the displayed tool path on Windows This shows the tool path in the help, and changed in 67b180c0e263. uirepr() already does the same thing, but that undoes the mangling in its call to repr().

contrib: add a helper script that help to build interesting repositories The script is dedicated to building a couple of repositories that should be interesting to run discovery from one another. It seems a common enough need to contribute it upstream.

py3: listify filter() to call len() on it Differential Revision: https://phab.mercurial-scm.org/D5354

loggingutil: document openlogfile() This function will be used later for command-server logging.

loggingutil: extract openlogfile() and proxylogger to new module This module isn't placed under the "utils" package since it needs "ui" to process things. It's called "loggingutil", not "logutil" because the word "log" is too obscure in our codebase.

blackbox: pass in options to _openlogfile() as arguments This prepares for extracting utility function from the blackbox module.

blackbox: just try writing to repo.vfs and update lastlogger on success This is simpler and more robust. Before, an empty ".hg" directory would be created if it's removed after checking vfs.isdir('.').

vfs: add option to not create parent directories implicitly In blackbox, we don't want to create a ".hg" directory by mistake. This provides a race-safe option to achieve that.

repo: add a `wcachevfs` to access the `.hg/wcache/` directory This wvfs will allow us to migrate various cache to the new `wcache` directory. Helping with cache issues with "share".

cache: create `wcache` directory at init time The cache directory will be needed very quickly, so it seems simpler to create it early to make sure it has the same owner and permission than the other directory in the repository.

cache: create `cache` directory at init time The cache directory will be needed very quickly, so it seems simpler to create it early to make sure it has the same owner and permission than the other directory in the repository.

check-exec: write file in 'wcache' instead of 'cache' Some cache are relevant or affected by the working copy used. So the `.hg/cache` directory is not the best place for them because multiple shared repository can end up fighting over them. To address this issue, we introduce a new 'wcache' directory to host this kind of cache. The first user are the `checkisexec` type file. These files describe property of the working copy and fit the use-case well.

mmapindex: set default to 1MB mmapping index is more efficient if we only need a small part of it. The 1MB value has been picked arbitrarily, a lower value might be better. On a large repository with a 60MB index, we see the following performance gain: hg perfindex before: ! wall 0.032023 comb 0.040000 user 0.000000 sys 0.040000 (best of 100) after: ! wall 0.000196 comb 0.000000 user 0.000000 sys 0.000000 (best of 1060) The speed boost benefit all cases, including the one where the full index needs to be parsed. hg perfindex --rev 0 before: ! wall 0.040673 comb 0.030000 user 0.000000 sys 0.030000 (best of 100) after ! wall 0.010713 comb 0.020000 user 0.010000 sys 0.010000 (best of 212) This gain reflect in higher level operation: hg perfbookmarks --clear-revlogs before: ! wall 0.161339 comb 0.160000 user 0.130000 sys 0.030000 (best of 56) after: ! wall 0.123228 comb 0.120000 user 0.120000 sys 0.000000 (best of 68)

mmapindex: move the 'mmapindexthreshold' option out of experimental The option is useful and should be advertised more. We move it out of experimental as a first step. The `storage` section is selected as this is related to how the storage is accessed. A new 'performance' section might be more appropriate. We move from 'mmapindexthreshold` to `mmap-threshold` as non-index item are also suitable for mmap (eg: the rev-branch-cache). If relevant, we can introduce sub-option `mmap-threshold.revlog-index` later.

perf: add a --rev attribute to perfindex This allow for benchmarking the time necessary to look for other version than the tip.

perf: update perfindex to be more realistic The previous code was creating a revlog manually, we now use the actual `localrepo` method to create it. We have to jump though extra hops to work around the impact of filecache.

match: drop unnecessary wrapping of regex in group It seems the regexes have been wrapped in an unnamed group since b6c42714d900 (Add locate command., 2005-07-05). In that commit, the grouping was needed because there was a "head" ('^') added before the group and a "tail" (os.sep) added after it. It seems the head was moved inside the group in 1c0c413cccdd (Get add and locate to use new repo and dirstate walk code., 2005-07-18) and the tail was moved inside the group in 89985a1b3427 (Clean up walk and changes code to use normalised names properly., 2005-07-31), So it seems to me that we've carried around the unnecessary group for 13 years. This patch removes it. Differential Revision: https://phab.mercurial-scm.org/D5352

match: use _BASE_SIZE instead of magic value 4 Differential Revision: https://phab.mercurial-scm.org/D5351

match: make "groupsize" include the trailing "|" I think this is a little easier to follow and it will simplify later patches too. Differential Revision: https://phab.mercurial-scm.org/D5350

match: fix an unaligned (but harmless) indent Differential Revision: https://phab.mercurial-scm.org/D5349

match: raise an Abort error instead of OverflowError This case of OverflowError (one single pattern being too large) has never been properly caught in the past.

match: avoid translating glob to matcher multiple times for large sets For hgignore with many globs, the resulting regexp might not fit under the 20K length limit. So the patterns need to be broken up in smaller pieces. Before this change, the logic was re-starting the full process from scratch for each smaller pieces, including the translation of globs into regexp. Effectively doing the work over and over. If the 20K limit is reached, we are likely in a case where there is many such glob, so exporting them is especially expensive and we should be careful not to do that work more than once. To work around this, we now translate glob to regexp once and for all. Then, we assemble the resulting individual regexp into valid blocks. This raises a very significant performance win for large `.hgignore file`: Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) After: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100)

match: extract function that group regexps

match: test for overflow error in pattern If a single pattern is too large to handle, we raise an exception. This case is now doctested.

match: extract a literal constant into a symbolic one

tests: apply binary mode to output in seq.py I noticed this when playing with running tests using WSL, and iterating over the output yielded '0\r', '1\r',... Most of the other *.py tools do this, and `seq` on MSYS lacks '\r' in the output, so this is more consistent.

perf: add a `--clear-caches` to `perfbranchmapupdate` This flag will help to measure the time we spend loading various cache that support the branchmap update. Example for an 500 000 revisions repository: hg perfbranchmapupdate --base 'not tip' --target 'tip' ! wall 0.000860 comb 0.000000 user 0.000000 sys 0.000000 (best of 336) hg perfbranchmapupdate --base 'not tip' --target 'tip' --clear-caches ! wall 0.029494 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)

perf: start from an existing branchmap if possible If the --base set if a superset of one of the cached branchmap, we should use as a starting point. This greatly help the overall runtime of `hg perfbranchmapupdate` For example, for a repository with about 500 000 revisions, using this trick make the command runtime move from about 200 second to about 10 seconds. A 20x gain.

perf: rely on repoview for perfbranchmapupdate Using 'repoview' matching the base and target subset make the benchmark more realistic. It also unlocks optimization to make the command initialization faster.

perf: pre-indent some code in `perfbranchmapupdate` This make the next patch easier to read.

perf: add a `perfbranchmapupdate` command This command benchmark the time necessary to update the branchmap between two sets of revisions. This changeset introduce a first version, doing nothing fancy regarding cache or other internal details.

push: config option to control behavior when pushing to a publishing server Pushing to a publishing server by mistake can lead to a difficult situation to solve because evolution doesn't work on public changesets. This new experimental config tries to help avoiding unintentionally (or at least being aware of) pushing to publishing remotes. `hg push --publish` can be used to make push succeed even when auto-publish is set to 'abort'.

narrowcommands: remove an unrequired `repo.narrowpats` call We call that few lines above and do nothing significant in between which can change the narrowpats. So let's use values returned by that call. Differential Revision: https://phab.mercurial-scm.org/D5348

manifest: reject lines shorter than 42 bytes, not 22 Yuya correctly spotted during the review of f27f8e9ef1e73 that we're dealing with hexlified hashes here, and so it should be 42 bytes not 22. Differential Revision: https://phab.mercurial-scm.org/D5347

blackbox: initialize logger with repo instance The blackboxlogger is unusable without a repo. Let's simply initialize it with a repo instance.

blackbox: do not nullify repo to deactivate the logger on failure The _repo will be a mandatory attribute. Instead, make the logger to not track any events.

blackbox: extract global last logger to proxylogger class So the blackboxlogger can be instantiated with a repo.

ui: pass in bytes opts dict to logger.log() This is the convention of the Mercurial API.

ui: pass in formatted message to logger.log() This makes sure that all logger instances will handle the message arguments properly.

blackbox: send debug message to logger by core ui Since the core ui.log() may recurse into ui.log() through ui.debug(), it must guard against recursion. The ui extension class can finally be removed.

blackbox: change the way of deactivating the logger on write error This prepares for the upcoming code move. The recursion guard will be ported to the core ui.

match: remove obsolete catching of OverflowError Since 0f6a1bdf89fb (match: handle large regexes, 2007-08-19), we catch an OverflowError from the regex engine and split up the regex if that happens. In 59a9dc9562e2 (ignore: split up huge patterns, 2008-02-11), that was extended to raise an OverflowError in our code even if the regex engine doesn't raise it. It's unclear if there was a range of regex sizes where the OverflowError would be raised from the regex engine but that were still below the limit we added in our code. Either way, both limitations were probably removed in Python 2.7.4 when the regex code width was extended from 16bit to 32bit (or Py_UCS4) integer (thanks to Yuya for finding that out). If at least the first limitation was removed, we no longer should be using OverflowError for flow control, so this patch changes that. Differential Revision: https://phab.mercurial-scm.org/D5309

sparse: raise a move verbose index error from the C code If we don't like a value we should print it.

narrow: drop the bundle2 capability since we have server capabilities (BC) This patch drops the narrow bundle2 capabilities since we introduced narrow server capabilities which are more nice and now used everywhere. I am not sure what it can affect, so on safe side I marked this as BC. Also I removed the NARROWCAP constant as that kind of conflicts with the same name constant in wireprototypes.py. Differential Revision: https://phab.mercurial-scm.org/D4892

vfs: extract the audit path logic into a submethod This will make it possible to apply it in more cases.

subrepo-git: use an official origvfs when appropriate The origvfs has the auditor properly set and can move file without issue. The current code is currently working without errors because rename are not audited, yet.

revert: extract origvfs logic in a sub-function The subrepo's "revert" logic could benefit from it.

vfs: treat 'undo.' file the same as 'journal.' file They are the same kind of file, they are protected by the store lock, but directly lives inside the '.hg' directory. No warnings were ever raised about them because `vfs.rename` is not audited. Something we are trying to change.

perf: add a perfignore command The command is meant to benchmark operations related to hgignore. Right now the command is benchmarking the loading time of the hgignore rules.

py3: use pycompat.xrange instead of xrange xrange does not exists on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5302

store: write fncache only once if there are both adds and removes Differential Revision: https://phab.mercurial-scm.org/D5307

perf: disable revlogs clearing in `perftags` by default This aligns things with what `perfbookmarks` does. I decided to disable the revlogs clearing by default to focus on the core logic by default, ignoring side effects. If we prefer to emphasize the side effect, we can instead keep this on in `perftags` and enable it by default in `perfbookmarks`.

perf: add a `clear-revlogs` flag to `perfbookmarks` This flag (off by default) makes it possible to enable the refresh of the changelog and revlog. This is useful to check for costly side effects of bookmark loading. Usually, these side effects are shared with other logics (eg: tags). example output in my mercurial repo (with 1 bookmark, so not a great example): $ hg perfbookmarks ! wall 0.000044 $ hg perfbookmarks --clear-revlogs ! wall 0.001380

tags: cache `repo.changelog` access when checking tags nodes The tags reading process checks if the nodes referenced in tags exist. Caching the access to `repo.changelog` provides a large speedup for repositories with many tags. running `hg perftags` in a large private repository before: ! wall 0.393464 comb 0.390000 user 0.330000 sys 0.060000 (median of 25) after: ! wall 0.267711 comb 0.270000 user 0.210000 sys 0.060000 (median of 38)

perf: add a `clear-revlogs` flag to `perftags` This flag (on by default) makes it possible to disable the refresh of the changelog and revlog. This is useful to check for the time spent in the core tags logic without the associated side effects. Usually, these side effects are shared with other logics (eg: bookmarks). Example output in my Mercurial repository $ hg perftags ! wall 0.017919 comb 0.020000 user 0.020000 sys 0.000000 (best of 141) $ hg perftags --no-clear-revlogs ! wall 0.012982 comb 0.010000 user 0.010000 sys 0.000000 (best of 207)

perf: stop creating new revlog by hand in perftags It's better to let the repository logic create its own object. We now just clear the cache. New object will be automatically created from there.

revlog: update the documentation for `trim_endidx` The function role drifted since the function was commented.

revlog: properly detect corrupted revlog in `index_get_length` Pointed out by Yuya Nishihara.

perf: rename `perfhelper-tracecopies` to `perfhelper-pathcopies` The command it supports is called `perfpathcopies`. It seems better to align the names.

perf: add a docstring to `perfpathcopies` This will help people to find this command.

revlog: update the docstring of `ancestors` to match reality Code using this method expect the revision to be (reverse) sorted. As pointed by Yuya Nishihara, the docstring should reflect that.

remotefilelog: fix typo in docstring Differential Revision: https://phab.mercurial-scm.org/D5306

store: append to fncache if there are only new files to write Before this patch, if we have to add a new entry to fncache, we write the whole fncache again which slows things down on large fncache which have millions of entries. Addition of a new entry is common operation while pulling new files or commiting a new file. This patch adds a new fncache.addls set which keeps track of the additions happening and store them. When we write the fncache, we will just read the addls set and append those entries at the end of fncache. We make sure that the entries are new entries by loading the fncache and making sure entry does not exists there. In future if we can check if an entry is new without loading the fncache, that will speed up things more. Performance numbers for commiting a new file: mercurial repo before: 0.08784651756286621 after: 0.08474504947662354 mozilla-central before: 1.83314049243927 after: 1.7054164409637451 netbeans before: 0.7953150272369385 after: 0.7202838659286499 pypy before: 0.17805707454681396 after: 0.13431048393249512 In our internal repo, the performance improvement is in seconds. I have used octobus's ASV perf benchmark thing to get the above numbers. I also see some minute perf improvements related to creating a new commit without a new file, but I believe that's just some noise. Differential Revision: https://phab.mercurial-scm.org/D5301

py3: fix couple of division operator to do integer divison Differential Revision: https://phab.mercurial-scm.org/D5305

py3: use dict.items() instead of dict.iteritems() dict.iteritems() does not exist on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5304

py3: convert strings to bytes in tests/test-remotefilelog-histpack.py Differential Revision: https://phab.mercurial-scm.org/D5303

py3: replace str() with pycompat.bytestr() or ('%d' % int) After this patch, test-fastannotate-diffopts.t is about to pass. There are some extra newlines in the output. Differential Revision: https://phab.mercurial-scm.org/D5300

perf: add a `--timing` argument to `perfhelper-tracecopies` The new argument will help picking better pair for benchmarking. See documentation for details.

ui: manage logger instances and event filtering by core ui The setup code in blackbox needs more tweaks since it has lots of black magics. I'll fix them by follow-up patches. To be clear, the goal of this series is to provide a proper way for command server to install its own logger. I need it to debug in-memory repository cache.

extensions: add "uipopulate" hook, called per instance, not per process In short, this is the "reposetup" function for ui. It allows us to modify ui attributes without extending ui.__class__. Before, the only way to do that was to abuse the config dictionary, which is copied across ui instances. See the next patch for usage example.

hgweb: load globally-enabled extensions explicitly Before, extensions were loaded as a side effect of hg.repository() if the hgweb was executed as a CGI/WSGI. I want to make it explicit so that another ui hook can be inserted after extensions.loadall().

tests: disable remotefilelog on Windows I've spent a non trivial amount of time trying to eliminate the test errors, but it's looking like this is pretty dependent on Unix support. For example, there are attempts to delete open files, and uses of threads that report I/O attempts on closed files. (Maybe this is a race condition? Don't we usually use processes as workers on Windows?) In any event, I don't want real new errors elsewhere to be masked by these known problems. For some reason $CACHEDIR is reported as missing in test-remotefilelog-repack.t, but it actually exists in the hgcloneshallow call inside shallowutil.mkstickygroupdir(). By the time the process exits, it's gone. I don't see it being removed by code that calls 'rmdir' or 'remove' in the extension itself.

perf: run 'setup' function during stub run The benchmarked function might need the content of the setup to be run in order to function properly.

perf: fallback to subset if ondisk cache is missing in perfbranchmapload If there is no branchmap on disk for that filter, it means that the cache from some subset's filter is relevant for this one. We look for it instead of aborting. That way it is much simpler to run the command in an automated way. We can now add it to `test-contrib-perf.t`.

perf: prewarm the branchmap in perfbranchmapload It is not very interesting to have the command randomly failing because the branchmap for the tested filter happens to be cold. So we make sure to have a valid up to date branchmap before going further. The data might still be missing from disk if a subset was equivalent. See next changeset for details and fix.

localrepo: correct docstring of filectx() The same reason as b6c2543e1dd8. It can't be any changeset specifiers but revision number.

annotate: remove dead code to not convert path to relative path It's annotate. There should be at least one file path specified.

annotate: start pager soon after command options are validated It helps extracting helper class.

annotate: check and update command options first It helps extracting helper class.

annotate: format local tables in less-dense form I'll move these to a class, where one-extra indent will be needed.

perf: rename version flag of perfchangegroupchangelog to cgversion The version flag conflicts with the global version flag and does not work. Differential Revision: https://phab.mercurial-scm.org/D5297 Differential Revision: https://phab.mercurial-scm.org/D5298

procutil: don't allow the main 'hg' script to be treated as the Windows exe Previously, there were a handful of errors like this: $ hg prefetch --repack (running background incremental repack) + abort: %1 is not a valid Win32 application + [255] CreateProcess() doesn't append .exe when `lpApplicationName` contains a path, and a python script isn't directly executable.

remotefilelog: drop some compatibility cruft for finding the hg exeutable

branchmap: build the revbranchcache._namesreverse() only when required On big repositories with a lot of named branches and that also increasing over time, building of this dict can be expensive and shows up in profile. For our internal repository, this saves ~0.05 seconds. Thanks to Yuya for suggesting using util.propertycache() and util.clearcachedproperty(). Differential Revision: https://phab.mercurial-scm.org/D5291

sparse-revlog: put the native implementation of slicechunktodensity to use When possible, the C implementation of delta chain slicing will be used. providing a large boost in performance for this operation. To take a practical example of restoring manifest revision '59547c40bc4c' for a reference NetBeans repository (using sparse-revlog). The media time of the step `slice-sparse-chain` of `perfrevlogrevision` improve from 0.660 ms to 0.098 ms; The full series move delta chain slicing from 1.120 ms to 0.098 ms; Implementing _slicechunktosize into C would yield further improvements. However, the performance seems good enough for now.

cext: increment the `parsers` module version We introduced a new `slicechunktodensity` method to the index.

sparse-revlog: introduce native (C) implementation of slicechunktodensity This is a C implementation of `_slicechunktodensity` in the `mercurial/revlogutils/deltas.py` file. The algorithm involves a lot of integer manipulation and low-level access to index data. Having a C implementation of it raises a large performance improvement. See later changeset in this series for details.

sparse-revlog: add a `trim_endidx` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This function implement a subpart of the python function `_trimchunk` in `mercurial/revlogutils/deltas.py`. Handling of actual Python objects is left to the caller function.

sparse-revlog: add a `index_segment_span` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

sparse-revlog: add a `index_get_length` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

sparse-revlog: add a `index_get_start` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

perf: add --clear-revlog flag to branchmapload Having the changelog index already loaded when loading the branchmap can have a large impact on performance. Example runs (large private repository): hg perfbranchmapload -f base ! wall 0.116722 comb 0.120000 user 0.110000 sys 0.010000 (best of 59) hg perfbranchmapload -f base --clear-revlogs ! wall 0.258246 comb 0.230000 user 0.220000 sys 0.010000 (best of 31)

perf: introduce a function to fully "unload" a changelog The function remove various attributes and caches related to changelog. This is getting a common requirement.

perf: use an explicit function in perfbranchmapload This make things clearer.

perf: update function name to match `perfbranchmapload` command Having function with the same name as the command is simpler.

perf: fix a minor typo in perfbranchmapload

context: floor adjustlinkrev graph walk during copy tracing The `_adjustlinkrev` method gains an optional "stoprev" argument. The linkrev adjustment will give up once this floor is reached. The relevant functions using `_adjustlinkrev` are updated to pass an appropriate value in the copy tracing code. In some private repository, about 10% of the status call triggered the pathological case addressed by this change. The speedup varies from one call to another, the best-observed win is moving from 170s to 11s. The effect of this change can be seen in the public pypy repository, running the following command: hg perftracecopies --source 83c9ff0c0206 --destination 59c79103d5b0 before: 3.401753 seconds after: 2.634897 seconds (-23%)

context: small refactoring of `isintroducedafter` This make the next change clearer.

context: split `introrev` logic in a sub function We want to add a mechanism to stop iteration early associated to intro rev early in some case. However, it does not make sense to expose it in the public `filectx` API. So we split the code into an internal method instead.

context: introduce an `isintroducedafter` method and use it in copies Right now, copy tracing make effort to not traverse the graph too much to save performance. It uses a "limit" acting as a floor revision past which data are no longer relevant to the current copy tracing. However, to enforce this limit, it does a call to `filectx.rev()` and that call can trigger a graph traversal on its own. That extra graph traversal is unaware of the current limit and can become very expensive. That cost is increased by the nature of work done in adjust link rev, we are not only walking down the graph, we are also checking the affected file for each revision we walk through. Something significantly more expensive than the walk itself. To work around this we need to make the `filectx` operation aware of the current limit. The first step is to introduce a dedicated method: `isintroducedafter`. We'll then rework that method logic to stop traversal as soon as possible.

context: take advantage of `_descendantrev` in introrev if available Before this changeset, `_descendantrev` was ignored and `introrev` could return a "wrong" result. I was previously fine because there seems to be no existing code using both `introrev` and `_descendantrev` at the same time. However, we would like to change that.

context: spell out the logic around linkrev adjustement starting point We make the intent of the `_changeid` and `_changectx` checks explicit. The same logic was previously performed by the `self.rev()` call. The new code is a bit redundant, but much clearer.

perf: add a new `perfhelper-tracecopies` command The command is not measuring performance itself, it digs interesting statistic to help pick good arguments for the `perfcopytrace` command.

narrow: extract helper for parsing narrowspec file This will be used for parsing a file that's stored in a different location. Differential Revision: https://phab.mercurial-scm.org/D5277

py3: cast attribute name to sysstr in clearcachedproperty()

py3: whitelist three more passing test caught by buildbot The number of failing test is back to two digits. Differential Revision: https://phab.mercurial-scm.org/D5293

tests: make test-check-module-imports more robust It failed for me without this in this way: tests/test-commandserver.t:19: relative import of stdlib module tests/test-lfs-serve.t:108: relative import of stdlib module tests/test-lfs-serve.t:255: relative import of stdlib module tests/test-lfs-serve.t:362: relative import of stdlib module tests/test-lfs-serve.t:406: relative import of stdlib module tests/test-lock.py:9: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-lrucachedict.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-match.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-remotefilelog-datapack.py:15: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-remotefilelog-histpack.py:14: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-simplekeyvaluefile.py:4: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-sshserver.py:6: imports not lexically sorted: silenttestrunner < unittest, True, True This is because every module is considered a stdlib module, because the stdlib_prefixes is /usr, and my repo is in /usr/local/home, which means that sys.path contains a couple of /usr/local/home/../hg/.. entries that count as "in the stdlib". Fix this by preventing any path in sys.path that's inside the mercurial source from being considered "in the stdlib". Differential Revision: https://phab.mercurial-scm.org/D5294

push: add --publish flag to change phase of pushed changesets This flag is copied from topic extension, where it proved to be useful. It makes all pushed changesets public on both ends if the push succeeds. Doesn't currently work for any subrepos that are implicitly pushed together with their owner repo.

match: reformat `syntaxes` dictionary for better maintainability

match: provide and use a quick way to escape a single byte The previous function has a lot of overhead (including being a function). In the `_globre` case, we always escape a single byte. So we provide a dictionary dedicated to this use case. We directly use the dictionary to avoid a function call, these are expensive in Python. Again, this raise a very significant performance gain: Before: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100) After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146) Total improvement for the full series: Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146)

perf: make `clearfilecache` helper work with any object If the object is not a localrepo, it won't have an `unfiltered` method (and won't need one).

perf: move some of the perftags benchmark to the setup function Creating fresh objects and clearing the cache should not be part of the timing. before: ! wall 0.020851 comb 0.020000 user 0.020000 sys 0.000000 (median of 138) after: ! wall 0.018740 comb 0.020000 user 0.020000 sys 0.000000 (median of 141)

perf: use the new setup function in "perfbookmarks" This command was picked arbitrarily to display the usefulness of the new feature. In my Mercurial repository (with very few bookmarks), moving cache cleanup in the dedicated setup function has a visible and stable effect on the benchmark number. before: ! wall 0.000061 comb 0.000000 user 0.000000 sys 0.000000 (median of 40837) after: ! wall 0.000058 comb 0.000000 user 0.000000 sys 0.000000 (median of 40500)

perf: add a `setup` argument to run code outside of the timed section With this new argument, it is possible to perform special setup and cleanup outside of code actually timed. This is useful to provide more accurate benchmark.

perf: explicitly pass title as a keyword argument in `perfdiffwd` This will help to update the timer function arguments in a later changeset.

ui: unify argument name of ui.log() It's called an "event" in both blackbox and logtoprocess.

logtoprocess: extract logger class from ui wrapper It provides the same interface as the blackboxlogger. The ui wrapper will be removed shortly.

merge with stable

py3: use node.hex(m.digest()) instead of m.hexdigest() hashlib.sha1.hexdigest() returns str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5287

tests: add Balto configuration file I have been developing a Mercurial test runner plugin for being able to run Mercurial tests with Balto (https://bitbucket.org/lothiraldan/balto/src/default/). Balto requires a configuration file so let's include it, I have added the required information in the configuration file as comments. I hope Balto would be an helpful tool for other people than me. Differential Revision: https://phab.mercurial-scm.org/D5283

tests: make test-alias.t pass with re2 Locally, these "non-ASCII character in alias" errors don't show up, though I get them when the alias is defined at the command line rather than in an hgrc. The brokenness comes from the fact that hgrcs are parsed with regexes, and re/re2 differ in this way: $ python -c 'import re; print(re.compile("(.*)").match("aaa\xc0bbbb").groups())' ('aaa\xc0bbbb',) $ python -c 'import re2; print(re2.compile("(.*)").match("aaa\xc0bbbb").groups())' ('aaa',) Apparently re2 stops when it encounters invalid utf8 (which I suppose makes sense given that '.' matches what appears to be a codepoint rather than a byte). This is presumably a bug in hg, but not very important, so just change the test to stick to valid utf8. Differential Revision: https://phab.mercurial-scm.org/D5288

context: remove seemingly impossible code branch I'm not a Python expert, but I can't think of a way that the following branch can ever be hit: def _changeid(self): if r'_changeid' in self.__dict__: return self._changeid It seems to me that if that condition is true, then this function would not have been called. The only exception I can think of is if a reference to the function had been stored beforehand, something like this: c = fctx.__dict__['_changeid'] fctx._changeid c() But that seems like very unlikely code to exist. The condition was added in 921b64e1f7b9 (filecontext: use 'is not None' to check for filelog existence, 2013-05-01) as a "bonus" change (in addition to what the patch was actually about) Differential Revision: https://phab.mercurial-scm.org/D5289

py3: use pycompat.byteskwargs() to covert keys of kwargs to bytes Differential Revision: https://phab.mercurial-scm.org/D5286

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D5285

py3: add 9 new passing tests caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5284

branch: allow changing branch of merge commits with --rev Tests show that changing branch of merge commits works fine with evolution and without, so let's allow it. Other safeguards should prevent users from shooting themselves in the foot.

lfs: ensure that the return of urlopener.open() is closed No problem observed, just an oversight noticed while reading documentation.

changegroup: avoid instantiating storage if we are not using it Differential Revision: https://phab.mercurial-scm.org/D5280

http: allow 'auth.prefix' to have a username consistent with the URI It may be a little weird to put a username in the prefix, but the documentation doesn't disallow it, and silently disallowing it has caused confusion[1]. The username must match what is passed in (which seems to be from the URI via a circuitous route), as well as 'auth.username' if it was specified. I thought about printing a warning for a mismatch, but we already don't print a warning if the 'auth.username' and URI username don't match. This change allows the first and second last new test cases to work as expected. It looks like this would have been a problem since at least 0593e8f81c71. [1] https://www.mercurial-scm.org/pipermail/mercurial/2018-November/051069.html

lfs: make the exception messages consistent I don't love that it repeats 'HTTP Error' in an already long message, but I doubt that we should assume that it will always say that on the original exception message.

lfs: handle URLErrors to add additional information Sometimes the blob server is hit first (e.g. on push), and sometimes it's hit last (e.g. pull). Throw in depth first subrepo operations, and things quickly get insane. It wasn't even mentioning LFS, so just saying "connection refused" can be confusing- especially if the blob server is a secondary server and connecting to the repo server works. The exception handler for the transfer handler will print the full path to the blob, but that seems fine given that it might be necessary to debug a second server. (We don't yet support a standalone blob server, so the handler for the Batch API will cover 99.9% of the current problems. But it might as well be handled now while I'm thinking about it.) The function for translating to a message was mostly borrowed from scmutil.catchall().

lfs: improve the hints for common errors in the Batch API The previous message was too debug-ish and less action oriented than a hint should be. The remaining errors that aren't handled are more along the lines of programming errors (not using POST, bad accept type, etc), so I'm not bothering with that. The friendly errors purposely use `self.baseurl` instead of the full Batch API endpoint because I'd expect some copy/paste/modify on the part of the user here, and it would be more confusing if '/objects/batch' magically appeared, but shouldn't be used in the config setting. It still seems like the right thing for debugging in the catchall case.

lfs: provide more Batch API error info via a hint in the raised exception A coworker had a typo in `lfs.url`, forgot it was even set because usually the blob server is inferred, and then got a 404. It would have been easier to debug with the failing URL printed.

scmutil: display the optional hint when handling StorageError in catchall() Other than CensoredNodeError (which is also a StorageError), it looks like all exceptions with a hint display them. I'm not sure that it makes sense to have a hint for censored nodes, so I'm not bothering with that. It looks like nobody is using this yet, as the tests don't change.

sparse-revlog: align endrevidx usages in the _slicechunktosize All "startrevidx..endrevidx" ranges in this function are now half-open.

sparse-revlog: use `span` variable as intended The variable was planned to be used in the while condition but was not used yet.

tests: stabilize test-commandserver.t on Windows It looks like new test coverage in 054d0fcba2c4, rather than a code change.

histedit: conditionalize the imports of 'fcntl' and 'termios' The recent import of chistedit in c36175456350 made Windows sad. I'm not sure if there's other stuff that needs to be done here (e.g. change the default interface), but this makes the tests run again. It would have been nicer if the error message indicated these modules were the problem, but instead it said "*** failed to import extension histedit: No module named histedit". I'm not sure if there's anything we can do about that.

logtoprocess: update commandfinish options arguments d2c997b8001f changed the logtoprocess API with the effect of not exposing the positional arguments to the logtoprocess scripts anymore. We have some scripts that use the duration and return code of the "commandfinish" event to monitor hg calls. Update the logging of the "commandfinish" to expose those values as options argument, which will be accessible as `OPT_RETURN_CODE` and `OPT_DURATION` in logtoprocess arguments. The code has been formatted with Black. Differential Revision: https://phab.mercurial-scm.org/D5282

rebase: fix two ui.logs to actually have text when using default blackbox log Some implementations of ui.log record structured information along with the ui.log which can be used for metrics, but ui.log() as implemented by the blackbox logging does not do anything special with this, and we end up with a log line with no text (not even a line break) so it ends up looking something like: date time user @node (pid) [rebase]> date time user @node (pid) ... Differential Revision: https://phab.mercurial-scm.org/D5279

wireprotov2server: let repo.narrowmatch(match) do matcher intersection This is supported since 4fd0fac48922 (localrepo: allow narrowmatch() to accept matcher to intersect with, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D5281

blackbox: extract function to test if log event is tracked This will be a required method of the logger interface.

blackbox: initialize inlog flag properly And ditch the "bb" prefix as it's no longer a ui extension class.

blackbox: initialize repo attribute properly And ditch the "bb" prefix as it's no longer a ui extension class.

blackbox: unindent "if True" block

blackbox: extract logger class from ui wrapper This moves most functions to new blackboxlogger class. The ui wrapper will be removed later.

blackbox: rename variables to prepare extracting core logic from ui wrapper I'm going to add ui.setlogger() function so that I can enable logging feature in command server without extending ui.__class__. This prepares for it. "self" will be a logger instance, so this patch renames some of them to "ui".

sparse-revlog: rework the way we enforce chunk size limit We move from a O(N) algorithm to a O(log(N)) algorithm. The previous algorithm was traversing the whole delta chain, looking for the exact point where it became too big. This would result in most of the delta chain to be traversed. Instead, we now use a "binary" approach, slicing the chain in two until we have a chunk of the appropriate size. We still keep the previous algorithm for the snapshots part. There are few of them and they are large bits of data distant from each other. So the previous algorithm should work well in that case. To take a practical example of restoring manifest revision '59547c40bc4c' for a reference NetBeans repository (using sparse-revlog). The media time of the step `slice-sparse-chain` of `perfrevlogrevision` improve from 1.109 ms to 0.660 ms.

doctest: add a `issnapshot` method to _testrevlog We'll need it soon.

tests: add `revlogutils.deltas` module to doctests The doctest in these module have been from `mercurial.revlog` but the module was not added to the doctests. Spotted by Yuya Nishihara.

merge with stable

mergetools: adjust Beyond Compare config on Mac/Linux Set the labels of the Linux and Mac versions of Beyond Compare from Mercurial's builtin variables, same as the Windows version. Differential Revision: https://phab.mercurial-scm.org/D5255

rewriteutil: move publicrevs closer to where it's used

requires: use atomictemp=True when writing .hg/requires We use an unusual file system at Google that allows writes (and renames) but not deletions (for certain paths). That causes problems when writing the requires files without atomictemp=True. There doesn't appear to be any real drawbacks to using atomictemp, so I'm hoping we can just change it in core. Differential Revision: https://phab.mercurial-scm.org/D5274

blackbox: extract _log() function which is called after lastui is resolved This makes sure that self is the solo ui instance used in _log().

blackbox: inline temporary variables which are referenced only once

blackbox: simply update global lastui variable at once

blackbox: consolidate conditions for early return Just pick the lastui only if it is usable.

blackbox: remove redundant check for unassigned repo Since ui._bbvfs is looked through ui._bbrepo, the repo instance should exist if ui._bbvfs isn't None.

tests: fix bytes/str issue I introduced when adding this test # skip-blame just b prefixes for py3 Differential Revision: https://phab.mercurial-scm.org/D5271

shelve: use matcher to restrict prefetch to just the modified files Shelve currently operates by: - make a temp commit - identify all the bases necessary to shelve, put them in the bundle - use exportfile to export the temp commit to the bundle ('file' here means "export to this fd", not "export this file") - remove the temp commit exportfile calls prefetchfiles, and prefetchfiles uses a matcher to restrict what files it's going to prefetch; if it's not provided, it's alwaysmatcher. This means that `hg shelve` in a remotefilelog repo can possibly download the file contents of everything in the repository, even when it doesn't need to. It luckily is restricted to the narrowspec (if there is one), but this is still a lot of downloading that's just unnecessary, especially if there's a "smart" VCS-aware filesystem involved. exportfile is called with exactly one revision to emit, so we're just restricting it to prefetching the files from that revision. The base revisions having separate files should not be a concern since they're handled already; example: commit 10 is draft and modifies foo/a.txt and foo/b.txt commit 11 is draft and modifies foo/a.txt my working directory that I'm shelving modifies foo/b.txt By the time we get to exportfile, commit 10 and 11 are already handled, so the matcher only specifying foo/b.txt does not cause any problems. I verified this by doing an `hg unbundle` on the bundle that shelve produces, and getting the full contents of those commits back out, instead of just the files that were modified in the shelve. Differential Revision: https://phab.mercurial-scm.org/D5268

revlog: automatically read from opened file handles The revlog reading code commonly opens a new file handle for reading on demand. There is support for passing a file handle to revlog.revision(). But it is marked as an internal argument. When revlogs are written, we write() data as it is available. But we don't flush() data until all revisions are written. Putting these two traits together, it is possible for an in-process revlog reader during active writes to trigger the opening of a new file handle on a file with unflushed writes. The reader won't have access to all "available" revlog data (as it hasn't been flushed). And with the introduction of the previous patch, this can lead to the revlog raising an error due to a partial read. I witnessed this behavior when applying changegroup data (via `hg pull`) before issue6006 was fixed via different means. Having this and the previous patch in play would have helped cause errors earlier rather than manifesting as hash verification failures. While this has been a long-standing issue, I believe the relatively new delta computation code has tickled it into being more common. This is because the new delta computation code will compute deltas in more scenarios. This can lead to revlog reading. While the delta computation code is probably supposed to reuse file handles, it appears it isn't doing so in all circumstances. But the issue runs deeper than that. Theoretically, any code can access revision data during revlog writes. It appears we were just getting lucky that it wasn't. (The "add revision callback" passed to addgroup() provides an avenue to do this.) If I changed the revlog's behavior to not cache the full revision text or to clear caches after revision insertion during addgroup(), I was able to produce crashes 100% of the time when writing changelog revisions. This is because changelog's add revision callback attempts to resolve the revision data to access the changed files list. And without the revision's fulltext being cached, we performed a revlog read, which required opening a new file handle. This attempted to read unflushed data, leading to a partial read and a crash. This commit teaches the revlog to store the file handles used for writing multiple revisions during addgroup(). It also teaches the code for resolving a file handle when reading to use these handles, if available. This ensures that *any* reads (regardless of their source) use the active writing file handles, if available. These file handles have access to the unflushed data because they wrote it. This allows reads to complete without issue. Differential Revision: https://phab.mercurial-scm.org/D5267

revlog: detect incomplete revlog reads _readsegment() is supposed to return N bytes of revlog revision data starting at a file offset. Surprisingly, its behavior before this patch never verified that it actually read and returned N bytes! Instead, it would perform the read(), then return whatever data was available. And even more surprisingly, nothing in the call chain appears to have been validating that it received all the data it was expecting. This behavior could lead to partial or incomplete revision chunks being operated on. This could result in e.g. cached deltas being applied against incomplete base revisions. The delta application process would happily perform this operation. Only hash verification would detect the corruption and save us. This commit changes the behavior of raw revlog reading to validate that we actually read() the number of bytes that were requested. We will raise a more specific error faster, rather than possibly have it go undetected or manifest later in the call stack, at delta application or hash verification. Differential Revision: https://phab.mercurial-scm.org/D5266

revlog: use single file handle when de-inlining revlog _getsegmentforrevs() will eventually call into _datareadfp() to resolve a file handle to read revision data. If no file handle is passed into _getsegmentforrevs(), it opens a new one. Explicit is better than implicit. This commit changes _enforceinlinesize() to open a file handle explicitly when converting inline revlogs to split revlogs and to pass this file handle into _getsegmentforrevs(). I haven't measured, but this change should improve performance, as we no longer reopen the revlog for reading for every revision in the revlog when it is converted from inline to split. Instead, we open it at most once and use it for the duration of the operation. That being said, I /think/ the chunk cache may mitigate the number of file opens required. Differential Revision: https://phab.mercurial-scm.org/D5265

store: raise ProgrammingError if unable to decode a storage path Right now, the function magically return False which is dangerous, so let's raise ProgrammingError. Suggested by Augie in D5139. Differential Revision: https://phab.mercurial-scm.org/D5264

tests: document a known failing interaction between narrow and lfs This is one of the two remaining aborts I found looking into issue5794. I've got no idea what's wrong with the hook, since the changes there fixed the other two problems noted in that bug report. It seems like it might go away when the narrow issue is fixed, but let's make sure this doesn't get lost. The stacktrace for the hook seems to indicate that the missing file *is* in ctx: remote: Traceback (most recent call last): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in checkrequireslfs remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr> remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 191, in filectxislfs remote: return _islfs(self.filelog(), self.filenode()) remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode remote: return self._filenode remote: File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__ remote: result = self.func(obj) remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode remote: return self._filelog.lookup(self._fileid) remote: File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 68, in lookup remote: self._revlog.indexfile) remote: File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 218, in fileidlookup remote: raise error.LookupError(fileid, identifier, _('no match found')) remote: LookupError: data/inside2/f.i@f59b4e021835: no match found

logtoprocess: drop support for ui.log() call with invalid msg arguments (BC) Before, the logtoprocess extension put a formatted message into $MSG1, and its arguments to $MSG2... If the specified arguments couldn't be formatted because of a caller bug, an unformatted message was passed in to $MSG1 instead of exploding. This behavior doesn't make sense. Since I'm planning to formalize the ui.log() interface such that we'll no longer have to extend the ui class, I want to remove any features not conforming to the ui.log() API. So this patch removes the support for ill-formed arguments, and $MSG{n} (where n > 1) parameters which seems useless as long as the message can be formatted. The $MSG1 variable isn't renamed for the maximum compatibility. In future patches, a formatted msg will be passed to a processlogger object, instead of overriding the ui.log() function. .. bc:: The logtoprocess extension no longer supports invalid ``ui.log()`` arguments. A log message is always formatted and passed in to the ``$MSG1`` environment variable.

py3: byte-stringify inline extension in test-logtoprocess.t

logtoprocess: rewrite dict building in py3-compatible way

logtoprocess: leverage procutil.shellenviron() to stringify variables (BC) This should make the extension more Py3 friendly. The environment variables of the main process are copied to the dict by shellenviron(). .. bc:: Boolean options passed to the logtoprocess extension are now formatted as ``0`` or ``1`` instead of ``None``, ``False``, or ``True``.

templater: check invalid use of list expression properly (issue5920) The error message is still cryptic, but it should be better.

py3: add b suffix to make sure file is opened in bytes mode Differential Revision: https://phab.mercurial-scm.org/D5263

py3: don't use dict.iterkeys() dict.iterkeys() is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5262

py3: use node.hex(h.digest()) instead of h.hexdigest() hashlib.sha1.hexdigest() returns str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5261

py3: make second argument of fdopen() a str # skip-blame as just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5260

py3: fix keyword arguments handling in hgext/remotefilelog/ Keys of kwargs on Python 3 should be strings. This patch fixes them by appending r'' prefixes, and using pycompat.byteskwargs() and pycompat.strkwargs(). Differential Revision: https://phab.mercurial-scm.org/D5259

py3: add __pycache__ to .hgignore Something changed with Python 3 just like million on things. Maybe they should named Python 3 as anaconda or cobra instead of just increasing the version number. This makes test-custom-filters.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5039

sparse-revlog: stop using a heap to track selected gap Same logic as for 'gapsheap', we don't actually need a heap.

sparse-revlog: stop using a heap to track gaps The heap doesn't bring any performance advantage as we can simply sort the final list. Moreover, the lesser complexity helps a lot when we later implement it in C.

sparse-revlog: fast-path before computing payload size In this fast-path case, we do not need to compute the full delta chain payload. Since it comes with a significant cost, we avoid doing so if possible.

sparse-revlog: drop unused deltainfo parameter from segmentspan We no longer need to deal with slicing logic including uncommitted revision. We drop the associated code.

sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity We no longer need to deal with slicing logic including uncommitted revision. We drop the associated code.

sparse-revlog: skip the span check in the sparse-revlog case This significantly improves the performance on unbundling on smaller repositories. Mercurial: unbundling 1K revisions no-sparse-revlog: 500 ms sparse-revlog-before: 689 ms sparse-revlog-after: 484 ms Pypy: unbundling 1K revisions no-sparse-revlog: 1.242 s sparse-revlog-before: 1.135 s sparse-revlog-after: 0.860 s NetBeans: unbundling 1K revisions no-sparse-revlog: 1.386 s sparse-revlog-before: 2.368 s sparse-revlog-after: 1.191 s Mozilla: unbundling 1K revisions no-sparse-revlog: 3.103 s sparse-revlog-before: 3.367 s sparse-revlog-after: 3.093 s

histedit: import chistedit curses UI from hg-experimental I don't tend to like curses interfaces, but this gets enough use at work that it seems like it's worth bringing into core. This is a minimal import from hg-experimental revision 4c7f33bf5f00, in that I've done the smallest amount of code movement and editing in order to import the functionality. .. feature:: `hg histedit` will now present a curses UI if curses is available and `ui.interface` or `ui.interface.histedit` is set to `curses`. Differential Revision: https://phab.mercurial-scm.org/D5146

manifest: perform cheap checks before potentially allocating memory Differential Revision: https://phab.mercurial-scm.org/D5258

manifest: also reject obviously-too-short lines when parsing lines Differential Revision: https://phab.mercurial-scm.org/D5257

manifest: make sure there's a filename before bothering to look for newline There's no valid manifest that would have no characters before the NUL byte on a line, and this fixes some erratic timeouts in the fuzzer. Differential Revision: https://phab.mercurial-scm.org/D5256

revlog: replace PyInt_AS_LONG with a more portable helper function PyInt_AS_LONG disappears on Python, and our previous #define was producing some problems on Python 3. Let's give up and make an inline helper function that makes this more sane. Differential Revision: https://phab.mercurial-scm.org/D5235

help: document weird behavior of uisetup() While refactoring ui.log() API, it was really annoying that uisetup() is actually an extsetup() the phase 1. Let's document that. I'm planning to add another uisetup() which is called per ui instance, though I can't think of a good name for it.

help: merge section about uisetup() and extsetup() They are technically the same callback, called only once per process. The section name "ui setup" is confusing, so shouldn't be used.

help: add internals.extensions topic I think it's better to include the API overview in core as we now have the internals section in our help system. Retrieved from the wiki, and formatted as reST. Several wiki links are removed since they are invalid in the help. The sections about example extension are removed at all as they seemed too verbose. https://www.mercurial-scm.org/wiki/WritingExtensions

commandserver: send raw progress information to message channel This is pretty basic implementation to support GUI progress bar.

commandserver: make getpass() request distinct from normal prompt Otherwise, GUI clients would have to parse the prompt text.

commandserver: attach prompt default and choices to message These attributes are important to provide a GUI prompt to user.

test-commandserver: clean up quoting and location of dbgui extension This helps embedding '$' in the script.

ui: extract helpers to write message with type or label This provides a 'type' attribute to command-server clients, which seems more solid than relying on 'ui.<type>' labels. In future patches, type='progress' will be added to send raw progress information.

commandserver: add experimental option to use separate message channel This is loosely based on the idea of the TortoiseHg's pipeui extension, which attaches ui.label to message text so the command-server client can capture prompt text, for example. https://bitbucket.org/tortoisehg/thg/src/4.7.2/tortoisehg/util/pipeui.py I was thinking that this functionality could be generalized to templating, but changed mind as doing template stuff would be unnecessarily complex. It's merely a status message, a simple serialization option should suffice. Since this slightly changes the command-server protocol, it's gated by a config knob. If the config is enabled, and if it's supported by the server, "message-encoding: <name>" is advertised so the client can stop parsing 'o'/'e' channel data and read encoded messages from the 'm' channel. As we might add new message encodings in future releases, client can specify a list of encoding names in preferred order. This patch includes 'cbor' encoding as example. Perhaps, 'json' should be supported as well.

commandserver: fix reference before assignment error in pipeservice cleanup Spotted by the next patch.

dispatch: pass around ui.fmsg channel This will be set by the command server. See the next patch.

test-subrepo: document why share magically works even if subrepos are missing I was confused how it's working while reviewing fb490d798be0, "share: reload repo after adjusting it in postshare()."

packaging: add Fedora 29 target Differential Revision: https://phab.mercurial-scm.org/D5254

store: pass in decoded filename to narrow matcher

test-narrow-clone-stream: include no usefncache case which is broken Since 9aeb9e2d28a7, encoded filenames are filtered by the narrow matcher, which is clearly wrong.

test-narrow-clone-stream: include uppercase letter in filename This makes encoded filenames differ from the original names.

phase: use the `phases.cmdphasenames` constant to walk available command flags This reverts 1ea6772fb415.

phases: add a list of names usable by the phase command This is useful in TortoiseHg for example, which has until recently just added all names in the `phasenames` list to the context menu that changes the phase. The ones not supported by the command would just error out.

transaction: display data about why the transaction failed to rollback We saw more of these a while back. Having more data available would be nice.

merge with stable

remotefilelog: avoid accessing repo instance after dispatch Upstream commit c5e6c1ba1c79 (hg: don't reuse repo instance after unshare(), 2018-09-12) poisoned the repo instance after unshare(). That made `hg unshare` fail with remotefilelog because we tried to close the fileserverclient after dispatch by accessing it via the repo. This patch fixes that by storing the reference to the fileserverclient at the beginning of dispatch. An analogous patch was sent for remotefilelog version in FB's hg-experimental as D5246. Differential Revision: https://phab.mercurial-scm.org/D5253

share: avoid a now-unnecessary reload of the repo hg.share() now returns an already-reloaded repo, so let's just use that. (This would have failed test-subrepo-recursion.t without the previous patch.) Differential Revision: https://phab.mercurial-scm.org/D5252

share: reload repo after adjusting it in postshare() When sharing a repo that's using remotefilelog, the update that happens at the end of the `hg share` call does not see the remote repo path that's copied in hg.postshare(). This patch reloads the repo after hg.postshare() to address that. This changes a subrepo test case. Note that `hg share -U; hg co tip` worked there before, so I don't see see why `hg share` should fail. I also don't know what a "locally referenced subrepo". So maybe this is fixing a bug? Hopefully it's not breaking something someone actually cares about at least. Maybe someone who knows and cares about subrepos can review this. Differential Revision: https://phab.mercurial-scm.org/D5251

tests: don't load strip extension via mq extension for remotefilelog tests The proper way to get the `hg strip` command has been via the "strip" extension since 2013. Differential Revision: https://phab.mercurial-scm.org/D5250

remotefilelog: remove unused configs for {data,history}pack version Differential Revision: https://phab.mercurial-scm.org/D5249

tests: remove obsolete PYTHONPATH override from remotefilelog tests They were added in https://bitbucket.org/facebook/hg-experimental/commits/fdcad37a6a68cfbaf5920f3eeaa0cc9cae42bd26, which says this: When remotefilelog moved from its own repo, the tests needed to be updated to adjust the PYTHONPATH to ensure the in-repo remotefilelog was loaded instead of the system one. This meant any local runs of remotefilelog tests would've been using the system remotefilelog unless the user had manually set the PYTHONPATH themselves. That doesn't seem relevant with remotefilelog in core. Differential Revision: https://phab.mercurial-scm.org/D5248

tests: drop obsolete "hginit" alias in remotefilelog tests Differential Revision: https://phab.mercurial-scm.org/D5247

fix: add extra field to fixed revisions to avoid creating obsolescence cycles The extra field prevents sequential invocations of fix from producing the same hash twice. Previously, this could cause problems because it would create an obsolescence cycle instead of the expected new successor. This change also adds an explicit check for whether a new revision should be committed. Until now, the code relied on memctx.commit() to quietly do nothing if the node already exists. Because of the new extra field, this no longer covers the case where we don't want to replace an unchanged node. Differential Revision: https://phab.mercurial-scm.org/D5245

cleanup: use revision numbers instead of hashes in test output This allows changes to the hashes produced by fix to not needlessly modify this area of the test. Differential Revision: https://phab.mercurial-scm.org/D5244

tests: make the commands.resolve.confirm tests not pollute the test-wide hgrc I want to add tests after these tests that create a new repo and use it, and was confused for a bit as to why they were seeing different behavior than I expected. Differential Revision: https://phab.mercurial-scm.org/D5242

tests: fix a couple typos in test-resolve.t comments and add a comment Differential Revision: https://phab.mercurial-scm.org/D5241

fix: add suboption for configuring execution order of tools This could be accomplished by using wrapper scripts, but that would diminish the usefulness of the incremental formatting logic. Configuring execution order along with other things in the hgrc is probably more convenient anyway. This change highlights some awkwardness with suboptions and default values, which should be addressed separately. Differential Revision: https://phab.mercurial-scm.org/D5237

tests: fix up some import statements caught by Python 3 I'm curious how the import checker manages to be so much more pedantic in Python 3, but not enough to bother exploring. Differential Revision: https://phab.mercurial-scm.org/D5240

tests: replace `tac` reimplementation by `sort -r` `sort -r` is better code "formatter" than `tac` since it's stable. It's also portable so we don't need to reimplement it in Python. Differential Revision: https://phab.mercurial-scm.org/D5239

tests: work around `tac` not being portable Introduce a tac.py helper and use it. Sigh. Differential Revision: https://phab.mercurial-scm.org/D5238

revlog: give formatting to clang-format This has two lines that are poorly formatted, both right after PyObject_HEAD macros. It's possible that in the future there will be a feature to tell clang-format that a token includes an end-of-line character (in this case the ;), but for now I'm willing to live with two poorly-formatted lines in a struct definition in exchange for not thinking about how this file is formatted. Differential Revision: https://phab.mercurial-scm.org/D5234

revlog: add blank line in comment to help clang-format Differential Revision: https://phab.mercurial-scm.org/D5233

revlog: add a comment to help clang-format produce less-awful results Differential Revision: https://phab.mercurial-scm.org/D5232

py3: ratchet caught two more passing tests Differential Revision: https://phab.mercurial-scm.org/D5236

perf: add a lazydeltabase option to perfrevlogwrite The option controls if we'll test the provided delta first. This is the current default behavior for unbundle.

perf: add `storage` as possible source for perfrevlogwrite This source will use the stored delta.

perf: add `parent-smallest` as possible source for perfrevlogwrite This source will use the smallest of the possible diff against parent.

perf: add `parent-2` as possible source for perfrevlogwrite This source will use a diff against p2 if it exists and fall back to p1 otherwise.

perf: add `parent-1` as possible source for perfrevlogwrite This source will use a diff against p1 in all case.

perf: add the notion of "source" to perfrevlogwrite We want to test performance associated witch various way to add a new revision. They will be specified using this new argument.

perf: only display the total time for perfrevlogwrite if quiet This provide a simple way to get an overview of the total performance.

perf: offer full details in perfrevlogwrite This will be useful for people who want to study the timing pattern more closely.

perf: introduce a perfrevlogwrite command The command record times taken by adding many revisions to a revlog. Timing each addition, individually. The "added revision" are recreations of the original ones. To time each addition individually, we have to handle the timing and the reporting ourselves. This command is introduced to track the impact of sparse-revlog format on delta computations at initial storage time. It starts with the full text, a situation similar to the "commit". Additions from an existing delta are better timed with bundles. The complaints from `check-perf-code.py` are not relevant. We are accessing and "revlog" opener, not a repository opener.

tests: fix config knob in test-narrow-clone-stream.t Two patches landed in parallel and had a semantic conflict. This resolves the mess and leaves us with passing tests. Differential Revision: https://phab.mercurial-scm.org/D5231

remotefilelog: fix various whitespace issues in docstring Differential Revision: https://phab.mercurial-scm.org/D5230

ui: add config knob to redirect status messages to stderr (API) This option can be used to isolate structured output from status messages. For now, "stdio" (stdout/err pair) and "stderr" are supported. In future patches, I'll add the "channel" option which will send status messages to a separate command-server channel with some metadata attached, maybe in CBOR encoding. This is a part of the generic templating plan: https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Sanity_check_output .. api:: Status messages may be sent to a dedicated stream depending on configuration. Don't use ``ui.status()``, etc. as a shorthand for conditional writes. Use ``ui.write()`` for data output.

ui: hide fin/fout/ferr attributes behind @property functions This allows keeping references to fout/ferr/fin which are updated when these properties are changed. See the next patch.

ui: label prompt and echo messages I'm going to add a dedicated command-server channel for status messages, which carries metadata alongside a message text. 'ui.*' label provides a hint how message text should be processed.

ui: add inner function to select write destination I'm going to add a config knob to redirect any status messages to stderr. This function helps to switch underlying file objects. # no-check-commit because of existing write_err() function

ui: remove _write() and _write_err() functions

ui: move pre/post processes from low-level write()s to _writenobuf() This helps adding a dedicated stream for status/error messages. I don't want to add _write*() function per stream.

ui: pass in file object to _writenobuf() See the subsequent patches for why. The "if" block in _writenobuf() will be removed soon.

perf: fix perfrevlogrevisions --reverse Currently, 'endrev' equals `len(revlog)`, a revision that does not exist. When asking for the reverse order, the arguments passed to xrange are `xrange(len(revlog), startrev)` which then crash. We need to offset 'endrev' by one so we don't crash anymore. Also, we offset 'startrev' to ensure we get the same number of revisions with and without the `--reverse` option. Differential Revision: https://phab.mercurial-scm.org/D5228

procutil: import concerns about creationflags on Windows from D1701 I don't have the need anymore for the change in D1701 nor the time to investigate the changes on all supported Windows platforms. I import the stuff I learned on D1701 in the `runbgcommand` so the next people working on it can starts from there. Differential Revision: https://phab.mercurial-scm.org/D5229

localrepo: extract loading of hgrc files to standalone function Various 3rd party extensions supplement where per-repo config data lives. Looking at their sources, they resort to unorthodox means to inject the config data. And the way they do it is susceptible to corner cases. e.g. not processing automatic extension loads, not reacting to new or disabled extensions in configs, etc. This commit extracts the core logic of loading hgrc files into a standalone function so there is a clear function that can be monkeypatched to inject per-repo config data at repository open time. Differential Revision: https://phab.mercurial-scm.org/D5221

revsets: make bookmark/named('re:nonexistent') not abort (issue6018) (BC) Foozy documented the differences between revsets branch(), tag(), bookmark(), and named() in eeb5d5ab14a6 (revset: raise RepoLookupError to make present() predicate continue the query, 2015-01-31). He seemed to want tag() to change behavior to not error out on non-matching regular expressions. I think it's instead bookmark() and named() that should not error out. So that's what this patch does. Differential Revision: https://phab.mercurial-scm.org/D5220

fix: rename :fileset subconfig to :pattern This name was always inaccurate, since the config accepts any pattern. Hopefully so few people use this right now that it won't matter, but there will now be a warning if the old config name is used. Differential Revision: https://phab.mercurial-scm.org/D5226

fix: add a config to abort when a fixer tool fails This allows users to stop and address tool failures before proceeding, instead of the default behavior of continuing to apply any tools that didn't fail. For example, a code formatting tool could fail if you have syntax errors, and you might want your repo to stay in its current state while you fix the syntax error before re-running 'hg fix'. It's conceivable that this would even be necessary for the correctness of some fixer tools across a chain of revisions. Differential Revision: https://phab.mercurial-scm.org/D5200

perf: measure slicing time in perfrevlogrevision Slicing a sparse delta chain can be expensive. We now benchmark the associated time.

perf: teach perfrevlogrevision about sparse reading Before this change, chunks were always read in a single block. Even in the sparse-read/sparse-revlog case. This gave a false view of the performance and could lead to memory consumption issue.

perf: use the same timer for all section of perfrevlogrevision Otherwise the -T json output is invalid.

obsutil: clarify the access to "repo" We use the variable multiple times and we might use it even more in the future. We use a temporary variable instead.

pycompat: adding Linux detection and fixing Mac Python 3 recommends detecting OSs with the prefix of the platform, but we were comparing the full string for macOS. We also didn't have Linux detection, which is convenient for extensions to use (rather than have some OSs detected by hg and some by the extension). Reference: https://docs.python.org/3/library/sys.html#sys.platform Differential Revision: https://phab.mercurial-scm.org/D5227

remotefilelog: fix various minor py3 problems # skip-blame b prefixes and pycompat.long, nothing remotely interesting Differential Revision: https://phab.mercurial-scm.org/D5223

remotefilelog: rip out a missed mention of lz4 Differential Revision: https://phab.mercurial-scm.org/D5222

ui: wrap whole _write() block with timeblockedsection I think the cost of color labeling is negligible compared to the I/O syscalls. Let's simply wrap the whole write() function so that we can eliminate _write() and _write_err() in later changeset.

ui: indent _writenobuf() to prepare moving bits from _write() functions

ui: simply concatenate messages before applying color labels This should be cheaper in space than applying labels for each message.

ui: simplify interface of low-level write() functions _write() and _write_err() will be replaced with fout.write() and ferr.write() respectively. This is the first step.

ui: factor out function that writes data to fout/ferr with labeling I'm thinking of adding an option to send status messages to stderr (or a dedicated command-server channel) so that structured output (e.g. JSON) would never be interleaved with non-formatter output. A unified write() interface helps to do that.

ui: consolidate places where _progclear() is called The progress bar has to be cleared when we start writing some data to the output stream. Let's make it always triggered immediately before switching by _colormode, so that we can easily factor out helper functions.

templater: compute revset lazily This speeds up e.g. "{ifcontains(rev, revset('::.'), ...)}" in common cases where 'rev' is near the working parent. The templater API is ugly, but it helps here. 'f' can be either a generator or a function returning a generator.

remotefilelog: drop compat code for "getbundle_shallow" wireprotocol command Doing some annotate on hgexperimental shows that getbundle_shallow used to exist in 2013 or before. We don't have any pre-2013 remotefilelog users except Fb themselves and I doubt they are going to use in-core remotefilelog. So it's safe to remove this. Differential Revision: https://phab.mercurial-scm.org/D5193

tests: remove lz4 as dependency while running tests One of the previous patch authored by Augie rips out the lz4 dependency and things should work without it. Now there are just 2-3 tests failing because of same change in emitrevisions() API. Differential Revision: https://phab.mercurial-scm.org/D5192

remotefilelogserver: add a matcher argument to _walkstreamfiles() Implementing narrow stream clones in core, I added an optional matcher argument. The function in remotefilelogserver.py does not know about that argument and does not accept that and hence some tests fails. Differential Revision: https://phab.mercurial-scm.org/D5191

shallowutil: introduce a helper function isenabled() This patch introduces a inenabled() function which will check whether remotefilelog is enabled or not. The function is then also used at all the places where check whether remotefilelog is enabled or not. The new function makes code easy to read without need to understand what is the constant involved and why we are checking repo.requirements. Differential Revision: https://phab.mercurial-scm.org/D5190

remotefilelog: remove some BC code related to streamclone Since remotefilelog is now moved to core and we have streamclone._walkstreamfiles() in core, we don't need to have the backward compatibility code. People with old mercurial version should use remotefilelog from hg-experimental repo as IMO remotefilelog will go under a good refactoring and old clients will break. # no-check-commit foo_bar function name Differential Revision: https://phab.mercurial-scm.org/D5189

remotefilelog: add some docstring Differential Revision: https://phab.mercurial-scm.org/D5131

remotefilelog: rename wireproto methods and remaining capabilities This is a mess, in part because there should be more constants throughout. I know we typically do exp- instead of the x_ business in this change, but I also had to use this in some function names, so I figured until I can break that coupling I'd go with this. If it's too unpleasant during review, let me know and I can probably clean it up some more. # no-check-commit due to new foo_bar naming - too hard to avoid right now :( Differential Revision: https://phab.mercurial-scm.org/D5129

remotefilelog: consolidate and rename on-disk store requirement The value of this constant appeared in too many places. While we're here, rename it to be more consistent with our naming conventions for experimental functionality. Differential Revision: https://phab.mercurial-scm.org/D5128

remotefilelog: consolidate and rename bundle2 capability Bonus: we catch a spot where the shallowrepo requirement was being misused as the bundle2 capability. Differential Revision: https://phab.mercurial-scm.org/D5127

remotefilelog: rename capability for legacy ssh file fetching method I think I want to delete this, but for now we'll just rename this so it's easy to tell apart from other strings that have the same value. Differential Revision: https://phab.mercurial-scm.org/D5126

remotefilelog: rip out lz4 support Some methods are left teased out so it's easier to build a migration extension from v1 packs to v2. I also anticipate those methods will be of use if we get around to adding configurable compression engines before we end up jettisoning packs. I think it would make sense to register one-shot `compress` and `decompress` methods on our compression engines in util.py, but indygreg mentioned that might not be straightforward. Differential Revision: https://phab.mercurial-scm.org/D4944

remotefilelog: bump pack file formats to version 2 Drop support for format 0 and 1 packs. Version 2 will be different in that it'll use zlib compression instead of lz4, as the latter isn't really suitable for core. If I can make it work, I'll make compression engines pluggable, but the logic for handling these files is spread out enough it's a little tricky to get that right. Test changes are only due to pack files being renamed. The contents of the packs are the same. Differential Revision: https://phab.mercurial-scm.org/D4943

remotefilelog: avoid hard-coding pack names in tests I'm about to alter the pack format, which alters the content-addressed name of the pack file. In order to reduce my own confusion I found it helpful to use an ls | head pipeline since we always want to inspect the newest pack. Differential Revision: https://phab.mercurial-scm.org/D4942

remotefilelog: remove now-unused wirepack code Differential Revision: https://phab.mercurial-scm.org/D5124

remotefilelogserver: remove pack-serving functionality Differential Revision: https://phab.mercurial-scm.org/D5123

remotefilelog: remove support for fetching packs We've never used this at Google, and it's strictly an optimization that indygreg and I are confident we can handle more elegantly later. Let's rip it out to keep the surface area of the imported remotefilelog small. Some tests have changes because the tests for regular packs and repacking were mixed with tests of fetching packs. I'm pretty sure I've done the right thing here. Differential Revision: https://phab.mercurial-scm.org/D5122

remotefilelog: jettison example cache client We can always reconstruct it from history if it becomes interesting. Differential Revision: https://phab.mercurial-scm.org/D4941

remotefilelog: prune obsolete method Differential Revision: https://phab.mercurial-scm.org/D4940

logtoprocess: use new runbgcommand from procutil Differential Revision: https://phab.mercurial-scm.org/D4939

procutil: port over windows encoding fixes from logtoprocess Differential Revision: https://phab.mercurial-scm.org/D5121

remotefilelog: transplant runbgcommand to procutil While cleaning up the deprecated runshellcommand I noticed a near-clone of this in logtoprocess, so I'm standardizing on what appears to be the newer one by moving it to procutil. Differential Revision: https://phab.mercurial-scm.org/D4938

remotefilelog: remove function that was described as deprecated Differential Revision: https://phab.mercurial-scm.org/D4937

remotefilelog: import pruned-down remotefilelog extension from hg-experimental This is remotefilelog as of my recent patches for compatibility with current tip of hg, minus support for old versions of Mercurial and some FB-specific features like their treemanifest extension and fetching linkrev data from a patched phabricator. The file extutil.py moved from hgext3rd to remotefilelog. This is not yet ready to be landed, consider it a preview for now. Planned changes include: * replace lz4 with zstd * rename some capabilities, requirements and wireproto commands to mark them as experimental * consolidate bits of shallowutil with related functions (eg readfile) I'm certainly open to other (small) changes, but my rough mission is to land this largely as-is so we can use it as a model of the functionality we need going forward for lazy-fetching of file contents from a server. # no-check-commit because of a few foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D4782

store: introduce _matchtrackedpath() and use it to filter store files This patch introduces a function to filter store files on the basis of the path which they are tracking. The function assumes that the entries can be of two types, 'meta/*' and 'data/*' which means it will just work on revlog based storage and not with another storage ways. For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the beginning and the end then pass that to matcher. For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from beginning and end then call matcher.visitdir() with it to make sure all the parent directories are also downloaded. Since the storage filtering for narrow stream clones is implemented with this patch, we remove the un-implemented error message, add some more tests and add the treemanifest case to tests too. The tests demonstrate that it works correctly. After this patch, we have now narrow stream clones working. Narrow stream clones are a very important feature for large repositories who have good internet connection because they use streamclones for cloning and if they do normal narrow clone, that takes more time then a full streamclone. Also narrow-stream clone will drastically speed up clone timings. Differential Revision: https://phab.mercurial-scm.org/D5139

narrow: only send includepats and excludepats if they are not empty If we send an empty includepats or excludepats argument to getbundle, it's translated to `['']` on the server which causes problems because even though it's empty, bool of that value if True and we end up creating differencematcher with narrowspec.match() which results in unexpected behavior. Differential Revision: https://phab.mercurial-scm.org/D5138

exchange: pass includepats and excludepats as arguments to getbundle() This will help in implementing narrow stream clones. Also narrow extension used to add these arguments, now we add them by default if they are not empty. Since reading includepats and excludepats on the server only works when narrow is enabled, we check if narrow if enabled or not before passing them. Differential Revision: https://phab.mercurial-scm.org/D5119

catapult: add a bit more documentation on how to use catapult tracing Differential Revision: https://phab.mercurial-scm.org/D5217

catapult: introduce HGTESTCATAPULTSERVERPIPE to control run-tests' tracing If unset, it will inherit its value from HGCATAPULTSERVERPIPE (which hg itself also respects). By setting only HGTESTCATAPULTSERVERPIPE, we can get per-command breakdowns of the test runtimes for the whole test suite without overloading the trace file with the contents of the tracing from hg (such as demandimport, etc.) Differential Revision: https://phab.mercurial-scm.org/D5216

catapult: rename 'active' to 'activetrace'; this isn't storing a boolean state (This is an attempt to make it more obvious that the indentation is correct, to prevent a repeat of 97f52862b1bd) Differential Revision: https://phab.mercurial-scm.org/D5215

catapult: fix broken run-tests catapult tracing Commit 97f52862b1bd thought this block was misindented, but it was not; the indentation after that commit means we never log a catapult trace for individual commands. Also, we've never emitted the "end" trace line when the test was finished, so the trace viewer claims that these "did not finish", fixing that as well. Differential Revision: https://phab.mercurial-scm.org/D5214

test: fix self._testdir to use the right mercurial library during testing Currently if you run tests from some other directory other than the `../tests/`, you will get a warning stating ``` warning: Testing with unexpected mercurial lib: mercurial (expected /tmp/hgtests.xxxxxx/install/lib/python/mercurial) ``` This is because the current directory being added to the 'PATH', if the `self._testdir != runtestdir`, owing to this line ``` if self._testdir != runtestdir: path = [self._testdir] + path ``` Also say you ran the tests from the hg base directory, because directory is being added in the PATH (see the above snippet, at that stage the `self._testdir` has the value as `cwd`, owing to a faulty initialization). And since the current directory already has the 'hg', that is used in place of the hg that is installed for the testing purposes in `/tmp/hgtests.xxxxxx/...`. Differential Revision: https://phab.mercurial-scm.org/D5199

tests: conditonalize a difference in test-merge-tools.t on Windows

statprof: clean up unicode/bytes a little I'm not really sure how this worked before, but something perturbed it and what I've got in this change I believe is a little tidier. This fixes test-profile.t on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5210

statprof: add a couple of asserts to avoid storing unicodes I don't feel strongly about this change, so it'd be fine to drop it during review, but it was _extremely_ helpful in drafting the next change. Differential Revision: https://phab.mercurial-scm.org/D5209

tests: add some helpful `|| cat` bits to test-profile.t This way if the hg invocation crashes, you get to see a stacktrace without having to edit the test. Differential Revision: https://phab.mercurial-scm.org/D5208

tests: skip wireproto clientreactor tests on Python 3.6.0-3.6.3 inclusive See comment next to the if statement for sad details. Differential Revision: https://phab.mercurial-scm.org/D5207

contrib: tweak import-checker to always use bytes for module names We could go the other route and make embedded() return this as a bytes, but then it would be harder to portably doctest, so I prefer this option. Differential Revision: https://phab.mercurial-scm.org/D5206

contrib: fix import-checker to not b'' module names on Python 3 Caught by the doctests. Differential Revision: https://phab.mercurial-scm.org/D5205

tests: add a critical flush() to run-tests.py to make output stable on py3 Differential Revision: https://phab.mercurial-scm.org/D5204

merge-tools: when calling external merge tool, describe the resolve inputs It is a common complaint that a user will be running some operation (histedit, rebase, evolve, etc.), get into a merge-conflict situation, and not understand what they are seeing - it is possible that the merge tool is configured to display the hash, but it's difficult for most merge tools to display a good snippet of the description. In the worst case, configuring this template will lead to output that is immediately covered by a terminal application, maybe the user can hit ctrl-z to see it. In the common case, the output will be in a terminal window and a GUI program will start, and it should be possible to view both the terminal and the GUI program at the same time. Differential Revision: https://phab.mercurial-scm.org/D5094

templatekw: deprecate p1rev/p2rev/p1node/p2node in favor of p1/p2

templatekw: add p1/p2 keywords which switches the current ctx This is just an example of mappingdict. I have no idea what should be displayed as {p1|json}. Currently it is an empty dict since {'ctx'} isn't displayable, which is clearly useless and should be changed later.

templater: add wrapper for a single template mapping This can be used to nest template mappings without inserting a sequence-like layer. See the next patch for example.

templatekw: extract internal "{rev}:{node|formatnode}" template to constant This will be used later.

test: remove changes preparing the drop of `HGMERGE` These changes are no longer necessary so we better drop them.

run-tests: define the default merge tool through configuration Using the `HGMERGE` environment variable generates confusion as it overrides any value set through configuration. By setting the default value through the default HGRC, the usual overriding rules apply and it should reduce confusion.

run-tests: explicitly declare the list of dropped environment variable It will make the list clearer and more maintainable.

test: drop usage of `HGMERGE` in `test-lfconvert.t` Using configuration instead of environment variable has a strange effect on the output, but no impact on the end result.

test: drop usage of `HGMERGE` in `test-merge-types.t` Using configuration instead of environment variable has a strange effect on the output, but no impact on the end result.

test: use the "correct" merge tool in `test-rebase-detach.t` The value of `HGMERGE` overrides the value passed through --config. We are about to drop this `HGMERGE` environment variable so we update the value used in config to match what is currently in use. This is another example of confusion introduced by the `HGMERGE` environment variable.

test: also deactivate `ui.merge` when testing merge tool We are about to introduce a default config for `ui.merge`. Since `test-merge-tools.t` is about testing merge configuration, we need to make sure it won't interfere.

tests: adjust merge tool config in test-largefiles-update.t Previously, the `ui.merge=internal:fail` configuration was shadowed by the the `HGMERGE=internal:merge` environment variable. We would like to remove this usage of `HGMERGE` variable because it is error-prone (eg: this very test). We start by cleaning up test independently to avoid large churn when the change happens.

extensions: include current version in "invalid version" message It is "not so rare" for the mercurial version to be badly detected at build time. In such case, version check for extensions gets confused. To help pinpoint the error, we now include the Mercurial's version number in the error message.

obsolete: prefetch the repo.obsstore used in phasedivergence loop The speedup is probably quite negligible, but it cannot hurt.

obsutil: prefetch method in allpredecessors loop We don't expect a massive speedup from this, but the change was laying around in my repository and it cannot hurt.

merge with stable

beautifygraph: don't substitute anything for 'X' in rendered graphs It looks like we never actually render graphs this way, although there's an example in a comment next to the code that prevents it from being needed (see graphmod.ascii()). Differential Revision: https://phab.mercurial-scm.org/D5103

py3: do not stringify integers in revlog stats by '%s'

py3: convert revlog stats to a dict of (bytes, int) pairs Py_DECREF(t) is replaced with Py_CLEAR(t) so that t is set to NULL once decrefed. Otherwise, it would be excessively decrefed if a subsequent PyBytes_FromString() failed.

py3: roll up threading.Thread constructor args into **kwargs The constructor doesn't have a `verbose` keyword argument in py3.

tests: glob over a single quote vs double quote difference on Windows

py3: port test-log-exthook.t to Python 3 For once an easy one. Differential Revision: https://phab.mercurial-scm.org/D5184

extensions: fix up many many debug logs that use %r This gets us a bunch closer on Python 3, but I'll still have to use a ton of sad globs. A previous version of this patch tried to preserve the %r formatting, but upon review Yuya noted that special characters in extension names is very unlikely, so we can just use %s and not sweat the quoting. Differential Revision: https://phab.mercurial-scm.org/D5002

blackbox: add configitem for format of log timestamps Sometimes blackbox logs are used to report performance problems, but the timestamps are only at second granularity, so often the timings have to stated separately by the reporter. This is inconvenient and error-prone, so I would like to include %f in the date format. This patch makes that possible.

narrow: replace filtering in list comprehension by set operations I didn't think of this while reviewing the patch. Differential Revision: https://phab.mercurial-scm.org/D5188

storage: update sqlitestore to use the new `deltamode` parameter While updating the filelog class, I forget to update the sqlitestore, tests are now passing with this patch. Differential Revision: https://phab.mercurial-scm.org/D5185

narrow: rework logic to check whether we need to widen and narrow This patch reworks logic which calculates whether we need to extend or narrow our working copy or not. We filter the addincludes, removeincludes, addexcludes and removeexcludes passed from user to the actual added and removed includes and excludes. What that means is a user can pass an already included path as addincludes, a path which is not included as removeincludes etc. In such situations the old logic use to think we need to do some work, whereas we don't need to do that work. In old logic, even if we don't have anything new to include but it believes we need to call widen, this adds some good amount of work on large repository. A widen calls involves computing incomming csets, calling the narrow_widen() which in non-ellipses cases goes through all the set of csets which are available which can take ~2-3 mins on large repos. Those 2-3 minutes are spend on doing nothing which a client can prevent by checking is there really anything which needs to be included. The tests changes shows that we don't go to the server anymore in such cases which is nice. Differential Revision: https://phab.mercurial-scm.org/D5183

tests: show that adding an already included path still calls narrow_widen() This patch adds tests demonstrating that we still go to the server in non-ellipses widening when we have that path already on the client and there is nothing new to download. The next patch will try to make client side logic smart and not go to the server if we don't need to download anything. Differential Revision: https://phab.mercurial-scm.org/D5182

graft: introduce --base option for using custom base revision while merging The graft command usually performs an internal merge of the current parent revision with the graft revision, using p1 of the grafted revision as base for the merge. As a trivial extension of this, we introduce the --base option to allow for using another base revision. This can be used as a building block for grafting and collapsing multiple changesets at once, or for grafting the resulting change from a merge as a single simple change. (This is kind of similar to backout --parent ... only different: this graft base must be an ancestor, but is usually *not* a parent.) This is probably an advanced use case, and we do thus not show it in the non-verbose help.

changegroup: add a option to create bundle with full snapshot only This is easy to implement now and can be useful for benchmarking.

changegroup: allow to force delta to be against p1 This new developer option is useful to general more "generic" bundle. Without this option, a bundle generated from the repository use deltas similar to the one stored in the specific repository it was generated from. This makes performance testing a bit tricky. Using deltas similar to the final result means all delta stored in the bundle can be applied to the target repository without any further processing (except for the rare case of a full snapshot). The application of such bundles (almost) never exercises the (slower) path of searching for a new valid delta. This result in unrealistic and too favorable timing and profile. Instead, we introduce an option to make sure all revisions are stored as a delta against p1. It might not be the best generation option, but it guarantees that the content will be "generic", not favoring a specific target.

storage: also use `deltamode argument` for ifiledata Now that lower level uses such argument, we can propagate the change to higher layers.

changegroup: refactor emitrevision to use a `deltamode` argument This new argument gathers the semantic of `sendfulltext` and `deltaprevious` in a single value. We are about to introduce a new type of constraints. Avoiding yet another argument sounds like a plus.

branchmap: do not specify changelog as an argument Since (unfiltered)repo.changelog lookup gets as fast as __dict__ lookup, there's no point to pass in changelog instance. $ hg perfbranchmap --clear-revbranch -R mozilla-central ! base (orig) wall 20.593091 comb 20.600000 user 20.520000 sys 0.080000 (best of 3) (this) wall 20.129126 comb 20.130000 user 20.020000 sys 0.110000 (best of 3) This backs out most of the changes in 76d4272bd57b and 47c03042cd1d.

filecache: unimplement __set__() and __delete__() (API) Implementing __set__() implies that the descriptor can't be overridden by obj.__dict__, which means any property access involves slow function call. "Data descriptors with __set__() and __get__() defined always override a redefinition in an instance dictionary. In contrast, non-data descriptors can be overridden by instances." https://docs.python.org/2.7/reference/datamodel.html#invoking-descriptors This patch basically backs out 236bb604dc39, "scmutil: update cached copy when filecached attribute is assigned (issue3263)." The problem described in issue3263 (which is #3264 in Bugzilla) should no longer happen since repo._bookmarkcurrent has been moved to repo._bookmarks.active. We still have a risk of introducing similar bugs, but I think that's the cost we have to pay. $ hg perfrevset 'branch(tip)' -R mercurial (orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66) (prev) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81) (this) wall 0.099038 comb 0.110000 user 0.100000 sys 0.010000 (best of 93)

filecache: use try-except for faster __dict__ lookup Python function call is slow, and the cost could be significant here. $ hg perfrevset 'branch(tip)' -R mercurial (orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66) (this) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81)

help: displaying extension commands by default Differential Revision: https://phab.mercurial-scm.org/D5156

help: displaying documented aliases by default This makes aliases be displayed in "hg help" when they have a :doc config entry, and also allows them to be assigned to a category with :category. Differential Revision: https://phab.mercurial-scm.org/D5087

help: allow hiding of help topics Differential Revision: https://phab.mercurial-scm.org/D5077

help: allow commands to be hidden This is useful in enterprise environments where some workflows are discouraged. Differential Revision: https://phab.mercurial-scm.org/D5076

py3: add one more passing test to whitelist Caught by python 3 builder. Differential Revision: https://phab.mercurial-scm.org/D5175

py3: make sure we pass sysstr in sqlite3.connect() Differential Revision: https://phab.mercurial-scm.org/D5174

archive: use manifest.matches() to simplify and speed up matching manifest.matches() can avoid walking paths the user did not want to archive. Differential Revision: https://phab.mercurial-scm.org/D5178

archive: create alwaysmatcher when no matcher provided Differential Revision: https://phab.mercurial-scm.org/D5177

archive: change "matcnfn" argument to a real matcher All callers seem to be passing a real matcher, not just a function. We were also passing it into match.subdirmatcher(), which assumes it is a matcher. Differential Revision: https://phab.mercurial-scm.org/D5176

histedit: crashing with a more useful error message on empty defaultrev Before this, `hg --config histedit.defaultrev= histedit` would crash with File "destutil.py", line 385, in desthistedit if revs: UnboundLocalError: local variable 'revs' referenced before assignment Differential Revision: https://phab.mercurial-scm.org/D5543

Added signature for changeset 197f092b2cd9

Added tag 4.8.2 for changeset 197f092b2cd9

server: always close http socket if responding with an error (issue6033) It's possible for hgweb to respond _very_ early with an error if we're catching certain types of errors. When we do, we need to tell the client the socket is toast when there's a POST involved because otherwise there can be lingering POST data on the socket that will confuse any future requests on the socket. This manifested as a flaky failure on Linux in an lfs extension test and a reliable failure on FreeBSD. With this patch applied, test-lfs-serve-access.t now passes for me on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5498

match: fix assertion for fileset with no context (issue6046) A falsy changectx should be allowed.

templatekw: fix documentation typos

update: do not pass in user revspec as default destination (issue6044) When the revsingle() was introduced at 61c0df2b089a, it couldn't handle revspec=0 (not '0') properly. That's probably why the default was set to rev. This is technically BC since "hg update ''" was identical to "hg update '.'" whereas "hg update -r ''" is "hg update", but I believe that's a bug given no test fails with this change.

revlog: cache delta base value under -1 Such base are invalid so we better report them early.

revlog: catch revlog corruption in index_baserev A revision cannot use a base above itself, it can only happens one corrupted repository. Ignoring such corrupted could lead to infinite loop.

phabricator: properly encode boolean types in the request body I tripped over this playing with `hg debugcallconduit` to query for valid reviewers. If the JSON on stdin is written as 'True' or 'False', python complains it isn't valid JSON. If it's written as 'true' or 'false', it made it to the server, but got kicked back with this: abort: Conduit Error (ERR-CONDUIT-CORE): Error while reading "isBot": Expected boolean (true or false), got something else. The test isn't really relevant here (the code can be reverted, and it will pass), but this gives us coverage for the debug command.

worker: do not swallow exception occurred in main process Before, SystemExit(255) would be most likely raised since the worker processes were terminated by the main process and the status would be set to 255 in response. We should instead re-raise the exception occurred first. It's pretty hard to debug problems like the issue 6035 with no traceback.

run-tests: fix permission to clean up unreadable directories I found many hgtests.* directories left in $TMPDIR, which couldn't be deleted because test-ssh-repoerror.t created some directories with a-rx mode.

context: reimplement memfilectx.cmp() If I added a sanity check to basefilectx, test-context.py exploded. This patch copies the naive implementation from overlayworkingfilectx.

windows: ensure pure posixfile fd doesn't escape by entering context manager There are tests in test-revlog-mmapindex.t and test-rebase-mq-skip.t that are fixed by this, but we usually don't use --pure on Windows. For whatever reason, the remaining --pure failures are various errors like $ENOTDIR$ and "Access is denied" have a trailing '.'.

vfs: ensure closewrapbase fh doesn't escape by entering context manager I'm not sure if there's a problem in practice here, as there's no test failure either way. The __exit__() and close() methods raise an exception, so maybe __exit__() and close() are being called directly on the underlying handle when delayclosedfile is used on a context manager? I doubt that was intended.

windows: ensure mixedfilemodewrapper fd doesn't escape by entering context mgr Otherwise it seems that the special read and write handling would be bypassed.

py3: ensure the proxied Windows fd doesn't escape by entering context manager The purpose of the proxy class is to provide the `name` attribute which contains the file path. But in tests that used a context manager, it still blew up complaining that 'int' doesn't have a 'startswith' function.

test: fix test-http-bad-server with current python 2.7 https://github.com/python/cpython/pull/2825 changed the exception message for empty http status line. Differential Revision: https://phab.mercurial-scm.org/D5412

hgweb: register web.comparisoncontext to the config table This was caught in some server side logging added to debug py3 issues.

Added signature for changeset 1c8c54cf9725

Added tag 4.8.1 for changeset 1c8c54cf9725

rebase: fix path auditing to audit path relative to repo root (issue5818) Before this patch, when rebasing a file called "foo/bar", we would check e.g. if "/foo" (i.e. rooted at the file system root) was a symlink. Differential Revision: https://phab.mercurial-scm.org/D5361

tests: show bad path auditing in in-memory rebase Thanks to Yuya for providing this test case in https://bz.mercurial-scm.org/show_bug.cgi?id=5818. Differential Revision: https://phab.mercurial-scm.org/D5368

tests: add a missing "cd .." to test-rebase-inmemory.t Differential Revision: https://phab.mercurial-scm.org/D5367

rust: fix possible out-of-bounds read through index_get_parents() index_get_parents() is an internal function, which doesn't check if the specified rev is valid. If rustlazyancestors() were instantiated with an invalid stoprev, it would access to invalid memory region. This is NOT a security fix as there's no Python code triggering the bug, but included in this series to not give a notion about the memory issue fixed by the previous patch.

revlog: fix out-of-bounds access by negative parents read from revlog (SEC) 82d6a35cf432 wasn't enough. Several callers don't check negative revisions but for -1 (nullrev), which would directly lead to out-of-bounds read, and buffer overflow could follow. RCE might be doable with carefully crafted revlog structure, though I don't think this would be useful attack surface.

rebase: fix dir/file conflict detection when using in-mem merge Differential Revision: https://phab.mercurial-scm.org/D5360

tests: show that in-mem rebase does not find path dir/file conflicts Differential Revision: https://phab.mercurial-scm.org/D5359

extdiff: register the configuration generated commands with a help category Otherwise, 'extdiff' shows up under file management and the rest of the commands are at the bottom under 'Uncategorized'.

rebase: abort in-mem rebase if there's a dirty merge state In-memory merge uses the on-disk merge state, so we should not allow it run in-memory merge when the merge state is not clean. We should probably not use the on-disk merge state when running in-memory merge, but chaning that is not suitable for the stable branch. Differential Revision: https://phab.mercurial-scm.org/D5357

rebase: preserve working copy when redoing in-mem rebase on disk When in-memory rebase runs into conflicts, we retry it on disk. But before we do that, we abort the in-memory rebase. That is done because even though it's mostly in memory, there are still a few state files written (e.g. the merge state). We should make it not write those files so we don't need to abort, but for the stable branch, let's explicitly clear the state we need to clear instead of running the usual abort code. Differential Revision: https://phab.mercurial-scm.org/D5356

tests: show that in-mem rebase falling back loses state Both working copy changes and the merge state is lost when in-memory rebase falls back to on-disk mode. Differential Revision: https://phab.mercurial-scm.org/D5355

commandserver: get around ETIMEDOUT raised by selectors2 selector.select() should exits with an empty event list on timed out, but selectors2 raises OSError if timeout expires while recovering from EINTR. Spotted while debugging new chg feature.

selectors2: backport minimal fix of timeout handling from 2.0.1 The original code would raise TypeError since OSError() doesn't support keyword arguments. We can't simply import the selectors 2.0.1, which still spawns "uname -p" through platform.system(). We could switch to the unreleased version, but I decided to not right now to minimize the change.

tests: sniff for libfuzzer actually being available in test-fuzz-targets.t When I upgraded the FreeBSD buildbot to 11.2 it seems we picked up clang6, but the default clang on FreeBSD doesn't include libfuzzer. I can't find a way to sniff for libfuzzer without running a compile, so here we are. Differential Revision: https://phab.mercurial-scm.org/D5270

tests: sniff for /usr/local/bin/gmake and use it in test-fuzz-targets.t This isn't as robust as it probably should be, but for now it'll get the job done on the buildbots. Differential Revision: https://phab.mercurial-scm.org/D5269

tests: stabilize test-inherit-mode.t on FreeBSD and macOS (issue6026) Symbolic links are funny permissions-wise, but on the linked issue Yuya has convinced me that we can ignore this permissions issue on macOS (FreeBSD allows setting permissions bits but ignores them) and we'll be in fine shape.

wireprotov2peer: wait for initial object before resolving future As part of rolling out wireprotov2 with redirect support, I encountered an edge case with regards to future resolution. Essentially, the initial response frame from the server did not fully decode the initial CBOR object. The frame wasn't marked as EOS. In the previous code, we resolved the future for the request to response.objects(), which mapped to the commandresponse instance which would eventually produce a redirect. Upon receiving subsequent data, the initial CBOR object containing the redirect would be decoded and we'd process the redirect. However, the future would already have been resolved with the initial commandresponse.objects() and the client iterating over the objects wouldn't receive any objects from the redirect because the redirect was populating a different commandresponse instance! This commit changes the logic so we don't resolve futures until the initial CBOR response object is fully decoded or until EOS occurs. In cases where there is an empty or partial frame associated with a redirect, the future will now resolve with the commandresponse containing the proper series of decoded objects.

wireprotov2peer: always return a bool from _processredirect() Without this, we may stop servicing the redirect response if the future has already been resolved. And the future will often be resolved very early, since many consumers iterate the decoded CBOR object stream and expect data to lazily arrive.

tests: stabilize the recent checkexec changes on Windows This goes with bd0874977a5e.

checkexec: create destination directory if necessary Since 460733327640, a "share" use the cache of the source repository. A side effect is that no `.hg/cache` directory exists in the "share" anymore. As a result, the checkexec logic can't use it to create its temporary file and have to use the working copy for that. This is suboptimal, it pollutes the working copy and prevents them to keep the file around in cache. We do not want to use the cache directory for the share target, it might be on a different file system. So instead, we (try to) create the directory if it is missing. This is a simple change that fixes the current behavior regression on stable. On default, we should probably ensure the proper directories are created when initializing the repository. We should also introduce a 'wcache' directory to hold cache file related to the working copy. This would clarify the cache situation regarding shares. The tests catch a couple of other affected cases.

graft: do not try to skip rev derived from ancestor more than once (issue6024) We check 'x in revs' in other cases, so let's do the same. The test case credits to Tom Prince.

subrepo: print the status line before creating the peer for better diagnostics I ran into a problem where I tried updating to a different branch, and the process appeared to hang. It turned out that the subrepo revision wasn't available locally, and I must have originally cloned it from an `hg serve -S` on a machine that currently wasn't serving anything. It took 2+ minutes to timeout, and didn't mention what it was connecting to even then. There are a couple of other issues in this scenario too. - The repo is dirty after the failed checkout because the top level repo is updated first. We should probably make 2 passes- top down to pull everything needed, and then do an update once everything is in place. - Something must be reading .hgsubstate from wdir because if the same merge command is run after the timeout, a prompt is issued that the local and remote subrepo diverged, instead of hanging. But it lists the local version and remote version as having the same hash.

tests: allow for 100% of profiled time in sleep in test-profile.t I'm getting an annoying failure in this test on our builder, and I *think* what's happening is that the profiler is taking _just_ long enough to start that we're spending 100% of the profiled time in the sleep function, which was causing the leading space to not be printed since the 100 was in the first column of output. Differential Revision: https://phab.mercurial-scm.org/D5272

copystore: provide unit to ui.makeprogress()

verify: provide unit to ui.makeprogress()

tests: fix wireproto redirection test on systems without tls1.2 Our automated package builder has some ancient configuration that lacks modern TLS, which is how we noticed this. Tested: the test now passes on both macOS High Sierra (has tls1.2) and Ubuntu Trusty (which does not).

phabricator: ensure the command summaries are available in extension help Previously, `hg help phabricator` listed the 3 supported commands at the bottom of the extension help, but said "no help text available".

hgweb: cast bytearray to bytes PEP-3333 seems to indicate that bytes is the only allowed type that can be used to express the output of a WSGI application. And some WSGI environments seem to enforce this (mod_wsgi does). This commit universally casts bytearray instances to bytes to appease the WSGI specification. I found this because wireprotov2 is emitting bytearray instances. I'd like to keep things that way because the way it builds a data structure, bytearray is more efficient. I'd rather keep the low-level code efficient (and using bytearray) and cast at the edges than impose a performance penalty on code that may run outside WSGI contexts.

help: unjumble the list of default config values for `internals.config`

tweakdefaults: remove commands.resolve.mark-check=abort, it is too broken See issue6020 for the current case. I don't want to continue attempting to fix this on the stable branch, so I'm removing from tweakdefaults and will send fixes meant for the default branch and 4.9. Differential Revision: https://phab.mercurial-scm.org/D5225

resolve: when resolve.mark-check=abort, downgrade to warning if pats specified Previously, with --config resolve.mark-check=abort, running `hg resolve -m foo` would abort and emit a message saying to use --all. This command does not work, though: `hg resolve -m foo --all`, and it's really weird for --all to be the "--force" flag. My original goal with the option was to make it so that `hg resolve -m` (no filename arguments) was safer, which is why --all is used; in my mind, `hg resolve -m foo` should always mark it as resolved, and `--all` is how you specify "all the files", so that's why I chose `hg resolve -m --all` as the way out of `hg resolve -m` aborting. This commit makes all of this work the way it was meant to in my head :) Differential Revision: https://phab.mercurial-scm.org/D5218

Added signature for changeset a91a2837150b

Added tag 4.8 for changeset a91a2837150b

rust: fix signature of rustlazyancestors_init() function Obviously, sizeof(int) != mem::size_of::<usize>() on amd64, though the argument would be passed in 64-bit register anyway.

tests: require SQLite 3.8.3+ as sqlitestore relies on "WITH" clause The test fails on gcc112 because the SQLite is too old. https://sqlite.org/changes.html#version_3_8_3

relnotes: various tweaks for release notes Stop filtering out commits that are expected to be covered by releasenotes extension: now we want two lists, one for WhatsNew and one for ReleaseX.Y. Use `only(stoprev, startrev)` to make `relnotes -h` output be actually true about what revisions are included. More filter rules, mostly obvious. More classifying rules to have less things in "unsorted". Looks like nargs=1 was just making args.startrev and args.stoprev be lists for no reason. BC and API sections are renamed to what we're using on the WhatsNew page, and also just skipped if empty.

delta: skip "empty delta" optimisation for non-general case (issue6006) Non-general delta repository cannot delta against anything than prev. So even if the delta to prev is empty we should use it. This is similar to the change made in bafa1c4bb7a8. Differential Revision: https://phab.mercurial-scm.org/D5201

narrow: fix copies._fullcopytracing() narrowspec filtering in graft case I broke this too in 707c3804e607 (narrow: move copies overrides to core, 2018-09-28). Hopefully I'm done fixing things broken by that commit now. Differential Revision: https://phab.mercurial-scm.org/D5213

tests: demonstrate broken copies._fullcopytracing() Turns out copies._fullcopytracing() was also broken. Differential Revision: https://phab.mercurial-scm.org/D5212

narrow: make copies.pathcopies() filter with narrowspec again I broke this in 707c3804e607 (narrow: move copies overrides to core, 2018-09-28). Differential Revision: https://phab.mercurial-scm.org/D5203

tests: demonstrate broken copies.pathcopies() Differential Revision: https://phab.mercurial-scm.org/D5202

setup: explain to distutils how we write rc versions When we use a rc version number (e.g. 4.8rc0), bdist_msi is using distutils.StrictVersion to parse it into a tuple of numbers. By default, StrictVersion.version_re only recognizes [ab] for alpha/beta, where mercurial may use '-rc' or 'rc'. This change makes StrictVersion parse correctly our version numbers, so that bdist_msi doesn't fail on rc versions.

changegroup: restore default node ordering (issue6001) Changeset db5501d9 changed the default node ordering from "storage" to "linearize". While the new API is more explicit and cleaner, the "linearize" order is problematic on certain repositories like netbeans where it makes bundling slower the more nodes we bundle. Pushing and pulling 100 changesets was ~20% slower and pushing and pulling 1000 changesets was ~600% slower. A very quick analysis of profile traces showed that the pull operation was taking more time creating the delta. Putting back the old default order seems to be the safe option. With more time during the next cycle, we can understand better the impact of sorting with the DAG order by default, the source of the regression and how to mitigate it. /!\ We are still waiting for the full performance impact but with this patch, bundling and pulling locally (not on the performance workstation) 1000 changesets on the netbeans repository is as fast as before the regression. Differential Revision: https://phab.mercurial-scm.org/D5196

changegroup: introduce an explicit linear sorting We still need to linearize the revisions in some cases, introduce an explicit `linear` sorting before changing back the default order. Differential Revision: https://phab.mercurial-scm.org/D5195

fix: disable use of thread-based worker getfixes() accesses to repo, changectx, filectx, etc., so I believe there are code paths triggering data race. Mercurial API isn't thread safe in general.

configitems: rename the config to prevent adding an alias in future Right now the config option looks like: [experimental.server] stream-narrow-clones= which does not match how config options are generally defined in core. So let's rename this to: [experimental] server.stream-narrow-clones= before the new release so that we don't have to add an alias in future for this. Differential Revision: https://phab.mercurial-scm.org/D5198

sparse-revlog: only refine delta candidates in the sparse case (issue6006) Starting with 5aef5afa8654, a valid delta parent might be "refined". This allows repository using sparse-revlog to produce better delta chain by using better intermediate snapshot base. However, this refining step was performed in all cases, including for repository not using sparse-revlog. This could produce a strange chain in the general delta case and corrupted repository in the non-general delta case. We now skip this step unless sparse-revlog is in use. In issue 6006, Yuya Nishihara provided a test case using an external repository, so we did not include it. Finding "laboratory" condition to reproduce this case and implementing an efficient test reproducing it is a bit tricky. We do not foresee to have the time to provide one by the release date. Differential Revision: https://phab.mercurial-scm.org/D5197

http: work around custom http client classes that refuse extra attrs I have no idea what is going on with our custom http client code at Google, but it chokes on these extra attributes we're tucking on http clients. Since it feels more than a little wrong to just stuff extra data on a client, let's degrade gracefully when the client class refuses the attributes.

crecord: make nextsametype() check that parent item exists (issue6009) Items that represent files in curses interface don't have parents.

help: describe what ui.tweakdefaults changes, concretely Currently, one has to look at the code. A couple things are suboptimal: - probably not translatable - lines don't get wrapped (a couple are a bit too long) but it seems to better this way than without help at all. Differential Revision: https://phab.mercurial-scm.org/D5187

logexchange: convert paths to unix when detecting the active path This fixes the problem in the tests[1] where Windows was showing the whole path as the remotename for local repositories. Somebody with a better understanding of this extension should probably take a deeper look. There may be other cases that need to be converted- specifically the `elif not instance` and the missing `else` cases in activepath(). I also noticed when adding debug prints that the absolute path is stored in the file, probably not normalized. (It's wrapped up in $TESTTMP.) [1] https://buildbot.mercurial-scm.org/builders/Win7%20x86_64%20hg%20tests/builds/1042/steps/run-tests.py%20%28python%202.7.13%29/logs/stdio

help: update the default value specified for `profiling.time-track` I tried conditionalizing this in a `.. container::` block, but that seemed to add an extra blank line between the main text and the parenthetical.

profiling: revert the default mode back to 'cpu' on Windows On Windows, os.times() only returns user and system times. Real elapsed time is 0. That results in no actual times reported, an end wall time of 0.000000, and seemingly randomly sorted stack frames. This at least provides test stability in test-profile.t. I kind of think that `default=pycompat.iswindows and 'cpu' or 'real'` would be a better way to set the default in configitems, but I didn't see any other examples of this, and thought maybe there's a reason for that. That might allow plugging the value into the help text automatically- the documented default wasn't updated in db0dba2d157d.

phase: add an archived phase This phase allows for hidden changesets in the "user space". It differs from the "internal" phase which is intended for internal by-product only. There have been discussions at the 4.8 sprint to use such phase to speedup cleanup after history rewriting operation. Shipping it in the same release as the 'internal-phase' groups the associated `requires` entry. The important bit is to have support for this phase in the earliest version of mercurial possible. Adding the UI to manipulate this new phase later seems fine. The current plan for archived usage and user interface are as follow. On a repository with internal-phase on and evolution off: * history rewriting command set rewritten changeset in the archived phase. (This mean updating the cleanupnodes method). * keep `hg unbundle .hg/strip-backup/X.hg` as a way to restore changeset for now (backup bundle need to contains phase data) * [maybe] add a `hg strip --soft` advance flag (a light way to expose the feature without getting in the way of a better UI) Mercurial 4.8 freeze is too close to get the above in by then. We don't introduce a new repository `requirement` as we reuse the one introduced with the 'archived' phase during the 4.8 cycle.

exewrapper: apply clang-format to silence test-check-clang-format.t

Added signature for changeset 956ec6f1320d

Added tag 4.8rc0 for changeset 956ec6f1320d

merge to stable for 4.8 release freeze

shortest: never emit 0-length prefix even if unique It turned out that the pure version of our code for finding the shortest unique nodeid prefix would return a 0-length string if that was unique (because there was at most one revision in the disambiguation set). That's kind of correct, but it can't be used as input, so we shouldn't return it. Let's just adjust the given minlength up to at least 1. This fixes test-template-functions.t, which was failing in pure mode. Differential Revision: https://phab.mercurial-scm.org/D5181

logtoprocess: sends the canonical command name to the subprocess One of the use-case of logtoprocess is to monitor command duration. With the current code, we only get whatever command name the user typed (either abbreviated or aliased). This makes analytics on the collected data more difficult. Stores the canonical command name in the request object. Pass the stored canonical name in the `req.ui.log("commandfinish", ...)` call as keyword argument to not break potential string formatting. Pass the value as the environment variable named `LTP_COMMAND` to the called script. Differential Revision: https://phab.mercurial-scm.org/D4820

logtoprocess: fix message formatting The logtoprocess used to try formatting the message using keyword options instead of always using the rest of the arguments. Update it to match blackbox behavior. Differential Revision: https://phab.mercurial-scm.org/D5180

profiling: move default mode to "real" time Mercurial operations involve a lot of disks or network access. These impact command runtime significantly and it seems important to report them in our default profiling output. Having the right default means that we don't forget them when asking people to produces profiling traces or when doing profiling ourselves. Moving to "real time" by default will remove the need to think about activating it on most occasions. The "CPU" time-based profiling is still accessible when necessary.

streamclone: abort when client needs to handle obsmarkers, but doesn't When client doesn't have any of obsolescence markers exchange capabilities, then it's safe to say it can't handle obsmarkers. However, if it understands even one format version, then stream clones are fine -- client can use "obsmarkers" bundle2 part.

streamclone: include obsstore file into stream bundle if client can read it

setup: build exewrapper with Unicode support on py3 I didn't see a compiler switch documented anywhere, but I diffed the command line for full VC++ project when toggling between MBCS and Unicode. This is all they do.

exewrapper: convert to _tcsxxx functions for Unicode compatability This fixes more than 50 tests on py3 on Windows when enabled, mostly hooks and such that invoked `hg` directly. 187 left to go. I skipped doing the abort printing with Unicode because of apparent issues with MinGW [1]. It may be moot though, as MinGW isn't listed as a supported compiler after 3.4 [2]. [1] https://stackoverflow.com/questions/17700797/printf-wprintf-s-s-ls-char-and-wchar-errors-not-announced-by-a-compil [2] https://wiki.python.org/moin/WindowsCompilers

exewrapper: drop an unused variable

commands: restore compatibility for "^cmd" registration (issue6005) This is done at loading time, where ui is available.

exchangev2: support fetching shallow files history This commit teaches the exchangev2 client code to handle fetching shallow files data. Only shallow fetching of files data is supported: shallow fetching of changeset and manifest data is explicitly not yet supported. Previously, we would fetch file revisions for changesets that were received by the current pull operation. In the new model, we calculate the set of "relevant" changesets given the pull depth and only fetch files data for those changesets. We also teach the "filesdata" command invocation to vary parameters as needed. The implementation here is far from complete or optimal. Subsequent pulls will end up re-fetching a lot of files data. But the application of this data should mostly be a no-op on the client, so it isn't a big deal. Depending on the order file revisions are fetched in, revisions could get inserted with the wrong revision number relationships. I think the best way to deal with this is to remove revision numbers from storage and to either dynamically derive them (by reconstructing a DAG from nodes/parents) or remove revision numbers from the file storage interface completely. A missing API that we'll likely want to write pretty soon is "ensure files for revision(s) are present." We can kind of cajole exchangev2.pull() to do this. But it isn't very efficient. For example, in simple cases like widening the store to obtain data for a single revision, it is probably more efficient to walk the manifest and find exactly which file revisions are missing and to make explicit requests for just their data. In more advanced cases, asking the server for all files data may be more efficient, even though it requires sending data the client already has. There is tons of room for future experimentation here. And TBH I'm not sure what the final state will be. Anyway, this commit gets us pretty close to being able to have shallow and narrow checkouts with exchangev2/sqlite storage. Close enough that a minimal extension should be able to provide fill in the gaps until the code in core stabilizes and there is a user-facing way to trigger the narrow/shallow bits from `hg clone` without also implying using of the narrow extension... Differential Revision: https://phab.mercurial-scm.org/D5169

sqlitestore: support for storing revisions without their parents This commit kinda/sorta implements the equivalent of ellipsis nodes for the SQLite storage backend. Without implementing full blown ellipsis nodes (and the necessary support for them in the wire protocol), we instead teach the store to rewrite the p1 and p2 nodes to nullid when the incoming parent isn't in the local store. This allows servers to remain dumb and send the real parent and have the clients deal with the missing parent problem. This obviously isn't ideal because a benefit of ellipsis nodes is we can insert a fake parent to ellide missing changesets. But neither solution is ideal because it drops the original parent from storage. We could probably teach the SQLite store to retain the original parent and handle missing parents at read time. However, parent revisions are stored as integers and it isn't trivial to store an "empty" revision in the store yet, which would be necessary to represent the "missing" parent. The store is somewhat intelligent in trying to remove the missing parents metadata when the revision is re-added. But, revision numbers will be all messed up in that case, so I'm not sure it is worth it. At some point we'll likely want to remove the concept of revision numbers from the database and have the store invent them at index generation time. Or even better, we can do away with revision numbers from the file storage interface completely. We'll get there eventually... Differential Revision: https://phab.mercurial-scm.org/D5168

wireprotov2: support exposing linknode of file revisions When supporting shallow file storage, clients may fetch file revisions by changeset. But they may not readily know which changeset introduced a specific file revision. The "linknode" is used to record which changeset introduces which file revision. This commit teaches the "filedata" and "filesdata" wire protocol commands to expose the linknode for file revisions. The implementation is likely wrong when hidden changesets are in play, since the linknode may refer to a hidden changeset. We can deal with this problem later. Differential Revision: https://phab.mercurial-scm.org/D5167

localrepo: support marking repos as having shallow file storage Various operations against repositories need to know if repository storage is full or partial. For example, a checkout (including possibly a widening of a sparse checkout), needs to know if it can assume all file revisions are available or whether to look for missing revisions first. This commit lays the plumbing for doing that. We define a repo creation option that indicates that shallow file storage is desired. The SQLite store uses this creation option to add an extra repo requirement indicating file storage is shallow. A new repository feature has been added to indicate that file storage is shallow. The SQLite store adds this feature when the shallow file store requirement is present. Code can now look at repo.features to determine if repo file storage may be shallow and take additional actions if so. While we're here, we also teach the SQLite store to handle the narrow repo requirement, which gets added when making narrow clones. Differential Revision: https://phab.mercurial-scm.org/D5166

repository: teach addgroup() to receive data with missing parents The way the narrow extension works today, the server rewrites outgoing changegroup data to lie about parents when the parents data is missing. It adds the ellipsis flag to the revision so it can be recorded as such in the revlog. In the new wire protocol, such rewriting does not occur on the server (at least not yet anyway). Instead, it is up to the client to recognize when it has received a revision without its parents. This means rewriting will be performed on the client. Furthermore, the mechanism for storing a shallow revision may differ from store to store. For example, the revlog store uses the ellipsis flag to denote a revision's parents have been rewritten. But a non-revlog store may wish to store things differently. And, some stores may not even support receiving shallow revision data! Therefore, it makes sense for the store itself to be making decisions about what to do when they receive revision data without their parents. This commit teaches the addgroup() bulk insert method to accept a boolean argument that indicates whether the incoming data may lack parent revisions. This flag can be set when receiving "shallow" data from a remote. The revlog implementation of this method has been taught to rewrite the missing parent(s) to nullid and to add the ellipsis flag to the revision when a missing parent is encountered. But it only does this if ellipsis flags are enabled on the repo and the incoming data is marked as possibly shallow. An error occurs otherwise. Differential Revision: https://phab.mercurial-scm.org/D5165

commands: support passing depth to hg.clone() This will allow extensions to add --depth or other arguments to control depth fetching. Differential Revision: https://phab.mercurial-scm.org/D5164

filelog: add a hasnode() method (API) Missing in the file storage interface is the ability to query whether a specified value is a known node. This commit defines that interface member and implements it on the revlog and sqlite file stores. Storage unit tests have been added. The revlog implementation is a bit more complicated because index lookups don't consistently raise the same exception. For SQLite, we can simply look for a key in a dict. Differential Revision: https://phab.mercurial-scm.org/D5163

lfs: consult the narrow matcher when extracting pointers from ctx (issue5794) I added a testcase for lfs to all narrow tests, and the following failed: test-narrow-acl.t test-narrow-exchange.t test-narrow-patterns.t test-narrow-strip.t test-narrow-trackedcmd.t test-narrow-widen.t test-narrow.t The first two still have errors in the pretxnchangegroup on clone and (receiving a) push, which I'm still looking into (4d63f3bc1e1a fixed something in this area already). These two modified tests seem to cover the things that failed in the remaining narrow tests, i.e. `hg tracked` and `hg strip`, so I didn't bother enabling the testcases elsewhere. Maybe we should, but it's 68 tests total.

statprof: fix overflow while skipping boilerplate parts I got IndexError randomly because of stack[i] where i = len(stack).

statprof: fix indent level of fp.write() (issue6004) It was changed at 9d3034348c4f by mistake.