py3: stop subscripting socket.error In 3.3 and later, this is now an alias for OSError. I hacked up the server code enough that I was able to trigger the exception handler in server.py from test-http-bundle1.t. Other instances of this either subscript through the `args` member, or reference the errno or strerror attributes. Note that on Windows, the errno value seems to reflect the Winsock error, so the various tests for EPIPE seem like they would always fail. But that seems to be the case in py2 as well.

test: update test-logtoprocess.t to not requires less The test previously assumed that a working pager was present in the test environment. Update it to use the fakepager instead. Differential Revision: https://phab.mercurial-scm.org/D5399

py3: quote several instances of $PYTHON for Windows Python3 lives in Program Files by default. The last remaining unquoted instance is in test-hghave.t. I can't figure out how to quote that, as it either complains: 'c:\\Program' is not recognized as an internal or external command, operable program or batch file. or $TESTTMP.sh: line 22: "C:/Program Files/Python37/python.exe": $ENOENT$

py3: adapt test-largefiles-wireproto.t to python3

py3: byteify test-push-race.t

py3: convert filename to bytes in test-hardlinks.t Windows was complaining about passing str to CreateFileA() in win32.py.

narrow: keep bookmarks temporarily stripped for as long as commits are The narrow extension also has support for shallowness and for inserting older commits on pull. It works by temporarily stripping newer commits, adding the older commits, then re-applying the stripped bundle. The regular Mercurial server uses that when you widen, although it shouldn't be necessary there. Our Google-internal server does it when the user requests an older commit. Our Google-internal tests fail since 7caf632e30c3 (filecache: unimplement __set__() and __delete__() (API), 2018-10-20). I haven't quite understood the problem, but it's related to the way we temporarily hide bookmarks while the commits they point to are stripped. When a transaction is started, Mercurial tries to read various things from the repo for the transaction summary. That leads to computation of hidden commits, which leads to an attempt to find commits pinned by bookmarks. This is the reason we temporarily hide the bookmarks. I think the aforementioned commit makes the restored bookmarks visible earlier than before (which seems like an improvement), so we can no longer incorrectly rely on the repo._bookmarks field being cached too long (IIUC). This patch makes it so we restore the temporarily hidden bookmarks only after the temporary bundle has been re-applied. It also adapts the code to update the repo.__bookmarks field using the pattern described in the aforementioned commit instead of writing directly to the fiels. I have spent many hours trying to understand what was going on here, but I still don't know if this can also happen without our custom server. So this patch unfortunately does not add any tests; I have only been able to test the fix using our Google-internal tests. Differential Revision: https://phab.mercurial-scm.org/D5398

narrow: drop obsolete support for old Mercurial Not needed since narrow is shipped with core. Differential Revision: https://phab.mercurial-scm.org/D5397

tests: test that narrow preserves bookmarks on widen When widening inserts older commits in the changelog, we have to preserve bookmarks so they are not removed by the call to repair.strip() we temporarily do. We didn't have any tests for that before. Differential Revision: https://phab.mercurial-scm.org/D5396

rebase: clarify that commits that become empty are skipped Our message just said that it created no changes, but we didn't explicitly say that that meant that the the commit was not rebased. It also wasn't clear why it created no changes, so this patch also clarifies that that was because the changes were already in the destination. Differential Revision: https://phab.mercurial-scm.org/D5395

rebase: format commit in usual way when explaining that it became empty We use the _ctxdesc() for other similar messages, so it seems we should use it here too. Differential Revision: https://phab.mercurial-scm.org/D5394

rust: propagate error of index_get_parents() properly Before, rustla_contains() would return 0 on error, and the exception would be cleared or noticed somewhere else. We need to propagate the error from AncestorsIterator up to the FFI surface.

rust: look up HgRevlogIndex_GetParents() from symbol table And removes the unused index_get_parents_checked() function. I expect the Index struct will be turned into a pyobject type, though I haven't written any PoC-level patches yet.

revlog: add public CPython function to get parent revisions Since this is a public function, it validates the input revision, and supports nullrev. index_get_parents_checked() will be replaced by this function.

revlog: rename indexType to HgRevlogIndex_Type as it's a global symbol It follows the CPython naming convention, but uses Hg instead of Py.

revlog: export symbol of indexType The idea is to wrap the index object with rust-cpython. I haven't tried it, but it should be doable. We'll probably need a better interface than raw function pointers to do more in Rust.

setup: fix path of rust source dependencies Appears that they were screwed up while moving the source files around. Since rust-direct-ffi sources are automatically added to the depends list, we only need to list hg-core sources.

rebase: remove now-unnecessary arguments to _abort() Since _abort() is now an instance method, we don't need to pass state that's available on "self". Differential Revision: https://phab.mercurial-scm.org/D5393

rebase: move abort() onto rebaseruntime The function depends on a lot of the state of rebaseruntime, so it makes sense for it to be an instance method. This will let us remove many of the arguments to the method. That will be done in a later patch. Differential Revision: https://phab.mercurial-scm.org/D5392

cleanupnodes: trust caller when "moves" is not None If "moves" (indicating how to move bookmarks) is None, we fill it out based on "replacements" (indicating which obsmarkers to add). If "moves" is not None, we would still add items based on "replacements". This makes it impossible to pass "moves={}" and not move bookmarks, which surprised me. The only caller that currently passes a value for "moves" was the rebase extension and there we were already adding bookmark moves corresponding to obsmarker additions, so it should not be impacted. Differential Revision: https://phab.mercurial-scm.org/D5391

shelve: change transaction description from "commit" to "shelve" "commit" was probably a copy&paste mistake. Differential Revision: https://phab.mercurial-scm.org/D5390

shelve: drop unnecessary backup of dirstate for phase-based case Regular shelve has a hack using an uncommitted transaction that's then aborted at the end of the operation. It preserves the dirstate across the abort, however, by saving a backup copy of it. Phase-based shelve instead commits the transaction, so the hack shouldn't be necessary there. Differential Revision: https://phab.mercurial-scm.org/D5389

tests: split test-shelve.t in two test-shelve.t dominated run time for all shelve tests. Before: # Ran 9 tests, 1 skipped, 0 failed. real 0m43.568s user 2m15.822s sys 0m40.857s After: # Ran 11 tests, 1 skipped, 0 failed. real 0m24.574s user 2m21.354s sys 0m40.435s Differential Revision: https://phab.mercurial-scm.org/D5388

remotefilelog: rely on progress helper for keeping track of position The progress helper class keeps track of its current position, so we don't need a "count" variable for that. Differential Revision: https://phab.mercurial-scm.org/D5386

remotefilelog: reduce use of "count" container We had already given the initial value of "count[0]" a name, so just use that when applicable. Differential Revision: https://phab.mercurial-scm.org/D5385

remotefilelog: replace a "a=[expr]; b=a[0]" by "b=expr; a = [b]" Simpler is better. Differential Revision: https://phab.mercurial-scm.org/D5384

remotefilelog: remove an unnecessary update of "count" container The "count" container is never used after this point, so there is no need to update it. Differential Revision: https://phab.mercurial-scm.org/D5383

remotefilelog: avoid temporarily using "count" variable as synonym for "total" The "count" variable is generally used for updating progress, but early in fileserverclient.request(), its used to mean the total count. We already have another "total" variable for that, so it seems much clearer to use that. Differential Revision: https://phab.mercurial-scm.org/D5382

remotefilelog: use progress helper in fileserverclient Differential Revision: https://phab.mercurial-scm.org/D5381

remotefilelog: use progress helper in shallowrepo Differential Revision: https://phab.mercurial-scm.org/D5380

remotefilelog: use progress helper in shallowbundle Differential Revision: https://phab.mercurial-scm.org/D5379

remotefilelog: use progress helper in repack Differential Revision: https://phab.mercurial-scm.org/D5378

remotefilelog: use progress helper in remotefilelogserver Differential Revision: https://phab.mercurial-scm.org/D5377

remotefilelog: use progress helper in basestore Differential Revision: https://phab.mercurial-scm.org/D5376

remotefilelog: use progress helper when analyzing repos Differential Revision: https://phab.mercurial-scm.org/D5375

fastannotate: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5374

debugbuildannotatecache: use progress helper Differential Revision: https://phab.mercurial-scm.org/D5373

upgrade: clarify "aggressivemergedelta" handling We rename "aggressivemergedelta" argument to "forceaggressivemergedelta". The previous argument naming could infer an absolute control on the behavior. However, the code show we respect the config option if set.

upgrade: test that fncache is valid after repository upgrade The behavior was correct. Now it is correct and tested.

tests: update `rm` invocation for new location of checklink This should fix the test on FreeBSD. Differential Revision: https://phab.mercurial-scm.org/D5387

bookflow: new extension for bookmark-based branching This extension should be helpful for feature branches - based workflows. At my company we first considered branches, but weren't sure about creating a lot of permanent objects. We tried bookmarks, but found some scenarios to be difficult to control. The main problem, was the active bookmark being moved on update. Disabling that made everything a lot more predictable. Bookmarks move on commit, and updating means switching between them. The extension also implements a few minor features to better guide the workflow: - hg bookmark NAME can be ambiguous (create or move), unlike hg branch. The extension requires -rev to move. - require an active bookmark on commit. - some bookmarks can be protected (like @), useful for teams, that require code reviews. - block creation of new branches. The initial implementation requires no changes in the core, but it does rely on some implementation details. I thought it may be useful to discuss the functionality first, and then focus making the code more robust. Differential Revision: https://phab.mercurial-scm.org/D4312

rust: make clean takes care of rust/target This isn't the prettiest way of doing it, but it doesn't require looking up cargo, or wondering whether that should be part of setup.py clean. Differential Revision: https://phab.mercurial-scm.org/D5369

rust: rename local variables in AncestorsIterator::next It was confusing to have p1 and parents.1 ; (p1, p2) is clearer. Differential Revision: https://phab.mercurial-scm.org/D5365

tests: stabilize test-inherit-mode.t on FreeBSD and macOS again This is the same fix as 90e26ef4cbb1, just repeated on the new file location. Differential Revision: https://phab.mercurial-scm.org/D5371

tests: drop redundant "#if execbit" from test-rebase-inmemory.t The whole file is already guarded by "#require execbit". Differential Revision: https://phab.mercurial-scm.org/D5366

narrow: don't resurrect old commits when narrowing (don't strip obsmarkers) If you have an old obsolescence-chain of commits that has been pruned and you narrow your repo so that some of those commits get stripped (because they affected the removed paths), then we would currently resurrect the commit that came before (along the obsmarker chain) the last stripped commit. That happens by the usual rules for obsmarker-stripping. However, it's quite surprising when it happens when you narrow your repo. This patch makes narrowing not strip obsmarkers. Differential Revision: https://phab.mercurial-scm.org/D5364

commandserver: turn server debug messages into logs They were ui.debug() just because commandserver.log() was noop if no client connected.

commandserver: add config knob for various logging options The default rotating options are copied from the blackbox extension.

commandserver: expand log path for convenience This allows us to set the log path relative to $XDG_RUNTIME_DIR, for instance. [cmdserver] log = $XDG_RUNTIME_DIR/chg/server.log

commandserver: switch logging facility to ui.log() interface The "pager subcommand" message is removed since ui isn't accessible there. I think that's okay as cmdtable[cmd]() will call attachio() and some debug message will be printed.

commandserver: install logger to record server events through canonical API The global commandserver.log() will be replaced with this.

commandserver: enable logging when server process started This allows us to keep track of server events before client connects to the server. Tests will be added later. Currently there's no log() call to check if things are working well.

test-commandserver: change way of triggering early crash Future patches will move the logging facility out of the server class, so cmdserver.log can't be (ab)used for this purpose. Instead, let's hook the factory function to raise exception.

loggingutil: add basic logger backends These classes will be used in command server. They are similar to the blackboxlogger, but it can't be factored out since the blackbox is so tightly coupled with a repo object.

hgweb: register web.comparisoncontext to the config table This was caught in some server side logging added to debug py3 issues.

merge with stable

Added signature for changeset 1c8c54cf9725

Added tag 4.8.1 for changeset 1c8c54cf9725

rebase: fix path auditing to audit path relative to repo root (issue5818) Before this patch, when rebasing a file called "foo/bar", we would check e.g. if "/foo" (i.e. rooted at the file system root) was a symlink. Differential Revision: https://phab.mercurial-scm.org/D5361

tests: show bad path auditing in in-memory rebase Thanks to Yuya for providing this test case in https://bz.mercurial-scm.org/show_bug.cgi?id=5818. Differential Revision: https://phab.mercurial-scm.org/D5368

tests: add a missing "cd .." to test-rebase-inmemory.t Differential Revision: https://phab.mercurial-scm.org/D5367

rust: fix possible out-of-bounds read through index_get_parents() index_get_parents() is an internal function, which doesn't check if the specified rev is valid. If rustlazyancestors() were instantiated with an invalid stoprev, it would access to invalid memory region. This is NOT a security fix as there's no Python code triggering the bug, but included in this series to not give a notion about the memory issue fixed by the previous patch.

revlog: fix out-of-bounds access by negative parents read from revlog (SEC) 82d6a35cf432 wasn't enough. Several callers don't check negative revisions but for -1 (nullrev), which would directly lead to out-of-bounds read, and buffer overflow could follow. RCE might be doable with carefully crafted revlog structure, though I don't think this would be useful attack surface.

rust: peek_mut optim for lazy ancestors This is one of the two optimizations that are also present in the Python code: replacing pairs of pop/push on the BinaryHeap by single updates, hence having it under the hood maintain its consistency (sift) only once. On Mozilla central, the measured gain (see details below) is around 7%. Creating the PeekMut object by calling peek_mut() right away instead of peek() first is less efficient (gain is only 4%, stats not included). Our interpretation is that its creation has a cost which is vasted in the cases where it ends by droping the value (Peekmut::pop() just does self.heap.pop() anyway). On the other hand, the immutable peek() is very fast: it's just taking a reference in the underlying vector. The Python version still has another optimization: if parent(current) == current-1, then the heap doesn't need to maintain its consistency, since we already know that it's bigger than all the others in the heap. Rust's BinaryHeap doesn't allow us to mutate its biggest element with no housekeeping, but we tried it anyway, with a copy of the BinaryHeap implementation with a dedicaded added method: it's not worth the technical debt in our opinion (we measured only a further 1.6% improvement). One possible explanation would be that the sift is really fast anyway in that case, whereas it's not in the case of Python, because it's at least partly done in slow Python code. Still it's possible that replacing BinaryHeap by something more dedicated to discrete ordered types could be faster. Measurements on mozilla-central: Three runs of 'hg perfancestors' on the parent changeset: Moyenne des médianes: 0.100587 ! wall 0.100062 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.135804 comb 0.130000 user 0.130000 sys 0.000000 (max of 98) ! wall 0.102864 comb 0.102755 user 0.099286 sys 0.003469 (avg of 98) ! wall 0.101486 comb 0.110000 user 0.110000 sys 0.000000 (median of 98) ! wall 0.096804 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.132235 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.100258 comb 0.100300 user 0.096000 sys 0.004300 (avg of 100) ! wall 0.098384 comb 0.100000 user 0.100000 sys 0.000000 (median of 100) ! wall 0.099925 comb 0.100000 user 0.100000 sys 0.000000 (best of 98) ! wall 0.133518 comb 0.140000 user 0.130000 sys 0.010000 (max of 98) ! wall 0.102381 comb 0.102449 user 0.098265 sys 0.004184 (avg of 98) ! wall 0.101891 comb 0.090000 user 0.090000 sys 0.000000 (median of 98) Mean of the medians: 0.100587 On the present changeset: ! wall 0.091344 comb 0.090000 user 0.090000 sys 0.000000 (best of 100) ! wall 0.122728 comb 0.120000 user 0.110000 sys 0.010000 (max of 100) ! wall 0.093268 comb 0.093300 user 0.089300 sys 0.004000 (avg of 100) ! wall 0.092567 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.093294 comb 0.080000 user 0.080000 sys 0.000000 (best of 100) ! wall 0.144887 comb 0.150000 user 0.140000 sys 0.010000 (max of 100) ! wall 0.097708 comb 0.097700 user 0.093400 sys 0.004300 (avg of 100) ! wall 0.094980 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) ! wall 0.091262 comb 0.090000 user 0.080000 sys 0.010000 (best of 100) ! wall 0.123772 comb 0.130000 user 0.120000 sys 0.010000 (max of 100) ! wall 0.093188 comb 0.093200 user 0.089300 sys 0.003900 (avg of 100) ! wall 0.092364 comb 0.100000 user 0.090000 sys 0.010000 (median of 100) Mean of the medians is 0.0933 Differential Revision: https://phab.mercurial-scm.org/D5358