worker: manually buffer reads from pickle stream My previous fix (D8051, cb52e619c99e, which added Python's built-in buffering to the pickle stream) has the problem that the selector will ignore the buffer. When multiple pickled objects are read from the pipe into the buffer at once, only one object will be loaded. This can repeat until the buffer is full and delays the processing of completed items until the worker exits, at which point the pipe is always considered readable and all remaining items are processed. This changeset reverts D8051, removing the buffer again. Instead, on Python 3 only, we use a wrapper to modify the "read" provided to the Unpickler to behave more like a buffered read. We never read more bytes from the pipe than the Unpickler requests, so the selector behaves as expected. Also add a test case for "pickle data was truncated" issue. https://phab.mercurial-scm.org/D8051#119193 Differential Revision: https://phab.mercurial-scm.org/D8076

py3: __repr__ needs to return str, not bytes Differential Revision: https://phab.mercurial-scm.org/D8089

config: also respect HGRCSKIPREPO in the zeroconf extension Differential Revision: https://phab.mercurial-scm.org/D8075

config: also respect HGRCSKIPREPO in hgwebdir_mod Differential Revision: https://phab.mercurial-scm.org/D8074

config: also respect HGRCSKIPREPO in `dispatch._getlocal` For some reason, we are also reading the local config in that function. Differential Revision: https://phab.mercurial-scm.org/D8073

config: add a function in `rcutil` to abstract HGRCSKIPREPO We wil need to respect this environment variable in more place. Differential Revision: https://phab.mercurial-scm.org/D8072

packaging: make the path to Win32 requirements absolute when building WiX Otherwise this broke automation when not launched from `contrib/packaging`. Differential Revision: https://phab.mercurial-scm.org/D8068

resourceutil: blacken

merge with stable

rebase: abort if the user tries to rebase the working copy I think it's more correct to treat `hg rebase -r 'wdir()' -d foo` as `hg co -m foo`, but I'm instead making it error out. That's partly because it's probably what the user wanted (in the case I heard from a user, they had done `hg rebase -s f` where `f` resolved to `wdir()`) and partly because I don't want to think about more complicated cases where the user specifies the working copy together with other commits. Differential Revision: https://phab.mercurial-scm.org/D8057

tests: add tests for rebasing wdir() revision Differential Revision: https://phab.mercurial-scm.org/D8056

merge: when rename was made on both sides, use ancestor as merge base When both sides of a merge have renamed a file to the same place, we would treat that as a "both created" action in merge.py. That means that we'd use an empty diffbase. It seems better to use the copy source as diffbase. That can be done by simply dropping code that prevented us from doing that. I think I did it that way in 57203e0210f8 (copies: calculate mergecopies() based on pathcopies(), 2019-04-11) only to preserve the existing behavior. I also suspect it was just an accident that it behaved that way before that commit. Note that until fa9ad1da2e77 (merge: start using the per-side copy dicts, 2020-01-23), it was non-deterministic (depending on iteration order of the `allsources` set in `copies._fullcopytracing()`) which source was used in the affected test case in test-rename-merge1.t. We could easily have fixed that by sorting them, but now we can instead detect the case (the TODO added in the previous patch). Differential Revision: https://phab.mercurial-scm.org/D7974

absorb: graduate -i flag from experimental The interactive mode seems to work well. I have previously thought that `-i` should be what `-e` does, but the current behavior matches what other `-i` flags do (select a subset of the hunks), so I think that is what we want. Differential Revision: https://phab.mercurial-scm.org/D8055

rust-cpython: remove PySharedRefCell and its companion structs Also updates py_shared_iterator!() documentation accordingly.

rust-cpython: switch to upstreamed version of PySharedRefCell Our PyLeaked is identical to cpython::UnsafePyLeaked. I've renamed it because it provides mostly unsafe functions.

rust-cpython: rename inner_shared() to inner() The "shared" accessor will be automatically generated, and will have the same name as the data itself.

rust-cpython: use PyList.insert() instead of .insert_item() Silences the deprecated warning. https://github.com/dgrunwald/rust-cpython/commit/e8cbe864841714c5555db8c90e057bd11e360c7f

rust-cpython: bump cpython to 0.4 to switch to upstreamed PySharedRef

rust: update dependencies For no particular reason, but just because I'll bump the rust-cpython version.

Added signature for changeset 7f5410dfc8a6

Added tag 5.3 for changeset 7f5410dfc8a6

rust-dirstatemap: add missing @propertycache While investigating a regression on `hg update` performance introduced by the Rust `dirstatemap`, two missing `@propertycache` were identified when comparing against the Python implementation. This adds back the first one, that has no observable impact on behavior. The second one (`nonnormalset`) is going to be more involved, as the caching has to be done from the Rust side of things. Differential Revision: https://phab.mercurial-scm.org/D8047

worker: Use buffered input from the pickle stream On Python 3, "pickle.load" will raise an exception ("_pickle.UnpicklingError: pickle data was truncated") when it gets a short read, i.e. it receives fewer bytes than it requested. On our build machine, Mercurial seems to frequently hit this problem while updating a mozilla-central clone iff it gets scheduled in batch mode. It is easy to trigger with: #wipe the workdir rm -rf * hg update null chrt -b 0 hg update default I've also written the following program, which demonstrates the core problem: from __future__ import print_function import io import os import pickle import time obj = {"a": 1, "b": 2} obj_data = pickle.dumps(obj) assert len(obj_data) > 10 rfd, wfd = os.pipe() pid = os.fork() if pid == 0: os.close(rfd) for _ in range(4): time.sleep(0.5) print("First write") os.write(wfd, obj_data[:10]) time.sleep(0.5) print("Second write") os.write(wfd, obj_data[10:]) os._exit(0) try: os.close(wfd) rfile = os.fdopen(rfd, "rb", 0) print("Reading") while True: try: obj_copy = pickle.load(rfile) assert obj == obj_copy except EOFError: break print("Success") finally: os.kill(pid, 15) The program reliably fails with Python 3.8 and succeeds with Python 2.7. Providing the unpickler with a buffered reader fixes the issue, so let "os.fdopen" create one. https://bugzilla.mozilla.org/show_bug.cgi?id=1604486 Differential Revision: https://phab.mercurial-scm.org/D8051

packaging: lowercase the `contrib` and `templates` directories with Inno I have no idea why these (and `contrib/vim`) were leading with uppercase with Inno, but not WiX. It probably doesn't matter too much, but might be a problem with `templates` if the user enabled case sensitivity on NTFS. Differential Revision: https://phab.mercurial-scm.org/D8063