grep: warn on censored revisions instead of erroring out We need most of the grep logic to go through in case we encounter a censored revision, so we just return a None body for a censored node, and we stop just short of trying to record matches with the contents of that censored body. The other parts such as recording that the censored file has been considered at this revision needs to go into the proper dicts. I have also gotten weary of all the abbreviations, so while I did a small refactor to move the file-data-getting operation into a common function, I also expanded the abbreviations of the relevant variables within this little function. Hopefully some day this helps someone figure out what all the abbreviations mean. Although the censoring docs currently state that some commands error out or are ignored depending on the `censor.policy` config, I cannot see a benefit for grep to ever stop dead in its tracks when a censored revision is encountered. I will also amend the docs to indicate that some commands, such as grep, unconditionally ignore censored revisions.

py3: enable legacy fs encoding to fix filename compatibility on Windows This patch is untested. I just followed the instruction: https://docs.python.org/3/whatsnew/3.6.html#pep-529-change-windows-filesystem-encoding-to-utf-8

rust-cpython: run cargo fmt

rust-cpython: remove useless PyResult<> from leak_immutable() The caller should know if the shared data is mutably borrowed or not.

rust-cpython: remove useless PyRefMut wrapper

rust-cpython: drop manual management of mutably_borrowed RefCell::borrow() should guarantee there's no mutable borrow. As a follow up, maybe PySharedState can be a pure data structure + function. Most ref-sharing business has already been moved to PySharedRef* and PyLeaked*.

rust-cpython: leverage RefCell::borrow() to guarantee there's no mutable ref Since the underlying value can't be mutably borrowed by PyLeaked, we don't have to manage yet another mutably-borrowed state. We can just rely on the RefCell implementation. Maybe we can add try_leak_immutable(), but this patch doesn't in order to keep the patch series not too long.

rust-cpython: remove useless Option<$leaked> from py_shared_iterator We no longer need to carefully drop the iterator when it's consumed. Mutation is allowed even if the iterator exists. There's a minor behavior change: next(iter) may return/raise something other than StopIteration if it's called after the iterator has been fully consumed, and if the Rust object isn't a FusedIterator.

rust-cpython: allow mutation unless leaked reference is borrowed In other words, mutation is allowed while a Python iterator holding PyLeaked exists. The iterator will be invalidated instead. We still need a borrow_count to prevent mutation while leaked data is dereferenced in Rust world, but most leak_count business is superseded by the generation counter. decrease_leak_count(py, true) will be removed soon.

rust-cpython: add generation counter to leaked reference This counter increments on borrow_mut() to invalidate existing leaked references. This is modeled after the iterator invalidation in Python. The other checks will be adjusted by the subsequent patches.

rust-cpython: add stub wrapper that'll prevent leaked data from being mutated In order to allow mutation of PySharedRefCell value while PyLeaked reference exists, we need yet another "borrow" scope where mutation is prohibited. try_borrow<'a> and try_borrow_mut<'a> defines the "borrow" scope <'a>. The subsequent patches will implement leak counter based on this scope. PyLeakedRef<T> and PyLeakedRefMut<T> could be unified to PyLeakedRef<&T> and PyLeakedRef<&mut T> respectively, but I didn't do that since it seemed a bit weird that deref_mut() would return a mutable reference to an immutable reference.

rust-cpython: rename PyLeakedRef to PyLeaked This series will make PyLeaked* behave more like a Python iterator, which means mutation of the owner object will be allowed and the leaked reference (i.e. the iterator) will be invalidated instead. I'll add PyLeakedRef/PyLeakedRefMut structs which will represent a "borrowed" state, and prevent the underlying value from being mutably borrowed while the leaked reference is in use: let shared = self.inner_shared(py); let leaked = shared.leak_immutable(); { let leaked_ref: PyLeakedRef<_> = leaked.borrow(py); shared.borrow_mut(); // panics since the underlying value is borrowed } shared.borrow_mut(); // allowed The relation between PyLeaked* structs is quite similar to RefCell/Ref/RefMut, but the implementation can't be reused because the borrowing state will have to be shared across objects having no lifetime relation. PyLeaked isn't named as PyLeakedCell since it isn't actually a cell in that leaked.borrow_mut() will require &mut self.

fix: fix grammar/typos in hg help -e fix

progress: flush stderr after clearing On python3, ui.stderr is buffered, it seems, so we need to flush it to actually get the progress bar off the screen. This is important since ui.write() will call into progbar.clear() if it thinks there's a progress bar on the screen, with the intent that the next thing it outputs is at the beginning of the line (instead of at the end of the progress bar line). Without the flush, we buffer up the clearing of the screen, and we get some really weird/corrupt output. Differential Revision: https://phab.mercurial-scm.org/D7784

subrepo: fix a crash when archiving an svn or git subrepo Only hgsubrepos have a repository attribute. This is pretty hacky, but probably the best we can do on stable. Pushing the lfstatus check down into the wrapper for hgsubrepo (and dropping the check for lfstatus at the top of `hgsubrepoarchive()`) resulted in various test failures because: 1) hgsubrepoarchive isn't returning the number of files archived at the bottom, resulting in an error about += NoneType 2) These copypasta archive wrappers don't use progress bars 3) Largefiles are *not* currently archived when using extdiff (68822b7cdd01), but pushing this context manager down into the subrepo resulted in it apparently doing so (as evidenced by progress bars being dropped) The other uses of `lfstatus()` are not in the substate processing loop, so they shouldn't be an issue. I initially put testcases in this test for largefiles-{on,off}, and it flagged a bunch of exit code differences for `cat` and `diff`, so I backed that off. Differential Revision: https://phab.mercurial-scm.org/D7714

Added signature for changeset b4c82b704180

Added tag 5.2.2 for changeset b4c82b704180

py3: force bytestr conversion of "reason" in scmutil.callcatch() For instance, reason may be an InvalidURL as shown in added test.

rust-cpython: do not convert warning pattern to utf-8 bytes On Unix, both Rust Path and Mercurial expect a locale-dependent bytes, and we don't support Windows yet. Differential Revision: https://phab.mercurial-scm.org/D7614

rust-cpython: import utils::files::* function at module level IIRC, it's common in Rust to call functions with the module prefix. Differential Revision: https://phab.mercurial-scm.org/D7613

py3: send bytes from Rust-created warning patterns Python code expects bytes in both Python 2 and Python 3, so we should send bytes. Differential Revision: https://phab.mercurial-scm.org/D7612

py3: pass bytes to `configint` and `configbool` Both functions require bytes, even in Python 3. Differential Revision: https://phab.mercurial-scm.org/D7611

rust-threads: force Rayon to respect the worker count in config As per the inline comment, this is a workaround because Rust code does not yet know how to read config files. Differential Revision: https://phab.mercurial-scm.org/D7610

remotefilelog: remove the `ensurestart` usage The option does not seems to do what we want since it does not wait on the background process actually starting (eg: taking the lock). And the race have been dealt with another way anyway. This effectively back out changeset 2c74337e6483. Differential Revision: https://phab.mercurial-scm.org/D7590

remotefilelog: remove most of the debugwait* command in the tests Since we use a dedicated developer config, we do not need them anymore. Differential Revision: https://phab.mercurial-scm.org/D7589

remotefilelog: use the correct redirection in tests The test were using '2>%1' which just redirect stderr in the "%1" file. The intend was '2>&1' which redirect stderr on stdin. This impact tests because some commit were picking up this '%1' files. Differential Revision: https://phab.mercurial-scm.org/D7588

remotefilelog: remove most sleep in tests Now that we removed most of the races, we can remove the infamous sleeps. Differential Revision: https://phab.mercurial-scm.org/D7587

remotefilelog: have command wait for background task in the tests We leverage the feature introduced in the two previous commit to have the main command wait for the background task to be finished before returning. All test still pass and instability seems to have gone away. Most of the sleep and explicit wait are now useless and will be cleaned up in later changesets. Differential Revision: https://phab.mercurial-scm.org/D7586

remotefilelog: add a developer option to wait for background processes In order to block the main command on the subprocess exiting, we ensure the repo's ui object will call the subprocess.wait() method to ensure the top-level hg process doesn't exit until all background processes have also done so. Currently, in the tests, most operation spawning background process as followed by commands waiting for these operations to complete. However this waiting is racy. First because it seems like we can start waiting before the background operation actually start, in which case it is prematurely detected as "done". Second, because some commands may spawn multiple background operation for the same operation (eg: rebase can apparently trigger multiple prefetch). The current approach could be updated to maybe handle the first issue, but the second one will never be properly handled. In most case, we do not care that the bg process keep running after the command end. (Since we explicitly wait for them to end before doing anything else). So we add an option to wait on the background process before exiting the command. We'll put it in use in the next changeset. Differential Revision: https://phab.mercurial-scm.org/D7585

procutil: add a option to not fully detach background process When writing test, it is useful to make code pretending to spawn a fully detached process while still actually waiting for it to finish. We add the option to `procutil.runbgcommand`, to do so. We will use this in remote-filelog tests. Differential Revision: https://phab.mercurial-scm.org/D7584

Added signature for changeset a50fecefa691

Added tag 5.2.1 for changeset a50fecefa691