censor: rename `rl` to `revlog` in the main function Now that the bulk of the code moved to smaller function we are less restricted on line length and we can use more explicite naming. Differential Revision: https://phab.mercurial-scm.org/D10902

censor: extract the part about creating and opening new files in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. The last part is the file copying and opening logic. It now have its own function. Differential Revision: https://phab.mercurial-scm.org/D10901

censor: extract the part about recomputing delta in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. It is now the turn of the logic that recompute new delta to replace the one based on the revision that are going away. Differential Revision: https://phab.mercurial-scm.org/D10900

censor: extract the part about writing the other revision in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. We continue with a small function that add a non-censored revision back to the revlog. Differential Revision: https://phab.mercurial-scm.org/D10899

censor: extract the part about writing the censored revision in a function The v2_censor function is huge, now that its content has settled a bit it is a good time to split individual part inside dedicated function. We start with a small function that process the censored revision. Differential Revision: https://phab.mercurial-scm.org/D10898

censor: put the tuple of open files in an explicit variable This will be simpler to pass these file around in future changesets. Differential Revision: https://phab.mercurial-scm.org/D10897

revlog: rewrite `censors.py` to `rewrite.py` The logic we use for censoring revision will be mostly common with the one we needs for stripping. So we rename the module to `rewrite` to better match its future content. Differential Revision: https://phab.mercurial-scm.org/D10896

copies: Keep changelog sidedata file open during copy tracing Instead of having a callback that opens and closes that file many times, a add and use a context manager method on the `revlog` class that keeps files open for its duration. Differential Revision: https://phab.mercurial-scm.org/D10888

purge: restore the `testedwith` declaration Otherwise, this extension gets highlighted as a 3rd party extension when an exception occurs. Differential Revision: https://phab.mercurial-scm.org/D10895

rust: remove dead utils module This was introduced a while back while not being imported anywhere in the module system. How pertinent this Facebook module was to the codebase was debatable at the time, nowadays with the Rust VFS layer having laid its foundations, it is clear that we should drop it. Differential Revision: https://phab.mercurial-scm.org/D10890

rebase: use str-keyed opts in remaining places Differential Revision: https://phab.mercurial-scm.org/D10892

rebase: keep str-keyed opts long enough to make `action` a str This is just another little py3 cleanup. Differential Revision: https://phab.mercurial-scm.org/D10891

benchmarks: restore `output` variable lost in D10884 Noticed by pyflakes (which I didn't have installed for Python 3.9 when I sent D10884). Differential Revision: https://phab.mercurial-scm.org/D10894

censor: implement censoring for revlogv2 It is a bit verbose and rough, but it works. Most of that logic can be common for `stripping`, so we can expect more refactoring of that code to accommodate both needs. However I wanted to keep this changesets "simple enough" and before moving forward. We also need to properly delete the older index/data/sidedata file, but this has implication for streaming clone and transaction, so this will come later. Differential Revision: https://phab.mercurial-scm.org/D10869

deltas: at a `target_rev` parameter to finddeltainfo Otherwise, recomputing a delta for a revision might result in a delta against a later revision or a full snapshot thinking we are appending a new revision. We will make use of this during censoring (and later, stripping). Differential Revision: https://phab.mercurial-scm.org/D10887

revlog: list older-but-still-around file in `files` These older files will eventually be removed, but we have to take them in account until they are. So we update `files` to keep fncache happy. Differential Revision: https://phab.mercurial-scm.org/D10868

revlog: add docket method to request new content files If we want to write content into new files, we need to be able to ask for them. Differential Revision: https://phab.mercurial-scm.org/D10867

revlog: add a way to keep track of older uids in the docket When the revlog content is rewritten, we will use new files, to avoid truncating the previous ones. We need some way to keep track of the older files before we clean them up. Differential Revision: https://phab.mercurial-scm.org/D10866

revlog: factor the logic to determine the delta compression out This make the logic clearly isolated and documented and it will help use to reuse it during censors/strip. Differential Revision: https://phab.mercurial-scm.org/D10889

nodemap: fix some comment formatting Black seems unhappy about it. Differential Revision: https://phab.mercurial-scm.org/D10865

revlogv2: fix `hg verify` with revlog v2 We added a bunch of new files but we never teached some internal about it. This is now fixed. Differential Revision: https://phab.mercurial-scm.org/D10858

revlogv2: also test that local clone works This is now the case so lets add a test. Differential Revision: https://phab.mercurial-scm.org/D10857

clone: also report the bookmark file as copied This is a small UI adjustement, but this is easy enough to do. Differential Revision: https://phab.mercurial-scm.org/D10856

clone: reuse the stream clone logic for local clone Streaming clone and local (non `--pull`) clone do mostly the same thing, however they were using different logic to do so. This means the logic frequently went out of sync and that new case had to be dealt with twice. This is fragile and anoying. So we replace this with a re-use of the logic we use for streaming clone. I can see various test changes: - a more precise progress output, - armless fncache loading during clone, - fncache is no longer hardlinked (since we write it by hand). I am not reinstalling the `reposimplestore` specific output, as far as I understand this variant have been broken for years. Differential Revision: https://phab.mercurial-scm.org/D10855

copyfiles: add a way to relax the file system checking for hardlink This is critical for transaction file, less for hardlink/copy clone as we are about to do. Since `pure` build does not have a `getfstype` implementation this would disable hardlink clone for all pure build. So we add a parameter to control that extra check. Differential Revision: https://phab.mercurial-scm.org/D10854

copyfile: add a option callback for failed hardlinking Local clone, adjust its UI depending on the success of using hardlinking, so we add a small callback making it possible for `copyfile` to signal if the requested hardlinking failed. Differential Revision: https://phab.mercurial-scm.org/D10853

streamingclone: extract the scanning part from the generation part We will reuse the scanning part for local clone, so we need it in a dedicated function. Differential Revision: https://phab.mercurial-scm.org/D10852

vfs: add a `register_file` method on the vfs class This is used by the fncache vfs to register new file. Until now, `fncache` have been doing this "automatically" by monitoring write pattern. However this is fragile and when we copy files in place by other means, we need something more robuts. So we add an explicit method to do so. Differential Revision: https://phab.mercurial-scm.org/D10851

clone: use "official" API to create local clone destination This make sure we have a properly created, fully functional repository early. This will be useful to simply the hardlink/copy phases of the local clone to make it share more of its logic with the similar "stream" cloning. This has a minor impact of the test and the resulting repository has is better initialized (eg: the `wcache` directory is pre-created.) Differential Revision: https://phab.mercurial-scm.org/D10850

localrepo: introduce a clone_requirements function This function take a source repository and return a relevant set of requirements that should be used by a copy clone. This will help make the creation of the destination repository during copy clone simpler. Differential Revision: https://phab.mercurial-scm.org/D10849

createrepository: allow to directly pass the target requirements This is useful when doing a local clone that copies store contents, it will requires the destination to use the very same store requirements so directly providing them will be simpler and safer Differential Revision: https://phab.mercurial-scm.org/D10848

narrowbundle: use new context manager for silencing the ui Same reasoning as the previous change. This affects a few tests because of the hack from d7304434390f (changegroup: move message about added changes to transaction summary, 2019-09-08). Differential Revision: https://phab.mercurial-scm.org/D10886

debugbackupbundle: use new context manager for silencing the ui A difference between setting `ui.quiet` and using `ui.silent()` is that the latter also silences `ui.write()` calls. That's practically always what one wants, including here, I think. Differential Revision: https://phab.mercurial-scm.org/D10885

ui: add a context manager for silencing the ui (pushbuffer+popbuffer) We often silence the ui by calling `ui.pushbuffer()` followed (a later in the code) by `ui.popbuffer()`. These places can be identified by the fact that they ignore the output returned from `ui.popbuffer()`. Let's create a context manager for these cases, to avoid repetition, and to avoid accidentally leaving the ui silent on exceptions. I deliberately called the new function `silent()` instead of `buffered()`, because it's just an implementation detail that it uses `pushbuffer()` and `popbuffer()`. We could later optimize it to not buffer the output. Differential Revision: https://phab.mercurial-scm.org/D10884

shelve: don't swallow errors that happen when reverting the working copy Same reasoning as the parent commit. The code in question here comes from fb8065de47b0 (unshelve: silence internal revert, 2014-08-15), which says that it was to prepare for some upcoming revert changes. I suppose those changes have happened by now but we fixed something about it, because I don't see any unwanted changes in tests (no changes at all in tests, in fact). Differential Revision: https://phab.mercurial-scm.org/D10883

histedit: don't swallow errors that happen when updating the working copy Commit 34165875fa5d (histedit: limit updated and merging output to important updates, 2016-02-03) added a call to `ui.pushbuffer(labeled=True, errors=True)`. I don't know if there was a reason for the `errors=True` (nor the `labeled=True` bit, which I removed in the previous commit). It seems bad to just discard the errors, so let's not do that. By not doing that, we can also remove a workaround for the lack of errors from `test-removeemptydirs.t`. Differential Revision: https://phab.mercurial-scm.org/D10882

histedit: don't record labels for output that we will discard Differential Revision: https://phab.mercurial-scm.org/D10881

dog-raft: work with str-keyed opts throughout Differential Revision: https://phab.mercurial-scm.org/D10864

cmdutil: make resolvecommitoptions() work on str-keyed opts As with `checknotesize()`, I also changed to snake_case while at it, to help extensions a little. Differential Revision: https://phab.mercurial-scm.org/D10863

cmdutil: make amend() take str-keyed opts I couldn't think of a reasonable change to the signature here, so unfortunately it keeps the existing one. Differential Revision: https://phab.mercurial-scm.org/D10862

commit: keep opts dict str-keyed a bit longer Differential Revision: https://phab.mercurial-scm.org/D10861

dog-raft: keep opts dict str-keyed a bit longer Differential Revision: https://phab.mercurial-scm.org/D10860

cmdutil: make checknotesize() work on str-keyed opts Since this patch changes the behavior of the function, I consider it a bonus if the signature also changes so that extensions can detect which version they're dealing with (for the record, I don't have any such extensions). Therefore, I also dropped an unused `ui` argument and renamed the function to use snake_case. Differential Revision: https://phab.mercurial-scm.org/D10859

amend: work mostly with str-type **opts for simplicity Differential Revision: https://phab.mercurial-scm.org/D10880

revlog: use file read caching for sidedata The previous changeset extracted this caching logic from the revlog class into a new class. Adding a new instance of that class allows using the same logic for side data. Differential Revision: https://phab.mercurial-scm.org/D10879

revlog: Extract low-level random-access file read caching logic The `revlog` class does many things, among which fulfilling requests for arbitrary byte slices from the revlog "data file" by reading a larger chunk and caching it in memory, in order to reduce the number of system calls. This extracts that logic into a new class, so that it may later also be used for the side-data file (with another instance of that class). The copyright notice of the new file does not include a date or author name since such information tend not to be kept up-to-date: https://www.linuxfoundation.org/en/blog/copyright-notices-in-open-source-software-projects/ Differential Revision: https://phab.mercurial-scm.org/D10878

typing: disable warnings for a couple of Windows specific attributes Differential Revision: https://phab.mercurial-scm.org/D10870

typing: suppress a false error in mercurial/revlogutils/docket.py on py2 `ord()` wants bytes or str on py3, so I'm guessing it got confused by passing a single byte instead of a one byte string. But this seems to work on 2.7.18 anyway. Differential Revision: https://phab.mercurial-scm.org/D10876

transaction: simplify `undo.backupfiles` file creation with a variable This make the code shorter and clearer. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10846

transaction: narrow the error filtering when failing to rename undo file Having inconsistent "undo" files can be quite serious so we narrow the error filtering to the intent that the comment explain. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10845

transaction: clarify a conditionnal about version check Let us move the short branch early. Differential Revision: https://phab.mercurial-scm.org/D10844

transaction: extract message about different version in a constants This make the code a bit simpler and clearer. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10843

transaction: explain why some recovery failed Right now we issue a message about failing to recover some file, but not why. It seems useful to add some information about that. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10842

copyfiles: deal with existing file when hardlinking If the hardlinking fails, we fallback to `shutil.copy`, but do not consider future hardlinking doomed. This is a slight improvement from the current situation, we still avoid hardliking in a case we might be able to do it. However this does not have an impact of the rest of the operation. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10841

clone: cleanup the "cleanup dir" logic used during local clone With the previous code, there was a small windows were we could delete too much data. This should no longer be the case. (This is an opportunity improvement while looking at something next to that.) Differential Revision: https://phab.mercurial-scm.org/D10840

config: add an experimental option to list all known config That option is not ready for prime-time, hence the `exp-` prefix. However, this is a good base to start going toward completion. This is also quite useful for developer on its own for now. Differential Revision: https://phab.mercurial-scm.org/D10356

revlog: byteify a few error strings Differential Revision: https://phab.mercurial-scm.org/D10875

revlog: fix a typo closing the wrong file Caught by pytype as possibly being None, but the file is closed in the previous conditional. Differential Revision: https://phab.mercurial-scm.org/D10874

py3: byteify the version string passed to the deprecation warning method The other callers were already correct. Differential Revision: https://phab.mercurial-scm.org/D10873

urlutil: byteify several localized messages Flagged by pytype. Differential Revision: https://phab.mercurial-scm.org/D10872

urlutil: use the deprecation warning function on the `ui` object PyCharm flagged this as not existing, and I suspect a copy/paste error. Differential Revision: https://phab.mercurial-scm.org/D10871

dirstate-v2: Store a hash of ignore patterns (.hgignore) Later, this help extend `read_dir` caching to directories that contain ignored files (but no unknown files). Such cache must be invalidated when ignore patterns change since a formerly-ignored file might become unknown. This helps the default configuration of `hg status` where unknown files must be listed, but ignored files are not. Differential Revision: https://phab.mercurial-scm.org/D10836

rhg: propogate error coming from HgError::Abort to CommandError Differential Revision: https://phab.mercurial-scm.org/D10839

rhg: add exit code to HgError::Abort() My previous attempts to have rhg end with correct exit code was more of bug hunting. I found cases which were failing and fixed them. But as one might expect, more tests started failing. Let's add exit code `HgError::Abort()` and make it users explicitly tell what exit code they want. Differential Revision: https://phab.mercurial-scm.org/D10838

rhg: split non_repo_config and `--config` loading in different functions This will help us in better handling of error caused when trying to load `--config` values. Differential Revision: https://phab.mercurial-scm.org/D10837

rhg: look for repository in ancestors also instead of cwd only Last patch introduced config reading at startup to parse value of `--repository` flag. However, that patch only tried to check for current repository at current working directory and not it's ancestors. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D10767

rhg: read [paths] for `--repository` value hg parses `-R` and `--repository` CLI arguments "early" in order to know which local repository to load config from. (Config can then affect whether or how to fall back.) The value of of those arguments can be not only a filesystem path, but also an alias configured in the `[paths]` section. This part was missing in rhg and this patch implements that. The current patch still lacks functionality to read config of current repository if we are not at root of repo. That will be fixed in upcoming patches. A new crate `home` is added to get path of home directory. Differential Revision: https://phab.mercurial-scm.org/D10296

merge: with stable

revlog: allow to pass an existing docket to `_loadindex()` This will be when switching undelying files, for examples during censors and strip operation with rvlog-v2. Differential Revision: https://phab.mercurial-scm.org/D10800

revlog: add a ways to blacklist some revision when searching for a delta This will be useful to recompute appropriate deltas one the fly during censor/strip operation with revlog-v2. Differential Revision: https://phab.mercurial-scm.org/D10799

util: add `nb_bytes` argument to `copyfile` to partially copy a file When set, this allow to copy only the first `nb_bytes` of a file. This will be useful for censor/strip operation with revlogv2. Differential Revision: https://phab.mercurial-scm.org/D10798

revlog: move `revisioninfo` in `revlogutils` We will need it in other utility module. So lets extract it from `revlog.py`, the module is too large already anyway. Differential Revision: https://phab.mercurial-scm.org/D10797

revlog: use the `entry` function in bundlerepo We can use some of the default value again! Differential Revision: https://phab.mercurial-scm.org/D10796

revlog: use entry in revlogv0.py This make things clearer and we can reply on some of the default value directly. Differential Revision: https://phab.mercurial-scm.org/D10795

revlog: use the new `entry` function in revlog.py This just make the construction of the tuple clearer. Differential Revision: https://phab.mercurial-scm.org/D10794

revlog: add a function to build index entry tuple Keeping index entry as tuple make sense for performance reason, however it does not means we need to manually build that tuple for all piece of python code that are not performance critical. So we add a nice function responsible to build the tuple using argument using explicit keyword argument. Differential Revision: https://phab.mercurial-scm.org/D10793

revlog: move `offset_type` to `revlogutils` This multiple module are using this so it make sense to move it at the utility level. Differential Revision: https://phab.mercurial-scm.org/D10792

revlog: move entry documentation alongside new related constants Accessing individual index-entry element is usually done using integer directly. This is presumably for "performance reasons". However as the index entry gain more and more element it seems useful to get the option to use symbolic constant to access item, for both clarify and versatility. We will probably keep using integer for performance critical path, but less critical code may start using them now that they are declared. Differential Revision: https://phab.mercurial-scm.org/D10791

censor: drop size limitation on the tombstone This limitation seems to date back from a previous implementation of censors, were the revision were censored" in place. For a long while the implementation actually involved rewriting the revlog before replacing the original data. So it looks like we can safely remove this limitation. The tests suite agrees. Differential Revision: https://phab.mercurial-scm.org/D10790

revlog: move censoring code in a dedicated module This code is quite specific and we are about to add more of it for revlog-v2 (and other derived version). So we move this code in a dedicated module in `mercurial/revlogutils/`. This looks like a good fit. The diff is huge because I used `hg copy` to create the new file so that we keep the history of the censor code. Differential Revision: https://phab.mercurial-scm.org/D10789

censor: reduce risk of censor test blowing up output limit The censored file is really big, and we keep using `hg cat` on it, with the assumption that it is censored. However, when it is not censored, the amount of extra line output is huge and quickly saturate the backlog of terminal and CI system. Differential Revision: https://phab.mercurial-scm.org/D10788

revlog: store sidedata in their own file This makes sidedata manipulation simpler and results in more compact data when traversing either data or sidedata. Differential Revision: https://phab.mercurial-scm.org/D10787

revlog: fix docket.date_filepath docstring This was copy-pasted too quickly. Differential Revision: https://phab.mercurial-scm.org/D10786

revlog: use dedicated code for reading sidedata We are about to introduce a new, dedicated, file to store sidedata. Before doing so, we make sidedata reading go through different code as reading data chunk. This will simplify some of the complexity of the next changesets. The reading is very simple right now and will need some improvement later to reuse some of the caching strategy we use for the data file. Differential Revision: https://phab.mercurial-scm.org/D10785

revlog: simplify "partial read" error message We are about to reuse this message in more places and the current handling is pretty hard to read. So we eat two pie with one stone and clean up this. Differential Revision: https://phab.mercurial-scm.org/D10784

revlog: no longer return sidedata from `_revisiondata` All users have been migrated. When they was introduced sidedata where grouped with the actual revision data and unpacking one came with the other. Sidedata moved be stored "independently" and it no longer make sense to retrieve both at the same time unconditionnaly. Differential Revision: https://phab.mercurial-scm.org/D10783

revlog: use `self.sidedata` directly in `revlog.clone` We want to change `revlog._revisiondata` signature, so we need to migrate users away. Differential Revision: https://phab.mercurial-scm.org/D10782

revlog: use `self.sidedata` directly to construct changelogrevision We want to change `revlog._revisiondata` signature, so we need to migrate users away. Differential Revision: https://phab.mercurial-scm.org/D10781

revlog: implement sidedata without using _revisiondata When they was introduced sidedata where grouped with the actual revision data and unpacking one came with the other. Sidedata moved be stored "independently" and it no longer make sense to retrieve both at the same time unconditionnaly. We start with changeset the implementation of the `revlog.sidedata` command to no longer use `self._revisiondata`. More users need to be migrated to direct usage of this `revlog.sidedata` method. This will be done in the coming changesets. Differential Revision: https://phab.mercurial-scm.org/D10780

revlogv2: drop "sidedatacompression" from the list of todos This was implemented in c4dbb7636a12 and 07641bafa646, but I forgot to drop it from the todo-list. Differential Revision: https://phab.mercurial-scm.org/D10779

rust: Use a maintained crate for SHA-1 hashing https://crates.io/crates/rust-crypto hasn’t been updated in 5 years. This doesn’t neccesarily mean there’s anything wrong with it, but if something comes up it’s preferable to rely on libraries that have active maintainers. Use https://crates.io/crates/sha-1 from https://github.com/RustCrypto instead Differential Revision: https://phab.mercurial-scm.org/D10835

rust: Parse "subinclude"d files along the way, not later When parsing a `.hgignore` file and encountering an `include:` line, the included file is parsed recursively right then in a depth-first fashion. With `subinclude:` however included files were parsed (recursively) much later. This changes it to be expanded during parsing, like `.hgignore`. The motivation for this is an upcoming changeset that needs to detect changes in which files are ignored or not. The plan is to hash all ignore files while they are being read, and store that hash in the dirstate (in v2 format). In order to allow a potential alternative implementations to read that format, the algorithm to compute that hash must be documented. Having a well-defined depth-first ordering for the tree of (sub-)included files makes that easier. Differential Revision: https://phab.mercurial-scm.org/D10834

rust: Make some file path parameters less generic These are not widely used APIs that benefit from being maximally flexible, taking an explicit `&Path` borrow is fine and simplifies their internals. Differential Revision: https://phab.mercurial-scm.org/D10833

convert-bazaar: use breezy package instead of old bzr one Breezy is the most recent of the two, and works on Python 3 while being compatible with the (old) Bazaar file format. This patch contains a variety of unicode <-> bytes changes, API breakage fixing, restoring failing imports and changing the executable from `bzr` to `brz`. I recommend using the debian packages for `brz` and `python3-breezy` (3.1+), because the pip package seems to be haunted by radioactive dragons. Differential Revision: https://phab.mercurial-scm.org/D10513

bzr: only support bzr conversion on Python3 Python 2 support will be dropped from Mercurial before anyone needs to do a bzr -> hg conversion on Python 2 again. Bazaar tests were broken with the new lib anyway, which we'll get into in the next patches. Differential Revision: https://phab.mercurial-scm.org/D10512

rewriteutil: add pointer to help text when rewrite would cause divergence The evolve extension's version of the hint has this pointer. I missed it when I moved it to core. This is pretty much the same as D10723. Turns out I missed two places :( Differential Revision: https://phab.mercurial-scm.org/D10832