verify: show progress while verifying dirlogs In repos with treemanifests, the non-root-directory dirlogs often have many more total revisions than the root manifest log has. This change adds progress out to that part of 'hg verify'. Since the verification is recursive along the directory tree, we don't know how many total revisions there are at the beginning of the command, so instead we report progress in units of directories, much like we report progress for verification of files today. I'm not very happy with passing both 'storefiles' and 'progress' into the recursive calls. I tried passing in just a 'visitdir(dir)' callback, but the results did not seem better overall. I'm happy to update if anyone has better ideas.

verify: check for orphaned dirlogs We already report orphaned filelogs, i.e. revlogs for files that are not mentioned in any manifest. This change adds checking for orphaned dirlogs, i.e. revlogs that are not mentioned in any parent-directory dirlog. Note that, for fncachestore, only files mentioned in the fncache are considered, there's not check for files in .hg/store/meta that are not mentioned in the fncache. This is no different from the current situation for filelogs.

verify: check directory manifests In repos with treemanifests, there is no specific verification of directory manifest revlogs. It simply collects all file nodes by reading each manifest delta. With treemanifests, that's means calling the manifest._slowreaddelta(). If there are missing revlog entries in a subdirectory revlog, 'hg verify' will simply report the exception that occurred while trying to read the root manifest: manifest@0: reading delta 1700e2e92882: meta/b/00manifest.i@67688a370455: no node This patch changes the verify code to load only the root manifest at first and verify all revisions of it, then verify all revisions of each direct subdirectory, and so on, recursively. The above message becomes b/@0: parent-directory manifest refers to unknown revision 67688a370455 Since the new algorithm reads a single revlog at a time and in order, 'hg verify' on a treemanifest version of the hg core repo goes from ~50s to ~14s. As expected, there is no significant difference on a repo with flat manifests.

hg: perform update after pulling during clone with share (issue5103) When pooled storage is enabled, `hg clone` will initialize a repo from a local repo using the store sharing mechanism then pull from the originally requested repo. Before this patch, the working directory update occurred between these steps. This meant that we would only update to revisions that were already present in the local pooled storage. This patch moves the update to after we pull from the originally requested repository so we may check out a revision that didn't yet exist locally. In other words, it makes the behavior like normal `hg clone`.

hg: extract post share update logic into own function A future patch will introduce a new caller that needs to perform an update. Extract the code so we don't duplicate it.

merge: perform background file closing in batchget As 2fdbf22a1b63 demonstrated with stream clones, closing files on background threads on Windows can yield a significant speedup because closing files that have been created/appended to is slow on Windows/NTFS. Working directory updates can write thousands of files. Therefore it is susceptible to excessive slowness on Windows due to slow file closes. This patch enables background file closing when performing working directory file writes. The impact when performing an `hg up tip` on mozilla-central (136,357 files) from an empty working directory is significant: Before: 535s (8:55) After: 133s (2:13) Delta: -402s (6:42) That's a 4x speedup! By comparison, that same machine can perform the same operation in ~15s on Linux. So Windows went from ~35x to ~9x slower. Not bad but there's still work to do. As a reminder, background file closing is only activated on Windows because it is only beneficial on that platform. So this patch shouldn't change non-Windows behavior at all. It's worth noting that non-Windows systems perform working directory updates with multiple processes. Unfortunately, worker.py doesn't yet support Windows. So, there is still plenty of room for making working directory updates faster on Windows. Even if multiple processes are used on Windows, I believe background file closing will still provide a benefit, as individual processes will still be slowed down by the file close bottleneck (assuming the I/O system isn't saturated).

merge: indent code in batchget() To make the next patch easier to read.

localrepo: support background closing for wwrite() So working copy update can pass it in.

scmutil: support background closing for write() Upcoming patches will add background file closer support to working copy update. This patch adds some plumbing to prepare for that.

chg: hold a lock file before connected to server This is a part of the one server per config series. In multiple-server setup, multiple clients may try to start different servers (on demand) at the same time. The old lock will not guarantee a client to connect to the server it just started, and is not crash friendly. This patch addressed above issues by using flock and does not release the lock until the client actually connects to the server or times out.

serve: allow --daemon-postexec to be 'unlink:path' or 'none' This patch changes the format of value of --daemon-postexec. Now it can be either to unlink a file, or a no-op. The following patch will make chg to use the no-op option.

serve: rename --daemon-pipefds to --daemon-postexec (BC) Initially we use --daemon-pipefds to pass file descriptors for synchronization. Later, in order to support Windows, --daemon-pipefds is changed to accept a file path to unlink instead. The name is outdated since then. chg client is designed to use flock, which will be held before starting a server and until the client actually connects to the server it started. The unlink synchronization approach is not so helpful in this case. To address the issues, this patch renames pipefds to postexec and the following patch will allow the value of --daemon-postexec to be things like 'unlink:/path/to/file' or 'none'.

largefiles: don't explicitly list optional parameters that are not used This makes it easier for changes to the API.

revert: properly revert to ancestor of p2 during merge (issue5052) During merge, added (from one perspective) file can be reported as "modified". To work around that, revert was testing if modified file were present in the parent manifest and marking them as "added" in this case. However, we should be checking against the target revision manifest instead. Otherwise see file as "newly added" even if they exist in the target revision. That revert behavior regressed in 06fbd9518bc5.

help: hg.intevation.de is new primary name of hg.intevation.de (and new cert) Adjust the examples (prefix and hostfingerprints) in help/config.txt https://hg.intevation.de/ is now served with a new certificate that is signed by a commercial CA, so all nearly all browsers will accept it automatically. Listing both names, hg.intevation.de and hg.intevation.org, in the section [hostfingerprints] allows using both without configuring web.cacerts and without changing existing https URLs in the [paths] section.

rebase: explicitly test abort from ambiguous destination We want to explicitly test the next behavior. We add this test in its own changeset to make the test change from the behavior change as clean as possible.

rebase: choose default destination the same way as 'hg merge' (BC) This changeset finally make 'hg rebase' choose its default destination using the same logic as 'hg merge'. The previous default was "tipmost changeset on the current branch", the new default is "the other head if there is only one". This change has multiple consequences: - Multiple tests which were not rebasing anything (rebasing from tipmost head) are now rebasing on the other "lower" branch. This is the expected new behavior. - A test is now explicitly aborting when there is too many heads on the branch. This is the expected behavior. - We gained a better detection of the "nothing to rebase" case while performing 'hg pull --rebase' so the message have been updated. Making clearer than an update was performed and why. This is beneficial side-effect. - Rebasing from an active bookmark will behave the same as 'hg merge' from a bookmark.

rebase: add potential divergent commit hashes to error message (issue5086)

hgwebdir_wsgi: update script and expand help I've updated the script to reflect changes in Mercurial and to include a much more through installation guide with configuration examples and details on how to configure IIS. I've used the script to set up a working server from scratch.

bundlerepo: properly handle hidden linkrev in filelog (issue4945) The bundlerepository have to do some special magic to handle linkrev of the bundlerepo filerev. That logic was done from a repoview and obsolescence marker affecting bundled changeset could lead to a crash. We now ensure we operate on unfiltered repository.

rebase: adds storing collapse message (issue4792) Before this patch collapse message wasn't stored so when you ran into the merge conflict while rebasing, running rebase --continue didn't remember the message and always opened editor to fill commit message. This patch adds saving collapse message in .hg/last-message.txt and restoring it later when needed.

test-automv: fix inline config settings for 5ec1ce8fdf0a

automv: use 95 as the default similarity threshold The motivation for the change from 100 to 95 is included in a comment. * Updated the tests to include a change to a moved file that still should be caught as a move. * Use ui.configint() to non-integer configuration entries more gracefully. Also complain if a similarity outside of the acceptable range is set.

bookmarks: add 'hg push -B .' for pushing the active bookmark (issue4917)

worker: change partition strategy to every Nth element The only consumer of the worker pool code today is `hg update`. Previously, the algorithm to partition work to each worker process preserved input list ordering. We'd take the first N elements, then the next N elements, etc. Measurements on mozilla-central demonstrate this isn't an optimal partitioning strategy. I added debug code to print when workers were exiting. When performing a working copy update on a previously empty working copy of mozilla-central, I noticed that process lifetimes were all over the map. One worker would complete after 7s. Many would complete after 12s. And another worker would often take >16s. This behavior occurred for many worker process counts and was more pronounced on some than others. What I suspect is happening is some workers end up with lots of small files and others with large files. This is because the update code passes in actions according to sorted filenames. And, directories under tend to accumulate similar files. For example, test directories often consist of many small test files and media directories contain binary (often larger) media files. This patch changes the partitioning algorithm to select every Nth element from the input list. Each worker thus has a similar composition of files to operate on. The result of this change is that worker processes now all tend to exit around the same time. The possibility of a long pole due to being unlucky and receiving all the large files has been mitigated. Overall execution time seems to drop, but not by a statistically significant amount on mozilla-central. However, repositories with directories containing many large files will likely show a drop. There shouldn't be any regressions due to partial manifest decoding because the update code already iterates the manifest to determine what files to operate on, so the manifest should already be decoded.

run-tests: allow run-tests.py to run tests outside current directory When reloading tests, run-tests.py was assuming that it could look up the test by the basename, which only works if you are running tests which are in the current directory. This patch changes that lookup to use the full path. This is all that was needed, and does not appear to cause any problems for any of the existing testing work flows based on running the suggested commands at the top of run-tests.py. Motivation: In order to test Mercurial with Hypothesis (according to https://www.mercurial-scm.org/wiki/HypothesisPlan) it is useful to be able to generate temporary test files and execute them. Generating temporary files in the tests/ directory leads to a lot of suboptimal clutter.

histedit: unifying the way replacements are computed for abort and success This is a part of a bigger refactoring effort with the ultimate goal of better understanding of how histedit works and fixing issue4800.

templater: factor out type conversion of revset() result This makes it clear why we have to do repo[int(x)].