bookmarks: prefetch 'lookup' outside of the loop Skipping the attribute lookup up raise a significant speedup. Example on a repository with about 4000 bookmarks. Before: ! wall 0.026027 comb 0.020000 user 0.020000 sys 0.000000 (best of 112) After: ! wall 0.021580 comb 0.020000 user 0.020000 sys 0.000000 (best of 134) (This is also in its own changeset to clarify the perf win from another coming changesets)

perf: add a perfbookmarks command A new command dedicated to benchmark of bookmark initialization.

perfphases: add a flag to also include file access time The flag purges all phases data so we'll have to read the file from disk again.

perf: have a generic "clearstorecache" function There are multiple places where we will want to purge some store cache. So we promote the existing _clearobsstore function to a something reusable.

localrepo: move filtername to __init__ This is obviously an instance attribute, not a type attribute. The modern Python style is to use __init__ for defining these. This exposes statichttprepo as inheriting from localrepository without calling its __init__. As a result, its __init__ defines a lot of variables that methods on localrepository's methods need. But factoring the common bits into a separate class is for another day.

obsolete: move obsstore creation logic from localrepo This code has more to do with obsolete.py than localrepo.py. Let's move it there.

match: allow pats to be None match.match already interprets "!bool(patterns)" as matching everything (but includes and excludes still apply). We might as well allow None, which lets us simplify some callers a bit. I originally wrote this patch while trying to change match.match(patterns=[]) to mean to match no patterns. This patch is one step towards that goal. I'm not sure it'll be worth the effort to go all the way there, but I think this patch still makes sense on its own.

tests: add a test for installing hg with pip in a virtualenv Since we're doing so much clever junk in our setup.py, let's have a test that exercises it. Thanks to Matt Harbison for testing this on Windows and verifying that installenv/*/hg would work as a way to work around bin being called Scripts on Windows.

hghave: add check for virtualenv

setup: introduce dummy copies of setuptools flags Since we're filtering out some egg gunk, we need to emulate these flags which disable eggs so that pip still works.

extensions: catch uisetup and extsetup failures and don't let them break hg Otherwise users of the patience diff extension will be unable to run anything at all in hg 4.3 until they figure out what's broken.

tests: add test demonstrating how broken third-party extensions can get I intend to fix this, but will do the fix as a separate change to make the behavior change obvious. This was inspired by some users having the patience diff extension, which broke when we moved bdiff.so so thoroughly the users can't even run 'hg debuginstall'.

extensions: move wrapfilecache function from fsmonitor It makes more sense to put this in core, so other extensions can trivially get access to it without having to rely on importing fsmonitor.

chmod: create a new file when flags are set on a hardlinked file For performance reasons we have several repositories where the files in the working directory of 1 repo are hardlinks to the files of the other repo When an update in one repo results in a chmod of a such a file, the hardlink has to be deleted and replaced by a regular file to make sure that the change does not happen in the other repo

run-tests: make --restart work with output dir

run-tests: output coverage to output dir There do not appear to be any tests for this, and I've never used either of these options before, but this works.

run-tests: write JSON reports to output dir

run-tests: write test times to output dir

run-tests: allow specifying an output dir to write .errs to I'm trying to use run-tests.py on a read-only file system. This series allows that to happen.

run-tests: add information about skipped tests to XUnit output The XUnit spec supports skipped tests.

run-tests: wrap failures in an XUnit 'failure' element This is closer to what most XUnit consumers can understand.

discovery: log discovery result in non-trivial cases We log the discovery summary, the number of roundtrips and the elapsed time. This is useful to understand where slow push might come from when lloking at the blackbox.

discovery: include timing in the debug output Having such date easily available is useful. It also prepare the inclusion of some discovery related data in blackbox.

tests: remove sys.executable from "required tools" In practice this doesn't appear to have been true for some time - we reference Python using the $PYTHON variable in all the tests now (which we have to for PyPy and Python 3), and I've been using ~/.../python.exe to test with tip of the cpython 3.6 release branch while working on manifest tests in Python 3 and everything seems to be just fine. The only real observable difference from this change is that I stop getting a warning about python.exe not being a thing on $PATH, which seems like an improvement.

perfbranchmap: add an option to purge the revbranch cache The perf extension needs to be able to reflect this reality too. (eg: 4s vs 60s on a Million-ish revisions repository).

push: add a way to allow concurrent pushes on unrelated heads Client has a mechanism for the server to check that nothing changed server side since the client prepared a push. That check is wide and any head changed on the server will lead to an aborted push. We introduce a way for the client to send a less strict checking. That logic will check that no heads impacted by the push have been affected. If other unrelated heads (including named branches heads) have been affected, the push will proceed. This is very helpful for repositories with high developers traffic on different heads, a common setup. That behavior is currently controlled by an experimental option. The config should live in the "server" section but bike-shedding of the name will happen in the next changesets. Servers advertise this capability through a new bundle2 capability 'checkeads', using the value 'related'. The 'test-push-race.t' is updated to check that new capabilities on the documented cases.

headsummary: expose the 'discardedheads' set in the headssummary That information will be useful to detect push race on related part of the history. See next changeset for details.

checkheads: perform obsolescence post processing directly in _headssummary The goal is to have the function directly return something meaningful and useful for the whole pull. Note: we skip adding post-processing in '_oldheadssummary' because if a client is too old for branchmap it will be too old for obsolescence too.

headssummary: directly feed the function with the 'pushop' object Our goal is to be able to perform the post processing directly into the '_headssummary' function. However before this patch the '_headsummary' function only had access to repo, remote, outgoing while the '_postprocessobsolete' function takes a 'pushop' object. Experience shows that having the 'pushop' object helps extensions so we update '_headssummary' to take a pushop object as argument.

checkheads: gather the postprocessing with other obsolescence specific code We extract this function from the loop and gather it with the rest of the obsolescence specific code. That will help to clarify the move of the whole logic inside the "heads summary" computation.

run-tests: add a way to list tests, with JSON and XUnit support Some test runners are interested in listing tests, so they can do their own filtering on top (usually based on attributes like historically observed runtime). Add support for that.

run-tests: install hg after computing tests to run We're going to add a way to list tests, and we don't need to install hg for that.

run-tests: make time field optional for xunit report We're going to use XUnit to list tests, and we don't have a time field in that case.

run-tests: factor out json write code into another method We're going to use this code to output a JSON-formatted listing of tests.

run-tests: factor out xunit write code into another method We're going to use this code to output an XUnit-formatted listing of tests.

revset: lookup descendents for negative arguments to ancestor operator Negative offsets to the `~` operator now search for descendents. The search is aborted when a node has more than one child as we do not have a definition for 'nth child'. Optionally we can introduce such a notion and take the nth child ordered by rev number. The current revset language does provides a short operator for ancestor lookup but not for descendents. This gives user a simple revset to move to the previous changeset, e.g. `hg up '.~1'` but not to the 'next' changeset. With this change userse can now use `.~-1` as a shortcut to move to the next changeset. This fits better into allowing users to specify revisions via revsets and avoiding the need for special `hg next` and `hg prev` operations. The alternative to negative offsets is adding a new operator. We do not have many operators in ascii left that do not require bash escaping (',', '_', and '/' come to mind). If we decide that we should add a more convenient short operator such as ('/', e.g. './1') we can later add it and allow ascendents lookup via negative numbers.

update: show the commit to which we updated in case of multiple heads (BC) Currently when we have multiple heads on the same branch, update tells us that there some more heads for the current branch but does not tells us the head to which the repository has been updated to. It makes more sense showing the head we updated to and then telling there are some more heads.

revlog: skeleton support for version 2 revlogs There are a number of improvements we want to make to revlogs that will require a new version - version 2. It is unclear what the full set of improvements will be or when we'll be done with them. What I do know is that the process will likely take longer than a single release, will require input from various stakeholders to evaluate changes, and will have many contentious debates and bikeshedding. It is unrealistic to develop revlog version 2 up front: there are just too many uncertainties that we won't know until things are implemented and experiments are run. Some changes will also be invasive and prone to bit rot, so sitting on dozens of patches is not practical. This commit introduces skeleton support for version 2 revlogs in a way that is flexible and not bound by backwards compatibility concerns. An experimental repo requirement for denoting revlog v2 has been added. The requirement string has a sub-version component to it. This will allow us to declare multiple requirements in the course of developing revlog v2. Whenever we change the in-development revlog v2 format, we can tweak the string, creating a new requirement and locking out old clients. This will allow us to make as many backwards incompatible changes and experiments to revlog v2 as we want. In other words, we can land code and make meaningful progress towards revlog v2 while still maintaining extreme format flexibility up until the point we freeze the format and remove the experimental labels. To enable the new repo requirement, you must supply an experimental and undocumented config option. But not just any boolean flag will do: you need to explicitly use a value that no sane person should ever type. This is an additional guard against enabling revlog v2 on an installation it shouldn't be enabled on. The specific scenario I'm trying to prevent is say a user with a 4.4 client with a frozen format enabling the option but then downgrading to 4.3 and accidentally creating repos with an outdated and unsupported repo format. Requiring a "challenge" string should prevent this. Because the format is not yet finalized and I don't want to take any chances, revlog v2's version is currently 0xDEAD. I figure squatting on a value we're likely never to use as an actual revlog version to mean "internal testing only" is acceptable. And "dead" is easily recognized as something meaningful. There is a bunch of cleanup that is needed before work on revlog v2 begins in earnest. I plan on doing that work once this patch is accepted and we're comfortable with the idea of starting down this path.

check-code: ban grep's context flags (-A/-B/-C) since they're not on Solaris

patchbomb: avoid -r and -B options at the same time

debugbundle: add --part-type flag to emit only named part types This removes the need in the tests for grep -A, which is not supported on Solaris.

test-obsolete-bundle-strip: add globs for Windows

obsstore: move header encoding to a separate function This patch moves encodeheader from encodemarkers. So markers and header could be encoded separately.

obsstore: move _version to a propertycache This makes sure _version is correct even if "_all" is not called.

obsstore: separate marker parsing from obsstore reading This allows us to get raw obsstore content without parsing any markers. Reading obsstore is much cheaper than parsing markers.

obsstore: move marker version reading to a separate function This allows us to read marker version without reading markers.

obsstore: minor optimization for the obsolete revset Use local variables in a loop.

dispatch: do not close stdout and stderr, just flush() instead Since 3a4c0905f357 "util: always force line buffered stdout when stdout is a tty", we have two file objects attached to the same STDOUT_FILENO. If one is closed, the underlying file descriptor is also closed, and writing to the other file object would crash the Python interpreter in a hard way, at least on Windows. So, it seems safer to not close the standard streams. This also matches the behavior of the default sys.stdout/stderr.close(), which never close the FILE* streams in C layer. https://hg.python.org/cpython/file/v2.7.13/Python/sysmodule.c#l1401

windows: do not close stdout on flush() failure It's been there since e817c68edfed (2007-02-19), but seems wrong since any I/O operations to a closed file would raise ValueError, not IOError. We should keep the file object open even if the underlying file descriptor is half dead.

test-obsolete-bundle-strip: do not include \n in filename (issue5586)

revlog: add support for partial matching of wdir node id The idea is simple. If the given node id prefix is 'ff...f', add +1 to the number of matches (e.g. ambiguous if partial + maybewdir > 1). This patch also fixes id() revset and shortest() template since _partialmatch() can raise WdirUnsupported exception.

revset: add support for branch(wdir()) and wdir() & branch()

dirstate: add docstring for invalidate This always confuses me, and we already have a docstring on localrepo.invalidatedirstate.

tests: simplify and clarify test-obsolete-bundle-strip.t a little

tests: add missing parens in test-obsolete-bundle-strip.t The "(not ${revs})" was missing parens around ${revs}, so when revs was "A + B", it became "(not A + B)" when actually "(not (A + B))" was intended. Fixing that leads to some more testing of strip. Similarly, the parens were missing in "${revs}::", making it "A + B::" instead of "(A + B)::". Thanks to Yuya for noticing this part. This did not affect any existing tests.

test-clonebundles: conditionalize output for Windows This was the fallout from 9d6031df42c8 (on stable). Now that individual lines can be conditionalized, it seems better to be explicit, rather than mash all of this into one regex. "getaddrinfo failed" was added in 4e566f513fd8 to support Windows.

tests: adjust recent output changes for Windows

killdaemons: close pid file before killing processes With #serve enabled on Windows, I was getting occasional stacktraces like this: Errored test-hgweb-json.t: Traceback (most recent call last): File "./run-tests.py", line 724, in run self.tearDown() File "./run-tests.py", line 805, in tearDown killdaemons(entry) File "./run-tests.py", line 540, in killdaemons logfn=vlog) File "...\tests\killdaemons.py", line 94, in killdaemons os.unlink(pidfile) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: '...\\hgtests.zmpqj3\\child80\\daemon.pids' Adrian suggested using util.posixfile, which works. However, the 'mercurial' package isn't in sys.path when invoking run-tests.py, and it isn't clear that hacking[1] it in is a good thing (especially for test-run-tests.t, which uses an installation in a temp folder). I tried using ProcessMonitor to figure out what the other process is, but that monitoring slows things down to such a degree that the issue doesn't occur. I was ready to blame the virus scanner, but it happens without that too. Looking at the code, I don't see anything that would have the pid file open. But I was able to get through about 20 full test runs without an issue with this minor change, whereas before it was pretty certain to hit this at least once in two or three runs. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097907.html

win32: drop a py26 daemonizing hack I'm not sure what the referenced hang specifically was, but the whole test suite (with #serve) still runs on python 2.7.13. Aside from no longer prepending "cmd.exe /c", this backs out ca6aa8362f33. I'm trying to track down a rare failure of TerminateProcess() with an access error, and I've seen random extra python processes hanging around after running tests sometimes, so this might help. However, e48cb1c7a902 forces this change. Since the pid object is no longer converted to a string, the cmd.exe pid was being saved instead of the hg pid, and none of the daemons were being killed.

checkheads: simplify the code around obsolescence post-processing The 'discardedheads' return become unused and the relationship between newheads and newhs can be clarified. Our next goal here is to be able to extract the _postprocessobsolete call outside of the loop. We keep returning the 'discardedheads' because we'll start using it again soon in this series.