bookmarks: rely on exception for malformed lines Since we already have an exception context open, for other thing, we can simplify the code a bit and rely on exception handling for invalid lines. Speed is not the main motivation for this changes. However as I'm in the middle of benchmarking things we can see a small positive impact. Before: ! wall 0.009358 comb 0.000000 user 0.000000 sys 0.000000 (best of 303) After: ! wall 0.009173 comb 0.010000 user 0.010000 sys 0.000000 (best of 310)

bookmarks: explicitly convert to 'node' during initialization We know the content of the file is supposed to be full hex. So we can do the translation ourselves and directly check if the node is known. As nice side effect we now have proper error handling for invalid node value. Before: ! wall 0.021580 comb 0.020000 user 0.020000 sys 0.000000 (best of 134) After: ! wall 0.009342 comb 0.010000 user 0.010000 sys 0.000000 (best of 302)

bookmarks: prefetch 'lookup' outside of the loop Skipping the attribute lookup up raise a significant speedup. Example on a repository with about 4000 bookmarks. Before: ! wall 0.026027 comb 0.020000 user 0.020000 sys 0.000000 (best of 112) After: ! wall 0.021580 comb 0.020000 user 0.020000 sys 0.000000 (best of 134) (This is also in its own changeset to clarify the perf win from another coming changesets)

perf: add a perfbookmarks command A new command dedicated to benchmark of bookmark initialization.

perfphases: add a flag to also include file access time The flag purges all phases data so we'll have to read the file from disk again.

perf: have a generic "clearstorecache" function There are multiple places where we will want to purge some store cache. So we promote the existing _clearobsstore function to a something reusable.

localrepo: move filtername to __init__ This is obviously an instance attribute, not a type attribute. The modern Python style is to use __init__ for defining these. This exposes statichttprepo as inheriting from localrepository without calling its __init__. As a result, its __init__ defines a lot of variables that methods on localrepository's methods need. But factoring the common bits into a separate class is for another day.

obsolete: move obsstore creation logic from localrepo This code has more to do with obsolete.py than localrepo.py. Let's move it there.

match: allow pats to be None match.match already interprets "!bool(patterns)" as matching everything (but includes and excludes still apply). We might as well allow None, which lets us simplify some callers a bit. I originally wrote this patch while trying to change match.match(patterns=[]) to mean to match no patterns. This patch is one step towards that goal. I'm not sure it'll be worth the effort to go all the way there, but I think this patch still makes sense on its own.

tests: add a test for installing hg with pip in a virtualenv Since we're doing so much clever junk in our setup.py, let's have a test that exercises it. Thanks to Matt Harbison for testing this on Windows and verifying that installenv/*/hg would work as a way to work around bin being called Scripts on Windows.

hghave: add check for virtualenv

setup: introduce dummy copies of setuptools flags Since we're filtering out some egg gunk, we need to emulate these flags which disable eggs so that pip still works.

extensions: catch uisetup and extsetup failures and don't let them break hg Otherwise users of the patience diff extension will be unable to run anything at all in hg 4.3 until they figure out what's broken.

tests: add test demonstrating how broken third-party extensions can get I intend to fix this, but will do the fix as a separate change to make the behavior change obvious. This was inspired by some users having the patience diff extension, which broke when we moved bdiff.so so thoroughly the users can't even run 'hg debuginstall'.

extensions: move wrapfilecache function from fsmonitor It makes more sense to put this in core, so other extensions can trivially get access to it without having to rely on importing fsmonitor.

chmod: create a new file when flags are set on a hardlinked file For performance reasons we have several repositories where the files in the working directory of 1 repo are hardlinks to the files of the other repo When an update in one repo results in a chmod of a such a file, the hardlink has to be deleted and replaced by a regular file to make sure that the change does not happen in the other repo

run-tests: make --restart work with output dir

run-tests: output coverage to output dir There do not appear to be any tests for this, and I've never used either of these options before, but this works.

run-tests: write JSON reports to output dir

run-tests: write test times to output dir

run-tests: allow specifying an output dir to write .errs to I'm trying to use run-tests.py on a read-only file system. This series allows that to happen.

run-tests: add information about skipped tests to XUnit output The XUnit spec supports skipped tests.

run-tests: wrap failures in an XUnit 'failure' element This is closer to what most XUnit consumers can understand.

discovery: log discovery result in non-trivial cases We log the discovery summary, the number of roundtrips and the elapsed time. This is useful to understand where slow push might come from when lloking at the blackbox.

discovery: include timing in the debug output Having such date easily available is useful. It also prepare the inclusion of some discovery related data in blackbox.

tests: remove sys.executable from "required tools" In practice this doesn't appear to have been true for some time - we reference Python using the $PYTHON variable in all the tests now (which we have to for PyPy and Python 3), and I've been using ~/.../python.exe to test with tip of the cpython 3.6 release branch while working on manifest tests in Python 3 and everything seems to be just fine. The only real observable difference from this change is that I stop getting a warning about python.exe not being a thing on $PATH, which seems like an improvement.

perfbranchmap: add an option to purge the revbranch cache The perf extension needs to be able to reflect this reality too. (eg: 4s vs 60s on a Million-ish revisions repository).

push: add a way to allow concurrent pushes on unrelated heads Client has a mechanism for the server to check that nothing changed server side since the client prepared a push. That check is wide and any head changed on the server will lead to an aborted push. We introduce a way for the client to send a less strict checking. That logic will check that no heads impacted by the push have been affected. If other unrelated heads (including named branches heads) have been affected, the push will proceed. This is very helpful for repositories with high developers traffic on different heads, a common setup. That behavior is currently controlled by an experimental option. The config should live in the "server" section but bike-shedding of the name will happen in the next changesets. Servers advertise this capability through a new bundle2 capability 'checkeads', using the value 'related'. The 'test-push-race.t' is updated to check that new capabilities on the documented cases.

headsummary: expose the 'discardedheads' set in the headssummary That information will be useful to detect push race on related part of the history. See next changeset for details.

checkheads: perform obsolescence post processing directly in _headssummary The goal is to have the function directly return something meaningful and useful for the whole pull. Note: we skip adding post-processing in '_oldheadssummary' because if a client is too old for branchmap it will be too old for obsolescence too.

headssummary: directly feed the function with the 'pushop' object Our goal is to be able to perform the post processing directly into the '_headssummary' function. However before this patch the '_headsummary' function only had access to repo, remote, outgoing while the '_postprocessobsolete' function takes a 'pushop' object. Experience shows that having the 'pushop' object helps extensions so we update '_headssummary' to take a pushop object as argument.

checkheads: gather the postprocessing with other obsolescence specific code We extract this function from the loop and gather it with the rest of the obsolescence specific code. That will help to clarify the move of the whole logic inside the "heads summary" computation.

run-tests: add a way to list tests, with JSON and XUnit support Some test runners are interested in listing tests, so they can do their own filtering on top (usually based on attributes like historically observed runtime). Add support for that.

run-tests: install hg after computing tests to run We're going to add a way to list tests, and we don't need to install hg for that.

run-tests: make time field optional for xunit report We're going to use XUnit to list tests, and we don't have a time field in that case.

run-tests: factor out json write code into another method We're going to use this code to output a JSON-formatted listing of tests.

run-tests: factor out xunit write code into another method We're going to use this code to output an XUnit-formatted listing of tests.

revset: lookup descendents for negative arguments to ancestor operator Negative offsets to the `~` operator now search for descendents. The search is aborted when a node has more than one child as we do not have a definition for 'nth child'. Optionally we can introduce such a notion and take the nth child ordered by rev number. The current revset language does provides a short operator for ancestor lookup but not for descendents. This gives user a simple revset to move to the previous changeset, e.g. `hg up '.~1'` but not to the 'next' changeset. With this change userse can now use `.~-1` as a shortcut to move to the next changeset. This fits better into allowing users to specify revisions via revsets and avoiding the need for special `hg next` and `hg prev` operations. The alternative to negative offsets is adding a new operator. We do not have many operators in ascii left that do not require bash escaping (',', '_', and '/' come to mind). If we decide that we should add a more convenient short operator such as ('/', e.g. './1') we can later add it and allow ascendents lookup via negative numbers.

update: show the commit to which we updated in case of multiple heads (BC) Currently when we have multiple heads on the same branch, update tells us that there some more heads for the current branch but does not tells us the head to which the repository has been updated to. It makes more sense showing the head we updated to and then telling there are some more heads.

revlog: skeleton support for version 2 revlogs There are a number of improvements we want to make to revlogs that will require a new version - version 2. It is unclear what the full set of improvements will be or when we'll be done with them. What I do know is that the process will likely take longer than a single release, will require input from various stakeholders to evaluate changes, and will have many contentious debates and bikeshedding. It is unrealistic to develop revlog version 2 up front: there are just too many uncertainties that we won't know until things are implemented and experiments are run. Some changes will also be invasive and prone to bit rot, so sitting on dozens of patches is not practical. This commit introduces skeleton support for version 2 revlogs in a way that is flexible and not bound by backwards compatibility concerns. An experimental repo requirement for denoting revlog v2 has been added. The requirement string has a sub-version component to it. This will allow us to declare multiple requirements in the course of developing revlog v2. Whenever we change the in-development revlog v2 format, we can tweak the string, creating a new requirement and locking out old clients. This will allow us to make as many backwards incompatible changes and experiments to revlog v2 as we want. In other words, we can land code and make meaningful progress towards revlog v2 while still maintaining extreme format flexibility up until the point we freeze the format and remove the experimental labels. To enable the new repo requirement, you must supply an experimental and undocumented config option. But not just any boolean flag will do: you need to explicitly use a value that no sane person should ever type. This is an additional guard against enabling revlog v2 on an installation it shouldn't be enabled on. The specific scenario I'm trying to prevent is say a user with a 4.4 client with a frozen format enabling the option but then downgrading to 4.3 and accidentally creating repos with an outdated and unsupported repo format. Requiring a "challenge" string should prevent this. Because the format is not yet finalized and I don't want to take any chances, revlog v2's version is currently 0xDEAD. I figure squatting on a value we're likely never to use as an actual revlog version to mean "internal testing only" is acceptable. And "dead" is easily recognized as something meaningful. There is a bunch of cleanup that is needed before work on revlog v2 begins in earnest. I plan on doing that work once this patch is accepted and we're comfortable with the idea of starting down this path.

check-code: ban grep's context flags (-A/-B/-C) since they're not on Solaris

patchbomb: avoid -r and -B options at the same time

debugbundle: add --part-type flag to emit only named part types This removes the need in the tests for grep -A, which is not supported on Solaris.

test-obsolete-bundle-strip: add globs for Windows

obsstore: move header encoding to a separate function This patch moves encodeheader from encodemarkers. So markers and header could be encoded separately.

obsstore: move _version to a propertycache This makes sure _version is correct even if "_all" is not called.

obsstore: separate marker parsing from obsstore reading This allows us to get raw obsstore content without parsing any markers. Reading obsstore is much cheaper than parsing markers.

obsstore: move marker version reading to a separate function This allows us to read marker version without reading markers.

obsstore: minor optimization for the obsolete revset Use local variables in a loop.

dispatch: do not close stdout and stderr, just flush() instead Since 3a4c0905f357 "util: always force line buffered stdout when stdout is a tty", we have two file objects attached to the same STDOUT_FILENO. If one is closed, the underlying file descriptor is also closed, and writing to the other file object would crash the Python interpreter in a hard way, at least on Windows. So, it seems safer to not close the standard streams. This also matches the behavior of the default sys.stdout/stderr.close(), which never close the FILE* streams in C layer. https://hg.python.org/cpython/file/v2.7.13/Python/sysmodule.c#l1401

windows: do not close stdout on flush() failure It's been there since e817c68edfed (2007-02-19), but seems wrong since any I/O operations to a closed file would raise ValueError, not IOError. We should keep the file object open even if the underlying file descriptor is half dead.

test-obsolete-bundle-strip: do not include \n in filename (issue5586)

revlog: add support for partial matching of wdir node id The idea is simple. If the given node id prefix is 'ff...f', add +1 to the number of matches (e.g. ambiguous if partial + maybewdir > 1). This patch also fixes id() revset and shortest() template since _partialmatch() can raise WdirUnsupported exception.

revset: add support for branch(wdir()) and wdir() & branch()

dirstate: add docstring for invalidate This always confuses me, and we already have a docstring on localrepo.invalidatedirstate.

tests: simplify and clarify test-obsolete-bundle-strip.t a little

tests: add missing parens in test-obsolete-bundle-strip.t The "(not ${revs})" was missing parens around ${revs}, so when revs was "A + B", it became "(not A + B)" when actually "(not (A + B))" was intended. Fixing that leads to some more testing of strip. Similarly, the parens were missing in "${revs}::", making it "A + B::" instead of "(A + B)::". Thanks to Yuya for noticing this part. This did not affect any existing tests.

test-clonebundles: conditionalize output for Windows This was the fallout from 9d6031df42c8 (on stable). Now that individual lines can be conditionalized, it seems better to be explicit, rather than mash all of this into one regex. "getaddrinfo failed" was added in 4e566f513fd8 to support Windows.

tests: adjust recent output changes for Windows

killdaemons: close pid file before killing processes With #serve enabled on Windows, I was getting occasional stacktraces like this: Errored test-hgweb-json.t: Traceback (most recent call last): File "./run-tests.py", line 724, in run self.tearDown() File "./run-tests.py", line 805, in tearDown killdaemons(entry) File "./run-tests.py", line 540, in killdaemons logfn=vlog) File "...\tests\killdaemons.py", line 94, in killdaemons os.unlink(pidfile) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: '...\\hgtests.zmpqj3\\child80\\daemon.pids' Adrian suggested using util.posixfile, which works. However, the 'mercurial' package isn't in sys.path when invoking run-tests.py, and it isn't clear that hacking[1] it in is a good thing (especially for test-run-tests.t, which uses an installation in a temp folder). I tried using ProcessMonitor to figure out what the other process is, but that monitoring slows things down to such a degree that the issue doesn't occur. I was ready to blame the virus scanner, but it happens without that too. Looking at the code, I don't see anything that would have the pid file open. But I was able to get through about 20 full test runs without an issue with this minor change, whereas before it was pretty certain to hit this at least once in two or three runs. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097907.html

win32: drop a py26 daemonizing hack I'm not sure what the referenced hang specifically was, but the whole test suite (with #serve) still runs on python 2.7.13. Aside from no longer prepending "cmd.exe /c", this backs out ca6aa8362f33. I'm trying to track down a rare failure of TerminateProcess() with an access error, and I've seen random extra python processes hanging around after running tests sometimes, so this might help. However, e48cb1c7a902 forces this change. Since the pid object is no longer converted to a string, the cmd.exe pid was being saved instead of the hg pid, and none of the daemons were being killed.

checkheads: simplify the code around obsolescence post-processing The 'discardedheads' return become unused and the relationship between newheads and newhs can be clarified. Our next goal here is to be able to extract the _postprocessobsolete call outside of the loop. We keep returning the 'discardedheads' because we'll start using it again soon in this series.

checkheads: drop now unused filtering of 'unsyncedheads' Now that unsynced heads are no longer in the function inputs or returns, we can simplify the code a bit.

checkheads: clarify that we no longer touch the head unknown locally Since c6cb21ddf74a, heads unknown locally no longer get any post processing from obsolescence markers. We clarify this fact by only feeding the list of locally known new heads to the function. This simplification of the input will help moving that post-processing earlier in the function.

headssummary: ensure all returned lists are sorted This is a simple step that will help to keep a stable output in coming refactoring.

discovery: also use lists for the returns of '_oldheadssummary' The '_headssummary' function is documenting and using list objects in its return. We now use them in _oldheadssummary too for consistency. This does not affect any usages of these values.

test: add a push race case where the updated head is obsoleted This is the mirror of the previously added case. We check the case where the racing-push obsoletes a head while the raced-push updates that same head.

test: add a push race case where obsoleted head is updated We check the case where the raced-push obsoletes a head while the racing-push updates that same head.

keepalive: set buffering=True to do more efficient reads of headers Support for buffering was added to python in d09d6fe31b61, first released with python2.7. Without this, the entirety of the response headers is read byte-by-byte (it does more efficient reads when it gets to the non-header part of the response).

keepalive: pass the correct arguments to HTTPResponse python2.7's httplib.HTTPResponse takes the arguments in the following order: sock, debuglevel, strict, method, buffering This was previously passing them in as positional and skipped strict, so we set strict=method. I'm explicitly setting strict=True now to preserve the previous behavior that has been there since this file was created.

help: update the color documentation for Windows 10 ANSI support It looks like only the initial release of Windows 10 lacked support for this functionality. [1][2] Since that build is no longer supported, I didn't bother getting very specific, to keep the help text less cluttered. [1] https://github.com/symfony/symfony/issues/17499#issuecomment-243481052 [2] https://en.wikipedia.org/wiki/Windows_10_version_history

color: enable ANSI support on Windows 10 This will display color if "color.mode=ansi", and default to 'ansi' if the mode is set to 'auto'. The 'debugcolor' command also reflects this policy. Previously, "color.mode=ansi" on Windows printed jibberish around the normal text. Using ANSI color is better, as it avoids the normal loss of color when the default pager is enabled on Windows. See also issue5570. When the underlying function fails (e.g. when run on older Windows), 'auto' still falls back to 'win32'. Apparently, Microsoft originally had this feature turned on by default, and then made it opt-in[1]. Therefore, not enabling it unconditionally seems safer. Instead, only do it after processing the existing check for support in a Unix-like environment. [1] https://github.com/symfony/symfony/issues/17499#issuecomment-243481052

win32: add a method to enable ANSI color code processing on Windows 10 SetConsoleMode() fails with an invalid parameter error if given this option prior to Windows 10, so indicate that to the caller instead of doing explicit version checks.

merge with stable

export: map wctx.node() to 'ff...' node id (issue5438)

revset: add support for integer and hex wdir identifiers As I said before, partial 'ff...' hash isn't supported yet.

localrepo: map integer and hex wdir identifiers to workingctx changectx.__init__() is slightly modified to take str(wdirrev) as a valid integer revision (and raise WdirUnsupported exception.) Test will be added by the next patch.

revlog: map rev(wdirid) to WdirUnsupported exception This will allow us to map repo["ff..."] to workingctx. _partialmatch() will be updated later. I tried "return wdirrev" in place of raising the exception, but earlier exception seemed better.

scmutil: introduce binnode(ctx) as paired function with intrev(ctx) It seemed silly to convert ctx.hex() back to binary to use node.hex/short(), or to use [:12] instead of node.short() because ctx.node() could be None. Eventually I want to change wctx.rev() and wctx.node() to return wdirrev and wdirid respectively, but that's quite big API breakage and can't be achieved without some compatibility wrappers.

merge: use scmutil.intrev() to sort ctx objects This moves wctx to the last, but that shouldn't matter. Only the order of stored revisions is important.

scmutil: pass ctx object to intrev() This makes it slightly easier to sort basectx objects by key=scmutil.intrev. We're most likely to have ctx objects where changectx/workingctx abstraction is necessary, so this won't increase the abstraction overhead.

setup: do not overwrite local __modulepolicy__.py on out-of-source build Since the default policy is selected depending on setup options, "make install" shouldn't overwrite in-source __modulepolicy__.py generated by "make local".

Added signature for changeset c850f0ed54c1

Added tag 4.2.1 for changeset c850f0ed54c1

merge with stable

status: don't crash if a lookup file disappears This can happen if another process (even another hg process!) comes along and removes the file at that time. This partly resolves issue5584, but not completely -- a bogus dirstate update can still happen. However, the full fix is too involved for stable.

match: simplify nevermatcher Most of it does the same as its superclass, so it can simply be removed. It also seems to make more sense for it to use relative paths, as we do for everything except alwaysmatcher, although nevermatcher.uipath() will probably never get called anyway, so it won't matter.

annotate: restructure formatter output to be nested list (BC) Annotate data should be in [(file, [line...])...] form, but there was no API to represent such data structure when I ported it to formatter. Now we have fm.nested() and the -T option is still experimental, so we can fix the data format.

annotate: rename formatter variable So we can add a nested 'fm' of narrow scope.

setup: prevent setuptools from laying an egg Previously, test-hghave.t was failing on Windows (and on Linux if $FORCE_SETUPTOOLS was set) with the following: --- c:/Users/Matt/Projects/hg/tests/test-hghave.t +++ c:/Users/Matt/Projects/hg/tests/test-hghave.t.err @@ -19,7 +19,11 @@ > foo > EOF $ run-tests.py $HGTEST_RUN_TESTS_PURE test-hghaveaddon.t + warning: Testing with unexpected mercurial lib: c:\Users\Matt\Projects\hg\mercurial + (expected ...\hgtests.mu9rou\install\lib\python\mercurial) . + warning: Tested with unexpected mercurial lib: c:\Users\Matt\Projects\hg\mercurial + (expected ...\hgtests.mu9rou\install\lib\python\mercurial) Augie relayed concerns[1] about the first attempt at this, which also excluded 'install_egg_info'. All that needs to be excluded to avoid the egg and make the test work is to filter out 'bdist_egg'. (Actually, the body of this class could simply be 'pass', and 'bdist_egg' still isn't run. But that seems to magical.) Also note that prior to this (and still now), `make clean` doesn't delete the 'mercurial.egg-info' that is generated by `make install`. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097668.html # no-check-commit

bitmanipulation: add missing include of string.h That's where memcpy() is declared.

py3: add test to show `hg update` and `hg identify` works on Python 3

py3: add a test to show `hg diff` works on Python 3

py3: implement __bytes__ for committablectx Before this method, calling bytes on workingctx or memctx calls basectx.__bytes__ since the magic method was not defined for this class. When it calls the method from basectx class, it returns TypeError because None is passed into it. After this commit `hg update -C` works on Python 3 if eol is not enabled.

py3: convert exception to bytes to pass into ui.warn() Here encoding.strtolocal() is used because exc maybe an IOError which could contain a valid non-ascii unicode.

py3: convert bool variables to bytes before passing into ui.debug() We can't pass unicodes to ui.debug() and hence we need to convert things to bytes before passing them.