revlog: skeleton support for version 2 revlogs There are a number of improvements we want to make to revlogs that will require a new version - version 2. It is unclear what the full set of improvements will be or when we'll be done with them. What I do know is that the process will likely take longer than a single release, will require input from various stakeholders to evaluate changes, and will have many contentious debates and bikeshedding. It is unrealistic to develop revlog version 2 up front: there are just too many uncertainties that we won't know until things are implemented and experiments are run. Some changes will also be invasive and prone to bit rot, so sitting on dozens of patches is not practical. This commit introduces skeleton support for version 2 revlogs in a way that is flexible and not bound by backwards compatibility concerns. An experimental repo requirement for denoting revlog v2 has been added. The requirement string has a sub-version component to it. This will allow us to declare multiple requirements in the course of developing revlog v2. Whenever we change the in-development revlog v2 format, we can tweak the string, creating a new requirement and locking out old clients. This will allow us to make as many backwards incompatible changes and experiments to revlog v2 as we want. In other words, we can land code and make meaningful progress towards revlog v2 while still maintaining extreme format flexibility up until the point we freeze the format and remove the experimental labels. To enable the new repo requirement, you must supply an experimental and undocumented config option. But not just any boolean flag will do: you need to explicitly use a value that no sane person should ever type. This is an additional guard against enabling revlog v2 on an installation it shouldn't be enabled on. The specific scenario I'm trying to prevent is say a user with a 4.4 client with a frozen format enabling the option but then downgrading to 4.3 and accidentally creating repos with an outdated and unsupported repo format. Requiring a "challenge" string should prevent this. Because the format is not yet finalized and I don't want to take any chances, revlog v2's version is currently 0xDEAD. I figure squatting on a value we're likely never to use as an actual revlog version to mean "internal testing only" is acceptable. And "dead" is easily recognized as something meaningful. There is a bunch of cleanup that is needed before work on revlog v2 begins in earnest. I plan on doing that work once this patch is accepted and we're comfortable with the idea of starting down this path.

check-code: ban grep's context flags (-A/-B/-C) since they're not on Solaris

patchbomb: avoid -r and -B options at the same time

debugbundle: add --part-type flag to emit only named part types This removes the need in the tests for grep -A, which is not supported on Solaris.

test-obsolete-bundle-strip: add globs for Windows

obsstore: move header encoding to a separate function This patch moves encodeheader from encodemarkers. So markers and header could be encoded separately.

obsstore: move _version to a propertycache This makes sure _version is correct even if "_all" is not called.

obsstore: separate marker parsing from obsstore reading This allows us to get raw obsstore content without parsing any markers. Reading obsstore is much cheaper than parsing markers.

obsstore: move marker version reading to a separate function This allows us to read marker version without reading markers.

obsstore: minor optimization for the obsolete revset Use local variables in a loop.

dispatch: do not close stdout and stderr, just flush() instead Since 3a4c0905f357 "util: always force line buffered stdout when stdout is a tty", we have two file objects attached to the same STDOUT_FILENO. If one is closed, the underlying file descriptor is also closed, and writing to the other file object would crash the Python interpreter in a hard way, at least on Windows. So, it seems safer to not close the standard streams. This also matches the behavior of the default sys.stdout/stderr.close(), which never close the FILE* streams in C layer. https://hg.python.org/cpython/file/v2.7.13/Python/sysmodule.c#l1401

windows: do not close stdout on flush() failure It's been there since e817c68edfed (2007-02-19), but seems wrong since any I/O operations to a closed file would raise ValueError, not IOError. We should keep the file object open even if the underlying file descriptor is half dead.

test-obsolete-bundle-strip: do not include \n in filename (issue5586)

revlog: add support for partial matching of wdir node id The idea is simple. If the given node id prefix is 'ff...f', add +1 to the number of matches (e.g. ambiguous if partial + maybewdir > 1). This patch also fixes id() revset and shortest() template since _partialmatch() can raise WdirUnsupported exception.

revset: add support for branch(wdir()) and wdir() & branch()

dirstate: add docstring for invalidate This always confuses me, and we already have a docstring on localrepo.invalidatedirstate.

tests: simplify and clarify test-obsolete-bundle-strip.t a little

tests: add missing parens in test-obsolete-bundle-strip.t The "(not ${revs})" was missing parens around ${revs}, so when revs was "A + B", it became "(not A + B)" when actually "(not (A + B))" was intended. Fixing that leads to some more testing of strip. Similarly, the parens were missing in "${revs}::", making it "A + B::" instead of "(A + B)::". Thanks to Yuya for noticing this part. This did not affect any existing tests.

test-clonebundles: conditionalize output for Windows This was the fallout from 9d6031df42c8 (on stable). Now that individual lines can be conditionalized, it seems better to be explicit, rather than mash all of this into one regex. "getaddrinfo failed" was added in 4e566f513fd8 to support Windows.

tests: adjust recent output changes for Windows

killdaemons: close pid file before killing processes With #serve enabled on Windows, I was getting occasional stacktraces like this: Errored test-hgweb-json.t: Traceback (most recent call last): File "./run-tests.py", line 724, in run self.tearDown() File "./run-tests.py", line 805, in tearDown killdaemons(entry) File "./run-tests.py", line 540, in killdaemons logfn=vlog) File "...\tests\killdaemons.py", line 94, in killdaemons os.unlink(pidfile) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: '...\\hgtests.zmpqj3\\child80\\daemon.pids' Adrian suggested using util.posixfile, which works. However, the 'mercurial' package isn't in sys.path when invoking run-tests.py, and it isn't clear that hacking[1] it in is a good thing (especially for test-run-tests.t, which uses an installation in a temp folder). I tried using ProcessMonitor to figure out what the other process is, but that monitoring slows things down to such a degree that the issue doesn't occur. I was ready to blame the virus scanner, but it happens without that too. Looking at the code, I don't see anything that would have the pid file open. But I was able to get through about 20 full test runs without an issue with this minor change, whereas before it was pretty certain to hit this at least once in two or three runs. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097907.html

win32: drop a py26 daemonizing hack I'm not sure what the referenced hang specifically was, but the whole test suite (with #serve) still runs on python 2.7.13. Aside from no longer prepending "cmd.exe /c", this backs out ca6aa8362f33. I'm trying to track down a rare failure of TerminateProcess() with an access error, and I've seen random extra python processes hanging around after running tests sometimes, so this might help. However, e48cb1c7a902 forces this change. Since the pid object is no longer converted to a string, the cmd.exe pid was being saved instead of the hg pid, and none of the daemons were being killed.

checkheads: simplify the code around obsolescence post-processing The 'discardedheads' return become unused and the relationship between newheads and newhs can be clarified. Our next goal here is to be able to extract the _postprocessobsolete call outside of the loop. We keep returning the 'discardedheads' because we'll start using it again soon in this series.

checkheads: drop now unused filtering of 'unsyncedheads' Now that unsynced heads are no longer in the function inputs or returns, we can simplify the code a bit.