manifest: change manifestlog mancache to be directory based In the last patch we added a get() function that allows fetching directory level treemanifestctxs. It didn't handle caching at directory level though, so we need to change our mancache to support multiple directories.

manifest: add manifestlog.get to obtain subdirectory instances Previously manifestlog only allowed obtaining root level manifests. Future patches will need direct access to subdirectory manifests as part of changegroup creation, so let's add a get() function that knows how to deal with subdirectories.

manifest: throw LookupError if node not in revlog When accessing a manifest via manifestlog[node], let's verify that the node actually exists and throw a LookupError if it doesn't. This matches the old read behavior, so we don't accidentally return invalid manifestctxs. We do this in manifestlog instead of in the manifestctx/treemanifestctx constructors because the treemanifest code currently relies on the fact that certain code paths can produce treemanifests without touching the revlogs (and it has tests that verify things work if certain revlogs are missing entirely, so they break if we add validation that tries to read them).

revlog: optimize _chunkraw when startrev==endrev In many cases, _chunkraw() is called with startrev==endrev. When this is true, we can avoid an extra index lookup and some other minor operations. On the mozilla-unified repo, `hg perfrevlogchunks -c` says this has the following impact: ! read w/ reused fd ! wall 0.371846 comb 0.370000 user 0.350000 sys 0.020000 (best of 27) ! wall 0.337930 comb 0.330000 user 0.300000 sys 0.030000 (best of 30) ! read batch w/ reused fd ! wall 0.014952 comb 0.020000 user 0.000000 sys 0.020000 (best of 197) ! wall 0.014866 comb 0.010000 user 0.000000 sys 0.010000 (best of 196) So, we've gone from ~25x slower than batch to ~22.5x slower. At this point, there's probably not much else we can do except implement an optimized function in the index itself, including in C.

revlog: inline start() and end() for perf reasons When I implemented `hg perfrevlogchunks`, one of the things that stood out was N * _chunk() calls was ~38x slower than 1 _chunks() call. Specifically, on the mozilla-unified repo: N*_chunk: 0.528997s 1*_chunks: 0.013735s This repo has 352,097 changesets. So the average time per changeset comes out to: N*_chunk: 1.502us 1*_chunks: 0.039us If you extrapolate these numbers to a repository with 1M changesets, that comes out to 1.502s versus 0.039s, which is significant. At these latencies, Python attribute lookups and function calls matter. So, this patch inlines some code to cut down on that overhead. The impact of this patch on N*_chunk() calls is clear: ! wall 0.528997 comb 0.520000 user 0.500000 sys 0.020000 (best of 19) ! wall 0.367723 comb 0.370000 user 0.350000 sys 0.020000 (best of 27) So, we go from ~38x slower to ~27x. A nice improvement. But there's still a long way to go. It's worth noting that functionality like revsets perform changelog lookups one revision at a time. So this code path is worth optimizing.

revlog: reorder index accessors to match data structure order Index entries are ordered tuples. We have accessors in the revlog class to map tuple offsets to names. To help reinforce the order, reorder the methods so they match the order of elements in the tuple. While I'm here, also sneak in some minimal documentation.

color: add the ability to display configured style to 'debugcolor' The 'hg debugcolor' command gains a '--style' flag to display all the configured labels and their styles. This have many benefits: * discovering documented label, * checking consistency between label's style, * showing the actual style of a label.

color: sort output of 'debugcolor' The previous ordering were provided by the set. The new output is more stable and rational. In addition we have some logic to keep the '_background' version together to help readability.

color: extract color and effect display from 'debugcolor' We are about to introduce a second mode for 'hg debugcolor' that would list the known label and their configuration, so we split the code related to color and effect out of the main function.

color: restore _style global after debugcolor ran Before this change, running 'debugcolor' would destroy all color style for the rest of the process life. We now properly backup and restore the variable content. Using a global variable is sketchy in general and could probably be removed. However, this is a quest for another adventure.

color: add basic documentation to 'debugcolor' This does not hurt.

tests: merge 'test-push-hook-lock.t' into 'test-push.t' That test file is very small and is merge with the new 'test-push.t'. No logic is changed. We don't register this as a copy because is actually a "ypoc" merging two file together without replacing the destination and Mercurial cannot express that.

tests: merge 'test-push-validation.t' into 'test-push.t' That test file is very small and is merge with the new 'test-push.t'. No logic is changed but repository name are update to avoid collision. We don't register this as a copy because is actually a "ypoc" merging two file together without replacing the destination and Mercurial cannot express that.

test: rename 'test-push-r.t' to 'test-push.t' We do not have a simple test for 'hg push' but we have multiple tiny tests for various aspect of it. We'll unify them into a single file, and we start with 'test-push-r.t'. The code is unchanged but we renamed the repository used to avoid collision with other tests we'll import in coming changesets. Test timing for the record: start end cuser csys real Test 1.850 2.640 0.650 0.090 0.790 test-push-validation.t 2.640 3.520 0.760 0.090 0.880 test-push-hook-lock.t 0.000 1.850 1.560 0.210 1.850 test-push-r.t

tests: simplify command script in 'test-push-r.t' I came across this code by chance. The script of this test is a bit messy with a lot of unnecessary intermediate commands. We simplify the script and unify repository access through '-R'. In the process the update after the unbundle is dropped as it does not add anything to the tests.

py3: use encoding.environ in ui.py Using source transformer we add b'' everywhere. So there are no chances that those bytes string will work with os.environ on Py3 as that returns a dict of unicodes. We are relying on the errors, even though no error is raised even in future, these pieces of codes will tend to do wrong things. if statements can result in wrong boolean and certain errors can be raised while using this piece of code. Let's not wait for them to happen, fix what is wrong. If this patch goes in, I will try to do it for all the cases. Leaving it as it is buggy.

py3: make scmposix.userrcpath() return bytes We are making sure that we deal with bytes as much we can. This is a part of fixing functions so that they return bytes if they have to. Used encoding.environ to return bytes. After this patch, scmposix.userrcpath() returns bytes and scmutil.osrcpath() will also return bytes if the platform is posix. Functions is scmposix returns bytes on Python 3 now.

adjustlinkrev: remove unnecessary parameters Since adjustlinkrev has "self", and is a method of a filectx object, it does not need path, filelog, filenode. They can be fetched from the "self" easily.

repair: make strip() return backup file path narrowhg wants to strip some commits and then re-apply them after applying another bundle. Having repair.strip() return the bundle path will be helpful for it.

rebase: check for conflicts before continuing When there are unresolved merge conflicts, there is no reason to make the user wait for rebase to process all of the already rebased commits just to complain that it cannot do anything. Abort early.

cmdutil: refactor checkunresolved localrepo.commit had code to check for unresolved merge conflicts, it would be helpful for at least rebase to be able to use that code without calling commit().

rebase: rename merge to mergemod

context: make sure __str__ works, also when there is no _changectx Before, it could crash when trying to print the wrong kind of object at the wrong time.

largefiles: clarify variable name holding file mode A follow-up to c01acee367ec. 'st' sounds like the whole stat result while 'mode' is a better name for the actual file mode.

changegroup: use changelogrevision() Using offsets for accessing changelog entries isn't very readable. As a bonus, changelog.changelogrevision() also accepts a revision, so we don't need to perform the inline node resolution either.

changegroup: cache changelog and manifestlog outside of loop History has taught us that repo.changelog can add significant overhead to loops. So cache the changelog instance outside of the loop to avoid the lookup. While we're here, do the same for manifestlog, since each loop would otherwise initialize a new manifestlog instance.

make: targets for building packages for ubuntu yakkety Ubuntu 16.10 Yakkety Yak is out. Let's support it.

util: put compression code next to each other ctxmanager was injecting itself between the compression and decompression code. Let's restore some order.

hgweb: make log streams compatible with command server Even though it would be useless to start a web server by a command server, it should be doable in principle. Also, we can't use sys.stdout/err directly on Python 3 because they are unicode streams.

cmdserver: write channel header and payload by a single write() call This makes a channeledoutput thread-safe as long as the underlying fwrite() is thread-safe. Both POSIX and Windows implementations are documented as MT-safe. MT-safety is necessary to use ui.fout and ui.ferr in hgweb.

histedit: use ui.fin to read commands from stdin stdin is wrapped by channeledinput in command-server session.

convert: have debugsvnlog obtain standard streams from ui This will help porting to Python 3, where sys.stdin/out/err are unfortunately unicode streams so we can't use them directly.

convert: remove superfluous setbinary() calls from debugsvnlog a3fe91b4f6eb made standard streams set to binary mode globally.

tests: explicitly use ls profiler In preparation for making the statprof profiler the default.

statprof: pass data structure to display functions Currently, statprof maintains a global "state" variable that is used by several functions. Global variables hinder adaptability of code. So pass state to display functions so we can make changes to how "state" works in future patches.

statprof: use print function

statprof: use absolute_imports As part of this, we modify import order to satisfy our import checker.

statprof: require paths to save or load profile data Upstream appears to aggressively save statprof data in a well-defined home directory path. Change the code to not do that. We also change file saving to fail if an error has occurred instead of silently failing. Callers can catch the exception. This behavior is more suitable for a generic "library" module.

statprof: fix flake8 warnings My local flake8 hook informed me of these warnings in the upstream code. Fix them.

statprof: vendor statprof.py Vendored from https://bitbucket.org/facebook/hg-experimental changeset 73f9db47ae5a1a9fa29a98dfe92d557ad51234c3 without modification. This introduces a number of code style violations. The file already has the magic words to skip test-check-code.t. I'll make additional changes to clean up the test-check-py3-compat.t warnings and to change some behavior in the code that isn't suitable for general use. test-check-commit.t also complains about numerous things. But there's nothing we can do if we're importing as-is.

merge with stable

Added signature for changeset eab274469952

Added tag 4.0 for changeset eab274469952

merge with i18n

i18n-ja: synchronized with 69ffbbe73dd0

tag: clarify warning about making a tag on a branch head Currently the warning is ambiguous about whether the new tag (possibly specified via --rev) is being added on a branch head or whether the working directory is based on a branch head. Clarify the error message to eliminate this ambiguity.

contrib: check reference to old selenic.com domain Now, all URL in Mercurial source tree should refer mercurial-scm.org domain instead of selenic.com. *.po files are ignored in this patch, because they might contain msgid/msgstr coming from old source files. This ignorance seems safe enough, because such msgstr should be ignored at runtime, because: - msgid corresponded to it should be invalid, or - msgstr itself should be marked as fuzzy at synchronized to recent hg.pot If any additional examination for *.po files is needed in the future, let i18n/check-translation.py achieve such examination. BTW, some binary files (e.g. *.png) are meaningless for checking reference to old domain in this patch, but aren't ignored like as *.po files, because excluding multiple suffixes is difficult for regexp matching.

check-code: discard filtering result of previous check for independence Before this patch, check-code.py applies filtering on the file content, to which filtering of previous check is already applied. This might hide issues, which should be detected by a subsequent check in "checks" list. Fortunately, this problem hasn't appeared, because there is no overlapping of filename matching (examined in the order below). 1. *.py or *.cgi 2. test-* (not *.t suffix) 3. *.c or *.h 4. *.t 5. *.txt 6. *.tmpl For example, adding a test, which wants to examine raw comment text in *.py files, at the end of current "checks" list doesn't work as expected, because a filter for *.py files normalizes comment text in them. Putting such test at the beginning of "checks" list also resolves this problem, but such dependence on the order decreases maintainability of check-code.py itself. This patch discards filtering result of previous check at the beginning of each checks, for independence of each checks.

help: replace selenic.com by mercurial-scm.org in man pages Source code repository and mailing list services have been already migrated to mercurial-scm.org domain.

help: replace selenic.com by mercurial-scm.org in command examples Source code repository service of Mercurial itself has been already migrated to mercurial-scm.org domain.

hghave: fix 'rmcwd' to ensure temporary directory is removed On platforms where cwd can't be removed, it should try rmdir() after chdir to the original cwd.

httppeer: make __del__ access to self.urlopener more safe Some errors could in some cases show unfortunate scary and confusing warnings from the httppeer delstructors: abort: nodename nor servname provided, or not known Exception AttributeError: "'httpspeer' object has no attribute 'urlopener'" in <bound method httpspeer.__del__ of <mercurial.httppeer.httpspeer object at 0x106e1f5d0>> ignored``` To mute that, take 7b15dd9125b3 to the next level and use getattr in __del__.

tests: test preserving execbit changes at amending only on execbit platform

tests: put temporary file outside the working directory for test portability test-largefiles-update.t creates temporary file exec-bit.patch inside the working directory for no-execbit platform specific test, but subsequent tests aren't aware of it. On execbit platform, subsequent tests can run successfully, because exec-bit.patch isn't created. But on no-execbit platform, this temporary file makes subsequent tests show "? exec-bit.patch" at each "hg status".

tests: avoid quoting of commit messages for test portability journal extension uses util.shellquote() to record command line, but result of it depends on runtime platform: double quotation is used on Windows and OpenVMS, but single quotation is used otherwise. test-journal-share.t sometimes specifies commit messages including white space on command line. It makes journal output depend on runtime platform, but commit message itself isn't important in this test case.

tests: use basic format code "%Y" instead of "%s" for test portability On Windows, strftime() doesn't support format code "%s", and it causes "invalid format string" error. https://msdn.microsoft.com/en-us/library/fe06s4ak.aspx test-command-template.t examines not seconds value in UTC, but arithmetic calculation. Therefore, using format code "%Y" instead of "%s" should be reasonable. FYI: - Python standard library reference doesn't list "%s" up in format code list required for "C standard (1989 version)", even though it also mentions that additional format codes are required for "C standard (1999 version)" https://docs.python.org/2.7/library/datetime.html#strftime-and-strptime-behavior - The Open Group Base Specifications Issue 7 (IEEE Std 1003.1-2008, 2016 Edition) doesn't require strftime to support format code "%s" http://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html - "man strftime" of (Open/Oracle) Solaris and Mac OS X (= UNIX certified OSs) describes about format code "%s"

tests: add test-commit-interactive-curses.t "require tic" for test portability Standard library of Python on Windows doesn't have curses module.

tests: use "?" to glob both ":" and ";" in output for test portability If environment variable looks like PATH or so (e.g. any of components joined by ":" contains "/"), ":" in it is replaced with ";" by MinGW at spawning Windows native process, to follow path concatenation style of Windows. Therefore, "bundle:../full.hg" is converted into "bundle;..\full.hg" on MinGW. Difference between "/" and "\" is automatically ignored by "(glob)", but difference between ":" and ";" should be globed explicitly.

tests: invoke printenv.py via sh -c for test portability On Windows platform, invoking printenv.py directly via hook is problematic, because: - unless binding between *.py suffix and python runtime, application selector dialog is displayed, and running test is blocked at each printenv.py invocations - it isn't safe to assume binding between *.py suffix and python runtime, because application binding is easily broken For example, installing IDE (VisualStudio with Python Tools, or so) often requires binding between source files and IDE itself. This patch invokes printenv.py via sh -c for test portability. This is a kind of follow up for d19787db6fe0, which eliminated explicit "python" for printenv.py. There are already other 'sh -c "printenv.py"' in *.t files, and this fix should be reasonable. This changes were confirmed in cases below: - without any application binding for *.py suffix - with binding between *.py suffix and VisualStudio This patch also replaces "echo + redirection" style with "heredoc" style, because: - hook command line is parsed by cmd.exe as shell at first, and - single quotation can't quote arguments on cmd.exe, therefore, - "printenv.py foobar" should be quoted by double quotation, but - nested quoting (or tricky escaping) isn't readable

largefiles: handle that a found standin file doesn't exist when removing it I somehow ended up in a situation where hg crashed on an unlink I introduced in 328545c7d8a1. I don't know how it happened and can't reproduce it. It seems like it only can happen when the file is removed between the time of check in a working directory context walk that finds a standin file, and the time of use when we try to remove it because the corresponding largefile doesn't exist. But better safe than sorry: replace the plain unlink with unlinkpath with ignoremissing=True. That will also remove remaining empty directories, which arguably is more correct.

templater: use unfiltered changelog to calculate shortest() at constant time cl._partialmatch() can be pretty slow if hidden revisions are involved. This patch cancels the slowdown introduced by the previous patch by using an unfiltered changelog, which means shortest(node) isn't always the shortest. The result isn't perfect, but seems okay as long as shortest(node) is short enough to type and can be used as an identifier. (with hidden revisions) % hg log -R hg-committed -r0:20000 -T '{node|shortest}\n' --time > /dev/null (.^^) time: real 1.530 secs (user 1.480+0.000 sys 0.040+0.000) (.^) time: real 43.080 secs (user 43.060+0.000 sys 0.030+0.000) (.) time: real 1.680 secs (user 1.650+0.000 sys 0.020+0.000)

templater: do not use index.partialmatch() directly to calculate shortest() cl.index.partialmatch() isn't a drop-in replacement for cl._partialmatch(). It has no knowledge about hidden revisions, and it raises ValueError if a node shorter than 4 chars is given. Instead, use index.partialmatch() through cl._partialmatch(), which has no such problems and gives the identical result with/without --pure. The test output was sampled with --pure without this patch, which shows the most correct result. However, we'll need to switch to using an unfiltered changelog because _partialmatch() of a filtered changelog can be an order of magnitude slower. (with hidden revisions) % hg log -R hg-committed -r0:20000 -T '{node|shortest}\n' --time > /dev/null (.^) time: real 1.530 secs (user 1.480+0.000 sys 0.040+0.000) (.) time: real 43.080 secs (user 43.060+0.000 sys 0.030+0.000)

tests: run "cwd was removed" test only if cwd can actually be removed On some platforms, cwd can't be removed. In which case, util.unlinkpath() continues with no error since the failure of directory removal isn't critical. So it doesn't make sense to run the test added by 90a6c18a7c1d on those platforms. OTOH, we need to run the test in test-rebase-scenario-global.t since the repository is referenced after that.

merge: avoid superfluous filemerges when grafting through renames (issue5407) This is a fix for a regression introduced by the patches for issue4028. The test changes are due to us doing fewer _checkcopies searches now, which makes some test outputs revert to the pre-issue4028 behavior. That issue itself remains fixed, we only skip copy tracing for files where it isn't relevant. As a nice side effect, this makes copy detection much faster when tracing backwards through lots of renames.

sslutil: guard against broken certifi installations (issue5406) Certifi is currently incompatible with py2exe; the Python code for certifi gets included in library.zip, but not the cacert.pem file - and even if it were included, SSLContext can't load a cacert.pem file from library.zip. This currently makes it impossible to build a standalone Windows version of Mercurial. Guard against this, and possibly other situations where a module with the name "certifi" exists, but is not usable.

revset: don't cache abstractsmartset min/max invocations infinitely There was a "leak", apparently introduced in ab66c1dee405. When running: hg = hglib.open('repo') while True: hg.log("max(branch('default'))") all filteredset instances from branch() would be cached indefinitely by the @util.cachefunc annotation on the max() implementation. util.cachefunc seems dangerous as method decorator and is barely used elsewhere in the code base. Instead, just open code caching by having the min/max methods replace themselves with a plain lambda returning the result.

merge with i18n

i18n-pt_BR: synchronized with 7b428b00a1d4

dirstate: fix debug.dirstate.delaywrite to use the new "now" after sleeping It seems like the a regression has sneaked into debug.dirstate.delaywrite in 6c6b48aca328. It would sleep until no files were modified "now" any more, but when writing the dirstate it would use the old "now" and still mark files as 'unset' instead of recording the timestamp that would make the file show up as clean instead of unknown. Instead of getting a new "now" from the file system, we trust the computed end time as the new "now" and thus cause the actual modification time to be writiten to the dirstate. debug.dirstate.delaywrite is undocumented and only used in test-largefiles-update.t . All tests seems to work fine for me without debug.dirstate.delaywrite . Perhaps because it not really worked as intended without the fix in this patch, and code and tests thus have evolved to do fine without it? It could thus perhaps make sense to drop usage of this setting in the tests. That could speed the test up a bit. This functionality (or something very similar) can however apparently be very convenient in setups where checking dirty-ness is expensive - such as when using large files and have slow file filesystems or are CPU constrained. Now it works and we can try it. (But ideally, for the largefile use case, it should probably only delay lfdirstate writes - not ordinary dirstate.)

tests: fix test-casefolding.t The message had changed, but the test was not updated. This test does not run on Linux, but failed on my Mac.

commands: print security protocol support in debuginstall Over the past week I've had to instruct multiple people to run Python code to query the ssl module to see what TLS protocol support is present. I think it would be useful for `hg debuginstall` to print this info to make it easier to access and debug why Mercurial is complaining about using an insecure TLS 1.0 protocol. Ideally we'd also print the path to the CA cert bundle. But the APIs for querying that in sslutil can emit warnings, making it slightly more difficult to integrate into `hg debuginstall`. That work will have to wait for another day.

manifest: make treemanifestctx store the repo Same as in the last commit, the old treemanifestctx stored a reference to the revlog. If the inmemory revlog became invalid, the ctx now held an old copy and would be incorrect. To fix this, we need the ctx to go through the manifestlog for each access. This is the same pattern that changectx already uses (it stores the repo, and accesses commit data through self._repo.changelog).

manifest: make manifestctx store the repo The old manifestctx stored a reference to the revlog. If the inmemory revlog became invalid, the ctx now held an old copy and would be incorrect. To fix this, we need the ctx to go through the manifestlog for each access. This is the same pattern that changectx already uses (it stores the repo, and accesses commit data through self._repo.changelog).

manifest: make manifestlog a storecache The old @property on manifestlog was broken. It meant that we would always recreate the manifestlog instance, which meant the cache was never hit. Since we'll eventually remove repo.manifest and make manifestlog the only property, let's go ahead and make manifestlog the @storecache property, have manifestlog own the manifest instance, and have repo.manifest refer to it via manifestlog. This means all accesses go through repo.manifestlog, which is now invalidated correctly.

manifest: move manifest creation to a helper function A future patch will be moving manifest creation to be inside manifestlog as part of improving our cache guarantees. bundlerepo and unionrepo currently rely on being able to hook into manifest creation, so let's temporarily move the actual manifest creation to a helper function for them to intercept. In the future manifest.manifest() will disappear entirely and this can disappear.

Added signature for changeset 438173c41587

Added tag 4.0-rc for changeset 438173c41587

merge default into stable for 4.0 code freeze

merge with i18n

i18n-pt_BR: synchronized with 149433e68974

changegroup: increase write buffer size to 128k By default, Python defers to the operating system for choosing the default buffer size on opened files. On my Linux machine, the default is 4k, which is really small for 2016. This patch bumps the write buffer size when writing changegroups/bundles to 128k. This matches the 128k read buffer we already use on revlogs. It's worth noting that this only impacts when writing to an explicit file (such as during `hg bundle`). Buffers when writing to bundle files via the repo vfs or to a temporary file are not impacted. When producing a none-v2 bundle file of the mozilla-unified repository, this change caused the number of write() system calls to drop from 952,449 to 29,788. After this change, the most frequent system calls are fstat(), read(), lseek(), and open(). There were 2,523,672 system calls after this patch (so a net decrease of ~950k is statistically significant). This change shows no performance change on my system. But I have a high-end system with a fast SSD. It is quite possible this change will have a significant impact on network file systems, where extra network round trips due to excessive I/O system calls could introduce significant latency.

changegroup: skip delta when the underlying revlog do not use them Revlog can now be configured to store full snapshot only. This is used on the changelog. However, the changegroup packing was still recomputing deltas to be sent over the wire. We now just reuse the full snapshot directly in this case, skipping delta computation. This provides use with a large speed up(-30%): # perfchangegroupchangelog on mercurial ! wall 2.010326 comb 2.020000 user 2.000000 sys 0.020000 (best of 5) ! wall 1.382039 comb 1.380000 user 1.370000 sys 0.010000 (best of 8) # perfchangegroupchangelog on pypy ! wall 5.792589 comb 5.780000 user 5.780000 sys 0.000000 (best of 3) ! wall 3.911158 comb 3.920000 user 3.900000 sys 0.020000 (best of 3) # perfchangegroupchangelog on mozilla central ! wall 20.683727 comb 20.680000 user 20.630000 sys 0.050000 (best of 3) ! wall 14.190204 comb 14.190000 user 14.150000 sys 0.040000 (best of 3) Many tests have to be updated because of the change in bundle content. All theses update have been verified. Because diffing changelog was not very valuable, the resulting bundle have similar size (often a bit smaller): # full bundle of mozilla central with delta: 1142740533B without delta: 1142173300B So this is a win all over the board.

revlog: make 'storedeltachains' a "public" attribute The next changeset will make that attribute read by the changegroup packer. We make it "public" beforehand.

manifest: don't store None in fulltextcache When we read a value from fulltextcache, we expect it to be an array, so we should not store None in it. Found while working on narrowhg.

copies: improve assertions during copy recombination - Make sure there is nothing to recombine in non-graftlike scenarios - More pythonic assert syntax

treemanifest: fix bad argument order to treemanifestctx Found by running tests with _treeinmem (both of them) modified to be True.

wireproto: compress data from a generator Currently, the "getbundle" wire protocol command obtains a generator of data, converts it to a util.chunkbuffer, then converts it back to a generator via the protocol's groupchunks() implementation. For the SSH protocol, groupchunks() simply reads 4kb chunks then write()s the data to a file descriptor. For the HTTP protocol, groupchunks() reads 32kb chunks, feeds those into a zlib compressor, emits compressed data as it is available, and that is sent to the WSGI layer, where it is likely turned into HTTP chunked transfer chunks as is or further buffered and turned into a larger chunk. For both the SSH and HTTP protocols, there is inefficiency from using util.chunkbuffer. For SSH, emitting consistent 4kb chunks sounds nice. However, the file descriptor it is writing to is almost certainly buffered. That means that a Python .write() probably doesn't translate into exactly what is written to the I/O layer. For HTTP, we're going through an intermediate layer to zlib compress data. So all util.chunkbuffer is doing is ensuring that the chunks we feed into the zlib compressor are of uniform size. This means more CPU time in Python buffering and emitting chunks in util.chunkbuffer but fewer function calls to zlib. This patch introduces and implements a new wire protocol abstract method: compresschunks(). It is like groupchunks() except it operates on a generator instead of something with a .read(). The SSH implementation simply proxies chunks. The HTTP implementation uses zlib compression. To avoid duplicate code, the HTTP groupchunks() has been reimplemented in terms of compresschunks(). To prove this all works, the "getbundle" wire protocol command has been switched to compresschunks(). This removes the util.chunkbuffer from that command. Now, data essentially streams straight from the changegroup emitter to the wire, possibly through a zlib compressor. Generators all the way, baby. There were slim to no performance changes on the server as measured with the mozilla-central repository. This is likely because CPU time is dominated by reading revlogs, producing the changegroup, and zlib compressing the output stream. Still, this brings us a little closer to our ideal of using generators everywhere.

revset: optimize for destination() being "inefficient" destination() will scan through the whole subset and read extras for each revision to get its source.

copies: make _checkcopies handle copy sequences spanning the TCA (issue4028) When working in a rotated DAG (for a graftlike merge), there can be files that are renamed both between the base and the topological CA, and between the TCA and the endpoint farther from the base. Such renames span the TCA (and thus need both passes of _checkcopies to be fully detected), but may not necessarily be divergent. Make _checkcopies return "incomplete copies" and "incomplete divergences" in this case, and let mergecopies recombine them once data from both passes of _checkcopies is available. With this patch, all known cases involving renames and grafts pass. (Developed together with Pierre-Yves David)

checkcopies: add logic to handle remotebase As the two _checkcopies passes' ranges are separated by tca, not base, only one of the two passes will actually encounter the base. Pass "remotebase" to the other pass to let it know not to expect passing over the base. This is required for handling a few unusual rename cases.

mergecopies: add logic to process incomplete data We first combine incomplete copies on the two sides of the topological CA into complete copies. Any leftover incomplete copies are then combined with the incomplete divergences to reconstruct divergences spanning over the topological CA. Finally we promote any divergences falsely flagged as incomplete to full divergences. Right now, there is nothing generating incomplete copy/divergence data, so this code does nothing. Changes to _checkcopies to populate these dicts are coming later in this series.

checkcopies: handle divergences contained entirely in tca::ctx During a graftlike merge, _checkcopies runs from ctx to tca, possibly passing over the merge base. If there is a rename both before and after the base, then we're actually dealing with divergent renames. If there is no rename on the other side of tca, then the divergence is contained entirely in the range of one _checkcopies invocation, and should be detected "in the loop" without having to rely on the other _checkcopies pass.

update: enable copy tracing for backwards and non-linear updates As a followup to the issue4028 series, this fixes a variant of the issue that can occur when updating with uncommited local changes. The duplicated .hgsub warning is coming from wc.dirty(). We would previously skip this call because it's only relevant when we're going to perform copy tracing, which we didn't do before. The change to the update summary line is because we now treat the rename as a proper rename (which counts as a change), rather than an add+delete pair (which counts as a change and a delete).

bashcompletion: allow skipping completion for 'hg status' On systems with large repositories and slow disks, the calls to 'hg status' make autocomplete annoyingly slow. This fix makes it possible to avoid the slowdown.

tests: add more test coverage of phase changes when pushing Prepare for test coverage of phase updates with future push --readonly option, both with and without actually pushing changesets.

mergecopies: invoke _computenonoverlap for both base and tca during merges The algorithm of _checkcopies can only walk backwards in the DAG, never forward. Because of this, the two _checkcopies patches need to run from their respective endpoints to the TCA to cover the entire subgraph where the merge is being performed. However, detection of files new in both endpoints, as well as directory rename detection, need to run with respect to the merge base, so we need lists of new files both from the TCA's and the merge base's viewpoint to correctly detect renames in a graft-like merge scenario. (Series reworked by Pierre-Yves David)

copies: make it possible to distinguish betwen _computenonoverlap invocations _computenonoverlap needs to be invoked twice during a graft, and debugging messages should be distinguishable between the two invocations

copies: make _checkcopies handle simple renames in a rotated DAG This introduces a distinction between "merge base" and "topological common ancestor". During a regular merge, these two are identical. Graft, however, performs a merge in a rotated DAG, where the merge base will not be a common ancestor at all in the original DAG. To correctly find copies in case of a graft, we need to take both the merge base and the topological CA into account, and track any renames between them in reverse. Fortunately we can detect this in advance, see comment in the code about "backwards". This patch only supports finding non-divergent renames contained entirely between the merge base and the topological CA. Further patches are coming to support more complex cases. (Pierre-Yves David was involved in the cleanup of this patch.)

copies: compute a suitable TCA if base turns out to be unsuitable This will be used later in an update to _checkcopies. (Pierre-Yves David was involved in the cleanup of this patch.)

copies: detect graft-like merges Right now, nothing changes as a result of this, but we want to handle grafts differently from ordinary merges later. (Series developed together with Pierre-Yves David)

tests: introduce tests for grafting through renames These cover all currently known cases of renames being grafted, or changes being grafted through renames. Right now, most of these cases are broken. Later patches in this series will make them behave correctly. The testcases heavily rely on each other, which would make it very difficult to separate them and add them one-by-one for each case fixed by a patch. Separating them should perhaps be a 4.1 task, if it doesn't slow down the tests too much. (Developed together with Pierre-Yves David)

largefiles: fix 'deleted' files sometimes persistently appearing with R status A code snippet that has been around since largefiles was introduced was wrong: Standins no longer found in lfdirstate has *not* been removed - they have probably just been deleted ... or not created. This wrong reporting did that 'up -C' didn't undo the change and didn't sync the two dirstates. Instead of reporting such files as removed, propagate the deletion to the standin file and report the file as deleted.

largefiles: more safe handling of interruptions while updating modifications Largefiles are fragile with the design where dirstate and lfdirstate must be kept in sync. To be less fragile, mark all clean largefiles as unsure ("normallookup") before updating standins. After standins have been updated and we know exactly which largefile standins actually was changed, mark the unchanged largefiles back to clean ("normal"). This will make the failure mode more safe. If interrupted, the next command will continue to perform extra hashing of all largefiles. That will do that all largefiles that are out of sync with their standin will be marked dirty and they will show up in status and can be cleaned with update --clean.

largefiles: test coverage of fatal interruption of update Test using existing changesets in a clean working directory, revealing problems with files that don't show up as modified or do show up as removed when they just not have been written yet.

checkcopies: add a sanity check against false-positive copies When grafting a copy backwards through a rename, a copy is wrongly detected, which causes the graft to be applied inappropriately, in a destructive way. Make sure that the old file name really exists in the common ancestor, and bail out if it doesn't. This fixes the aggravated case of bug 5343, although the basic issue (failure to duplicate the copy information) still occurs.

exchange: refactor APIs to obtain bundle data (API) Currently, exchange.getbundle() returns either a cg1unpacker or a util.chunkbuffer (in the case of bundle2). This is kinda OK, as both expose a .read() to consumers. However, localpeer.getbundle() has code inferring what the response type is based on arguments and converts the util.chunkbuffer returned in the bundle2 case to a bundle2.unbundle20 instance. This is a sign that the API for exchange.getbundle() is not ideal because it doesn't consistently return an "unbundler" instance. In addition, unbundlers mask the fact that there is an underlying generator of changegroup data. In both cg1 and bundle2, this generator is being fed into a util.chunkbuffer so it can be re-exposed as a file object. util.chunkbuffer is a nice abstraction. However, it should only be used "at the edges." This is because keeping data as a generator is more efficient than converting it to a chunkbuffer, especially if we convert that chunkbuffer back to a generator (as is the case in some code paths currently). This patch refactors exchange.getbundle() into exchange.getbundlechunks(). The new API returns an iterator of chunks instead of a file-like object. Callers of exchange.getbundle() have been updated to use the new API. There is a minor change of behavior in test-getbundle.t. This is because `hg debuggetbundle` isn't defining bundlecaps. As a result, a cg1 data stream and unpacker is being produced. This is getting fed into a new bundle20 instance via bundle2.writebundle(), which uses a backchannel mechanism between changegroup generation to add the "nbchanges" part parameter. I never liked this backchannel mechanism and I plan to remove it someday. `hg bundle` still produces the "nbchanges" part parameter, so there should be no user-visible change of behavior. I consider this "regression" a bug in `hg debuggetbundle`. And that bug is captured by an existing "TODO" in the code to use bundle2 capabilities.

mergecopies: rename 'ca' to 'base' This variable was named after the common ancestor. It is actually the merge base that might differ from the common ancestor in the graft case. We rename the variable before a larger refactoring to clarify the situation. Similar rename was also applied to 'checkcopies' in a prior changeset.

copies: move variable document from checkcopies to mergecopies It appears that 'mergecopies' is the function consuming these data so we move the documentation there.

checkcopies: pass data as a dictionary of dictionaries more are coming

checkcopies: move 'movewithdir' initialisation right before its usage The 'movewithdir' had a lot of related logic all around the 'mergecopies'. However it is actually never containing anything until the very last loop in that function. We move the (simplified) variable definition there for clarity

cmdutil: satisfy expections in dirstateguard.__del__, even if __init__ fails Python "delstructors" are terrible - this one because it assumed that __init__ had completed before it was called. That would not necessarily be the case if the repository was read only or broken and saving the dirstate thus failed in unexpected ways. That could give confusing warnings about missing '_active' after failures. To fix that, make sure all member variables are "declared" before doing anything that possibly could fail. [Famous last words.]

util: increase filechunkiter size to 128k util.filechunkiter has been using a chunk size of 64k for more than 10 years, also in years where Moore's law still was a law. It is probably ok to bump it now and perhaps get a slight win in some cases. Also, largefiles have been using 128k for a long time. Specifying that size multiple times (or forgetting to do it) seems a bit stupid. Decreasing it to 64k also seems unfortunate. Thus, we will set the default chunksize to 128k and use the default everywhere.

largefiles: always use filechunkiter when iterating files Before, we would sometimes use the default iterator over large files. That iterator is line based and would add extra buffering and use odd chunk sizes which could give some overhead. copyandhash can't just apply a filechunkiter as it sometimes is passed a genuine generator when downloading remotely.

revset: for x^2, do not take null as a valid p2 revision Since we don't count null p2 revision as a parent, x^2 should never return null even if null is explicitly populated.

revset: make follow() reject more than one start revisions Taking only the last revision is inconsistent because ancestors(set) follows all revisions given, and theoretically follow(startrev=set) == ancestors(set). I'm planning to add a support for multiple start revisions, but that won't fit to the 4.0 time frame. So reject multiple revisions now to avoid future BC. len(revs) might be slow if revs were large, but we don't care since a valid revs should have only one element.

bundle2: only emit compressed chunks if they have data This is similar to 58467204cac0. Not all calls into the compressor return compressed data, as the compressor may buffer compressed output internally. It is cheaper to check for empty chunks than to send empty chunks through the generator. When generating a gzip-v2 bundle of the mozilla-unified repo, this change results in 50,093 empty chunks not being sent through the generator (out of 1,902,996 total input chunks).

color: add some documentation for custom terminfo codes

color: debugcolor should emit the user-defined colors This also fixes a long-standing bug that reversed the sense of the color name and the label used to print it, which was never relevant before.

color: ignore effects missing from terminfo If terminfo mode is in effect, and an effect is used which is missing from the terminfo database, simply silently ignore the request, leaving the output unaffected rather than causing a crash.

color: allow for user-configurable terminfo codes for effects If the entry in the terminfo database for your terminal is missing some attributes, it should be possible to create them on the fly without resorting to just making them a color. This change allows you to have [color] terminfo.<effect> = <code> where <effect> might be something like "dim" or "bold", and <code> is the escape sequence that would otherwise have come from a call to tigetstr(). If an escape character is needed, use "\E". Any such settings will override attributes that are present in the terminfo database.