crecord: use scmutil.termsize()

scmutil: extend termwidth() to return terminal height, renamed to termsize() It appears crecord.py has its own termsize() function. I want to get rid of it. The fallback height is chosen from the default of cmd.exe on Windows, and VT100 on Unix.

scmutil: clarify that we explicitly do termwidth - 1 on Windows I was a bit confused since we didn't add 1 to the width, which is different from the example shown in StackOverflow. http://stackoverflow.com/a/12642749

scmutil: remove superfluous indent from termwidth()

scmutil: narrow ImportError handling in termwidth() The array module must exist. It's sufficient to suppress the ImportError of termios. Also salvaged the comment why we have to handle AttributeError, from 7002bb17cc5e.

scmutil: make termwidth() obtain stdio from ui I'm getting rid of direct sys.stderr|out|in references so Py3 porting will be slightly easier.

scmutil: move util.termwidth() I'm going to get rid of sys.stderr|out|in references from posix.termwidth(). In order to do that, termwidth() needs to take a ui, but functions in util.py shouldn't depend on a ui object. So moves termwidth() to scmutil.py.

bdiff: don't check border condition in loop `plast = a + len - 1`. So, this "for" loop iterates from "a" to "plast", inclusive. So, `p == plast` can only be true on the final iteration of the loop. So checking for it on every loop iteration is wasteful. This patch simply decreases the upper bound of the loop by 1 and adds an explicit check after iteration for the `p == plast` case. We can't simply add 1 to the initial value for "i" because that doesn't do the correct thing on empty input strings. `perfbdiff -m 3041e4d59df2` on the Firefox repo becomes significantly faster: ! wall 0.072763 comb 0.070000 user 0.070000 sys 0.000000 (best of 100) ! wall 0.053221 comb 0.060000 user 0.060000 sys 0.000000 (best of 100) For the curious, this code has its origins in 8b067bde6679, which is the changeset that introduced bdiff.c in 2005. Also, GNU diffutils is able to perform a similar line-based diff in under 20ms. So there's likely more perf wins to be found in this code. One of them is the hashing algorithm. But it looks like mpm spent some time testing hash collisions in d0c48891dd4a. I'd like to do the same before switching away from lyhash, just to be on the safe side.

perf: add perfbdiff bdiff shows up a lot in profiling. I think it would be useful to have a perf command that runs bdiff over and over so we can find hot spots.

help: show help for disabled extensions (issue5228) This patch does not exactly solve issue5228 but it results in a better condition on this issue. For disabled extensions, we used to parse the module and get the first occurrences of docstring and then return the first line of that as an introductory heading of extension. This is what we get today. This patch returns the whole docstring of the module as a help for extension, which is more informative. There are some modules which don't have much docstring at top level except the heading so those are unaffected by this change. To follow the existing trend of showing commands either we have to load the extension or have a very ugly parsing method which don't even assure correctness.

py3: make scmutil.rcpath() return bytes This patch make sure scmutil.rcpath() returns bytes independent of which platform is used on Python 3. If we want to change type for windows we can just conditionalize the return variable.

py3: use pycompat.ossep at certain places Certain instances of os.sep has been converted to pycompat.ossep where it was sure to use bytes only. There are more such instances which needs some more attention and will get surely.

py3: have pycompat.ospathsep and pycompat.ossep We needed bytes version of os.sep and os.pathsep in py3 as they return unicodes.

py3: add a bytes version of os.name os.name returns unicodes on py3. Most of our checks are like os.name == 'nt' Because of the transformer, on the right hand side we have b'nt'. The condition will never satisfy even if os.name returns 'nt' as that will be an unicode. We either need to encode every occurence of os.name or have a new variable which is much cleaner. Now we have pycompat.osname. There are around 53 occurences of os.name in the codebase which needs to be replaced by pycompat.osname to support Python 3.

py3: make util.datapath a bytes variable In this patch we make util.datapath a bytes variable, but we have to pass a unicode to gettext.translation otherwise it will cry. Used pycompat.fsdecode() to decode it back to unicode as it was converted to bytes using pycompat.fsencode().

py3: add os.fsdecode() as pycompat.fsdecode() We need to use os.fsdecode() but this was not present in Python 2. So added the function in pycompat.py

statprof: return state from stop() I don't like global variables. Have stop() return the captured state so callers can pass data to the display function.

hghave: check darcs version more strictly test-convert-darcs.t suddenly started failing on my Debian sid machine. The reason was Darcs was upgraded from 2.12.0 to 2.12.4 so the original pattern got to match the last two digits. Fix the pattern to match 2.2+.

tests: silence output of darcs command It appears darcs is more verbose by default these days. I got test failure with Darcs 2.12.4.

manifest: remove manifest.readshallowdelta This removes manifest.readshallowdelta and converts its one consumer to use manifestlog instead.

manifest: get rid of manifest.readshallowfast This removes manifest.readshallowfast and converts it's one user to use manifestlog instead.

manifest: add shallow option to treemanifestctx.readdelta and readfast The old manifest had different functions for performing shallow reads, shallow readdeltas, and shallow readfasts. Since a lot of the code is duplicate (and since those functions don't make sense on a normal manifestctx), let's unify them into flags on the existing readdelta and readfast functions. A future diff will change consumers of these functions to use the manifestctx versions and will delete the old apis.

manifest: change manifestlog mancache to be directory based In the last patch we added a get() function that allows fetching directory level treemanifestctxs. It didn't handle caching at directory level though, so we need to change our mancache to support multiple directories.

manifest: add manifestlog.get to obtain subdirectory instances Previously manifestlog only allowed obtaining root level manifests. Future patches will need direct access to subdirectory manifests as part of changegroup creation, so let's add a get() function that knows how to deal with subdirectories.

manifest: throw LookupError if node not in revlog When accessing a manifest via manifestlog[node], let's verify that the node actually exists and throw a LookupError if it doesn't. This matches the old read behavior, so we don't accidentally return invalid manifestctxs. We do this in manifestlog instead of in the manifestctx/treemanifestctx constructors because the treemanifest code currently relies on the fact that certain code paths can produce treemanifests without touching the revlogs (and it has tests that verify things work if certain revlogs are missing entirely, so they break if we add validation that tries to read them).

revlog: optimize _chunkraw when startrev==endrev In many cases, _chunkraw() is called with startrev==endrev. When this is true, we can avoid an extra index lookup and some other minor operations. On the mozilla-unified repo, `hg perfrevlogchunks -c` says this has the following impact: ! read w/ reused fd ! wall 0.371846 comb 0.370000 user 0.350000 sys 0.020000 (best of 27) ! wall 0.337930 comb 0.330000 user 0.300000 sys 0.030000 (best of 30) ! read batch w/ reused fd ! wall 0.014952 comb 0.020000 user 0.000000 sys 0.020000 (best of 197) ! wall 0.014866 comb 0.010000 user 0.000000 sys 0.010000 (best of 196) So, we've gone from ~25x slower than batch to ~22.5x slower. At this point, there's probably not much else we can do except implement an optimized function in the index itself, including in C.

revlog: inline start() and end() for perf reasons When I implemented `hg perfrevlogchunks`, one of the things that stood out was N * _chunk() calls was ~38x slower than 1 _chunks() call. Specifically, on the mozilla-unified repo: N*_chunk: 0.528997s 1*_chunks: 0.013735s This repo has 352,097 changesets. So the average time per changeset comes out to: N*_chunk: 1.502us 1*_chunks: 0.039us If you extrapolate these numbers to a repository with 1M changesets, that comes out to 1.502s versus 0.039s, which is significant. At these latencies, Python attribute lookups and function calls matter. So, this patch inlines some code to cut down on that overhead. The impact of this patch on N*_chunk() calls is clear: ! wall 0.528997 comb 0.520000 user 0.500000 sys 0.020000 (best of 19) ! wall 0.367723 comb 0.370000 user 0.350000 sys 0.020000 (best of 27) So, we go from ~38x slower to ~27x. A nice improvement. But there's still a long way to go. It's worth noting that functionality like revsets perform changelog lookups one revision at a time. So this code path is worth optimizing.

revlog: reorder index accessors to match data structure order Index entries are ordered tuples. We have accessors in the revlog class to map tuple offsets to names. To help reinforce the order, reorder the methods so they match the order of elements in the tuple. While I'm here, also sneak in some minimal documentation.

color: add the ability to display configured style to 'debugcolor' The 'hg debugcolor' command gains a '--style' flag to display all the configured labels and their styles. This have many benefits: * discovering documented label, * checking consistency between label's style, * showing the actual style of a label.

color: sort output of 'debugcolor' The previous ordering were provided by the set. The new output is more stable and rational. In addition we have some logic to keep the '_background' version together to help readability.

color: extract color and effect display from 'debugcolor' We are about to introduce a second mode for 'hg debugcolor' that would list the known label and their configuration, so we split the code related to color and effect out of the main function.

color: restore _style global after debugcolor ran Before this change, running 'debugcolor' would destroy all color style for the rest of the process life. We now properly backup and restore the variable content. Using a global variable is sketchy in general and could probably be removed. However, this is a quest for another adventure.

color: add basic documentation to 'debugcolor' This does not hurt.

tests: merge 'test-push-hook-lock.t' into 'test-push.t' That test file is very small and is merge with the new 'test-push.t'. No logic is changed. We don't register this as a copy because is actually a "ypoc" merging two file together without replacing the destination and Mercurial cannot express that.

tests: merge 'test-push-validation.t' into 'test-push.t' That test file is very small and is merge with the new 'test-push.t'. No logic is changed but repository name are update to avoid collision. We don't register this as a copy because is actually a "ypoc" merging two file together without replacing the destination and Mercurial cannot express that.

test: rename 'test-push-r.t' to 'test-push.t' We do not have a simple test for 'hg push' but we have multiple tiny tests for various aspect of it. We'll unify them into a single file, and we start with 'test-push-r.t'. The code is unchanged but we renamed the repository used to avoid collision with other tests we'll import in coming changesets. Test timing for the record: start end cuser csys real Test 1.850 2.640 0.650 0.090 0.790 test-push-validation.t 2.640 3.520 0.760 0.090 0.880 test-push-hook-lock.t 0.000 1.850 1.560 0.210 1.850 test-push-r.t

tests: simplify command script in 'test-push-r.t' I came across this code by chance. The script of this test is a bit messy with a lot of unnecessary intermediate commands. We simplify the script and unify repository access through '-R'. In the process the update after the unbundle is dropped as it does not add anything to the tests.

py3: use encoding.environ in ui.py Using source transformer we add b'' everywhere. So there are no chances that those bytes string will work with os.environ on Py3 as that returns a dict of unicodes. We are relying on the errors, even though no error is raised even in future, these pieces of codes will tend to do wrong things. if statements can result in wrong boolean and certain errors can be raised while using this piece of code. Let's not wait for them to happen, fix what is wrong. If this patch goes in, I will try to do it for all the cases. Leaving it as it is buggy.

py3: make scmposix.userrcpath() return bytes We are making sure that we deal with bytes as much we can. This is a part of fixing functions so that they return bytes if they have to. Used encoding.environ to return bytes. After this patch, scmposix.userrcpath() returns bytes and scmutil.osrcpath() will also return bytes if the platform is posix. Functions is scmposix returns bytes on Python 3 now.

adjustlinkrev: remove unnecessary parameters Since adjustlinkrev has "self", and is a method of a filectx object, it does not need path, filelog, filenode. They can be fetched from the "self" easily.

repair: make strip() return backup file path narrowhg wants to strip some commits and then re-apply them after applying another bundle. Having repair.strip() return the bundle path will be helpful for it.

rebase: check for conflicts before continuing When there are unresolved merge conflicts, there is no reason to make the user wait for rebase to process all of the already rebased commits just to complain that it cannot do anything. Abort early.

cmdutil: refactor checkunresolved localrepo.commit had code to check for unresolved merge conflicts, it would be helpful for at least rebase to be able to use that code without calling commit().

rebase: rename merge to mergemod

context: make sure __str__ works, also when there is no _changectx Before, it could crash when trying to print the wrong kind of object at the wrong time.

largefiles: clarify variable name holding file mode A follow-up to c01acee367ec. 'st' sounds like the whole stat result while 'mode' is a better name for the actual file mode.

changegroup: use changelogrevision() Using offsets for accessing changelog entries isn't very readable. As a bonus, changelog.changelogrevision() also accepts a revision, so we don't need to perform the inline node resolution either.

changegroup: cache changelog and manifestlog outside of loop History has taught us that repo.changelog can add significant overhead to loops. So cache the changelog instance outside of the loop to avoid the lookup. While we're here, do the same for manifestlog, since each loop would otherwise initialize a new manifestlog instance.

make: targets for building packages for ubuntu yakkety Ubuntu 16.10 Yakkety Yak is out. Let's support it.

util: put compression code next to each other ctxmanager was injecting itself between the compression and decompression code. Let's restore some order.

hgweb: make log streams compatible with command server Even though it would be useless to start a web server by a command server, it should be doable in principle. Also, we can't use sys.stdout/err directly on Python 3 because they are unicode streams.

cmdserver: write channel header and payload by a single write() call This makes a channeledoutput thread-safe as long as the underlying fwrite() is thread-safe. Both POSIX and Windows implementations are documented as MT-safe. MT-safety is necessary to use ui.fout and ui.ferr in hgweb.

histedit: use ui.fin to read commands from stdin stdin is wrapped by channeledinput in command-server session.

convert: have debugsvnlog obtain standard streams from ui This will help porting to Python 3, where sys.stdin/out/err are unfortunately unicode streams so we can't use them directly.

convert: remove superfluous setbinary() calls from debugsvnlog a3fe91b4f6eb made standard streams set to binary mode globally.

tests: explicitly use ls profiler In preparation for making the statprof profiler the default.

statprof: pass data structure to display functions Currently, statprof maintains a global "state" variable that is used by several functions. Global variables hinder adaptability of code. So pass state to display functions so we can make changes to how "state" works in future patches.

statprof: use print function

statprof: use absolute_imports As part of this, we modify import order to satisfy our import checker.

statprof: require paths to save or load profile data Upstream appears to aggressively save statprof data in a well-defined home directory path. Change the code to not do that. We also change file saving to fail if an error has occurred instead of silently failing. Callers can catch the exception. This behavior is more suitable for a generic "library" module.

statprof: fix flake8 warnings My local flake8 hook informed me of these warnings in the upstream code. Fix them.

statprof: vendor statprof.py Vendored from https://bitbucket.org/facebook/hg-experimental changeset 73f9db47ae5a1a9fa29a98dfe92d557ad51234c3 without modification. This introduces a number of code style violations. The file already has the magic words to skip test-check-code.t. I'll make additional changes to clean up the test-check-py3-compat.t warnings and to change some behavior in the code that isn't suitable for general use. test-check-commit.t also complains about numerous things. But there's nothing we can do if we're importing as-is.

merge with stable

Added signature for changeset e69874dc1f4e

Added tag 4.0.2 for changeset e69874dc1f4e

i18n-ja: synchronized with 5f33116cd787

posix: make poll() restart on interruption by signal (issue5452) select() is a notable example of syscalls which may fail with EINTR. If we had a SIGWINCH handler installed, ssh would crash when the terminal window was resized. This patch fixes the problem.

demandimport: do not raise ImportError for unknown item in fromlist This is the behavior of the default __import__() function, which doesn't validate the existence of the fromlist items. Later on, the missing attribute is detected while processing the import statement. https://hg.python.org/cpython/file/v2.7.13/Python/import.c#l2575 The comtypes library relies on this (maybe) undocumented behavior, and we got a bug report to TortoiseHg, sigh. https://bitbucket.org/tortoisehg/thg/issues/4647/ The test added at 26a4e46af2bc verifies the behavior of the import statement, so this patch only adds the test of __import__() function and works around CPython/PyPy difference.

hgweb: add missing slash to file log url in rss style

Added signature for changeset b3b1ae98f6a0

Added tag 4.0.1 for changeset b3b1ae98f6a0

i18n-pt_BR: synchronized with 819f96b82fa4

shelve: add missing space in help text The change is trivial and unlikely to have been translated so we update translation files too.

scmutil: ignore EPERM at os.utime, which avoids ambiguity at closing According to POSIX specification, just having group write access to a file causes EPERM at invocation of os.utime() with an explicit time information (e.g. working on the repository shared by group access permission). To ignore EPERM at closing file object in such case, this patch makes checkambigatclosing._checkambig() use filestat.avoidambig() introduced by previous patch. Some functions below imply this code path at truncation of an existing (= might be owned by another user) file. - strip() in repair.py, introduced by e38d85be978f - _playback() in transaction.py, introduced by 599912a62ff6 This is a variant of issue5418.

vfs: ignore EPERM at os.utime, which avoids ambiguity at renaming (issue5418) According to POSIX specification, just having group write access to a file causes EPERM at invocation of os.utime() with an explicit time information (e.g. working on the repository shared by group access permission). To ignore EPERM at renaming in such case, this patch makes vfs.rename() use filestat.avoidambig() introduced by previous patch.

util: add utility function to skip avoiding file stat ambiguity if EPERM Now, advancing stat.st_mtime by os.utime() is used to avoid file stat ambiguity. But according to POSIX specification, utime(2) with an explicit time information is permitted only for a process with: - the effective user ID equal to the user ID of the file, or - appropriate privileges http://pubs.opengroup.org/onlinepubs/9699919799/functions/utime.html Therefore, just having group write access to a file causes EPERM at applying os.utime() on it (e.g. working on the repository shared by group access permission). This patch adds class filestat utility function avoidamgig() to avoid file stat ambiguity but skip it if EPERM. It is reasonable to always ignore EPERM, because utime(2) causes EPERM only in the case described above (EACCES is used only for utime(2) with NULL).

hgweb: cache fctx.parents() in annotate command (issue5414) 9c37df347485 introduced a call to fctx.parents() for each line in annotate output. This function call isn't cheap, as it requires linkrev adjustment. Since multiple lines in annotate output tend to belong to the same file revision, a cache of fctx.parents() lookups for each input should be effective in the common case. So we implement one. Since the cache has to precompute parents so an aborted generator doesn't leave an incomplete cache, we could just return a list. However, we preserve the generator for backwards compatibility. The effect of this change when requesting /annotate/96ca0ecdcfa/ browser/locales/en-US/chrome/browser/downloads/downloads.dtd on the mozilla-aurora repo is significant: p1(9c37df347485) 5.5s 9c37df347485: 66.3s this patch: 10.8s We're still slower than before. But only by ~2x instead of ~12x. On the tip revisions of layout/base/nsCSSFrameConstructor.cpp file in the mozilla-unified repo, time went from 12.5s to 14.5s and back to 12.5s. I'm not sure why the mozilla-aurora repo is so slow. Looking at the code of basefilectx.parents(), there is room for further improvements. Notably, we still perform redundant calls to filelog.renamed() and basefilectx._parentfilectx(). And basefilectx.annotate() also makes similar calls, so there is potential for object reuse. However, introducing caches here are not appropriate for the stable branch.

Added signature for changeset eab274469952

Added tag 4.0 for changeset eab274469952

merge with i18n

i18n-ja: synchronized with 69ffbbe73dd0

tag: clarify warning about making a tag on a branch head Currently the warning is ambiguous about whether the new tag (possibly specified via --rev) is being added on a branch head or whether the working directory is based on a branch head. Clarify the error message to eliminate this ambiguity.

contrib: check reference to old selenic.com domain Now, all URL in Mercurial source tree should refer mercurial-scm.org domain instead of selenic.com. *.po files are ignored in this patch, because they might contain msgid/msgstr coming from old source files. This ignorance seems safe enough, because such msgstr should be ignored at runtime, because: - msgid corresponded to it should be invalid, or - msgstr itself should be marked as fuzzy at synchronized to recent hg.pot If any additional examination for *.po files is needed in the future, let i18n/check-translation.py achieve such examination. BTW, some binary files (e.g. *.png) are meaningless for checking reference to old domain in this patch, but aren't ignored like as *.po files, because excluding multiple suffixes is difficult for regexp matching.

check-code: discard filtering result of previous check for independence Before this patch, check-code.py applies filtering on the file content, to which filtering of previous check is already applied. This might hide issues, which should be detected by a subsequent check in "checks" list. Fortunately, this problem hasn't appeared, because there is no overlapping of filename matching (examined in the order below). 1. *.py or *.cgi 2. test-* (not *.t suffix) 3. *.c or *.h 4. *.t 5. *.txt 6. *.tmpl For example, adding a test, which wants to examine raw comment text in *.py files, at the end of current "checks" list doesn't work as expected, because a filter for *.py files normalizes comment text in them. Putting such test at the beginning of "checks" list also resolves this problem, but such dependence on the order decreases maintainability of check-code.py itself. This patch discards filtering result of previous check at the beginning of each checks, for independence of each checks.

help: replace selenic.com by mercurial-scm.org in man pages Source code repository and mailing list services have been already migrated to mercurial-scm.org domain.

help: replace selenic.com by mercurial-scm.org in command examples Source code repository service of Mercurial itself has been already migrated to mercurial-scm.org domain.

hghave: fix 'rmcwd' to ensure temporary directory is removed On platforms where cwd can't be removed, it should try rmdir() after chdir to the original cwd.

httppeer: make __del__ access to self.urlopener more safe Some errors could in some cases show unfortunate scary and confusing warnings from the httppeer delstructors: abort: nodename nor servname provided, or not known Exception AttributeError: "'httpspeer' object has no attribute 'urlopener'" in <bound method httpspeer.__del__ of <mercurial.httppeer.httpspeer object at 0x106e1f5d0>> ignored``` To mute that, take 7b15dd9125b3 to the next level and use getattr in __del__.

tests: test preserving execbit changes at amending only on execbit platform

tests: put temporary file outside the working directory for test portability test-largefiles-update.t creates temporary file exec-bit.patch inside the working directory for no-execbit platform specific test, but subsequent tests aren't aware of it. On execbit platform, subsequent tests can run successfully, because exec-bit.patch isn't created. But on no-execbit platform, this temporary file makes subsequent tests show "? exec-bit.patch" at each "hg status".

tests: avoid quoting of commit messages for test portability journal extension uses util.shellquote() to record command line, but result of it depends on runtime platform: double quotation is used on Windows and OpenVMS, but single quotation is used otherwise. test-journal-share.t sometimes specifies commit messages including white space on command line. It makes journal output depend on runtime platform, but commit message itself isn't important in this test case.

tests: use basic format code "%Y" instead of "%s" for test portability On Windows, strftime() doesn't support format code "%s", and it causes "invalid format string" error. https://msdn.microsoft.com/en-us/library/fe06s4ak.aspx test-command-template.t examines not seconds value in UTC, but arithmetic calculation. Therefore, using format code "%Y" instead of "%s" should be reasonable. FYI: - Python standard library reference doesn't list "%s" up in format code list required for "C standard (1989 version)", even though it also mentions that additional format codes are required for "C standard (1999 version)" https://docs.python.org/2.7/library/datetime.html#strftime-and-strptime-behavior - The Open Group Base Specifications Issue 7 (IEEE Std 1003.1-2008, 2016 Edition) doesn't require strftime to support format code "%s" http://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html - "man strftime" of (Open/Oracle) Solaris and Mac OS X (= UNIX certified OSs) describes about format code "%s"

tests: add test-commit-interactive-curses.t "require tic" for test portability Standard library of Python on Windows doesn't have curses module.

tests: use "?" to glob both ":" and ";" in output for test portability If environment variable looks like PATH or so (e.g. any of components joined by ":" contains "/"), ":" in it is replaced with ";" by MinGW at spawning Windows native process, to follow path concatenation style of Windows. Therefore, "bundle:../full.hg" is converted into "bundle;..\full.hg" on MinGW. Difference between "/" and "\" is automatically ignored by "(glob)", but difference between ":" and ";" should be globed explicitly.

tests: invoke printenv.py via sh -c for test portability On Windows platform, invoking printenv.py directly via hook is problematic, because: - unless binding between *.py suffix and python runtime, application selector dialog is displayed, and running test is blocked at each printenv.py invocations - it isn't safe to assume binding between *.py suffix and python runtime, because application binding is easily broken For example, installing IDE (VisualStudio with Python Tools, or so) often requires binding between source files and IDE itself. This patch invokes printenv.py via sh -c for test portability. This is a kind of follow up for d19787db6fe0, which eliminated explicit "python" for printenv.py. There are already other 'sh -c "printenv.py"' in *.t files, and this fix should be reasonable. This changes were confirmed in cases below: - without any application binding for *.py suffix - with binding between *.py suffix and VisualStudio This patch also replaces "echo + redirection" style with "heredoc" style, because: - hook command line is parsed by cmd.exe as shell at first, and - single quotation can't quote arguments on cmd.exe, therefore, - "printenv.py foobar" should be quoted by double quotation, but - nested quoting (or tricky escaping) isn't readable

largefiles: handle that a found standin file doesn't exist when removing it I somehow ended up in a situation where hg crashed on an unlink I introduced in 328545c7d8a1. I don't know how it happened and can't reproduce it. It seems like it only can happen when the file is removed between the time of check in a working directory context walk that finds a standin file, and the time of use when we try to remove it because the corresponding largefile doesn't exist. But better safe than sorry: replace the plain unlink with unlinkpath with ignoremissing=True. That will also remove remaining empty directories, which arguably is more correct.

templater: use unfiltered changelog to calculate shortest() at constant time cl._partialmatch() can be pretty slow if hidden revisions are involved. This patch cancels the slowdown introduced by the previous patch by using an unfiltered changelog, which means shortest(node) isn't always the shortest. The result isn't perfect, but seems okay as long as shortest(node) is short enough to type and can be used as an identifier. (with hidden revisions) % hg log -R hg-committed -r0:20000 -T '{node|shortest}\n' --time > /dev/null (.^^) time: real 1.530 secs (user 1.480+0.000 sys 0.040+0.000) (.^) time: real 43.080 secs (user 43.060+0.000 sys 0.030+0.000) (.) time: real 1.680 secs (user 1.650+0.000 sys 0.020+0.000)

templater: do not use index.partialmatch() directly to calculate shortest() cl.index.partialmatch() isn't a drop-in replacement for cl._partialmatch(). It has no knowledge about hidden revisions, and it raises ValueError if a node shorter than 4 chars is given. Instead, use index.partialmatch() through cl._partialmatch(), which has no such problems and gives the identical result with/without --pure. The test output was sampled with --pure without this patch, which shows the most correct result. However, we'll need to switch to using an unfiltered changelog because _partialmatch() of a filtered changelog can be an order of magnitude slower. (with hidden revisions) % hg log -R hg-committed -r0:20000 -T '{node|shortest}\n' --time > /dev/null (.^) time: real 1.530 secs (user 1.480+0.000 sys 0.040+0.000) (.) time: real 43.080 secs (user 43.060+0.000 sys 0.030+0.000)

tests: run "cwd was removed" test only if cwd can actually be removed On some platforms, cwd can't be removed. In which case, util.unlinkpath() continues with no error since the failure of directory removal isn't critical. So it doesn't make sense to run the test added by 90a6c18a7c1d on those platforms. OTOH, we need to run the test in test-rebase-scenario-global.t since the repository is referenced after that.

merge: avoid superfluous filemerges when grafting through renames (issue5407) This is a fix for a regression introduced by the patches for issue4028. The test changes are due to us doing fewer _checkcopies searches now, which makes some test outputs revert to the pre-issue4028 behavior. That issue itself remains fixed, we only skip copy tracing for files where it isn't relevant. As a nice side effect, this makes copy detection much faster when tracing backwards through lots of renames.

sslutil: guard against broken certifi installations (issue5406) Certifi is currently incompatible with py2exe; the Python code for certifi gets included in library.zip, but not the cacert.pem file - and even if it were included, SSLContext can't load a cacert.pem file from library.zip. This currently makes it impossible to build a standalone Windows version of Mercurial. Guard against this, and possibly other situations where a module with the name "certifi" exists, but is not usable.

revset: don't cache abstractsmartset min/max invocations infinitely There was a "leak", apparently introduced in ab66c1dee405. When running: hg = hglib.open('repo') while True: hg.log("max(branch('default'))") all filteredset instances from branch() would be cached indefinitely by the @util.cachefunc annotation on the max() implementation. util.cachefunc seems dangerous as method decorator and is barely used elsewhere in the code base. Instead, just open code caching by having the min/max methods replace themselves with a plain lambda returning the result.

merge with i18n

i18n-pt_BR: synchronized with 7b428b00a1d4

dirstate: fix debug.dirstate.delaywrite to use the new "now" after sleeping It seems like the a regression has sneaked into debug.dirstate.delaywrite in 6c6b48aca328. It would sleep until no files were modified "now" any more, but when writing the dirstate it would use the old "now" and still mark files as 'unset' instead of recording the timestamp that would make the file show up as clean instead of unknown. Instead of getting a new "now" from the file system, we trust the computed end time as the new "now" and thus cause the actual modification time to be writiten to the dirstate. debug.dirstate.delaywrite is undocumented and only used in test-largefiles-update.t . All tests seems to work fine for me without debug.dirstate.delaywrite . Perhaps because it not really worked as intended without the fix in this patch, and code and tests thus have evolved to do fine without it? It could thus perhaps make sense to drop usage of this setting in the tests. That could speed the test up a bit. This functionality (or something very similar) can however apparently be very convenient in setups where checking dirty-ness is expensive - such as when using large files and have slow file filesystems or are CPU constrained. Now it works and we can try it. (But ideally, for the largefile use case, it should probably only delay lfdirstate writes - not ordinary dirstate.)

tests: fix test-casefolding.t The message had changed, but the test was not updated. This test does not run on Linux, but failed on my Mac.

commands: print security protocol support in debuginstall Over the past week I've had to instruct multiple people to run Python code to query the ssl module to see what TLS protocol support is present. I think it would be useful for `hg debuginstall` to print this info to make it easier to access and debug why Mercurial is complaining about using an insecure TLS 1.0 protocol. Ideally we'd also print the path to the CA cert bundle. But the APIs for querying that in sslutil can emit warnings, making it slightly more difficult to integrate into `hg debuginstall`. That work will have to wait for another day.

manifest: make treemanifestctx store the repo Same as in the last commit, the old treemanifestctx stored a reference to the revlog. If the inmemory revlog became invalid, the ctx now held an old copy and would be incorrect. To fix this, we need the ctx to go through the manifestlog for each access. This is the same pattern that changectx already uses (it stores the repo, and accesses commit data through self._repo.changelog).

manifest: make manifestctx store the repo The old manifestctx stored a reference to the revlog. If the inmemory revlog became invalid, the ctx now held an old copy and would be incorrect. To fix this, we need the ctx to go through the manifestlog for each access. This is the same pattern that changectx already uses (it stores the repo, and accesses commit data through self._repo.changelog).

manifest: make manifestlog a storecache The old @property on manifestlog was broken. It meant that we would always recreate the manifestlog instance, which meant the cache was never hit. Since we'll eventually remove repo.manifest and make manifestlog the only property, let's go ahead and make manifestlog the @storecache property, have manifestlog own the manifest instance, and have repo.manifest refer to it via manifestlog. This means all accesses go through repo.manifestlog, which is now invalidated correctly.

manifest: move manifest creation to a helper function A future patch will be moving manifest creation to be inside manifestlog as part of improving our cache guarantees. bundlerepo and unionrepo currently rely on being able to hook into manifest creation, so let's temporarily move the actual manifest creation to a helper function for them to intercept. In the future manifest.manifest() will disappear entirely and this can disappear.

Added signature for changeset 438173c41587

Added tag 4.0-rc for changeset 438173c41587

merge default into stable for 4.0 code freeze

merge with i18n

i18n-pt_BR: synchronized with 149433e68974

changegroup: increase write buffer size to 128k By default, Python defers to the operating system for choosing the default buffer size on opened files. On my Linux machine, the default is 4k, which is really small for 2016. This patch bumps the write buffer size when writing changegroups/bundles to 128k. This matches the 128k read buffer we already use on revlogs. It's worth noting that this only impacts when writing to an explicit file (such as during `hg bundle`). Buffers when writing to bundle files via the repo vfs or to a temporary file are not impacted. When producing a none-v2 bundle file of the mozilla-unified repository, this change caused the number of write() system calls to drop from 952,449 to 29,788. After this change, the most frequent system calls are fstat(), read(), lseek(), and open(). There were 2,523,672 system calls after this patch (so a net decrease of ~950k is statistically significant). This change shows no performance change on my system. But I have a high-end system with a fast SSD. It is quite possible this change will have a significant impact on network file systems, where extra network round trips due to excessive I/O system calls could introduce significant latency.

changegroup: skip delta when the underlying revlog do not use them Revlog can now be configured to store full snapshot only. This is used on the changelog. However, the changegroup packing was still recomputing deltas to be sent over the wire. We now just reuse the full snapshot directly in this case, skipping delta computation. This provides use with a large speed up(-30%): # perfchangegroupchangelog on mercurial ! wall 2.010326 comb 2.020000 user 2.000000 sys 0.020000 (best of 5) ! wall 1.382039 comb 1.380000 user 1.370000 sys 0.010000 (best of 8) # perfchangegroupchangelog on pypy ! wall 5.792589 comb 5.780000 user 5.780000 sys 0.000000 (best of 3) ! wall 3.911158 comb 3.920000 user 3.900000 sys 0.020000 (best of 3) # perfchangegroupchangelog on mozilla central ! wall 20.683727 comb 20.680000 user 20.630000 sys 0.050000 (best of 3) ! wall 14.190204 comb 14.190000 user 14.150000 sys 0.040000 (best of 3) Many tests have to be updated because of the change in bundle content. All theses update have been verified. Because diffing changelog was not very valuable, the resulting bundle have similar size (often a bit smaller): # full bundle of mozilla central with delta: 1142740533B without delta: 1142173300B So this is a win all over the board.

revlog: make 'storedeltachains' a "public" attribute The next changeset will make that attribute read by the changegroup packer. We make it "public" beforehand.

manifest: don't store None in fulltextcache When we read a value from fulltextcache, we expect it to be an array, so we should not store None in it. Found while working on narrowhg.