util: use string.hexdigits instead of defining it ourselves This resolves some Python 3 weirdness.

util: correct check of sys.version_info sys.version is a string, and shouldn't be compared against a tuple for version comparisons. This was always true, so we were never disabling gc on 2.6. >>> (2, 7) >= '2.7' True >>> (2, 6) >= '2.7' True

py3: switch to .items() using transformer .iteritems() don't exist in Python 3 world. Used the transformer to replace .iteritems() to .items()

py3: handle multiple arguments in .encode() and .decode() There is a case and more can be present where these functions have multiple arguments. Our transformer used to handle the first argument, so added a loop to handle more arguments if present.

py3: convert to unicode to pass into encode() encoding.encoding is bytes, we need to pass it to encode() which accepts unicodes in py3, so used pycomapt.sysstr() Also this can't be done using transformer as that only transforms the string values not variables.

templater: use "needle" and "haystack" as (meta-)variables for ifcontains() It wasn't immediately clear if it's supposed to look for "search" in "thing" or "thing" in "search".

copies: mark checkcopies as internal with the _ prefix

copies: split u1/u2 to u1u/u2u and u1r/u2r These will be made different in case of grafts by another patch in this series.

copies: style fixes and add comment

copies: limit is an optimization, and doesn't provide guarantees

revset: do not rewrite ':y' to '0:y' (issue5385) That's no longer valid since the revision 0 may be hidden. Bypass validating the existence of '0' and filter it by spanset.

revset: extract function that creates range set from computed revisions So we can pass m=0 to _makerangeset() even if the revision 0 is hidden. Hidden revisions are filtered by spanset.

lazymanifest: write a more efficient, pypy friendly version of lazymanifest

hg: set default path correctly when doing a clone+share (issue5378) Before, if performing a clone+share from a repo that was itself using shared storage, the share code would copy paths.default from the underlying repo being shared, not from the source given by the user. This patch teaches hg.clonewithshare to resolve paths.default and pass it to share so it can be written to the hgrc accordingly.

annotate: calculate line count correctly Before this patch, the "lines" function inside "annotate" returns 1 for empty text (''). This patch makes it 0. Because the function should match mdiff.splitnewlines (used by mdiff.allblocks), or s.splitlines (used at the end of the "annotate" method). Both len(mdiff.splitnewlines('')) and len(''.splitlines(True)) are 0. This issue was discovered while testing fastannotate [1]. I could not find a test case to reveal this issue. However in theory this could reduce memory usage a little bit, and avoids surprises when people are touching this area in the future. [1]: https://bitbucket.org/facebook/hg-experimental/commits/525b3b98e93a

py3: use unicode in is_frozen() imp.is_frozen() doesnot accepts bytes on Python 3. It does accept both bytes and strings on Python 2.

py3: use unicodes in __slots__ __slots__ doesnot accepts bytes on Python 3.

merge with stable

url: fix crash by empty path with #fragments Before, "#foo" paths made hg crash. We've moved the #fragment parsing at 64fbd0de9773, but we shouldn't set path to None too early. This patch just removes the "if not path:" block since that's checked a few lines later.

py3: make i18n use encoding.environ

py3: provide encoding.environ which is a dict of bytes This can't be moved to pycompat.py since we need encoding.tolocal() to build bytes dict from unicode os.environ.

py3: convert encoding name and mode to str Otherwise tolocal() and fromlocal() wouldn't work on Python 3. Still tolocal() can't make a valid localstr object because localstr inherits str, but it can return some object without raising exceptions. Since Py3 bytes() behaves much like bytearray() than str() of Py2, we can't simply do s/str/bytes/g. I have no good idea to handle str/bytes divergence.

pycompat: extract function that converts attribute or encoding name to str This will be used to convert encoding.encoding to a str acceptable by Python 3 functions. The source encoding is changed to "latin-1" because encoding.encoding can have arbitrary bytes. Since valid names should consist of ASCII characters, we don't care about the mapping of non-ASCII characters so long as invalid names are distinct from valid names.

Added signature for changeset 149433e68974

pycompat: provide 'ispy3' constant We compare version_info at several places, which seems enough to define a constant.

Added tag 3.9.2 for changeset 149433e68974

extensions: add a note about debug output during extensions search These messages do not show up when one use '--debug'. This is quite confusing so we clarify the situation next to the 'ui.debug' call.

extensions: fix a debug message when searching for extensions The "next" value was wrong. When 'hgext.NAME' is not found we now search for 'hgext3rd.NAME'.

dirstate: rebuild should update dirstate properly Updating dirstate by simply adding and dropping files from self._map doesn't keep the other maps updated (think: _dirs, _copymap, _foldmap, _nonormalset) thus introducing cache inconsistency. This is also affecting the debugstate tests since now we don't even try to set correct mode and mtime for the files because they are marked dirty anyway and will be checked during next status call.

histedit: avoid converting nodeid to context and back again It looks like this became unnecessary in e767f5aba810 (histedit: fix preventing strips during histedit, 2015-04-04).

help: mark boolean flags with [no-] to explain that they can be negated That is, help gets tweaked thus: global options ([+] can be repeated): -v --[no-]verbose enable additional output Other proposals have included: global options ([+] can be repeated, options marked [?] are boolean flags): -v --verbose[?] enable additional output and global options ([+] can be repeated, options marked [^] are boolean flags): -v --verbose[^] enable additional output which avoid the unfortunate visual noise in this patch. In this version's favor, it's consistent with what I'm used to seeing in man pages and similar documentation venues.

mdiff: remove unused parameter 'refine' from allblocks()

demandimport: error out early on missing attribute of non package (issue5373) If the parent module isn't a package, all valid attributes must be obtained from it. We can raise ImportError early if any attributes not found.

demandimport: add 'nt' to ignore list (issue5373) pathlib2 tries to import nt. Since it is a built-in module, there should be no performance penalty. https://github.com/mcmtroffaes/pathlib2/blob/release/2.2.0/pathlib2.py#L33

demandimport: add '_ctypes.pointer' to ignore list on PyPy The pointer module is shadowed by a subsequent import. Our demand importer can't handle this because both sub modules and attributes live in the same namespace. https://bitbucket.org/pypy/pypy/src/release-5.0.1/lib_pypy/_ctypes/__init__.py#__init__.py-5

hgweb: make anchor name actually match its href on help index page

perf: add perfchangegroupchangelog command This command can be used for testing the performance of producing the changelog portion of a changegroup. We could use additional perf* commands for testing other parts of changegroup. Those can be written another time, when they are needed. (And those may want to refactor the changegroup generation API so code can be reused.) Speaking of code reuse, yes, this command does reinvent a small wheel. I didn't want to scope bloat to change the changegroup API because that will invite bikeshedding.

perf: add --reverse to perfrevlog It can be useful to know how fast we can read revisions from a revlog in reverse. This operation tends to occur in `hg log` commands, for example.

log: copy the way of ancestor traversal to --follow matcher (issue5376) We can't use fctx.linkrev() because follow() revset tries hard to simulate the traversal of changelog DAG, not filelog DAG. This patch fixes _makefollowlogfilematcher() to walk file ancestors in the same way as revset._follow(). I'll factor out a common function in future patches.

log: unroll loop that populates file paths for --patch --follow matcher We can't handle the first fctx in the same manner as its ancestors. Also, I think the original code was too tricky.

wireproto: rename argument to groupchunks() groupchunks() is a generic "turn a file object into a generator" function. It isn't limited to changegroups. Rename the argument and update the docstring to reflect this.

tests: actually test non-generaldelta variant for stream clones 608cabec1b15 accidentally made both tests test generaldelta repos. Restore the test for a non-gd repo.

revlog: document high frequency of code execution Recording my notes while working on performance optimization.

revlog: make code in builddelta() slightly easier to read self.compress() is destructured into its components. "l" is renamed to "deltalen."

py3: update test expectation on stable

grep: rewrite help to better document current (confusing) behavior

templates: add built-in files() function We already support multiple primitive for listing files, which were affected by the current changeset. This patch adds files() which returns files of the current changeset matching a given pattern or fileset query via the "set:" prefix.

rebase: rebase changesets in topo order (issue5370) (BC) There are two reasons that rebase should be done this way: 1. This would make rebasing faster because it would minimize the total number of files to be checked out in the process, as it don't need to switch back and forth between branches. 2. It makes resolving conflicts easier as user has a better context. This commit changes the behavior in "Test multiple root handling" of test-rebase-obsolete.t. It is an expected change which reflects the new behavior that commits in a branch are grouped together when rebased.

dispatch: make hg --profile wrap reposetup Move profiling.maybeprofile() from _runcommand to _dispatch() so that profiler output will include reposetup.

dispatch: change indentation level in _dispatch() Add an if True: placeholder for a profiling context manager that will be added in the next commit, for the purpose of reducing size of the diff due to trivial indentation changes. This change should be a no-op.

log: drop outdated optimization to walk revisions in reverse order Since revset is computed lazily, there would be no (or little) benefit to reverse 'revs' temporarily.

graphlog: preserve topo sort even if additional filter options specified Use ordered=revset.followorder instead. This change is logically the same as fa5e4f58dfbc.

transaction: open a file with checkambig=True to avoid file stat ambiguity Before this patch, if steps below occurs at "the same time in sec", all of mtime, ctime and size are same between (1) and (3). 1. append data to revlog-style file (and close transaction) 2. discard appended data by truncation of rollback 3. append same size but different data to revlog-style file again Therefore, cache validation doesn't work after (3) as expected. To avoid file stat ambiguity around truncation, this patch opens a file with checkambig=True. This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

repair: open a file with checkambig=True to avoid file stat ambiguity Before this patch, if steps below occurs at "the same time in sec", all of mtime, ctime and size are same between (1) and (3). 1. append data to revlog-style file (and close transaction) 2. discard appended data by truncation of strip 3. append same size but different data to revlog-style file again Therefore, cache validation doesn't work after (3) as expected. To avoid such file stat ambiguity around truncation, this patch opens a file with checkambig=True. This patch also introduces "with" statement style, to ensure immediate invocation of close() after truncation, because closing file is the only trigger to check (and get rid of) file stat ambiguity. This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

changelog: specify checkambig=True to revlog.__init__, to avoid ambiguity If steps below occurs at "the same time in sec", all of mtime, ctime and size are same between (1) and (3). 1. append data to 00changelog.i (and close transaction) 2. discard appended data by truncation (strip or rollback) 3. append same size but different data to 00changelog.i again Therefore, cache validation doesn't work after (3) as expected. To avoid such file stat ambiguity around truncation, this patch specifies checkambig=True to revlog.__init__(). This makes revlog write changes out with checkambig=True. Even though changes of 00changelog.i themselves are written out at changelog._finalize(), this checkambig=True is needed, because revlog.checkinlinesize(), which is invoked at the end of changelog._finalize(), might replace already changed 00changelog.i by converted one. Even after this patch, avoiding file stat ambiguity of 00changelog.i around truncation isn't yet completed, because truncation side isn't aware of this issue. This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

changelog: specify checkambig=True to avoid ambiguity around truncation If steps below occurs at "the same time in sec", all of mtime, ctime and size are same between (1) and (3). 1. append data to 00changelog.i (and close transaction) 2. discard appended data by truncation (strip or rollback) 3. append same size but different data to 00changelog.i again Therefore, cache validation doesn't work after (3) as expected. To avoid such file stat ambiguity around truncation, this patch specifies checkambig=True for renaming or opening to write changes out at finalization. Even after this patch, avoiding file stat ambiguity of 00changelog.i around truncation isn't yet completed, because truncation side isn't aware of this issue. This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

manifest: specify checkambig=True to revlog.__init__, to avoid ambiguity If steps below occurs at "the same time in sec", all of mtime, ctime and size are same between (1) and (3). 1. append data to 00manifest.i (and close transaction) 2. discard appended data by truncation (strip or rollback) 3. append same size but different data to 00manifest.i again Therefore, cache validation doesn't work after (3) as expected. To avoid such file stat ambiguity around truncation, this patch specifies checkambig=True to revlog.__init__(). This makes revlog write changes out with checkambig=True. Even after this patch, avoiding file stat ambiguity of 00manifest.i around truncation isn't yet completed, because truncation side isn't aware of this issue. This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

revlog: specify checkambig at writing to avoid file stat ambiguity This allows revlog-style files to be written out with checkambig=True easily. Because avoiding file stat ambiguity is needed only for filecache-ed manifest and changelog, this patch does: - use False for default value of checkambig - focus only on writing changes of index file out This patch also adds optional argument checkambig to _divert/_delay for changelog, to safely accept checkambig specified in revlog layer. But this argument can be fully ignored, because: - changes are written into other than index file, if name != target - changes are never written into index file, otherwise (into pending file by _divert, or into in-memory buffer by _delay) This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

vfs: use checkambigatclosing in checkambig=True but atomictemp=False case In Mercurial source tree, opening a file in "a"/"a+" mode like below doesn't specify atomictemp=True for vfs, and this avoids file stat ambiguity check by atomictempfile. - writing changes out in revlog layer uses "a+" mode - truncation in repair.strip() uses "a" mode - truncation in transaction._playback() uses "a" mode If steps below occurs at "the same time in sec", all of mtime, ctime and size are same between (1) and (3). 1. append data to revlog-style file (and close transaction) 2. discard appended data by truncation (strip or rollback) 3. append same size but different data to revlog-style file again Therefore, cache validation doesn't work after (3) as expected. This patch uses checkambigatclosing in checkambig=True but atomictemp=False case, to check (and get rid of) file stat ambiguity at closing. This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan

scmutil: add file object wrapper class to check ambiguity at closing In Mercurial source tree, opening a file in "a"/"a+" mode like below doesn't specify atomictemp=True for vfs, and this avoids file stat ambiguity check by atomictempfile. - writing changes out in revlog layer uses "a+" mode - truncation in repair.strip() uses "a" mode - truncation in transaction._playback() uses "a" mode If steps below occurs at "the same time in sec", all of mtime, ctime and size are same between (1) and (3). 1. append data to revlog-style file (and close transaction) 2. discard appended data by truncation (strip or rollback) 3. append same size but different data to revlog-style file again Therefore, cache validation doesn't work after (3) as expected. This patch adds file object wrapper class checkambigatclosing to check (and get rid of) ambiguity at closing. It is used by vfs in subsequent patch. This is a part of ExactCacheValidationPlan. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan BTW, checkambigatclosing is tested in test-filecache.py, even though it doesn't use filecache itself, because filecache assumes that file stat ambiguity never occurs (and there is no another test-*.py related to filecache).

scmutil: factor out common logic of delayclosedfile to reuse it This is a preparation for the subsequent patch, which adds another proxy class for a file object.

spartan: remove unused templates

monoblue: remove unused templates

gitweb: remove unused templates

paper: remove unused templates

help: add sections for revsets

help: move revsets.## documentation into infix section

help: clarify quotes are needed for filesets.size expressions

paper: remove unused template "changelogchild"

monoblue: remove unused template "changelogchild"

gitweb: remove unused template "changelogchild"

monoblue: tweak branch table's last row's cell colspan on summary page This table only has 3 columns, so max sensible colspan is 3.

gitweb: tweak branch table's last row's cell colspan on summary page This table only has 3 columns, so max sensible colspan is 3.

tests: clarify demandimport disabled state

demandimport: add trailing comma

tests: favor single quotes for wrapping hg help ...

samplehgrcs: use single quotes in use warning

util: use single quotes in use warning

obsolete: use single quotes in use warning

localrepo: use single quotes in use warning

help: use single quotes in use warning

discovery: use single quotes in use warning

serve: use single quotes in use warning

bundle: use single quotes in use warning

histedit: use single quotes in use warning

keyword: use single quotes in use warning

mq: use single quotes in use warning

pager: use single quotes in use warning

rebase: use single quotes in use warning

push: update help hint to point to config.paths section

update: use single quotes in use warning

remove: specify hg in added warning

manifest: add manifestlog.add This adds a simple add() function to manifestlog. This lets us convert more uses of repo.manifest to use repo.manifestlog, so we can further break our dependency on the manifest class.

manifest: move manifest.add onto manifestrevlog This moves add and _addtree onto manifestrevlog. manifestrevlog is responsible for all serialization decisions, so therefore the add function should live on it. This will allow us to call add() from manifestlog, which lets us further break our dependency on manifest.

manifest: remove dependency on treeinmem from manifest.add Currently manifest.add uses the treeinmem option to know if it can call fastdelta on the given manifest instance. In a future patch we will be moving add() to be on the manifestrevlog, so it won't have access to the treeinmem option anymore. Instead, let's have it actually check if the given manifest instance supports the fastdelta operation. This also means that if treemanifest or any implementation eventually implements fastdelta(), it will automatically benefit from this code path.

manifest: move treeinmem onto manifestlog A previous patched moved all the serialization related options onto manifestrevlog (since it is responsible for serialization). Let's move the treeinmem option on manifestlog, since it is responsible for materialization decisions. This reduces the number of dependencies manifestlog has on the old manifest type as well, so we can eventually make them completely independent of each other.

copy: document current behavior of 'hg cp --after' I'm about to propose an output change here, but the existing behavior was untested!

crecord: add an event that scrolls the selected line to the top of the screen Using ctrl-l for this purpose seems to be a fairly widely used practice, presumably following emacs. This doesn't scroll the selected line all the way to the top of the window, instead it leaves a 3 line buffer for context. Use curses.unctrl() to resolve keypressed to '^L' to avoid hard-coding hexadecimal key codes.

log: drop hack to fix order of revset (issue5100) Specify ordered=revset.followorder instead. This patch effectively backs out c407583cf5f6. revs.sort(reverse=True) is replaced by revs.reverse() because the matcher should no longer reorder revisions.

revset: add option to make matcher takes the ordering of the input set This allows us to evaluate match(subset) as if 'subset & expr', which will be the complete fix for the issue5100.

strip: don't use "full" and "partial" to describe bundles The partial bundle is not a subset of the full bundle, and the full bundle is not full in any way that i see. The most obvious interpretation of "full" I can think of is that it has all commits back to the null revision, but that is not what the "full" bundle is. The "full" bundle is simply a backup of what the user asked us to strip (unless --no-backup). The "partial" bundle contains the revisions we temporarily stripped because they had higher revision numbers that some commit that the user asked us to strip. The "full" bundle is already called "backup" in the code, so let's use that in user-facing messages too. Let's call the "partial" bundle "temporary" in the code.

strip: clarify that user action is required to recover temp bundle If strip fails when applying the temporary bundle, the commits in the temporary bundle have not yet been applied, so the user will almost definitely want to apply the bundle. We should be more clear to the user about that than our current "partial bundle stored in...". Note that we will probably not be able to recover it automatically, since whatever made it fail (e.g. a hook) will most likely make it fail again. We need to give control back to the user to fix the problem before trying again.

strip: report both bundle files in case of exception (issue5368) If strip fails while recovering the temporary bundle (e.g. because a hook fails), we tell the user only about the backup bundle, not about the temporary bundle. Since the user did not ask to strip the commits in the temporary bundle, that's the more important bundle to mention, so let's do that (and also mention the backup bundle as usual).

strip: simplify some repeated conditions We check "if saveheads or savebases" in several places to see if we should or have created a bundle of the changesets to apply after truncating the revlogs. One of the conditions is actually just "if saveheads", but since there can't be savebases without saveheads, that is effectively the same condition. It seems simpler to check only once and from then on see if we created the file.

config: add template support V2: - Limit escaping to plain formatting only - Use the formatter consistently (no more ui.debug) - Always include 'name' and 'value' V3: - Always convert 'value' to string (this also makes sure we handle functions) - Keep real debug message as ui.debug for now - Add additional tests. Note: I'm not quite sure about the best approach to handling the 'print the full config' case. For me, it printed the 'ui.promptecho' key at the end. I went with globs there as that at least tests the json display reliably. Example output: [ { "name": "ui.username", "source": "/home/mathias/.hgrc:2", "value": "Mathias De Maré <mathias.demare@gmail.com>" } ]

formatter: introduce isplain() to replace (the inverse of) __nonzero__() (API) V2: also remove and replace __nonzero__

diffopts: notice a negated boolean flag in diffopts This means that if you have git-diffs enabled by default (pretty common) and you hit the rare (but real) case where a git-diff breaks patch(1) or some other tool, you can easily disable it by just specifying --no-git on the command line. I feel a little bad about the isinstance() check, but some values in diffopts are not booleans and so we need to preserve false iff the flag is a boolean flag: failing to do this means we end up with empty string defaults for flags clobbering meaningful values from the [diff] section in hgrc.

flags: allow specifying --no-boolean-flag on the command line (BC) This makes it much easier to enable some anti-foot-shooting features (like update --check) by default, because now all boolean flags can be explicitly disabled on the command line without having to use HGPLAIN or similar. Flags which don't deserve this treatment can be removed from consideration by adding them to the nevernegate set in fancyopts. This doesn't make it any easier to identify when a flag is set: opts still always gets filled in, either with the user-specified flag value or with the default from the flags list in the command table. Improving that would probably clean things up a bit, but for now if you want a boolean flag and care if it was explicitly false or default false (or true, but nobody uses that functionality because before now it was nonsense) you need to use None as your default rather than True or False. This doesn't (yet) update help output, because I'm not quite sure how to do that cleanly.

revset: make sort() noop depending on ordering requirement (BC) See the previous patch for why.

revset: make reverse() noop depending on ordering requirement (BC) Because smartset.reverse() may modify the underlying subset, it should be called only if the set can define the ordering. In the following example, 'a' and 'c' is the same object, so 'b.reverse()' would reverse 'a' unexpectedly. # '0:2 & reverse(all())' <filteredset <spanset- 0:2>, # a <filteredset # b <spanset- 0:2>, # c <spanset+ 0:9>>>

revset: fix order of nested 'range' expression (BC) Enforce range order only if necessary as the comment says "carrying the sorting over would be more efficient."

revset: forward ordering requirement to argument of present() present() is special in that it returns the argument set with no modification, so the ordering requirement should be forwarded. We could make present() fix the order like orset(), but that would be silly because we know the extra filtering cost is unnecessary.

crecord: delete commented line

manifest: move dirlog up to manifestrevlog This removes dirlog and its associated cache from manifest and puts it in manifestrevlog. The notion of there being sub-logs is specific to the revlog implementation, and therefore belongs on the revlog class. This patch will enable future patches to move the serialization logic for manifests onto manifestrevlog, which will allow us to move manifest.add onto manifestlog in a way that it just calls out to manifestrevlog for the serialization.

manifest: move revlog specific options from manifest to manifestrevlog The manifestv2 and treeondisk options are specific to how we serialize the manifest into revlogs, so let's move them onto the manifestrevlog class. This will allow us to add a manifestlog.add() function in a future diff that will rely on manifestrevlog to make decisions about how to serialize the given manifest to disk. We have to move a little bit of extra logic about the 'dir' as well, since it is used in conjunction with the treeondisk option to decide the revlog file name. It's probably good to move this down to the manifestrevlog class anyway, since it's specific to the revlog.

manifest: adds manifestctx.readfast This adds a copy of manifest.readfast to manifestctx.readfast and adds a consumer of it. It currently looks like duplicate code, but a future patch causes these functions to diverge as tree concepts are added to the tree version.

manifest: add manifestctx.readdelta() This adds an implementation of readdelta to the new manifestctx class and adds a couple consumers of it. This currently appears to have some duplicate code, but future patches cause this function to diverge when things like "shallow" are introduced.

merge with stable

rebase: make debug logging more consistent We emit some lines that mix revision numbers with nodeids, which makes little sense to me.

revset: fix order of nested '_(|int|hex)list' expression (BC) This fixes the order of 'x & (y + z)' where 'y' and 'z' are trivial, and the other uses of _list()-family functions. The original functions are renamed to '_ordered(|int|hex)list' to say clearly that they do not follow the subset ordering.