py3: convert system strings to bytes in doctest of formatter.py

py3: rewrite stdout hack of doctest by using ui.pushbuffer() We can't use pycompat.stdout here because the doctest runner replaces sys.stdout with a string buffer.

py3: use bytes os.sep in doctest of pathutil.py

py3: fix doctests in patch.py to be compatible with Python 3 We were lucky that parsepatch() could concatenate a character slice as if it were a list of chunks.

py3: stop using bytes[n] in patch.py

error: move patch.PatchError so it can easily implement __bytes__ (API)

templater: extract shortest() logic from template function It can be useful for extensions to be able to produce the shortest unambiguous hash (including the in-tree "show" extension). That logic is currently inside the shortest() template function. Let's move it out of the templater. I've put it on revlog since it's closely related to revlog._partialmatch. We may also want a convenience method on context, but I'll leave that for a later patch. Differential Revision: https://phab.mercurial-scm.org/D724

bundle2: raise a more helpful error if building a bundle part header fails I've tripped on this several times now, and am tired of debugging. Now the header parts are part of the error message when the ''.join() fails, which makes debugging obvious.

hgwebdir: read 'web.template' untrusted The 'hgweb_mod.py' version of this read it untrusted. For consistency we align the two versions of this code.

configitems: register the 'web.templates' config

configitems: register the 'web.style' config

configitems: register the 'web.stripes' config

configitems: register the 'web.refreshinterval' config

configitems: register the 'web.prefix' config

configitems: register the 'web.port' config

configitems: register the 'web.ipv6' config

configitems: register the 'web.errorlog' config

configitems: register the 'web.encoding' config

configitems: register the 'web.description' config

configitems: register the 'web.descend' config

configitems: register the 'web.deny_read' config

configitems: register the 'web.csp' config

configitems: register the 'web.collapse' config

configitems: register the 'web.certificate' config

configitems: register the 'web.cacerts' config

configitems: register the 'web.baseurl' config

configitems: register the 'web.allow_read' config

configitems: register the 'web.allow_archive' config

configitems: register the 'web.address' config

configitems: register the 'web.accesslog' config

web: use '_unset' default value for proxy config method This special value is needed to make sure registered default value are taken in account.

tests: use int() instead of long() in test-pathencode.py

tests: add xrange alias for test-pathencode.py

python3: another 3 tests whitelisted

repair: reliably obtain bytestr of node ids

bundles: turn nbchanges int into a bytestr using pycompat.bytestr Fixes some python 3 failures.

exchange: hit opargs with pycompat.strkwargs before **-ing it Fixes Python 3 problems.

revlog: update signature of dummy addgroup() in bundlerepo and unionrepo Per c8b6ed51386b, 2f5c45fe3a3b and 00e3f909907f.

py3: use 'surrogatepass' error handler to process U+DCxx transparently It's disallowed by default on Python 3. https://docs.python.org/3/library/codecs.html#error-handlers

py3: don't pass bytes to array.array()

py3: wrap bytes in encoding.from/toutf8b() with bytestr

py3: iterate bytes as a byte string in store.lowerencode()

py3: use bytechr() in store._buildlowerencodefun()

store: give name to lowerencode function lambda function isn't easy to track in traceback.

py3: iterate bytes as a byte string in dagparser.py

py3: wrap string constants in dagparser.py with bytestr()

py3: drop use of str() in dagparser.py

dagparser: fix variable name in error message There's no variable named 'type'.

py3: convert function name to bytes in ui.configwith()

drawdag: port to python 3

drawdag: add a couple of doctests to help with python3 porting

drawdag: tagsmod.tag() takes a list of names, not a single name We were getting lucky on Python 2 since we have only one-byte names, but on Python 3 badness was happening.

bruterebase: port to python 3

exchange: use '%d' % x instead of str(x) to encode ints Recommended by Yuya instead of using pycompat.bytestr() in this case.

posix: always pass a native str to unicodedata.normalize's first arg

posix: use slicing to grab a single byte out of a bytes in HFS+ normcase code

encoding: ensure getutf8char always returns a bytestr, never an int

posix: fix HFS+ normcase doctest to produce valid bytes literals in Python 3 We were previously getting lucky on Python 2.

tests: add and remove some (glob) markers I have no idea if these changes are reasonable, but they look like they'd help on the Windows buildbot.

Added signature for changeset 920977f72c7b

Added tag 4.3.2 for changeset 920977f72c7b

merge with i18n

i18n-pt_BR: synchronized with 850d2ec2cf6a

hgwebdir: read 'web.template' untrusted The 'hgweb_mod.py' version of this read it untrusted. For consistency we align the two versions of this code.

uncommit: move fb-extension to core which uncommits a changeset uncommit extension in fb-hgext adds a uncommit command which by default uncommits a changeset and move all the changes to the working directory. If file names are passed, uncommit moves the changes from those files to the working directory and left the changeset with remaining committed files. The uncommit extension in fb-hgext does not creates an empty commit like the one in evolve extension unless user has specified ui.alllowemptycommit to True. The test file added is a combination of tests from test-uncommit.t, test-uncommit-merge.t and test-uncommit-bookmark.t from fb-hgext. .. feature:: A new uncommit extension which provides `hg uncommit` using which one can uncommit part or all of the changeset. This command undoes the effect of a local commit, returning the affected files to their uncommitted state. Differential Revision: https://phab.mercurial-scm.org/D529

show: use consistent (and possibly shorter) node lengths `hg show` makes heavy use of shortest() to limit the length of the node hash. For the "stack" and "work" views, you are often looking at multiple lines of similar output for "lines" of work. It is visually appeasing for things to vertically align. A naive use of {shortest(node, N)} could result in variable length nodes and for the first character of the description to vary by a column or two. We implement a function to determine the longest shortest prefix for a set of revisions. The new function is used to determine the printed node length for all `hg show` views. .. feature:: show: use consistent node length in views Our previous shortest node length of 5 was arbitrarily chosen. shortest() already does the work of ensuring that a partial node isn't ambiguous with an integer revision, which is our primary risk of a collision for very short nodes. It should be safe to go with the shortest node possible. Existing code is also optimized to handle nodes as short as 4. So, we decrease the minimum hash length from 5 to 4. We also add a test demonstrating that prefix collisions increase the node length. .. feature:: show: decrease minimum displayed hash length from 5 to 4 Differential Revision: https://phab.mercurial-scm.org/D558

show: pass the minimum length for nodes as a template keyword This will allow us to make the displayed length configurable and/or dynamic. Differential Revision: https://phab.mercurial-scm.org/D556

cmdutil: allow extra properties to be added to each context The changeset displayer allows setting extra keywords to be available to the templating layer. This patch adds an argument to displaygraph() to pass a dict of extra properties to be available to every changeset. Differential Revision: https://phab.mercurial-scm.org/D555

dirstate: perform transactions with _map using single call, where possible This is in the same style as https://phab.mercurial-scm.org/D493. In general, this replaces patterns such as: ``` f in self._map: entry = self._map[f] ``` with: ``` entry = self._map.get(f): if entry is not None: # use entry ``` Test Plan: `make tests` Differential Revision: https://phab.mercurial-scm.org/D663

extensions: register config item early Config items are likely to be used in during extensions setup. So we much register them before that. For example this apply to the 'win32text.warn' options.

extensions: factor extra data loading out Some of the extra data need to be registered earlier than they currently are (eg: config items). We first factor out the logic to registered them in a small function before reusing it in the next changeset.

configitems: register the 'win32text.warn' config

configitems: register the 'mq.secret' config

configitems: register the 'mq.plain' config

configitems: register the 'mq.keepchanges' config

configitems: register the 'mq.git' config

configitems: register the 'win32mbcs.encoding' config

copytrace: move fast heuristic copytracing algorithm to core copytrace extension in fb-hgext has a heuristic implementation of copy tracing which is faster than the current copy tracing. The heuristic limits the search of copies to just files that are either: 1) Renames in the same directory 2) Moved to other directory with same name The default copytrace implementation is very slow as it finds all the new files that were added from merge base up to the head commit and for each file it checks whether it this was copied or moved version of a different file. Stash@fb did analysis for the above heuristics on the fb repo and found that among 2,443,768 moves/copies there are only 32,234 moves/copies which does not fall under the above heuristics which is approx. 0.013 of total copies. This patch moves the heuristics algorithm under config `experimental.copytrace=heuristics`. While moving fbext to core, this patch removes couple of less useful config options named `sourcecommitlimit` and `maxmovescandidatestocheck`. Tests are also added for the heuristics algorithm, which are basically copied from fbext/tests/test-copytrace.t. The tests follow a pattern creating a server repo and then cloning to a local repo to create public and draft changesets, the distinction which will be useful in upcoming patches. After this patch `experimental.copytrace` has the following behaviour: 1) `off`: turns off copytracing 2) `heuristics`: use the heuristic algorithm added in this patch. 3) everything else: use the full copytracing algorithm .. feature:: A new fast heuristic algorithm for copytracing which assumes that the files moves are either:: 1) Renames in the same directory 2) Moves in other directories with same names You can use this algorithm by setting `experimental.copytrace=heuristics`. Differential Revision: https://phab.mercurial-scm.org/D623

configitems: register the 'convert.svn.startrev' config

configitems: register the 'convert.svn.debugsvnlog' config

configitems: register the 'convert.skiptags' config

configitems: register the 'convert.p4.startrev' config

configitems: register the 'convert.localtimezone' config

configitems: register the 'convert.ignoreancestorcheck' config

configitems: register the 'convert.hg.usebranchnames' config

configitems: register the 'convert.hg.tagsbranch' config

configitems: register the 'convert.hg.startrev' config

configitems: register the 'convert.hg.sourcename' config

configitems: register the 'convert.hg.saverev' config

configitems: register the 'convert.hg.revs' config

configitems: register the 'convert.hg.ignoreerrors' config

configitems: register the 'convert.hg.clonebranches' config

configitems: register the 'convert.git.skipsubmodules' config

configitems: register the 'convert.git.similarity' config

configitems: register the 'convert.git.saverev' config

configitems: register the 'convert.git.renamelimit' config

configitems: register the 'convert.git.remoteprefix' config

configitems: register the 'convert.git.findcopiesharder' config

configitems: register the 'convert.git.extrakeys' config

configitems: register the 'convert.git.committeractions' config

configitems: register the 'convert.cvsps.mergeto' config

configitems: register the 'convert.cvsps.mergefrom' config

configitems: register the 'convert.cvsps.fuzz' config

configitems: register the 'convert.cvsps.cache' config The convert extensions has code a bit all around, but it seems simpler to use a central declaration of options at the root first.

bundle2: move exception handling into part iterator As part of separating the part iteration logic from the part handling logic, let's move the exception handling to the part iterator class. Differential Revision: https://phab.mercurial-scm.org/D705

bundle2: move part counter to partiterator As part of moving the part iterator logic to a separate class, let's move the part counting logic and the output for it. Differential Revision: https://phab.mercurial-scm.org/D704

bundle2: move part iterator a separate class Currently, the part iterator logic is tightly coupled with the part handling logic, which means it's hard to replace the part handling logic without duplicating the part iterator bits. In a future diff we'll want to be able to replace all part handling, so let's begin refactoring the part iterator logic to it's own class. Differential Revision: https://phab.mercurial-scm.org/D703

changegroup: add source parameter to generatemanifests Extensions, like remotefilelog, will want to look at the source of a pull when determining what manifests to add to a changegroup. For instance, on push they will include everything, while on pull they won't. Differential Revision: https://phab.mercurial-scm.org/D686

changegroup: remove changegroup dependency from revlog.addgroup Previously revlog.addgroup would accept a changegroup and a linkmapper and use it to iterate of the deltas. As part of untangling the revlog-changegroup interdependency, let's move the changegroup delta iteration logic to it's own function and pass the simple iterator to the revlog instead. This will make it easier to introduce non-revlogs stores in the future, without reinventing any changegroup specific logic. Differential Revision: https://phab.mercurial-scm.org/D688

revlog: refactor chain variable Previously the addgroup loop would set chain to be the result of self._addrevision(node,...). Since _addrevision now always returns the passed in node, we can drop that behavior and just always set chain = node in the loop. This will be useful in a future patch where we refactor the cg.deltachunk logic to another function and therefore chain disappears entirely from this function. Differential Revision: https://phab.mercurial-scm.org/D699

scmutil: don't append .orig to backups in origbackuppath (BC) When ui.origbackuppath is set, .orig files are stored outside of the working copy, however they still have a .orig suffix appended to them. This can cause unexpected conflicts, particularly when tracked files or directories have .orig at the end. This change removes the .orig suffix from files stored in an out-of-tree origbackuppath. Test Plan: Update and run unit tests. Differential Revision: https://phab.mercurial-scm.org/D679

ssh: fix flakey ssh errors on BSD systems This is a trivial backport of c037fd655b47 performed by augie@google.com, but the change is still really Durham's not mine, so I [augie] am leaving him as the author.

repair: preserve phase also when not using generaldelta (issue5678) It seems like we used to pick the oldest possible version of the changegroup to use for bundles created by the repair module (used e.g. by "hg strip" and for temporary bundles by "hg rebase"). I tried to preserve that behavior when I created the changegroup.safeversion() method in 3b2ac2115464 (changegroup: introduce safeversion(), 2016-01-19). However, we have recently chagned our minds and decided that these commands are only used locally and downgrades are unlikely. That decicion allowed us to start adding obsmarker and phase information to these bundles. However, as the bug report shows, it means we get different behavior e.g. when generaldelta is not enabled (because when it was enabled, it forced us to use bundle2). The commit that actually caused the reported bug was 8e3021fd1a44 (strip: include phases in bundle (BC), 2017-06-15). So, since we now depend on having more information in the bundles, let's make sure we instead pick the newest possible changegroup version. Differential Revision: https://phab.mercurial-scm.org/D715

tests: add test for issue5678 In addition to a test case for the direct problem described in the bug report, this also adds a test case showing how obsmarkers can also get lost when not using generaldelta. Differential Revision: https://phab.mercurial-scm.org/D714

merge: move cwd-missing detection to helper functions This will exist in two places with defered writes, so we want to avoid duplication. Differential Revision: https://phab.mercurial-scm.org/D626

doctest: enable tests by default on Python 3 Still several tests fail mostly because of the string issues, sigh. I'll fix them one by one.

doctest: normalize b'', u'' and exception output on Python 3 The idea is described in the following page. https://dirkjan.ochtman.nl/writing/2014/07/06/single-source-python-23-doctests.html # no-check-commit

doctest: coerce dict.keys() to list Otherwise it would be printed as odict_keys([...]) on Python 3.

doctest: upgrade old-style "except" clause

doctest: use print_function and convert bytes to unicode where needed