tests: add lots of globs and conditional output lines The test now passes on Python 3. I'm going to do one follow-up where I'll adjust some grep calls to improve our checking of things on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5004

releasenotes: fix remaining bytes/unicode issues caught by tests All tests now pass. Differential Revision: https://phab.mercurial-scm.org/D5054

relnotes: port to Python 3 The big annoyance here was having to feed textwrap unicodes instead of bytes, but it all seems to work. Differential Revision: https://phab.mercurial-scm.org/D5053

py3: use __code__ instead of func_code to test if func is cachable Surprisingly, this fixes test-cache-abuse.t, which would generate a different revbranch cache if filtered set had no cache.

py3: pass in system string to vars(branchmap).__contains__()

test-bookmarks-pushpull: use correct tmp dir for no-bm-move.sh test-bookmarks-pushpull.t writes a temporary file in $TESTDIR instead of $TESTTMP. Make it use $TESTTMP instead. Differential Revision: https://phab.mercurial-scm.org/D5057

py3: add some glob to make output compatible with python 3 This makes test-pathconflicts-basic.t and test-origbackup-conflict.t passing on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5042

py3: add a b'' prefix in tests/test-requires.t This makes the test pass on Python 3. # skip-blame because just b'' prefix. Differential Revision: https://phab.mercurial-scm.org/D5041

py3: use '%d' for rev nums instead of '%s' This makes test-close-head.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5040

py3: more globing of things to make output compatible between py2 and py3 This also makes the test-ssh-repoerror.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5038

py3: use stringutil.pprint() instead of '%r' This makes test-pager-legacy.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5037

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D5036

py3: add some py3 specific output to test This makes the test work on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5035

py3: use stringutil.pprint() in color.py instead of '%r' This fixes the test-status-color.t on python 3. Differential Revision: https://phab.mercurial-scm.org/D5034

py3: add b'' prefixes in tests/test-progress.t This makes the test pass on Python 3. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5033

py3: add b'' prefixes to tests/test-keyword.t This test is now pretty close to passing on Python 3. # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D5032

py3: use stringutil.pprint() if we are printing bool values Differential Revision: https://phab.mercurial-scm.org/D5031

py3: glob some difference between py2 and py3 output On py2, the directory names are under quotes and on py3 they are not and I don't know why. Differential Revision: https://phab.mercurial-scm.org/D5030

py3: byteify tests/wireprotosimplecache.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5029

py3: add b'' prefixes in tests/test-help.t This takes us close to make the test passing. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5028

py3: use sys.stdout.buffer for binary output in tests/svnxml.py Just following 6c0b1d964537ab62d76c208f5f04ab414814c94e here. Differential Revision: https://phab.mercurial-scm.org/D5027

py3: use '%s' instead of %r in hgext/convert/subversion.py Using '%r' on bytes in python 3 adds b'' prefixes to output which is bad. I don't see a reason why we want to repr() the string here, so just switched to '%s'. This fixes some output failure in one of the subversion test. Differential Revision: https://phab.mercurial-scm.org/D5026

py3: make test-contrib-perf.t work on python 3 Differential Revision: https://phab.mercurial-scm.org/D5025

py3: add 3 more passing tests to whitelist caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5024

py3: add some r'' prefixes in hgext/narrow/narrowwirepeer.py This fixes test-narrow-widen-no-ellipsis.t on Python 3. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5023

py3: r'' prefix default values for mimetypes.guess_mime() As suggested by @yuja in D4967. Differential Revision: https://phab.mercurial-scm.org/D5020

absorb: update help text Update the help text to reflect the recent changes to absorb in D4999. The `-p` option actually only affects the `-a` option: without `-a`, the changes are printed; with `-a` the changes are only printed if `-p` is specified. Reword the help text for `-p` to reflect this. Differential Revision: https://phab.mercurial-scm.org/D5022

fsmonitor: use vfs instead of opener (issue5938) "opener" of localrepository object was dropped at Mercurial 4.3 (or a7e210167c28). "vfs" should be used instead. wlock is required to write into a file under .hg directory. For efficiency, we should change _cmpsets() from: 1. acquire wlock 2. open log file under .hg directory with write mode 3. compare between result of watchman and Mercurial's dirstate logic 4. write out error info into a file, if error is detected 5. release wlock to: 1. compare between result of watchman and Mercurial's dirstate logic 2. acquire wlock, if error is detected 3. open and write error info into a file 4. release wlock But this is another issue.

tests: configure fsmonitor.mode=paranoid always if fsmonitor is used This forces fsmonitor extension execute "paranoid" code path. Strict speaking, we should make fsmonitor-run-tests.py accept own specific options, but there is no code path, which is disabled in "paranoid" mode, at least now. Therefore, this solution seems reasonable enough.

tests: drop files from exclusion list in test-check-module-imports.t Now, these files has no style issue at checking with import-checker.py. BTW, tests/test-imports-checker.t is still excluded, because almost all code fragments in it has intentional importing style violation. Using NO_CHECK_EOF instead of EOF as heredoc limit mark can make import-checker.py ignore such fragments. But keeping these fragments checkable seems still useful: for example, test-imports-checker.t can be used to test whether import-checker.py can detect erroneous code fragment in test script as expected.

tests: use NO_CHECK_EOF for fragments used to test importing via extension test-extension.t is excluded in test-check-module-imports.t, because import-checker.py reports that some python code fragments in it does not import modules in recommended style. These code fragments are used to test importing modules relatively and absolutely via extension (e.g. issue4029, issue5208, and so on). Test specific package hierarchy becomes complicated for this purpose, and it seems reasonable to avoid checking these code fragments with import-checker.py. But on the other hand, other code fragments in test-extension.t should be checked by import-checker.py. Therefore, this patch uses NO_CHECK_EOF limit mark only for python code fragments, which are used to test importing via extension in test-extension.t. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: use NO_CHECK_EOF for fragments having intentional error Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that code fragments embedded in these test scripts have syntax error. But these syntax error is intentional. This patch uses NO_CHECK_EOF instead of EOF as heredoc limit mark for such fragments, in order to make import-checker.py ignore them. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: fix style issue of importing order in test-lock.py test-lock.py is excluded in test-check-module-imports.t, because import-checker.py reports that some stdlibs are imported after local silenttestrunner module. I can not found out any reason why tests/silenttestrunner.py should be imported before some stdlibs from the point of functionality view.

tests: fix style issue of importing hgweb in embedded code fragments Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that hgweb and/or hgwebdir of mercurial.hgweb are not imported in recommended style. To fix this issues, this patch make python code fragments embedded in these files import hgweb from mercurial package at first, and refer hgweb and hgwebdir via imported hgweb.

py3: encode str to bytes These fields are str on Python 2 and 3. This module doesn't import any Mercurial modules. So I just did the str -> bytes inline. Differential Revision: https://phab.mercurial-scm.org/D5010

py3: sprinkle statprof.py with utf-8 encoding Trying to chase the long tail of unicode badness in this file due to frame / code objects holding str everywhere. Differential Revision: https://phab.mercurial-scm.org/D5016

py3: flush stdout This avoids buffering due to mixed print() and sys.stdout.write() by profiling code. And with this, test-profile.t passes on Python 3 \o/. Honestly, I'd be shocked if there weren't more Python 3 encoding issues lingering in the profiling code. That code isn't super high quality and doesn't seem to have thorough test coverage. Differential Revision: https://phab.mercurial-scm.org/D5018

py3: use raw strings in statprof.py This allows main() and argument parsing to work. Differential Revision: https://phab.mercurial-scm.org/D5017

py3: switch from print(..., file=) to write() Because Python 3's print() doesn't like bytes. Differential Revision: https://phab.mercurial-scm.org/D5015

py3: use %d in a few places Differential Revision: https://phab.mercurial-scm.org/D5014

py3: use raw strings for stack names These are used for simple compares. We don't need to normalize to bytes. As part of this, I expanded a set literal so entries are 1 per line. Differential Revision: https://phab.mercurial-scm.org/D5013

py3: use sysbytes for converting code attributes Differential Revision: https://phab.mercurial-scm.org/D5012

py3: use write() instead of print() Because print() expects str and we want to write bytes. There should be no functional changes as part of this. Differential Revision: https://phab.mercurial-scm.org/D5011

py3: use %d to format ints Differential Revision: https://phab.mercurial-scm.org/D5009

py3: convert sorting field to sysstr This is used as part of an attribute name check and needs to be native str. Differential Revision: https://phab.mercurial-scm.org/D5008

py3: encode json output to bytes and use write() Differential Revision: https://phab.mercurial-scm.org/D5007

absorb: prompt user to accept absorb changes by default Change the default absorb behaviour to print the changes out and then prompt the user if they want to absorb the changes or not. Providing the `-p` option still prints the result and exits. A new `-a` option is provided which applies the changes without printing them, giving the equivalent of the old behaviour. Differential Revision: https://phab.mercurial-scm.org/D4999

templatefuncs: add truncate parameter to pad Add a truncate option to pad that additionally truncates the text to the pad width if it is wider. Since color codes can cause a problem with this, when the text is truncated, the color codes are also stripped. Users of the truncate option should label the text outside the pad. Differential Revision: https://phab.mercurial-scm.org/D5000

absorb: print summary of changesets affected Differential Revision: https://phab.mercurial-scm.org/D4998

absorb: use a formatter to generate output Change absorb to use a formatter to generate its output. This allows the use of templates to customize the output. Differential Revision: https://phab.mercurial-scm.org/D4997

tests: fix up test-bad-extension.t's inline extension Upcoming patches will be simpler, I think. Differential Revision: https://phab.mercurial-scm.org/D5003

polib: update to latest release 1.0.7 (upstream rev d75ce6dbbc2a) # no-check-commit third-party code that doesn't match our style Differential Revision: https://phab.mercurial-scm.org/D5001

tests: glob away some annoying py3 differences Differential Revision: https://phab.mercurial-scm.org/D4994

tests: fix a repr on python3 in test-extension.t Differential Revision: https://phab.mercurial-scm.org/D4993

tests: so many b prefixes in test-extension.t # skip-blame because it's b prefixes Differential Revision: https://phab.mercurial-scm.org/D4992

tests: ensure print() statements in test-extension.t all flush Differential Revision: https://phab.mercurial-scm.org/D4991

revlog: update pure nodecache start lookup offset on insertion test-storage.py is failing in pure builds because the population of the node cache isn't accounting for the new starting offset after a revlog insertion. This commit updates the node cache start offset to account for insertions. I'm not 100% convinced this is the ideal solution. But it works and seems correct. Differential Revision: https://phab.mercurial-scm.org/D4996

exchangev2: use filesdata filesdata is a more efficient mechanism for bulk fetching files data for a range of changesets. Let's use it in exchangev2. With this change, a client performing a full clone of mozilla-unified transmits substantially fewer bytes across the wire: before: 139,124,863 bytes sent after: 20,522,499 bytes sent The bulk of the remaining bytes is likely the transfer of ~1M nodes for changesets and manifests. We can eliminate this by making requests in terms of node ranges instead of explicit node lists... Differential Revision: https://phab.mercurial-scm.org/D4982

wireprotov2: define and implement "filesdata" command Previously, the only way to access file revision data was the "filedata" command. This command is useful to have. But, it only allowed resolving revision data for a single file. This meant that clients needed to send 1 command for each tracked path they were seeking data on. Furthermore, those commands would need to enumerate the exact file nodes they wanted data for. This approach meant that clients were sending a lot of data to remotes in order to request file data. e.g. if there were 1M file revisions, we'd need at least 20,000,000 bytes just to encode file nodes! Many clients on the internet don't have that kind of upload capacity. In order to limit the amount of data that clients must send, we'll need more efficient ways to request repository data. This commit defines and implements a new "filesdata" command. This command allows the retrieval of data for multiple files by specifying changeset revisions and optional file patterns. The command figures out what file revisions are "relevant" and sends them in bulk. The logic around choosing which file revisions to send in the case of haveparents not being set is overly simple and will over-send files. We will need more smarts here eventually. (Specifically, the client will need to tell the server which revisions it knows about.) This work is deferred until a later time. Differential Revision: https://phab.mercurial-scm.org/D4981

wireprotov2: extract file object emission to own function An upcoming commit will introduce another caller. Differential Revision: https://phab.mercurial-scm.org/D4980