py3: use sys.stdout.buffer for binary output in tests/svnxml.py Just following 6c0b1d964537ab62d76c208f5f04ab414814c94e here. Differential Revision: https://phab.mercurial-scm.org/D5027

py3: use '%s' instead of %r in hgext/convert/subversion.py Using '%r' on bytes in python 3 adds b'' prefixes to output which is bad. I don't see a reason why we want to repr() the string here, so just switched to '%s'. This fixes some output failure in one of the subversion test. Differential Revision: https://phab.mercurial-scm.org/D5026

py3: make test-contrib-perf.t work on python 3 Differential Revision: https://phab.mercurial-scm.org/D5025

py3: add 3 more passing tests to whitelist caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5024

py3: add some r'' prefixes in hgext/narrow/narrowwirepeer.py This fixes test-narrow-widen-no-ellipsis.t on Python 3. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5023

py3: r'' prefix default values for mimetypes.guess_mime() As suggested by @yuja in D4967. Differential Revision: https://phab.mercurial-scm.org/D5020

absorb: update help text Update the help text to reflect the recent changes to absorb in D4999. The `-p` option actually only affects the `-a` option: without `-a`, the changes are printed; with `-a` the changes are only printed if `-p` is specified. Reword the help text for `-p` to reflect this. Differential Revision: https://phab.mercurial-scm.org/D5022

fsmonitor: use vfs instead of opener (issue5938) "opener" of localrepository object was dropped at Mercurial 4.3 (or a7e210167c28). "vfs" should be used instead. wlock is required to write into a file under .hg directory. For efficiency, we should change _cmpsets() from: 1. acquire wlock 2. open log file under .hg directory with write mode 3. compare between result of watchman and Mercurial's dirstate logic 4. write out error info into a file, if error is detected 5. release wlock to: 1. compare between result of watchman and Mercurial's dirstate logic 2. acquire wlock, if error is detected 3. open and write error info into a file 4. release wlock But this is another issue.

tests: configure fsmonitor.mode=paranoid always if fsmonitor is used This forces fsmonitor extension execute "paranoid" code path. Strict speaking, we should make fsmonitor-run-tests.py accept own specific options, but there is no code path, which is disabled in "paranoid" mode, at least now. Therefore, this solution seems reasonable enough.

tests: drop files from exclusion list in test-check-module-imports.t Now, these files has no style issue at checking with import-checker.py. BTW, tests/test-imports-checker.t is still excluded, because almost all code fragments in it has intentional importing style violation. Using NO_CHECK_EOF instead of EOF as heredoc limit mark can make import-checker.py ignore such fragments. But keeping these fragments checkable seems still useful: for example, test-imports-checker.t can be used to test whether import-checker.py can detect erroneous code fragment in test script as expected.

tests: use NO_CHECK_EOF for fragments used to test importing via extension test-extension.t is excluded in test-check-module-imports.t, because import-checker.py reports that some python code fragments in it does not import modules in recommended style. These code fragments are used to test importing modules relatively and absolutely via extension (e.g. issue4029, issue5208, and so on). Test specific package hierarchy becomes complicated for this purpose, and it seems reasonable to avoid checking these code fragments with import-checker.py. But on the other hand, other code fragments in test-extension.t should be checked by import-checker.py. Therefore, this patch uses NO_CHECK_EOF limit mark only for python code fragments, which are used to test importing via extension in test-extension.t. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: use NO_CHECK_EOF for fragments having intentional error Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that code fragments embedded in these test scripts have syntax error. But these syntax error is intentional. This patch uses NO_CHECK_EOF instead of EOF as heredoc limit mark for such fragments, in order to make import-checker.py ignore them. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: fix style issue of importing order in test-lock.py test-lock.py is excluded in test-check-module-imports.t, because import-checker.py reports that some stdlibs are imported after local silenttestrunner module. I can not found out any reason why tests/silenttestrunner.py should be imported before some stdlibs from the point of functionality view.

tests: fix style issue of importing hgweb in embedded code fragments Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that hgweb and/or hgwebdir of mercurial.hgweb are not imported in recommended style. To fix this issues, this patch make python code fragments embedded in these files import hgweb from mercurial package at first, and refer hgweb and hgwebdir via imported hgweb.

py3: encode str to bytes These fields are str on Python 2 and 3. This module doesn't import any Mercurial modules. So I just did the str -> bytes inline. Differential Revision: https://phab.mercurial-scm.org/D5010

py3: sprinkle statprof.py with utf-8 encoding Trying to chase the long tail of unicode badness in this file due to frame / code objects holding str everywhere. Differential Revision: https://phab.mercurial-scm.org/D5016

py3: flush stdout This avoids buffering due to mixed print() and sys.stdout.write() by profiling code. And with this, test-profile.t passes on Python 3 \o/. Honestly, I'd be shocked if there weren't more Python 3 encoding issues lingering in the profiling code. That code isn't super high quality and doesn't seem to have thorough test coverage. Differential Revision: https://phab.mercurial-scm.org/D5018

py3: use raw strings in statprof.py This allows main() and argument parsing to work. Differential Revision: https://phab.mercurial-scm.org/D5017

py3: switch from print(..., file=) to write() Because Python 3's print() doesn't like bytes. Differential Revision: https://phab.mercurial-scm.org/D5015

py3: use %d in a few places Differential Revision: https://phab.mercurial-scm.org/D5014

py3: use raw strings for stack names These are used for simple compares. We don't need to normalize to bytes. As part of this, I expanded a set literal so entries are 1 per line. Differential Revision: https://phab.mercurial-scm.org/D5013

py3: use sysbytes for converting code attributes Differential Revision: https://phab.mercurial-scm.org/D5012

py3: use write() instead of print() Because print() expects str and we want to write bytes. There should be no functional changes as part of this. Differential Revision: https://phab.mercurial-scm.org/D5011

py3: use %d to format ints Differential Revision: https://phab.mercurial-scm.org/D5009

py3: convert sorting field to sysstr This is used as part of an attribute name check and needs to be native str. Differential Revision: https://phab.mercurial-scm.org/D5008

py3: encode json output to bytes and use write() Differential Revision: https://phab.mercurial-scm.org/D5007

absorb: prompt user to accept absorb changes by default Change the default absorb behaviour to print the changes out and then prompt the user if they want to absorb the changes or not. Providing the `-p` option still prints the result and exits. A new `-a` option is provided which applies the changes without printing them, giving the equivalent of the old behaviour. Differential Revision: https://phab.mercurial-scm.org/D4999

templatefuncs: add truncate parameter to pad Add a truncate option to pad that additionally truncates the text to the pad width if it is wider. Since color codes can cause a problem with this, when the text is truncated, the color codes are also stripped. Users of the truncate option should label the text outside the pad. Differential Revision: https://phab.mercurial-scm.org/D5000

absorb: print summary of changesets affected Differential Revision: https://phab.mercurial-scm.org/D4998

absorb: use a formatter to generate output Change absorb to use a formatter to generate its output. This allows the use of templates to customize the output. Differential Revision: https://phab.mercurial-scm.org/D4997

tests: fix up test-bad-extension.t's inline extension Upcoming patches will be simpler, I think. Differential Revision: https://phab.mercurial-scm.org/D5003

polib: update to latest release 1.0.7 (upstream rev d75ce6dbbc2a) # no-check-commit third-party code that doesn't match our style Differential Revision: https://phab.mercurial-scm.org/D5001

tests: glob away some annoying py3 differences Differential Revision: https://phab.mercurial-scm.org/D4994

tests: fix a repr on python3 in test-extension.t Differential Revision: https://phab.mercurial-scm.org/D4993

tests: so many b prefixes in test-extension.t # skip-blame because it's b prefixes Differential Revision: https://phab.mercurial-scm.org/D4992

tests: ensure print() statements in test-extension.t all flush Differential Revision: https://phab.mercurial-scm.org/D4991

revlog: update pure nodecache start lookup offset on insertion test-storage.py is failing in pure builds because the population of the node cache isn't accounting for the new starting offset after a revlog insertion. This commit updates the node cache start offset to account for insertions. I'm not 100% convinced this is the ideal solution. But it works and seems correct. Differential Revision: https://phab.mercurial-scm.org/D4996

exchangev2: use filesdata filesdata is a more efficient mechanism for bulk fetching files data for a range of changesets. Let's use it in exchangev2. With this change, a client performing a full clone of mozilla-unified transmits substantially fewer bytes across the wire: before: 139,124,863 bytes sent after: 20,522,499 bytes sent The bulk of the remaining bytes is likely the transfer of ~1M nodes for changesets and manifests. We can eliminate this by making requests in terms of node ranges instead of explicit node lists... Differential Revision: https://phab.mercurial-scm.org/D4982

wireprotov2: define and implement "filesdata" command Previously, the only way to access file revision data was the "filedata" command. This command is useful to have. But, it only allowed resolving revision data for a single file. This meant that clients needed to send 1 command for each tracked path they were seeking data on. Furthermore, those commands would need to enumerate the exact file nodes they wanted data for. This approach meant that clients were sending a lot of data to remotes in order to request file data. e.g. if there were 1M file revisions, we'd need at least 20,000,000 bytes just to encode file nodes! Many clients on the internet don't have that kind of upload capacity. In order to limit the amount of data that clients must send, we'll need more efficient ways to request repository data. This commit defines and implements a new "filesdata" command. This command allows the retrieval of data for multiple files by specifying changeset revisions and optional file patterns. The command figures out what file revisions are "relevant" and sends them in bulk. The logic around choosing which file revisions to send in the case of haveparents not being set is overly simple and will over-send files. We will need more smarts here eventually. (Specifically, the client will need to tell the server which revisions it knows about.) This work is deferred until a later time. Differential Revision: https://phab.mercurial-scm.org/D4981

wireprotov2: extract file object emission to own function An upcoming commit will introduce another caller. Differential Revision: https://phab.mercurial-scm.org/D4980

wireprotov2: change how revisions are specified to changesetdata Right now, we have a handful of arguments for specifying the revisions whose data should be returned. Defining how all these arguments interact when various combinations are present is difficult. This commit establishes a new, generic mechanism for specifying revisions. Instead of a hodgepodge of arguments defining things, we have a list of dicts that specify revision selectors. The final set of revisions is a union of all these selectors. We implement support for specifying revisions based on: * An explicit list of changeset revisions * An explicit list of changeset revisions plus ancestry depth * A DAG range between changeset roots and heads If you squint hard enough, this problem has already been solved by revsets. But I'm reluctant to expose revsets to the wire protocol because that would require servers to implement a revset parser. Plus there are security and performance implications: the set of revision selectors needs to be narrowly and specifically tailored for what is appropriate to be executing on a server. Perhaps there would be a way for us to express the "parse tree" of a revset query, for example. I'm not sure. We can explore this space another time. For now, the new mechanism should bring sufficient flexibility while remaining relatively simple. The selector "types" are prefixed with "changeset" because I plan to add manifest and file-flavored selectors as well. This will enable us to e.g. select file revisions based on a range of changeset revisions. Differential Revision: https://phab.mercurial-scm.org/D4979

wireprotov2: stop sending phase updates for base revisions This feature is broken and doesn't work properly in all scenarios. e.g. if we have the following DAGs: client server D draft C draft C draft B draft B public A public A public The current code would only send the phase data for C. The client wouldn't see that B moved from draft to public. This feature will be restored in a future commit. For now, it is making refactoring of how revisions are specified in the wire protocol a bit difficult... Differential Revision: https://phab.mercurial-scm.org/D4978

debugcommands: support wrapping long lines If a line within a block is indented more than the line that came before, we automatically concatenate it with the previous line. This allows us to pretty format data. This will make tests easier to read. At some point we may just want to evaluate entire blocks as Python code or something, as even with this change, things aren't perfect, as we can't e.g. have formatting like: foo eval:[ True ] But this is strictly better than before, where we couldn't wrap long lines. Differential Revision: https://phab.mercurial-scm.org/D4977

exchangev2: honor server advertised manifestdata recommended batch size Let's plug the client up to the server-advertised recommended batch size for manifestdata requests. Differential Revision: https://phab.mercurial-scm.org/D4976

wireprotov2: advertise recommended batch size for requests Currently, exchangev2 hardcodes the batch size for how many revisions to fetch per command request. A single value is not appropriate for every repository because some repositories may have a drastically different "shape" from other repositories. e.g. a repo with lots of small files may benefit from larger batch sizes than a repo with lots of large files. And depending on caching used by the server, the server may wish to control the number of commands (to e.g. mitigate overhead of following content redirects). This commit teaches wireprotov2 commands to declare extra metadata which is advertised as part of the command descriptor. The manifestdata command has been taught to advertise a recommended batch size for requests. Differential Revision: https://phab.mercurial-scm.org/D4975

httppeer: expose API descriptor on httpv2peer The API descriptor in wireprotov2 is much more expressive than space-delimited tokens and it will be difficult to define methods to query it in all of the ways we'll want to query it. So let's just declare defeat and expose the API descriptor on the peer instance. As part of this, we define a new interface for version 2 peers, fulfilling a TODO in the process. Differential Revision: https://phab.mercurial-scm.org/D4974

tests: use baseurl instead of advertisedbaseurl The distinction matters for e.g. hosts behind load balancers. But for the test environment, it doesn't matter. For whatever reason, advertisedbaseurl is resolving to http://1.0.0.127.in-addr.arpa:$HGPORT on my MBP. This hostname fails to resolve, causing the test to fail. No clue what's up with that behavior. Differential Revision: https://phab.mercurial-scm.org/D4973

py3: another one started passing Differential Revision: https://phab.mercurial-scm.org/D4990

py3: one more passing test Differential Revision: https://phab.mercurial-scm.org/D4989

scmutil: wrap locker information in bytestr before repr()ing it Differential Revision: https://phab.mercurial-scm.org/D4988

py3: more passing tests Differential Revision: https://phab.mercurial-scm.org/D4987

py3: whitelist another passing test Differential Revision: https://phab.mercurial-scm.org/D4986

obsolete: convert error string to a sysstr This should be okay because we're just %-formatting a hash into a localized string. Differential Revision: https://phab.mercurial-scm.org/D4985

tests: ensure the python 3 passing test list stays sorted Differential Revision: https://phab.mercurial-scm.org/D4984

py3: sort passing test list Differential Revision: https://phab.mercurial-scm.org/D4983

py3: tweak stdout writing in test-hgweb-no-path-info.t We want to write bytes for convenience. This requires sys.stdout.buffer. But using sys.stdout.buffer introducing buffered output. So we sprinkle code with sys.stdout.flush() to force immediate writes. After all that, Python 3 was emitting b'' prefixed output for errors. So we only print errors if there were some. There aren't, so b'' don't come into play and output is identical in Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D4972

py3: use .+ instead of .* in regexp pattern Python 3.7 changed the behavior of re.sub(). See https://bugs.python.org/issue33585. The new code should work on old and new Pythons. Differential Revision: https://phab.mercurial-scm.org/D4971

py3: coerce bytestr to bytes to appease urllib.parse.quote_from_bytes() Differential Revision: https://phab.mercurial-scm.org/D4969

py3: pass str and return bytes from mimetypes.guess_type() This function wants a str (which represents a path) and returns a str. We normalize input to str and output to bytes. Differential Revision: https://phab.mercurial-scm.org/D4967

py3: use bytes literal in test-hgweb-json.t # skip-blame just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4970