wireprotoserver: rename webproto to httpv1protocolhandler This matches our naming convention for the SSH server's protocol handler. Differential Revision: https://phab.mercurial-scm.org/D2218

wireproto: improve docstring for "hello" Differential Revision: https://phab.mercurial-scm.org/D2217

httppeer: remove httpspeer All it did was verify at construction time that Mercurial supports TLS. instance() is what's used to construct peer instances. So we can just inline this check into that function and do away with the type variant. Differential Revision: https://phab.mercurial-scm.org/D2216

httppeer: remove redundant code to fetch capabilities _fetchcaps() is called by httppeer.instance(), which is the only instantiator of httppeer. Since _fetchcaps() always sets self._caps and since 197d10e157ce removed the fallback for cases where the remote doesn't support capabilities, we can remove some dead code from httppeer.capabilities(). Differential Revision: https://phab.mercurial-scm.org/D2215

httppeer: change logic around argument handling The code to process arguments only makes sense if there are arguments. So change an "else" to "elif args", remove an "if" that isn't necessary, and add some docs for good measure. Differential Revision: https://phab.mercurial-scm.org/D2214

tests: test using both versions of SSH protocol Now that the version 2 of the SSH protocol is usable in core, we can start actively testing it more widely outside of low-level protocol tests. We add #testcases variants to a handful of tests so we exercise both version 1 and version 2 of the SSH protocol when testing. This will allow us to more easily find regressions and variances as protocol 2 is developed. It will also make it easier to continue testing with protocol version 1 once version 2 is enabled by default. There are a handful of tests using ssh:// that should also gain test variances. One - test-push-race.t - already has a #testcases. This would require combinatorial cases. I didn't want to go down that rabbit hole, so that test is unchanged. Thinking aloud, there is probably an opportunity to automatically run tests with multiple server/protocol implementations. Ideally any test that performed server interaction would run with all supported server implementations and protocols so we could find variances between servers and protocols. But this has been a long-standing issue with our test harness. I don't think it is an easily solved problem. But it would be nice... Differential Revision: https://phab.mercurial-scm.org/D2206

sshpeer: log remote capabilities after protocol upgrade This helps reduce variance with version 1 and will help prevent test output divergence as we start testing protocol version 2 more widely. Differential Revision: https://phab.mercurial-scm.org/D2205

wireprotoserver: handle SSH protocol version 2 upgrade requests This commit teaches the SSH server to recognize the "upgrade" request line that clients send when they wish to switch the channel to version 2 of the SSH protocol. Servers don't honor upgrade requests unless an experimental config option is set. Since the built-in server now supports upgrade requests, our test server to test the handshake has been deleted. Existing tests use the built-in server and their output doesn't change. The upgrade is handled in our state machine. The end result is a bit wonky, as the server transitions back to version 1 state immediately after upgrading. But this will change as soon as version 2 has an actual protocol that differs from version 1. Tests demonstrating that the new server is a bit more strict about the upgrade handshake have been added. Differential Revision: https://phab.mercurial-scm.org/D2204

wireprotoserver: move SSH server operation to a standalone function The server-side processing logic will soon get a bit more complicated in order to handle protocol switches. We will use a state machine to help make the transitions clearer. To prepare for this, we move SSH server operation into a standalone function. We structure it as a very simple state machine. It only has two states for now, with one state containing the bulk of the logic. But things will evolve shortly. Differential Revision: https://phab.mercurial-scm.org/D2203

py3: stringify integer with %d instead of bytes() The unbundle wire protocol command now returns a properly formatted reply in Python 3. Differential Revision: https://phab.mercurial-scm.org/D2274

py3: add b'' to test-sshserver.py # skip-blame because adding b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2273

py3: add b'' to config options in test extension # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2272

manifest: add support for including directories outside narrowspec When using tree manifests and the client doesn't have a directory, we have two choices for what to do with "hg manifest" output: 1) ignore the directory, and 2) include the directory (not files within it). For "hg files", we decided to ignore the directories (and files) outside the narrowspec. If we choose to not include directories outside the narrowspec, then I think we should also make sure we don't include files outside the narrowspec. I also think we should add --outside-narrow flag (or other name). Thus, whichever way we go, I think we should have a way of displaying paths (files or directories) outside the narrowspec. For that we'll need to handle the 't' flag that narrowhg uses, and that's what this patch adds support for. Differential Revision: https://phab.mercurial-scm.org/D2235

narrow: restrict manifest iteration by using manifest.walk(matcher) This is only for root nodes, so it shouldn't really matter (they're rarely huge), but seems cleaner. Differential Revision: https://phab.mercurial-scm.org/D2234

narrow: only diff manifest part within narrowspec when generating changegroup Since 959ebff3505a (manifest: add match argument to diff and filesnotin, 2017-03-07), we have a more efficient way of diffing manifests while applying a matcher. Let's use that while generating narrowed changegroups, so we avoid diffing parts of the manifest that don't match the narrowspec. Differential Revision: https://phab.mercurial-scm.org/D2233

cmdutil: build "%m" (desc|firstline) in makefilename()

cmdutil: rewrite makefilename() to use ctx methods

cmdutil: pass ctx to makefileobj() in place of repo/node pair (API)

cmdutil: pass ctx to makefilename() in place of repo/node pair (API)

cmdutil: make node parameter of makefileobj() mandatory (API) (repo, node) pair will be replaced with ctx, so makefilename() can be easily ported to templater.

progress: use '%*d' to pad progress value Follows up 7f5108e58083. The problem of '% Nd' is that ' ' means we want {' ' or '-'} as a sign character. We should instead write '%Nd' to pad up to N characters with space, and N can be '*'.

py3: stringify IOError/OSError without loosing local character Follows up fa4d333cac58. An environment error may contain non-ascii characters on Windows, which should be encoded to a platform-native string.

dirstate: drop explicit files that shouldn't match (BC) (issue4679) Before, wctx.walk() could include files excluded by -X pattern, which disagrees with wctx.matches() and ctx.walk()/matches() behavior. This patch fixes the problem by testing stat results against the matcher if the matcher may contain false paths. I have no idea if the fix should be made before the workaround for case- insensitive filesystems, but that shouldn't matter since match.anypats() means 'not match.isexact()'. This patch also makes narrow and sparse extensions to not exclude explicit paths on walk() because they appear to depend on the buggy behavior. More detailed analysis about this issue by Martin von Zweigbergk: "I think it's just an unintended consequence of how the dirstate walk works, but I'm not sure. The exception for explicit files also bothered me when I was working on the matcher code a year or so ago. I actually added the exception to the matcher code because I thought it was always working like that (not just for dirstate) in a83a7d27911e (match: handle excludes using new differencematcher, 2017-05-16). It was only recently that Yuya realized that it used to be inconsistent and that I probably made it consistently bad because I didn't realize it was inconsistent to start with, see 821d8a5ab4ff (match: do not weirdly include explicit files excluded by -X option, 2018-01-16)." .. bc:: Working-directory commands now respect ``-X PATTERN`` no matter if PATTERN matches explicitly-specified FILEs. For example, ``hg add foo -X foo`` no longer add the file ``foo``.

log: add TODO comments about --line-range processing

log: factor out function to feed revisions to displayer

graphlog: unblock --line-range option It should work now.

graphlog: deduplicate preprocessing of log command

hgweb: show users recorded in obsolescence markers It's useful to see who obsoleted a commit, because it's not always done by its author (hg-committed is a good example, because people rebase stacks of commits made by various people). Usernames are obfuscated, but look correct (e.g. "test" is "test").

revert: use an exact matcher in interactive diff selection (issue5789) When going through _performrevert() in the interactive case, we build a matcher with files to revert and pass it patch.diff() for later selection of diff hunks to revert. The files set used to build the matcher comes from dirstate and accounts for patterns explicitly passed to revert ('hg revert -i <file>') and, in case of nonexistent pattern, this set is empty (which is expected). Unfortunately, the matcher built from scmutil.match(ctx, []) is wrong as it leads patch.diff() to rebuild a 'changes' tuple with dirstate information, ignoring user-specified pattern. This leads to the situation described in issue5789, where one gets prompted about reverting files unrelated to specified patterns because they made a typo or so. We fix this by building an exact matcher with the correct set of file paths (built earlier). Thanks to Yuya Nishihara for suggesting the correct fix.

tests: add a test demonstrate that 'revert -i' ignores nonexistent patterns As described in issue5789, when revert is called through: hg revert -i <some file that does not exist> we'd expect the command to abort early. Currently, it just warns about missing file but prompt about files unrelated to user arguments.

py3: slice over bytes to prevent getting ascii values Differential Revision: https://phab.mercurial-scm.org/D2271

py3: converts bytes to pycompat.bytestr to get bytechrs while enumerating Differential Revision: https://phab.mercurial-scm.org/D2270

py3: use pycompat.{bytes|str}kwargs in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2268

py3: replace file() with open() in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2267

py3: use "%d" for converting int to bytes in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2266

py3: open files in bytes mode in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2265

py3: use pycompat.bytestr instead of str Differential Revision: https://phab.mercurial-scm.org/D2264

py3: use "%d" to convert integers to bytes Differential Revision: https://phab.mercurial-scm.org/D2263

py3: add couple of missing b'' in fakemergerecord.py # skip-blame as we are just adding b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2262

py3: add b'' to make sure regex pattern are bytes in hgweb/webutil.py # skip-blame because we are just adding b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2260

py3: add b'' prefixes in test-worker.t # skip-blame because we are just adding b'' Differential Revision: https://phab.mercurial-scm.org/D2258

py3: add b'' prefixes to make printrevset.py work in test-glog.t # skip-blame because we are just adding b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2257

py3: slice over bytes to prevent getting ascii values Differential Revision: https://phab.mercurial-scm.org/D2256

python3: whitelist another 5 passing tests Differential Revision: https://phab.mercurial-scm.org/D2252

py3: introduce and use pycompat.getargspec This is getfullargspec on py3, which means we can't use namedtuple named accessors for all fields (eg keywords from getargspec is varkw from getfullargspec, with the same meaning). Solves some warning issues on Python 3. I didn't clean up httpclient because that's vendored code I think we should probably discard, and I didn't touch the manpage generator for now either. Differential Revision: https://phab.mercurial-scm.org/D2251

merge: coerce nodes to bytes, not str Differential Revision: https://phab.mercurial-scm.org/D2250

py3: whitelist more passing tests I had some good success tonight with histedit tests. test-histedit-fold.t is now low-hanging fruit: it is only failing because the logic for setting environment variables in hook.py is slightly busted. Specifically, hook.py line 142 should be making sure it puts strings in the env dict, not bytes, to avoid b'' goo in environment variables. Differential Revision: https://phab.mercurial-scm.org/D2249

histedit: binascii.unhexlify (aka node.bin) throws new exception type on py3 Lucky for us, the exception type exists on 2.7, so we can include it in the except block without any extra work. Differential Revision: https://phab.mercurial-scm.org/D2244

treemanifest: add an optimized __nonzero__() We use bool(manifest) in at least some places: localrepo.py:1730 hgweb/webcommands.py:524 Since the treemanifest class doesn't define __nonzero__() (before this patch), bool(manifest) will instead call __len__(), which can be slow for treemanifests. This patch may make a noticeable difference in the localrepo case above, but that only happens when committing a merge and I haven't timed it. Note that Durham already added a __nonzero__ implementation to manifestdict in b19291e5d506 (manifest: add __nonzero__ method, 2016-11-03). Differential Revision: https://phab.mercurial-scm.org/D2232

tests: port fakemergerecord to python3 Differential Revision: https://phab.mercurial-scm.org/D2248

tests: add some b-prefixes on local script in test-histedit-arguments # skip-blame its just more b prefixes and a file in binary mode Differential Revision: https://phab.mercurial-scm.org/D2247

tests: add bytes prefixes to test-histedit-fold.t's local extension # skip-blame b prefixes Differential Revision: https://phab.mercurial-scm.org/D2246

histedit: fix silly bug that was unpacking a bytestr before writing it I have this foggy notion that popbuffer() might have returned a list in the past, but it doesn't anymore, and this was breaking on Python 3. As a bonus, it's probably a ton faster on Python 2 now. Differential Revision: https://phab.mercurial-scm.org/D2243

py3: whitelist another 8 passing tests Differential Revision: https://phab.mercurial-scm.org/D2242

histedit: modernize write of histedit-last-edit file Differential Revision: https://phab.mercurial-scm.org/D2241

histedit: convert bool to bytestring manually Yes, this format has 'True' and 'False' in it. I'm sorry, as it's almost certainly my fault. Differential Revision: https://phab.mercurial-scm.org/D2240

narrowcommands: more byteskwargs cleanup Differential Revision: https://phab.mercurial-scm.org/D2239

narrowmerge: iterate over a copy of dict items so we can mutate the dict Differential Revision: https://phab.mercurial-scm.org/D2238

contrib: ban $RANDOM using check-code Differential Revision: https://phab.mercurial-scm.org/D2237

tests: remove use of bashism $RANDOM to fix test-narrow.t on non-bash shells This test passed on our workstations, including on Macs, so we never noticed, but it fails on both the Linux and FreeBSD buildbots. Today I learned about a bash feature, wherein $RANDOM gives a random int every time you read it. check-code rule to follow. Differential Revision: https://phab.mercurial-scm.org/D2236

narrow: fix for getting the username when running http server Differential Revision: https://phab.mercurial-scm.org/D2231

tests: remove code to support Mercurial 4.3 Now that narrow lives in core, we don't need the legacy support. Differential Revision: https://phab.mercurial-scm.org/D2202

narrowspec: move module into core Having support for parsing the narrow specification in core is necessary for moving many other parts of narrow to core. We do still want to harmonize the narrow spec with the sparse spec. And the format needs to be documented. But this shouldn't hold up the code moving to core. Differential Revision: https://phab.mercurial-scm.org/D2201

hg: move share._getsrcrepo into core The fact we were calling this from extensions was a sign that it should live in core. We were also able to remove some extra attribute aliases from the share extension. Differential Revision: https://phab.mercurial-scm.org/D2200

tests: glob over line number Differential Revision: https://phab.mercurial-scm.org/D2199

tests: remove references to bundle2-exp config option This option was deleted a while ago. We don't even alias it in core. Differential Revision: https://phab.mercurial-scm.org/D2198

py3: whitelist another six passing tests Differential Revision: https://phab.mercurial-scm.org/D2230

narrowrepo: filter() is a generator on py3, wrap in list() Was at the top of Python 3 exceptions. Differential Revision: https://phab.mercurial-scm.org/D2229

narrowcommands: use pycompat.{bytes,str}kwargs Differential Revision: https://phab.mercurial-scm.org/D2228

debugcommands: mergestate version is an int, use %d on it Differential Revision: https://phab.mercurial-scm.org/D2227

progress: use %d to format ints instead of %s Due to behavioral changes between '% Ns' and '% Nd' this has some unfortunate extra dancing. I'm not sure of a better way to solve this problem. Differential Revision: https://phab.mercurial-scm.org/D2226

progress: determine padding width portably Differential Revision: https://phab.mercurial-scm.org/D2225

branchmap: wrap builtin exception in bytes for logging Differential Revision: https://phab.mercurial-scm.org/D2224

remotenames: add three new revsets related to remotenames This patch adds three new revsets 'remotenames', 'remotebookmarks' and 'remotebranches' which will return changesets which have remotenames, remotebookmarks and remotebranches on them respectively. The remotenames revset exist in hgremotenames extension and is moved from there whereas rest of the two revsets are introduced in this patch. hgremotenames: https://bitbucket.org/seanfarley/hgremotenames Differential Revision: https://phab.mercurial-scm.org/D2002

convert: fix line ending of mapfile and commit.desc file Follows up 42a393ea56d2. CRLF vs LF doesn't really matter as we do strip() or rstrip() on read, but mixing them isn't nice. So let's restore the old behavior. I don't know whether CVS/Root, CVS/Repository, and ~/.cvspass are written in native line ending, so I leave them read as binary files.

py3: port string formatting `skipped` is a set of integer revs here. So cast integers to strings using %d. None doesn't cast to an empty string on Python 3. So use '' explicitly. Differential Revision: https://phab.mercurial-scm.org/D2146

bdiff: write a native version of splitnewlines ./hg perfunidiff mercurial/manifest.py 0 --count 500 --profile before: ! wall 0.309280 comb 0.350000 user 0.290000 sys 0.060000 (best of 32) ./hg perfunidiff mercurial/manifest.py 0 --count 500 --profile after: ! wall 0.241572 comb 0.260000 user 0.240000 sys 0.020000 (best of 39) so it's about 20% faster. I hate Python. I wish we could usefully write this in Rust, but it doesn't look like that's realistic without using the cpython crate, which I'd still like to avoid. Differential Revision: https://phab.mercurial-scm.org/D1973

label: enforce the lack of leading or trailing white space In practice, all commands create label are currently striping external white space. Let us enforce this logic at a lower level before starting to rely on it elsewhere.

buildrpm: bump bundled docutils version to 0.14 when building for centos{5,6} when building rpm packages for centos 5 and 6, we bundle a mercurial-specific version of docutils in /opt/python-hg/lib/python2.7/site-packages/docutils Let's bump the embedded docutils version from 0.12 (released in 2014) to 0.14 (latest as of today)

buildrpm: bump bundled python version to 2.7.14 when building for centos{5,6} when building rpm packages for centos 5 and 6, we bundle a mercurial-specific version of python 2.7 in /opt/python-hg Let's bump the embedded python version from 2.7.10 (released in 2015) to 2.7.14 (latest as of today)

merge: invoke scmutil.fileprefetchhooks() prior to applying updates This moves the file list calculation into core, so other extensions don't need to duplicate it.

revert: drop the remnant of the prefetchfiles hook

cat: migrate to the fileprefetch callback mechanism

archive: migrate to the fileprefetch callback mechanism

lfs: migrate to the fileprefetch callback mechanism

cmdutil: convert the prefetchfiles() hook to a callback mechanism (API) Yuya suggested a list of callbacks instead of function wrapping. This means that one extension can't block another from processing the list. .. api:: File prefetching is now handled by registering a callback with scmutil.fileprefetchhooks.

test-narrow: partially stabilize on Windows test-narrow-strip.t is still broken. I don't see any feature condititionals, so it might be an actual problem.

py3: whitelist test-convert-clonebranches.t Differential Revision: https://phab.mercurial-scm.org/D2223

convert: make hg sha1 regex consistently be a bytes Differential Revision: https://phab.mercurial-scm.org/D2222

convcmd: pass encoding name as a sysstr Differential Revision: https://phab.mercurial-scm.org/D2221

convert: open all files in binary mode Differential Revision: https://phab.mercurial-scm.org/D2220

py3: whitelist another 11 passing tests Differential Revision: https://phab.mercurial-scm.org/D2213

util: convert traceback-related sysstrs to sysbytes in getstackframes With this change, test-devel-warnings.t passes except for differences in how ProgrammingError is formatted and one mysterious traceback inside of transaction.__del__. Differential Revision: https://phab.mercurial-scm.org/D2212

util: format line number of stack trace using %d Required on Python 3, functional on Python 2. Differential Revision: https://phab.mercurial-scm.org/D2211

util: call warnings.warn() with a sysstr in nouideprecwarn This preserves nouideprecwarn wanting a bytes, which is consistent with the rest of hg. Differential Revision: https://phab.mercurial-scm.org/D2210

ui: convert stack traces to sysbytes before logging They're coming back as unicodes, so sysbytes seems like the best we can do. This is like D2171 and D2172, but those fail on Python 2.7.5 (seriously!), so this is my version of the same change. I actually wrote this before reviewing those, then discarded it, then came back to it after finding out 2.7.5 is a silly place. Differential Revision: https://phab.mercurial-scm.org/D2209