py3: explicitly convert result of dict.items() into list Differential Revision: https://phab.mercurial-scm.org/D2320

py3: use pycompat.byteskwargs() to convert opts keys to bytes Differential Revision: https://phab.mercurial-scm.org/D2319

wireprotoserver: py3 helpfully calls adds HTTP_ to CONTENT_LENGTH Just handle both with a membership check, preferring the HTTP_ namespaced version. Sigh. Differential Revision: https://phab.mercurial-scm.org/D2310

tests: port inline Python in test-http-branchmap.t to Python 3 Differential Revision: https://phab.mercurial-scm.org/D2309

hgweb_mod: partially undo 8afc25e7effc to fix py3 We now bytes-ify the entire request, so this is wrong. Differential Revision: https://phab.mercurial-scm.org/D2308

tags: don't feed both int and None to min() They're not comparable on Python 3, and we can just map None to 0. Differential Revision: https://phab.mercurial-scm.org/D2307

tests: fix get-with-headers.py on python3 when writing to stdout Differential Revision: https://phab.mercurial-scm.org/D2306

webcommands: use pycompat.strkwargs as needed Differential Revision: https://phab.mercurial-scm.org/D2305

webcommands: unpack contents of length-1 dict portably ((k,v),) = dict.items() was suggested by some friends, but I feel like that's maybe too clever? Differential Revision: https://phab.mercurial-scm.org/D2304

bundle2: use pycompat.strkwargs as needed Differential Revision: https://phab.mercurial-scm.org/D2303

tests: add b prefixes to test-rebase-dest.t # skip-blame just b prefixes Differential Revision: https://phab.mercurial-scm.org/D2302

rebase: sort roots by revision Previously we were sorting them by whatever implicit sort order we got between changectx instances. Sorting by rev at least makes some sense, so I chose that. Differential Revision: https://phab.mercurial-scm.org/D2301

httppeer: use %d to format int Differential Revision: https://phab.mercurial-scm.org/D2300

extensions: reject any unicode strings in tables before loading This allows us to test hg on Python 3 without disabling third-party extensions which could pollute cmdtable for example.

py3: make test-extdiff.t pass on Python 3

py3: make regexp patterns bytes in extdiff

py3: fix bytes-unicode dance while building docstring of extdiff

py3: factor out byterepr() which returns an asciified value on py3

remotenames: don't use the default value of logfmt for namespaces logfmt is the format which is used to format the log output for that namespace. This patch passes "remote {bookmark|branch}: %s" as the logfmt. Space is not added after bookmark and branch to make output consistent with other details. Still this is not the best output. We may need to wrap getlogcolumns() to change spacing in the in built columns to match the remotenames one. lognames are also deleted as they are superseded by logfmt. Differential Revision: https://phab.mercurial-scm.org/D2277

py3: whitelist test-pull-http.t There's a lot of work left to do on network-related stuff, but I at least got one more test passing. Differential Revision: https://phab.mercurial-scm.org/D2296

scmutil: bytes-ify IOErrors before wrapping them in abort message Differential Revision: https://phab.mercurial-scm.org/D2292

py3: port f to Python 3 This involved a lot of b'' literals, conversion from %s to %d, and using a hashing mechanism that returns bytes instead of str. Differential Revision: https://phab.mercurial-scm.org/D2299

py3: replace file() with open() in test-convert-git.t file() is not present in Python 3. It also makes sure we write bytes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D2133

bundle: updates the help text for hg bundle (issue5744) Differential Revision: https://phab.mercurial-scm.org/D2278

py3: get bytes-repr of network errors portably This resolves a lot of weird issues in Python 3 around error strings. Differential Revision: https://phab.mercurial-scm.org/D2295

hgweb: open server logs in binary mode This is consistent with when we're logging to stdout, so we don't have to do something annoyingly complicated in the logging infrastructure. Differential Revision: https://phab.mercurial-scm.org/D2294

tests: add some b prefixes in test-http-bundle1.t # skip-blame just some b prefixes Differential Revision: https://phab.mercurial-scm.org/D2293

hgweb: correctly bytes-ify status, not string-ify Differential Revision: https://phab.mercurial-scm.org/D2291

printenv: port to python3 Differential Revision: https://phab.mercurial-scm.org/D2290

py3: whitelist another eight passing tests Differential Revision: https://phab.mercurial-scm.org/D2281

remotenames: port partway to python3 by using collections.MutableMapping test-logexchange.t doesn't pass after this, but at least the remotenames extension can be imported. Differential Revision: https://phab.mercurial-scm.org/D2280

tests: avoid referring to pvec in demandimport test Nothing else currently uses pvec, so we may want to delete it (but Augie Fackler says he may want to use it). To enable deletion, this patch replaces it by the error module in the demandimport test (any module works). However, since the error module had already been loaded at this point in the test (via the util module), I moved it earlier in the test so it's still not loaded (although I'm not sure if that's even relevant to the test). Differential Revision: https://phab.mercurial-scm.org/D2287

help: use cmdutil.parsealiases() to resolve command name This seems slightly better than parsing '^command|name' string by using an ad-hoc pattern.

help: list deprecated and experimental extensions if --verbose This is common in the other help topics.

hgweb: translate Abort in help command to 404 error c3784e3c3e8d changed the exception type to be raised if unknowncmd=False.

templatekw: add {reporoot} keyword The "%b" in filename format string will be replaced with "{reporoot|basename}".

templatefilters: add dirname() filter This is different from stripdir() in that the last component is always removed.

templater: add function to help substituting patterns in template string This will be used to rewrite a filename pattern to a template string.

templater: extract function scanning template string This provides a tokenizer-level view of template fragments, and will be used to substitute patterns in outermost 'string' tokens.

tests: expand test coverage for updating phases Consolidating the tests demonstrated that there are behavior differences when pushing phases between bundle1 and bundle2. A reason for this is the behavior of legacy pushes where the client queries the state of phases and then conditionally updates phases after an "unbundle" is processed. This behavior is expected. The tests were incomplete because they only tested the case of a publishing repo. In this commit, we add a variant for a non-publishing repo. We still see some differences between the legacy and bundle2 exchanges. But they are less pronounced. The behavior of not firing a pushkey hook when phases are updated as part of changegroup application feels weird to me. I'm not sure if this is a feature or a bug. By the time the "pushkey" or "phases" bundle2 part is applied, the phases have already been moved on a publishing repository. We fire the "pushkey" hook regardless, even though it would be a no-op. This is the part that feels the most buggy.

tests: consolidate test-push-http.t and test-push-http-bundle1.t These tests were initially copies of each other. Now that we have #testcases support in .t tests, we can consolidate them. The changes to test-push-http.t reflect the differences between that file and test-push-http-bundle1.t. The variances in phases push behavior are the biggest differences. The test will be updated in a subsequent commit to make the differences more clear and to expand test coverage. For now, let's just port the differences verbatim to get the tests consolidated.

tests: port value-less unbundle capability test to test-push-http.t This test is present in test-push-http-bundle1.t. Let's add it to test-push-http.t to further unify the tests.

tests: add phase testing to test-push-http-bundle1.t test-push-http.t and test-push-http-bundle1.t were initially copies. Now that we have support for inline test variants, we can combine them. One of the variances between the tests is testing of phase moving. We add the missing code to test-push-http-bundle1.t.

cmdutil: narrow scope of KeyError in makefilename() It seemed too broad to catch a standard Python exception.

node: make bin() be a wrapper instead of just an alias This includes a full backout of 59affe7e and 30d0cb27. Per the review of the former, we'd rather adapt the API to behave like it used to (at least for now), and take a second run at it if it shows up in our performance numbers. I ran perfrevlogindex with and without this change and it didn't make a measurable difference, so maybe it's fine (despite my intuition to the contrary). Differential Revision: https://phab.mercurial-scm.org/D2279

tests: actually check that HGDEMANDIMPORT=disable disables demandimport At the point in the test where we were checking that the 'node' got eagerly loaded, it had already been loaded (via the pvec module), so our check wasn't doing anything (i.e. the test would pass even if you removed the line that set HGDEMANDIMPORT=disable). Let's move this test earlier so it tests what it was meant to test. Differential Revision: https://phab.mercurial-scm.org/D2286

debugcommands: print out the editor that was searched for (post shlexsplit) A user was in #mercurial and had the following settings in their hgrc: [ui] editor = C:\home\npp\notepad++.exe -multiInst -nosession After shlexsplit, the first argument was mangled into C:homenppnotepad++.exe, which was quite unlikely to exist. It took many back-and-forths to identify that adding " characters around the exe would fix the issue; we were thinking that it's because something was incorrectly *not* splitting and adding/moving the " characters fixed the split boundaries, but when testing afterward it appears that it's just mangled. I considered adding an informational if pycompat.iswindows and \ in the string about this issue, but was worried that might have too many false positives and did not do so at this time. Differential Revision: https://phab.mercurial-scm.org/D1808

merge with stable

date: fix parsing months Thanks nemo for discovering this on #mercurial IRC channel. Test Plan: Add a test. It fails before this patch: ``` + hg: parse error: invalid date: 'Feb 2018' + hg: parse error: invalid date: 'Apr 2018' + hg: parse error: invalid date: 'Jun 2018' + hg: parse error: invalid date: 'Sep 2018' + hg: parse error: invalid date: 'Nov 2018' ``` Differential Revision: https://phab.mercurial-scm.org/D2289

hgweb: show dates recorded in obsolescence markers Dates of operations that obsolete commits are also important enough to be shown, but maybe not as important as original commit dates, that and also for brevity is why they are wrapped in a <span> with only "age" class. Provided that JS is enabled, such elements only show age (e.g. "3 months ago"), and the exact date is visible in a tooltip. Commits dates, on the other hand, show both date and age (the latter in parenthesis). Example result for f1a0933ce59e from hg-committed: obsolete: pruned by Yuya Nishihara <yuya@tcha.org> 13 days ago

gitweb: make span.age CSS selector more specific This set of CSS properties is only used on /changelog page to make age elements float to the left and be fixed in size, but span.a is too broad of a selector, because we don't want to apply these properties to all <span> elements that have age class.

wireprotoserver: return to using iscmd() method This was teased out in part so remotefilelog could disable an old protocol method over http. It got dropped accidentally in the recent refactor, but the code was all still present so it's easy to support for now. I think once we land remotefilelog, we should probably inline this function. Differential Revision: https://phab.mercurial-scm.org/D2285

githelp: cast commands to bytes This is more compatible with Python 3. Differential Revision: https://phab.mercurial-scm.org/D2165

py3: use util.forcebytestr instead of str for converting errors to bytes Differential Revision: https://phab.mercurial-scm.org/D2269

py3: use pycompat.bytestr to convert str to bytes Differential Revision: https://phab.mercurial-scm.org/D2261

mpatch: allow clang-format oversight Differential Revision: https://phab.mercurial-scm.org/D2183

base85: allow clang-format oversight Differential Revision: https://phab.mercurial-scm.org/D2182

charencode: allow clang-format oversight Nice and easy. Differential Revision: https://phab.mercurial-scm.org/D2181

formatting: enforce system headers before local headers Differential Revision: https://phab.mercurial-scm.org/D2275

wireprotoserver: add version to HTTP protocol name (API) This matches what we did for the SSH protocol handler in ac33dc94e1d5. .. api:: HTTP protocol handlers now advertises its internal name as ``http-v1`` instead of ``http``. Differential Revision: https://phab.mercurial-scm.org/D2219

wireprotoserver: rename webproto to httpv1protocolhandler This matches our naming convention for the SSH server's protocol handler. Differential Revision: https://phab.mercurial-scm.org/D2218

wireproto: improve docstring for "hello" Differential Revision: https://phab.mercurial-scm.org/D2217

httppeer: remove httpspeer All it did was verify at construction time that Mercurial supports TLS. instance() is what's used to construct peer instances. So we can just inline this check into that function and do away with the type variant. Differential Revision: https://phab.mercurial-scm.org/D2216

httppeer: remove redundant code to fetch capabilities _fetchcaps() is called by httppeer.instance(), which is the only instantiator of httppeer. Since _fetchcaps() always sets self._caps and since 197d10e157ce removed the fallback for cases where the remote doesn't support capabilities, we can remove some dead code from httppeer.capabilities(). Differential Revision: https://phab.mercurial-scm.org/D2215

httppeer: change logic around argument handling The code to process arguments only makes sense if there are arguments. So change an "else" to "elif args", remove an "if" that isn't necessary, and add some docs for good measure. Differential Revision: https://phab.mercurial-scm.org/D2214

tests: test using both versions of SSH protocol Now that the version 2 of the SSH protocol is usable in core, we can start actively testing it more widely outside of low-level protocol tests. We add #testcases variants to a handful of tests so we exercise both version 1 and version 2 of the SSH protocol when testing. This will allow us to more easily find regressions and variances as protocol 2 is developed. It will also make it easier to continue testing with protocol version 1 once version 2 is enabled by default. There are a handful of tests using ssh:// that should also gain test variances. One - test-push-race.t - already has a #testcases. This would require combinatorial cases. I didn't want to go down that rabbit hole, so that test is unchanged. Thinking aloud, there is probably an opportunity to automatically run tests with multiple server/protocol implementations. Ideally any test that performed server interaction would run with all supported server implementations and protocols so we could find variances between servers and protocols. But this has been a long-standing issue with our test harness. I don't think it is an easily solved problem. But it would be nice... Differential Revision: https://phab.mercurial-scm.org/D2206

sshpeer: log remote capabilities after protocol upgrade This helps reduce variance with version 1 and will help prevent test output divergence as we start testing protocol version 2 more widely. Differential Revision: https://phab.mercurial-scm.org/D2205

wireprotoserver: handle SSH protocol version 2 upgrade requests This commit teaches the SSH server to recognize the "upgrade" request line that clients send when they wish to switch the channel to version 2 of the SSH protocol. Servers don't honor upgrade requests unless an experimental config option is set. Since the built-in server now supports upgrade requests, our test server to test the handshake has been deleted. Existing tests use the built-in server and their output doesn't change. The upgrade is handled in our state machine. The end result is a bit wonky, as the server transitions back to version 1 state immediately after upgrading. But this will change as soon as version 2 has an actual protocol that differs from version 1. Tests demonstrating that the new server is a bit more strict about the upgrade handshake have been added. Differential Revision: https://phab.mercurial-scm.org/D2204

wireprotoserver: move SSH server operation to a standalone function The server-side processing logic will soon get a bit more complicated in order to handle protocol switches. We will use a state machine to help make the transitions clearer. To prepare for this, we move SSH server operation into a standalone function. We structure it as a very simple state machine. It only has two states for now, with one state containing the bulk of the logic. But things will evolve shortly. Differential Revision: https://phab.mercurial-scm.org/D2203

py3: stringify integer with %d instead of bytes() The unbundle wire protocol command now returns a properly formatted reply in Python 3. Differential Revision: https://phab.mercurial-scm.org/D2274

py3: add b'' to test-sshserver.py # skip-blame because adding b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2273

py3: add b'' to config options in test extension # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D2272

manifest: add support for including directories outside narrowspec When using tree manifests and the client doesn't have a directory, we have two choices for what to do with "hg manifest" output: 1) ignore the directory, and 2) include the directory (not files within it). For "hg files", we decided to ignore the directories (and files) outside the narrowspec. If we choose to not include directories outside the narrowspec, then I think we should also make sure we don't include files outside the narrowspec. I also think we should add --outside-narrow flag (or other name). Thus, whichever way we go, I think we should have a way of displaying paths (files or directories) outside the narrowspec. For that we'll need to handle the 't' flag that narrowhg uses, and that's what this patch adds support for. Differential Revision: https://phab.mercurial-scm.org/D2235

narrow: restrict manifest iteration by using manifest.walk(matcher) This is only for root nodes, so it shouldn't really matter (they're rarely huge), but seems cleaner. Differential Revision: https://phab.mercurial-scm.org/D2234

narrow: only diff manifest part within narrowspec when generating changegroup Since 959ebff3505a (manifest: add match argument to diff and filesnotin, 2017-03-07), we have a more efficient way of diffing manifests while applying a matcher. Let's use that while generating narrowed changegroups, so we avoid diffing parts of the manifest that don't match the narrowspec. Differential Revision: https://phab.mercurial-scm.org/D2233

cmdutil: build "%m" (desc|firstline) in makefilename()

cmdutil: rewrite makefilename() to use ctx methods

cmdutil: pass ctx to makefileobj() in place of repo/node pair (API)

cmdutil: pass ctx to makefilename() in place of repo/node pair (API)

cmdutil: make node parameter of makefileobj() mandatory (API) (repo, node) pair will be replaced with ctx, so makefilename() can be easily ported to templater.

progress: use '%*d' to pad progress value Follows up 7f5108e58083. The problem of '% Nd' is that ' ' means we want {' ' or '-'} as a sign character. We should instead write '%Nd' to pad up to N characters with space, and N can be '*'.

py3: stringify IOError/OSError without loosing local character Follows up fa4d333cac58. An environment error may contain non-ascii characters on Windows, which should be encoded to a platform-native string.

dirstate: drop explicit files that shouldn't match (BC) (issue4679) Before, wctx.walk() could include files excluded by -X pattern, which disagrees with wctx.matches() and ctx.walk()/matches() behavior. This patch fixes the problem by testing stat results against the matcher if the matcher may contain false paths. I have no idea if the fix should be made before the workaround for case- insensitive filesystems, but that shouldn't matter since match.anypats() means 'not match.isexact()'. This patch also makes narrow and sparse extensions to not exclude explicit paths on walk() because they appear to depend on the buggy behavior. More detailed analysis about this issue by Martin von Zweigbergk: "I think it's just an unintended consequence of how the dirstate walk works, but I'm not sure. The exception for explicit files also bothered me when I was working on the matcher code a year or so ago. I actually added the exception to the matcher code because I thought it was always working like that (not just for dirstate) in a83a7d27911e (match: handle excludes using new differencematcher, 2017-05-16). It was only recently that Yuya realized that it used to be inconsistent and that I probably made it consistently bad because I didn't realize it was inconsistent to start with, see 821d8a5ab4ff (match: do not weirdly include explicit files excluded by -X option, 2018-01-16)." .. bc:: Working-directory commands now respect ``-X PATTERN`` no matter if PATTERN matches explicitly-specified FILEs. For example, ``hg add foo -X foo`` no longer add the file ``foo``.

log: add TODO comments about --line-range processing

log: factor out function to feed revisions to displayer

graphlog: unblock --line-range option It should work now.

graphlog: deduplicate preprocessing of log command

hgweb: show users recorded in obsolescence markers It's useful to see who obsoleted a commit, because it's not always done by its author (hg-committed is a good example, because people rebase stacks of commits made by various people). Usernames are obfuscated, but look correct (e.g. "&#116;&#101;&#115;&#116;" is "test").

revert: use an exact matcher in interactive diff selection (issue5789) When going through _performrevert() in the interactive case, we build a matcher with files to revert and pass it patch.diff() for later selection of diff hunks to revert. The files set used to build the matcher comes from dirstate and accounts for patterns explicitly passed to revert ('hg revert -i <file>') and, in case of nonexistent pattern, this set is empty (which is expected). Unfortunately, the matcher built from scmutil.match(ctx, []) is wrong as it leads patch.diff() to rebuild a 'changes' tuple with dirstate information, ignoring user-specified pattern. This leads to the situation described in issue5789, where one gets prompted about reverting files unrelated to specified patterns because they made a typo or so. We fix this by building an exact matcher with the correct set of file paths (built earlier). Thanks to Yuya Nishihara for suggesting the correct fix.

tests: add a test demonstrate that 'revert -i' ignores nonexistent patterns As described in issue5789, when revert is called through: hg revert -i <some file that does not exist> we'd expect the command to abort early. Currently, it just warns about missing file but prompt about files unrelated to user arguments.

py3: slice over bytes to prevent getting ascii values Differential Revision: https://phab.mercurial-scm.org/D2271

py3: converts bytes to pycompat.bytestr to get bytechrs while enumerating Differential Revision: https://phab.mercurial-scm.org/D2270

py3: use pycompat.{bytes|str}kwargs in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2268

py3: replace file() with open() in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2267

py3: use "%d" for converting int to bytes in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2266

py3: open files in bytes mode in transplant.py Differential Revision: https://phab.mercurial-scm.org/D2265