shortest: move some safe code out of exception block The RevlogError and WdirUnsupported could be raised by _partialmatch(), but not by the rest of isvalid(), so let's move the rest out to make it clearer. Differential Revision: https://phab.mercurial-scm.org/D3458

revlog: don't say "not found" on internal error If index_node() returned NULL, then index_find_node() and and nt_partialmatch() used to return -2 to signal that the node was not found. However, we were passing in a revnum to index_node() that we knew should exist, so the only reason it could return NULL was due to some internal error or perhaps out of memory. Let's not use "not found" for these cases. I suppose we never noticed this because these error never happen in practice. I think there are more places where we should error out instead of reporting that the node was not found, but the cases mentioned above were all I cared about right now (because using the same error code for all failures simplified some future patches). Differential Revision: https://phab.mercurial-scm.org/D3457

revlog: extract function for getting node from known-to-exist rev Many of the calls to index_node() (which converts a rev to a nodeid) are done with a rev that's know to exist. If the function fails, there's something really wrong and we should just abort. This was done in only one place. This patch starts by extracting that code to a function that we can reuse in later patches. Differential Revision: https://phab.mercurial-scm.org/D3456

shortest: make {shortest("fffffffff")} work again {shortest("fffffffff")} should shorten it to the shortest unambiguous prefix for the working directory. It used to do that until I broke it in 7b2955624777 (scmutil: make shortesthexnodeidprefix() take a full binary nodeid, 2018-04-14), when we started returning the full hex nodeid for any working directory prefix shorter than 40 hex digits. This patch fixes it by catching WdirUnsupported specifically. Differential Revision: https://phab.mercurial-scm.org/D3455

revlog: use radix tree also for matching keys shorter than 4 hex digits I don't know what the reason for the 4-digit limit was, and I can't think of any real disadvantages of using the radix tree also when the requested minimum length is short. This speeds up `hg log -T '{shortest(node,1)}\n'` from 2m16s to 4.5s by making that not fall back to pure code. Differential Revision: https://phab.mercurial-scm.org/D3453

context: convert to hex for error message only for 20-byte changeid Now that 20-byte strings unambiguously mean binary (or a bug), we can specialize the conversion to hex for that case. Differential Revision: https://phab.mercurial-scm.org/D3452

context: clarify that only one attempt is made to interpret changeid We can now tell what type of revision specifier we have just by looking at it (we no longer attempt to interpret it in one way after the other -- that's now in scmutil.revsymbol()). Let's clarify this in the code by swithing to if/elif. Differential Revision: https://phab.mercurial-scm.org/D3451

context: only bother looking for broken dirstate for 20-byte changeid If we fail to look up a changeid in changectx.__init__, we check if it exactly matches any of the dirstate parents, and if it does, we print a more specific message ("working directory has unknown parent '...'!" instead of "unknown revision '...'"). The dirstate parents are always 20 bytes, so there's no need to check for a match when the given changeid is not 20 bytes. (And now that all the other allowed forms of changeid have been moved out of the constructor, there's no risk that a changeid that did match a dirstate parent was actually a valid bookmark.) Differential Revision: https://phab.mercurial-scm.org/D3450

revset: pass in lookup function to matchany() (issue5879) Silly mistake in f83cb91b052e.

test-hgweb: add test for foo-bar name lookup This is broken since f83cb91b052e "revset: pass in lookup function instead of repo (API)."

context: drop support for looking up context by ambiguous changeid (API) This removes support for using the changectx constructor (and thereby repo[x]) for looking up contexts by a stringified int, a namespace key (e.g. a bookmark), or a partial hex nodeid. This means that e.g. repo[<hex nodeid>] will now fail even if a bookmark with the same name exists (which is a good thing IMO). It also means that doing repo[<non-existent node>] no longer ends up loading namespaces (which was a surprising side-effect of creating of failing to create a context object that I recently ran into while debugging something unrelated to this series). Differential Revision: https://phab.mercurial-scm.org/D3449

tests: pass parent revision as integer to repo[x] in test-context-metadata.t Support for looking up by stringified int is going away. It's already deprecated, but I think I didn't notice this case because the test case pipes through `grep`. Differential Revision: https://phab.mercurial-scm.org/D3448

bookmarks: cache reverse mapping (issue5868) I chose a simpler implementation. If the initial cost of building reverse mapping is significant, we'll have to move it under @propertycache. The nodemap could be a dict of sets, but I think keeping a sorted list is better since each node is likely to have zero/one bookmark. Micro-benchmark with 1001 bookmarks and 1001 revisions: $ for n in `seq 0 1000`; do touch $n; hg book book$n; hg ci -qAm$n; done $ hg bookmarks --time > /dev/null (orig) time: real 0.040 secs (user 0.050+0.000 sys 0.000+0.000) (new) time: real 0.040 secs (user 0.040+0.000 sys 0.010+0.000) $ hg log -T '{bookmarks}\n' --time > /dev/null (orig) time: real 0.160 secs (user 0.160+0.000 sys 0.000+0.000) (new) time: real 0.090 secs (user 0.100+0.000 sys 0.000+0.000)

bookmarks: make argument names of _set/_del() more specific

bookmarks: extract function that looks up bookmark names by node

bookmarks: hide dict behind bmstore class This should make it clearer that the bmstore doesn't expose all dict APIs.

tests: remove pid file by default Previously, killdaemons.py would kill PIDs listed in a file then leave the file lingering around. If the PIDs are killed, then there's no point leaving the PID file around. In the worst case, a later invocation of killdaemons.py (run-tests.py invokes killdaemons.py after running a test) could kill a separate process whose PID conflicted with a previously-killed process. By removing the PID file, we eliminate this possibility. Some tests were manually removing the PID file after calling killdaemons.py. So we update these tests to not do this. Differential Revision: https://phab.mercurial-scm.org/D3443

tests: remove #require killdaemons The killdaemons hghave feature has returned True since it was introduced in 448d0c452140. As such, "#require killdaemons" has no effect and is superfluous. So we remove instances of it. Differential Revision: https://phab.mercurial-scm.org/D3442

pycompat: export queue module instead of symbols in module (API) Previously, pycompat and util re-exported individual symbols from the queue module. This had the side-effect of forcing the loading of the queue module whenever pycompat/util was imported. These symbols aren't used very often. So importing the module to get a handle on the symbols is wasteful. This commit changes pycompat so it no longer exports the individual symbols in the queue module. Instead, we make the imported module a "public" symbol. We drop the individual symbol aliases from the util module. All consumers are updated to use pycompat.queue.* instead. This change makes 300 invocations of `hg log -r. -T '{rev}\n'` a little faster: before: 18.44s after: 17.87s Differential Revision: https://phab.mercurial-scm.org/D3441

demandimport: make module ignores a set (API) The list of modules to ignore is used for membership testing. Yet it is defined as a list. Sets are more efficient for membership testing. So this commit converts the module list to a set. Since we took an API hit, I renamed the variable to further clarify the change. This appears to reduce the CPU time for running 300 invocations of `hg log -r. -T '{rev}'` on my i7-6700K: before: 18.64s after: 18.44s Differential Revision: https://phab.mercurial-scm.org/D3440

bundle2: mark the bundle2 part as advisory (issue5872) It blocks old clients to read bundle including this part. Differential Revision: https://phab.mercurial-scm.org/D3481

debugbundle: also display if a part is mandatory or advisory Most parts are mandatory but when introducing new parts, they should be advisory if included by default or old clients won't be able to process it. Differential Revision: https://phab.mercurial-scm.org/D3480

cmdutil: remove unused 'confirmopts' constant Follows up f10cb49951e1.

formatter: ditch namedtuple in favor of attr

log: consume --stat/patch options at constructor of changesetprinter The variable name, self.diffopts, was confusing. Let's split it to two booleans.

log: cache diffopts instance It appears that calling patch.diff*opts() repeatedly has some cost. $ hg log -T '{rev}\n' -R mercurial --time > /dev/null (orig) time: real 4.430 secs (user 4.370+0.000 sys 0.050+0.000) (new) time: real 1.950 secs (user 1.880+0.000 sys 0.060+0.000) 'diffopts or {}' isn't necessary as patch.diff*opts() accepts opts=None.

merge with stable

Added signature for changeset 6614cac550ae

Added tag 4.6 for changeset 6614cac550ae

filelog: don't crash on invalid copy metadata (issue5748) "copy" and "copyrev" are both supposed to appear next to each other. However, a user report demonstrated a crash that indicates that something in the wild is producing "copy" without "copyrev" (probably `hg convert`). While we should definitely fix the source of the bad metadata, the bad code causing the crash is already in the wild and who knows how many repositories are impacted. So let's be more defensive when accessing the file revision metadata.

httppeer: detect redirect to URL without query string (issue5860) 197d10e157ce subtly changed the HTTP peer's handling of HTTP redirects. Before that changeset, we instantiated an HTTP peer instance and performed the capabilities lookup with that instance. The old code had the following relevant properties: 1) The HTTP request layer would automatically follow HTTP redirects. 2) An encountered HTTP redirect would update a peer instance variable pointing to the repo URL. 3) The peer would automagically perform a "capabilities" command request if a caller requested capabilities but capabilities were not yet defined. The first HTTP request issued by a peer is for ?cmd=capabilities. If the server responds with an HTTP redirect to a ?cmd=capabilities URL, the HTTP request layer automatically followed it, retrieved a valid capabilities response, and the peer's base URL was updated automatically so subsequent requests used the proper URL. In other words, things "just worked." In the case where the server redirected to a URL without the ?cmd=capabilities query string, the HTTP request layer would follow the redirect and likely encounter HTML. The peer's base URL would be updated and the unexpected Content-Type would raise a RepoError. We would catch RepoError and immediately call between() (testing the case for pre 0.9.1 servers not supporting the "capabilities" command). e.g. try: inst._fetchcaps() except error.RepoError: inst.between([(nullid, nullid)]) between() would eventually call into _callstream(). And _callstream() made a call to self.capable('httpheader'). capable() would call self.capabilities(), which would see that no capabilities were set (because HTML was returned for that request) and call the "capabilities" command to fetch capabilities. Because the base URL had been updated from the redirect, this 2nd "capabilities" command would succeed and the client would immediately call "between," which would also succeed. The legacy handshake succeeded. Only because "capabilities" was successfully executed as a side effect did the peer recognize that it was talking to a modern server. In other words, this all appeared to work accidentally. After 197d10e157ce, we stopped calling the "capabilities" command on the peer instance. Instead, we made the request via a low-level opener, detected the redirect as part of response handling code, and passed the redirected URL into the constructed peer instance. For cases where the redirected URL included the query string, this "just worked." But for cases where the redirected URL stripped the query string, we threw RepoError and because we removed the "between" handshake fallback, we fell through to the "is a static HTTP repo" check and performed an HTTP request for .hg/requires. While 197d10e157ce was marked as backwards incompatible, the only intended backwards incompatible behavior was not performing the "between" fallback. It was not realized that the "between" command had the side-effect of recovering from an errant redirect that dropped the query string. This commit restores the previous behavior and allows clients to handle a redirect that drops the query string. In the case where the request is redirected and the query string is dropped, we raise a special case of RepoError. We then catch this special exception in the handshake code and perform another "capabilities" request against the redirected URL. If that works, all is well. Otherwise, we fall back to the "is a static HTTP repo" check. The new code is arguably better than before 197d10e157ce, as it is explicit about the expected behavior and we avoid performing a "between" request, saving a server round trip. Differential Revision: https://phab.mercurial-scm.org/D3433

hgweb: prevent triggering dummy href="#" handler Follow up for the previous patch.

paper: add href="#" to links with click handlers This restores the styling that was accidentally removed by the previous change to these files. Differential Revision: https://phab.mercurial-scm.org/D3438

paper: don't register click handlers with inline javascript (issue5812) The use of inline href="javascript:" undermines CSP policies that don't allow inline javascript. This commit changes the registering of the diffstat and line wrapping toggle handlers to the the global DOMContentLoaded handler, thus eliminating all inline javascript from the paper template. Differential Revision: https://phab.mercurial-scm.org/D3437

hgweb: allow Content-Security-Policy header on 304 responses (issue5844) A side-effect of 98baf8dea553 was that the Content-Security-Policy header was set on all HTTP responses by default. This header wasn't in our list of allowed headers for HTTP 304 responses. This would trigger a ProgrammingError when a 304 response was issued via hgwebdir. This commit adds Content-Security-Policy to the allow list of headers for 304 responses so we no longer encounter the error. Differential Revision: https://phab.mercurial-scm.org/D3436

hgweb: discard Content-Type header for 304 responses (issue5844) A side-effect of 98baf8dea553 was that hgwebdir always sets a global default for the Content-Type header. HTTP 304 responses don't allow the Content-Type header. So a side-effect of this change was that HTTP 304 responses served via hgwebdir resulted in a ProgrammingError being raised. This commit teaches our 304 response issuing code to drop the Content-Type header. Differential Revision: https://phab.mercurial-scm.org/D3435

tests: add tests demonstrating ISE for HTTP 304 responses with hgwebdir There are two separate failures here. One for the Content-Type header. Another for the Content-Security-Policy header. Differential Revision: https://phab.mercurial-scm.org/D3434

hgweb: guard against empty Content-Length header Discussion in issue 5860 seems to indicate this can occur. Differential Revision: https://phab.mercurial-scm.org/D3432

test-push-http: do not clear pid file It's okay now, but we'll end up leaking daemon processes if we add some more.

debugcolor: fix crash by empty styles (issue5856)

tests: explicitly define compression engines for tests The zstd compression engine requires C extensions and isn't present in pure Python builds. The compression engine list leaks into the server capabilities string. Unless we're testing functionality specific to a compression format, the set of compression formats supported by a server doesn't matter much. So this commit explicitly defines the server's compression engines for some tests so behavior is consistent between pure and non-pure builds. Differential Revision: https://phab.mercurial-scm.org/D3431

tests: update no-zstd branch of test-treediscovery.t as in 330ada7e8ea5 This side of the test got overlooked. We should probably consider having a way to run some of our tests through a "no-zstd" case just like we run some things through a "no-obsmarkers" case, but that's not an appropriate thing for stable. Differential Revision: https://phab.mercurial-scm.org/D3430

tests: glob away content-length changes relating to missing zstd bindings This doesn't fix everything in these two tests around missing zstd: we still get some changes in the CBOR payload in ways that I think we probably shouldn't bother to glob around. Maybe we should just disable zstd support in some of these lower-level wireproto tests? Differential Revision: https://phab.mercurial-scm.org/D3429

revlog: make pure version of _partialmatch() support 40-byte hex nodeids Without this patch, test-histedit-arguments.t would fail when run with --pure. It turned out to be because the pure version of _partialmatch() does not support full 40-byte hex nodeids. When histedit's instructions include things like "pick tip", it resolves the "tip" revision early to a full nodeid (but plain hex nodeid prefixes are not resolved to full nodeids). Then the nodeid (full or not) is looked up using to a full nodeid later. This step is what fails in pure mode. It has been failing since my c4131138eadb (histedit: look up partial nodeid as partial nodeid, 2018-04-06). I haven't verified, but I suspect histedit instructions like "pick <full hex nodeid>" would have been failing before my commit too, though. The fix is trivial: change a "< 40" to "<= 40". Differential Revision: https://phab.mercurial-scm.org/D3428

hgweb: reuse body file object when hgwebdir calls hgweb (issue5851) An unintended side-effect of f0a851542a05 was that the request body file object (which uses a util.cappedreader) was constructed twice when hgwebdir called into hgweb. Since we attempt to read all remaining data from this file object when Content-Length is defined and since there were two instances of this object and the client supplied no additional data to read, this resulted in deadlock. The fix implemented in this commit is to reuse the request body file object when it is passed from hgwebdir to hgweb. A test demonstrating `hg clone` and `hg push` via hgwebdir has been added. Without this patch, the test hangs when doing `hg clone`. Surprisingly, this must mean that we have effectively no test coverage of the wire protocol when run via hgwebdir. Differential Revision: https://phab.mercurial-scm.org/D3427

remotenames: mark the extension as EXPERIMENTAL I still don't feel confident about locking the behavior of all the things in the remotenames extension. Moreover the extension was introduced in this cycle only. Let's mark this extension EXPERIMENTAL for now so that we can change things especially the storage layer if required in next cycle. I will like to use cbor at storage layer too. Differential Revision: https://phab.mercurial-scm.org/D3426

tests: fix test-check-commit.t when all commits are public I'm 99% sure this is a portable use of /bin/[, and it seems to fix the issue I noticed on the buildbot on my machine.

import: fix crash on --exact check of empty commit (issue5702)

tests: mark test-check-interfaces.py as requiring a repo This was failing our 4.6rc1 build like this: mercurial.error.RepoError: repository /tmp/build-debs.zMTRhC/src-4.6rc1 not found Differential Revision: https://phab.mercurial-scm.org/D3425

sshpeer: reflect actual command activity one handshake The output from devel-peer-request is expected to give data about request and roundtrip done to the server. Changeset a9cffd14aa04 changed some of that by grouping hello and between commands call. However, the old sequence of command was "emulated" in sshpeer. Update the sshpeer to reflect this grouping of commands and update the tests that use it.

tests: drop a useless glob in test-infinite-bundlestore.t With the previous breakage tamed, the lack of test output difference was causing the test runner to report "no result code from test" because of this glob.

infinitepush: ensure fileindex bookmarks use '/' separators (issue5840) After loading up with status messages, I noticed that the subsequent matcher was rejecting 'scratch\mybranch' on Windows. No bookmarks were reported back, and the tests subsequently failed. I did a search for 'match', and nothing else looks like it needs to be fixed up, but someone who understands this code should also take a look. I also tried setting `infinitepush.branchpattern=re:scratch\\.*` in library-infinitepush.sh without this change, but that didn't work. Still, should we ban '\' in these bookmarks to avoid confusion? I thought I saw code that sandwiches a pattern between 're:^' and '.*', so perhaps regex characters will need special care? I also noticed comments in externalbundlestore.{read,write} that it won't work on Windows because of opening an open file. But I don't see a test failure, so this may lack test coverage.

interfaceutil: module to stub out zope.interface The startup time of `hg` increased during the 4.6 development cycle. A cause of that was importing more modules and doing more work at module import time. The import of zope.interface and the declaring of various interfaces is partially responsible for the startup time regression. Our current usage of zope.interface doesn't do much at run time: we are merely declaring interfaces and stating that certain types implement various interfaces. Core Mercurial is not (yet) using of any of zope.interface features that actually require that interface plumbing be defined. The only place we actually need the interface metadata is in test-check-interfaces.py. This commit establishes a new interfaceutil module. It exposes the subset of the zope.interface API that we currently use. By default, the APIs no-op. But if an environment variable is set, we export the real zope.interface APIs. Existing importers of zope.interface have been converted to use the new module. test-check-interfaces.py has been updated to define the environment variable so the real zope.interface is used. The net effect of this change is we stop importing 9 zope.interface.* modules and we no longer perform interface bookkeeping when registering interfaces. On my i7-6700K on Linux, a shell loop that runs `hg log -r .` 300 times on a repo with 1 commit shows a significant CPU time improvement (average of 4 runs): 4.5: 14.814s before: 19.028s after: 16.945s And with `run-tests.py -j10` (single run): 4.5: ~3100s (~51.7m) before: ~4450s (~74.2m) after: ~3980s (~66.3m) So this claws back about half of the regressions in 4.6. Differential Revision: https://phab.mercurial-scm.org/D3419

test-fix: normalize precision of mtime copied by 'cp -p' Appears that MSYS cp only copies mtime in seconds.

merge stable heads

Added signature for changeset 1ec874717d8a

Added tag 4.6rc1 for changeset 1ec874717d8a

internals: correct capitalization of 'compression' stream level parameter

test-check-code: prevent from adding Python modules shadowed by ancient C

diffhelper: rename module to avoid conflicts with ancient C module (issue5846) Historically we had had C extensions in mercurial/, which shadows the pure Python modules of the same name forever unless we do clean build/install. I'm sloppy to think about new name, so just dropped the "s".

diffhelpers: backport 9e40bc4c1bde from C implementation 9e40bc4c1bde just says "harden testhunk." I don't think this would be the case, but it makes some sense to avoid negative index.

version: make parser more robust for rc variants and ill-formed strings

phabricator: specify some metadata compatibly with arc Differential Revision: https://phab.mercurial-scm.org/D3414

diff: restore original color scheme for worddiff I'm not young. "red dim" is nearly invisible, and "red" vs "red dim" is too subtle to see difference. And, "underline" is necessary to highlight whitespace changes.

test-fix: fix use of 'f --newer' to check that foo.whole is not updated Here we want to assert not 'mtime(foo.whole.orig) < mtime(foo.whole)'. The condition has to be inverted since 'f --newer' says "newer" if the mtimes match. Alternatively, we could insert 'sleep 2' before 'cp' to ensure that foo.whole.orig is newer than foo.whole, but a fewer sleeps should be better.

context: translate FilteredIndex/LookupError at repo[changeid] (API) This partially backs out ecd3f6909184. It seems layering violation for repo[changeid] to raise storage-level exceptions transparently. Otherwise, we would have to rewrite callers to catch all of them. try: repo[rev_or_node] except (error.RepoLookupError, error.FilteredIndexError, error.FilteredLookupError): pass This would also fix filectx._changectx(), which catches FilteredRepoLookupError to fall back to the unfiltered path.

inifinitepush: fix filebundlestore to close file

notify: only notify for non-filtered revision This should fix issue5821 for hook targeting individual revision.

notify: access the initial revision on an unfiltered repository (issue5821) This should avoid crash when the first revision pushed end up being hidden.

notify: use changelog API to iterate over revision number (issue5821) This will avoid iterating over filtered revision.

infinitepush: open files in binary mode This fixes the scary looking abort in test-infinitepush-ci.t when unbundling: --- tests/test-infinitepush-ci.t +++ tests/test-infinitepush-ci.t.err @@ -84,15 +84,12 @@ $ hg unbundle .hg/scratchbranches/filebundlestore/a4/c2/a4c202c147a9c4bb91bbadb56321fc5f3950f7f2 adding changesets - adding manifests - adding file changes - added 1 changesets with 1 changes to 1 files - new changesets 6cb0989601f1 - (run 'hg update' to get a working copy) - - $ hg glog - o 1:6cb0989601f1 added a - | public + transaction abort! + rollback completed + abort: stream ended unexpectedly (got 68 bytes, expected 218759168) + [255] + + $ hg glog @ 0:67145f466344 initialcommit public This was found by grepping for '"r', "'r", '"w' and "'w" after manually creating a bundle from the same revision, diffing against the corrupt one, and seeing CRs sprinkled around. Sadly, the missing bookmarks are still a problem in the two remaining test failures.

tests: stabilize test-fix.t for Windows

Added signature for changeset ed5448edcbfa

Added tag 4.6rc0 for changeset ed5448edcbfa

merge with default to begin 4.6 freeze # no-check-commit because of many vendored packages

rebase: introduce support for automatically rebasing orphan changes _destautorebase(SRC) is based on the _destrestack(SRC) revset from fbamend. The supporting _possibledestination function is extracted from evolve, with minor cleanups. We've considered some alternatives here: * This change, but with --auto as the flag name. We're hedging our bets on this a little in this change so that if this ends up being the wrong direction we haven't burned the valauble --auto name on rebase. * --destination auto: I've got reservations about the discoverability of this, and we don't currently have a good story for a revset alias of sorts that changes behavior depending on the context in which it's used. * A "rebase presets" feature, where we could use the currently-an-error positional argument space for the rebase command to define presets, so that users could define a 'linearize' preset that specifies --revision='orphan()-obsolete()' and --dest=_destautoorphanrebase(SRC). Personally, I find the third option somewhat appealing, but am hesitant to "spend" the functionality space of positional arguments to the rebase command. We should revisit the way we expose this functionality sometime in the 4.7 cycle once we've had a chance to vet the implementation of the functionality. Differential Revision: https://phab.mercurial-scm.org/D2668

py3: whitelist six new passing tests Differential Revision: https://phab.mercurial-scm.org/D3406

wireproto: rename wireproto to wireprotov1server (API) We have wireprotov2server, wireprotov1peer, and wireprotov2peer. wireproto only contains server functionality. So it makes sense to rename it to wireprotov1server so the naming aligns with everything else. Differential Revision: https://phab.mercurial-scm.org/D3400

wireproto: move version 2 commands dict to wireprotov2server This was the final piece of version 2 referenced in wireproto. The break between server implementations is now much cleaner. Differential Revision: https://phab.mercurial-scm.org/D3399

wireproto: move supportedcompengines out of wireproto This function is used by both version 1 and version 2. It belongs in a common module. "wireprototypes" may not be the best module name. I may rename it... Differential Revision: https://phab.mercurial-scm.org/D3398

wireproto: reimplement dispatch() for version 2 server The code is minimal. I'm trying to create a cleaner break between version 1 and version 2 server code. Differential Revision: https://phab.mercurial-scm.org/D3397

wireproto: move command registration types to wireprototypes These are shared across wire protocol implementations. wireprototypes is our module for common code. Differential Revision: https://phab.mercurial-scm.org/D3396

wireproto: make version 2 @wireprotocommand an independent function Previously, the code for this decorator was shared between version 1 and version 2 commands. Very few parts of the function were identical. So I don't think sharing is justified. wireprotov2server now has its own @wireprotocommand decorator function. Because the decorator is no longer shared, code for configuring the transport policy has been removed. i.e. commands must have separate implementations for each wire protocol version. Differential Revision: https://phab.mercurial-scm.org/D3395

wireproto: don't pass transportpolicy argument The default is version 1 only. So we don't need to pass this argument when declaring commands. Differential Revision: https://phab.mercurial-scm.org/D3394

forget: rename --confirm to --interactive Differential Revision: https://phab.mercurial-scm.org/D3405

notify: add maxdiffstat option to truncate long file lists Large scale changes like a new GCC version can easily result in 1MB+ emails due to diffstat alone. The new maxdiffstat option truncates the list similar to what maxdiff already provides for the diffs. Differential Revision: https://phab.mercurial-scm.org/D3402

log: fix crash on empty revision with --copies switch If a revset is empty, .max() raises ValueError. I don't see any reason to recompute the revs, so I made it reuse the one returned by logcmdutil.getrevs(). If no revs specified by command line, the endrev will be smartset.spanset(repo) + 1, which is basically the same as len(repo), the default of getrenamedfn(). If --follow specified, revs.max() points to the working parent, which seems more correct.

revsetlang: do not pass in non-bytes to parse() Since parse() isn't a simple function, we shouldn't expect it would raise TypeError or ValueError for invalid inputs. Before, TypeError was raised at 'if pos != len(spec)', which was quite late to report an error. This patch also makes tokenize() detect invalid object before converting it to a py3-safe bytes. Spotted while adding the 'revset(...)' hack to _parsewith().

fix: use templater to substitute values in command string bytes.format() isn't supported on Python 3. Luckily, our template syntax is similar so we can reuse it. We need a hack to disable \-escapes as '\' is a directory separator on Windows.

logcmdutil: pass formatter to jsonchangeset as argument And rename the class. Now we can reuse it for 'log -Tcbor' (and '-Tpickle' if we want.)

logcmdutil: rewrite jsonchangeset printer to be backed by jsonformatter This is a bit slower than the original implementation, but I don't think that would actually matter. It's still faster than full templating. $ hg log -Tjson -r0:5000 --time > /dev/null (orig) time: real 1.550 secs (user 1.500+0.000 sys 0.040+0.000) (new) time: real 1.810 secs (user 1.740+0.000 sys 0.070+0.000) cf. $ hg log -Tdefault -r0:5000 --time > /dev/null time: real 4.980 secs (user 4.850+0.000 sys 0.130+0.000) $ hg log -r0:5000 --time > /dev/null time: real 2.340 secs (user 2.220+0.000 sys 0.100+0.000) $ hg log -r0:5000 -q --time > /dev/null time: real 0.750 secs (user 0.670+0.000 sys 0.070+0.000) The test output changes because keys are sorted alphabetically.

logcmdutil: remove unused attribute 'cache' from jsonchangeset printer Perhaps it's a copy-pasta of changeset_templater at bd15932846a4.

formatter: convert timestamp to int Spotted while porting jsonchangeset printer to formatter. A timestamp may be float type, but its meaningful part is just int.

bundle: introduce per-engine compression level If experimental.bundlecomplevel.$engine is set, prefer it over the generic experimental.bundlecomplevel. Given that compression levels have widely different meanings across engines, this allows much saner configuration. Differential Revision: https://phab.mercurial-scm.org/D3393

bundlespec: drop externalnames flag Always provide the human readable version of compression and version. Add the translated wire format name in the new wirecompression and wireversion fields. Differential Revision: https://phab.mercurial-scm.org/D3392

revlog: make shortest() take a full binary nodeid (API) Follow-up to 7b2955624777 (scmutil: make shortesthexnodeidprefix() take a full binary nodeid, 2018-04-14). Differential Revision: https://phab.mercurial-scm.org/D3403

lfs: add the 'Authorization' property to the Batch API response, if present The client copies all of these properties under 'header' to the HTTP Headers of the subsequent GET or PUT request that it performs. That allows the Basic HTTP authentication used to authorize the Batch API request to also authorize the upload/download action. There's likely further work to do here. There's an 'authenticated' boolean key in the Batch API response that can be set, and there is an 'LFS-Authenticate' header that is used instead of 'WWW-Authenticate'[1]. (We likely need to support both, since some hosting solutions are likely to only respond with the latter.) In any event, this works with SCM Manager, so there is real world benefit. I'm limiting the headers returned to 'Basic', because that's all the lfs spec calls out. In practice, I've seen gitbucket emit custom header content[2]. [1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md#response-errors [2] https://github.com/gitbucket/gitbucket/blob/35655f33c7713f08515ed640ece0948acd6d6168/src/main/scala/gitbucket/core/servlet/GitRepositoryServlet.scala#L119

lfs: enable the final download count status message At this point, I think all of the core commands are prefetching, except grep and verify. Verify will need some special handling, in case the revlogs are corrupt. Grep has an issue that still needs to be debugged, but we probably need to give the behavior some thought too- it would be a shame to have to download everything in order to search. I think the benefit of having this info for all commands outweighs extra printing in a command that is arguably not well behaved in this context anyway.

diff: invoke the file prefetch hook By invoking it this deep within the command, we pick up both subrepo and hgweb support, as well as --patch support for commands that implement logopts.

export: invoke the file prefetch hook cmdutil.exportfile() is only called by shelve, mq and patchbomb. Those are unlikely to mix with lfs, but it may as well be invoked there for completeness.

scmutil: teach the file prefetch hook to handle multiple commits The remainder of the commands that need prefetch deal with multiple revisions. I initially coded this as a separate hook, but then it needed a list of files to handle `diff` and `grep`, so it didn't seem worth keeping them separate. Not every matcher will emit bad file messages (some are built from a list of files that are known to exist). But it seems better to filter this in one place than to push this on either each caller or each hook implementation.

tests: arrange for a server in wireproto-command-capabilities.t to be killed The stray servers were piling up after the test harness exited. On Windows, this means the *.pyd files can't be rebuilt, which is why the build warning count dropped to 1 recently.

revset: skip legacy lookup for revspec wrapped in 'revset(...)' Currently, multiple labels can take forms that can be confused with revset (eg: "rev(0)" is a valid tag). Since we look up for tags before evaluating revset, this means a tag can shadow a valid revset at any time. We now enforce the strict revset parsing when wrapped with 'revset(...)'. For now, This only work on a whole revspec (but can be used within the revset without effect). This might change in the future if we improve the implementation. The feature is undocumented for now, keeping it in the experimental namespace. In case a better approach to achieve the same goal is found. The syntax looks like a revset but is not implemented as such for now. Since the goal is to avoid some preprocessing that happens before revset parsing, we cannot simply implement it as a revset predicate. There was other approaches discussed over the mailing-list but they were less convincing. Having a configuration flag to disable legacy lookup have been considered but discarded. There are too many common uses of ambiguous identifier (eg: '+', '-' or '..') to have the legacy lookup mechanism turned off. In addition, the approach can control the parsing of each revset, making it more flexible. For example, a revset used as the value of an existing configuration option (eg: pushrev) could enforce its resolution as a revset (by using the prefix) while user inputs would still use the legacy lookup. In addition of offering a way to unambiguously input a revset, this prefix allow skipping the name lookup providing a significant speedup in some case.

revset: add more test to show current behaviors with label looking like revset There are very few constraints on what character can be put into tags and other labels. We add more tests showing some of extreme cases that user can currently use.

commands: use command executor interface Differential Revision: https://phab.mercurial-scm.org/D3391

exchange: use command executor interface for calling listkeys So the requests are compatible with version 2 peers. Differential Revision: https://phab.mercurial-scm.org/D3390

forget: add --confirm option Also added confirmopts in cmdutil.py Differential Revision: https://phab.mercurial-scm.org/D2934

largefiles: opts appears to already be bytes in this instance test-largefiles.t now passes. Differential Revision: https://phab.mercurial-scm.org/D3364

tests: port inline extensions in test-hook.t to py3 This test *almost* passes now, but some import errors print very differently in ways that seem at least somewhat important. Differential Revision: https://phab.mercurial-scm.org/D3363

tests: manually print list in test-hook.t Changes the output a bit, but not in an important way. Differential Revision: https://phab.mercurial-scm.org/D3362

hook: also use pprint on lists for stable output on py2/3 Differential Revision: https://phab.mercurial-scm.org/D3361

hook: use stringutil.pprint instead of reinventing it Differential Revision: https://phab.mercurial-scm.org/D3360

stringutil: make b prefixes on string output optional I need this to preserve some behavior in hook.py. Differential Revision: https://phab.mercurial-scm.org/D3359

stringutil: teach pprint how to format None Differential Revision: https://phab.mercurial-scm.org/D3358

tests: update inline extensions in test-bundle2-exchange.t to py3 The test doesn't pass for what superficially look like good reasons. We'll need to come back to it later. # skip-blame because it's b prefixes and a couple of b'N' instead of str(N) Differential Revision: https://phab.mercurial-scm.org/D3357

wsgicgi: un-do some prior porting work that is now wrong The Python 3 WSGI behavior is that the environ dict should be full of unicodes. We previously tried Too Hard here, so we unwind that bit of porting. Also add some bytesurl() encodes on status and headers. test-clone-cgi.t now passes. Differential Revision: https://phab.mercurial-scm.org/D3356

hgweb_mod: inform hgweb class about paths actually being bytes Differential Revision: https://phab.mercurial-scm.org/D3355

hgweb: inform hgweb.hgweb() entrypoint that paths should be bytes Differential Revision: https://phab.mercurial-scm.org/D3354

tests: port inline cgi script in test-largefiles.t to python 3 Differential Revision: https://phab.mercurial-scm.org/D3353

lfcommands: use %d on known-int in format string Differential Revision: https://phab.mercurial-scm.org/D3352