show: tweak plain abort language for clarity

tests: print Unix style paths in *.py tests These tests don't support (glob). I didn't audit all tests, but these ones were failing.

revlog: make "size" diverge from "rawsize" Previously, revlog.size equals to revlog.rawsize. However, the flag processor framework could make a difference - "size" could mean the length of len(revision(raw=False)), while "rawsize" means len(revision(raw=True)). This patch makes it so. This corrects "hg status" output when flag processor is involved. The call stack looks like: basectx.status -> workingctx._buildstatus -> workingctx._dirstatestatus -> workingctx._checklookup -> filectx.cmp -> filelog.cmp -> filelog.size -> revlog.size

test-flagprocessor: add a case about hg status This shows how "hg status" is wrong - nothing changed but the file is labeled as "M".

test-http-proxy: add the Windows variant of "connection refused" The full error is "No connection could be made because the target machine actively refused it".

test-fileset: eliminate a duplicate test that was conditionalized for output

test-fileset: glob away hash differences There are various files committed above that can't be used on Windows because of the name, or being a symlink.

test-bundle: glob away a URL protocol separator MSYS thinks the ':' is a Unix path separator, and replaces it with ';'.

templatekw: clarify the result of {latesttag} when no tag exists My initial expectation was that the list would be empty, and therefore detectable with {if()}. The map for {latesttag()} is populated with real values in this case (except {tag}), so it probably doesn't make any sense to change this to an empty list.

hgweb: position the "followlines" box close to latest cursor position

hgweb: add a floating tooltip to invite on followlines action In followlines.js, we create a <div id="followlines-tooltip"> element to draw attention of users on "followlines" feature. The element shows up on hover of source lines after one second and follows the cursor. After first click (start line selection), the text changes and indicates that next click will terminate selection.

zstd: vendor python-zstandard 0.8.1 This contains a fix for compilation on BSDs. #no-check-commit

fsmonitor: match watchman and filesystem encoding watchman's paths encoding can differ from filesystem encoding. For example, on Windows, it's always utf-8. Before this patch, on Windows, mismatch in path comparison between fsmonitor state and osutil.statfiles would yield a clean status for added/modified files. In addition to status reporting wrong results, this leads to files being discarded from changesets while doing history editing operations such as rebase. Benchmark: There is a little overhead at module import: python -m timeit "import hgext.fsmonitor" Windows before patch: 1000000 loops, best of 3: 0.563 usec per loop Windows after patch: 1000000 loops, best of 3: 0.583 usec per loop Linx before patch: 1000000 loops, best of 3: 0.579 usec per loop Linux after patch: 1000000 loops, best of 3: 0.588 usec per loop 10000 calls to _watchmantofsencoding: python -m timeit -s "from hgext.fsmonitor import _watchmantofsencoding, _fixencoding" "fname = '/path/to/file'" "for i in range(10000):" " if _fixencoding: fname = _watchmantofsencoding(fname)" Windows (_fixencoding is True): 100 loops, best of 3: 19.5 msec per loop Linux (_fixencoding is False): 100 loops, best of 3: 3.08 msec per loop

pull: abort pull --update if config requires destination (issue5528)

tests: move update requiredest test to own test file More tests for this flag are coming in upcoming patches.

py3: add pycompat.unicode and add it to importer On python 3, builtins.unicode does not exist.

py3: add a bytes version of urllib.parse.urlencode() to pycompat.py urllib.parse.urlencode() returns unicodes on Python 3. This commit adds a method which will take its output and encode it to bytes so that we can use bytes consistently.

py3: replace str() with bytes()

metadataonlyctx: replace "changeset()[0]" to "manifestnode()" As Yuya pointed out [1], "changeset()[0]" could be simplified to "manifestnode()". I didn't notice that method earlier. It should definitely be used - it's easier to read, and faster. [1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-March/095716.html

test-check-pylint: match its output "pylint --version" shows: pylint 2.0.0, astroid 1.5.0 Python 2.7.13 (default, Dec 21 2016, 07:16:46) [GCC 6.2.1 20160830] I got "Your code has been rated at 10.00/10" every time and didn't know how to turn it off. Therefore the fix.

test-flagprocessor: remove unnecessary greps The "2>&1 | egrep ..." code is used for removing uninteresting parts from tracebacks. Now the test does not dump tracebacks, they can be removed.

bundlerepo: use raw revision in revdiff() This is similar to "revlog: use raw revisions in revdiff". revdiff() generates raw text used in revlog directly. This makes test-flagprocessor.t happy.

bundlerepo: fix raw handling in revision() Similar to fixes in revlog.py, this patch uses "rawtext" to explicitly label contents expected to be raw, and makes sure content stored in _cache is raw text. Now test-flagprocessor.t points us to another issue.

bundlerepo: build revlog index with flags This fixes bundlerevlog.flags(rev) for any revisions provided by the bundle. Now test-flagprocessor.t points us to another issue.

bundlerepo: make baserevision return raw text "baserevision" returns the text that will be used to apply deltas. Since deltas are against raw texts, "baserevision" should return raw text. Now test-flagprocessor.t points us to a new error.

test-flagprocessor: add tests about bundlerepo This shows flag processor is broken with a bundle repo. The test creates non-liner history to exercise code path where the deltaparent cannot be reused.

test-flagprocessor: use changegroup3 in bundle2 This will force "hg bundle" to use changegroup3 in the test. It is important since only changegroup3 preserves revlog flags.

bundle: allow bundle command to use changegroup3 in tests Since bundle2 writes changegroup version, we can just reuse the bundle2 format for changegroup3. This won't enable the bundle command to write changegroup3 in the wild, since exchange.parsebundlespec only returns changegroup2. It unlocks tests to override exchange.parsebundlespec and get "hg bundle" write changegroup3.

tests: add per-line output conditionals for Windows

run-tests: support per-line conditional output in tests Duplicating entire tests just because the output is different is both error prone and can make the tests harder to read. This harnesses the existing '(?)' infrastructure, both to improve readability, and because it seemed like the path of least resistance. The form is: $ test_cmd output (hghave-feature !) # required if hghave.has_feature(), else optional out2 (no-hghave-feature2 !) # req if not hghave.has_feature2(), else optional I originally extended the '(?)' syntax. For example, this: 2 r4/.hg/cache/checkisexec (execbit ?) pretty naturally reads as "checkisexec, if execbit". In some ways though, this inverts the meaning of '?'. For '(?)', the line is purely optional. In the example, it is mandatory iff execbit. Otherwise, it is carried forward as optional, to preserve the test output. I tried it the other way, (listing 'no-exec' in the example), but that is too confusing to read. Kostia suggested using '!', and that seems fine.

test-run-tests: pad the failure test to preserve the run order Test size seems to dictate the order in which the tests are run, and the next patch will add to test-success.t. Similar to c0cecc153d25.

run-tests: prevent a (glob) declaration from reordering (?) lines Previously, if a series of optional output lines marked with '(?)' had a (glob) in one of the first lines, the output would be reordered such that it came last if none of the lines were output. The (re) declaration wasn't affected, which was helpful in figuring this out. There were no tests for '(re) (?)' so add that to make sure everything plays nice.

py3: use pycompat.byteskwargs() to convert opts to bytes We have converted opts to unicodes before passing them.

test-check-code: do not use xargs We have too many files, and passing them via arguments could cause strange errors on some platforms [1]. Since check-code.py can now take "-" and read file names from stdin, use it instead of xargs to avoid the argv size limit. [1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096346.html

check-code: use "-" to specify a list of files from stdin This will be used by the next patch.

perf: add historical portability for util.timer util.timer has been introduced in ae5d60bb and used in perf.py since 22fbca1d. For historical portability, forcibly define util.timer in perf.py

diff: add --binary option for git mode diffs This patch adds --binary option to `hg diff` and `hg export` to allow more control about when binary diffs are displayed in Git mode as well as some tests to verify it behaves correctly (issue5510).

patch: make diff in git mode respect --text option (issue5510) This changeset makes patch respect -a/--text option in --git mode by aligning its behaviour with git itself.

py3: have registrar process docstrings in bytes Mixing bytes and unicode creates a mess. Do things in bytes as possible. New sysbytes() helper only takes care of ASCII characters, but avoids raising nasty unicode exception. This is the same design principle as sysstr().

localrepo: fix deprecation version for 'repo._link' The patch lingered for a while and nobody noticed when it was resubmitted.

localrepo: fix deprecation version for 'repo.join' The patch lingered for a while and nobody noticed when it was resubmitted.

tests: make zstd-related output optional Caught by the buildbot with --pure, fix suggested by indygreg.

check-code: update test IP address enforcement checks Instead of mentioning 127.0.0.1, we should use $LOCALIP. Anytime $LOCALIP appears in output, we should make sure we use (glob) on that line of output so that weird environments that do remapping jiggery pokery (such as our FreeBSD buildbot that's in a jail) don't get spurious test failures.

tests: fix missing (glob) annotations in test-push-http.t

tests: fix missing (glob) annotations in test-push-http-bundle1.t

tests: fix missing (glob) annotations in test-https.t

tests: fix missing (glob) annotations in test-bundle2-exchange.t

util: fix %-formatting on docstring by moving a closing parenthesis We have to do the % formatting over the sysstr, since the things we're going to splat into it are themselves sysstrs. This is probably technically wrong-ish, since bt is probably actually a bytestr here, but this fixes the immediate issue, which was that hg was broken on Python 3.

revset: stop supporting plain list as input set (API) There was no deprecwarn(), but this is the same kind of API compatibility as the one removed by the previous patch.

revset: stop supporting predicate that returns plain list (API) It's said to be removed after 3.9.

mdiff: add a hunkinrange helper function This factors out hunk filtering logic by line range that is similar in mdiff.blocksinrange() and hgweb.webutil.diffs().

templater: provide loop counter as "index" keyword This was originally written for JSON templating where we would have to be careful to not add extra comma, but seems generally useful. Inner loop started by % operator has its own counter.

templater: rename variable "i" to "v" in runmap() I want to reuse "i" for index.

formatter: reorder code that builds template mapping This makes the future patch slightly simpler.

revlog: avoid applying delta chain on cache hit Previously, revlog.revision(raw=False) may try to apply the delta chain on _cache hit. That happens if flags are non-empty. This patch makes rawtext reused so delta chain application is avoided. "_cache" and "rev" are moved a bit to avoid unnecessary assignments.

revlog: indent block to make review easier

revlog: avoid calculating "flags" twice in revision() This is more consistent with other code in "revision()" - prefer performance to code length.

revlog: use raw revision for rawsize When writing the revlog-ng index, the third field is len(rawtext). See revlog._addrevision: textlen = len(rawtext) .... e = (offset_type(offset, flags), l, textlen, base, link, p1r, p2r, node) self.index.insert(-1, e) Therefore, revlog.index[rev][2] returned by revlog.rawsize should be len(rawtext), where "rawtext" is revlog.revision(raw=True). Unfortunately it's hard to add a test for this code path because "if l >= 0" catches most cases.

revsetlang: enable optimization of 'x + y' expression It's been disabled since 4d1e56b29a91, but it can be enabled now as the ordering requirement is resolved at analyze().

repair: use rawvfs when copying extra store files If we use the normal vfs, store encoding will be applied when we .join() the path to be copied. This results in attempting to copy a file that (likely) doesn't exist. Using the rawvfs operates on the raw file path, which is returned by vfs.readdir(). Users at Mozilla are encountering this, as I've instructed them to run `hg debugupgraderepo` to upgrade to generaldelta. While Mercurial shouldn't deposit any files under .hg/store that require encoding, it is possible for e.g. .DS_Store files to be created by the operating system.

tests: add test demonstrating buggy path handling `hg debugupgraderepo` is currently buggy with regards to path handling when copying files in .hg/store/. Specifically, it applies the store filename encoding to paths instead of operating on raw files. This commit adds a test demonstrating the buggy behavior.

repair: iterate store files deterministically An upcoming test will add a 2nd file. Since readdir() is non-deterministic, add a sorted() to make traversal deterministic.

zstd: vendor python-zstandard 0.8.0 Commit 81e1f5bbf1fc54808649562d3ed829730765c540 from https://github.com/indygreg/python-zstandard is imported without modifications (other than removing unwanted files). Updates relevant to Mercurial include: * Support for multi-threaded compression (we can use this for bundle and wire protocol compression). * APIs for batch compression and decompression operations using multiple threads and optimal memory allocation mechanism. (Can be useful for revlog perf improvements.) * A ``BufferWithSegments`` type that models a single memory buffer containing N discrete items of known lengths. This type can be used for very efficient 0-copy data operations. # no-check-commit

commands: update help for "unbundle" Similar to the recent change to "bundle," this command no longer just deals with "changegroup" data.

commands: update help for "bundle" We now have a dedicated help topic to describe bundle specification strings. Let's update `hg bundle`'s documentation to reflect its existence. While I was hear, I also tweaked some wording which I felt was out of date and needed tweaking. Specifically, `hg bundle` no longer just deals with "changegroup" data: it can also generate files that have non-changegroup data.

help: document bundle specifications I softly formalized the concept of a "bundle specification" a while ago when I was working on clone bundles and stream clone bundles and wanted a more robust way to define what exactly is in a bundle file. The concept has existed for a while. Since it is part of the clone bundles feature and exposed to the user via the "-t" argument to `hg bundle`, it is something we need to support for the long haul. After the 4.1 release, I heard a few people comment that they didn't realize you could generate zstd bundles with `hg bundle`. I'm partially to blame for not documenting it in bundle's docstring. Additionally, I added a hacky, experimental feature for controlling the compression level of bundles in 76104a4899ad. As the commit message says, I went with a quick and dirty solution out of time constraints. Furthermore, I wanted to eventually store this configuration in the "bundlespec" so it could be made more flexible. Given: a) bundlespecs are here to stay b) we don't have great documentation over what they are, despite being a user-facing feature c) the list of available compression engines and their behavior isn't exposed d) we need an extensible place to modify behavior of compression engines I want to move forward with formalizing bundlespecs as a user-facing feature. This commit does that by introducing a "bundlespec" help page. Leaning on the just-added compression engine documentation and API, the topic also conveniently lists available compression engines and details about them. This makes features like zstd bundle compression more discoverable. e.g. you can now `hg help -k zstd` and it lists the "bundlespec" topic.

util: document bundle compression An upcoming patch will add support for documenting bundle specifications in more detail. As part of this, we'd like to enumerate available bundle compression formats. In order to do this, we need to provide the help mechanism a dict of names and objects with docstrings. This patch adds docstrings to compengine.bundletype and adds a function for retrieving a dict of them. The code is not yet used.

tests: store ETag when using --headeronly Previously, --headeronly would prevent --twice from working because the ETag wasn't stored when --headeronly was used. This feels like a bug. That feeling is reaffirmed by the fact that this change doesn't regress any tests.

hgweb: extract path traversal checking into standalone function A common exploit in web applications that access paths is to insert path separator strings like ".." to try to get the server to serve up files it shouldn't. We have code for detecting this in staticfile(). A subsequent commit will need to perform this test as well. Since this is security code, let's factor the check so we don't have to reinvent the wheel.

hgweb: use context manager for file I/O

tags: rename "head" to "node" where we don't care Followup to 5eb4d206202b (tags: extract fnode retrieval into its own function, 2017-03-28) in which the "for head in head" became "for head in nodes".

manifest: update comment to be about bytearray Looks like a leftover from 2a18e9e6ca43 (py3: use bytearray() instead of array('c', ...) constructions, 2017-03-12).

check-code: fix "covert" typo

hgweb: rename linerangelog.js as followlines.js So that the file name matches both the feature name and user facing vocabulary (e.g. the revset function).

hgweb: rely on a specific class to change cursor type in followlines UI The previous CSS rule would also apply in pages where followlines UI was not available (e.g. "changeset" view at /rev/<node>/). We insert a "followlines-select" class in JavaScript on actually selectable lines and restrict the CSS selector to use it.

hgweb: use a function expression for the install listener of followlines UI We define the listener of document's "DOMContentLoaded" inline in registration and use a function expression (anonymous) with everything inside. This makes it clearer that this file is not a library of JavaScript functions but rather an executable script. (Most of changes consists of reindenting the "followlinesBox" function, so mostly white space changes.)

formatter: use templatefilters.json() Now _jsonifyobj() is identical to templatefilters.json(paranoid=False).

templatefilters: use list comprehension in json() Not important, but the code slightly looks better.

templatefilters: unroll handling of None/False/True It doesn't make sense to use a dict here.

templatefilters: drop callable support from json() This backs out ae5447de4c11. A callable should be evaluated beforehand by templater.runsymbol().

ui: use bytes IO and convert EOL manually in ui.editor() Text IO sucks on Python 3 as it must be a unicode stream. We could introduce a wrapper that converts unicode back to bytes, but it wouldn't be simple to handle offsets transparently from/to underlying IOBase API. Fortunately, we don't need to process huge text files, so let's stick to bytes IO and convert EOL in memory.

util: add helper to convert between LF and native EOL See the next patch for why.

util: extract pure tolf/tocrlf() functions from eol extension This can be used for EOL conversion of text files.

pycompat: provide bytes os.linesep

pycompat: introduce identity function as a compat stub I was sometimes too lazy to use 'str' instead of 'lambda a: a'. Let's add a named function for that purpose.

test-blackbox: glob away quoting differences on Windows Windows uses double quotes in these places.

test-subrepo: update output for Windows

test-serve: disable unfixable tests on Windows These tests would run if hghave.has_serve() were enabled on Windows. Windows has no issue allowing an unpriviledged process to open port 13, so it doesn't abort. The other tests are related to how MSYS tries to be helpful and converts Unix constructs to the Windows equivalent. There isn't any way to disable this behavior, though it supposedly doesn't happen if the exe is linked against the MSYS library.

test-serve: kill daemons before deleting the access and error logs On Windows, `rm` will fail with 'Permission denied' if another process has it open. It looks like the rollback test was missing the kill entirely.

test-serve: make the 'listening at *' lines optional The daemonized serve process doesn't print these lines out (see 448d0c452140). I was able to get it to with the following hack: diff --git a/mercurial/win32.py b/mercurial/win32.py --- a/mercurial/win32.py +++ b/mercurial/win32.py @@ -418,6 +418,11 @@ return str(ppid) def spawndetached(args): + + import subprocess + return subprocess.Popen(args, cwd=pycompat.getcwd(), env=encoding.environ, + creationflags=subprocess.CREATE_NEW_PROCESS_GROUP).pid + # No standard library function really spawns a fully detached # process under win32 because they allocate pipes or other objects # to handle standard streams communications. Passing these objects However, MSYS translates --prefixes starting with '/' to 'C:/MinGW/msys/1.0', which changes the output. The output isn't so important that I want to spend a bunch of time on this, and risk breaking some subtle behavior of `hg serve -d` with the more complicated code.

test-http: update output for Windows The http test simply wasn't updated in 161ab32b44a1 for Windows. It looks like the https test meant to glob away the error message in 3e2d8120528b, but forgot the '*', and was subsequently removed in 408f2202bd80.

tests: quote paths in shell script hooks Without the quoting, MSYS will remove the '\' directory separators, and the repo can't be opened.

tests: add globs for Windows

show: new extension for displaying various repository data Currently, Mercurial has a number of commands to show information. And, there are features coming down the pipe that will introduce more commands for showing information. Currently, when introducing a new class of data or a view that we wish to expose to the user, the strategy is to introduce a new command or overload an existing command, sometimes both. For example, there is a desire to formalize the wip/smartlog/underway/mine functionality that many have devised. There is also a desire to introduce a "topics" concept. Others would like views of "the current stack." In the current model, we'd need a new command for wip/smartlog/etc (that behaves a lot like a pre-defined alias of `hg log`). For topics, we'd likely overload `hg topic[s]` to both display and manipulate topics. Adding new commands for every pre-defined query doesn't scale well and pollutes `hg help`. Overloading commands to perform read-only and write operations is arguably an UX anti-pattern: while having all functionality for a given concept in one command is nice, having a single command doing multiple discrete operations is not. Furthermore, a user may be surprised that a command they thought was read-only actually changes something. We discussed this at the Mercurial 4.0 Sprint in Paris and decided that having a single command where we could hang pre-defined views of various data would be a good idea. Having such a command would: * Help prevent an explosion of new query-related commands * Create a clear separation between read and write operations (mitigates footguns) * Avoids overloading the meaning of commands that manipulate data (bookmark, tag, branch, etc) (while we can't take away the existing behavior for BC reasons, we now won't introduce this behavior on new commands) * Allows users to discover informational views more easily by aggregating them in a single location * Lowers the barrier to creating the new views (since the barrier to creating a top-level command is relatively high) So, this commit introduces the `hg show` command via the "show" extension. This command accepts a positional argument of the "view" to show. New views can be registered with a decorator. To prove it works, we implement the "bookmarks" view, which shows a table of bookmarks and their associated nodes. We introduce a new style to hold everything used by `hg show`. For our initial bookmarks view, the output varies from `hg bookmarks`: * Padding is performed in the template itself as opposed to Python * Revision integers are not shown * shortest() is used to display a 5 character node by default (as opposed to static 12 characters) I chose to implement the "bookmarks" view first because it is simple and shouldn't invite too much bikeshedding that detracts from the evaluation of `hg show` itself. But there is an important point to consider: we now have 2 ways to show a list of bookmarks. I'm not a fan of introducing multiple ways to do very similar things. So it might be worth discussing how we wish to tackle this issue for bookmarks, tags, branches, MQ series, etc. I also made the choice of explicitly declaring the default show template not part of the standard BC guarantees. History has shown that we make mistakes and poor choices with output formatting but can't fix these mistakes later because random tools are parsing output and we don't want to break these tools. Optimizing for human consumption is one of my goals for `hg show`. So, by not covering the formatting as part of BC, the barrier to future change is much lower and humans benefit. There are some improvements that can be made to formatting. For example, we don't yet use label() in the templates. We obviously want this for color. But I'm not sure if we should reuse the existing log.* labels or invent new ones. I figure we can punt that to a follow-up. At the aforementioned Sprint, we discussed and discarded various alternatives to `hg show`. We considered making `hg log <view>` perform this behavior. The main reason we can't do this is because a positional argument to `hg log` can be a file path and if there is a conflict between a path name and a view name, behavior is ambiguous. We could have introduced `hg log --view` or similar, but we felt that required too much typing (we don't want to require a command flag to show a view) and wasn't very discoverable. Furthermore, `hg log` is optimized for showing changelog data and there are things that `hg display` could display that aren't changelog centric. There were concerns about using "show" as the command name. Some users already have a "show" alias that is similar to `hg export`. There were also concerns that Git users adapted to `git show` would be confused by `hg show`'s different behavior. The main difference here is `git show` prints an `hg export` like view of the current commit by default and `hg show` requires an argument. `git show` can also display any Git object. `git show` does not support displaying more complex views: just single objects. If we implemented `hg show <hash>` or `hg show <identifier>`, `hg show` would be a superset of `git show`. Although, I'm hesitant to do that at this time because I view `hg show` as a higher-level querying command and there are namespace collisions between valid identifiers and registered views. There is also a prefix collision with `hg showconfig`, which is an alias of `hg config`. We also considered `hg view`, but that is already used by the "hgk" extension. `hg display` was also proposed at one point. It has a prefix collision with `hg diff`. General consensus was "show" or "view" are the best verbs. And since "view" was taken, "show" was chosen. There are a number of inline TODOs in this patch. Some of these represent decisions yet to be made. Others represent features requiring non-trivial complexity. Rather than bloat the patch or invite additional bikeshedding, I figured I'd document future enhancements via TODO so we can get a minimal implmentation landed. Something is better than nothing.

test-revlog-raw: remove duplicated option

test-revlog-raw: fix "genbits" implementation The "genbits" implementation is actually incorrect. This patch fixes it. A good "genbits" implementation should pass the below assertion: n = 3 # or other number l = list(genbits(n)) assert 2**(n*2) == len(set((l[i]<<n)+l[i+1] for i in range(len(l)-1))) An assertion is added to make sure "genbits" won't work unexpectedly.