tests: add test demonstrating buggy path handling `hg debugupgraderepo` is currently buggy with regards to path handling when copying files in .hg/store/. Specifically, it applies the store filename encoding to paths instead of operating on raw files. This commit adds a test demonstrating the buggy behavior.

repair: iterate store files deterministically An upcoming test will add a 2nd file. Since readdir() is non-deterministic, add a sorted() to make traversal deterministic.

zstd: vendor python-zstandard 0.8.0 Commit 81e1f5bbf1fc54808649562d3ed829730765c540 from https://github.com/indygreg/python-zstandard is imported without modifications (other than removing unwanted files). Updates relevant to Mercurial include: * Support for multi-threaded compression (we can use this for bundle and wire protocol compression). * APIs for batch compression and decompression operations using multiple threads and optimal memory allocation mechanism. (Can be useful for revlog perf improvements.) * A ``BufferWithSegments`` type that models a single memory buffer containing N discrete items of known lengths. This type can be used for very efficient 0-copy data operations. # no-check-commit

commands: update help for "unbundle" Similar to the recent change to "bundle," this command no longer just deals with "changegroup" data.

commands: update help for "bundle" We now have a dedicated help topic to describe bundle specification strings. Let's update `hg bundle`'s documentation to reflect its existence. While I was hear, I also tweaked some wording which I felt was out of date and needed tweaking. Specifically, `hg bundle` no longer just deals with "changegroup" data: it can also generate files that have non-changegroup data.

help: document bundle specifications I softly formalized the concept of a "bundle specification" a while ago when I was working on clone bundles and stream clone bundles and wanted a more robust way to define what exactly is in a bundle file. The concept has existed for a while. Since it is part of the clone bundles feature and exposed to the user via the "-t" argument to `hg bundle`, it is something we need to support for the long haul. After the 4.1 release, I heard a few people comment that they didn't realize you could generate zstd bundles with `hg bundle`. I'm partially to blame for not documenting it in bundle's docstring. Additionally, I added a hacky, experimental feature for controlling the compression level of bundles in 76104a4899ad. As the commit message says, I went with a quick and dirty solution out of time constraints. Furthermore, I wanted to eventually store this configuration in the "bundlespec" so it could be made more flexible. Given: a) bundlespecs are here to stay b) we don't have great documentation over what they are, despite being a user-facing feature c) the list of available compression engines and their behavior isn't exposed d) we need an extensible place to modify behavior of compression engines I want to move forward with formalizing bundlespecs as a user-facing feature. This commit does that by introducing a "bundlespec" help page. Leaning on the just-added compression engine documentation and API, the topic also conveniently lists available compression engines and details about them. This makes features like zstd bundle compression more discoverable. e.g. you can now `hg help -k zstd` and it lists the "bundlespec" topic.

util: document bundle compression An upcoming patch will add support for documenting bundle specifications in more detail. As part of this, we'd like to enumerate available bundle compression formats. In order to do this, we need to provide the help mechanism a dict of names and objects with docstrings. This patch adds docstrings to compengine.bundletype and adds a function for retrieving a dict of them. The code is not yet used.

tests: store ETag when using --headeronly Previously, --headeronly would prevent --twice from working because the ETag wasn't stored when --headeronly was used. This feels like a bug. That feeling is reaffirmed by the fact that this change doesn't regress any tests.

hgweb: extract path traversal checking into standalone function A common exploit in web applications that access paths is to insert path separator strings like ".." to try to get the server to serve up files it shouldn't. We have code for detecting this in staticfile(). A subsequent commit will need to perform this test as well. Since this is security code, let's factor the check so we don't have to reinvent the wheel.

hgweb: use context manager for file I/O

tags: rename "head" to "node" where we don't care Followup to 5eb4d206202b (tags: extract fnode retrieval into its own function, 2017-03-28) in which the "for head in head" became "for head in nodes".

manifest: update comment to be about bytearray Looks like a leftover from 2a18e9e6ca43 (py3: use bytearray() instead of array('c', ...) constructions, 2017-03-12).

check-code: fix "covert" typo

hgweb: rename linerangelog.js as followlines.js So that the file name matches both the feature name and user facing vocabulary (e.g. the revset function).

hgweb: rely on a specific class to change cursor type in followlines UI The previous CSS rule would also apply in pages where followlines UI was not available (e.g. "changeset" view at /rev/<node>/). We insert a "followlines-select" class in JavaScript on actually selectable lines and restrict the CSS selector to use it.

hgweb: use a function expression for the install listener of followlines UI We define the listener of document's "DOMContentLoaded" inline in registration and use a function expression (anonymous) with everything inside. This makes it clearer that this file is not a library of JavaScript functions but rather an executable script. (Most of changes consists of reindenting the "followlinesBox" function, so mostly white space changes.)

formatter: use templatefilters.json() Now _jsonifyobj() is identical to templatefilters.json(paranoid=False).

templatefilters: use list comprehension in json() Not important, but the code slightly looks better.

templatefilters: unroll handling of None/False/True It doesn't make sense to use a dict here.

templatefilters: drop callable support from json() This backs out ae5447de4c11. A callable should be evaluated beforehand by templater.runsymbol().

ui: use bytes IO and convert EOL manually in ui.editor() Text IO sucks on Python 3 as it must be a unicode stream. We could introduce a wrapper that converts unicode back to bytes, but it wouldn't be simple to handle offsets transparently from/to underlying IOBase API. Fortunately, we don't need to process huge text files, so let's stick to bytes IO and convert EOL in memory.

util: add helper to convert between LF and native EOL See the next patch for why.

util: extract pure tolf/tocrlf() functions from eol extension This can be used for EOL conversion of text files.

pycompat: provide bytes os.linesep

pycompat: introduce identity function as a compat stub I was sometimes too lazy to use 'str' instead of 'lambda a: a'. Let's add a named function for that purpose.

test-blackbox: glob away quoting differences on Windows Windows uses double quotes in these places.

test-subrepo: update output for Windows

test-serve: disable unfixable tests on Windows These tests would run if hghave.has_serve() were enabled on Windows. Windows has no issue allowing an unpriviledged process to open port 13, so it doesn't abort. The other tests are related to how MSYS tries to be helpful and converts Unix constructs to the Windows equivalent. There isn't any way to disable this behavior, though it supposedly doesn't happen if the exe is linked against the MSYS library.

test-serve: kill daemons before deleting the access and error logs On Windows, `rm` will fail with 'Permission denied' if another process has it open. It looks like the rollback test was missing the kill entirely.

test-serve: make the 'listening at *' lines optional The daemonized serve process doesn't print these lines out (see 448d0c452140). I was able to get it to with the following hack: diff --git a/mercurial/win32.py b/mercurial/win32.py --- a/mercurial/win32.py +++ b/mercurial/win32.py @@ -418,6 +418,11 @@ return str(ppid) def spawndetached(args): + + import subprocess + return subprocess.Popen(args, cwd=pycompat.getcwd(), env=encoding.environ, + creationflags=subprocess.CREATE_NEW_PROCESS_GROUP).pid + # No standard library function really spawns a fully detached # process under win32 because they allocate pipes or other objects # to handle standard streams communications. Passing these objects However, MSYS translates --prefixes starting with '/' to 'C:/MinGW/msys/1.0', which changes the output. The output isn't so important that I want to spend a bunch of time on this, and risk breaking some subtle behavior of `hg serve -d` with the more complicated code.