sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

stable: merge heads

Added signature for changeset 943c91326b23

Added tag 4.2.3 for changeset 943c91326b23

ssh: unban the use of pipe character in user@host:port string This vulnerability was fixed by the previous patch and there were more ways to exploit than using '|shellcmd'. So it doesn't make sense to reject only pipe character. Test cases are updated to actually try to exploit the bug. As the SSH bridge of git/svn subrepos are not managed by our code, the tests for non-hg subrepos are just removed. This may be folded into the original patches.

ssh: quote parameters using shellquote (SEC) This patch uses shellquote to quote ssh parameters more strictly to avoid shell injection.

subrepo: add tests for git rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for svn rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for hg rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

push: add tests for unsafe ssh url (SEC)

pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

Added signature for changeset 5544af862286

Added tag 4.3 for changeset 5544af862286

chg: define _GNU_SOURCE to allow CentOS 5 compilation Without this flag, compilation fails with: hgclient.c: In function 'hgc_open': hgclient.c:466: error: 'O_DIRECTORY' undeclared (first use in this function) hgclient.c:466: error: (Each undeclared identifier is reported only once hgclient.c:466: error: for each function it appears in.) Differential Revision: https://phab.mercurial-scm.org/D260

centos5: use vault urls CentOS 5 has reached end-of-life. A working build for it is still possible using 'vault.centos.org'. Differential Revision: https://phab.mercurial-scm.org/D261

ui: make sure buffer is flushed before waiting for user input (issue5587) Without this patch on Windows 'hg ci -i' hangs waiting for user input and "examine changes to 'file'? [Ynesfdaq?]" is never displayed (at least if the diff is sufficiently small). When Ctrl+C is pressed, this prompt becomes visible, which suggests that the buffer just wasn't flushed. I've never seen this happening on Linux, but this looks harmless enough to not platform-gate it.

util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) It appears that overriding __setitem__() doesn't work as documented on PyPy. Let's patch it as before bd0fd3ff9916. https://docs.python.org/2/library/collections.html#ordereddict-examples-and-recipes The issue was ui.configitems() wasn't ordered correctly, so the pull command was wrapped in different order.

test-revlog-raw: close file handles explicitly (issue5644)

tests: clear __pycache__ for PyPy compatibility (issue5638) (issue5642)

test-extension: glob module name in error message for PyPy

rebase: sort collapsed revisions in commit message (issue5643) This fixes the test instability.

i18n: use actual filename, in which function is defined, for hg.pot Before this patch, source filename for msgid in hg.pot file becomes incorrect, if a function is defined in file A, but detected in dict in file B, For example, almost all debug* commands are defined in debugcommands.py, but hggettext detects them in "table" of commands.py. Therefore, docstring fragments of debug* commands are marked as "defined in commands.py" in hg.pot file. This is serious problem for translation, because the cost to find out original location of texts increases very much.

ui: enable pager always for explicit --pager=on (issue5580) Before this patch, explicit --pager=on is unintentionally ignored by any disabling factor, even if priority of it is less than --pager=on (e.g. "[ui] paginate = off").

rebase: use one dirstateguard for when using rebase.singletransaction This was previously landed as 2519994d25ca but backed out in b63351f6a2 because it broke hooks mid-rebase and caused conflict resolution data loss in the event of unexpected exceptions. This new version adds the behavior back but behind a config flag, since the performance improvement is notable in large repositories. The old commit message was: Recently we switched rebases to run the entire rebase inside a single transaction, which dramatically improved the speed of rebases in repos with large working copies. Let's also move the dirstate into a single dirstateguard to get the same benefits. This let's us avoid serializing the dirstate after each commit. In a large repo, rebasing 27 commits is sped up by about 20%. I believe the test changes are because us touching the dirstate gave the transaction something to actually rollback. (grafted from 9e3dc3a1638b9754b58a0cb26aaa75d868058109) (grafted from 7d38b41d2266d9a02a15c64229fae0da5738dcec) Differential Revision: https://phab.mercurial-scm.org/D135

README: make paragraphs before example code end with "::" for reST syntax After this patch, README as reST text can be rendered into HTML more appropriately. This is useful for hosting servers, which render README file as a part of index HTML of its directory (for example, Bitbucket).

commit: don't let failed commit with --addremove update dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D204

tests: demonstrate that failed "hg ci -A" updates dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D203

tests: fix test-flagprocessor.t flakiness The test for duplicate flag processors depended on the timestamps being set in the dirstate to work. If the time between the the previous failed commit (which would set the timestamp, due to bug 5645) and the attempted commit with the duplicate flag processors was small enough, it would fail. The failure was caused by a call to commands.status() early in the commit process. If the dirstate did not have the timestamp set, it would need to fetch the file content to compare with. Since two flag processors had been registered, it would attempted to base64 decode the contents twice, which would of course fail. This patch adds a "hg debugrebuilddirstate" to make it deterministic and also replaces the test case's "hg commit" by simply "hg status", since that will trigger reading of the contents and thereby use of the flag processors as noted above. Differential Revision: https://phab.mercurial-scm.org/D202

tests: clarify that duplicate flag processors is not an error The flag processors test for duplicate processors for a single flag was misleading because the file from the previous test case caused it to fail (making the "echo 'this should fail' > file" part irrelevant). Let's remove the leftover from the previous test case to make it clear that duplicate flag processors results only in a warning. Note that duplicate flag processors would have resulted in a failure (not just a warning) until ea1c2eb7abd3 (extensions: catch uisetup and extsetup failures and don't let them break hg, 2017-06-06). I remember expressing my concern about ending up with half-loaded extensions. It would be pretty unfortunate to have double-encoded revlog content enter a repo, so maybe we should reconsider? Differential Revision: https://phab.mercurial-scm.org/D201

tests: avoid unexpected result at invocation of *.py file on Windows Invocation of "diff tool.py" in test-extdiff.t tests whether shellquote() is applied on specified command as expected. But direct invocation of "*.py" file might cause unexpected result on Windows according to suffix binding. For example, starting IDE, showing dialog to choose program to be used, and so on. In such case, running test-extdiff.t is easily timed out. This patch uses intermediate *.bat file on Windows, to avoid such unexpected result. Naming that intermediate file as "diff tool.bat" is enough to test applying shellquote().

tests: use pdiff instead of diff for portability "diff" command might cause redundant message, "No differences encountered" on Solaris for example. But suppressing option like "-q" isn't portable, because POSIX specification doesn't define it. pdiff script was introduced by b59ef0c21405 to stabilize output of standard diff command on each platforms.

tests: make pdiff return appropriate exit code at comparison of files Before this patch, pdiff script returns 0, even if diff is detected. This issue doesn't cause failure of tests using it, if it is invoked via extdiff extension, because extdiff itself examines changes between specified revisions and decides exit code. BTW, this patch ignores recursive comparison case, because: - there is no portable way for current while-read based implementation to return 1 at detecting changes - it isn't yet needed to replace direct "diff -r" invocation by pdiff for portability

merge with stable

tests: make test-profile.t pass if statprof didn't collect samples We had the first 3 tests in test-profile.t fail because the output didn't match. I have not yet confirmed that this was because no samples were collected, but we shouldn't require samples to be collected for the test to pass either way. Differential Revision: https://phab.mercurial-scm.org/D199

pypy: fix failing test-devel-warnings.t with Pypy5.6.0 In Pypy 5.6.0, traceback exception classes are not displayed with their full qualified name. Instead of displaying mercurial.error.ProgrammingError, Pypy displays ProgrammingError. Update the test to support both version.

pypy: fix failing test files with Pypy5.6.0 Pypy 5.6.0 saves cached bytecode files in __pycache__ directory, clean them in tests to fix loading old test extensions code. Doing so should also helps for Python3.x migration.

commitextras: make sure keys are not empty Thanks to Yuya for catching this.

make: add new directory 'hgdemandimport' to cleanbutpackages rule This fixes the build process for ppa (at least in my simple check). Perhaps we should loop through all directory that aren't 'packages'?

statichttprepo: implement wlock() (issue5613) statichttprepo inherits from localrepository. In doing so, it obtains default implementations of various methods, like wlock(). Before this change, tags cache writing would call repo.wlock(). This failed on statichttprepo due to localrepository's wlock() looking for an instance attribute that doesn't exist on statichttprepo (statichttprepo doesn't call localrepository.__init__). We /could/ define missing attributes until the base wlock() works. However, a statichttprepo is remote and read-only and can't be locked. The class already has a lock() that short circuits. So it makes sense to implement a short-circuited wlock() as well. That is what this patch does. LockError is expected to be raised when locking fails. The constructor takes a number of arguments that are local repository centric. Rather than rework LockError to not require them (which would not be appropriate for stable), this commit populates dummy values. I don't believe they'll ever be seen by the user, as lock failures on static http repos should be limited to well-defined (and tested) scenarios. We can and should revisit the LockError type to improve this.

test: add tests for branches and tags with static http repo (issue5613) Mercurial currently fails to clone by revision on static http repos with tags. The added tests demonstrate this.

commitextras: make sure keys contains ascii letters, numbers, '_' and '-' only

phabricator: convert unicode to binary when writing patches This is a quick fix to make `hg phabread D189` work. It seems we might want to replace all `r''` to `u''`, and add more `encoding.*to*` to be more explicit when interacting with `json` module. Differential Revision: https://phab.mercurial-scm.org/D192

setup: silence warning of unknown option python_requires on distutils /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'

setup: forcibly include cext/pure packages in py2exe (issue5625) Since these modules are implicitly imported by policy.importmod(), py2exe can't track them statically. cffi modules are excluded for now because they wouldn't be useful in frozen (i.e. CPython) environment.

setup: do not select hg executable that prints unexpected warnings Otherwise the subsequent hg.run() would fail. This factors out the filtering function so the same rule should apply.

docker: pass proxy arguments to docker process Building in Docker when behind a proxy requires passing the proxy variables. Differential Revision: https://phab.mercurial-scm.org/D190

makefile: build on CentOS 6 with a Python 2.7 Differential Revision: https://phab.mercurial-scm.org/D189

osx: wire up genosxversion script The only version strings that are changed are the ones baked into the .pkg - hg's self-reported version string doesn't change, so users will still see our mostly-pip-compatible version strings. For reference, the part of our versioning setup that's not PEP440 compatible is the RC releases - those should be .rc0 insted of -rc. It's too late to change that for the 4.3 cycle, so I'll worry about fixing that during the 4.4 cycle.

osx: new script for generating OS X package versions If you're shipping prerelease or rc packages using Munki, you'll eventually discover that Munki's version comparison logic is not as good as pip's. In theory we should be able to fix Munki, but it seems entirely reasonable to produce version strings that sort reasonably under these conditions. Since the requried logic not brief, add a new script and some tests of that logic. A followup change will wire this into the Makefile.

tests: update mac packaging test expectations Some more files are included these days.