releasenotes: add import check for fuzzywuzzy This patch adds the has_fuzzywuzzy for import check of external dependency fuzzywuzzy.

run-tests: don't drop optional lines after a missing unconditional line The previous behavior was to bail out when hitting 'awol', so 'missing (?)' was also dropped.

run-tests: drop required (feature !) style lines when the output is missing Essentially, these were acting as a verbose (?) flag, since they weren't being dropped when required. Foozy has a nice description [1]. Basically, a couple more places needed to check the features before treating it as optional. I don't like how test-run-tests.py had to be hacked, but _hghave() can't be made a static method. The test change was a change while developing `debugssl`, prior to tightening up the cases where the message is printed, that this fix would have caught. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101941.html

status: avoid recursing into ignored directory with "--terse u" Let "isignoreddir" function first check that supplied directory is itself ignored before walking recursively into its content. Otherwise, the command is awfully slow when one has an ignored directory with a lot of content. Update and rephrase function docstring accordingly.

pushvars: move fb extension pushvars to core pushvars extension in fbext adds a --pushvars flag to push command using which one send strings to server which becomes environment variables there prepended with HG_USERVAR_. These variables can then be used to run hooks on the server. The extension is moved directly to core and unbundling of the strings and converting them to environment variables at server is disabled by default for security reasons. One can turn that on by following config: [push] pushvars.server = true This patch also adds the test for the extension. Differential Revision: https://phab.mercurial-scm.org/D210

phabricator: update diff property even if we choose not to create a new diff The diff property contains metadata like "HG Node". Previously we skip uploading a new diff if we are sure that the old patch and new patch have a same content. That has issues when a pusher adds an obsmarker using the old "HG Node" stored in the old diff. This patch adds logic to update the diff property so "HG Node" gets updated to prevent that issue. Differential Revision: https://phab.mercurial-scm.org/D229

phabricator: use Phabricator's last node information This makes it more strict when checking whether or not we should update a Differential Revision. For example, a) Alice updates D1 to content 1. b) Bob updates D1 to content 2. c) Alice tries to update D1 to content 1. Previously, `c)` will do nothing because `phabsend` detects the patch is not changed. A more correct behavior is to override Bob's update here, hence the patch. This also makes it possible to return a reaonsable "last node" when there is no tags but only `Differential Revision` commit messages. Test Plan: ``` for i in A B C; do echo $i > $i; hg ci -m $i -A $i; done hg phabsend 0:: # D40: created # D41: created # D42: created echo 3 >> C; hg amend; hg phabsend . # D42: updated hg tag --local --hidden -r 2 -f D42 # move tag to the previous version hg phabsend . # D42: skipped (previously it would be "updated") rm -rf .hg; hg init hg phabread --stack D42 | hg import - hg phabsend . # D42: updated hg tag --local --remove D42 hg commit --amend hg phabsend . # D42: updated (no new diff uploaded, previously it will upload a new diff) ``` The old diff object is now returned, which could be useful in the next patch. Differential Revision: https://phab.mercurial-scm.org/D121

phabricator: add --confirm option to phabsend command This adds a --confirm flag similar to the confirm flag of `hg email` using which one can confirm the changesets before they get emailed. The confirm flag will show the changesets and ask for confirmation before sending them. Differential Revision: https://phab.mercurial-scm.org/D218

evolution: rename bumped to phase-divergent Rename bumped to phase-divergent in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D216

evolution: rename divergent to content-divergent Rename divergent to content-divergent in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D215

py3: use bytes IO to write sample hgrc Unicode sucks. Stop using Text IO and manually convert line endings.

buildrpm: do not break in presence of custom user configs For example, if "hg log" was defined as an alias: # /etc/mercurial/hgrc [alias] log = log --graph the buildrpm script would be surprised by log messages formatted in unexpected ways, and bail out. This patch sets HGPLAIN, effectively resetting all the user configs, including log output, to a common state, making the build more predictable across all the possible environments.

sparse: treat paths as cwd-relative This commit makes it so sparse treats passed paths as CWD-relative, not repo-root-realive. This is a more intuitive behavior in my (and some other FB people's) opinion. This is breaking change however. My hope here is that since sparse is experimental, it's ok to introduce BCs. The reason (glob)s are needed in the test is this: in these two cases we do not supply path together with slashes, but `os.path.join` adds them, which means that under Windows they can be backslashes. To demonstrate this behavior, one could remove the (glob)s and run `./run-tests.py test-sparse.t` from MinGW's terminal on Windows.

match: expose some data and functionality to other modules This patch makes sure that other modules can check whether patterns are CWD-relative.

sparse: properly error out when absolute paths are used Current logic is misleading (it says it drops only absolute paths, but it actually drops all of them), not cross-platform (does not support Windows) and IMO just wrong (as it should just error out if absolute paths are given). This commit fixes it.

py3: convert arbitrary exception object to byte string more reliably Our exception types implement __bytes__(), which should be tried first. Do lossy encoding conversion as a last resort.

build: delay version computation on macOS builds The way HGVER is evaluated now, it'll be evaluated at the beginning of the make execution - with this change, it's evaluated when it gets to that command, at which point the version file it's looking for is sure to exist and be up-to-date. Differential Revision: https://phab.mercurial-scm.org/D224

color: remove warnings if term is not formatted (==dumb or !ui.formatted()) If the user sets color.mode=terminfo, and then runs in the shell inside of emacs (so TERM=dumb), the previous behavior was that it would warn about no terminfo entry for setab/setaf, and then warn about 'failed to set color mode to terminfo'. The first warning is silenced by carrying 'formatted' through to _terminfosetup, the second is silenced by using 'formatted' instead of ui.formatted(). If --color=on (or ui.color=always) is specified, this will still warn, since the formatted boolean is set to true in these cases. Differential Revision: https://phab.mercurial-scm.org/D223

rpms: add chg I'm not sure if there's a reason chg is not added by default. If not, I would like to propose adding in this patch. Differential Revision: https://phab.mercurial-scm.org/D220

tests: demonstrate crash when trying to rebase merge without its parents As the test case shows, when "hg rebase -d G -r 'B + D + F'" is run on the following graph, we crash with traceback. It's reasonable to fail because we can not easily produce a correct rebased F. The problem is what diff to apply to either the rebased B or the rebased D. We could potentially produce the result by e.g. applying the (F-D) diff to the rebased B and then applying the reverse (E-D) diff on top, but that could result in merge conflicts in each of those steps, which we don't have a way of dealing with. So for now, let's just add a test case to demonstrate that we crash (i.e. the AssertionError is clearly incorrect since the user can run into it). F /| C E | | B D G \|/ A Differential Revision: https://phab.mercurial-scm.org/D212

tests: demonstrate broken rebase of merge with p1's successor in dest The fix in 8ede973597fd (rebase: handle successor targets (issue5198), 2016-04-11) only fixed the case where p2's successor was in the destination, and only when the successor was exactly the destination (i.e. not when the successor was an ancestor of it). This patch adds a test case for when p1's successor is in the destination. It adds another one for when the successor is an ancestor of the destination. To do that simply, it also rewrites the test case using drawdag. Differential Revision: https://phab.mercurial-scm.org/D211

bundle2: obtain repr() of exception in a python3-safe way This was exposed by other problems in bundle generation, but I'm not sure how to test it for now.

bundle2: use bytestr() instead of str() to convert part id to bytes This was exposed by trying to run previously-passing Python 3 tests.

bundle2: work around zip() being lazy in Python 3

bundle2: look for __next__ as well as next to identify iterators In Python 3, next is called __next__ and this was failing to catch some iterators.

bundle2: use modern Python division This was failing on Python 3 because the / was returning a float, which was then making the __mul__ on a bytes sad.

changegroup: wrap some ** expansions in strkwargs

obsolete: reuse _fm1metapair for computing _fm1metapairsize It's evaluated at import time, so it seems silly to not reuse the constant name.

evolution: rename unstable to orphan Rename unstable to orphan in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D214

evolution: rename trouble(s) to instability Rename trouble(s) to instability in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D213

bundle2: load hookargs from bundleoperation into transaction when started one When a transaction is started, we must load the hookargs from the bundleoperation object to the transaction so that they can be used in the transaction. Also this patch makes sure no more hookargs are added to the bundleoperation object once the transaction starts. This is a part of porting fb extension bundle2hooks to core. Differential Revision: https://phab.mercurial-scm.org/D209

bundle2: add the capability to store hookargs on bundle operation object There are extensions like pushrebase, pushvars which run hooks on a server before taking the lock. Since the lock is not taken, transaction is not there, so the hookargs can't be stored on the transaction. Adding hooksargs to bundle operation object will help in running hooks before taking the lock. This is a part of moving fb's extension bundle2hooks to core. Differential Revision: https://phab.mercurial-scm.org/D208

rebase: use one dirstateguard for when using rebase.singletransaction This was previously landed as 2519994d25ca but backed out in b63351f6a2 because it broke hooks mid-rebase and caused conflict resolution data loss in the event of unexpected exceptions. This new version adds the behavior back but behind a config flag, since the performance improvement is notable in large repositories. The old commit message was: Recently we switched rebases to run the entire rebase inside a single transaction, which dramatically improved the speed of rebases in repos with large working copies. Let's also move the dirstate into a single dirstateguard to get the same benefits. This let's us avoid serializing the dirstate after each commit. In a large repo, rebasing 27 commits is sped up by about 20%. I believe the test changes are because us touching the dirstate gave the transaction something to actually rollback. (grafted from 9e3dc3a1638b9754b58a0cb26aaa75d868058109) (grafted from 7d38b41d2266d9a02a15c64229fae0da5738dcec) Differential Revision: https://phab.mercurial-scm.org/D135

merge with stable

tests: invoke run-tests.py in test-hghave using $PYTHON (issue5697) Some platforms (notably pkgsrc on NetBSD) only provide versioned Python interpreters (eg `python2.7` exists, but `python` does not), which exposes this error. We want to be running run-tests.py with the python given in $PYTHON, since that might be `pypy` or `python3`.

Added signature for changeset 2f427b57bf90

Added tag 4.3.3 for changeset 2f427b57bf90

rebase: move bookmarks with --keep (issue5682) This is a regression caused by 3b7cb3d17137. We have documented the behavior in rebase help: Rebase will destroy original commits unless you use "--keep". It will also move your bookmarks (even if you do). So let's restore the old behavior. It is done by changing `scmutil.cleanupnodes` to accept more information so a node could have different "movement destination" from "successors". It also helps simplifying the callsite as a side effect - the special bookmark movement logic in rebase is removed. Differential Revision: https://phab.mercurial-scm.org/D727

cleanupnodes: rename "mapping" to "replacements" The next patch will pass in overrides for bookmark moves, which is a mapping itself, so let's rename "mapping" to "replacements" to distinguish them better. Differential Revision: https://phab.mercurial-scm.org/D749

cleanupnodes: separate out bookmark destination calculation from actual update We will soon want to pass in overrides for bookmark movements and this will make that patch simpler. I also think this makes the code easier to follow regardless of the later patch. Differential Revision: https://phab.mercurial-scm.org/D748

ui: check for progress singleton when clearing progress bar (issue5684) A combination of wrapping `ui` and progress bars interrupted by exceptions can lead to the progress bar not being cleared when the exception error is printed. This results in corrupted-looking output like this: ``` updating [===============================> ] 1/2u nresolved conflicts (see hg resolve, then hg rebase --continue) ``` This is because in `ui._progclear`, we only check the local reference to the progress bar, not whether or not there is an instance of the singleton. When a progress bar is interrupted by an exception, the exception printing in `scmutil.callcatch` uses the original instance of the `ui` object, not the wrapped copy that has `_progbar` set. When consider whether or not to clear the progress bar, check for the existence of the singleton, rather than just whether or not we have a local reference to it. Differential Revision: https://phab.mercurial-scm.org/D743

Added signature for changeset 920977f72c7b

Added tag 4.3.2 for changeset 920977f72c7b

merge with i18n

i18n-pt_BR: synchronized with 850d2ec2cf6a

hgwebdir: read 'web.template' untrusted The 'hgweb_mod.py' version of this read it untrusted. For consistency we align the two versions of this code.

ssh: fix flakey ssh errors on BSD systems This is a trivial backport of c037fd655b47 performed by augie@google.com, but the change is still really Durham's not mine, so I [augie] am leaving him as the author.

repair: preserve phase also when not using generaldelta (issue5678) It seems like we used to pick the oldest possible version of the changegroup to use for bundles created by the repair module (used e.g. by "hg strip" and for temporary bundles by "hg rebase"). I tried to preserve that behavior when I created the changegroup.safeversion() method in 3b2ac2115464 (changegroup: introduce safeversion(), 2016-01-19). However, we have recently chagned our minds and decided that these commands are only used locally and downgrades are unlikely. That decicion allowed us to start adding obsmarker and phase information to these bundles. However, as the bug report shows, it means we get different behavior e.g. when generaldelta is not enabled (because when it was enabled, it forced us to use bundle2). The commit that actually caused the reported bug was 8e3021fd1a44 (strip: include phases in bundle (BC), 2017-06-15). So, since we now depend on having more information in the bundles, let's make sure we instead pick the newest possible changegroup version. Differential Revision: https://phab.mercurial-scm.org/D715

tests: add test for issue5678 In addition to a test case for the direct problem described in the bug report, this also adds a test case showing how obsmarkers can also get lost when not using generaldelta. Differential Revision: https://phab.mercurial-scm.org/D714

mq: create non-lossy patches, also with custom global diff configuration Users with custom [diff] configuration most certainly didn't intend it to make mq lose changes. It could: * git is handled perfectly fine. * nobinary could make mq leave some files out from the patches. * noprefix could make mq itself (and probably also other tools) fail to apply patches without the usual a/b prefix. * ignorews, ignorewsamount, or ignoreblanklines could create patches with missing whitespace that could fail to apply correctly. Thus, when refreshing patches, use patch.difffeatureopts, optionally with git as before, but without the config options for whitespace and format changing that most likely will cause loss or problems. (patch.diffopts is just patch.difffeatureopts with all options enabled and can be replaced with that.)

mq: test coverage of how [diff] configuration influence can break mq patches

debugssl: allow a URL to be specified without a local repository This was the original intent, but I bungled the logic. Otherwise if there is a certificate chain issue, the repository can't be cloned in order for there to be a repo object. I think I missed this case because I was inside of a Mercurial clone as I was originally developing and testing this.

record: fix revert -i for lines without newline (issue5651) This is a regression caused by 66117dae87f9. Code prior to 66117dae87f9 seems to miss the "\ No newline at end of file" line. Differential Revision: https://phab.mercurial-scm.org/D528

templatekw: choose {latesttag} by len(changes), not date (issue5659) As Augie reported in the bug, the current heuristic of choosing the best tag of a merge commit by taking the one with newest tag (in terms of tagging date) currently fails in the Mercurial repo itself. Copying the example from Yuya: $ hg glog -T '{node|short} {latesttag}+{latesttagdistance}\n' \ -r '4.2.3: & (merge() + parents(merge()) + tag())' o 02a745c20121 4.2.3+5 |\ | o 86aca74a063b 4.2.3+4 | |\ | | o e6d8ee3c9ec3 4.3-rc+109 | | | | | ~ o | a3ce07e2dde5 4.3.1+2 : | o | 3fee7f7d2da0 4.3.1+0 |/ o 98e990bb7330 4.2.3+3 |\ | ~ o 506d7e48fbe6 4.2.3+2 : o 943c91326b23 4.2.3+0 | ~ It seems to me like the best choice is the tag with the smallest number of changes since it (across all paths, not the longest single path). So that's what this patch does, even though it's costly. Best-of-5 timings for Yuya's command above shows a slowdown from 1.293s to 1.610s. We can optimize it later. Differential Revision: https://phab.mercurial-scm.org/D447

tests: use graph log in {latesttag} tests The tests are much easier to read if one does not have to re-read the setup part all the time to understand the graph shape. Differential Revision: https://phab.mercurial-scm.org/D446

ui: restore behavior to ignore some I/O errors (issue5658) e9646ff34d55 and 1bfb9a63b98e refactored ui methods to no longer silently swallow some IOError instances. This is arguably the correct thing to do. However, it had the unfortunate side-effect of causing StdioError to bubble up to sensitive code like transaction aborts, leading to an uncaught exceptions and failures to e.g. roll back a transaction. This could occur when a remote HTTP or SSH client connection dropped. The new behavior is resulting in semi-frequent "abandonded transaction" errors on multiple high-volume repositories at Mozilla. This commit effectively reverts e9646ff34d55 and 1bfb9a63b98e to restore the old behavior. I agree with the principle that I/O errors shouldn't be ignored. That makes this change... unfortunate. However, our hands are tied for what to do on stable. I think the proper solution is for the ui's behavior to be configurable (possibly via a context manager). During critical sections like transaction rollback and abort, it should be possible to suppress errors. But this feature would not be appropriate on stable.

tests: test behavior of IOError during transactions (issue5658) ui._write(), ui._write_err(), and ui.flush() all trap IOError and re-raise as error.StdioError. If a caller doesn't catch StdioError when writing to stdio, it could bubble all the way to dispatch. This commit adds tests for I/O failures around various transaction operations. The most notable badness is during abort. Here, an uncaught StdioError will result in incomplete transaction rollback, requiring an `hg rollback` to recover. This can result in a client "corrupting" a remote repo via terminated HTTP and SSH socket.

log: mention ui.logtemplate in the help text

hg: avoid relying on errno numbers / descriptions A few tests hardcode errno numbers and/or descriptions in the output, causing test failures on platforms where these values are different. Differential Revision: https://phab.mercurial-scm.org/D362

hg: tolerate long vs. int in test-context.py The file times here can be longs instead of ints on some platforms, which will cause a test failure due to these printing with an L suffix; instead always format with %d which will produce the same output in either case. Differential Revision: https://phab.mercurial-scm.org/D361

setup: fix installing in a mingw environment The addition, in 9a4adc76c88a, of a hack for the MSVC compiler class was overwriting the original class for the Mingw32CCompiler class, leading to an error when the HackedMingw32CCompiler is instantiated. Differential Revision: https://phab.mercurial-scm.org/D329

osx: delay version computation on macOS builds Before this patch, HGVER would be evaluated at the beginning of the make execution, and would be unset because build/mercurial/ doesn't exist yet at that point. Now we compute the version after the `make install` run has completed. This is backported to stable from 8626b44516c1, but that revision had an error in the shell invocation syntax.

branchmap: revert c34532365b38 for Python 2.7 compatibility Old versions of python 2.7 don't like that the second argument to struct.unpack_from is a bytearray, so the change removing the util.buffer around that argument in branchmap broke running on older versions of python 2.7. Differential Revision: https://phab.mercurial-scm.org/D330

Added signature for changeset 3fee7f7d2da0

Added tag 4.3.1 for changeset 3fee7f7d2da0

ssh: unban the use of pipe character in user@host:port string This vulnerability was fixed by the previous patch and there were more ways to exploit than using '|shellcmd'. So it doesn't make sense to reject only pipe character. Test cases are updated to actually try to exploit the bug. As the SSH bridge of git/svn subrepos are not managed by our code, the tests for non-hg subrepos are just removed. This may be folded into the original patches.

ssh: quote parameters using shellquote (SEC) This patch uses shellquote to quote ssh parameters more strictly to avoid shell injection.

subrepo: add tests for git rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for svn rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for hg rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

push: add tests for unsafe ssh url (SEC)

pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

stable: merge heads

Added signature for changeset 943c91326b23

Added tag 4.2.3 for changeset 943c91326b23

ssh: unban the use of pipe character in user@host:port string This vulnerability was fixed by the previous patch and there were more ways to exploit than using '|shellcmd'. So it doesn't make sense to reject only pipe character. Test cases are updated to actually try to exploit the bug. As the SSH bridge of git/svn subrepos are not managed by our code, the tests for non-hg subrepos are just removed. This may be folded into the original patches.

ssh: quote parameters using shellquote (SEC) This patch uses shellquote to quote ssh parameters more strictly to avoid shell injection.

subrepo: add tests for git rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for svn rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for hg rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

push: add tests for unsafe ssh url (SEC)

pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

Added signature for changeset 5544af862286

Added tag 4.3 for changeset 5544af862286