i18n: make hggettext use original docstring to compute offset Before this patch, hggettext uses __doc__ of each functions to compute offset of document text. But __doc__ of many functions is already modified by decorators in registrar (e.g. @templatekeyword adds ":NAME: " prefix to it), and hggettext can not find it out in original source. This causes many "unknown offset in ..." warning at "make update-pot", and leaving them might cause overlooking serious problems. This patch makes hggettext use original docstring, which decorators in registrar save into _origdoc, to compute offset. Even after this patch, there are still a few "unknown offset in ..." warning at "make update-pot" for specific reasons. These will be fixed later one by one.

run-tests: also color the summary messages (skipped, failed...)

crecord: fixes the formatting of the select status in the status line The status line in the crecord has the "space" status field which has variable length depending on the length of the status label in the language of choice. In English, the status labels are "space: deselect" and "space:select". The "deselect" label is 2 glyphs longer. This makes the terminal output jump around if the terminal width is just right so that the shorter label makes the status line 1 line long, and the longer label makes it 2 lines long. This patch formats the selected status into a fixed-width field. The field width is the maximum of the lengths of the two possible labels, to account for differing translations and label lengths. This should make the label behavior uniform across localizations. There does not seem to be a test for crecord, so I verified the change manually with a local build of 'hg'.

cext: move PyInt macros to charencode.c properly Python3 build was broken at e9996bd7203f.

py3: change encoding.localstr to a subclass of bytes, not str

bundle2: relax the condition to update transaction.hookargs This is just a micro optimization. If hookargs is empty, nothing should be necessary.

bundle2: raise ProgrammingError for invalid call of addhookargs() It should be hard error. Also fixed the error message as s/hooks/hookargs/.

merge: removed sorting in casefolding detection, for a slight performance win It was not required for the correctness of the algorithm. Differential Revision: https://phab.mercurial-scm.org/D30

tests: verify that peer instances only expose interface members Our abstract interfaces are more useful if we guarantee that implementations conform to certain rules. Namely, we want to ensure that objects implementing interfaces don't expose new public attributes that aren't part of the interface. That way, as long as consumers don't access "internal" attributes (those beginning with "_") then (in theory) objects implementing interfaces can be swapped out and everything will "just work." We add a test that enforces our "no public attributes not part of the abstract interface" rule. We /could/ implement "interface compliance detection" at run-time. However, that is littered with problems. The obvious solutions are custom __new__ and __init__ methods. These rely on derived types actually calling the parent's implementation, which is no sure bet. Furthermore, __new__ and __init__ will likely be called before instance-specific attributes are assigned. In other words, they won't detect public attributes set on self.__dict__. This means public attribute detection won't be robust. We could work around lack of robust self.__dict__ public attribute detection by having our interfaces implement a custom __getattribute__, __getattr__, and/or __setattr__. However, this incurs an undesirable run-time penalty. And, subclasses could override our custom method, bypassing the check. The most robust solution is a non-runtime test. So that's what this commit implements. We have a generic function for validating that an object only has public attributes defined by abstract classes. Then, we instantiate some peers and verify a newly constructed object plays by the rules. Differential Revision: https://phab.mercurial-scm.org/D339

wireproto: use new peer interface The wirepeer class provides concrete implementations of peer interface methods for calling wire protocol commands. It makes sense for this class to inherit from the peer abstract base class. So we change that. Since httppeer and sshpeer have already been converted to the new interface, peerrepository is no longer adding any value. So it has been removed. httppeer and sshpeer have been updated to reflect the loss of peerrepository and the inheritance of the abstract base class in wirepeer. The code changes in wirepeer are reordering of methods to group by interface. Some Python code in tests was updated to reflect changed APIs. .. api:: peer.peerrepository has been removed. Use repository.peer abstract base class to represent a peer repository. Differential Revision: https://phab.mercurial-scm.org/D338

httppeer: use peer interface This is similar to what we did to sshpeer. Quirks and all. Differential Revision: https://phab.mercurial-scm.org/D337

sshpeer: use peer interface We need the same @property conversion of ui like we did for localpeer. We renamed _capabilities() to capabilities() to satisfy the new naming requirement. However, since we're inheriting from wireproto.wirepeer which inherits from peer.peerrepository and provides its own code accessing _capabilities(), we need to keep the old alias around. This wonkiness will disappear once wirepeer is cleaned up in subsequent commits. We also implement methods for basepeer that are identical to the defaults in peer.peerrepository in preparation for the removal of peerrepository. Differential Revision: https://phab.mercurial-scm.org/D336

localrepo: use peer interfaces We now have a formal abstract base class for peers. Let's transition the peer classes in localrepo to it. As part of the transition, we reorder methods so they are grouped by interface and match the order they are defined in the interface. We also had to change self.ui from an instance attribute to a property to satisfy the @abstractproperty requirement. As part of this change, we uncover the first "bug" as part of enforcing interfaces: stream_out() wasn't implemented on localpeer! This isn't technically a bug since the repo isn't advertising the stream capability, so clients shouldn't be attempting to call it. But I don't think there's a good reason why this is the case. We implement a dummy method to satisfy the interface requriements. We can make localpeer instances streamable as a future enhancement. # no-check-commit Differential Revision: https://phab.mercurial-scm.org/D335

repository: implement generic capability methods on peer class These methods are part of the peer interface, are generic, and can be implemented in terms of other members of the peer interface. So we implement them on the peer base class as a convenience. The implementation is essentially copied from peer.py. The code in peer.py will eventually be deleted. Differential Revision: https://phab.mercurial-scm.org/D334

repository: formalize wire protocol interface There are a well-defined set of commands constituting the wire protocol. Interaction with these and methods for calling them in batches are exposed via methods on peer instances. Let's formalize support for these features in abstract classes. The command parts come from the existing wireproto.wirepeer class. The batch methods come from peer.peerrepository. Ample documentation has been added as part of defining the interfaces. # no-check-commit Differential Revision: https://phab.mercurial-scm.org/D333

repository: formalize peer interface with abstract base class There are various interfaces for interacting with repositories and peers. They form a contract for how one should interact with a repo or peer object. The contracts today aren't very well-defined or enforced. There have been several bugs over the years where peers or repo types have forgotten to implement certain methods. In addition, the inheritance of some classes is wonky. For example, localrepository doesn't inherit from an interface and the god-object nature of that class means the repository interface isn't well-defined. Other repository types inherit from localrepository then stub out methods that don't make sense (e.g. statichttprepository re-defining locking methods to fail fast). Not having well-defined interfaces makes implementing alternate storage backends, wire protocol transports, and repository types difficult because it isn't clear what exactly needs to be implemented. This patch starts the process of attempting to establish more order to the type system around repositories and peers. Our first patch starts with a problem space that already has a partial solution: peers. The peer.peerrepository class already somewhat defines a peer interface. But it is missing a few things and the total interface isn't well-defined because it is combined with wireproto.wirepeer. Our newly-established basepeer class uses the abc module to declare an abstract base class with the properties and methods that a generic peer must implement. We create a new class that inherits from it. This class will hold our other future abstract base classes / interfaces so we can expose a unified base class/interface. We don't yet use the new interface because subsequent additions will break existing code without some refactoring first. A new module (repository.py) was created to hold the interfaces. I could have put things in peer.py. However, I have plans to eventually add interfaces to define repository and storage types. These almost certainly require a new module. And I figured having all the interfaces live in one module makes sense. So I created repository.py to be that future home. Differential Revision: https://phab.mercurial-scm.org/D332

util: make nogc effective for CPython 279cd80059d4 made `util.nogc` a no-op. It was to optimize PyPy. But it slows down CPython if many objects (like 300k+) are created. For example, running `hg log -r .` without extensions in `hg-committed` with 14k+ obsmarkers have the following times: before | after hg | chg | hg | chg ----------------------------- 1.262 | 0.860 | 1.077 | 0.619 (seconds, best of 20 runs) Therefore let's re-enable nogc for CPython. Differential Revision: https://phab.mercurial-scm.org/D402

scmutil: use util.shellquote instead of %r Changes some output, but also resolves differences with Python 3. Differential Revision: https://phab.mercurial-scm.org/D301

context: fix troubled deprecation troubled has been renamed into isunstable but troubled was calling unstable instead. Fix the mistake. Differential Revision: https://phab.mercurial-scm.org/D384

test-pushvars: invoke shell script hook via `sh` for Windows Invoking *.sh on Windows leads to the "what program should open this?" prompt, which stalls the test and led to the recent series of exceptions on the Windows test machine as the runner times out.

test-sparse: drop unnecessary globs

exchange: simplify unbundle locking using context managers Differential Revision: https://phab.mercurial-scm.org/D393

util: add base class for transactional context managers We have at least three types with a close() and a release() method where the close() method is supposed to be called on success and the release() method is supposed to be called last, whether successful or not. Two of them (transaction and dirstateguard) already have identical implementations of __enter__ and __exit__. Let's extract a base class for this, so we reuse the code and so the third type (transactionmanager) can also be used as a context manager. Differential Revision: https://phab.mercurial-scm.org/D392

exchange: remove need for "locked" variable The transactionmanager() constructor just assigned a few variables and cannot fail, so it's safe to move it inside the earlier try/except. Differential Revision: https://phab.mercurial-scm.org/D391

exchange: drop now-unnecessary "local" from lock name variables Since fda0867cfe03 (exchange: drop support for lock-based unbundling (BC), 2017-08-06), there is no more remote locking. Differential Revision: https://phab.mercurial-scm.org/D390

exchange: remove 'locallocked' member from pushop object The variable has been used only within a single function since 94e2862dbcfb (push: elevate phase transaction to cover entire operation, 2014-11-21), so there's no need to keep it on the pushop object. Differential Revision: https://phab.mercurial-scm.org/D389

rebase: change "result would have 3 parent" error message (BC) The old error message "cannot use revision REV as base, result would have 3 parents" is confusing - why use REV as base? why add a new parent?. This patch changes it to "cannot move parent", which seems better. Differential Revision: https://phab.mercurial-scm.org/D342

drawdag: allow override file contents via comments This makes drawdag more flexible, and allow us to create non-clean merges in test cases. Differential Revision: https://phab.mercurial-scm.org/D385

phabricator: add --amend option to phabsend Previously `hg phabsend` was imitating `hg email` and won't mutate changesets. That works fine with reviewer-push workflow, reviewers run `phabread`, `import`. However, it does not work well with author-push workflow. Namely, the author needs to run extra commands to get the right commit message, and remove the local tag after push. This patch solves those issues by adding the `--amend` option, so local changesets will have the right commit message, and tags become unnecessary. Test Plan: Given the following DAG: o 17 o 16 | o 15 | @ 14 |/ o 13 o 12 Run `hg phabsend '(13::)-17' --amend`, check the new DAG looks like: o 21 | o 20 | @ 19 |/ o 18 | o 17 | x 16 | x 13 |/ o 12 And commit messages are updated to contain the `Differential Revision` lines. Use `phabread` to make sure Phabricator has the amended node recorded. Also check `phabsend .` followed by a `phabsend . --amend`, the commit message will be updated and the tag will be removed. Differential Revision: https://phab.mercurial-scm.org/D122

rebase: rewrite core algorithm (issue5578) (issue5630) "defineparents" is the core algorithm of rebase. The old code has too many tech debts (like outdated comments, confusing assertions, etc) and is very error-prone to be improved. This patch rewrites "defineparents" to make the code easier to reason about, and solve a bunch of issues, including: - Assertion error: no base found (demonstrated by D212, issue5578) - Asymmetric result (demonstrated by D211, "F with one parent") - Wrong new parent (demonstrated by D262, "C':A'A'") - "revlog index out of range" (demonstrated by D262, issue5630) - "nothing to merge" (demonstrated by D262) As a side effect, merge base decision has been made more solid - rebase now prints out explicitly what could go wrong when it cannot find a unique suitable merge base. .. fix:: Issue 5578, Issue 5630 Core rebase algorithm has been rewritten to be more robust. Differential Revision: https://phab.mercurial-scm.org/D21

phabricator: remove an unnecessary writediffproperties call This was introduced by D229. Thanks Yuya for finding it! Differential Revision: https://phab.mercurial-scm.org/D366

releasenotes: minor bug fix for index error while serializing Differential Revision: https://phab.mercurial-scm.org/D356

label: rename log.trouble into log.instability The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D259

label: rename trouble.X into instability.X The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D258

label: rename changeset.troubled into changeset.unstable The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D257

test: add a test for stabilization related labels Differential Revision: https://phab.mercurial-scm.org/D256

obsolete: rename bumped volatile set into phasedivergent volatile set The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D255

obsolete: rename divergent volatile set into contentdivergent volatile set The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D254

obsolete: rename unstable volatile set into orphan volatile set The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D253

revset: rename bumped into phasedivergent Don't touch bumped volatile set name, only the revset name. The volatile set name will be updated in a later patch. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D252

revset: remane divergent into contentdivergent Don't touch divergent volatile set name, only the revset name. The volatile set name will be updated in a later patch. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D251

revset: rename unstable into orphan Don't touch unstable volatile set name, only the revset name. The volatile set name will be updated in a later patch. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D250

test: update evolution config evolution* config has been rewritten in stabilization* in the previous patch, update tests file to use the new names. Differential Revision: https://phab.mercurial-scm.org/D249

config: rename evolution config into stabilization Use aliases for backward-compatibility. Though I'm not sure how to emit compatibility warnings with aliases. Test configuration are updated in the next patch. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D248

morestatus: move fb extension to core by plugging to `hg status --verbose` morestatus extension in fbext use to show more context about the state of the repo like the repository is in a unfinished merge state, or a rebase is going on, or histedit is going on, listing the files which need to be resolved and also suggesting ways to handle the situation. This patch moves the extension directly to core by plugging it into the --verbose flag of the status command. So now if you are in any unfinished state and you do hg status -v, it will show you details and help related to the state. The extension in fbext also shows context about unfinished update state which is not ported to core as that plug in hooks to update command which need to be tackled somewhat differently. The following configuration will turn the behaviour on by default [commands] status.verbose = 1 You can also skip considering some states like bisect as follows: [commands] status.skipstates=bisect This patch also adds test for the feature. .. feature:: ``hg status -v`` can now show unfinished state. For example, when in an unfinished rebase state, ``hg status -v`` might show:: # The repository is in an unfinished *rebase* state. # No unresolved merge conflicts. # To continue: hg rebase --continue # To abort: hg rebase --abort Differential Revision: https://phab.mercurial-scm.org/D219

bundle2: fix transaction availability detection Changeset 5fc4ddfbe626 introduce more complex logic around 'bundleoperation.gettransaction'. In that process it turns the old "attribute" into a proper method which breaks the code that detects the "transaction availability". The change was visible in 'test-acl.t', fixing this reverts the test changes. Differential Revision: https://phab.mercurial-scm.org/D303

fsmonitor: correct an error message Without the change, the error looks like: warning: Watchman unavailable: "watchman" executable not in PATH (%s), while executing [Errno 2] No such file or directory With the change, it now looks like: warning: Watchman unavailable: "watchman" executable not in PATH ([Errno 2] No such file or directory) Differential Revision: https://phab.mercurial-scm.org/D322

sshpeer: make instance attributes and methods internal Peer types are supposed to conform to a formal interface defined by peer.peerrepository and wireproto.wirepeer. Every "public" attribute on *peer types makes it harder to understand what attributes are part of the interface and what are instance specific. This commit converts a number of "public" instance attributes and methods on sshpeer to internal so they can't be confused to be part of the peer API. The URL-related instance attributes were introduced in 876333a295ff in 2005. AFAICT most of them aren't used and could potentially be removed. But I kept them around anyway. I also reorded some code to make things slightly easier to read. .. api:: Rename attributes on sshpeer to reflect peer API Differential Revision: https://phab.mercurial-scm.org/D331

peer: remove non iterating batcher (API) The last use of this API was removed in b6e71f8af5b8 in 2016. While not formally deprecated, as of the last commit the code is no longer explicitly tested. I think the new API has existed long enough for people to transition to it. I also have plans to more formalize the peer API and removing batch() makes that work easier. I'm not convinced the current client-side API around batching is great. But it's the best we have at the moment. .. api:: remove peer.batch() Replace with peer.iterbatch(). Differential Revision: https://phab.mercurial-scm.org/D320

wireproto: overhaul iterating batcher code (API) The remote batching code is difficult to read. Let's improve it. As part of the refactor, the future returned by method calls on batchiter() instances is now populated. However, you still need to consume the results() generator for the future to be set. But at least now we can stuff the future somewhere and not have to worry about aligning method call order with result order since you can use a future to hold the result. Also as part of the change, we now verify that @batchable generators yield exactly 2 values. In other words, we enforce their API. The non-iter batcher has been unused since b6e71f8af5b8. And to my surprise we had no explicit unit test coverage of it! test-batching.py has been overhauled to use the iterating batcher. Since the iterating batcher doesn't allow non-batchable method calls nor local calls, tests have been updated to reflect reality. The iterating batcher has been used for multiple releases apparently without major issue. So this shouldn't cause alarm. .. api:: @peer.batchable functions must now yield exactly 2 values Differential Revision: https://phab.mercurial-scm.org/D319

wireproto: remove support for local results in @batchable (API) @peer.batchable decorated generator functions have two forms: yield value, None and yield args, future yield value These forms have been present since the decorator was introduced. There are currently no in-repo consumers of the first form. So this commit removes support for it. Note that remoteiterbatcher.submit() asserts the 2nd form. And b6e71f8af5b8 removed the last user of remotebatcher, forcing everyone to remoteiterbatcher. So anything relying on this in the wild would have been broken since b6e71f8af5b8. .. api:: @peer.batchable can no longer emit local values Differential Revision: https://phab.mercurial-scm.org/D318

wireproto: properly implement batchable checking remoteiterbatcher (unlike remotebatcher) only supports batchable commands. This claim can be validated by comparing their implementations of submit() and noting how remoteiterbatcher assumes the invoked method has a "batchable" attribute, which is set by @peer.batchable. remoteiterbatcher has a custom __getitem__ that was trying to validate that only batchable methods are called. However, it was only validating that the called method exists, not that it is batchable. This wasn't a big deal since remoteiterbatcher.submit() would raise an AttributeError attempting to `mtd.batchable(...)`. Let's fix the check and convert it to ProgrammingError, which may not have been around when this was originally implemented. Differential Revision: https://phab.mercurial-scm.org/D317

largefiles: remove remotestore.batch() This method was added in 9e1616307c4c. AFAICT it didn't do anything at inception. If it did, there was no test coverage for it because changing it to raise doesn't fail any tests at that revision. b6e71f8af5b8 later refactored all remote.batch() calls to remote.iterbatch(). So if this was somehow used, it isn't called any more because there are no calls to .batch() remaining in the repo. I suspect the original patch author got confused by the distinction between the peer/remote interface and the largefiles store. The lf store is a gateway to a peer instance. It exposes additional lf-specific methods to execute against a peer. However, it is not a peer and doesn't need to implement batch() because peer itself does that. Differential Revision: https://phab.mercurial-scm.org/D316

histedit: check first changeset for verb "roll" or "fold" (issue5498) If someone changes "pick" to "roll" or "fold" for the first changeset in a histedit rule Mercurial could remove a wrong changeset if the phase is non-public. roll or fold for the first changeset should be invalid.

encoding: drop circular import by proxying through '<policy>.charencode' I decided not to split charencode.c to new C extension module because it would duplicate binary codes unnecessarily.

policy: reroute proxy modules internally This allows us to split encoding functions from pure.parsers without doing that for cext.parsers. See the next patch for why.

cext: modernize charencode.c to use Py_ssize_t

cext: factor out header for charencode.c This merges a part of util.h with the header which should exist for charencode.c.

cext: split character encoding functions to new compilation unit This extracts charencode.c from parsers.c, which seems big enough for me to hesitate to add new JSON functions. Still charencode.o is linked to parsers.so to avoid duplication of binary codes.

cext: move _dict_new_presized() to header Prepares for splitting encoding functions from parsers.c.

ui: restore behavior to ignore some I/O errors (issue5658) e9646ff34d55 and 1bfb9a63b98e refactored ui methods to no longer silently swallow some IOError instances. This is arguably the correct thing to do. However, it had the unfortunate side-effect of causing StdioError to bubble up to sensitive code like transaction aborts, leading to an uncaught exceptions and failures to e.g. roll back a transaction. This could occur when a remote HTTP or SSH client connection dropped. The new behavior is resulting in semi-frequent "abandonded transaction" errors on multiple high-volume repositories at Mozilla. This commit effectively reverts e9646ff34d55 and 1bfb9a63b98e to restore the old behavior. I agree with the principle that I/O errors shouldn't be ignored. That makes this change... unfortunate. However, our hands are tied for what to do on stable. I think the proper solution is for the ui's behavior to be configurable (possibly via a context manager). During critical sections like transaction rollback and abort, it should be possible to suppress errors. But this feature would not be appropriate on stable.

tests: test behavior of IOError during transactions (issue5658) ui._write(), ui._write_err(), and ui.flush() all trap IOError and re-raise as error.StdioError. If a caller doesn't catch StdioError when writing to stdio, it could bubble all the way to dispatch. This commit adds tests for I/O failures around various transaction operations. The most notable badness is during abort. Here, an uncaught StdioError will result in incomplete transaction rollback, requiring an `hg rollback` to recover. This can result in a client "corrupting" a remote repo via terminated HTTP and SSH socket.

log: mention ui.logtemplate in the help text

hg: avoid relying on errno numbers / descriptions A few tests hardcode errno numbers and/or descriptions in the output, causing test failures on platforms where these values are different. Differential Revision: https://phab.mercurial-scm.org/D362

hg: tolerate long vs. int in test-context.py The file times here can be longs instead of ints on some platforms, which will cause a test failure due to these printing with an L suffix; instead always format with %d which will produce the same output in either case. Differential Revision: https://phab.mercurial-scm.org/D361

configitems: register the 'notify.test' config

configitems: register the 'notify.template' config

configitems: register the 'notify.style' config

configitems: register the 'notify.strip' config

configitems: register the 'notify.sources' config

configitems: register the 'notify.merge' config

configitems: register the 'notify.mbox' config

configitems: register the 'notify.maxsubject' config

configitems: register the 'notify.maxdiff' config

configitems: register the 'notify.fromauthor' config

configitems: register the 'notify.domain' config

configitems: register the 'notify.diffstat' config

configitems: register the 'notify.config' config

merge with stable

merge with stable

obsutil: rename allprecursors into allpredecessors Use util.nouideprecwarn because obsstore doesn't have easy access to an ui object. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D247

obsolete: rename precursor into predecessor in obsolete docstrings The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D246

obsstore: rename precursors into predecessors Use util.nouideprecwarn because obsstore doesn't have easy access to an ui object. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D245

obsolete: rename _addprecursors into _addpredecessors Use util.nouideprecwarn because _addpredecessors doesn't have easy access to an ui object. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D244

obsmarker: rename precnode into prednode Rename prednode (predecessors node) into precnode (precursors node) in markers class. Use util.nouideprecwarn because markers doesn't have easy access to an ui object. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D243

context: rename troubled into isunstable As we changed the meaning of unstable between the old vocabulary and the new one, we can't reuse the unstable method name at the risk of breaking extensions calling unstable and getting a wrong result. Instead rename troubled into isunstable so extensions will continue to work. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D242

context: rename bumped into phasedivergent Rename bumped context method into phasedivergent and add a deprecation warning on bumped. Only update all callers to keep the patch straightforward. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D241

context: rename divergent into contentdivergent Rename divergent context method into contentdivergent and add a deprecation warning on divergent. Only update all callers to keep the patch straightforward. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D240

context: rename unstable into orphan Rename unstable context method into orphan and add a deprecation warning on unstable. Only update all callers to keep the patch straightforward. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D239

context: rename troubles into instabilities Rename troubles context method into instabilities. Copy the old troubles method and add a deprecation warning. This way extensions calling troubles will see the deprecation warning but will not break due to new return values. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D238

repair: move manifest strip to a separate function This moves manifest stripping to a separate function so implementations of the manifest that don't support stripping can replace this function with a no-op. I considered adding a strip api to the manifestlog, so other implementations could make it a no-op there, but it seems like strip might be unique to the revlog implementation, and therefore shouldn't be present on the generic api. Differential Revision: https://phab.mercurial-scm.org/D292

repair: refactor broken linkrev collection This refactors broken linkrev collection such that manifest collection is in a separate function. This allows extensions to replace the manifest collection with a non-revlog oriented version. I considered moving the collect changes function onto the manifestlog itself, so it would be behind the abstraction, but since the store we're building doesn't even have the concept of strip, embeding that concept in the manifestlog api seemed odd. Differential Revision: https://phab.mercurial-scm.org/D291

tests: update duplicateoptions test to use bytestrings everywhere Test now passes on Python 3. Differential Revision: https://phab.mercurial-scm.org/D297

extensions: attempt to use non-deprecated inspect method Avoids some deprecation warnings when extension loading breaks. Differential Revision: https://phab.mercurial-scm.org/D295

tests: clean up many print statements to be print functions instead Differential Revision: https://phab.mercurial-scm.org/D294

tests: fix simple heredoc print statements to work on Py3 There are a handful left that require some more care. Differential Revision: https://phab.mercurial-scm.org/D293

obsolete: use bytes() instead of str() so the node is bytes on py3 I'm not sure this is right, since this should either be bytes or str to match what's going on in the revlog layer. Differential Revision: https://phab.mercurial-scm.org/D271

tests: fix test-notify.t to use $PYTHON In doing so, I consolidated the filtering logic. This resulted in some small test output changes, but I think the consistency throughout the test is worth it. Differential Revision: https://phab.mercurial-scm.org/D279

tests: make test-highlight code portable to python3 This is easier than trying to do some sort of check-code shenanigans. Differential Revision: https://phab.mercurial-scm.org/D278

dagparser: make print statement in doctest Py3 portable Differential Revision: https://phab.mercurial-scm.org/D277

i18n: fix check-translation.py to be less broken on Python 3 These are all simple one-argument print statements, so this syntax works the same way in 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D276

ui: refactor extractchoices so it doesn't break on Python 3 Differential Revision: https://phab.mercurial-scm.org/D275

obsutil: defend against succsmarkers() returning None I'm not sure if this is a realistic problem, but doing this avoids some pretty awful test failures on Python 3, and it looks like it should be harmless. Differential Revision: https://phab.mercurial-scm.org/D274

changegroup: more **kwargs Differential Revision: https://phab.mercurial-scm.org/D273

bundle2: convert ints to strings using pycompat.bytestring() Fixes some Python 3 regressions. We don't use %d here because the part id is actually an Optional[int]. It should always be initialized to a non-None value by the time this code executes, but we shouldn't blindly depend on that being the case. Differential Revision: https://phab.mercurial-scm.org/D272

tests: fix up test-run-tests failures on Python 3.6 Some of the recent work on run-tests.py didn't work on Python 3. This fixes the regressions. Differential Revision: https://phab.mercurial-scm.org/D270

template: rename troubles templatekw into instabilities Rename troubles template keyword into instabilities and add a deprecation warning on templatekw. Update default mapfile and test files to use the new template keyword. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D237

test-rebase: add a brute force test Rebase is becoming more complex and it looks like a good idea to try some brute force enumeration to cover cases that are hard to find manually. Using brute force to generate repos in different shapes and enumerating the rebase source and destination would generate too many cases that takes too long to compute. This patch limits the "brute force" to only the "rebase source" part. Repo and destination are still manual. The added test cases are crafted manually to reveal some behaviors that are not covered by other tests: - "revlog index out of range" crash - after rebase, p1 == p2, p2 != null - "nothing to merge" abort In the future we might want to add more tests here. For now I'm more interested in revealing interesting behaviors in a minified way. I tried some more complex cases but didn't find other interesting behaviors. Differential Revision: https://phab.mercurial-scm.org/D262

dirstate: simplify dirstate's __iter__ Probably also a performance win, but not measurable in perfdirstate. Differential Revision: https://phab.mercurial-scm.org/D269

repo: skip invalidation of changelog if it has 'delayed' changes (API) The changelog object can store recently added revisions in memory until the transaction is committed. We don't want to lose those changes even if repo.invalidate(clearfilecache=True), so let's skip the changelog when it has such 'delayed' changes. Differential Revision: https://phab.mercurial-scm.org/D152

httppeer: make several instance attributes internal (API) Peer instances are supposed to conform to a well-defined API so consumers can be agnostic about the underlying peer type. To reinforce this, this commit renames a handful of instance attributes on httpeer so they no longer have a "public" name. Differential Revision: https://phab.mercurial-scm.org/D268

httppeer: remove unused handler attribute The consumer of this attribute was removed by b6bd4ee6ed85 in 2008. Differential Revision: https://phab.mercurial-scm.org/D267

localrepo: remove unused requirements attributes on localpeer (API) The previous changeset removed the last consumer of requirements. I'm not sure when supportedformats became unused. But I couldn't find any obvious instances where it is being used. It likely stems from peers being derived from repository instances several years ago and is a holdover from that day. Differential Revision: https://phab.mercurial-scm.org/D266

exchange: access requirements on repo instead of peer As part of formalizing the peer interface, I audited for attribute accesses for non-internal names to find API violations. This uncovered the code changed in this commit. localpeer.requirements is just an alias to the repo's requirements attribute. So, change the code to get the data from the source instead of relying on a one-off attribute in the localpeer type. Differential Revision: https://phab.mercurial-scm.org/D265

exchange: drop support for lock-based unbundling (BC) Locking over the wire protocol and the "addchangegroup" wire protocol command has been deprecated since e8c4f3d3df8c, which was first part of Mercurial 0.9.1. Support for handling these commands from sshserver was dropped in 9f6e0e7ef828 in 2015, effectively locking out pre 0.9.1 clients from new servers. However, client-side code for calling lock and addchangegroup is still present in exchange.py and the various peer classes to facilitate pushing to pre 0.9.1 servers. The lock-based pushing mechanism is extremely brittle. 0.9.1 was released in July 2006 and I highly doubt anyone is still running such an ancient version of Mercurial on a server. I'm about to refactor the peer API and I don't think it is worth keeping support for this ancient protocol feature. So, this commit removes client support for the lock-based pushing mechanism. This means modern clients will no longer be able to push to pre 0.9.1 servers. Differential Revision: https://phab.mercurial-scm.org/D264

http: add a test of actually pushing with httppostargs This was previously untested. Sigh. Differential Revision: https://phab.mercurial-scm.org/D230

filemerge: move decorator definition for internal merge tools to registrar This patch also adds extra loading entry for internal merge tools to extensions.py, for similarity to other decorators defined in registrar.py. This patch uses "internalmerge" for decorator class name, instead of original "internaltool", because the latter is too generic. BTW, after this patch, 4-spaces indentation is added to the 1st line of internal merge tool description docstring, and this may make already translated entries in *.po fuzzy. Even though this indentation is required for "definition list" in reST syntax, absence of it has been overlooked, because help.makeitemsdoc() forcibly inserts it at generation of online help. But this forcible insertion causes formatting issue (I'll send another patch series for this). Therefore, this additional indentation should be reasonable.

releasenotes: add similarity check function to compare incoming notes It is possible that the incoming note fragments have some similar content as the existing release notes. In case of a bug fix, we match for issueNNNN in the existing notes. For other general cases, it makes use of fuzzywuzzy library to get a similarity score. If the score is above a certain threshold, we ignore the fragment, otherwise add it. But the score might be misleading for small commit messages. So, it uses similarity function only if the length of string (in words) is above a certain value. The patch adds tests related to its usage. But it needs improvement in the sense of combining incoming notes. We can use interactive mode for adding notes. Maybe we can do this if similarity is under a certain range.

releasenotes: add import check for fuzzywuzzy This patch adds the has_fuzzywuzzy for import check of external dependency fuzzywuzzy.

run-tests: don't drop optional lines after a missing unconditional line The previous behavior was to bail out when hitting 'awol', so 'missing (?)' was also dropped.

run-tests: drop required (feature !) style lines when the output is missing Essentially, these were acting as a verbose (?) flag, since they weren't being dropped when required. Foozy has a nice description [1]. Basically, a couple more places needed to check the features before treating it as optional. I don't like how test-run-tests.py had to be hacked, but _hghave() can't be made a static method. The test change was a change while developing `debugssl`, prior to tightening up the cases where the message is printed, that this fix would have caught. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/101941.html

status: avoid recursing into ignored directory with "--terse u" Let "isignoreddir" function first check that supplied directory is itself ignored before walking recursively into its content. Otherwise, the command is awfully slow when one has an ignored directory with a lot of content. Update and rephrase function docstring accordingly.

pushvars: move fb extension pushvars to core pushvars extension in fbext adds a --pushvars flag to push command using which one send strings to server which becomes environment variables there prepended with HG_USERVAR_. These variables can then be used to run hooks on the server. The extension is moved directly to core and unbundling of the strings and converting them to environment variables at server is disabled by default for security reasons. One can turn that on by following config: [push] pushvars.server = true This patch also adds the test for the extension. Differential Revision: https://phab.mercurial-scm.org/D210

phabricator: update diff property even if we choose not to create a new diff The diff property contains metadata like "HG Node". Previously we skip uploading a new diff if we are sure that the old patch and new patch have a same content. That has issues when a pusher adds an obsmarker using the old "HG Node" stored in the old diff. This patch adds logic to update the diff property so "HG Node" gets updated to prevent that issue. Differential Revision: https://phab.mercurial-scm.org/D229

phabricator: use Phabricator's last node information This makes it more strict when checking whether or not we should update a Differential Revision. For example, a) Alice updates D1 to content 1. b) Bob updates D1 to content 2. c) Alice tries to update D1 to content 1. Previously, `c)` will do nothing because `phabsend` detects the patch is not changed. A more correct behavior is to override Bob's update here, hence the patch. This also makes it possible to return a reaonsable "last node" when there is no tags but only `Differential Revision` commit messages. Test Plan: ``` for i in A B C; do echo $i > $i; hg ci -m $i -A $i; done hg phabsend 0:: # D40: created # D41: created # D42: created echo 3 >> C; hg amend; hg phabsend . # D42: updated hg tag --local --hidden -r 2 -f D42 # move tag to the previous version hg phabsend . # D42: skipped (previously it would be "updated") rm -rf .hg; hg init hg phabread --stack D42 | hg import - hg phabsend . # D42: updated hg tag --local --remove D42 hg commit --amend hg phabsend . # D42: updated (no new diff uploaded, previously it will upload a new diff) ``` The old diff object is now returned, which could be useful in the next patch. Differential Revision: https://phab.mercurial-scm.org/D121

phabricator: add --confirm option to phabsend command This adds a --confirm flag similar to the confirm flag of `hg email` using which one can confirm the changesets before they get emailed. The confirm flag will show the changesets and ask for confirmation before sending them. Differential Revision: https://phab.mercurial-scm.org/D218

evolution: rename bumped to phase-divergent Rename bumped to phase-divergent in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D216

evolution: rename divergent to content-divergent Rename divergent to content-divergent in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D215

py3: use bytes IO to write sample hgrc Unicode sucks. Stop using Text IO and manually convert line endings.

buildrpm: do not break in presence of custom user configs For example, if "hg log" was defined as an alias: # /etc/mercurial/hgrc [alias] log = log --graph the buildrpm script would be surprised by log messages formatted in unexpected ways, and bail out. This patch sets HGPLAIN, effectively resetting all the user configs, including log output, to a common state, making the build more predictable across all the possible environments.

sparse: treat paths as cwd-relative This commit makes it so sparse treats passed paths as CWD-relative, not repo-root-realive. This is a more intuitive behavior in my (and some other FB people's) opinion. This is breaking change however. My hope here is that since sparse is experimental, it's ok to introduce BCs. The reason (glob)s are needed in the test is this: in these two cases we do not supply path together with slashes, but `os.path.join` adds them, which means that under Windows they can be backslashes. To demonstrate this behavior, one could remove the (glob)s and run `./run-tests.py test-sparse.t` from MinGW's terminal on Windows.

match: expose some data and functionality to other modules This patch makes sure that other modules can check whether patterns are CWD-relative.

sparse: properly error out when absolute paths are used Current logic is misleading (it says it drops only absolute paths, but it actually drops all of them), not cross-platform (does not support Windows) and IMO just wrong (as it should just error out if absolute paths are given). This commit fixes it.

py3: convert arbitrary exception object to byte string more reliably Our exception types implement __bytes__(), which should be tried first. Do lossy encoding conversion as a last resort.

build: delay version computation on macOS builds The way HGVER is evaluated now, it'll be evaluated at the beginning of the make execution - with this change, it's evaluated when it gets to that command, at which point the version file it's looking for is sure to exist and be up-to-date. Differential Revision: https://phab.mercurial-scm.org/D224

color: remove warnings if term is not formatted (==dumb or !ui.formatted()) If the user sets color.mode=terminfo, and then runs in the shell inside of emacs (so TERM=dumb), the previous behavior was that it would warn about no terminfo entry for setab/setaf, and then warn about 'failed to set color mode to terminfo'. The first warning is silenced by carrying 'formatted' through to _terminfosetup, the second is silenced by using 'formatted' instead of ui.formatted(). If --color=on (or ui.color=always) is specified, this will still warn, since the formatted boolean is set to true in these cases. Differential Revision: https://phab.mercurial-scm.org/D223

rpms: add chg I'm not sure if there's a reason chg is not added by default. If not, I would like to propose adding in this patch. Differential Revision: https://phab.mercurial-scm.org/D220

tests: demonstrate crash when trying to rebase merge without its parents As the test case shows, when "hg rebase -d G -r 'B + D + F'" is run on the following graph, we crash with traceback. It's reasonable to fail because we can not easily produce a correct rebased F. The problem is what diff to apply to either the rebased B or the rebased D. We could potentially produce the result by e.g. applying the (F-D) diff to the rebased B and then applying the reverse (E-D) diff on top, but that could result in merge conflicts in each of those steps, which we don't have a way of dealing with. So for now, let's just add a test case to demonstrate that we crash (i.e. the AssertionError is clearly incorrect since the user can run into it). F /| C E | | B D G \|/ A Differential Revision: https://phab.mercurial-scm.org/D212

tests: demonstrate broken rebase of merge with p1's successor in dest The fix in 8ede973597fd (rebase: handle successor targets (issue5198), 2016-04-11) only fixed the case where p2's successor was in the destination, and only when the successor was exactly the destination (i.e. not when the successor was an ancestor of it). This patch adds a test case for when p1's successor is in the destination. It adds another one for when the successor is an ancestor of the destination. To do that simply, it also rewrites the test case using drawdag. Differential Revision: https://phab.mercurial-scm.org/D211

bundle2: obtain repr() of exception in a python3-safe way This was exposed by other problems in bundle generation, but I'm not sure how to test it for now.

bundle2: use bytestr() instead of str() to convert part id to bytes This was exposed by trying to run previously-passing Python 3 tests.

bundle2: work around zip() being lazy in Python 3

bundle2: look for __next__ as well as next to identify iterators In Python 3, next is called __next__ and this was failing to catch some iterators.

bundle2: use modern Python division This was failing on Python 3 because the / was returning a float, which was then making the __mul__ on a bytes sad.

changegroup: wrap some ** expansions in strkwargs

obsolete: reuse _fm1metapair for computing _fm1metapairsize It's evaluated at import time, so it seems silly to not reuse the constant name.

setup: fix installing in a mingw environment The addition, in 9a4adc76c88a, of a hack for the MSVC compiler class was overwriting the original class for the Mingw32CCompiler class, leading to an error when the HackedMingw32CCompiler is instantiated. Differential Revision: https://phab.mercurial-scm.org/D329

osx: delay version computation on macOS builds Before this patch, HGVER would be evaluated at the beginning of the make execution, and would be unset because build/mercurial/ doesn't exist yet at that point. Now we compute the version after the `make install` run has completed. This is backported to stable from 8626b44516c1, but that revision had an error in the shell invocation syntax.

evolution: rename unstable to orphan Rename unstable to orphan in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D214

evolution: rename trouble(s) to instability Rename trouble(s) to instability in all external user-facing output. Only update user-facing output for the moment, variables names, templates keyword and potentially configuration would be done in later series. The renaming is done according to https://www.mercurial-scm.org/wiki/CEDVocabulary. Differential Revision: https://phab.mercurial-scm.org/D213

bundle2: load hookargs from bundleoperation into transaction when started one When a transaction is started, we must load the hookargs from the bundleoperation object to the transaction so that they can be used in the transaction. Also this patch makes sure no more hookargs are added to the bundleoperation object once the transaction starts. This is a part of porting fb extension bundle2hooks to core. Differential Revision: https://phab.mercurial-scm.org/D209

bundle2: add the capability to store hookargs on bundle operation object There are extensions like pushrebase, pushvars which run hooks on a server before taking the lock. Since the lock is not taken, transaction is not there, so the hookargs can't be stored on the transaction. Adding hooksargs to bundle operation object will help in running hooks before taking the lock. This is a part of moving fb's extension bundle2hooks to core. Differential Revision: https://phab.mercurial-scm.org/D208

branchmap: revert c34532365b38 for Python 2.7 compatibility Old versions of python 2.7 don't like that the second argument to struct.unpack_from is a bytearray, so the change removing the util.buffer around that argument in branchmap broke running on older versions of python 2.7. Differential Revision: https://phab.mercurial-scm.org/D330

Added signature for changeset 3fee7f7d2da0

Added tag 4.3.1 for changeset 3fee7f7d2da0

ssh: unban the use of pipe character in user@host:port string This vulnerability was fixed by the previous patch and there were more ways to exploit than using '|shellcmd'. So it doesn't make sense to reject only pipe character. Test cases are updated to actually try to exploit the bug. As the SSH bridge of git/svn subrepos are not managed by our code, the tests for non-hg subrepos are just removed. This may be folded into the original patches.

ssh: quote parameters using shellquote (SEC) This patch uses shellquote to quote ssh parameters more strictly to avoid shell injection.

subrepo: add tests for git rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for svn rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for hg rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

push: add tests for unsafe ssh url (SEC)

pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

stable: merge heads

Added signature for changeset 943c91326b23

Added tag 4.2.3 for changeset 943c91326b23

ssh: unban the use of pipe character in user@host:port string This vulnerability was fixed by the previous patch and there were more ways to exploit than using '|shellcmd'. So it doesn't make sense to reject only pipe character. Test cases are updated to actually try to exploit the bug. As the SSH bridge of git/svn subrepos are not managed by our code, the tests for non-hg subrepos are just removed. This may be folded into the original patches.

ssh: quote parameters using shellquote (SEC) This patch uses shellquote to quote ssh parameters more strictly to avoid shell injection.

subrepo: add tests for git rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for svn rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

subrepo: add tests for hg rogue ssh urls (SEC) 'ssh://' has an exploit that will pass the url blindly to the ssh command, allowing a malicious person to have a subrepo with '-oProxyCommand' which could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' is able to execute arbitrary commands. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

push: add tests for unsafe ssh url (SEC)

pull: add tests for unsafe ssh url (SEC)

clone: add tests for unsafe ssh url (SEC)

sshpeer: check for safe ssh url (SEC) Checking in the sshpeer for a rogue ssh:// urls seems like the right place to do it (instead of whack-a-mole with pull, clone, push, etc).

ssh: ban any username@host or host that starts with - (SEC) This paranoia probably isn't required, but it can't hurt either.

util: add utility method to check for bad ssh urls (SEC) Our use of SSH has an exploit that will parse the first part of an url blindly as a hostname. Prior to this set of security patches, a url with '-oProxyCommand' could run arbitrary code on a user's machine. In addition, at least on Windows, a pipe '|' can be abused to execute arbitrary commands in a similar fashion. We defend against this by checking ssh:// URLs and looking for a hostname that starts with a - or contains a |. When this happens, let's throw a big abort into the user's face so that they can inspect what's going on.

pathauditor: disable cache of audited paths by default (issue5628) The initial attempt was to discard cache when appropriate, but it appears to be error prone. We had to carefully inspect all places where audit() is called e.g. without actually updating filesystem, before removing files and directories, etc. So, this patch disables the cache of audited paths by default, and enables it only for the following cases: - short-lived auditor objects - repo.vfs, repo.svfs, and repo.cachevfs, which are managed directories and considered sort of append-only (a file/directory would never be replaced with a symlink) There would be more cacheable vfs objects (e.g. mq.queue.opener), but I decided not to inspect all of them in this patch. We can make them cached later. Benchmark result: - using old clone of http://selenic.com/repo/linux-2.6/ (38319 files) - on tmpfs - run HGRCPATH=/dev/null hg up -q --time tip && hg up -q null - try 4 times and take the last three results original: real 7.480 secs (user 1.140+22.760 sys 0.150+1.690) real 8.010 secs (user 1.070+22.280 sys 0.170+2.120) real 7.470 secs (user 1.120+22.390 sys 0.120+1.910) clearcache (the other series): real 7.680 secs (user 1.120+23.420 sys 0.140+1.970) real 7.670 secs (user 1.110+23.620 sys 0.130+1.810) real 7.740 secs (user 1.090+23.510 sys 0.160+1.940) enable cache only for vfs and svfs (this series): real 8.730 secs (user 1.500+25.190 sys 0.260+2.260) real 8.750 secs (user 1.490+25.170 sys 0.250+2.340) real 9.010 secs (user 1.680+25.340 sys 0.280+2.540) remove cache function at all (for reference): real 9.620 secs (user 1.440+27.120 sys 0.250+2.980) real 9.420 secs (user 1.400+26.940 sys 0.320+3.130) real 9.760 secs (user 1.530+27.270 sys 0.250+2.970)

tests: show cache of audited paths is never invalidated

Added signature for changeset 5544af862286

Added tag 4.3 for changeset 5544af862286

chg: define _GNU_SOURCE to allow CentOS 5 compilation Without this flag, compilation fails with: hgclient.c: In function 'hgc_open': hgclient.c:466: error: 'O_DIRECTORY' undeclared (first use in this function) hgclient.c:466: error: (Each undeclared identifier is reported only once hgclient.c:466: error: for each function it appears in.) Differential Revision: https://phab.mercurial-scm.org/D260

centos5: use vault urls CentOS 5 has reached end-of-life. A working build for it is still possible using 'vault.centos.org'. Differential Revision: https://phab.mercurial-scm.org/D261

ui: make sure buffer is flushed before waiting for user input (issue5587) Without this patch on Windows 'hg ci -i' hangs waiting for user input and "examine changes to 'file'? [Ynesfdaq?]" is never displayed (at least if the diff is sufficiently small). When Ctrl+C is pressed, this prompt becomes visible, which suggests that the buffer just wasn't flushed. I've never seen this happening on Linux, but this looks harmless enough to not platform-gate it.

util: fix sortdict.update() to call __setitem__() on PyPy (issue5639) It appears that overriding __setitem__() doesn't work as documented on PyPy. Let's patch it as before bd0fd3ff9916. https://docs.python.org/2/library/collections.html#ordereddict-examples-and-recipes The issue was ui.configitems() wasn't ordered correctly, so the pull command was wrapped in different order.

test-revlog-raw: close file handles explicitly (issue5644)

tests: clear __pycache__ for PyPy compatibility (issue5638) (issue5642)

test-extension: glob module name in error message for PyPy

rebase: sort collapsed revisions in commit message (issue5643) This fixes the test instability.

i18n: use actual filename, in which function is defined, for hg.pot Before this patch, source filename for msgid in hg.pot file becomes incorrect, if a function is defined in file A, but detected in dict in file B, For example, almost all debug* commands are defined in debugcommands.py, but hggettext detects them in "table" of commands.py. Therefore, docstring fragments of debug* commands are marked as "defined in commands.py" in hg.pot file. This is serious problem for translation, because the cost to find out original location of texts increases very much.

ui: enable pager always for explicit --pager=on (issue5580) Before this patch, explicit --pager=on is unintentionally ignored by any disabling factor, even if priority of it is less than --pager=on (e.g. "[ui] paginate = off").

rebase: use one dirstateguard for when using rebase.singletransaction This was previously landed as 2519994d25ca but backed out in b63351f6a2 because it broke hooks mid-rebase and caused conflict resolution data loss in the event of unexpected exceptions. This new version adds the behavior back but behind a config flag, since the performance improvement is notable in large repositories. The old commit message was: Recently we switched rebases to run the entire rebase inside a single transaction, which dramatically improved the speed of rebases in repos with large working copies. Let's also move the dirstate into a single dirstateguard to get the same benefits. This let's us avoid serializing the dirstate after each commit. In a large repo, rebasing 27 commits is sped up by about 20%. I believe the test changes are because us touching the dirstate gave the transaction something to actually rollback. (grafted from 9e3dc3a1638b9754b58a0cb26aaa75d868058109) (grafted from 7d38b41d2266d9a02a15c64229fae0da5738dcec) Differential Revision: https://phab.mercurial-scm.org/D135

README: make paragraphs before example code end with "::" for reST syntax After this patch, README as reST text can be rendered into HTML more appropriately. This is useful for hosting servers, which render README file as a part of index HTML of its directory (for example, Bitbucket).

commit: don't let failed commit with --addremove update dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D204

tests: demonstrate that failed "hg ci -A" updates dirstate (issue5645) Differential Revision: https://phab.mercurial-scm.org/D203

tests: fix test-flagprocessor.t flakiness The test for duplicate flag processors depended on the timestamps being set in the dirstate to work. If the time between the the previous failed commit (which would set the timestamp, due to bug 5645) and the attempted commit with the duplicate flag processors was small enough, it would fail. The failure was caused by a call to commands.status() early in the commit process. If the dirstate did not have the timestamp set, it would need to fetch the file content to compare with. Since two flag processors had been registered, it would attempted to base64 decode the contents twice, which would of course fail. This patch adds a "hg debugrebuilddirstate" to make it deterministic and also replaces the test case's "hg commit" by simply "hg status", since that will trigger reading of the contents and thereby use of the flag processors as noted above. Differential Revision: https://phab.mercurial-scm.org/D202

tests: clarify that duplicate flag processors is not an error The flag processors test for duplicate processors for a single flag was misleading because the file from the previous test case caused it to fail (making the "echo 'this should fail' > file" part irrelevant). Let's remove the leftover from the previous test case to make it clear that duplicate flag processors results only in a warning. Note that duplicate flag processors would have resulted in a failure (not just a warning) until ea1c2eb7abd3 (extensions: catch uisetup and extsetup failures and don't let them break hg, 2017-06-06). I remember expressing my concern about ending up with half-loaded extensions. It would be pretty unfortunate to have double-encoded revlog content enter a repo, so maybe we should reconsider? Differential Revision: https://phab.mercurial-scm.org/D201

tests: avoid unexpected result at invocation of *.py file on Windows Invocation of "diff tool.py" in test-extdiff.t tests whether shellquote() is applied on specified command as expected. But direct invocation of "*.py" file might cause unexpected result on Windows according to suffix binding. For example, starting IDE, showing dialog to choose program to be used, and so on. In such case, running test-extdiff.t is easily timed out. This patch uses intermediate *.bat file on Windows, to avoid such unexpected result. Naming that intermediate file as "diff tool.bat" is enough to test applying shellquote().

tests: use pdiff instead of diff for portability "diff" command might cause redundant message, "No differences encountered" on Solaris for example. But suppressing option like "-q" isn't portable, because POSIX specification doesn't define it. pdiff script was introduced by b59ef0c21405 to stabilize output of standard diff command on each platforms.

tests: make pdiff return appropriate exit code at comparison of files Before this patch, pdiff script returns 0, even if diff is detected. This issue doesn't cause failure of tests using it, if it is invoked via extdiff extension, because extdiff itself examines changes between specified revisions and decides exit code. BTW, this patch ignores recursive comparison case, because: - there is no portable way for current while-read based implementation to return 1 at detecting changes - it isn't yet needed to replace direct "diff -r" invocation by pdiff for portability

merge with stable

tests: make test-profile.t pass if statprof didn't collect samples We had the first 3 tests in test-profile.t fail because the output didn't match. I have not yet confirmed that this was because no samples were collected, but we shouldn't require samples to be collected for the test to pass either way. Differential Revision: https://phab.mercurial-scm.org/D199

pypy: fix failing test-devel-warnings.t with Pypy5.6.0 In Pypy 5.6.0, traceback exception classes are not displayed with their full qualified name. Instead of displaying mercurial.error.ProgrammingError, Pypy displays ProgrammingError. Update the test to support both version.

pypy: fix failing test files with Pypy5.6.0 Pypy 5.6.0 saves cached bytecode files in __pycache__ directory, clean them in tests to fix loading old test extensions code. Doing so should also helps for Python3.x migration.

commitextras: make sure keys are not empty Thanks to Yuya for catching this.

make: add new directory 'hgdemandimport' to cleanbutpackages rule This fixes the build process for ppa (at least in my simple check). Perhaps we should loop through all directory that aren't 'packages'?

statichttprepo: implement wlock() (issue5613) statichttprepo inherits from localrepository. In doing so, it obtains default implementations of various methods, like wlock(). Before this change, tags cache writing would call repo.wlock(). This failed on statichttprepo due to localrepository's wlock() looking for an instance attribute that doesn't exist on statichttprepo (statichttprepo doesn't call localrepository.__init__). We /could/ define missing attributes until the base wlock() works. However, a statichttprepo is remote and read-only and can't be locked. The class already has a lock() that short circuits. So it makes sense to implement a short-circuited wlock() as well. That is what this patch does. LockError is expected to be raised when locking fails. The constructor takes a number of arguments that are local repository centric. Rather than rework LockError to not require them (which would not be appropriate for stable), this commit populates dummy values. I don't believe they'll ever be seen by the user, as lock failures on static http repos should be limited to well-defined (and tested) scenarios. We can and should revisit the LockError type to improve this.

test: add tests for branches and tags with static http repo (issue5613) Mercurial currently fails to clone by revision on static http repos with tags. The added tests demonstrate this.

commitextras: make sure keys contains ascii letters, numbers, '_' and '-' only

phabricator: convert unicode to binary when writing patches This is a quick fix to make `hg phabread D189` work. It seems we might want to replace all `r''` to `u''`, and add more `encoding.*to*` to be more explicit when interacting with `json` module. Differential Revision: https://phab.mercurial-scm.org/D192

setup: silence warning of unknown option python_requires on distutils /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'

setup: forcibly include cext/pure packages in py2exe (issue5625) Since these modules are implicitly imported by policy.importmod(), py2exe can't track them statically. cffi modules are excluded for now because they wouldn't be useful in frozen (i.e. CPython) environment.

setup: do not select hg executable that prints unexpected warnings Otherwise the subsequent hg.run() would fail. This factors out the filtering function so the same rule should apply.

docker: pass proxy arguments to docker process Building in Docker when behind a proxy requires passing the proxy variables. Differential Revision: https://phab.mercurial-scm.org/D190

makefile: build on CentOS 6 with a Python 2.7 Differential Revision: https://phab.mercurial-scm.org/D189

osx: wire up genosxversion script The only version strings that are changed are the ones baked into the .pkg - hg's self-reported version string doesn't change, so users will still see our mostly-pip-compatible version strings. For reference, the part of our versioning setup that's not PEP440 compatible is the RC releases - those should be .rc0 insted of -rc. It's too late to change that for the 4.3 cycle, so I'll worry about fixing that during the 4.4 cycle.

osx: new script for generating OS X package versions If you're shipping prerelease or rc packages using Munki, you'll eventually discover that Munki's version comparison logic is not as good as pip's. In theory we should be able to fix Munki, but it seems entirely reasonable to produce version strings that sort reasonably under these conditions. Since the requried logic not brief, add a new script and some tests of that logic. A followup change will wire this into the Makefile.

tests: update mac packaging test expectations Some more files are included these days.

tests: add support in run-tests.py for an environment variable to stop color I don't like the color output, so this gives me a way to easily never see it while still having pygments on my machine.

setup: fix mistake that prevented Python 3 from being excluded My intent was to only allow Python 3 if the right environment variable is set (for when people want to use `pip install .` on hg locally). This fixes the bug in my previous change. I verified that `python3.6 run-tests.py` still passes the tests that passed before, and that all tests pass on 2.7 (including our virtualenv-using installation test). Differential Revision: https://phab.mercurial-scm.org/D185

rebase: move bookmark to destination for commits becoming empty (issue5627) When rebasing a changeset X and that changeset becomes empty, we should move the bookmark on X to rebase destination. This is a regression caused by the scmutil.cleanupnodes refactoring for rebase. The `adjustdest` function calculates the destination of bookmark movement. It was back-ported from https://phab.mercurial-scm.org/D21. It might be slightly more powerful than the minimal requirement to solve this issue. For example, it's impossible for a merge changeset to become empty while any of its ancestors does not become empty, but the code could handle that case. Since the code is reasonably short and clean, and helps the upcoming D21 series, I'd like to check-in `adjustdest` now. Thanks Martin von Zweigbergk for spotting corner cases (-k and descendant with bookmarks) in this area!

setup: drop use of set literal so parsable on Python 2.6 Otherwise friendly error wouldn't be displayed.

setup: add extra logic to try and recommend a new pip on bad Python Modern pip can detect supported Python versions (which we now declare), and pull down a reasonable release. This trick was suggested in http://bit.ly/pycon2017-build-bridges, and seems like a good defensive maneuver so that when we want to move to Python 3 it's less risky for existing users. This moves the version-check logic after defining our printf function so we can print more informative messages.

setup: explicitly declare supported Python versions I think we should probably backport this to 4.2 as well, and do one more release there that explicitly declares 2.6 support. That way anyone stuck on Python 2.6 will end up getting the right hg if they use a modern pip to install. Users can still use `python setup.py` incantations to attempt installing Mercurial on unsupported Pythons, including 3.5 and 3.6. A followup change will switch to only doing our own Python-version-check logic if we're not being installed by a reasonable pip.

contrib: remove one test from the python 3 whitelist 439b4d005b4a introduced a use of 'hg revert --no-backup' which is currently broken. Rather than try and do a fix on the (complicated) revert code during the freeze, let's just give up on this test under Python 3 until later. Once we do fix things, the test ratchet script will re-enable it.

ui: fix configbytes isinstance check to look for bytes and not str Fixes configbytes on Python 3.

patch: update copying of dict keys and values to work on Python 3

phabricator: include the suggested arc config in the repo Test Plan: I uploaded this revision with `arc diff` Differential Revision: https://phab.mercurial-scm.org/D171

match: override visitdir() in nevermatcher to return False When we changed basematcher.visitdir() in cf15c3cc304c (match: make base matcher return True for visitdir, 2017-07-14), we forgot to add an override in nevermatcher. This led to tests failing in narrowhg. As Durham pointed out, it's high time to add unit tests for the matcher, so this patch also adds a first unit test. Differential Revision: https://phab.mercurial-scm.org/D151

wix: update list of necessary pyd files

run-tests: pre instantiate pygments objects This speeds up run-tests.py diff output by 10x, which affects developer experience significantly. As demonstrated by the following test: ``` #require pygments $ for i in `seq 1 200`; do > echo ' $ echo '$i >> test-a.t > echo ' wrong' >> test-a.t > done $ cat > walltime.py <<EOF > from __future__ import absolute_import, print_function > import os, sys, time > t1 = time.time() > os.system(' '.join(sys.argv[1:]) + ' >/dev/null 2>/dev/null') > t2 = time.time() > print('%0.2f' % (t2 - t1)) > EOF $ $PYTHON walltime.py $TESTDIR/run-tests.py -l --color=never $TESTTMP/test-a.t 0.24 Before this patch: $ $PYTHON walltime.py $TESTDIR/run-tests.py -l --color=always $TESTTMP/test-a.t 2.46 After this patch: $ $PYTHON walltime.py $TESTDIR/run-tests.py -l --color=always $TESTTMP/test-a.t 0.25 ```

contrib: drop Python 2.6 from manylinux1 wheel script

contrib: make build-linux-wheels.sh executable

test-dirstate-race: hide irrelevant hg status output See the explanation for more. Differential Revision: https://phab.mercurial-scm.org/D153

test-commit: stabilize for filesystems without symlink support