revset: infer ordering flag to teach if operation should define/follow order New flag 'order' is the hint to determine if a function or operation can enforce its ordering requirement or take the ordering already defined. It will be used to fix a couple of ordering bugs, such as: a) 'x & (y | z)' disregards the order of 'x' (issue5100) b) 'x & y:z' is listed from 'y' to 'z' c) 'x & y' can be rewritten as 'y & x' if weight(x) > weight(y) (a) and (b) are bugs of the revset core. Before this, there was no way to tell if 'orset()' and 'rangeset()' can enforce its ordering. These bugs could be addressed by overriding __and__() of the initial set to take the ordering of the other set: class fullreposet: def __and__(self, other): # allow other to enforce its ordering return other but it would expose (c), which is a hidden bug of optimize(). So, in either ways, optimize() have to know the current ordering requirement. Otherwise, it couldn't rewrite expressions by weights with no output change, nor tell how a revset function or operation should order the entries. 'order' is tri-state. It starts with 'define', and shifts to 'follow' by 'x & y'. It changes back to 'define' on function call 'f(x)' or function-like operation 'x (f) y' because 'f' may have its own ordering requirement for 'x' and 'y'. The state 'any' will allow us to avoid extra cost that would be necessary to constrain ordering where it isn't important, 'not x'.

revset: wrap arguments of 'or' by 'list' node This makes the number of 'or' arguments deterministic so we can attach additional ordering flag to all operator nodes. See the next patch. We rewrite the tree immediately after chained 'or' operations are flattened by simplifyinfixops(), so we don't need to care if arguments are stored in x[1] or x[1:].

journal: properly check for held lock (issue5349) The 'jlock' code meant to check for a held lock, but it actually just checking for a lock object. With CPython, this worked because the 'jlock' object is not referenced outside the '_write' function so reference counting would garbage collect it and the '_lockref' would return None. With pypy, the garbage collection would happen at an undefined time and the '_lockref' can still point to a 'jlock' object outside of '_write'. The right thing to do here is not only to check for a lock object but also to check if the lock is held. We update the code to do so and reuse a utility method that exist on 'localrepo' to help readability. This fix journal related tests with pypy.

ssl: handle a difference in SSLError with pypy (issue5348) The SSLError exception is a bit different with pypy (message is the first argument, not the second) This led the certificate error handling to crash when trying to extract the ssl error message. We now handle this different and 'test-https.t' is green again.

manifest: change manifestctx to not inherit from manifestdict If manifestctx inherits from manifestdict, it requires some weird logic to lazily load the dict if a piece of information is asked for. This ended up being complicated and unintuitive to use. Let's move the dict creation to .read(). This will make even more sense once we start adding readdelta() and other similar methods to manifestctx.

manifest: make one use of _mancache avoid manifestctxs In a future patch we will change manifestctx and treemanifestctx to no longer derive from manifestdict and treemanifest, respectively. This means that consumers of the _mancache will now need to be aware of the different between the two, until we get rid of the manifest entirely and the _mancache becomes only filled with ctxs. This fixes one case of it that can be fixed by using the other cache. Future patches will address the others uses using the upcoming manifestctx.read() function.

debugrevspec: add option to verify optimized result This provides a convenient way to diff "hg debugrevspec" outputs generated with/without --no-optimized option.

debugrevspec: add option to skip optimize() and evaluate unoptimized tree This will help debugging optimizer bugs. Maybe '--no-optimized' can be changed to '--optimized' (default: True) when flags series landed.

revset: remove showwarning option from expandaliases() Now all callers pass showwarning=ui.warn, so we no longer need the option to suppress warnings.

debugrevspec: evaluate tree built by itself Prepares for new option to evaluate an unoptimized tree. Since a revset expression is no longer parsed twice, alias warnings should be displayed at the first parsing stages. That's why showwarning=ui.warn is added.

localrepo: make _refreshfilecachestats unfiltered method to refresh correctly Before this patch, if transaction is started via "filtered repo" object, _refreshfilecachestats() at closing transaction doesn't refresh file stat of any @filecache properties correctly, because: - _refreshfilecachestats() omits refreshing file stat of a @filecache property, if it doesn't appear in self.__dict__ - if transaction is started via "filtered repo", _refreshfilecachestats() is applied on "filtered repo" because repo.transaction() adds "self._refreshfilecachestats" to post close procedures. repo.transaction() isn't unfiltered method, and "self" in it means "filtered repo" in this case. Transactions started by explicit repo.transaction() easily causes this situation. - _refreshfilecachestats() applied on "filtered repo" omits whole refreshing because @filecache properties are stored into "unfiltered repo", and appear only in self.__dict__ of "unfiltered repo". This incorrect refreshing causes unnecessary reloading from files. To refresh file stat of @filecache properties at closing transaction correctly, this patch makes _refreshfilecachestats() unfiltered method. This patch chooses making _refreshfilecachestats() unfiltered method instead of making transaction() unfiltered method, to reduce unexpected side effect.

streamclone: clear caches after writing changes into files for visibility Before this patch, streamclone-ed changes are invisible via @filecache properties to in-process procedures before closing transaction (e.g. pretxnclose python hook), if corresponded property is cached before consumev1(). Strictly speaking, caching should occur inside (store) lock for transaction. repo.invalidate() after closing transaction is too late to force @filecache properties to be reloaded from changed files at next access. For visibility of streamclone-ed changes to in-process procedures before closing transaction, this patch clears caches just after writing changes into files. BTW, regardless of changing in this patch, clearing cached properties in consumev1() causes inconsistency, if (1) transaction is started and (2) any @filecache property is changed before consumev1(). This patch also adds the comment to fix this (potential) inconsistency in the future.

localrepo: make invalidate avoid invalidating store inside transaction (API) Before this patch, invalidate() discards in-memory fncache changes, even inside transaction scope. Such changes should be written out at closing transaction. Otherwise, fncache might overlook newly added files. A file overlooked by fncache isn't accessible via store vfs, even if it actually exists in store. On the other hand, a non-existing file in fncache is less harmful, because fncachestore always examines whether a file actually exists or not before access. Therefore, discarding in-memory changes can be safely omitted. It is typical case that repo.invalidate() in streamclone is executed inside nested transaction. This patch makes invalidate() avoid invalidating store inside transaction. This patch focuses on describing only how invalidate() changes own behavior according to activity of transaction. Describing other detail of invalidate() in docstr will be done in another series, which refactors invalidate*() functions.

streamclone: force @filecache properties to be reloaded from file Before this patch, consumev1() invokes repo.invalidate() after closing transaction, to force @filecache properties to be reloaded from files at next access, because streamclone writes data into files directly. But this doesn't work as expected in the case below: 1. at closing transaction, repo._refreshfilecachestats() refreshes file stat of each @filecache properties with streamclone-ed files This means that in-memory properties are treated as valid. 2. but streamclone doesn't changes in-memory properties This means that in-memory properties are actually invalid. 3. repo.invalidate() just forces to examine file stat of @filecache properties at the first access after it Such examination should concludes that reloading from file isn't needed, because file stat was already refreshed at (1). Therefore, invalid in-memory cached properties (2) are unintentionally treated as valid (1). This patch invokes repo.invalidate() with clearfilecache=True, to force @filecache properties to be reloaded from file at next access. BTW, it is accidental that repo.invalidate() without clearfilecache=True in streamclone case seems to work as expected before this patch. If transaction is started via "filtered repo" object, repo._refreshfilecachestats() tries to refresh file stat of each @filecache properties on "filtered repo" object, even though all of them are stored into "unfiltered repo" object. In this case, repo._refreshfilecachestats() does nothing unintentionally, but this unexpected behavior causes reloading @filecache properties after repo.invalidate(). This is reason why this patch should be applied before making _refreshfilecachestats() correctly refresh file stat of @filecache properties.

manifest: backed out changeset bb3281b3fcaa There is some suspicious failure in evolution tests. This changeset was supposed to be dropped until we investigate.

manifest: backed out changeset b60a5fe98b73 There is some suspicious failure in evolution tests. This changeset was supposed to be dropped until we investigate.

debugrevspec: deprecate --optimize option This option has been superseded by '--show-stage NAME', and will cause confusion in future patches.

debugrevspec: add option to print parsed tree at given stages "-p <stage>" is useful for investigating parsing stages. With -p option, a transformed tree is printed no matter if it is changed or not, which allows us to know valid stage names by "-p all".

debugrevspec: transform and print parsed tree by stages Prepares for adding new option to print transformed tree at each stage.

debugrevspec: build parsed tree even if no --verbose specified Prepares for evaluating unoptimized tree. The output is suppressed by ui.note() anyway.

osutil: fix the declaration to work on different sizes of off_t

manifest: change manifestctx to not inherit from manifestdict If manifestctx inherits from manifestdict, it requires some weird logic to lazily load the dict if a piece of information is asked for. This ended up being complicated and unintuitive to use. Let's move the dict creation to .read(). This will make even more sense once we start adding readdelta() and other similar methods to manifestctx.

manifest: make uses of _mancache aware of contexts In a future patch we will change manifestctx and treemanifestctx to no longer derive from manifestdict and treemanifest, respectively. This means that consumers of the _mancache will now need to be aware of the different between the two, until we get rid of the manifest entirely and the _mancache becomes only filled with ctxs.

manifest: add treemanifestctx class Before we start using repo.manifestlog in the rest of the code base, we need to make sure that it's capable of returning treemanifests. As we add new functionality to manifestctx, we'll add it to treemanifestctx at the same time. We also comment out the manifestctx p1, p2, and linkrev fields for now, since we're not implementing them on treemanifest yet.

revset: add public function to create matcher from evaluatable tree "hg debugrevspec" will use it to evaluate unoptimized tree.

revset: make analyze() a separate step from optimize() This will allow us to evaluate unoptimized tree and compare the result with optimized one. The private _analyze() function isn't renamed since I'll add more parameters to it.

revset: extract tree transformation from optimize() This patch separates the simple tree transformation from the optimization step, which is called as _analyze() since I'll extend this function to infer ordering flags. I want to avoid making _optimize() more complicated. This will also allow us to evaluate unoptimized tree.

hghave: fix has_bzr() to not try to import RevisionSpec as module Since RevisionSpec isn't a module, "import bzrlib.revisionspec.RevisionSpec" would always raise ImportError.

update: label bookmark name in message We label bookmark name as such in various messages. This will help them to standout (or at least give the user the option to make them stand out). We use a distinct label for the 'active' bookmark, this can help users to catch bookmark operation affecting their working copy.

discovery: explicitly check for None in outgoing init f09d0004481c introduced default params for discovery.outgoing(), but it used a falsy check instead of an explicit check for None. The result is that callers that passed in an empty list would have that list overridden by the defaults, which is not the expected behavior. This was discovered by changes to the test-pushrebase.t test in Facebook's repository of mercurial extensions.