reachableroots: extend "revstates" to array of bit flags

reachableroots: rename "seen" array to "revstates" for future extension It will be an array of bit flags, SEEN | ROOT | REACHABLE.

reachableroots: give anonymous name to short-lived "numheads" variable I'll reuse it for the length of the roots list.

reachableroots: reduce nesting level by jumping to next iteration by continue This can eliminate lines over 80 columns. No code change except for the outermost "if" condition.

test-convert-git: work around output format changes in git git version 2.4.3: --- /home/augie/hg/tests/test-convert-git.t +++ /home/augie/hg/tests/test-convert-git.t.err @@ -659,7 +659,7 @@ $ touch a && git add a && git commit -am "commit a" [master (root-commit) 8ae5f69] commit a Author: nottest <test@example.org> 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 a $ cd .. $ git clone git-repo7 git-repo7-client git version 1.7.9.5: --- /home/augie/hg/tests/test-convert-git.t +++ /home/augie/hg/tests/test-convert-git.t.err @@ -659,7 +659,7 @@ $ touch a && git add a && git commit -am "commit a" [master (root-commit) 8ae5f69] commit a Author: nottest <test@example.org> - 1 file changed, 0 insertions(+), 0 deletions(-) + 0 files changed create mode 100644 a $ cd .. $ git clone git-repo7 git-repo7-client I don't know when this changed in git and am too lazy to try and bisect it, so just work around the change.

histedit: correct spelling etc in more comments Spotted during review of another patch.

histedit: add a missing "s" in a comment

convert: fix convert dropping p2 contents during filemap merge When converting a merge commit using a filemap convert (i.e. when moving contents from the root of the repo into subdir1/), convert would silently drop the entire contents of the target repo's p2. This was because when it built the target commit, it did so by taking the target p1 and adding only the files that changed in the source repo's merge commit. This breaks in the case where the target repo has files that are unrelated to the source repo (like in the case where you use convert to import a repo as a subdirectory of another). The fix is to use Mercurial's merge logic to detect which files in p2 we should carry over to the merge. It follows three rules: 1) if the file belongs to the source, don't try to merge it. Rely on the list of files provided to putcommit to be correct. 2) if the file requires merging or user input (change vs deleted), throw an exception. We don't have enough info to do this. 3) if p2 has the newest, non-merge-requiring version of the file, take it I've also added a test to cover this issue.

convert: implements targetfilebelongstosource for filemap source This is an implementation of the new targetfilebelongstosource() function for the filemapper. It simply checks if the given file name is prefixed by any of the rename destinations. It is not a perfect implementation since it doesn't account for the filemap specifying includes or excludes, but that makes the problem much harder, and this implementation should suffice for most cases.

convert: add function to test if file is from source This adds a base implementation of a function that tests if a given file from a target repo came from the source repo. This will be used later to detect which files did not come from the source repo during a merge, so we can merge those files correctly instead of dropping them.

revsetbenchmarks: run make after update so that C extensions are built

reachableroots: fix memleak of integer objects at includepath loop In the first visit loop, val is decref-ed correctly after PySet_Add(). Let's do the same for the includepath loop.

reachableroots: bail if integer object cannot be allocated This patch also replaces Py_XDECREF() by Py_DECREF() because we known "val" and "p" are not NULL. BTW, we can eliminate some of these allocation and error handling of int objects if the internal "seen" array has more information. For example, enum { SEEN = 1, ROOT = 2, REACHABLE = 4 }; /* ... build ROOT mask from roots argument ... */ if (seen[revnum + 1] & ROOT) { /* instead of PySet_Contains(roots, val) */ >From my quick hack, it is 2x faster.

devel-warn: issue a warning when writing bookmarks without holding the wlock I saw an issue in an extension that we develop where we were writing bookmarks without holding the wlock. Another extension was taking a lock at the same time and wiped out the bookmarks we were about to write. This patch adds a devel-warning to urge people to fix their invalid code.

merge with stable

rebase: lock the repo during the full rebase operation Running `hg pull --rebase` would move bookmarks without any repository locking. So we now lock the repository. For good measure and avoiding sneaky race conditions, we lock the repository for the whole operation. There is no code change besides the indentation.

update: wlock the repo for the whole 'hg update' command The update command is touching the repository and should lock it for the length of its operations. Equally importantly, it should lock the repository when it is writing bookmarks. It wasn't doing so until now, leaving doors open for all kinds of drunk beaver parties. This results in some minor tests changes, and the fixing of a couple of bugs from race conditions. Code does not receive any changes beside extra indentation.

help: fix typo in scripting documentation

hg: avoid auto sharing when the clone destination is remote Before this patch, when auto sharing is enabled, 'hg.clone()' tries to create local clone regardless of locality of the clone destination on the host, and causes failure. To avoid auto sharing when the clone destination is remote, this patch adds examination of 'islocal(dest)' before auto sharing in 'hg.clone()'. 'islocal(dest)' is examined after 'sharepool', because: - the former is more expensive than the latter - without enabling share extension, the later is always negative

largefiles: ensure lfutil.getstandinmatcher() only matches standins Previously, simply having the largefiles extension loaded without any largefiles added would crash when amending with -I. The problem was with no files in the matcher, the pattern list of files joined with 'standindir' was empty, and scmutil.match() would match everything. In lfutil.composestandinmatcher(), the match function is used to test if the file is a standin, and after getting a false positive, proceeds to call lfutil.splitstandin(). This returns None because it isn't a standin, which blows up when passed to rmatcher.matchfn(). Manually overriding _always in getstandinmatcher() probably isn't necessary anymore, but we leave well enough alone on stable. This regressed in ab618e52788a.

dirstate: add --minimal flag to debugrebuilddirstate On repositories with hundreds of thousands of files, hg debugrebuilddirstate causes every dirstate entry to be marked lookup, and the next hg status can take many minutes. This adds a --minimal flag that allows us to only rebuild the parts of the dirstate that are inconsistent. This follows two rules: 1) If a file is in the dirstate but not in the parent manifest, and it is not marked 'add', it is busted and we should drop it. 2) If a file is not in the dirstate at all, but it is in the parent manifest, it should be added to the dirstate and we need to mark it as lookup. This allows us to fix repositories where the dirstate doesn't match the manifest much more quickly. Tested by artificially adding bad dirstate entries (via code) for both cases above.

tests: make filterpyflakes.py read target files relatively to cwd Before this patch, 'filterpyflakes.py' reads target files relatively to own location. But this prevents third party tools from using it in own source tree, because their files are placed separately from 'filterpyflakes.py'. In fact, 'test-check-pyflakes.t', which is the only user of 'filterpyflakes.py', changes current working directory (cwd) to the root of "test target" source tree before using it. Therefore, composing the root of source tree in 'filterpyflakes.py' is redundant. This patch makes 'filterpyflakes.py' read target files relatively to cwd by invoking 'open()' without any path composition. This also removes importing 'os' module, because there is no user of it after this patch. This is a one of preparation of issue4677.

monoblue: remove duplicate font-family property from td.source

monoblue: remove unused elements and related css Since 91b0ada2d94b and c9b88695d894, when monoblue was introduced, the code this patch removes was untouched. Presumably, there supposed to be nice graphics in the screen corners, but there never were due to: - the css being commented out - ids of the elements and of the css selectors being different - and the png files absent The "corner" elements were unstyled and didn't affect the rest of the page, so I think it's safe to remove all this.

revpair: restrict odd-range handling to top-level x:y expression (issue4774) The odd-range hack was introduced by 2a0efa1112ac for backward compatibility, but it was too widely applied. I've checked cmdutil.revpair() at 1.6, and found that ".:", ":0" and ":" are also handled as pairs. So let's enable the hack only for "x:y", "x:", "y:" and ":". test-revset.t is updated because "tip^::tip^ or tip^" shouldn't be taken as an odd range. This patch adds "tip^:tip^" instead. This patch is written for the default branch because parse() of the stable branch lacks compatibility hack for "foo+bar" tag. If we want to mitigate the issue in stable, we can add something like "and '::' in revs[0]".

revpair: update test to make a difference if odd range not handled specially It was added at 2a0efa1112ac, but there was no difference between -r2 and -r2:2 because the working directory was clean.

reachableroots: verify type of each item of heads argument Though PyInt_AS_LONG() can return a value no matter if it isn't an int object, it could exceed the boundary of the underlying struct. I think C API should be defensive to such errors.

reachableroots: verify integer range of heads argument (issue4775) Now it raises IndexError instead of SEGV for 'wdir()' as it was before.

reachableroots: unify bail cases to raise exception correctly Before this patch, release_seen_and_tovisit did not return NULL, so the exception was not raised immediately. As Py_XDECREF() and free() are safe for NULL, we can simply bail in any case.

reachableroots: pass NULL to PySet_New() as it expects a pointer, not an int

merge with stable

copy: add flag for disabling copy tracing Copy tracing can be up to 80% of rebase time when rebasing stacks of commits in large repos (hundreds of thousands of files). This provides the option of turning off the majority of copy tracing. It does not turn off _forwardcopies() since that is used to carry copy information inside a commit across a rebase. This will affect the situation where a user edits a file, then rebases on top of commits that have moved that file. The move will not be detected and the user will have to manually resolve the issue (possibly by redoing the rebase with this flag off). The reason to have a flag instead of trying to fix the actual copy tracing performance is that copy tracing is fundamentally an O(number of files in the repo) operation. In order to know if file X in the rebase source was copied anywhere, we have to walk the filelog for every new file that exists in the rebase destination (i.e. a file in the destination that is not in the common ancestor). Without an index that lets us trace forward (i.e. from file Y in the common ancestor forward to the rebase destination), it will never be an O(number of changes in my branch) operation. In mozilla-central, rebasing a 3 commit stack across 20,000 revs goes from 39s to 11s.

strip: use the 'finally: tr.release' pattern during stripping The previous code, was calling 'abort' in all exception cases. This was wrong when an exception was raised by post-close callback on the transaction. Calling 'abort' on an already closed transaction resulted in a error, shadowing the original error. We now use the same pattern as everywhere else. 'tr.release()' will abort the transaction if we escape the scope without closing it. We add a test to make sure we do not regress.

changelog: trust C implementation of reachableroots more There are no remaining codepaths in reachableroots where it will return None, so just trust it completely and simplify this method. Result by revset ================ Revision: 0) Revision 1c75249e159b: style: adjust whitespaces in webutil.py 1) Revision d1d91b8090c6: changelog: trust C implementation of reachableroots more revset #0: 0::tip plain 0) 0.067684 1) 0.006622 9% revset #1: 0::@ plain 0) 0.068249 1) 0.009394 13% IOW this is a 10x speedup in my repo for hg itself for 0::tip and similar revsets now that the C code is correctly wired up.

reachableroots: return NULL if we're throwing an exception Based on my reading of [0] and surrounding sections, if we want an exception to be properly raised when something goes wrong in the C code, we need to make sure we return NULL here. Do so. https://docs.python.org/2/extending/extending.html#back-to-the-example

reachableroots: fix transposition of set and list types in PyArg_ParseTuple This is being masked by the function not properly returning NULL when it raises an exception, so the client code was just falling back to the native codepath when it got None back. A future change removes all reason for this C function to return None, which exposed this problem during development.

reachableroots: consistently use short-form of PyErr_NoMemory()

reachableroots: if allocating a new set fails, use PyErr_NoMemory() My inspection of the implementation of PySet_New() indicates that it does *not* reliably set an exception in the cases where it returns NULL (as far as I can tell it'll never do that!), so let's set that up ourselves.

reachableroots: default to the C implementation This patch is part of a series of patches to speed up the computation of revset.reachableroots by introducing a C implementation. The main motivation is to speed up smartlog on big repositories. At the end of the series, on our big repositories the computation of reachableroots is 10-50x faster and smartlog on is 2x-5x faster. Before this patch, reachableroots was computed in pure Python by default. This patch makes the C implementation the default and provides a speedup for reachableroots.

changelog: add way to call the reachableroots C implementation This patch is part of a series of patches to speed up the computation of revset.reachableroots by introducing a C implementation. The main motivation is to speed up smartlog on big repositories. At the end of the series, on our big repositories the computation of reachableroots is 10-50x faster and smartlog on is 2x-5x faster. This patch allows us to call the new C implementation of reachableroots from python by creating an entry point in the changelog class.

reachableroots: add a C implementation This patch is part of a series of patches to speed up the computation of revset.reachableroots by introducing a C implementation. The main motivation is to speed up smartlog on big repositories. At the end of the series, on our big repositories the computation of reachableroots is 10-50x faster and smartlog on is 2x-5x faster. This patch introduces a C implementation for reachableroots following closely the Python implementation but optimized by using C data structures.

revset: remove grandparent by using reachableroots This patch is part of a series of patches to speed up the computation of revset.reachableroots by introducing a C implementation. The main motivation is to speed up smartlog on big repositories. At the end of the series, on our big repositories the computation of reachableroots is 10-50x faster and smartlog on is 2x-5x faster. Before this patch, we had a custom computation for grandparent that was very close to the idea of reacheablerooots. This patch expresses grandparent with reachableroots to reduce the amount of code.

revset: rename revsbetween to reachableroots and add an argument This patch is part of a series of patches to speed up the computation of revset.revsbetween by introducing a C implementation. The main motivation is to speed up smartlog on big repositories. At the end of the series, on our big repositories the computation of revsbetween is 10-50x faster and smartlog on is 2x-5x faster. This patch rename 'revsbetween' to 'reachableroots' and makes the computation of the full path optional. This will allow graphlog to compute grandparents using 'reachableroots' and remove the need for a dedicated grandparent function.

revset: make revsbetween public This patch is part of a series of patches to speed up the computation of revset.revsbetween by introducing a C implementation. The main motivation is to speed up smartlog on big repositories. At the end of the series, on our big repositories the computation of revsbetween is 10-50x faster and smartlog on is 2x-5x faster. Later in this serie, we want to reuse the implementation of revsbetween in the changelog module, therefore, we make it public.

match: fix a caseonly rename + explicit path commit on icasefs (issue4768) The problem was that the former name and the new name are both normalized to the case in dirstate, so matcher._files would be ['ABC.txt', 'ABC.txt']. localrepo.commit() calls localrepo.status(), passing along the matcher. Inside dirstate.status(), _walkexplicit() simply grabs matcher.files() and processes those items. Since the old name isn't present, it is silently dropped. There's a fundamental tension here, because the status command should also accept files that don't match the filesystem, so we can't drop the normalization in status. The problem originated in baa11dde8c0e. Unfortunately with this change, the case of the old file must still be specified exactly, or the old file is again silently excluded. I went back to baa11dde8c0e^, and that had the same behavior, so we are no worse off. I'm open to ideas from a matcher or dirstate expert on how to fix that half.

style: adjust whitespaces in webutil.py Turns out, all this came from the single d605a82cf189.

merge with stable

convert: fix git copy file content conversions There was a bug in the git convert code where if you copied a file and modified the copy source in the same commit, and if the copy dest was alphabetically earlier than the copy source, the converted version would use the copy dest contents for both the source and the target. The root of the bug is that the git diff-tree output is formatted like so: :<mode> <mode> <oldhash> <newhash> <state> <src> <dest> :100644 100644 c1ab79a15... 3dfc779ab... C069 oldname newname :100644 100644 c1ab79a15... 03e2188a6... M oldname The old code would always take the 'oldname' field as the name of the file being processed, then it would try to do an extra convert for the newname. This works for renames because it does a delete for the oldname and a create for the newname. For copies though, it ends up associating the copied content (3dfc779ab above) with the oldname. It only happened when the dest was alphabetically before because that meant the copy got processed before the modification. The fix is the treat copy lines as affecting only the newname, and not marking the oldname as processed.

revset: prevent crash caused by empty group expression while optimizing "or" An empty group expression "()" generates None in AST, so it should be tested before destructuring a tuple. "A | ()" is still evaluated to an error because I'm not sure whether "()" represents an empty set or an empty expression (= a unit value). They are identical in "or" operation, but they should be evaluated differently in "and" operation. expression empty set unit value ---------- --------- ---------- () {} A A & () {} A A | () A A

revset: prevent crash caused by empty group expression while optimizing "and" An empty group expression "()" generates None in AST, so the optimizer have to test it before destructuring a tuple. The error message, "missing argument", is somewhat obscure, but it should be better than crash.

win32: use absolute_import

wireproto: use absolute_import

worker: use absolute_import

verify: use absolute_import

url: use absolute_import

ui: use absolute_import And we have no more ambiguous mixed imports \o/

unionrepo: use absolute_import

treediscovery: use absolute_import

transaction: use absolute_import

templater: use absolute_import

templatekw: use absolute_import

templatefilters: use absolute_import

tags: use absolute_import

tagmerge: use absolute_import

subrepo: use absolute_import

strutil: use absolute_import This file doesn't import anything. But chances are we'll eventually implement a static analysis check that ensures absolute_import is used. So we might as well add this.

statichttprepo: use absolute_import

sslutil: use absolute_import

sshserver: use absolute_import

sshpeer: use absolute_import

simplemerge: use absolute_import

setdiscovery: use absolute_import

repoview: use absolute_import

revset: use absolute_import

repair: use absolute_import

pushkey: use absolute_import

progress: use absolute_import

posix: use absolute_import

phases: use absolute_import

peer: use absolute_import

pathutil: use absolute_import

parser: use absolute_import

node: use absolute_import

namespaces: use absolute_import

minirst: use absolute_import

merge: use absolute_import

match: use absolute_import

mail: use absolute_import

lock: use absolute_import

i18n: use absolute_import

httppeer: use absolute_import

hook: use absolute_import

hbisect: use absolute_import

graphmod: use absolute_import

formatter: use absolute_import

filemerge: use absolute_import

filelog: use absolute_import

fancyopts: use absolute_import

extensions: use absolute_import

error: use absolute_import

discovery: use absolute_import

demandimport: use absolute_import

dagutil: use absolute_import

dagparser: use absolute_import

crecord: use absolute_import

hg: use absolute_import

fileset: use absolute_import

demandimport: support lazy loading for absolute_import Before, we didn't support lazy loading if absolute_import was in effect and a fromlist was used. This meant that "from . import X" wasn't lazy and performance could suffer as a result. With this patch, we now support lazy loading for this scenario. As part of developing this, I discovered issues when module names are defined. Since the enforced import style only allows "from X import Y" or "from .X import Y" in very few scenarios when absolute_import is enabled - scenarios where Y is not a module and thus there is nothing to lazy load - I decided to drop support for this case instead of chasing down the errors. I don't think much harm will come from this. But I'd like to take another look once all modules are using absolute_import and I can see the full extent of what is using names in absolute_import mode.

demandimport: support keyword arguments on _hgextimport __import__ supports keyword arguments since Python 2.5. This proxy should too. An upcoming patch will use this feature.

demandimport: refactor logic and add documentation demandimport doesn't currently support absolute imports (level >= 0). In preparation for this, we add some documentation and a code branch to handle the absolute_import case.

demandimport: add __future__ to ignore list __future__ is special. We should definitely not be trying to lazy load it.

demandimport: remove support for Python < 2.5 The removed code was to support an __import__ function that doesn't support the "level" argument. This argument was added in Python 2.5, which we no longer support.

dispatch: use absolute_import A mixed, ambiguous import has been removed!

commands: break import cycle This was the easiest link in the chain to break as there was only one use of commandserver in commands.py. Other files had multiple users.

cmdutil: break import cycle This was the easiest place to break the chain, as there were only 2 uses of the imported module in the file.

revset: make balanced addsets by orset() without using _combinesets() As scmutil.revrange() was rewritten to not use _combinesets(), we no longer need _combinesets().

revrange: evaluate all revset specs at once This provides an opportunity for revset to optimize given expressions. For example, "-r0 -r1 -r2" can be optimized to "_list(0 1 2)".

revset: add matchany() to construct OR expression from a list of specs This will allow us to optimize "-rREV1 -rREV2 ..." command-line options.

revset: split post-parsing stage from match() _makematcher() will be reused by new matchany(ui, specs, repo=None) function I'll add by the next patch.

filesets: ignore unit case in size() predicate for single value When specifying one plain value in size(), e.g. size(1k), fileset tries to guess the upper bound automatically (see the comment in _sizetomax()). It didn't ignore the specified unit's case, and so size("1 GB"), for example, produced this error: hg: parse error: couldn't parse size: 1 GB Let's do the same thing that util.sizetoint() does: .lower(). The two test lines without output just check that there are no parse errors.