largefiles: remove overly complex handling of ignored and unknown files

largefiles: introduce basic debugstate --large functionality Very useful for debugging largefiles "performance" issues. This is used for testing 03faf12fbee7.

util: fold ENOENT check into unlinkpath, controlled by new ignoremissing flag Refactor a common pattern.

merge with stable

largefiles: don't walk through all ignored files Problem: 'hg status' with largefiles enabled would walk through all the files that .hgignore said should be ignored. That made it slow if a lot of files were .hgignored or the cache was cold. It seems like there was a reason to this, but other improvements has rendered this unnecessary. Solution: .hgignore is now only ignored when that is requested (--ignore). This is a minimal 'stable' change. There is room for other improvement.

largefiles revert: update lfdirstate with result from first cleanliness check Largefiles revert do for some reason have two lfdirstates and lfdirstatestatus invocations in one function. The result from the first lfdirstate check was however not written back to the lfdirstate, and some files was thus checked twice.

largefiles status: update lfdirstate with result from cleanliness check Problem: 'hg status' kept checking largefiles with an unknown state until some other command wrote the updated dirstate. Solution: Add missing lfdirstate.write().

bundlerepo: don't return the peer without bundlerepo from getremotechanges Problem: getremotechanges would return the 'other' repo if nothing was incoming and there thus wasn't any bundle to base the repo on. The 'other' could be a http peer which only implement the functionality available over the http protocol. Transplant could thus fail with TypeError: argument of type 'httppeer' is not iterable Solution: Return the local repo instead of the remote peer if there is no reason to place a bundlerepo on top of the local repo.

strip: do not update branchcache during strip (issue3745) At this moment, the cache is invalid, and will be thrown away. Later the strip function will call the `localrepo.destroyed` method that will update the branchmap cache.

copies: do not track backward copies, only renames (issue3739) The inverse of a rename is a rename, but the inverse of a copy is not a copy. Presenting it as such -- in particular, stuffing it into the same dict as real copies -- causes bugs because other code starts believing the inverse copies are real. The only test whose output changes is test-mv-cp-st-diff.t. When a backwards status -C command is run where a copy is involved, the inverse copy (which was hitherto presented as a real copy) is no longer displayed. Keeping track of inverse copies is useful in some situations -- composability of diffs, for example, since adding "a" followed by an inverse copy "b" to "a" is equivalent to a rename "b" to "a". However, representing them would require a more complex data structure than the same dict in which real copies are also stored.

copies: make debug messages more sensible The -> in debug messages is currently overloaded to mean both source to dest and dest to source. To fix this, we add explicit labels and make the arrow direction consistent.

copies: separate moves via directory renames from explicit copies Currently the "copy" dict contains both explicit copies/moves made by a context and pending moves that need to happen because the other context moved the directory the file was in. For explicit copies, the dict stores a destination to source map, while for pending moves via directory renames, it stores a source to destination map. The merge code uses this fact in a non- obvious way to differentiate between these two cases. We make this explicit by storing these pending moves in a separate dict. The dict still has a source to destination map, but that is called out in the docstring.

merge with stable

branchmap: move validity logic in the object itself In several place, We check if a branchcache is still valid regarding the current state of the repository. This changeset puts this logic in a method of the object that can be reused when necessary. A branch map is considered valid whenever it is up to date or a strict subset of the repository state. The change will help making branchcache aware of filtered revision. The change in keyword is expected. the branch cache is actually invalid after the amend. The previous check did not detected it.

branchmap: make update a method

branchmap: make update responsible to update the cache key The update function have all necessary data to keep the branchcache key up to date with its value. This saves assignment to the cache key that each caller of update had to do by hand. The strip case is a bit more complicated to handles from inside the function but I do not expect any impact.

branchmap: factorise changelog access in update This both improves readability and performance. Access to changelog of filtered repository currently have a minor overhead.

branchmap: make write a method on the branchmap object

branchmap: simplify write signature All necessary data (cache value and key) are now stored in the branchcache object. Any extra parameter is superfluous.

branchmap: add the tiprev (cache key) on the branchmap object The actual cache key used on disk is the (tipnode, tiprev) pair. There is no reason not to use the revision number for the in memory version.

branchmap: add the tipnode (cache key) on the branchcache object Gathering data and cache key paves the way to a lot of simplification.

branchmap: store branchcache in a dedicated object Value and key of branchcache would benefit from being hold by the same object. Moreover some logic (update, write, validation) could be move on such object. The creation of this object is the first step toward this move. The result will clarify branchcache related code and hide most of the detail in the class itself. This encapsulation will greatly helps implementation of branchcache for filtered view of the repo.

branchmap: `stream_in` write remote branchcache to local one The previous code was writing it to a non existent `branchcache` attribute. We now write is to the proper `_branchcache` attribute and initialize the `_branchcachetip` at the same time. We keep writing it to disk, the previous code had this part right.

bookmarks: fix head selection for merge with two bookmarked heads A type mismatch caused the search for the other head to fail. The code is fragile, and instead it ended up using the 'first' bookmark head, but the ordering is undefined and it could thus randomly use the wrong bookmarkhead and fail with: $ hg up -q -C e@diverged $ hg merge abort: merging with a working directory ancestor has no effect

branchmap: extract updatebranchcache from repo

branchmap: extract _updatebranchcache from repo

branchmap: _updatebranchmap does not need to be filtered The `_updatebranchmap` method on repo does not need to be filtered as all callers are already handling filtering themself. The fact it is filtered may had even lead to buggy behaviors, but by chances the method make very sparse use of the repo object.

branchmap: extract read logic from repo

branchmap: extract write logic from localrepo

branchmap: create a mercurial.branchmap module This is the foundation stone for an extraction of branches map logic from local repository class. Most of the branches map logic have very few caller and therefor does not fit in the current criteria for code held by the localrepo class. Important change will be made to this code in relation with revision filtering. So we extract it in a dedicated module before adding additional complexity. Follow up commit do the actual code movement.

bundlerepo: use _cacheabletip mechanism in bundlerepo Instead of preventing any cache write we allow writing cache for all content of the original repo. The motivation for this change is to drop the custom _writebranchcache of bundlerepo to help extraction of the branchmap logic out of localrepo.

branchmap: merge _branchtags into updatebranchcache Now that nobody overwrite it, there is no reasons for `_branchtags` to remains separated from `updatebranchcache`.

branchmap: factorise access to changelog in updatebranchcache This prepares merge of `updatebranchcache` and `_branchtags`.

branchmap: simplify _branchtags using a new _cacheabletip method The current _branchtags method is a remnant of an older, much larger function. Its only remaining role is to help MQ to alter the version of branchcache we store on disk. As MQ mutates the repository it ensures persistent cache does not contain anything it is likely to alter. This changeset makes explicit the stable vs volatile part of repository and reduces the MQ specific code to the computation of this limit. The main _branchtags code now handles this possible limit in all cases. This will help to extract the branchmap logic from the repository. The new code of _branchtags is a bit duplicated, but as I expect major refactoring of this section I'm not keen to setup factorisation function here.

hook: disable demandimport before importing hooks This solved an obscure bug for me. In upgrading Distribute on the server, a patch was added to has a try: import a except ImportError thing that's only supposed to work with Python 3.3. I'm using 2.7. My hook failed with an ImportError because of this. It seems kind of sensible to turn off demandimport before importing the hook, since the except ImportError pattern is used quite a bit in Python code (including in other Distribute code).

test-pathencode: accept --seed parameter in hex as well test-pathencode.py outputs the seed value in hex if it finds a deviation. This change allows to specify the seed value as a command line parameter for test-pathencode.py in hex as well.

subrepo: append subrepo path to subrepo error messages This change appends the subrepo path to subrepo errors. That is, when there is an error performing an operation a subrepo, rather than displaying a message such as: pushing subrepo MYSUBREPO to PATH searching for changes abort: push creates new remote head HEADHASH! hint: did you forget to merge? use push -f to force mercurial will show: pushing subrepo MYSUBREPO to PATH searching for changes abort: push creates new remote head HEADHASH! (in subrepo MYSUBREPO) hint: did you forget to merge? use push -f to force The rationale for this change is that the current error messages make it hard for TortoiseHg (and similar tools) to tell the user which subrepo caused the push failure. The "(in subrepo MYSUBREPO)" message has been added to those subrepo methods were it made sense (by using a decorator). We avoid appending "(in subrepo XXX)" multiple times when subrepos are nexted by throwing a "SubrepoAbort" exception after the extra message is appended. The decorator will then "ignore" (i.e. just re-raise) the exception and never add the message again. A small drawback of this method is that part of the exception trace is lost when the exception is catched and re-raised by the annotatesubrepoerror decorator. Also, because the state() function already printed the subrepo path when it threw an error, that error has been changed to avoid duplicating the subrepo path in the error message. Note that I have also updated several subrepo related tests to reflect these changes.

tests: fix some slash-based Windows failures

inotify: on Python < 2.6, socket.error lacks errno

hidden: invalidate hiddenrevs when needed The `hiddenrevs` cache is volatile too (It use content from `obscache`). When unsure it is invalidated when necessary. In a near future, the cache will probably be moved to `revsfiltercache`

cache: group obscache and revsfiltercache invalidation in a single function Both caches are very volatile and needs invalidation on the same kind of event. revsfiltercache actually depends on the content of revsfiltercache.

clfilter: use filtering in `visibleheads` This is the second real use of changelog filtering. The change is very small to allow testing the new filter with a setup close to the original one. We replace custom post processing on `heads`function by call to the standard code pass on a filtering repo. In later coming will have wider usage of filtering that will make the dedicated function useless.

clfilter: use filtering in `visiblebranchmap` Here is the first real use of changelog filtering. The change is very small to allow testing the new filter with a setup close to the original one. We replace custom post processing on branchmap function by call to the standard code pass on a filtering repo. In later coming will have wider usage of filtering that will make the dedicated function useless.

clfilter: introduce a "unserver" filtering mode This mode is for repository used as a server. It filter secret and hidden changeset out. It is put to use in later changeset.

clfilter: add a cache on repo for set of revision to filter for a given set. Recomputing the filtered revisions at every access to changelog is far too expensive. This changeset introduce a cache for this information. This cache is hold by the repository (unfiltered repository) and invalidated when necessary. This cache is not a protected attribute (leading _) because some logic that invalidate it is not held by the local repo itself.

clfilter: add actual repo filtering mechanism We add a `filtered` method on repo. This method return an instance of `repoview` that behaves exactly as the original repository but with a filtered changelog attribute. Filters are identified by a "name". Planned filter are `unserved`, `hidden` and `mutable`. Filtering the repository in place what out of question as it wont not allows multiple thread to share the same repo. It would makes control of the filtering scope harder too. See the `repoview` docstring for details. A mechanism to compute filtered revision is also installed. Some caches will be installed in later commit.

inotify: pacify pestiferous pyflakes precipitously

tests: make test-inotify-issue1208.t disappear

posix: move server side of unix domain sockets out of inotify We also turn the unix domain socket into a class, so that we have a sensible place to hang its logically related attributes and behaviour. We'll shortly want to reuse this in other code.

inotify: don't fall over just because of a dangling symlink Previously, the inotify server failed to start if .hg/inotify.sock was a symlink that pointed to a non-existent path. This behaviour does not seem to make any sense. Now, if we encounter a broken symlink, we unlink it and continue.

test-inotify: test symlink indirection for unix sockets The inotify code performs a delicate dance to work around the 108-byte limit on unix domain socket path names on Linux. This change sets us up to safely refactor that code without breaking it. (It is redundant with part of test-inotify-issue1208.t, but we will shortly make that test go away.)

test-pathencode: compare current pathencoding implementations We already have two implementations of the pathencoding (C and Python) and this test can perfectly well be used to probabilistically test them instead of just wasting CPU cycles and test time.

rebase: use lazy ancestor membership testing For a repository with over 400,000 commits, rebasing one revision near tip, this avoids one walk up the DAG, speeding the operation up by around 0.8 seconds.

localrepo: use lazy ancestor membership testing For a repository with over 400,000 commits, rebasing one revision near tip, this avoids two treks up the DAG, speeding the operation up by around 1.6 seconds.

ancestor: add lazy membership testing to lazyancestors This also makes the perfancestorset command use lazy membership testing. In a linear repository with over 400,000 commits, without this patch, hg perfancestorset takes 0.80 seconds no matter how far behind we're looking. With this patch, hg perfancestorset -- X takes: Rev X Time -1 0.00s -4000 0.01s -20000 0.04s -80000 0.17s -200000 0.43s -300000 0.69s 0 0.88s Thus, for revisions close to tip, we're up to several orders of magnitude faster. At 0 we're around 10% slower.

revlog: move ancestor generation out to a new class This refactoring is to prepare for implementing lazy membership.

ignore: process hgignore files in deterministic order Previously, we processed them in whatever order the dict iterator gave us.

ignore: only read an ignore file once

ignore: refactor ignore into two functions This prepares us for eventually being able to hash the list of patterns in use.

clfilter: fix `nodemap` usage in `getbundle` With the current implementation, `changelog.nodemap` is not filtered. So some filtered changeset in common are not filtered by `n in nodemap`. This leads to crash lower in the stack when the bundle generation try to access those node on a filtered changelog.

test: fix truncated comment in test The push is made to ensure repository are related. The comment in the initial changeset got truncated somehow.

clfilter: ensure context raise RepoLookupError when the revision is filtered Currently the code path of `changectx(filteredrepo, rev)` call `filteredrepo.changelog.node(rev)`. When `rev` is filtered this raise an unhandled `IndexError`. This case now raise a `RepoLookupError` as other error case do.

revlog: remove incancestors since it is no longer used

transplant: replace incancestors uses with ancestors

revlog.ancestors: add support for including revs This is in preparation for an upcoming refactoring. This also fixes a bug in incancestors, where if an element of revs was an ancestor of another it would be generated twice.

perf: add command to test performance of membership in ancestor set The new command, perfancestorset, takes an argument denoting which revset to test the membership of. Currently this runs through all the ancestors and converts them into a set. The primary purpose of having this is to compare this approach, currently used in several places, against the upcoming lazy approach.

ancestor: move missingancestors doctest out into a separate file This is in preparation for upcoming patches which will reuse the same graph for tests.

merge with crew-stable

dirstate: remove obsolete comment from setbranch This comment should have been removed in b74361cf7c0a, when the call to scmutil.checknewlabel was removed.

dirstate: don't rename branch file if writing it failed

commands: 'hg bookmark NAME' should work even with ui.strict=True Before this patch, enabling strict command processing (ui.strict=True) meant that 'hg bookmark NAME', as referenced several places in the documentation, would not work. This adds 'bookmark' as an explicit alias to 'bookmarks'.

tests: improve test-mv-cp-st-diff.t readability

obsolete: refuse to push divergent changeset As other troubles `unstable` and `bumped`. Followup patches may simplify the push code with unification of "obsolescence troubles" handling.

obsolete: add a divergent method on context The same we have `unstable` and `bumped`. Convenient method to access troubles information in general may land later. This get actual use and testing in the next changesets.

obsolete: add revset and test for divergent changesets This changesets add a new `divergent()` revset similar to `unstable()` and `bumped()` one. Introducting this revset allows actuall test of the divergent detection.

obsolete: detect divergent changesets Divergent changeset are final successors (non obsolete) of a changeset who compete with another set of final successors for this same changeset. For example if you have two obsolescence markers A -> B and A -> C, B and C are both "divergent" because they compete to be the one true successors of A. Public revision can't be divergent. This function is used and tested in the next changeset.

obsolete: drop successors sets which are subset of another one If both "(B,)" and "(B, C)" are successors set of "A", "(B,)" is dropped. We won't be interrested in detection such divergence scenario.

obsolete: compute successors set Successors set are an important part of obsolescence. It is necessary to detect and solve divergence situation. This changeset add a core function to compute them, a debug command to audit them and solid test on the concept. Check function docstring for details about the concept.

mq: don't fail when removing a patch without patch file from series file

largefiles: align rm warnings with warnings used in core

merge with stable

largefiles: commit directories that only contain largefiles (issue3548) If we pass a directory to commit whose only commitable files are largefiles, the core commit code aborts before finding the largefiles. So we do the following: For directories that only have largefiles as matches, we explicitly add the largefiles to the matchlist and remove the directory. In other cases, we leave the match list unmodified.

revset.children: ignore rev numbers that are too low This replaces unnecessary parentrevs() calls with calculating min(parentset). Even though the min operation is O(size of parentset), since parentrevs is relatively expensive, this tradeoff almost always works in our favour. In a repository with over 400,000 changesets, hg perfrevset "children(X)" takes: Set X Before After -1 0.51s 0.06s -1000: 0.55s 0.08s -10000: 0.56s 0.10s -100000: 0.60s 0.25s -100000:-99000 0.55s 0.19s 0:100000 0.60s 0.61s all() 0.72s 0.74s The relative performance is similar for Mercurial's own repository -- several times faster in most cases, slightly slower for revisions close to 0 and all().

perf: add a command to measure revset performance

run-tests: fix whitespace nonsense

merge with mpm

run-tests: on windows, put correct python at front of PATH The older approach of trying to copy the python executable into the test directory was doomed to fail. There remains one weakness with this approach: if you've run "make local", tests may pick up the wrong extension DLLs from inside the source tree. I don't know why this happens. A reasonable workaround for now is to test either using --local or with a working directory that does not contain built DLLs.

run-tests: fix exename on Windows

run-tests: support running tests in parallel on windows Previously, we used os.spawnvp, which doesn't exist on Windows, and isn't needed anyway (the command line begins with an absolute path). We also need a slightly more convoluted way to wait for processes without specifying an order on Windows, as it lacks os.wait.

makefile: allow local builds to work on windows/mingw32

check-code: make 'missing whitespace in assignment' more aggressive New warnings: > a.b=ab missing whitespace in assignment (the pattern did not accept '.' on the left hand side) > a=a missing whitespace in assignment (the right hand side pattern never matched a single character) > a=a + 7 missing whitespace in assignment (the pattern only matched one character after the identifier following =)

check-code: there must also be whitespace between ')' and operator The check pattern only checked for whitespace between keyword and operator. Now it also warns: > x = f(),7 missing whitespace after , > x = f()+7 missing whitespace in expression

rm: drop misleading 'use -f' hint for the rm --after 'not removing' warning A warning mentioning 'forgetting' or 'recording delete' would be more correct than 'not removing' but also more confusing.

contrib: state explicitly how hgtest.vim can be installed

tests: kill daemons early, making breaking at "Accept" prompt safe

run-tests: use correct python safely under --jobs

run-tests: check for the correct python when starting

run-tests: use correct python when run with --local

synthrepo: do not crash if a list is empty

hgwebdir: do not show RSS and Atom links for plain directories Up until now the templates that show RSS and Atom feeds on the "repository lists" (i.e. gitweb and monoblue) showed them for all entries, including regular folders. Clicking on those "folder RSS" links would result in an error page being shown. This patch hides those links for regular folders.

hgweb: add branches RSS and Atom feeds There were no RSS nor Atom feeds for the branches page. Different hgweb templates linked to different feeds on their branches page (some linked to the tags feed, some to the log feed and some to the unexisting branches feed).

push: reunite comment with the line of code it describes

bookmarks: spelling correction in docstring

merge: fix mistake in moved _checkcollision call from 5881d5b7552f

windows: correctly pass a mode to S_IFMT in statfiles

strip: make query to get new bookmark target cheaper The current query to get the new bookmark target for stripped revisions involves multiple walks up the DAG, and is really expensive, taking over 2.5 seconds on a repository with over 400,000 changesets even if just one changeset is being stripped. A slightly simplified version of the current query is max(heads(::<tostrip> - <tostrip>)) We make two observations here. 1. For any set s, max(heads(s)) == max(s). That is because revision numbers define a topological order, so that the element with the highest revision number in s will not have any children in s. 2. For any set s, max(::s - s) == max(parents(s) - s). In other words, the ancestor of s with the highest revision number not in s is a parent of one of the revs in s. Why? Because if it were an ancestor but not a parent of s, it would have a descendant that would be a parent of s. This descendant would have a higher revision number, leading to a contradiction. Combining these two observations, we rewrite the revset query as max(parents(<tostrip>) - <tostrip>) The time complexity is now linear in the number of changesets being stripped. For the above repository, the query now takes 0.1 seconds when one changeset is stripped. This speeds up operations that use repair.strip, like the rebase and strip commands.

graft: explicit current node tracking This changes graft to explicitly track the progression of commits it makes, and updates it's idea of the current node based on it's last commit, rather than from the working copy parent. This should have no effect on the value of current since we were reading the working copy parent immediately after commiting to it. The motivation for this change is that a subsequent patch will break the current node and working copy relationship. Splitting this out into a separate patch will make that one more readible.

graft: move commit info building This moves the logic for generating the commit metadata ahead of the merge operation. The only purposae of this patch is to make subsequent patches easier to read, and there should be no behavior changes.

merge with stable

merge: support calculating merge actions against non-working contexts This is not currently used. It is instead a pre-requisite to performing non-conflicting grafts in memory, which a subsequent patch will do.

merge: refactor action calculation into function This pulls the code used to calculate the changes that need to happen during merge.update() into a separate function. This is not useful on its own, but is instead preparatory to performing grafts in memory when there are no potential conflicts.

dirstate: inline more properties and methods in status hg perfstatus -u on a working directory with 170,000 files, without this change: ! wall 1.839561 comb 1.830000 user 1.120000 sys 0.710000 (best of 6) With this change: ! wall 1.804222 comb 1.790000 user 1.140000 sys 0.650000 (best of 6) hg perfstatus on the same directory, without this change: ! wall 1.016609 comb 1.020000 user 0.670000 sys 0.350000 (best of 10) With this change: ! wall 0.985573 comb 0.980000 user 0.650000 sys 0.330000 (best of 10)

perf: add option to perfstatus to get the status of unknown files When status needs to look at unknown files (e.g. when running hg status), it needs to use a completely different algorithm than when it doesn't (e.g. when running hg diff).

dirstate: test normalize is truthy instead of using a no-op lambda hg perfstatus -u on a working directory with 170,000 files, without this change: ! wall 1.869404 comb 1.850000 user 1.170000 sys 0.680000 (best of 6) With this change: ! wall 1.839561 comb 1.830000 user 1.120000 sys 0.710000 (best of 6)

merge with stable

hgweb: avoid generator exhaustion with branches

hgweb: fix iterator reuse in atom feed generation

tests: don't hardcode errno==2 for ENOENT Hurd seems to set ENOENT to 2 + 2**30, unlike everyone else.

osutil: tab damage, how i hate thee

osutil: write a C implementation of statfiles for unix This makes a big difference to performance. In a clean working directory containing 170,000 files, performance of "hg --time diff" improves from 2.38 seconds to 1.69.