vfs: add explanation about cost of checkambig=True in corner case

vfs: replace avoiding ambiguity in abstractvfs.rename with _avoidambig This centralizes common logic to forcibly avoid file stat ambiguity into _avoidambig(), which was introduced by previous patch.

vfs: copy if EPERM to avoid file stat ambiguity forcibly at closing Now, files (to be truncated) are opened with checkambig=True, only if localrepository caches it. Therefore, straightforward "copy if EPERM" is always reasonable to avoid file stat ambiguity at closing. This patch makes checkambigatclosing close wrapper copy the target file, and advance mtime on it after renaming, if EPERM. This can avoid file stat ambiguity, even if the target file is owned by another (see issue5418 and issue5584 for detail). This patch factors main logic out instead of changing checkambigatclosing._checkambig() directly, in order to reuse it.

transaction: apply checkambig=True only on limited files for similarity Now, transaction can determine whether avoidance of file stat ambiguity is needed for each files, by blacklist "checkambigfiles". For similarity to truncation in _playback(), this patch apply checkambig=True only on limited files in code paths below. - restoring files by util.copyfile(), in _playback() (checkambigfiles itself is examined at first, because it as a keyword argument might be None) - writing files at finalization of transaction, in _generatefiles() This patch reduces cost of checking stat at writing out and restoring files, which aren't filecache-ed.

transaction: avoid file stat ambiguity only for files in blacklist Advancing mtime by os.utime() fails for EPERM, if the target file is owned by another. bff5ccbe5ead and related changes made some code paths give advancing mtime up in such case, to fix issue5418. This causes file stat ambiguity (again), if it is owned by another. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan To avoid file stat ambiguity in such case, especially for .hg/dirstate, ed66ec39933f made vfs.rename() copy the target file, and advance mtime of renamed one again, if EPERM (see issue5584 for detail). But straightforward "copy if EPERM" isn't reasonable for truncation of append-only files at rollbacking, because rollbacking might cost much for truncation of many filelogs, even though filelogs aren't filecache-ed. Therefore, this patch introduces blacklist "checkambigfiles", and avoids file stat ambiguity only for files specified in this blacklist. This patch consists of two parts below, which should be applied at once in order to avoid regression. - specify 'checkambig=True' at vfs.open(mode='a') in _playback() according to checkambigfiles - invoke _playback() with checkambigfiles - add transaction.__init__() checkambigfiles argument, for _abort() - make localrepo instantiate transaction with _cachedfiles - add rollback() checkambigfiles argument, for "hg rollback/recover" - make localrepo invoke rollback() with _cachedfiles After this patch, straightforward "copy if EPERM" will be reasonable at closing the file opened with checkambig=True, because this policy is applied only on files, which are listed in blacklist "checkambigfiles".

localrepo: store path and vfs location of cached properties This information is used to make transaction handle these files specially, in order to avoid file stat ambiguity of them. Gathering information about cached files via annotation classes can avoid overlooking properties newly introduced in the future.

template: add successors template Add a 'successorssets' template that returns the list of all closest known sucessorssets for a changectx. The elements of the list are changesets. The "closest successors" are the first locally known revisions encountered while, walking successors markers. It uses successorsets previously modified to support the closest argument. This logic respect repository filtering. So hidden revision will be skipped by this logic unless --hidden is specified. Since we only display the visible predecessors, this template will not display anything in most case. It makes a good candidate for inclusion in the default log output. I updated the test-obsmarker-template.t test file introduced with the predecessors template to test successorssets template.

template: add tests for more complex cases We add new tests for improving the coverage of existing obs-markers related template (predecessors) and the new one we are introducing (successorssets).

obsolete: closest divergent support Add a closest argument to successorssets changing the definition of latest successors. With "closest=false" (current behavior), latest successors are "leafs" on the obsmarker graph. They don't have any successor and are known locally. With "closest=true", latest successors are the closest locally-known changesets that are visible in the repository or repoview. Closest successors can be then obsolete, orphan. This will be used in a later patch to show the closest successor of changesets with the successorssets template.

obsolete: pass cache argument of successors set explicitly We plan to add a new argument to successorsets. But first we need to update all callers to pass cache argument explicitly to avoid arguments confusion.

obsolete: small doc update for 'successorssets' Clarify successorssets documentation before we start updating the main function. This patch serie will introduce the successorssets template, the opposite of predecessor templates. Successors will use successorssets function and requires some improvement so before doing that, we clean up successorssets a bit.

phabricator: do not read a same revision twice It's possible to set up non-linear dependencies in Phabricator like: o D4 |\ | o D3 | | o | D2 |/ o D1 The old `phabread` code will print D1 twice. This patch adds de-duplication to prevent that. Test Plan: Construct the above dependencies in a Phabricator test instance and make sure the old code prints D1 twice while the new code won't.

patch: make parsepatch optionally trim context lines Previously there is a suspicious `if False and delta > 0` which dates back to the beginning of hgext/record.py (b2607267236d). The "trimming context lines" feature could be useful (and is used by the next patch). So let's enable it. This patch adds a new `maxcontext` parameter to `recordhunk` and `parsepatch`, changing the `if False` condition to respect it. The old `trimcontext` implementation is also wrong - it does not update `toline` correctly and it does not do the right thing for `before` context. A doctest was added to guard us from making a similar mistake again. Since `maxcontext` is set to `None` (unlimited), there is no behavior change.

phabricator: try to fetch differential revisions in batch Previously, we read Differential Revisions one by one by calling `differential.query`. Fetching them one by one is suboptimal. Unfortunately, there is no Conduit API that allows us to get a stack of diffids using a single API call. This patch tries to be smarter using a simple heuristic: when fetching D59 as a stack, previous IDs like D51, D52, D53, ..., D58 are likely belonging to a same stack so just fetch them as well. Since `differential.query` only returns cheap metadata without expensive diff content, it shouldn't be a big problem for the server. Using a test Phabricator instance, this patch reduces `phabread` reading a 10 patch stack from about 13 to 30 seconds to 8 seconds.

phabricator: avoid calling differential.getcommitmessage Previously, we call differential.getcommitmessage API to get commit messages. Now we read that from "Differential Revision" object fetched via "differential.query" API. This removes one API call per patch.

phabricator: rework phabread to reduce memory usage and round-trips This patch reworked phabread a bit so it fetches the lightweight "Differential Revision" metadata for a stack first. Then read other data. This allows the code to: a) send 1 request to get all `hg:meta` data instead of N requests b) patches are read in desired order, no need to buffer the output "b)" reduces the memory usage from O(N^2) to O(N) since we no longer keep old patch contents in memory. The above `N` means the number of patches in the stack.

phabricator: abort if phabsend gets empty revs Previously we didn't abort. Now we abort if revs is empty. This is consistent with "hg export" behavior. Maybe "return 1" is also a reasonable behavior, but that's inconsistent with the existing "hg export".

phabricator: do not upload new diff if nothing changes Previously, `phabsend` uploads new diffs as long as the commit hash changes. That's suboptimal because sometimes the diff is exactly the same as before, the commit hash change is caused by a parent hash change, or commit message change which do not affect diff content. This patch adds a check examining actual diff contents to skip uploading new diffs in that case.

phabricator: add node and p1 to hg:meta property The "hg:meta" property is for extra metadata to reconstruct the patch. Previously it does not have node or parent information since I think by reading the patch again, the commit message will be mangled (like, added the "Differential Revision" line) and we cannot preserve the commit hash. However, the "parent" information could be useful. It could be helpful to locate the "base revision" so in case of a conflict applying the patch directly, we might be able to use 3-way merge to resolve it correctly. Note: "local:commits" is an existing "property" used by Phabricator that has the node and parent information. However, it lacks of timezone information and requires "author" and "authorEmail" to be separated. So we are using a different "property" - "hg:meta" to be distinguished from "local:commits".

phabricator: check associated Differential Revision from commit message Previously, only tags can "associate" a changeset to a Differential Revision. But the usual pattern (arc patch or hg phabread) is to put the Differential Revision URL in commit message. This patch makes the code read commit message to find associated Differential Revision if associated tags are not found. This makes some workflows possible. For example, if the author loses their repo, or switch to another computer, they can continue download their own patches from Phabricator and update them without needing to manually create tags.

tests: replace yet more calls to `python` with $PYTHON These are some simple cases. More to come in a future change. Reviewers: krbullock Reviewed By: krbullock Differential Revision: https://phab.mercurial-scm.org/D4

tests: capitalize Python when it's not used as a command name This avoids some false positives in an upcoming check-code rule. Reviewers: krbullock Reviewed By: krbullock Differential Revision: https://phab.mercurial-scm.org/D3

merge with stable

vfs: drop the 'mustaudit' API There are no remaining users of 'mustaudit' so we can safely drop the API. External user are unlikely from a quick research so no deprecation is added.

streamclone: stop using 'vfs.mustaudit = False' Now that each call disable the auditing on its own, we can safely drop this the mustaudit usage. No other code is modified.

vfs: simplify path audit disabling in stream clone The whole 'mustaudit' API is quite complex compared to its actual usage by its unique user in stream clone. Instead we add a "auditpath" parameter to 'vfs.__call_'. The stream clone code then explicitly open files with path auditing disabled. The 'mustaudit' API will be cleaned up in the next changeset.

auditor: add simple comment about repo.auditor and al Every once in a while, I get confused by what these are. Let us add a comment.

tag: make sure the repository is locked when tagging Otherwise, writing localtag can happen without the lock.

test: glob a line number in test-devel-warnings.t This make is simpler to edit the extensions file without side effect.

test: add a small comment to explain a section of test-devel-warning This makes each test boundaries clearer.

obsolete: reports the number of local changeset obsoleted when unbundling This is a first basic visible usage of the changes tracking in the transaction. We adds a new function computing the pre-existing changesets obsoleted by a transaction and a transaction call back displaying this information. Example output: added 1 changesets with 1 changes to 1 files (+1 heads) 3 new obsolescence markers obsoleted 1 changesets The goal is to evolve the transaction summary into something bigger, gathering existing output there and adding new useful one. This patch is a good first step on this road. The new output is basic but give a user to the content of tr.changes['obsmarkers'] and give an idea of the new options we haves. I expect to revisit the message soon. The caller recording the transaction summary should also be moved into a more generic location but further refactoring is needed before it can happen.

transaction: track new obsmarkers in the 'changes' mapping The obsstore collaborate with transaction to make sure we track all the obsmarkers added during a transaction. This will be useful for various usages: hooks, caches, better output, etc. This is the seconds kind of data added to tr.changes (first one was added revisions)

configitems: register the 'progress.estimate' config

configitems: register the 'progress.clear-complete' config

configitems: register the 'progress.assume-tty' config

configitems: register the 'format.usestore' config

configitems: register the 'format.usegeneraldelta' config

configitems: register the 'format.usefncache' config

configitems: register the 'format.obsstore-version' config

configitems: register the 'factotum.service' config

configitems: register the 'factotum.mountpoint' config

configitems: register the 'factotum.executable' config

configitems: register the 'format.maxchainlen' config

configitems: register the 'format.manifestcachesize' config

configitems: register the 'format.generaldelta' config

configitems: register the 'format.dotencode' config

configitems: register the 'format.chunkcachesize' config

configitems: register the 'format.aggressivemergedeltas' config

configitems: gather comment related to 'worker.backgroundclosemaxqueue' Thanks to Yuya for pointing this out.

configitems: register the 'worker.numcpus' config

configitems: register the 'worker.backgroundclosethreadcount' config

configitems: register the 'worker.backgroundcloseminfilecount' config

configitems: register the 'worker.backgroundclosemaxqueue' config

configitems: register the 'patch.eol' config

configitems: register the 'server.zliblevel' config

configitems: register the 'server.validate' config

configitems: register the 'server.uncompressedallowsecret' config

configitems: register the 'server.preferuncompressed' config

configitems: register the 'server.maxhttpheaderlen' config

configitems: register the 'server.disablefullbundle' config

configitems: register the 'server.concurrent-push-mode' config

configitems: register the 'server.compressionengines' config

configitems: register the 'server.bundle1gd' config

configitems: register the 'server.bundle1' config

configitems: register the 'hostsecurity.disabletls10warning' config

configitems: register the 'hostsecurity.ciphers' config

configitem: create a new list of each 'acl.sources' access Thanks for goes to Yuya for spotting this.

dirstate: centralize _cwd handling into _cwd method Before this patch, immediate value is assigned to dirstate._cwd, if ui.forcecwd is specified at instantiation of dirstate. But this doesn't work as expected in some cases. For example, hgweb set ui.forcecwd after instantiation of repo object. If an extension touches repo.dirstate in its reposetup(), dirstate is instantiated without setting ui.forcecwd, and dirstate.getcwd() returns incorrect result. In addition to it, hgweb.__init__() can take already instantiated repo object, too. In this case, repo.dirstate might be already instantiated, even if all enabled extensions don't so in their own reposetup(). To avoid such issue, this patch centralizes _cwd handling into _cwd method. This issue can be reproduced by running test-hgweb-commands.t with fsmonitor-run-tests.py.

tests: add line specific for testing with fsmonitor

tests: make output lines conditional for testing with fsmonitor Repository cloned-bookmark-default and tobundle exist in the working directory of main test repository "repo". We should take care for them, because it is known issue that fsmonitor can't handle nested repositories. These nested repositories are cloned from "repo", and the number of unknown files = files in these repositories (including files under .hg) will be changed easily in the future. But testing with fsmonitor is not ordinary. Therefore, test-bookmarks.t with fsmonitor might be broken silently. This is reason why this patch uses "(glob)" for the number of unknown files in "hg summary" output. BTW, this patch doesn't use .hgignore to make test portable, because .hgignore might cause another issue related to "walk_on_invalidate" configuration of fsmonitor.

tests: add fsmonitor specific output lines at enabling largefiles Temporarily enabling largefiles causes these output lines, only if tests are executed with fsmonitor.

show: document why accidentally quadratic is (probably) acceptable

show: also catch AmbiguousCommand cmdutil.findcmd() can raise this as well. While we'll almost certainly never encounter this in the wild, guard against it regardless.

show: avoid extra list operations

show: tweak warning message '.' is "working directory parent" not "working directory."

revlog: add an experimental option to mitigated delta issues (issue5480) The general delta heuristic to select a delta do not scale with the number of branch. The delta base is frequently too far away to be able to reuse a chain according to the "distance" criteria. This leads to insertion of larger delta (or even full text) that themselves push the bases for the next delta further away leading to more large deltas and full texts. This full text and frequent recomputation throw Mercurial performance in disarray. For example of a slightly large repository 280 000 files (2 150 000 versions) 430 000 changesets (10 000 topological heads) Number below compares repository with and without the distance criteria: manifest size: with: 21.4 GB without: 0.3 GB store size: with: 28.7 GB without 7.4 GB bundle last 15 00 revisions: with: 800 seconds 971 MB without: 50 seconds 73 MB unbundle time (of the last 15K revisions): with: 1150 seconds (~19 minutes) without: 35 seconds Similar issues has been observed in other repositories. Adding a new option or "feature" on stable is uncommon. However, given that this issues is making Mercurial practically unusable, I'm exceptionally targeting this patch for stable. What is actually needed is a full rework of the delta building and reading logic. However, that will be a longer process and churn not suitable for stable. In the meantime, we introduces a quick and dirty mitigation of this in the 'experimental' config space. The new option introduces a way to set the maximum amount of memory usable to store a diff in memory. This extend the ability for Mercurial to create chains without removing all safe guard regarding memory access. The option should be phased out when core has a more proper solution available. Setting the limit to '0' remove all limits, setting it to '-1' use the default limit (textsize x 4).

tests: use system hg only if changelog or dirstate can't be read The bundled hg should work flawlessly in most cases. Make it depend on the external installation only if necessary since we can't control the whole environment. This patch doesn't implement the "exit 80" idea proposed by Jun. I don't want to keep the capability checking sync with the actual tests.

tests: restore workaround of obsolete warning from 3c9066ed557c It's simple and works well unless you are using third-party extensions that changes the store format.

tests: alias syshg and syshgenv so they can be switched conditionally

tests: actually restore the original environment before running syshg Since os.environ may be overridden in run-tests.py, several important variables such as PATH weren't restored. I don't like the idea of using the system hg *by default* because the executable and the configs are out of our control. But I don't mind as long as the tests pass.

Added signature for changeset 26c49ed51a69

Added tag 4.2.2 for changeset 26c49ed51a69

phabricator: add phabread command to read patches This patch adds a `phabread` command generating plain-text patches from Phabricator, suitable for `hg import`. It respects `hg:meta` so user and date information might be preserved. And it removes `Summary:` field name which makes the commit message a bit tidier. To support stacked diffs, a `--stack` flag was added to read dependent patches recursively.

phabricator: add phabsend command to send a stack The `phabsend` command is intended to provide `hg email`-like experience - sending a stack, setup dependency information and do not amend existing changesets. It uses differential.createrawdiff and differential.revision.edit Conduit API to create or update a Differential Revision. Local tags like `D123` are written indicating certain changesets were sent to Phabricator. The `phabsend` command will use obsstore and tags information to decide whether to update or create Differential Revisions.

phabricator: add a contrib script The default Phabricator client arcanist is not friendly to send a stack of changesets. It works better when a feature branch is reviewed as a single review unit. However, we want multiple revisions per feature branch. To be able to have an `hg email`-like UX to send and receive a stack of commits easily, it seems we have to re-invent things. This patch adds `phabricator.py` speaking Conduit API [1] in `contrib` as the first step. This may also be an option for people who don't want to run PHP. Config could be done in `hgrc` (instead of `arcrc` or `arcconfig`): [phabricator] # API token. Get it from https://phab.mercurial-scm.org/conduit/login/ token = cli-xxxxxxxxxxxxxxxxxxxxxxxxxxxx url = https://phab.mercurial-scm.org/ # callsign is used by the next patch callsign = HG This patch only adds a single command: `debugcallconduit` to keep the patch size small. To test it, having the above config, and run: $ hg debugcallconduit diffusion.repository.search <<EOF > {"constraints": {"callsigns": ["HG"]}} > EOF The result will be printed in prettified JSON format. [1]: Conduit APIs are listed at https://phab.mercurial-scm.org/conduit/

show: implement "stack" view People often want to know what they are working on *now*. As part of this, they also commonly want to know how that work is related to other changesets in the repo so they can perform common actions like rebase, histedit, and merge. `hg show work` made headway into this space. However, it is geared towards a complete repo view as opposed to just the current line of work. If you have a lot of in-flight work or the repo has many heads, the output can be overwhelming. The closest thing Mercurial has to "show me the current thing I'm working on" that doesn't require custom revsets is `hg qseries`. And this requires MQ, which completely changes workflows and repository behavior and has horrible performance on large repos. But as sub-optimal as MQ is, it does some things right, such as expose a model of the repo that is easy for people to reason about. This simplicity is why I think a lot of people prefer to use MQ, despite its shortcomings. One common development workflow is to author a series of linear changesets, using bookmarks, branches, anonymous heads, or even topics (3rd party extension). I'll call this a "stack." You periodically rewrite history in place (using `hg histedit`) and reparent the stack against newer changesets (using `hg rebase`). This workflow can be difficult because there is no obvious way to quickly see the current "stack" nor its relation to other changesets. Figuring out arguments to `hg rebase` can be difficult and may require highlighting and pasting multiple changeset nodes to construct a command. The goal of this commit is to make stack based workflows simpler by exposing a view of the current stack and its relationship to other releant changesets, notably the parent of the base changeset in the stack and newer heads that the stack could be rebased or merged into. Introduced is the `hg show stack` view. Essentially, it finds all mutable changesets from the working directory revision in both directions, stopping at a merge or branch point. This limits the revisions to a DAG linear range. The stack is rendered as a concise list of changesets. Alongside the stack is a visualization of the DAG, similar to `hg log -G`. Newer public heads from the branch point of the stack are rendered above the stack. The presence of these heads helps people understand the DAG model and the relationship between the stack and changes made since the branch point of that stack. If the "rebase" command is available, a `hg rebase` command is printed for each head so a user can perform a simple copy and paste to perform a rebase. This view is alpha quality. There are tons of TODOs documented inline. But I think it is good enough for a first iteration.

tests: demonstrate inconsistencies with dirty state in various commands Not only is the output of these commands inconsistent with respect to each other when a file is deleted, they are internally inconsistent depending upon whether the deleted file is in the top level repo or a subrepo. It seemed easier to show the problems, rather than describe them. The original goal was to fix the summary command with respect to deleted files. I haven't fixed any of the other issues yet, in case anybody believes the current subrepo behavior is correct. I think a natural understanding of clean/dirty is that they are two opposite values of a single binary repo state. If `hg update --clean -r .` changes a file, then naturally that repo was dirty, and `hg update --check` should have blocked it. Deleted files are special, in that they don't block a commit. But they make the filesystem content not the same as a clean checkout.

check-config: syntax to allow inconsistent config values The ignore regular expression has been updated to detect "inconsistent config." If present, we track which configs have that set and we suppress the conflicting defaults error for those options. I also added named groups to the regexp to aid readability. A comment was added to profiling.py to make a desired inconsistent value error go away.

configitems: register the 'badserver.closebeforeaccept' config

configitems: register the 'badserver.closeaftersendbytes' config

configitems: register the 'badserver.closeafterrecvbytes' config

configitems: register the 'badserver.closeafteraccept' config

configitems: register the 'blackbox.logsource' config

configitems: register the 'blackbox.dirty' config

configitems: register the 'automv.similarity' config Default value has been ported.

acl: use configlist to retrieve the source config This is what the previous code was about.