demandimport: add __future__ to ignore list __future__ is special. We should definitely not be trying to lazy load it.

demandimport: remove support for Python < 2.5 The removed code was to support an __import__ function that doesn't support the "level" argument. This argument was added in Python 2.5, which we no longer support.

dispatch: use absolute_import A mixed, ambiguous import has been removed!

commands: break import cycle This was the easiest link in the chain to break as there was only one use of commandserver in commands.py. Other files had multiple users.

cmdutil: break import cycle This was the easiest place to break the chain, as there were only 2 uses of the imported module in the file.

revset: make balanced addsets by orset() without using _combinesets() As scmutil.revrange() was rewritten to not use _combinesets(), we no longer need _combinesets().

revrange: evaluate all revset specs at once This provides an opportunity for revset to optimize given expressions. For example, "-r0 -r1 -r2" can be optimized to "_list(0 1 2)".

revset: add matchany() to construct OR expression from a list of specs This will allow us to optimize "-rREV1 -rREV2 ..." command-line options.

revset: split post-parsing stage from match() _makematcher() will be reused by new matchany(ui, specs, repo=None) function I'll add by the next patch.

filesets: ignore unit case in size() predicate for single value When specifying one plain value in size(), e.g. size(1k), fileset tries to guess the upper bound automatically (see the comment in _sizetomax()). It didn't ignore the specified unit's case, and so size("1 GB"), for example, produced this error: hg: parse error: couldn't parse size: 1 GB Let's do the same thing that util.sizetoint() does: .lower(). The two test lines without output just check that there are no parse errors.

copies: use absolute_import

config: use absolute_import

changelog: use absolute_import

changegroup: use absolute_import

bundlerepo: use absolute_import

bundle2: use absolute_import

branchmap: use absolute_import

bookmarks: use absolute_import

archival: use absolute_import

ancestor: use absolute_import A few months ago, import-checker.py was taught to enforce a more well-defined import style for files with absolute_import. However, we stopped short of actually converting source files to use absolute_import because of problems with certain files. Investigation revealed the following problems with switching to absolute_import universally: 1) import cycles result in import failure on Python 2.6 2) undetermined way to import C/pure modules While these problems need to be solved, they can be put off. This patch starts a series of converting files to absolute_import that won't exhibit any of the aforementioned problems.

discovery: always use batching now that all peers support batching Some peers will transparently downgrade batched requests to non-batched ones, but that simplifies code for everyone using batching.

wireproto: make wirepeer look-before-you-leap on batching This means that users of request batching don't need to worry themselves with capability checking. Instead, they can just use batching, and if the remote server doesn't support batching for some reason the wirepeer code will transparently un-batch the requests. This will allow for some slight simplification in a handful of places. Prior to this change, largefiles would have been silently broken against a server which did not support batching.

batching: migrate basic noop batching into peer.peer "Real" batching only makes sense for wirepeers, but it greatly simplifies the clients of peer instances if they can be ignorant to actual batching capabilities of that peer. By moving the not-really-batched batching code into peer.peer, all peer instances now work with the batching API, thus simplifying users. This leaves a couple of name forwards in wirepeer.py. Originally I had planned to clean those up, but it kind of unclarifies other bits of code that want to use batching, so I think it makes sense for the names to stay exposed by wireproto. Specifically, almost nothing is currently aware of peer (see largefiles.proto for an example), so making them be aware of the peer module *and* the wireproto module seems like some abstraction leakage. I *think* the right long-term fix would actually be to make wireproto an implementation detail that clients wouldn't need to know about, but I don't really know what that would entail at the moment. As far as I'm aware, no clients of batching in third-party extensions will need updating, which is nice icing.

parsers: fix memory leak in compute_phases_map_sets PySet_Add increments the reference of the added object to the set, see: https://hg.python.org/cpython/file/2.6/Objects/setobject.c#l379 Before this patch we were forgetting to decrement the reference count after adding objects to the phaseset. This patch fixes the issue and makes the reference count right so that these objects can be properly garbage collected.

histedit: do not stay on a cleaned nodes on abort There is case where nodes are neither in tmpnodes nor leaf but still get removed. For example, if you used the "edit" action, made a commit and run --abort. The commit you made is not tracked by histedit, yet it will likely be cleaned up with its parent. The commit may not tracked because no replacements computations are done in the --abort case.

histedit: also update away from tmpnodes The working copy may be on a tmpnodes, we need to update away before it is stripped from the repository.

histedit: use revset to check if we need to update during abort The for loop is already quite more complicated than necessary and we are about to add some logic. Instead, we use a simple revset. Revset laziness should provide us with similar performance.

histedit: remove useless 'else' clause This 'else: pass' clause have no effect. We drop it for clarity.

histedit: make cleanupnode more robust The goal of this function is to strip content out of the repository. We do not really care if this content is visible or cleanup node not and we should proceed anyway. None of the internal actions are subject to this, however, a third party extension running arbitrary commands during histedit is affected by this.

convert: fix git convert using servers branches The conversion from git to hg was reading the remote branch list directly from the origin server. If the origin's branch had moved forward since the last git fetch, it would return a git hash which didn't exist locally, and therefore the branch was not converted. This changes it to rely on the local repo's refs/remotes list of branches instead, so it's completely cut off from the server.

revrange: drop old-style parser in favor of revset (API) Now revset can parse nullary ":" operator and existing "foo+bar" tags, we don't need the old-style parser. This means scmutil.revsingle(), revpair() and revrange() no longer accept a binary nodeid. An integer revision is still allowed as it isn't ambiguous.

tag: do not pass binary nullid to scmutil.revsingle() Future patches will remove the old-style parser that happen to accept a binary nodeid. A binary nodeid shouldn't be passed to scmutil.revrange() because it is ambiguous. For example, bin('20' * 19 + '30') is valid binary nodeid, but it can also be parsed as a revset expression, '0'.

revset: port parsing rule of old-style ranges from scmutil.revrange() The old-style parser will be removed soon.

debugrevspec: pass lookup function to visualize fallback mechanism The next patch will move the exceptional parsing of old-style ranges to revset.tokenize(). This patch will allow us to see the result tree. Note that the parsing result of '-a-b-c-' is incorrect at this changeset. It will be fixed soon.

help: fix typo familar -> familiar

highlight: exit early on textual and unknown files (issue3005) When highlight extension encountered files that pygments didn't recognize, it used to fall back to text lexer. Also, pygments uses TextLexer for .txt files. This lexer is noop by design. On bigger files, however, doing the noop highlighting resulted in noticeable extra CPU work and memory usage: to show a 1 MB text file, hgweb required about 0.7s more (on top of ~3.8s, Q8400) and consumed about 100 MB of RAM more (on top of ~150 MB). Let's just exit the function when it's clear that nothing will be highlighted. Due to how this pygmentize function works (it modifies the template in-place), we can just return from it and everything else will work as if highlight extension wasn't enabled.

histedit: extract a simpler function to process replacement on abort The process replacement is building a full mapping to allow moving bookmarks and creating obsolescence marker. We do not need the full logic for abort so we extract it. It will be useful as abort is missing some data about the replacement and can crash when third party extensions push it a bit too far.

merge with stable

exchange: fix dead assignment The assignment of the value from bundle2.processbundle() to 'r' is unused. It is currently the same as its third argument (if given), and since that argument may eventually go away (according to the method's docstring), let's reassign the return value to 'op' instead to better prepare for that.

exchange: s/phase/bookmark/ in _pushb2bookmarks()

histedit: backout ebb5bb9bc32e The faulty changeset use obsolescence marker to roll the repository back on --abort. This is a problematic approach because --abort should be as close as an actually transaction rollback as possible stripping all created data from the repository (cf `hg rebase --abort` stripping all created changesets). Instead ebb5bb9bc32e made all content created during the aborted histedit still available in the repository adding obsolescence marker to make them hidden. This will cause trouble to evolution user as a re-run of the same histedit (with success) will likely result in the very same node to be "recreated" while obsolescence marker would be in place for them. And canceling an obsoletion is still a fairly complicated process. This also rollback using obsmarkers instead of strip to clean up temporary node on successful histedit run because the two change were not split in separated changeset. Rolling that part back does not have significant consequence a will have to be resubmitted independently

test-bookmarks.t: avoid nested repo This is (a) pretty unnecessary and (b) breaks tests for the third-party hgwatchman extension, which doesn't support nested repos.

revlog: remove unused shaoffset constants Call sites were removed at 61c9bc3da402, "revlog: remove lazy index".

revlog: correct comment about size of v0 index format

merge with stable

Added signature for changeset 21aa1c313b05

Added tag 3.5 for changeset 21aa1c313b05

i18n: fix unclosed inline span in pt_BR This was causing test-gendoc.t to complain: WARNING: Inline interpreted text or phrase reference start-string without end-string.

merge with i18n

i18n-ja: synchronized with 7fcad0c4ef8c

convert: when converting from Perforce use original local encoding by default On Windows Perforce command line client uses default system locale to encode output. Using 'latin_1' causes locale-specific characters to be replaced with question marks. With this patch we will use default locale by default whilst allowing to specify it explicity with 'convert.p4.encoding' config option. This is a potentially breaking change for any scripts relying on output treated as in 'latin_1' encoding. Also because hgext.convert.convcmd overwrites detected default system locale with UTF-8 we had to introduce an import cycle in hgext.convert.p4 to retrieve originally detected encoding from hgext.convert.convcmd.

i18n-pt_BR: synchronized with 3e84f40232c7

convert: when getting file from Perforce concatenate data at the end As it turned out, even when getting relatively small files, concatenating string data every time when new chunk is received is very inefficient. Maintaining a string list of data chunks and concatenating everything in one go at the end seems much more efficient - in my testing it made getting 40 MB file 7 times faster, whilst converting of a particularly big changelist with some big files went down from 20 hours to 3 hours.

help: scripting help topic There are a lot of non-human consumers of Mercurial. And the challenges and considerations for machines consuming Mercurial is significantly different from what humans face. I think there are enough special considerations around how machines consume Mercurial that a dedicated help topic is warranted. I concede the audience for this topic is probably small compared to the general audience. However, lots of normal Mercurial users do things like create one-off shell scripts for common workflows that I think this is useful enough to be in the install (as opposed to, say, a wiki page - which most users will likely never find). This text is by no means perfect. But you have to start somewhere. I think I did cover the important parts, though.

convert: document convert.hg.startrev

transplant: restore dirstate correctly at unexpected failure Before this patch, transplant can't restore dirstate as expected at failure other than one while patching. This causes: - unexpected file status - dirstate refers already rollback-ed parent (only at failure of transplanting the 2nd or later revision) To restore dirstate correctly also at unexpected failure, this patch encloses scope of store lock and transaction by 'dirstateguard'. This is temporary fixing for stable branch. See DirstateTransactionPlan wiki page for detail about the future plan to treat dirstate consistently around scope boundary of transaction. https://mercurial.selenic.com/wiki/DirstateTransactionPlan This patch also adds 'if lock' examination for safety 'lock.release()', because creating 'dirstateguard' object may fail unexpectedly (e.g. IOError for saving dirstate). BTW, in the test script, putting section header '[extensions]' into '.hg/hgrc' is needed to fix incomplete disabling 'abort' extension at 4d1382fd96ff.

localrepo: make journal.dirstate contain in-memory changes before transaction Before this patch, in-memory dirstate changes aren't written out at opening transaction, even though 'journal.dirstate' is created directly from '.hg/dirstate'. Therefore, subsequent 'hg rollback' uses incomplete 'undo.dirstate' to restore dirstate, if dirstate is changed and isn't written out before opening transaction. In cases below, the condition "dirstate is changed and isn't written out before opening transaction" isn't satisfied and this problem doesn't appear: - "wlock scope" and "transaction scope" are almost equivalent e.g. 'commit --amend', 'import' and so on - dirstate changes are written out before opening transaction e.g. 'rebase' (via 'dirstateguard') and 'commit -A' (by separated wlock scopes) On the other hand, 'backout' may satisfy the condition above. To make 'journal.dirstate' contain in-memory changes before opening transaction, this patch explicitly invokes 'dirstate.write()' in 'localrepository.transaction()'. 'dirstate.write()' is placed before not "writing journal files out" but "invoking pretxnopen hooks" for visibility of dirstate changes to external hook processes. BTW, in the test script, 'touch -t 200001010000' and 'hg status' are invoked to make file 'c' surely clean in dirstate, because "clean but unsure" files indirectly cause 'dirstate.write()' at 'repo.status()' in 'repo.commit()' (see fe03f522dda9 for detail) and prevents from certainly reproducing the issue.

dirstate: ensure mv source is marked deleted when walking icasefs (issue4760) Previously, importing a case-only rename patch on a case insensitive filesystem caused the original file to be marked as '!' in status. The source was being forgotten properly in patch.workingbackend.close(), but the call it makes to scmutil.marktouched() then put the file back into the 'n' state (but it was still missing from the filesystem). The cause of this was scmutil._interestingfiles() would walk dirstate, and since dirstate was able to lstat() the old file via the new name, was treating this as a forgotten file, not a removed file. scmutil.marktouched() re-adds forgotten files, so dirstate got out of sync with the filesystem. This could be handled with less code in the "kind == regkind or kind == lnkkind" branch of dirstate._walkexplicit(), but this avoids filesystem accesses unless case collisions occur. _discoverpath() is used instead of normalize(), since the dirstate case is given first precedence, and the old file is still in it. What matters is the actual case in the filesystem.

extdiff: allow modifications in subrepos to be copied back This check was a legacy bit from when the file data was being fetched manually with 'ctx[wfn]', but archive() does that now. 49966b5ab16f seems to indicate that this avoided a problem where a merge adds a file to another branch, and that test still passes. Unfortunately, I don't see a way to create a test that modifies the file in the temporary directory before the command exits. I wonder if the os.lstat() call needs to be wrapped in an exception handler for the case where archive didn't create a file because the file didn't exist in that revision. But I wasn't able to trigger a problem without it on a real repository.

ignore: fix path concatenation of .hgignore on Windows Since 3de48ff62733, .hgignore is ignored on Windows because a pat may have a drive letter, but pathutil.join is posixpath.join. "z:\foo\bar/z:\foo\bar\.hgignore" Instead, this patch uses os.path.join() and util.localpath() to process both parts as file-system paths. Maybe we can remove os.path.join() at dirstate._ignore because 'include:' is resolved relative to repo root? It was introduced by a04c7b74b3d5.

repair: fix typo in warning message

monoblue: link to branches on summary page "Tags" and "Bookmarks" sections on this page already have the similar links, and so does the "Branches" on summary page in gitweb, so let's do the same in monoblue.

monoblue: fix branches link on help topics page

hgweb: show ages in repos' Last modified column in monoblue and gitweb Index page, which shows the list of available repositories, has a column where the last modification date for each repo is shown. paper, coal and spartan already show the dates in relative format (e.g. "2 weeks ago"), because these styles have the required process_dates() js function call in their footer templates, which are included on every page. But monoblue and gitweb styles have more things in the footer templates, such as repo name and its atom/rss links, so they don't include the footer on index page (as this page doesn't have a single repo context). Let's call process_dates() without including the footer.

ignore: fix include: rules depending on current directory (issue4759) When reading pattern files, we just call open(path), which is relative to the current directory. Let's fix this by resolving the paths before attempting to read the file.

test: move ignore test run into a subdirectory Previously the hgignore test just called hg init in the test directory. A future patch needs to test hgignore stuff from outside of the repo, so let's move the entire test repo into a subdirectory.

monoblue: show relative age of bookmarks In b24e5a708fad, age calculation was made dynamic (i.e. in javascript), but for some reason bookmarkentry template in monoblue/map got a wrong class. It resulted in /summary and /bookmarks pages always showing exact dates for bookmarks, no age calculation was performed. Let's fix this by using "age" class that is already used in branchentry and tagentry templates in the same map file. As usual, the exact date for such elements is still available in title attribute, so it shows in a tooltip on hover.

highlight: produce correct markup when there's a blank line just before EOF Due to how the colorized output from pygments was stripped of <pre> elements, when there was an empty line at the end of a file, highlight extension produced an incorrect markup (no closing tags from the fileline/annotateline template). It wasn't usually noticeable, because browsers were smart enough to see where the missing tags should've been, but in monoblue style it resulted in the last line having twice the normal height. Instead of awkwardly trying to strip outer <pre></pre> tags, let's make the formatter with nowrap=True, which should do what we need in pygments since at least 0.5 (2006-10-30). Example from monoblue style: Before: <div class="source"> <div style="font-family:monospace" class="parity0"> <pre><a class="linenr" href="#l1" id="l1"> 1</a> </pre> </div> <div style="font-family:monospace" class="parity1"> <pre><a class="linenr" href="#l2" id="l2"> 2</a> </div> Now: <div class="source"> <div style="font-family:monospace" class="parity0"> <pre><a class="linenr" href="#l1" id="l1"> 1</a> </pre> </div> <div style="font-family:monospace" class="parity1"> <pre><a class="linenr" href="#l2" id="l2"> 2</a> </pre> </div> </div> (Notice the missing </pre></div> now in place)

test-check-config: convert directory separators to '/' for MSYS The output of the files command uses native separator. MSYS then seems to drop the '\' on Windows when invoking python: --- c:/Users/Matt/Projects/hg/tests/test-check-config-hg.t +++ c:/Users/Matt/Projects/hg/tests/test-check-config-hg.t.err @@ -6,22 +6,10 @@ $ hg files "set:(**.py or **.txt) - tests/**" | > xargs python contrib/check-config.py Traceback (most recent call last): File "contrib/check-config.py", line 93, in <module> sys.exit(main(sys.argv[1:])) File "contrib/check-config.py", line 24, in main for l in open(f): IOError: [Errno 2] No such file or directory: 'contriball-revsets.txt' [123]

test-subrepo: fix globs for Windows It looks like these changes originated with 2538b87660be. I'm not sure that it was intentional, but is seems harmless enough for an error message.

test-clone: fix globs for Windows

tests: diff -r on Solaris emits "Common subdirectories:" When using the -r option to Solaris diff, any directores that compare identically are mentioned in the output. We don't really care about these directories for the purposes of this test, so ignore them.

tests: work around differing hunk headers between GNU and Solaris diff The hunk headers specifying what lines the hunk apply to differ. They're irrelevant to the test, so just eliminate them from the output.

tests: use egrep with regular expression branches for compatibility GNU grep allows you to use "a\|b" in a regular expression to match either "a" or "b", but at least Solaris grep does not; only egrep allows for that. And egrep considers "a+" to be "a{1,}" instead of an "a" and a literal plus sign, so escape that as well.

parsers: silence warning of implicit integer conversion issued by clang "-Wshorten-64-to-32" is enabled by default on Mac OS X. Because "len" should be represented in 32bit integer, this patch simply cast ssize_t to int.

tests: disable test of buffer overflow in parsers.c if --pure It fails with AttributeError and there's no benefit to make it runnable with pure Python code.

win32: update Inno Setup script after the changes done in 95e042d77a5f

Added signature for changeset 96a38d44ba09

Added tag 3.5-rc for changeset 96a38d44ba09

merge with default for code freeze

tests: add a check-config pass

hgweb: document web.certificates option

shelve: make maxbackup doc check-config friendly

share: make option docs more check-config friendly

perf: mark experimental option presleep

check-config: don't continue prematurely The early check for section headers like ^``foo`` was missing actual options like ^``foo.bar``.

subrepo: mark internal-only option

progress: mark experimental option

censor: mark experimental option

bundle2: fix type of experimental option

merge: make merge.preferancestor type and default consistent (and mark it)

merge: mark ancient debugging option

email: fix config default value inconsistency

bookmarks: mark internal-only option

commit: mark internal-only option

localrepo: mark format options

formatter: mark developer options

http2: mark experimental and developer options

bookmarks: mark internal-only config option

filemerge: mark internal-only config option

profiler: mark developer-only config option

debugger: mark developer-only option (and rearrange comment)

commandserver: mark developer-only logging option

generaldelta: mark experimental reordering option

bundlerepo: mark internal-only config variable

win32text: mark deprecated extension option deprecated

transplant: mark some undocumented options deprecated

mq: tweak config reading to make check-config happy The not-really-a-bool handling here upsets the type checker.

patchbomb: make sure all users of smtp.verifycert agree on the default

patchbomb: mark ancient option deprecated This just exists for backwards compatibility with the earliest versions of patchbomb.

histedit: mark defaultrev option experimental

changegroup: compute seen files as changesets are added (issue4750) Before this patch, addchangegroup() would walk the changelog and compute the set of seen files between applying changesets and applying manifests. When cloning large repositories such as mozilla-central, this consumed a non-trivial amount of time. On my MBP, this walk takes ~10s. On a dainty EC2 instance, this was measured to take ~125s! On the latter machine, this delay was enough for the Mercurial server to disconnect the client, thinking it had timed out, thus causing a clone to abort. This patch enables the changelog to compute the set of changed files as new revisions are added. By doing so, we: * avoid a potentially heavy computation between changelog and manifest processing by spreading the computation across all changelog additions * avoid extra reads from the changelog by operating on the data as it is added The downside of this is that the add revision callback does result in extra I/O. Before, we would perform a flush (and subsequent read to construct the full revision) when new delta chains were created. For changelogs, this is typically every 2-4 revisions. Using the callback guarantees there will be a flush after every added revision *and* an open + read of the changelog to obtain the full revision in order to read the added files. So, this increases the frequency of these operations by the average chain length. In the future, the revlog should be smart enough to know how to read revisions that haven't been flushed yet, thus eliminating this extra I/O. On my MBP, the total CPU times for an `hg unbundle` with a local mozilla-central gzip bundle containing 251,934 changesets and 211,065 files did not have a statistically significant change with this patch, holding steady around 360s. So, the increased revlog flushing did not have an effect. With this patch, there is no longer a visible pause between applying changeset and manifest data. Before, it sure felt like Mercurial was lethargic making this transition. Now, the transition is nearly instantaneous, giving the impression that Mercurial is faster. Of course, eliminating this pause means that the potential for network disconnect due to channel inactivity during the changelog walk is eliminated as well. And that is the impetus behind this change.

revlog: add support for a callback whenever revisions are added A subsequent patch will add a feature that performs iterative computation as changesets are added from a changegroup. To facilitate this type of processing in a generic manner, we add a mechanism for calling a function whenever a revision is added via revlog.addgroup(). There are potential performance concerns with this callback, as using it will flush the revlog after every revision is added.

crecord: throws error instead of crashing for large diffs Before this patch, crecord was crashing for large diffs (30k lines on my laptop). This patch catches the exception raised in that case and use the error reporting mechanism introduced in the previous patch for notifying the user of the issue. It is not possible to add a test for that for now as we don't yet have full blown ui tests for the curses interface.

crecord: add error reporting for failure in curses interface initialization Before this patch, we couldn't report to the user any error that occurred: - after we enabled the curses interface but - before the interface is set up and drawn This patch, provides a way to set errors that happens during the initialization of the interface and log them once the curses interface has been displayed.

revset: parse nullary ":" operator as "0:tip" This is necessary for compatibility with the old-style parser that will be removed by future patches.

parser: take suffix action if no infix action is defined If no infix action is defined, a suffix action isn't ambiguous, so it should be taken no matter if the next token can be an operand. This is exactly the same flow as prefix/primary handling. This change has no effect now because all suffix tokens have infix actions.

parser: reorder infix/suffix handling to be similar to prefix/primary flow It can be exactly the same flow as the prefix/primary handling. A suffix action is accepted only if the next token never starts new term.

parser: resolve ambiguity where both prefix and primary actions are defined If both actions are defined, a primary-expression action is accepted only if the next token never starts new term. For example, parsed as primary expression: ":" # next token 'end' has no action "(:)" # next token ')' has no action ":+y" # next token '+' is infix operator parsed as prefix operator: ":y" # next token 'y' is primary expression ":-y" # next token '-' is prefix operator This is mostly the same resolution as the infix/suffix rules.

parser: separate actions for primary expression and prefix operator This will allow us to define both a primary expression, ":", and a prefix operator, ":y". The ambiguity will be resolved by the next patch. Prefix actions in elements table are adjusted as follows: original prefix primary prefix ----------------- -------- ----------------- ("group", 1, ")") -> n/a ("group", 1, ")") ("negate", 19) -> n/a ("negate", 19) ("symbol",) -> "symbol" n/a

changelog: update read pending documentation The pending index contains a full copy of the index + in-transaction data. We replace "extend" with "overwrite" to make this clearer.

extdiff: add support for subrepos Git and svn subrepo support is incomplete, because they don't support archiving the working copy.

extdiff: use archiver to take snapshots of committed revisions This is the last step before supporting extdiff -S. It maintains the existing behavior of diffing the largefile standins instead of the largefiles themselves. Note however that the standins are not updated immediately upon modification, so uncommitted largefile changes are ignored, as they previously were, even with the diff command.

largefiles: allow the archiving of largefiles to be disabled There are currently no users of this, but it is a necessary step before converting extdiff to use archive. It may be useful to add an argument to extdiff in the future and allow largefiles to be diffed, but archiving largefiles can have significant overhead and may not be very diffable, so archiving them by default seems wrong. It is a mystery to me why the lfstatus attribute needs to be set on the unfiltered repo. However if it is set on the filtered repo instead (and the filtered repo is passed to the original command), the lfstatus attribute is False in the overrides for archival.archive() and hgsubrepo.archive() when invoking the archive command. This smells like the buggy status behavior (see 67d63ec85eb7, which was reverted in df463ca0adef). Neither the status nor summary commands have this weird behavior in their respective overrides.

parsers: fix buffer overflow by invalid parent revision read from revlog If revlog file is corrupted, it can have parent pointing to invalid revision. So we should validate it before updating nothead[], phases[], seen[], etc. Otherwise it would cause segfault at best. We could use "rev" instead of "maxrev" as upper bound, but I think the explicit "maxrev" can clarify that we just want to avoid possible buffer overflow vulnerability.

histedit: mark temporary commits as obsolete when allowed to Before this patch, we were stripping temporary commits at the end of a histedit whether it was successful or not. If we can create obs markers, we should create them instead of stripping because it is faster and safer.

histedit: minor refactoring of createmarkers check We use a variable to store whether or not we can create obsolescence markers. It makes the patch series more readable as we are going to reuse this values in other places in the function.

crecord: fix issue when backgrounding editor would leave artefact Before this patch: - if a user was entering a commit message after having ran the curses interface - and then uses ctrl-z, followed by fg to put the editor in the background/foreground - then the curses interface would leave artefact on the screen of the editor, making entering the commit message a difficult task This happened because ncurses registers a signal handler for SIGTSTP and does not restore the original signal handler after running. More info at: http://stackoverflow.com/questions/31440392/ curses-wrapper-messing-up-terminal-after-background-foreground-sequence/ 31441709#31441709 This patch restores the original value of the signal handler after running the curses interface and therefore fixes this issue. It don't know how to add a test for this issue, I tested the scenario above manually and it works correctly with the patch.

censor: make various path forms available like other Mercurial commands Before this patch, censored file should be exactly "a path relative to repository root" regardless of current working directory, because "hg censor" passes "path" to "repo.file()" directly without any preparations. To make various path forms available like other Mercurial commands, this patch gets a target file path in the way of "hg parents FILE". Getting "wctx" is relocated to reuse "wctx" for efficiency.

convert: use 'default' for specifying branch name in branchmap (issue4753) A fix for issue2653 with 1d155582a8ea introduced a discrepancy how default branch should be denoted when converting with branchmap from different SCM. E.g. for Git and Mercurial you need to use 'default' whilst for Perforce and SVN you had to use 'None'. This changeset unifies 'default' for such purposes whilst falling back to 'None' when no 'default' mapping specified.

parser: extract function that tests if next token may start new term Future patches will separate primary expression and prefix operator actions. This function will be used to resolve ambiguity of them. This is a step to remove the old-style revexpr parser. We need both ":" and ":y" operators for backward compatibility.

parser: factor out function that parses right-hand side of prefix/infix ops These two had common pattern. The significant difference was just a result expression: prefix: (op-name, rhs) infix: (op-name, lhs, rhs)

parser: remove unused parameter 'pos' from _match() This backs out 9d1cf337a78d. The issue spotted by that changeset was addressed earlier by d4cafcb63f77.

parser: fill invalid infix and suffix actions by None This can simplify the expansion of (prefix, infix, suffix) actions.

parser: add comment about structure of elements to each table

shelve: omit incorrect 'commit' suggestion at 'hg shelve -i' Before this patch, 'hg shelve -i' under non-interactive mode suggests 'use commit instead', and it obviously incorrect, because what user wants to do isn't 'commit' but 'shelve'. To omit incorrect 'commit' suggestion at 'hg shelve -i', this patch specifies 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()'.

record: omit meaningless 'qrefresh' suggestion at 'hg qrefresh -i' Before this patch, 'hg qrefresh -i' under non-interactive mode suggests 'use qrefresh instead', and it obviously meaningless. To omit meaningless 'qrefresh' suggestion at 'hg qrefresh -i', this patch specifies 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()'.

record: omit meaningless 'qnew' suggestion at 'hg qnew -i' Before this patch, 'hg qnew -i' under non-interactive mode suggests 'use qnew instead', and it obviously meaningless. To omit meaningless 'qnew' suggestion at 'hg qnew -i', this patch adds internal function '_qrecord()' and specifies 'cmdsuggest' for each of 'qrecord' and 'qnew' separately.

record: omit meaningless 'commit' suggestion at 'hg commit -i' Before this patch, 'hg commit -i' under non-interactive mode suggests 'use commit instead', and it obviously meaningless. This patch makes 'record.record'()' examine 'ui.interactive()' and show suggestion by itself before calling 'commands.commit()'. This allows 'commands.commit()' to specify 'None' for 'cmdsuggest' argument of 'cmdutil.dorecord()' to omit meaningless 'commit' suggestion at 'hg commit -i'.

cmdutil: allow callers of cmdutil.dorecord to omit suggestion Interactive committing under non-interactive mode shows command suggestion, but sometimes it is meaningless. command suggestion usability ------------ ---------- ----------- record commit commit -i commit meaningless qrecord qnew qnew -i qnew meaningless qrefersh -i qrefresh meaningless shelve -i commit incorrect ------------ ---------- ----------- This patch allows callers of 'cmdutil.dorecord()' to omit meaningless suggestion by passing None or so for 'cmdsuggest' argument of it. This is a preparation for subsequent patches, which fix each suggestion issues above.

spartan: don't drop current revision in log/graph links Just to be consistent with log and shortlog links, graph links should have a revision context too. And the same goes for the graph page, where it's log and shortlog links that should have context.

hgk: tweak doc format for path option This lets the config checker see it.

acl: mark deprecated config option This option has been undocumented since day 0.

gpg: mention undocumented options

check-config: add config option checker This script scans files for lines that look like either ui.config usage or config variable documentation. It then ensures: - ui.config calls for each option agree on types and defaults - every option appears to be mentioned in documentation It doesn't complain about devel/experimental options and allows marking options that are not intended to be public. Since we haven't been able to come up with a good scheme for documenting config options at point of use, this will help close the loop of making sure all options that should be documented are.

bookmarks: clear active bookmark on non-linear update

convert: unescape Perforce-escaped special characters in filenames

convert: allow customizing git remote prefix Previously all git remotes were created as "remote/foo". This patch adds a configuration option for deciding what the prefix should be. This is useful if you want the bookmarks to be "origin/foo" like they are in git, or if you're integrating with the remotenames extension and don't want the local remote/foo bookmarks to overlap with the remote foo bookmarks.

help: rewrite template examples to not use shell escaping Though they work fine even on cmd.exe, these examples could lead to misguided understanding about the template syntax.

templater: unify "string" and "rawstring" "rawstring" was introduced by 5ab28a2e9962, but it's no longer necessary because c1975809a6b5 and fd5bc660c9f0 changed the way of processing string literals. This patch moves string decoding to the parsing phase as it was before: ('rawstring', s) -> ('string', s) ('string', s) -> ('string', s.decode('string-escape'))

templater: remove processing of "string" literals from tokenizer They are processed as "template" strings now.

templater: introduce one-pass parsing of nested template strings Instead of re-parsing quoted strings as templates, the tokenizer can delegate the parsing of nested template strings to the parser. It has two benefits: 1. syntax errors can be reported with absolute positions 2. nested template can use quotes just like shell: "{"{rev}"}" It doesn't sound nice that the tokenizer recurses into the parser. We could instead make the tokenize itself recursive, but it would be much more complicated because we would have to adjust binding strengths carefully and put dummy infix operators to concatenate template fragments. Now "string" token without r"" never appears. It will be removed by the next patch.

templater: check existence of closing brace of template string

templater: extract function that parses template string It will be called recursively to parse nested template strings.

templater: respect stop position while parsing template string It has no effect now because stop is len(tmpl).

hgweb: provide links to branches, tags and bookmarks by name (paper and coal) It's sometimes handy to, say, have a url always point to branch head, not just at the current branch head by node hash. Previously, this was only possible by manually editing url and replacing node hash with branch/tag/bookmark name. It wasn't very convenient, or easy - in case the name contained special characters that needed to be urlencoded. Let's have /branches, /tags and /bookmarks pages in paper and coal style provide links both to symbolic revisions and to node hashes. This feature was wished for in issue3594.

templates: introduce revescape filter for escaping symbolic revisions There needs to be a way to escape symbolic revisions containing forward slashes, but urlescape filter doesn't escape slashes at all (in fact, it is used in places where forward slashes must be preserved). The filter considers @ to be safe just for bookmarks like @ and @default to look good in urls.

hgweb: allow symbolic revisions with forward slashes in urls It's possible to have a branch/tag/bookmark with all kinds of special characters, such as {}/\!?. While not very conveniently, symbolic revisions with such characters work from command line if user correctly quotes the characters. These characters also work in hgweb, when they are properly encoded, with one exception: '/' (forward slash, urlencoded as '%2F'), which was getting decoded before hgweb could parse it as a part of PATH_INFO. Because of that, hgweb was seeing it as any other forward slash, that is, as just another url parts separator. For example, if user wanted to see the content of dir/file at bookmark 'feature/eggs', url could be: '/file/feature%2Feggs/dir/file'. But hgweb tried to find a revision 'feature' and get contents of 'eggs/dir/file'. To fix this, let's assume forward slashes are doubly-urlencoded (%252F), so CGI/WSGI server decodes it into %2F. Then we can decode %2F in the revision part of the url into an actual '/' character. Making hgweb produce such urls will be done in the next 2 patches.

convert: ignore case changes in vieworder for Perforce Perforce sometimes mixes the case resulting in files being ignored.

convert: if getting a file from Perforce fails try to get it one more time When converting a particularly large Perforce changelist (especially with some big files), it is very likely to run into an intermittent network issue (e.g. WSAECONNRESET or WSAETIMEDOUT) getting one of the files, which will result in the entire changelist converting being aborted. Which can be quite unfortunate since you might have waited hours getting all other files. To mitigate this let's attempt to get the file one more time, escalating original exception if that attempt fails.

shelve: keep old backups if timestamp can't decide exact order of them Before this patch, backups to be discarded are decided by steps below at 'hg unshelve' or so: 1. list '(st_mtime, filename)' tuples of each backups up 2. sort list of these tuples, and 3. discard backups other than 'maxbackups' ones at the end of list This doesn't work well in the case below: - "sort by name" order differs from actual backup-ing order, and - some of backups have same timestamp For example, 'test-shelve.t' satisfies the former condition: - 'default-01' < 'default-1' in "sort by name" order - 'default-1' < 'default-01' in actual backup-ing order Then, 'default-01' is discarded instead of 'default-1' unexpectedly, if they have same timestamp. This failure appears occasionally, because the most important condition "same timestamp" is timing critical. To avoid such unexpected discarding, this patch keeps old backups if timestamp can't decide exact order of them. Timestamp of the border backup (= the oldest one of recent 'maxbackups' ones) as 'bordermtime' is used to examine whether timestamp can decide exact order of backups.

subrepo: use vfs.dirname instead of os.path.dirname This patch uses "wvfs of the parent repository" ('pwvfs') instead of 'wvfs' of own repository, because 'self._path' is the path to this subrepository as seen from the parent repository.

vfs: add dirname

subrepo: use vfs.basename instead of os.path.basename

vfs: add basename

subrepo: use repo.pathto instead of util.pathto to simplify invocation This centralization into 'repo.pathto()' should reduce the cost of vfs migration around 'getcwd()' and so on in the future.

subrepo: prefetch ctx.repo() for efficiency and centralization 'subrepo.state()' refers same 'ctx.repo()' in many places and times.

revset: rename getkwargs() to getargsdict() This function was added recently at 48919d246a47, but its name was misleading because it processes both positional and keyword arguments.

revset: work around x:y range where x or y is wdir() All revisions must be contiguous in spanset, so we need the special case for the wdir revision.

revset: use integer representation of wdir() in revset This is the simplest way to handle wdir() revision in revset. None didn't work well because revset heavily depends on integer operations such as min(), max(), sorted(), x:y, etc. One downside is that we cannot do "wctx.rev() in set" because wctx.rev() is still None. We could wrap the result set by wdirproxyset that translates None to wdirrev, but it seems overengineered at this point. result = getset(repo, subset, tree) if 'wdir' in funcsused(tree): result = wdirproxyset(result) Test cases need the '(all() + wdir()) &' hack because we have yet to fix the bootstrapping issue of null and wdir.

localrepo: provide workingctx by integer revision This allows us to use the integer representation in revset. None doesn't work well while computing revset because revset heavily depends on and optimized for integer revisions. Still repo[wdirrev].rev() is None, which means the canonical form of the working-directory revision is None. This patch doesn't add the case for the wdirid because we can't handle short and ambiguous identifiers here. Perhaps, the wdirid will have to be handled in the changelog layer.

changeset_printer: change flush() to accept ctx instead of rev Because flush() is the function to write data buffered by show(ctx), flush(ctx) is more consistent than flush(rev). This makes sure that buffered header and hunk are always keyed by ctx.rev(). This patch will allow us to give an integer to the wdir while keeping wctx.rev() -> None.

changeset_printer: display wdirrev/wdirnode values for workingctx Because we want to eliminate "if"s in the default template, it makes sense to display wdirrev/wdirnode values for now. wdir() is still experimental, so the output of "log -r'wdir()'" may change in future.

hg: support for auto sharing stores when cloning Many 3rd party consumers of Mercurial have created wrappers to essentially perform clone+share as a single operation. This is especially popular in automated processes like continuous integration systems. The Jenkins CI software and Mozilla's Firefox release automation infrastructure have both implemented custom code that effectively perform clone+share. The common use case here is that clients want to obtain N>1 checkouts while minimizing disk space and network requirements. Furthermore, they often don't care that a clone is an exact mirror of a remote: they are simply looking to obtain checkouts of specific revisions. When multiple third parties implement a similar feature, it's a good sign that the feature is worth adding to the core product. This patch adds support for an easy-to-use clone+share feature. The internal "clone" function now accepts options to control auto sharing during clone. When the auto share mode is active, a store will be created/updated under the base directory specified and a new repository pointing to the shared store will be created at the path specified by the user. The share extension has grown the ability to pass these options into the clone command/function. No command line options for this feature are added because we don't feel the feature will be popular enough to warrant their existence. There are two modes for auto share mode. In the default mode, the shared repo is derived from the first changeset (rev 0) in the remote repository. This enables related repositories existing at different URLs to automatically use the same storage. In environments that operate several repositories (separate repo for branch/head/bookmark or separate repo per user), this has the potential to drastically reduce storage and network requirements. In the other mode, the name is derived from the remote's path/URL.

merge with stable

cmdutil: apply dirstate.normallookup on (maybe partially) committed files To detect change of a file without redundant comparison of file content, dirstate recognizes a file as certainly clean, if: (1) it is already known as "normal", (2) dirstate entry for it has valid (= not "-1") timestamp, and (3) mode, size and timestamp of it on the filesystem are as same as ones expected in dirstate This works as expected in many cases, but doesn't in the corner case that changing a file keeps mode, size and timestamp of it on the filesystem. The timetable below shows steps in one of typical such situations: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N *** *** - change "f" N - execute 'hg commit -i' - backup "f" with timestamp N - revert "f" by 'merge.update()' N with 'partially' - apply selected hunks N by 'patch.patch()' - 'repo.commit()' - 'dirstate.normal("f")' N N+1 - 'dirstate.write()' N N - restore "f" N+1 - restore timestamp of "f" N - 'hg status' shows "f" as "clean" N N N ---- ----------------------------------- ---- ----- ----- The most important point is that 'dirstate.write()' is executed at N+1 or later. This causes writing dirstate timestamp N of "f" out successfully. If it is executed at N, 'parsers.pack_dirstate()' replaces timestamp N with "-1" before actual writing dirstate out. This issue can occur when 'hg commit -i' satisfies conditions below: - the file is committed partially, and - mode and size of the file aren't changed before and after committing The root cause of this issue is that (maybe partially changed) files are restored with original timestamp but dirstate isn't updated for them. To detect changes of files correctly, this patch applies 'dirstate.normallookup()' on restored files. Status check is needed before 'dirstate.normallookup()', because status other than "n(ormal)" should be kept at failure of committing. This patch doesn't examine whether each files are committed fully or partially, because interactive hunk selection makes it difficult. After this change, timetable is changed as below: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N *** *** - change "f" N - execute 'hg commit -i' - backup "f" with timestamp N - revert "f" by 'merge.update()' N with 'partially' - apply selected hunks N by 'patch.internalpatch()' - 'repo.commit()' - 'dirstate.normal("f")' N N+1 - 'dirstate.write()' N N - restore "f" N+1 - restore timestamp of "f" N ----------------------------------- ---- ----- ----- - normallookup("f") -1 - release wlock - 'dirstate.write()' -1 -1 N ----------------------------------- ---- ----- ----- - 'hg status' shows "f" as "clean" -1 -1 N ---- ----------------------------------- ---- ----- ----- To reproduce this issue in tests certainly, this patch emulates some timing critical actions as below: - change "f" at N 'touch -t 200001010000' before command invocation changes mtime of "f" to "2000-01-01 00:00" (= N). - apply selected hunks at N 'patch.internalpatch()' with 'fakepatchtime.py' explicitly changes mtime of patched files to "2000-01-01 00:00" (= N). - 'dirstate.write()' at N+1 (or "not at N") 'pack_dirstate()' uses actual timestamp at runtime as "now", and it should be different from the "2000-01-01 00:00" of "f". BTW, in 'test-commit-interactive.t', files are sometimes treated as modified , even though they are just committed fully via 'hg commit -i' and 'hg diff' shows nothing for them. Enabling win32text causes EOL style mismatching below: - files are changed in LF style EOL => files restored after committing uses LF style EOL (1) - 'merge.update()' reverts files in CRLF style EOL - 'patch.internalpatch()' changes files in CRLF style EOL => 'dirstate.normal()' via 'repo.commit()' uses the size of files in CRLF style EOL (2) Therefore, fully committed files are treated as "modified", because 'lstat()' returns size of (1) restored files in LF style EOL, but dirstate expects size of (2) committed files in CRLF style EOL. After this patch, 'dirstate.normallookup()' on committed files forces subsequent 'hg status' to examine changes exactly, and fully committed files are treated as clean as expected. This is reason why this patch also does: - add some 'hg status' checking status of fully committed files - clear win32text configuration before size/timestamp sensitive examination

cmdutil: put recordfunc invocation into wlock scope for consistency Before this patch, 'recordfunc()' for interactive hunk selection does below outside wlock scope at 'hg commit -i' and so on: - backup files, which may be partially changed - apply selected hunks on files - restore files from backup-ed ones These should be executed inside wlock scope for consistency. To put them into wlock scope without largely changing indents in 'recordfunc()', this patch adds another wrapper function. This patch is also a preparation for subsequent patch fixing the issue to correctly recognize partially committed files as "modified".

context: write dirstate out explicitly at the end of markcommitted To detect change of a file without redundant comparison of file content, dirstate recognizes a file as certainly clean, if: (1) it is already known as "normal", (2) dirstate entry for it has valid (= not "-1") timestamp, and (3) mode, size and timestamp of it on the filesystem are as same as ones expected in dirstate This works as expected in many cases, but doesn't in the corner case that changing a file keeps mode, size and timestamp of it on the filesystem. The timetable below shows steps in one of typical such situations: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- * *** *** - 'hg transplant REV1 REV2 ...' - transplanting REV1 .... N - change "f", but keep size N (via 'patch.patch()') - 'dirstate.normal("f")' N *** (via 'repo.commit()') - transplanting REV2 - change "f", but keep size N (via 'patch.patch()') - aborted while patching N+1 - release wlock - 'dirstate.write()' N N N - 'hg status' shows "r1" as "clean" N N N ---- ----------------------------------- ---- ----- ----- The most important point is that 'dirstate.write()' is executed at N+1 or later. This causes writing dirstate timestamp N of "f" out successfully. If it is executed at N, 'parsers.pack_dirstate()' replaces timestamp N with "-1" before actual writing dirstate out. This issue can occur when 'hg transplant' satisfies conditions below: - multiple revisions to be transplanted change the same file - those revisions don't change mode and size of the file, and - the 2nd or later revision of them fails after changing the file The root cause of this issue is that files are changed without flushing in-memory dirstate changes via 'repo.commit()' (even though omitting 'dirstate.normallookup()' on files changed by 'patch.patch()' for efficiency also causes this issue). To detect changes of files correctly, this patch writes in-memory dirstate changes out explicitly after marking files as clean in 'committablectx.markcommitted()', which is invoked via 'repo.commit()'. After this change, timetable is changed as below: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- * *** *** - 'hg transplant REV1 REV2 ...' - transplanting REV1 .... N - change "f", but keep size N (via 'patch.patch()') - 'dirstate.normal("f")' N *** (via 'repo.commit()') ----------------------------------- ---- ----- ----- - 'dirsttate.write()' -1 -1 ----------------------------------- ---- ----- ----- - transplanting REV2 - change "f", but keep size N (via 'patch.patch()') - aborted while patching N+1 - release wlock - 'dirstate.write()' -1 -1 N - 'hg status' shows "r1" as "clean" -1 -1 N ---- ----------------------------------- ---- ----- ----- To reproduce this issue in tests certainly, this patch emulates some timing critical actions as below: - change "f" at N 'patch.patch()' with 'fakepatchtime.py' explicitly changes mtime of patched files to "2000-01-01 00:00" (= N). - 'dirstate.write()' via 'repo.commit()' at N 'fakedirstatewritetime.py' forces 'pack_dirstate()' to use "2000-01-01 00:00" as "now", only if 'pack_dirstate()' is invoked via 'committablectx.markcommitted()'. - 'dirstate.write()' via releasing wlock at N+1 (or "not at N") 'pack_dirstate()' via releasing wlock uses actual timestamp at runtime as "now", and it should be different from the "2000-01-01 00:00" of "f". BTW, this patch doesn't test cases below, even though 'patch.patch()' is used similarly in these cases: 1. failure of 'hg import' or 'hg qpush' 2. success of 'hg import', 'hg qpush' or 'hg transplant' Case (1) above doesn't cause this kind of issue, because: - if patching is aborted by conflicts, changed files are committed changed files are marked as CLEAN, even though they are partially patched. - otherwise, dirstate are fully restored by 'dirstateguard' For example in timetable above, timestamp of "f" in .hg/dirstate is restored to -1 (or less than N), and subsequent 'hg status' can detect changes correctly. Case (2) always causes 'repo.status()' invocation via 'repo.commit()' just after changing files inside same wlock scope. ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N *** *** - make file "f" clean N - execute 'hg foobar' .... - 'dirstate.normal("f")' N *** (e.g. via dirty check or previous 'repo.commit()') - change "f", but keep size N - 'repo.status()' (*1) (via 'repo.commit()') ---- ----------------------------------- ---- ----- ----- At a glance, 'repo.status()' at (*1) seems to cause similar issue (= "changed files are treated as clean"), but actually doesn't. 'dirstate._lastnormaltime' should be N at (*1) above, because 'dirstate.normal()' via dirty check is finished at N. Therefore, "f" changed at N (= 'dirstate._lastnormaltime') is forcibly treated as "unsure" at (*1), and changes are detected as expected (see 'dirstate.status()' for detail). If 'hg import' is executed with '--no-commit', 'repo.status()' isn't invoked just after changing files inside same wlock scope. But preceding 'dirstate.normal()' is invoked inside another wlock scope via 'cmdutil.bailifchanged()', and in-memory changes should be flushed at the end of that scope. Therefore, timestamp N of clean "f" should be replaced by -1, if 'dirstate.write()' is invoked at N. It means that condition of this issue isn't satisfied.

tests: add extension to emulate invoking internalpatch at the specific time This extension fakes "mtime" of patched files to emulate invoking 'patch.internalpatch()' at the specific time. This is useful to reproduce timing critical issues fixed in subsequent patches.

cmdutil: remove useless dirstate.normallookup() invocation in revert() Explicit 'dirstate.normallookup()' invocation in 'revert()' is useless now, because previous patch fixed the relevant issue by writing in-memory dirstate changes out at the end of dirty check. 'dirstate.normallookup()' invocation was introduced by 21b33f0460e0 to avoid occasional test failure (see issue4583 for detail). This is partial backout of it (added tests are still left).

merge: remove useless dirstate.normallookup() invocation in applyupdates() Explicit 'dirstate.normallookup()' invocation via 'dirtysubstate()' in 'applyupdates()' is useless now, because previous patch fixed the relevant issue by writing in-memory dirstate changes out at the end of dirty check. 'dirstate.normallookup()' invocation was introduced by 6becb9dbca25 to avoid occasional test failure. This is partial backout of it (added tests are still left).

context: write dirstate out explicitly after marking files as clean To detect change of a file without redundant comparison of file content, dirstate recognizes a file as certainly clean, if: (1) it is already known as "normal", (2) dirstate entry for it has valid (= not "-1") timestamp, and (3) mode, size and timestamp of it on the filesystem are as same as ones expected in dirstate This works as expected in many cases, but doesn't in the corner case that changing a file keeps mode, size and timestamp of it on the filesystem. The timetable below shows steps in one of typical such situations: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N -1 *** - make file "f" clean N - execute 'hg foobar' - instantiate 'dirstate' -1 -1 - 'dirstate.normal("f")' N -1 (e.g. via dirty check) - change "f", but keep size N N+1 - release wlock - 'dirstate.write()' N N - 'hg status' shows "f" as "clean" N N N ---- ----------------------------------- ---- ----- ----- The most important point is that 'dirstate.write()' is executed at N+1 or later. This causes writing dirstate timestamp N of "f" out successfully. If it is executed at N, 'parsers.pack_dirstate()' replaces timestamp N with "-1" before actual writing dirstate out. Occasional test failure for unexpected file status is typical example of this corner case. Batch execution with small working directory is finished in no time, and rarely satisfies condition (2) above. This issue can occur in cases below; - 'hg revert --rev REV' for revisions other than the parent - failure of 'merge.update()' before 'merge.recordupdates()' The root cause of this issue is that files are changed without flushing in-memory dirstate changes via 'repo.commit()' (even though omitting 'dirstate.normallookup()' on changed files also causes this issue). To detect changes of files correctly, this patch writes in-memory dirstate changes out explicitly after marking files as clean in 'workingctx._checklookup()', which is invoked via 'repo.status()'. After this change, timetable is changed as below: ---- ----------------------------------- ---------------- timestamp of "f" ---------------- dirstate file- time action mem file system ---- ----------------------------------- ---- ----- ----- N -1 *** - make file "f" clean N - execute 'hg foobar' - instantiate 'dirstate' -1 -1 - 'dirstate.normal("f")' N -1 (e.g. via dirty check) ----------------------------------- ---- ----- ----- - 'dirsttate.write()' -1 -1 ----------------------------------- ---- ----- ----- - change "f", but keep size N N+1 - release wlock - 'dirstate.write()' -1 -1 - 'hg status' -1 -1 N ---- ----------------------------------- ---- ----- ----- To reproduce this issue in tests certainly, this patch emulates some timing critical actions as below: - timestamp of "f" in '.hg/dirstate' is -1 at the beginning 'hg debugrebuildstate' before command invocation ensures it. - make file "f" clean at N - change "f" at N 'touch -t 200001010000' before and after command invocation changes mtime of "f" to "2000-01-01 00:00" (= N). - invoke 'dirstate.write()' via 'repo.status()' at N 'fakedirstatewritetime.py' forces 'pack_dirstate()' to use "2000-01-01 00:00" as "now", only if 'pack_dirstate()' is invoked via 'workingctx._checklookup()'. - invoke 'dirstate.write()' via releasing wlock at N+1 (or "not at N") 'pack_dirstate()' via releasing wlock uses actual timestamp at runtime as "now", and it should be different from the "2000-01-01 00:00" of "f". BTW, this patch also changes 'test-largefiles-misc.t', because adding 'dirstate.write()' makes recent dirstate changes visible to external process.

tests: add extension to emulate invoking dirstate.write at the specific time This extension fakes 'now' for 'parsers.pack_dirstate()' to emulate invoking 'dirstate.write()' at the specific time, only when 'dirstate.write()' is invoked via functions below: - 'workingctx._checklookup()' (= 'repo.status()') - 'committablectx.markcommitted()' This is useful to reproduce timing critical issues fixed in subsequent patches.

convert: handle copies when converting from Perforce (issue4744)

convert: add config for recording the source name This creates the convert.hg.sourcename config option which will embed a user defined name into each commit created by the convert. This is useful when using the convert extension to merge several repositories together and we want to record where each commit came from.

convert: support multiple specifed revs in git source This allows specifying multiple revs/branches to convert from a git repo.

convert: add support for specifying multiple revs Previously convert could only take one '--rev'. This change allows the user to specify multiple --rev entries. For instance, this could allow converting multiple branches (but not all branches) at once from git. In this first patch, we disable support for this for all sources. Future patches will enable it for select sources (like git).

monoblue: use padding instead of position for text in footer Some installations alter monoblue style and remove margins from body element (these margins have that dark gray background) to adapt hgweb instance to an already existing site design. However, the margins hid a quirk in page footer: a block of text needlessly popped out of the footer, and when margins were gone, the whole page got a vertical scroll bar because of that. Live example: https://hg.prosody.im/prosody-modules/ To remove the potential scroll bar, this block of text now uses left padding, which doesn't make it overflow the footer, but makes it achieve the otherwise same result visually.

monoblue: don't try to show repo on hgwebdir index page Index page shows a list of accessible repositories, it doesn't have a single-repo context.

convert: handle deleted files when converting from Perforce (issue4743)

bookmarks: change bookmark within a transaction For some time, bookmark can and should be moved in the transaction. This changeset migrates the 'hg bookmarks' commands to use a transaction. Tests regarding rollback and transaction hooks are impacted for obvious reasons. Some have to be slightly updated to keep testing the same things. Some can just be dropped because they do not make sense anymore.

bookmark: remove the "touch changelog" hack Any changes to bookmarks used to touch the changelog to ensure hgweb was reloaded. This was fairly hacky and stops working when bookmarks are moved as part of the transaction. As hgweb is now explicitly tracking bookmark changes, we can remove this hack (and no tests break).

convert: add config option for disabling ancestor parent checks When converting merge commits, convert checks if any of the parents are ancestors of any of the other parents. To do this, it builds an ancestor list for every commit in the repository. On large repos this can take a long time (30min+). Let's add an option for disabling this check to preserve performance. The downside of this is that it may create unnecessary parent connections when enabled (which is unfortunate, but not incorrect). To verify, I ran the convert tests with the flag enabled, and verified the graph changes were all just to add new parents that were ancestors of existing parents.

convert: add config to not convert tags In some cases we do not want to convert tags from the source repo to be tags in the target repo (for instance, in a large repository, hgtags cause scaling issues so we want to avoid them). This adds a config option to disable converting tags.

templatekw: make {rev} return wdirrev instead of None wdirrev/wdirnode identifiers are still experimental, but {node} is mapped to wdirnode. So {rev} should do the same for consistency. I'm not sure if templatekw can import scmutil. If not, we should move intrev() to node module.

changeset_printer: use node.wdirrev to calculate meaningful parentrevs Because we defined the working-directory revision is INT_MAX, it makes sense that "hg log -r 'wdir()'" displays the "parent:" field. This is the same for two revisions that are semantically contiguous but the intermediate revisions are hidden.

workingctx: use node.wdirid constant

node: define experimental identifiers for working directory The "ff..." node was introduced at 183965a00c76, and we also need an integer that can be processed in revset. We could use len(repo), but it would be likely to hide possible bugs. Instead, using INT_MAX, we can notice such bugs by IndexError, at the cost of handling non-contiguous revisions.

templatekw: apply manifest template only if ctx.manifestnode() exists This will prevent crash by "hg log -r 'wdir()' -Tdefault". We could use the pseudo ff... hash introduced by 183965a00c76, but it isn't proven idea yet. For now, I want to make "hg log" just works in order to test 'wdir()' revset. Note that unlike its name, "{manifest}" is not a list of files in that revision, but a pair of (manifestrev, manifestnode).

merge with stable

import-checker: exclude mercurial packages installed into the system path If mercurial was installed into a directory other than the site-packages, test-module-imports.t failed as 'mercurial.node' was listed in stdlib_modules: testpackage/latesymbolimport.py relative import of stdlib module Instead, we should exclude our packages explicitly.

import-checker: recurse into subtree of sys.path only if __init__.py exists We can't assume that the site-packages is the only directory that has Python files but is not handled as a package. For example, we have dist-packages directory on Debian.

hghave: allow adding customized features at runtime Before this patch, there is no way to add customized features to `hghave` without changing `hghave` and `hghave.py` themselves. This decreases reusability of `run-tests.py` framework for third party tools, because they may want to examine custom features at runtime (e.g. existence of some external tools). To allow adding customized features at runtime, this patch makes `hghave` import `hghaveaddon` module, only when `hghaveaddon.py` file can be found in directories below: - `TESTDIR` for invocation via `run-tests.py` - `.` for invocation via command line The path to the directory where `hghaveaddon.py` should be placed is added to `sys.path` only while importing `hghaveaddon`, because: - `.` may not be added to `PYTHONPATH` - adding additional path to `sys.path` may change behavior of subsequent `import` for other features `hghave` is terminated with exit code '2' at failure of `import hghaveaddon`, because exit code '2' terminates `run-tests.py` immediately. This is a one of preparations for issue4677.

import-checker.py: exit with code 0 if no error is detected Before this patch, `import-checker.py` exits with non-0 code, if no error is detected. This is unusual as Unix command. This change may be a one of preparations for issue4677, because this can avoid extra explanation about unusual exit code of `import-checker.py` for third party tool developers.

run-tests.py: add TESTDIR to PATH if it differs from RUNTESTDIR Before this patch, `RUNTESTDIR` is added to `PATH`, but `TESTDIR` isn't. This doesn't cause any problems, if `run-tests.py` runs in `tests` directory of Mercurial source tree. In this case, `RUNTESTDIR` should be equal to `TESTDIR`. On the other hand, if `run-tests.py` runs in `tests` of third party tools, commands in that directory should be executed with explicit `$TESTDIR/` prefix in `*.t` test scripts. This isn't suitable for the policy "drop explicit $TESTDIR from executables" of Mercurial itself (see 4d2b9b304ad0). BTW, 4d2b9b304ad0 describes that "$TESTDIR is added to the path" even though `TESTDIR` isn't added to `PATH` exactly speaking, because `TESTDIR` and `RUNTESTDIR` weren't yet distinguished from each other at that time. This is a one of preparations for issue4677.

run-tests.py: add RUNTESTDIR to refer `tests` of Mercurial Before this patch, there is no way to refer files under `tests` or so of Mercurial source tree, when `run-tests.py` runs in `tests` of third party tools. In this case, `TESTDIR` refers the latter `tests`. This prevents third party tools from using useful tools in Mercurial source tree (e.g. `contrib/check-code.py`). This patch adds `RUNTESTDIR` environment variable to refer `tests` of Mercurial source tree, in which `run-tests.py` now running is placed. For example, tests of third party tools can refer `contrib/check-code.py` in Mercurial source tree as `$RUNTESTDIR/../contrib/check-code.py`. BTW, for similarity with `TESTDIR` referring `test*s*` directory, newly added environment variable isn't named as `RUNTEST*S*DIR`. In addition to it, the corresponded local variable is also named as `runtestdir`. This is a one of preparations for issue4677.

run-tests.py: execute hghave by the path relative to run-tests.py Before this patch, `run-tests.py` executes `hghave` by the path relative to `TESTDIR` (= cwd of `run-tests.py` running). This prevents third party tools for Mercurial from running `run-tests.py`, which is placed in `tests` of Mercurial source tree, in `tests` of own source tree. In such cases, `TESTDIR` refers the latter `tests`, and `hghave` doesn't exist in it. This is a one of preparations for issue4677.

templatekw: make {latesttag} a hybrid list This maintains the previous behavior of expanding {latesttag} to a string containing all of the tags, joined by ':'. But now it also allows list type operations. I'm unsure if the plural handling is correct (i.e. it seems like it is usually "{foos % '{foo}'}"), but I guess we are stuck with this because the singular form previously existed.

templatekw: allow the caller of showlist() to specify the join() separator The keyword {latesttag} currently manually joins the list of tags using ':', which prevents a transparent switch over to a hybrid list.

archive: use {changessincelatesttag} to build the metadata file This isolates the current magic when dealing with wdir() and only().

templatekw: introduce the changessincelatesttag keyword Archive is putting a value with the same name in the metadata file, to count all of the changes not covered by the latest tag, instead of just along the longest path. It seems that this would be useful to have on the command line as well. It might be nice for the name to start with 'latesttag' so that it is grouped with the other tag keywords, but I can't think of a better name. The initial version of this counted a clean wdir() and '.' as the same value, and a dirty wdir() as the same value after it is committed. Yuya objected on the grounds of consistency [1]. Since revsets can be used to conditionally select a dirty wdir() or '.' when clean, I can build the version string I need and will defer to him on this. [1] https://www.selenic.com/pipermail/mercurial-devel/2015-June/071588.html

help: support 'hg help template.somekeyword' Previously the output was simply 'abort: help section not found'.

test-convert-git: use a relative gitmodule url The absolute URL was causing this error with 1.9.5 on Windows, which had a cascading effect: @@ -466,22 +466,24 @@ > url = $TESTTMP/git-repo5 > EOF $ git commit -a -m "weird white space submodule" - [master *] weird white space submodule (glob) - Author: nottest <test@example.org> - 1 file changed, 3 insertions(+) + fatal: bad config file line 6 in $TESTTMP/git-repo6/.gitmodules + [128] $ cd .. $ hg convert git-repo6 hg-repo6 initializing destination hg-repo6 repository scanning source... For reasons unknown, there is still this delta on Windows: @@ -490,7 +490,6 @@ $ git commit -q -m "missing .gitmodules" $ cd .. $ hg convert git-repo6 hg-repo6 --traceback - fatal: Path '.gitmodules' does not exist in '*' (glob) initializing destination hg-repo6 repository scanning source... sorting...

test-convert-git: stablize for git 1.7.7.6 The output has apparently changed slightly since this version. Since they are just commits without any obvious importance to the test, and I can't figure out how to glob them away, silence them. Sample diffs were like this: @@ -468,7 +468,7 @@ $ git commit -a -m "weird white space submodule" [master *] weird white space submodule (glob) Author: nottest <test@example.org> - 1 file changed, 3 insertions(+) + 1 files changed, 3 insertions(+), 0 deletions(-) $ cd .. $ hg convert git-repo6 hg-repo6 initializing destination hg-repo6 repository

hgweb: also monitor change to bookmarks This makes changes to bookmarks visible to hgweb through the official way. There is no change to tests because there is currently another hack in place to ensure the same behavior.

hgweb: also refresh the repo on changes to the obsstore Before this change, hgweb could miss update to the obsolescence markers store if that was the only change between two commands.

hgweb: use an extensible list of files to check for refresh The refresh feature was explicitly testing if '00changelog.i' and 'phaseroots' changed. This is overlooking other important information like bookmarks and obsstore (bookmark have their own hack to work around it). We move to a more extensible system with a list of files of interest that will be used to build the repo state. The system should probably move into a more central place so that the command server and other systems are able to use it. Extension writers will also be able to add entries to ensure that changes to extension data are properly detected. Also the current key (mtime, size) is notably weak for bookmarks and phases whose files can easily change content without effect on their size. Still, this patch seems like a valuable minimal first step.

hgweb: drop the default argument for get_stat This default argument is used twice and is making things confusing. Making it explicit helps to clarify coming changesets

revset: prefetch method in "parents" As already demonstrated, saving attribute lookup gains us some minor but noticeable performance improvements. revset #0: parents(all()) before) 0.024169 after ) 0.022756 94%

convert: handle deleted files when converting from Perforce (issue4743)

forget: add a note to the command help about remove

shelve: only keep the latest N shelve backups This will keep the backup directory from growing indefinitely. The number of backups to keep can be set using the shelve.maxbackups config option (defaults to 10 backups).

shelve: always backup shelves instead of deleting them Instead of being deleted, shelve files are now moved into the .hg/shelve-backup directory. This is designed similarly to how strip saves backups into .ht/strip-backup. The goal is to prevent data loss especially when using unshelve. There are cases in which a user can complete an unshelve but lose some of the data that was shelved by, for example, resolving merge conflicts incorrectly. Storing backups will allow the user to recover the data that was shelved, at the expense of using more disk space over time.

wireproto: remove a debug print This looks like someone forgot something here.

amend: move obsmarkers creation in the "new changeset" conditional We already check if we created a new changesets right above this piece of code, so we can just drop the condition and indent the markers creation.

amend: move createmarkers evaluation earlier The value is used at multiple points in the function. Retrieving the value in the middle of the transaction scope gives the false impression that it has a single user. We move it at the start of the function to clarify this.

wireproto: correctly escape batched args and responses (issue4739) This issue appears to be as old as wireproto batching itself: I can reproduce the failure as far back as 08ef6b5f3715 trivially by rebasing the test changes in this patch, which was back in the 1.9 era. I didn't test before that change, because prior to that the testfile has a different name and I'm lazy. Note that the test thought it was checking this case, but it actually wasn't: it put a literal ; in the arg and response for its greet command, but the mangle/unmangle step defined in the test meant that instead of "Fo, =;o" going over the wire, "Gp-!><p" went instead, which doesn't contain any special characters (those being [.=;]) and thus not exercising the escaping. The test has been updated to use pre-unmangled special characters, so the request is now "Fo+<:o", which mangles to "Gp,=;p". I have confirmed that the test fails without the adjustment to the escaping rules in wireproto.py. No existing clients of RPC batching were depending on the old behavior in any way. The only *actual* users of batchable RPCs in core were: 1) largefiles, wherein it batches up many statlfile calls. It sends hexlified hashes over the wire and gets a 0, 1, or 2 back as a response. No risk of special characters. 2) setdiscovery, which was using heads() and known(), both of which communicate via hexlified nodes. Again, no risk of special characters. Since the escaping functionality has been completely broken since it was introduced, we know that it has no users. As such, we can change the escaping mechanism without having to worry about backwards compatibility issues. For the curious, this was detected by chance: it happens that the lz4-compressed text of a test file for remotefilelog compressed to something containing a ;, which then caused the failure when I moved remotefilelog to using batching for file content fetching.

merge with stable

revset: port extra() to support keyword arguments This is an example to show how keyword arguments are processed.

revset: add function to build dict of positional and keyword arguments Keyword arguments will be convenient for functions that will take more than one optional or boolean flags. For example, file(pattern[, subrepos=false]) subrepo([[pattern], status]) Because I don't think all functions should accept key=value syntax, getkwargs() does not support variadic functions such as 'ancestor(*changeset)'. The core logic is placed in the parser module because keyword arguments will be more useful in the templater, where functions take more options. Test cases will be added by the next patch.

revset: add parsing rule for key=value pair It will be used as an keyword argument. Note that our "=" operator is left-associative. In general, the assignment operator is right-associative, but we don't care because it isn't allowed to chain "=" operations.

import-checker: establish modern import convention We introduce a new convention for declaring imports and enforce it via the import checker script. The new convention is only active when absolute imports are used, which is currently nowhere. Keying off "from __future__ import absolute_import" to engage the new import convention seems like the easiest solution. It is also beneficial for Mercurial to use this mode because it means less work and ambiguity for the importer and potentially better performance due to fewer stat() system calls because the importer won't look for modules in relative paths unless explicitly asked. Once all files are converted to use absolute import, we can refactor this code to again only have a single import convention and we can require use of absolute import in the style checker. The rules for the new convention are documented in the docstring of the added function. Tests have been added to test-module-imports.t. Some tests are sensitive to newlines and source column position, which makes docstring testing difficult and/or impossible.

import-checker: establish new function for verifying import conventions A future patch will formalize the modern import convention. In preparation for that, introduce a new wrapper function that will invoke the proper function.

import-checker: resolve relative imports "from . import X" will produce an ImportFrom ast node with .module = None. This resulted in a run-time error from attempting to concatenate None with a str. Another problem with relative imports is that the prefix may be dynamic based on the "level" attribute of the import. e.g. "from ." has level 1 and "from .." has level 2. We teach the "fromlocal" function how to cope with relative imports. Where appropriate, the consumer passes in the level so relative module names may be resolved properly.

templatekw: use a list of tags in getlatesttags() instead of joining them This will be used in the next patch. It also points out that the documentation for '{latesttag}' is not quite accurate, since it says "most recent global tag" (singular). I assume it is too radical of a change to convert it to a list of strings. At least ':' is currently a reserved character in tag names.

convert: improve support for unusual .gitmodules Previously convert would throw an exception if it encountered a git commit with a .gitmodules file that was malformed (i.e. was missing, but had submodule files, or was malformed). Instead of breaking the convert entirely, let's print error messages and move on.

convert: handle .gitmodules with non-tab whitespaces The old implementation assumed .gitmodules file lines always began with tabs. It can be any whitespace, so lets trim the lines appropriately.

convert: fix bug with converting the same commit twice Convert had a bug where it relied on repo.tip() to be the newly committed commit. This was not the case if the commit already existed in the repository (since repo.commitctx() did nothing, the tip() referenced some random other commit and the revmap got corrupted). This fixes it by using the node returned by repo.commitctx().

templater: remove workaround for escaped quoted string in quoted template This patch backs out 554d6fcc3c84 which should no longer be needed. The test for '{\"invalid\"}' is removed because the parser is permissive for \"...\" literal.

merge with stable

amend: stop updating the bookmarks twice There was code to move the bookmarks around both in the 'cmdutil' help and in the main 'commit' function. We kill the 'commit' version as it is performed outside the transaction. The debug note is moved into cmdutil.

amend: collaborate with the transaction when moving bookmarks We have code moving bookmarks from the old changeset to the new one within the transaction scope. Yet this code was still writing to disk instead of handing the change to the transaction. This changeset fixes this.

sshserver: drop ancient do_{lock,unlock,addchangegroup} methods These were marked as deprecated and dangerous way back in e8c4f3d3df8c, which was first included in Mercurial 0.9.1. While it's possible that clients from that long ago are still around somewhere, they're risky for servers in that they want to lock the repo, and then might leave it locked if they died before finishing their transaction. Given that it's been 9 years, let's go ahead and cut this last lingering tie with a basically-untested protocol.

wireproto: add config knob for http header length limit Well-behaved Mercurial clients will respect the httpheader capability by not sending http headers longer than the given limit in bytes. The limit is currently hard-coded at 1024 bytes, a safe value for any web server. Since parsing headers is a notable factor in web server performance, tuning header size can nontrivially improve performance for request-heavy operations (eg. obsolete marker negotiation). Exposing the maximum header length limit as a configuration setting is a simple way to enable such tuning.

archive: fix changesincelatesttag with wdir()

revset: fix a crash in parents() when 'wdir()' is in the set The crash was "TypeError: expected string or Unicode object, NoneType found" down in revlog.parentrevs(). This fixes heads() too (which is where I found it.)

workingctx: don't report the tags for its parents This fixes the bad distance calculation for '{latesttagdistance}' mentioned in the previous patch.

hgweb: don't try to show repo on repo-not-found page in monoblue

hgweb: fix help pages title in gitweb and monoblue Help pages in gitweb and monoblue were setting a completely inappropriate title: "Branches". Let's have a proper title (copy-pasted from paper style).

hgweb: don't show summary link as active on error pages in monoblue These two error pages in monoblue think they are the summary page, when they clearly aren't.

identify: build the tag list directly instead of using wctx.tags() The current implementation of workingctx.tags() returns the tags of the parents. This causes the calculation of {lastesttagdistance} from wdir() to be wrong. The value when updated to a tag is 0, but updated to the tag's child is 2, the child of that 3, and so on. This prepares for workingctx.tags() to not report the parent tags.

identify: avoid a crash when given '-r wdir()' The crash was 'NoneType is not subscriptable' in hexfunc(ctx.node()), because the node for wdir() is None. This can be avoided simply by detecting 'wdir()' and taking the existing path for no given revision.

tests: clean up duplicated output in test-subrepo-recursion progress We have fixed a bug where two progress instance were created and competed with each other (in 95f490136e75). But we did not updated the non-hardlink section of 'test-subrepo-recursion.t'. This patch fixes it.

Added signature for changeset 540cd0ddac49

Added tag 3.4.2 for changeset 540cd0ddac49

transplant: only pull the transplanted revision (issue4692) For some reason, transplant was pulling all remote revisions when transplanting from a remote repository (unless --branch was specified).

transplant: update test to use hash for remote transplant Transplant is apparently allowing using revision numbers when transplanting through http. I call this nonsense and update the test to use an explicit hash. This "feature" will break in the next change fixing another bug.

changegroup: properly compute common base in changeggroupsubset (issue4736) The computation of roots was buggy, any ancestor of a bundled merge which was also a descendant of the parents of a bundled revision were included as part of the bundle. We fix it and add a test for strip (which revealed the problem). Check the test for a practical usecase.

templater: parse \"...\" as string for 2.9.2-3.4 compatibility (issue4733) As of Mercurial 3.4, there were several syntax rules to process nested template strings. Unfortunately, they were inconsistent and conflicted each other. a. buildmap() rule - template string is _parsed_ as string, and parsed as template - <\"> is not allowed in nested template: {xs % "{f(\"{x}\")}"} -> parse error - template escaping <\{> is handled consistently: {xs % "\{x}"} -> escaped b. _evalifliteral() rule - template string is _interpreted_ as string, and parsed as template in crafted environment to avoid double processing of escape sequences - <\"> is allowed in nested template: {if(x, "{f(\"{x}\")}")} - <\{> and escape sequences in string literal in nested template are not handled well c. pad() rule - template string is first interpreted as string, and parsed as template, which means escape sequences are processed twice - <\"> is allowed in nested template: {pad("{xs % \"{x}\"}', 10)} Because of the issue of template escaping, issue4714, 7298da81f5a9 (in stable) unified the rule (b) to (a). Then, 576d6c74784b (in default) unified the rule (c) to (b) = (a). But they disabled the following syntax that was somewhat considered valid. {if(rev, "{if(rev, \"{rev}\")}")} {pad("{files % \"{file}\"}", 10)} So, this patch introduces \"...\" literal to work around the escaped-quoted nested template strings. Because this parsing rule exists only for the backward compatibility, it is designed to copy the behavior of old _evalifliteral() as possible. Future patches will introduce a better parsing rule similar to a command substitution of POSIX shells or a string interpolation of Ruby, where extra escapes won't be necessary at all. {pad("{files % "{file}"}", 10)} ~~~~~~~~~~~~~~~~~~ parsed as a template, not as a string Because <\> character wasn't allowed in a template fragment, this patch won't introduce more breakages. But the syntax of nested templates are interpreted differently by people, there might be unknown issues. So if we want, we could instead remove db7463aa080f, 890845af1ac2 and 7298da81f5a9 from the stable branch as the bug fixed by these patches existed for longer periods. 554d6fcc3c8, "strip single backslash before quotation mark in quoted template", should be superseded by this patch. I'll remove it later.

merge with stable

demandimport: support importing builtins for Python 3 __builtin__ was renamed to builtins in Python 3. If importing __builtin__ fails, fall back to importing builtins in order to support Python 3.

demandimport: alias __builtin__ as builtins Python 3 renamed the __builtin__ module to builtins. In preparation for supporting Python 3, alias the imported module as "builtins."

bufferedinputpipe: remove N^2 computation of buffer length (issue4735) The assumption that dynamically computing the length of the buffer was N^2, but negligible because fast was False. So we drop the dynamic computation and manually keep track of the buffer length.

bufferedinputpipe: remove an outdate comment This comment is the remains of a intermediate implementation using self._buffer += data This implementation never made it to the repository and we can safely drop the comment.

statichttprepo: kill off sopener

obsolete: replace references to 'sopener' with 'svfs'

exchange: replace references to 'sopener' with 'svfs'

localrepo: kill off sopener (API) sopener is deprecated since 7034365089bf (Mercurial 2.3). It's annoying for extension authors to have to deal with both. Let's just kill it off.

share: replace reference to 'sopener' with 'svfs' sopener is deprecated. It's annoying for extension authors to have to deal with both. Let's just kill it off.

archive: don't assume '.' is being archived for changessincelatesttag Hardcoding '.' is wrong, and yielded strange results when archiving old revisions. For example, when archiving the cset that adds the signature to 3.4 (c48850339988), the resulting value was previously 51 (the number of commits on stable between 3.4 and today), even though it was a direct descendant of a tag, with a {latesttagdistance} of 2. This still includes all other _ancestor_ paths not included in {latesttag}. Note that archiving wdir() currently blows up several lines above this when building the 'base' variable. Since wdir() isn't documented, ignore that it needs work to handle wdir() here for now.

hgweb: use css for stripey background in coal Since "b8ecc3830c89 or 25dae11bb044::c229a5e7511e" paper style used css for stripes in background for browsing files, for listing branches/tags/bookmarks, and so on. Since coal borrows many paper templates (e.g. shortlogentry.tmpl), it actually tried to do the same, but it didn't have the needed css classes. You can compare https://selenic.com/hg?style=coal with https://selenic.com/hg?style=paper and see how log view in coal style has plain white background, unlike the one in paper style. This wasn't intended. Let's copy css classes directly from style-paper.css and remove parity classes from elements that don't need them anymore. This makes plain white background have stripes again and makes coal/map even more similar to paper/map (which can ease porting changes or %including paper/map in future).

templatekw: correct typo in activebookmark documentation

merge with stable

check-code: detect legacy exception syntax We just rewrote all files to use modern exception syntax. Ban the old form. This will detect the "except type, instance" and "except (type1, type2), instance" forms.

global: mass rewrite to use modern exception syntax Python 2.6 introduced the "except type as instance" syntax, replacing the "except type, instance" syntax that came before. Python 3 dropped support for the latter syntax. Since we no longer support Python 2.4 or 2.5, we have no need to continue supporting the "except type, instance". This patch mass rewrites the exception syntax to be Python 2.6+ and Python 3 compatible. This patch was produced by running `2to3 -f except -w -n .`.

check-code: detect legacy octal syntax Now that we have mass rewriting all files to use the modern octal syntax, detect and ban the legacy syntax, which is no longer supported in Python 3.

global: mass rewrite to use modern octal syntax Python 2.6 introduced a new octal syntax: "0oXXX", replacing "0XXX". The old syntax is not recognized in Python 3 and will result in a parse error. Mass rewrite all instances of the old octal syntax to the new syntax. This patch was generated by `2to3 -f numliterals -w -n .` and the diff was selectively recorded to exclude changes to "<N>l" syntax conversion, which will be handled separately.

revert: change the direction of revert -i After the discussion on the list about hg revert -i, it seems like we are satisfied with what we called proposition 2. It shows the changes to revert in the same direction as hg diff. This patch makes it the default option. It changes all the + in - and vice versa in the tests for revert -i.

revert: change a test to make the change of direction of revert -i easier Currently we are handling editing of newly-added files with the interactive interface. We are not handling editing of deleted files. In the test for revert, we were editing a newly-added file. Since we want to change the direction of revert -i, this editing of a newly-added file will become editing of a deleted file. Since we don't support that, this patch changes the test to make the rest of the series cleaner.

parser: update documentation about tokenizer and elements

parser: accept iterator of tokens instead of tokenizer function and program This can simplify the interface of parse() function. Our tokenizer tends to have optional arguments other than the message to be parsed. Before this patch, the "lookup" argument existed only for the revset, and the templater had to pack [program, start, end] to be passed to its tokenizer.

verify: print hint to run debugrebuildfncache Corrupt fncache is now a recoverable operation. Inform the user how to recover from this warning.

repair: add functionality to rebuild fncache Currently, there is no way to recover from a missing or corrupt fncache file in place (a clone is required). For certain use cases such as servers and with large repositories, an in-place repair may be desirable. This patch adds functionality for in-place repair of the fncache. The `hg debugrebuildfncache` command is introduced. It ensures the fncache is up to date by reconstructing the fncache from all seen files encountered during a brute force traversal of the repository's entire history. The command will add missing entries and will prune excess ones. Currently, the command no-ops unless the repository has the fncache requirement. The command could later grow the ability to "upgrade" an existing repository to be fncache enabled, if desired. When testing this patch on a local clone of the Firefox repository, it removed a bunch of entries. Investigation revealed that removed entries belonged to empty (0 byte size) .i filelogs. The functionality for pruning fncache of stripped revlogs was introduced in f49d60fa40a5, so the presence of these entries likely predates this feature.

import: cross-reference patch.fuzz option from `hg help import`

import: cross-reference ui.patch option from `hg help import` This is a fairly obscure patch option. It seems reasonable to expose it a bit more from the help text for `hg import`.

doc: document the ui.patch option This option has been undocumented since e56c7e05c7e6 (July 2007).

revset: rework 'filteredset.last' 'isascending' and 'isdescending' are methods, not attributes. This led 'last()' to misbehave on some non-ascending filtered sets.

revset: improves time complexity of 'roots(xxx)' The canonical way of doing 'roots(X)' is 'X - children(X)'. This is what the implementation used to be. However, computing children is expensive because it is unbounded. Any changesets in the repository may be a children of '0' so you have to look at all changesets in the repository to compute children(0). Moreover the current revsets implementation for children is not lazy, leading to bad performance when fetching the first result. There is a more restricted algorithm to compute roots: roots(X) = [r for r in X if not parents(r) & X] This achieve the same result while only looking for parent/children relation in the X set itself, making the algorithm 'O(len(X))' membership operation. Another advantages is that it turns the check into a simple filter, preserving all laziness property of the underlying revsets. The speed is very significant and some laziness is restored. -) revset without 'roots(...)' to compare to base line 0) before this change 1) after this change revset #0: roots((tip~100::) - (tip~100::tip)) plain min last -) 0.001082 0.000993 0.000790 0) 0.001366 0.001385 0.001339 1) 0.001257 92% 0.001028 74% 0.000821 61% revset #1: roots((0::) - (0::tip)) plain min last -) 0.134551 0.144682 0.068453 0) 0.161822 0.171786 0.157683 1) 0.137583 85% 0.146204 85% 0.070012 44% revset #2: roots(tip~100:) plain min first last -) 0.000219 0.000225 0.000231 0.000229 0) 0.000513 0.000529 0.000507 0.000539 1) 0.000463 90% 0.000269 50% 0.000267 52% 0.000463 85% revset #3: roots(:42) plain min first last -) 0.000119 0.000146 0.000146 0.000146 0) 0.000231 0.000254 0.000253 0.000260 1) 0.000216 93% 0.000186 73% 0.000184 72% 0.000244 93% revset #4: roots(not public()) plain min first -) 0.000478 0.000502 0.000504 0) 0.000611 0.000639 0.000634 1) 0.000604 0.000560 87% 0.000558 revset #5: roots((0:tip)::) plain min max first last -) 0.057795 0.004905 0.058260 0.004908 0.038812 0) 0.132845 0.118931 0.130306 0.114280 0.127742 1) 0.111659 84% 0.005023 4% 0.111658 85% 0.005022 4% 0.092490 72% revset #6: roots(0::tip) plain min max first last -) 0.032971 0.033947 0.033460 0.032350 0.033125 0) 0.083671 0.081953 0.084074 0.080364 0.086069 1) 0.074720 89% 0.035547 43% 0.077025 91% 0.033729 41% 0.083197 revset #7: 42:68 and roots(42:tip) plain min max first last -) 0.006827 0.000251 0.006830 0.000254 0.006771 0) 0.000337 0.000353 0.000366 0.000350 0.000366 1) 0.000318 94% 0.000297 84% 0.000353 0.000293 83% 0.000351 revset #8: roots(0:tip) plain min max first last -) 0.002119 0.000145 0.000147 0.000147 0.000147 0) 0.047441 0.040660 0.045662 0.040284 0.043435 1) 0.038057 80% 0.000187 0% 0.034919 76% 0.000186 0% 0.035097 80% revset #0: roots(:42 + tip~42:) plain min max first last sort -) 0.000321 0.000317 0.000319 0.000308 0.000369 0.000343 0) 0.000772 0.000751 0.000811 0.000750 0.000802 0.000783 1) 0.000632 81% 0.000369 49% 0.000617 76% 0.000358 47% 0.000601 74% 0.000642 81%

revsetbenchmark: do not abort on failure to run a revset Instead of aborting the whole process, we just skip entry for revset that failed to run.

osutil: remove Python 2.4 errno conversion workaround

patch: remove email import workaround for Python 2.4 Python 2.6 provides access to the sub-modules just fine. This workaround is no longer needed since we no longer support Python 2.4.

check-commit: catch both patterns of double empty lines

revsetbenchmarks: ignore empty lines Before this change, empty lines were seen as an entry and the benchmark tried to run benchmark for "".

bundle2: reword debug message for invalid .hgtags data marmoute requested a follow-up to make the verb the first word.

bundle2.getunbundler: rename "header" to "magicstring" This is more consistent with the name used in the bundler class. Thanks goes to Martin von Zweigbergk for pointing this out.

ancestors: prefetch method outside of the loop 15412bba5a68 is yet another example where this is worthwhile when it comes to performance, we blindly do it for all 'lazyancestors' methods.

templater: fix handling of \-escapes in raw string literals The backslash character should start escape sequences no matter if a string is prefixed with 'r'. They are just not interpreted as escape sequences in raw strings. revset.tokenize() handles them correctly, but templater didn't. https://docs.python.org/2/reference/lexical_analysis.html#string-literals

templater: evaluate "query" argument passed to revset() revset() had the same issue as 9452112c8eb0. It crashed by passing non-string expression.

match: let 'path:.' and 'path:' match everything (issue4687) Previously, both queries exited with code 1, printing nothing. The pattern in the latter query is normalized to '.', so it is really the same case.

changelog: document the 'readpending' method I happen to have spent some time understanding this logic, so I'm leaving documentation for the next poor fellow.

revsets: use '&' instead of '.filter' in head More high level operations are more likely to be optimised.

fileset: don't suggest private or undocumented queries Unlike revsets, it looks like all of the filesets are documented, so there's really nothing to test. This is aimed more at parity with revsets and future-proofing.

revset: don't suggest private or undocumented queries I noticed when I mistyped 'matching', that it suggested '_matchfiles' as well. Rather than simply exclude names that start with '_', this excludes anything without a docstring. That way, if it isn't in the help text, it isn't suggested, such as 'wdir()'.

patch: add fuzz config flag (issue4697)

devel-warn: issue a warning for old style revsets We have move to smartset class more than a year ago, we now have the tool to aggressively nudge developer into upgrading their extensions.

devel-warn: move the develwarn function as a method of the ui object We are going to use this feature in more and more place. Having to import scmutil makes it an import cycle hell.

templates: fix missing closing brace in map-cmdline.status Currently the parser does not validate the last closing brace, but future patches will make it less permissive for this kind of syntax errors.

verify: clarify misleading fncache message This is a message about cache corruption, not repository corruption or actually missing files. Fix message and reduce to a warning.

phase: add a pointer to 'hg help phases' in the 'phase' help The command and the general topic have very similar name. The topic is pointing to the command, but the command was not pointing to the topic.

publishing: unconditionally trust publishing flag Unfortunately, there are currently no tests for this behavior because we lack a good way of testing trust.

publishing: use new helper method

publishing: add helper method to localrepo

revset: make use of natively-computed set for 'draft()' and 'secret()' If the computation of a set for each phase (done in C) is available, we use it directly instead of applying a simple filter. This give a massive speed-up in the vast majority of cases. On my mercurial repo with about 15000 out of 40000 draft changesets: revset: draft() plain min first last 0) 0.011201 0.019950 0.009844 0.000074 1) 0.000284 2% 0.000312 1% 0.000314 3% 0.000315 x4.3 Bad performance for "last" come from the handling of the 15000 elements set (memory allocation, filtering hidden changesets (99% of it) etc. compared to applying the filter only on a handfuld of revisions (the first draft changesets being close of tip). This is not seen as an issue since: * Timing is still pretty good and in line with all the other one, * Current user of Vanilla Mercurial will not have 1/3 of their repo draft, This bad effect disappears when phase's set is smaller. (about 200 secrets): revset: secret() plain min first last 0) 0.011181 0.022228 0.010851 0.000452 1) 0.000058 0% 0.000084 0% 0.000087 0% 0.000087 19%

revset: refactor the non-public phase code Code for draft and secret are the same. We'll make it more complex to take advantages of the set recomputed in C, so we first refactor the code to only have one place to update (and make sure all behave properly). We do not refactor the 'public()' code because it does not have a natively computed set.

revset: translate node directly with changelog in 'head' Using 'repo[X]' is much slower because it creates a 'changectx' object and goes though multiple layers of code to do so. It is also error prone if there is tags, bookmarks, branch or other names that could map to a node hash and take precedence (user are wicked). This provides a significant performance boost on repository with a lot of heads. Benchmark result for a repo with 1181 heads. revset: head() plain min last reverse 0) 0.014853 0.014371 0.014350 0.015161 1) 0.001402 9% 0.000975 6% 0.000874 6% 0.001415 9% revset: head() - public() plain min last reverse 0) 0.015121 0.014420 0.014560 0.015028 1) 0.001674 11% 0.001109 7% 0.000980 6% 0.001693 11% revset: draft() and head() plain min last reverse 0) 0.015976 0.014490 0.014214 0.015892 1) 0.002335 14% 0.001018 7% 0.000887 6% 0.002340 14% The speed up is visible even when other more costly revset are in use revset: head() and author("mpm") plain min last reverse 0) 0.105419 0.090046 0.017169 0.108180 1) 0.090721 86% 0.077602 86% 0.003556 20% 0.093324 86%

revset: use a baseset in _notpublic() The '_notpublic()' internal revset was "returning" a set. That was wrong. We now return a 'baseset' as appropriate. This has no effect on performance in most case, because we do the exact same operation than what the combination with a 'fullreposet' was doing. This as a small effect on some operation when combined with other set, because we now apply the filtering in all cases. I think the correctness is worth the impact on some corner cases. The optimizer should take care of these corner cases anyway. revset #0: not public() plain min max first last reverse 0) 0.000465 0.000491 0.000495 0.000500 0.000494 0.000479 1) 0.000484 0.000503 0.000498 0.000505 0.000504 0.000491 revset #1: (tip~1000::) - public() plain min max first last reverse 0) 0.002765 0.001742 0.002767 0.001730 0.002761 0.002782 1) 0.002847 0.001777 0.002776 0.001741 0.002764 0.002858 revset #2: not public() and branch("default") plain min max first last reverse 0) 0.012104 0.011138 0.011189 0.011138 0.011166 0.011578 1) 0.011387 94% 0.011738 105% 0.014220 127% 0.011223 0.011184 0.012077 revset #3: (not public() - obsolete()) plain min max first last reverse 0) 0.000583 0.000556 0.000552 0.000555 0.000552 0.000610 1) 0.000613 105% 0.000559 0.000557 0.000573 0.000558 0.000613 revset #4: head() - public() plain min max first last reverse 0) 0.010869 0.010800 0.011547 0.010843 0.010891 0.010891 1) 0.011031 0.011497 106% 0.011087 0.011100 0.011100 0.011085

contrib: clean up all-revsets.txt file I forgot to cleanup a handful of them when I originally created the file.

hgweb: link to revision by node hash in paper & coal Unlike other styles, paper and coal had only one link to current revision: in the sidebar. Since those links now use symbolic revisions after 3bb6f5f478a7, it's nice to have a link that allows going from /rev/tip to /rev/<tip hash>, for instance. Let's make the node hash in the page header that new link.

hgweb: link to revision by node hash in gitweb & monoblue This allows going from /rev/tip to /rev/<tip hash> with ease.

archive: report the node as "{p1node}+" when archiving a dirty wdir() This is more useful than reporting all 'f's, allowing the archive to be diffed against a specific revision to see what changed.

phase: remove a 'for x in "foo bar".split()' idiom in phasecache.replace I find this idiom fairly horrible.

phase: document the replace method This is a minor documentation update to answer a co-worker question.

revset: ensure we have loaded phases data in '_notpublic()' If we are the very first rev access (or if the phase cache just got invalidated) the phasesets will be None even if we support the native computation. So we explicitly trigger a computation if needed. This was not an issue before because requesting any phase information would have triggered such computation.

phase: rename getphaserevs to loadphaserevs This function is: - already loading the data in place, - used once in the code. So we drop the return value and change the name to make this obvious. We keep the function public because we'll have to use it in revset.

test-subrepo-recursion: more aggressively regex 'unzip -l' output for 10.10 The output on 10.10 looks like this: Length Date Time Name -------- ---- ---- ---- 172 01-01-80 00:00 .hg_archival.txt 10 01-01-80 00:00 .hgsub 45 01-01-80 00:00 .hgsubstate 3 01-01-80 00:00 x.txt 10 01-01-80 00:00 foo/.hgsub 45 01-01-80 00:00 foo/.hgsubstate 9 01-01-80 00:00 foo/y.txt 9 01-01-80 00:00 foo/bar/z.txt -------- ------- 303 8 files (2 digit year, shorter -- separators and closer columns). We don't care about any of that, so ignore it.

contrib: introduce an all-revsets.txt file This file should gather all revsets ever thought interesting by anyone. That way one can check the impact of a change when touching something revset-ish. See inline comments for details. This file have been refilled with all the entry I could automatically find from changeset descriptions. I assume we missed some not using 'revsetbenchmarks.py' output.

contrib: rename revsetbenchmarks.txt to 'base-revsets.txt' We rename the file and document its purpose. We'll be introducing another file gathering revsets useful for benchmark of the predicate themsleves in a coming changesets.

revsetbenchmarks: add main documention for the script This allow us to document the fact we can use comment in the file listing revsets.

hgweb: don't dereference symbolic revision in paper & coal style (issue2296) Let's make paper (and coal, since it borrows so much from paper) templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes. Some pages don't have permanent links to current node hash (so it's not very easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future patches.

hgweb: don't dereference symbolic revision in gitweb style Let's make gitweb templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes. Some pages don't have permanent links to current node hash (so it's not very easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future patches.

hgweb: don't dereference symbolic revision in monoblue style Let's make monoblue templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes. Some pages don't have permanent links to current node hash (so it's not very easy to go from /rev/tip to /rev/<tip hash>), this will be addressed in future patches.

hgweb: don't dereference symbolic revision in spartan style Let's make spartan templates use symbolic revision in navigation links. The majority of links (log, filelog, annotate, etc) still use node hashes, and many pages also have permanent link to current node hash (i.e. you can go from /rev/tip to /rev/<tip hash> without manual url editing), so it's safe to update navigation.

hgweb: provide symrev (symbolic revision) property to the templates One of the features of hgweb is that current position in repo history is remembered between separate requests. That is, links from /rev/<node_hash> lead to /file/<node_hash> or /log/<node_hash>, so it's easy to dig deep into the history. However, such links could only use node hashes and local revision numbers, so while staying at one exact revision is easy, staying on top of the changes is not, because hashes presumably can't change (local revision numbers can, but probably not in a way you'd find useful for navigating). So while you could use 'tip' or 'default' in a url, links on that page would be permanent. This is not always desired (think /rev/tip or /graph/stable or /log/@) and is sometimes just confusing (i.e. /log/<not the tip hash>, when recent history is not displayed). And if user changed url deliberately to say default instead of <some node hash>, the page ignores that fact and uses node hash in its links, which means that navigation is, in a way, broken. This new property, symrev, is used for storing current revision the way it was specified, so then templates can use it in links and thus "not dereference" the symbolic revision. It is an additional way to produce links, so not every link needs to drop {node|short} in favor of {symrev}, many will still use node hash (log and filelog entries, annotate lines, etc). Some pages (e.g. summary, tags) always use the tip changeset for their context, in such cases symrev is set to 'tip'. This is needed in case the pages want to provide archive links. highlight extension needs to be updated, since _filerevision now takes an additional positional argument (signature "web, req, tmpl" is used by most of webcommands.py functions). More references to symbolic revisions and related gripes: issue2296, issue2826, issue3594, issue3634.

archive: support 'wdir()' This is a step toward replacing the extdiff internals with archive, in order to support 'extdiff -S'. Only Mercurial subrepos are supported for now. If a file is missing from the filesystem, it is silently skipped. Perhaps it should warn, but it cannot abort when working with extdiff because deleting a file is a legitimate diff.

subrepo: allow a representation of the working directory subrepo Some code cannot handle a subrepo based on the working directory (e.g. sub.dirty()), so the caller must opt in. This will be useful for archive, and perhaps some other commands. The git and svn methods where this is used may need to be fixed up on a case by case basis.

templater: comment that gettemplate() has different name resolution order I've tried to unify gettemplate() with buildtemplate(), but it didn't go well because gettemplate() have to bypass mapping dict. For example, web templates have '{tags%changelogtag}' and 'changelogtag' is defined in both mapping, the default, and context.cache, sourced from map file. In general, mapping shadows context variables, but gettemplate() have to pick it from context.cache.

templater: drop strtoken argument from compiletemplate() There's no "rawstring" template now.

templater: do not reevaluate rawstring as template (BC) The previous patch made 'string' is always interpreted as a template. So this patch removes the special handling of r'rawstring' instead. Now r'' disables template processing at all.

templater: take any string literals as template, but not for rawstring (BC) This patch series is intended to unify the interpretation of string literals. It is breaking change that boldly assumes a. string literal "..." never contains template-like fragment or it is intended to be a template b. we tend to use raw string literal r"..." for regexp pattern in which "{" should have different meaning Currently, we don't have a comprehensible rule how string literals are evaluated in template functions. For example, fill() takes "initialindent" and "hangindent" as templates, but not for "text", whereas "text" is a template in pad() function. date(date, fmt) diff(includepattern, excludepattern) fill(text, width, initialident: T, hangindent: T) get(dict, key) if(expr, then: T, else: T) ifcontains(search, thing, then: T, else: T) ifeq(expr1, expr2, then: T, else: T) indent(text, indentchars, firstline) join(list, sep) label(label: T, expr: T) pad(text: T, width, fillchar, right) revset(query, formatargs...]) rstdoc(text, style) shortest(node, minlength) startswith(pattern, text) strip(text, chars) sub(pattern, replacement, expression: T) word(number, text, separator) expr % template: T T: interpret "string" or r"rawstring" as template This patch series adjusts the rule as follows: a. string literal, '' or "", starts template processing (BC) b. raw string literal, r'' or r"", disables both \-escape and template processing (BC, done by subsequent patches) c. fragment not surrounded by {} is non-templated string "ccc{'aaa'}{r'bbb'}" ------------------ *: template --- c: string --- a: template --- b: rawstring Because this can eliminate the compilation of template arguments from the evaluation phase, "hg log -Tdefault" gets faster. % cd mozilla-central % LANG=C HGRCPATH=/dev/null hg log -Tdefault -r0:10000 --time > /dev/null before: real 4.870 secs (user 4.860+0.000 sys 0.010+0.000) after: real 3.480 secs (user 3.440+0.000 sys 0.030+0.000) Also, this will allow us to parse nested templates at once for better error indication.

templater: move runtemplate function out of buildmap/runmap pair The next patch will introduce buildtemplate function that should be defined near runtemplate. But I don't want to insert it between buildmap and runmap.

phase: also overwrite phase's sets when replacing a phasecache We need to copy this new attributes around too. This fix an issue where phases data used by 'not public()' were not invalidated properly.

phase: invalidate the phase's set cache alongside the revs Invalidate was leaving set data around leading to possible bugs in revset.

phase: also copy phase's sets when copying phase cache We forgot to add such copy when we added the attributes.

verify: check the subrepository references in .hgsubstate While hopefully atypical, there are reasons that a subrepository revision can be lost that aren't covered by corruption of the .hgsubstate revlog. Such things can happen when a subrepo is amended, stripped or simply isn't pulled from upstream because the parent repo revision wasn't updated yet. There's no way to know if it is an error, but this will find potential problems sooner than when some random revision is updated. Until recently, convert made no attempt at rewriting the .hgsubstate file. The impetuous for this is to verify the conversion of some repositories, and this is orders of magnitude faster than a bash script from 0..tip that does an 'hg update -C $rev'. But it is equally useful to determine if everything has been pulled down before taking a thumb drive on the go. It feels somewhat wrong to leave this out of verifymod (mostly because the file is already read in there, and the final summary is printed before the subrepos are checked). But verifymod looks very low level, so importing subrepo stuff there seems more wrong.

context: override workingctx.hex() to avoid a crash Since node is None for workingctx, it can't use the base class implementation of 'hex(self.node())'. It doesn't appear that there are any current callers of this, but there will be when archive supports 'wdir()'. My first thought was to use "{p1node}+", but that would cause headaches elsewhere [1]. We should probably fix up localrepository.__getitem__ to accept this hash for consistency, as a followup. This works, if the full hash is specified: @@ -480,7 +480,7 @@ return dirstate.dirstate(self.vfs, self.ui, self.root, validate) def __getitem__(self, changeid): - if changeid is None: + if changeid is None or changeid == 'ff' * 20: return context.workingctx(self) if isinstance(changeid, slice): return [context.changectx(self, i) That differs from null, where it will accept any number of 0s, as long as it isn't ambiguous. [1] https://www.selenic.com/pipermail/mercurial-devel/2015-June/071166.html

convert: update 'intermediate-source' in the destination's extras dictionary

check-code: ban use of '[[ ]]' in tests

test-fileset: remove bashism, use test instead of '[[ ]]' Debian dash complains about it. $TESTTMP.sh: 213: $TESTTMP.sh: [[: not found

tests: test symbolic revision (de)reference in all hgweb styles Right now the way revisions get specified in hgweb urls is ignored, i.e. after revision is resolved, only its node hash (or sometimes local revision number) is used for all links in the templates. So, basically, every page for "tip" revision (or any other symbolic rev id) will dereference it: lose the nice symbolic name by putting node hash/local rev number in its place. The only exception so far is archive links on some pages: /archive/tip.{bz2,gz,zip}. The fact that this dereferencing is neither convenient nor intuitive is reflected in issue2296, issue2826 and issue3594. issue3634 also mentions this. But to fix this it's first needed to demonstrate and test the way templates currently form links. The new test file is separate from other hgweb tests, since it seems big and distinct enough. And it's so big because links are formed in each template independently, so it's necessary to test them all to avoid any inconsistent behavior.

dirstate: use a presized dict for the dirstate This uses a simple heuristic to avoid expensive resizes. On a real-world repo with around 400,000 files, perfdirstate: before: ! wall 0.155562 comb 0.160000 user 0.150000 sys 0.010000 (best of 64) after: ! wall 0.132638 comb 0.130000 user 0.120000 sys 0.010000 (best of 75) On another real-world repo with around 250,000 files: before: ! wall 0.098459 comb 0.100000 user 0.090000 sys 0.010000 (best of 100) after: ! wall 0.089084 comb 0.090000 user 0.080000 sys 0.010000 (best of 100)

parsers: add an API to create a new presized dict

parsers: factor out code to create a presized dict In upcoming patches we'll expose this as an API.

parsers: drop Python 2.4 compat in make_file_foldmap

progress: respect ui.quiet (issue4726) Caught when I was running the hgsubversion testsuite and it started printing progress bars over top of my test output lines.

merge with stable

match: don't remove '.' from _includeroots This makes _includeroots more like _fileroots and gives visitdir() a nice symmetry in the two. I'm hoping to later combine the two (_fileroots and _includeroots), and having them treated similarly should make that step easier to follow.

match: join two nested if-blocks Instead of if a: if b: return False let's write it if a and b: return False

match: drop optimization (?) of 'parentdirs' calculation It seems unlikely that the optimization to avoid calling util.finddirs twice will be noticeable, so let's drop it. This makes the two conditions for includes and regular patterns more similar.

match: break boolean expressions into one operand per line This makes it much easier to spot both the operators ('and'/'or') and the operands.

match: drop unnecessary removal of '.' from excluded roots The repo root is nothing special when it comes to what directories to visit: patterns like '-X relglob:*.py' should not exclude the top directory, while '-X path:.' should (pointless as such a pattern may be). The explicit removal of '.' from the set of excluded roots was probably there to avoid removing the the root directory when any patterns had been given, but since 20ad936ac5d2 (treemanifest: visit directory 'foo' when given e.g. '-X foo/ba?', 2015-05-27), we only exclude directories that should be completely excluded, so we no longer need to special-case the root directory.

changegroup: simplify by not reusing 'prog(ress)' instance Just create a new instance of the 'prog' class for each step instead of replacing its fields and resetting the counter.

changegroup: don't use 'repo' for non-repo 'self' 'repo' is a very confusing name to use for 'self', especially when it's not a repo. Also drop repo.ui member (a.k.a. self.ui) now that 'self' doesn't shadow outer 'repo' variable.

subrepo: introduce hgsubrepo._getctx() This code is already used in a couple of places, and will need to be used in others where wdir() support is needed. Trying to get the wdir() revision stored in self._substate[1] when creating the object in subrepo.subrepo() resulted in breaking various status and diff tests. This is a far less invasive change.

convert: apply the appropriate phases to the destination (issue4165) If the conversion doesn't change the hash, and the cset is public in the source, it should be public in the destination. (This can happen if file remapping is done that doesn't affect the initial commits.) This also propagates the secret phase from the source, regardless of the hash, because presumably the content is what is secret. Otherwise, the destination commit stays in the draft phase. Maybe any draft cset with an unchanged hash should be changed to public, because it has effectively been shared, but convert pretty strongly implies throwing away (or at least readonly archiving) the source repo. The change in the rollback output is because the name of the outer transaction is now 'convert', which seems more accurate. Unfortunately, the memctx won't indicate the hash prior to committing, so the proper phase can't be applied with the commit. The repo is already write locked in mercurial_sink.before().

convert: always track the hg source revision in the internal commit object This will be needed in the next patch to determine if phases need to be adjusted. The insertion of the source revision in 'extras{}' is still controlled by the config property.

repoview: invalidate 'visible' filtered revisions when bookmarks change Context: the result of computehidden, used to compute the 'visible' revisions is cached. Its output can change when: 1) new obsolete commits are created 2) new bookmarks are created or deleted 3) new tags are created or deleted 4) the parents of the working copy change We currently correctly invalidate the cache only in the case 1). This patch fixes the second case (bookmarks) by invalidating the cache once a bookmark is added or removed.

ui: flush stderr after printing a non-chained exception for Windows There were consistent test failures in test-bad-extension.t, because Windows buffers stderr when redirected to a file (per the comment in ui.write_err()). That resulted in failures like this: --- c:/Users/Matt/Projects/hg/tests/test-bad-extension.t +++ c:/Users/Matt/Projects/hg/tests/test-bad-extension.t.err @@ -23,11 +23,11 @@ Traceback (most recent call last): Exception: bit bucket overflow *** failed to import extension badext2: No module named badext2 - Traceback (most recent call last): - ImportError: No module named badext2 hg help [-ec] [TOPIC] show help for a given topic or a help overview + Traceback (most recent call last): + ImportError: No module named badext2 show traceback for ImportError of hgext.name if debug is set (note that --debug option isn't applied yet when loading extensions) Instead of inserting another flush immediately after the print, to go along with the one recently added prior to the print (see 3ff4b07412ad), funnel the output through ui.write_err(). The flush prior to printing the traceback only mentions that stdout needs to be flushed, and only stderr needs to be flushed after printing the traceback. ui.write_err() does both for us without needing to redocument the quirky Windows behavior. It will also clear any progress bar.

revset: use parentsets.min in _children As stated in the comment, using the smartset 'min' will give more opportunity to be smart. It give a small but significant boost to the performance. Most of the time is still spend doing the actual computation but at least we can scrap some performance when it makes sense. revset #0: roots(0:tip) plain 0) 0.046600 1) 0.044109 94%

revset: prefetch all attributes before loop in _revsbetween Python is slow at attributes lookup. No, really, I mean -slow-. prefetching these three methods give use a measurable performance boost. revset #0: 0::tip plain 0) 0.037655 1) 0.034290 91%

hgweb: add some in-code documentation for 'web.view' This documentation was mostly intended for the user helps. However given the lack of request for such feature, we should keep it un-documented. We stick the help text in the code as it could still be useful to fellow contributors.

bookmark: informs of failure to upgrade a bookmark When we explicitly requested to update a bookmark but the bookmark location was missing locally, we used to silently ignore the case. We now issue a message about it to point that something wrong is going on. By chance, we fixed all the cases where is case happened (for explicit pulling only, issue4700 is still open). But I think it is still valuable to have a warning in place in case such issue is reintroduced. This patch have been tested against issue4689 test (but without issue4689 fix). It give the better but expected failure seen below: > --- /home/pyd/src/mercurial-dev/tests/test-bookmarks-pushpull.t > +++ /home/pyd/src/mercurial-dev/tests/test-bookmarks-pushpull.t.err > @@ -337,12 +337,12 @@ > adding manifests > adding file changes > added 1 changesets with 1 changes to 1 files > - updating bookmark Y > + remote bookmark Y point to locally missing 0d60821d2197 > (run 'hg update' to get a working copy) > $ hg book > * @ 1:0d2164f0ce0d > X 1:0d2164f0ce0d > - Y 5:35d1ef0a8d1b > + Y 4:b0a5eff05604 > Z 1:0d2164f0ce0d > > Update a bookmark right after the initial lookup -r (issue4700) > @@ -387,12 +387,11 @@ > adding manifests > adding file changes > added 1 changesets with 1 changes to 1 files > - updating bookmark Y > (run 'hg update' to get a working copy) > $ hg book > * @ 1:0d2164f0ce0d > X 1:0d2164f0ce0d > - Y 6:0d60821d2197 > + Y 4:b0a5eff05604 > Z 1:0d2164f0ce0d > $ hg -R $TESTTMP/pull-race book > @ 1:0d2164f0ce0d

fancyopts: allow all callable as default parameter value The current fancyopts allows function as default parameter value but not other callables. By supporting other callables, we can have the benefits of e.g., custom __str__ method, which will be printed by 'hg help' as the default value.

templater: evaluate arguments passed to diff() appropriately Before this patch, diff() crashed by passing non-string expression because it didn't evaluate arguments at all.

parsers: do not cache RevlogError type (issue4451) Index lookups raise RevlogError when the lookup fails. The previous implementation was caching a reference to the RevlogError type in a static variable. This assumed that the "mercurial.error" module was only loaded once and there was only a single copy of it floating around in memory. Unfortunately, in some situations - including certain mod_wsgi configurations - this was not the case: the "mercurial.error" module could be reloaded. It was possible for a "RevlogError" reference from the first interpreter to be used by a second interpreter. While the underlying thing was a "mercurial.error.RevlogError," the object IDs were different, so the Python code in revlog.py was failing to catch the exception! This error has existed since the C index lookup code was implemented in changeset e8d37b78acfb, which was first released in Mercurial 2.2 in 2012. http://emptysqua.re/blog/python-c-extensions-and-mod-wsgi/#static-variables-are-shared contains more details. This patch removes the caching of the RevlogError type from the function. Since pretty much the entire function was refactored and the return value of the function wasn't used, I changed the function signature to not return anything. For reasons unknown to me, we were calling PyErr_SetObject() with the type of RevlogError and an instance of RevlogError. This was equivalent to the Python code "raise RevlogError(RevlogError)". This seemed wonky and completely unnecessary. The Python code only cares about the type of the exception, not its contents. So I got rid of this complexity. This is my first Python C extension patch. Please give extra scrutiny to it during review.

largefiles: restore the original converter class after lfconvert --to-normal Not sure how much this really matters, but much of the largefiles code that monkey-patches like this restores things when it finishes.

push: only say we are trying to push obsmarkers when we actually try The message was issued unconditionally. Move it inside the `if` that actually tries to push obsmarkers.

convert: support incremental conversion with hg subrepos This was implied in issue3486, which specifically asked for subrepo support in lfconvert. Now that lfconvert uses the convert extension internally when going to normal files, the issue is half fixed. But now even non largefile repos benefit when other transformations are needed. Supporting a full subrepo tree conversion from a single command doesn't seem reasonable, given the number of options that can be provided, and the transformations that would need to occur when entering a subrepo (consider 'filemap' paths). Instead, this allows the user to incrementally convert each hg subrepo from bottom up like so: # so convert knows the dest type when it sees a non empty dest dir $ hg init converted $ hg convert orig/sub1 converted/sub1 $ hg convert orig/sub2 converted/sub2 $ hg convert orig converted This allows different options to be applied to different subrepos more readily. It assumes the shamap is in the default location in each converted subrepo for simplicity. It also allows for a subrepo to be cloned into place, in case _it_ doesn't need a conversion. I was able to convert away from using largefiles/bfiles in several subrepos with this mechanism.

crecord: exit edition of hunk with non-zero status does not interrupt session Before this patch, when editing a hunk, if the editor was exiting with a non- zero status the crecord session would be aborted. This patch makes crecord handle the failure and goes back to the state before the failed edition. We use the new error reporting mechanism introduced in the previous patch to notify the user of the issue.

crecord: add mechanism for error reporting Before this patch, there was no way to report errors in the crecord ui. This patch introduces a new variable errorstr. If set, its content replaces the top banner displayed in crecord until a key is pressed.

crecord: extract ui refresh code after editing of hunk in a function This is useful to reuse the logic to implement the continuation of the crecord session if the user edits a hunk and the editor exits with a non-zero status.

revset: mark spots that use 'set' instead of 'smartset' Using smartset is better because we can do more optimisation on it. So we are marking the faulty spot for later processing.

revset: mark spot that feeds a set to a baseset Sets have non-defined order and this should break stuff, but as we are lucky fullreposet is also broken so the result is "not too bad". We should fix it anyway, but it is too much for my current plate.

revset: mark the fact we should use '&' instead of 'filter' in 'head' I do not have time to fix all this now, let's mark it for later.

revset: gratuitous formating fix in keyword You will be aligned.

revset: gratuitous code move in '_children' As 'cs' is empty as the time of the conditional, we can just return an empty 'baseset' and create the variable later.

revset: mark spots that should use 'smartset.min()' Using smartset's min will be significantly faster when the input set can provided an optimised answer. I do not have time to fix all of them but I'm marking the spot.

revset: mark the place where we are combining sets in the wrong direction We should always combine with subset as the left operand (to preserve the order). I do not have time to fix all of them so I'm just marking the spot.

revset: point out wrong behavior in fullreposet I cannot fix all issues in revset because I've got other things to do, but let's write down all the brokenness to help other people reading and fixing.

revsetbenchmarks: also display tag when printing a revision This is usually more useful information than the commit message.

revsetbenchmarks: clean up revsets that achieved with default variants We remove revset making use of min and max as this is covered by the variants. We could use variant for roots too, but it is not in the default so keep it here.

revsetbenchmarks: use combination variants in default set Now that we have them, let's make use of them.

revsetbenchmarks: support combining variants with "+" We need more advanced variants in some cases. For example, "The last rev of the sorted version". We introduce a syntax for this: `reverse+last` means `last(reverse(REVSET))`.

revsetbenchmarks: use many more variants by default So far the variants feature was introduced, but not used by default. We now use a set of basic variants by default.

revsetbenchmarks: display even more compact timing result We now use an 8 char display for timing (from 10), we add some logic to drop precision if the number grows too large (as we do not care about sub-0 digit in this case). This allow to pack more variants in a single screen.

revsetbenchmarks: allow running multiple variants per revset The current benchmarks were only testing the whole iteration. This is suboptimal because some changes are meaningful for things like first result, minimum or sorting. We introduce a "variants" feature that let you systematically add some variants to all revsets tested. A typical variants value would be 'plain,min,last,sort'. When testing 'all()' it will also provide testing for: - all() - min(all()) - last(all()) - sort(sort) and output: plain min last sort 0) 0.034568 0.037857 0.000074 0.034238 1) 0.011358 32% 0.020181 53% 0.000080 108% 0.011405 33% Using revsets (who hit the API) instead of the internal API add some overhead, but the overhead should be the same everywhere so it still allow comparison. This is is more simple to implement and allows comparison with older versions who do not have the same API.

revsetbenchmarks: display relative change when meaningful If the time difference is more than 5% from the previous run, we'll display relative information. This makes it much simpler to spot performance changes in a sea of benchmarks.

revsetbenchmarks: improve revision printing We now print the revision number and short hash inline. As a result we drop the crappy list printing.

revsetbenchmarks: hide most timing under a --verbose flag We mostly only care about total time. Dropping this output give us some room to display more useful information (like percentage different) in future changesets.

revsetbenchmarks: drop outdated comment We are no longer testing against system mercurial for quite some time.

revsetbenchmarks: fix argument parsing The file doc was saying something, the code was doing something else, the argument validation was doing a third thing. Doc and behavior now comply with the argument defined in the code.

revsetbenchmarks: use a more compact output format with a header We change the output from: revset #0: draft() 0) wall 0.011989 comb 0.010000 user 0.000000 sys 0.010000 (best of 177) 1) wall 0.012226 comb 0.010000 user 0.000000 sys 0.010000 (best of 193) 2) wall 0.011838 comb 0.020000 user 0.000000 sys 0.020000 (best of 208) to: revset #0: draft() wall comb user sys count 0) 0.012028 0.010000 0.000000 0.010000 170 1) 0.012218 0.010000 0.000000 0.010000 157 2) 0.012622 0.010000 0.000000 0.010000 189 This opens the road to more useful output.

revsetbenchmarks: clarify comment based on irc discussion

revsetbenchmarks: ensure all indexes have the same width This avoids an alignment glitch.

revsetbenchmarks: factor out result output into a function This will make update of the output easier.

revsetbenchmarks: parse perfrevset output into actual number We cannot just ask perfrevset to provide debug output because we usually want to compare output from old version of Mercurial that do not support it. So, we are using a regular expression. (/we now have \d problems/).

revsetbenchmarks: improve error output in case of failure This helps with diagnostics.

revsetbenchmarks: extract call to mercurial into a function This is a gratuitous change to make the code easier to look at.

phases: really fix native phase computation For some reason (probably rebase issue, leprechaun or badly resolved .rej) 1635579f9baf contains only half of the emailed patches and do not fix the bug. This patch adds the other half and enable the sweet native computation for real. As expected this provide massive speedup along the board. revset #0: not public() plain first 0) 0.011960 0.010523 1) 0.000465 3% 0.000492 4% revset #1: (tip~1000::) - public() plain first 0) 0.025700 0.025169 1) 0.002864 11% 0.001899 7% revset #2: not public() and branch("default") plain first 0) 0.022842 0.020863 1) 0.011418 49% 0.010948 52% However, it has a less impact (even bad) on first result time in simple situation. This comes from the overhead of building the set and filtering it. This is especially true on my Mercurial repository (used here) where about 1/3 of the changesets are non public and hidden. This could be mitigated by a caching of the set and a better usage of smartset in '_notpublic'. (But this won't happen in this patch because the win is massive everywhere else). revset #0: not public() last 0) 0.000081 1) 0.000493 x6.1 <-- bad impact revset #1: (tip~1000::) - public() last 0) 0.013966 1) 0.002737 19% revset #2: not public() and branch("default") last 0) 0.011021 1) 0.011038 The effect mostly disappear when the number of non-public changesets is small and/or the repo get bigger. Result for Mozilla central: Mozilla revset #0: not public() plain first last 0) 0.092787 0.084094 0.000080 1) 0.000054 0% 0.000083 0% 0.000083 revset #1: (tip~1000::) - public() plain first last 0) 0.215607 0.183996 0.124962 1) 0.031620 14% 0.006616 3% 0.031168 24% revset #2: not public() and branch("default") plain first last 0) 0.092626 0.082687 0.000162 1) 0.000139 0% 0.000165 0% 0.000167

hgweb: don't point file links at tip hash where it doesn't make sense Some pages, e.g. bookmarks, help and summary don't have a meaningful revision context: they always either show information about tip or about the whole repo (and not about any specific changeset). And error pages can just show hgweb error messages, not related to any repo or changeset. Having a hash in the links worked (even when '{node|short}' resolved to an empty string on error pages), but seeing pages without revision context provide links with hashes is a bit confusing (unless you keep current tip hash in your head at all times) and not consistent with other template styles and other links on the same page: they don't have a hash. Let's just link to '/file', which is equal to '/file/tip'.

hgweb: don't point graph links at tip hash where it doesn't make sense Some pages, e.g. bookmarks, help and summary don't have a meaningful revision context: they always either show information about tip or about the whole repo (and not about any specific changeset). And error pages can just show hgweb error messages, not related to any repo or changeset. When monoblue style was added in 91b0ada2d94b, however, all graph links had tried to point at some hash, and on such pages as described above it didn't make sense. On error pages '{node|short}' is empty string anyway. Of course, it worked, but seeing such pages without revision context provide links with hashes is a bit confusing (unless you keep current tip hash in your head at all times) and wasn't consistent with other template styles, other pages in monoblue and even other links on the same page. Let's just link to '/graph', which is equal to '/graph/tip'.

hgweb: put help link in paper/search.tmpl separately for consistency Just a cosmetic markup change, no .css changes required.

help: use 'color' as an example (instead of 'progress') Progress is now deprecated, using it as an example is suboptimal.

progress: deprecate the progress extension Activating it is a absolute no-op now.

progress: empty the extension of any logic The default value match the one enforce by the extension, we can remove that logic.

progress: move config help into core config help This is core feature now.

progress: display progress bars by default with core Mercurial As discussed multiple time during sprint, we should activate progress by default. Having progress bar significantly improve the user experience.

bundle2: provide number of changesets information to 'addchangegroup' We can now link the two efforts and provided more useful information when pulling changesets.

addchangegroup: accept an expected total number of changesets as argument Caller can optionally informs how much changesets are expected to be added. This will be used for a more useful progress bar output.

getbundle: add data about the number of changesets bundled We use an advisory parameters to carry the number of changesets bundled. This will be used for progress output.

tests: test basic template support for status

templates: add a default template style for status Color doesn't work yet, so no labels here.

formatter: add template support This lets all the non-log commands that use the formatter use templates. There are still some things that don't work, for instance: - color (needs "repo" in map) - shortest (needs "ctx" in map)

formatter: add a method to build a full templater from a -T option

formatter: move most of template option helper to formatter We want to share this function between formatter and cmdutils. It doesn't belong in templater because it imports knowledge of ui layers that shouldn't be there. We'd prefer cmdutil to layer on the formatter rather than vice-versa. Since the formatter is the handler for -T options for all non-log commands, let's move the helper there. We leave the bits specific to the old --style option behind.

color: copy docstring of label() template function to wrapper Otherwise label() wouldn't be listed in "hg help template" if color extension is enabled.

templater: make pad function evaluate both string and rawstring templates "pad" function and "rawstring" type were introduced in parallel, aa51392da507 in default and 5ab28a2e9962 in stable respectively. Therefore, "pad" function lacked handling of "rawstring" unintentionally.

largefiles: ignore hidden changesets with 'verify --large --lfa' Previously, if there were any hidden changesets, the --lfa argument would cause the command to abort with a hint about using --hidden when it tripped over a hidden changeset.

bundle2: clarify in docstring that header size is for a single header The docstring for the header size field currently says "The total number of Bytes used by the part headers", but the size is about a single header, so let's change it to "header".

bundle2: rename duplicate handlepushkeyreply to handleobsmarkerreply The function was only called through the parthandlermapping dict, so it was confusing but safe in practice.

changegroup: remove 'getchangegroupraw' function There is no remaining caller for this function.

exchange: expand usage of getchangegroupraw The 'getchangegroupraw' is very simple (two lines) so we inline it in its only caller. This exposes the 'outgoing' object of the part generator function, allowing us to add information on the number of changesets contained in the part in a later changeset. Such information is useful for progress bar.

getbundle: have a single getchangegroupraw call site Having a single call site will simplify the code and help with coming refactoring.

phases: abort the whole push if phases fail to update (BC) When using bundle2, the phase pushkey parts are now made mandatory. As a result, failure to update the bookmark server side will result in the transaction being aborted.

bookmarks: abort the whole push if bookmarks fails to update (BC) When using bundle2, the bookmark's pushkey parts are now made mandatory. As a result failure to update the bookmark server side will result in the transaction being aborted.

httppeer: allow extensions to replace urllib2.Request The authentication library my extension wants to use requires using a different opener and a different request builder. This change pulls the call to urllib2.Request out so that my extension can replace it just like it can replace urlopener.

progress: move all logic altering the ui object logic in mercurial.ui The ui object can take care of its progress object logic by itself. test-subrepo-recursion is modified because it is a bit sensitive to the "no progress bar" default. It will become unnecessary in the next step when progress will be on by default in core.

progress: move the singleton logic to the ui module The use of a singleton for all of progress handling is debatable (because config may vary). However this is how the extension has been doing it so far. We move that code into the ui module because this is where is should belong when progress is moved into core.

progress: move most extension code into a 'mercurial.progress' module This initiate the relocation of progress into core.

test-subrepo-recursion: restore globs for Windows

tests: restore 'python' and '$TESTDIR/' for dummyssh invocation This is a backout of 46727fea7a00, and a partial backout of c3ecbf694904. Windows won't execute 'dummyssh' directly, presumably because CreateProcess() doesn't know how to execute a bash script: $ hg clone -e "dummyssh" ssh://user@dummy/cloned sshclone remote: 'dummyssh' is not recognized as an internal or external command, remote: operable program or batch file. abort: no suitable response from remote hg! [255] With the restoration of python as the executable, $TESTDIR needs to be restored for these invocations, because python won't search $PATH for 'dummyssh': $ hg clone -e "python dummyssh" ssh://user@dummy/cloned sshclone remote: python: can't open file 'dummyssh': [Errno 2] No such file or directory abort: no suitable response from remote hg! [255]

perf: support -T for every perf commands We are already building a formatter, we can now pass it options the official way.

bundle2: convey PushkeyFailed error over the wire We add a way to convey the precise exception. This will allow better error message on the server.

bundle2: also capture reply capability on failure When unbundling over the wire is aborted, we have a mechanism to convey the error inside a bundle part. As we add support for more errors, we need to know if the client will support them. For this purpose, we duck punch the reply capabilities of the client on the raised extensions. This is similar to what is done to salvage the server output on error.

bundle2: add an 'error' capability This capability will be extended as new error type is introduced.

merge with stable

templater: introduce indent function

hgewb: disable progress when serving (issue4582) Before this patch, progress bar could be displayed when serving, creating hypothetical problems.

test-subrepo-recursion.t: fix progress output on no-hardlink systems

test-ssh: stablize for platform-specific shell quoting Windows and OpenVMS use double quotes instead of single.

push: catch and process PushkeyFailed error We add a way to register "pushkey failure callback" that will be used if the push is aborted by a pushkey failure. A part generator adding mandatory pushkey parts should register a failure callback for all of them. The callback will be in charge of generating a meaningful abort if this part fails. If no callback is registered, the error is propagated. Catch PushkeyFailed error in exchange.

bundle2: introduce a PushkeyFail error to abort unbundle on pushkey error The pushkey code is generic and the server side has little context on what the client is trying to achieve. Generating interesting error messages server side would be challenging. Instead we introduce a dedicated exception that carries more data. In particular, it carries the id of the part which failed that will allow clients to display custom error messages depending on the part intent. The processing and transfer-over-the-wire of this exception is to be implemented in coming changesets.

record: exiting editor with non-zero status should not stop recording session Before this patch, exiting a hunk edit in record with a non-zero status lead to the end of the recording session, losing previously-selected hunks to record. This patch introduces the more desirable behavior of warning the user and continuing the recording session.

progress: stop double-wrapping of ui class We were wrapping the ui class again and again (uisetup, reposetup, subrepo setup, remote repo setup, etc). We now avoid that. This has impact on tests that were double-printing data because of this.

bundle2: abort when a mandatory pushkey part fails So far, result of a pushkey operation had no consequence on the transaction (beside the change). We makes it respect the 'mandatory' flag of part so that failed pushkey call abort the whole transaction. This will allow rejecting changes (primary target: changesets) regarding phases or bookmark criteria in the future (when we will push such data in a mandatory part). We currently raise an abort error because all clients support it. We'll introduce a more precise error in the next changesets.

push: make pushkey part advisory The current behavior (with bundle1) is to let the rest of the push succeed if the pushkey call (phases, bookmarks) failed (this comes from the fact that each item is sent in its own command). We kept this behavior with bundle2, which is highly debatable, but let us keep thing as they are now as a start. We are about to enforce 'mandatory' pushkey part as 'mandatory' successful, so we need to marks parts as advisory to preserve the current (debatable) behavior.

bundle2: pull bookmark the old way if no bundle2 listkeys support (issue4701) All known server implementations have listkeys support with bundle2, but people in the process of implementing new servers may not. Let's be nice with them.

tests: simplify printenv calls Make printenv executable so that we don't need python, TESTDIR, or quoting.

tests: make printenv executable

tests: simplify calls to dummyssh dummyssh is marked executable and is in the path, no need for python, TESTDIR, or quotes.

tests: remove duplicate inline dummyssh in test-wireproto.t

tests: drop DAEMON_PIDS from killdaemons calls

tests: make killdaemons.py use DAEMON_PIDS by default

tests: drop explicit $TESTDIR from executables $TESTDIR is added to the path, so this is superfluous. Also, inconsistent use of quotes means we might have broken on tests with paths containing spaces.

templater: do not preprocess template string in "if" expression (issue4714) The problem was spotted at 5ab28a2e9962, that says "this patch invokes it with "strtoken='rawstring'" in "_evalifliteral()", because "t" is the result of "arg" evaluation and it should be "string-escape"-ed if "arg" is "string" expression." This workaround is no longer valid since 890845af1ac2 introduced strict parsing of '\{'. Instead, we should interpret bare token as "string" or "rawstring" template. This is what buildmap() does at parsing phase.

largefiles: use the optional badfn argument when building a matcher The monkey patching in cat() can't be fixed, because it still delegates to the original bad(). Overriding commands.cat() should go away in favor overriding cmdutil.cat() anyway, and that matcher can be wrapped with matchmod.badmatch().

mq: use the optional badfn argument when building a matcher

commands: use the optional badfn argument when building a matcher

scmutil: add an optional parameter to matcher factories for a bad() override Even though scmutil.matchandpats() is documented to warn about bad files, several callers silence the warning.

scmutil: use the optional badfn argument when building a matcher

context: add an optional constructor parameter for a match.bad() override Most matcher creation is done by way of a context.

match: add an optional constructor parameter for a bad() override This will be used to eliminate monkey patching of new matcher instances that weren't removed in 5984dd42e140::1a95c57959f6.

mail: pass ui to sslutil.wrapsocket() even if verifycert is off (issue4713) 21b536f01eda made 'ui' argument is passed via sslutil.sslkwargs(), but mailer doesn't call sslkwargs() if smtp.verifycert is off. So we have to put it in sslkwargs manually.

windows: use os.SEEK_END

merge with stable

crecord: fix three typos introduced while moving crecord into core When moving crecord to core, I did a search an replace to remove all of the variable with caps letter. Doing so, I unfortunately changed some commands with capital letter shortcut to their lowercase counterpart. I fixed one of these cases in 093d38165e5a. This patch fixes all of the remaining cases.

revlog: move size limit check to addrevision This lets us add the name of the indexfile to the message.

sshpeer: also use doublepipe for client to server communication This will allow even more real time output when the server issue output in the middle a stream push.

sshpeer: allow doublepipe on unbuffered main pipe The output pipe does not have manually managed read buffer (actually, no read anything). To also use the doublepipe for outgoing write (useful to consume remote output when pushing large set of data) we need the doublepipe to work on standard pipe too.

sshpeer: allow write operations through double pipe We have a shiny toy, lets make it wider.

sshpeer: rename 'size' to 'data' in doublepipe We are about to add 'write' support, the argument will be either an int or a string.

mq: ban \r and \n in patch names (issue4711) This is at best crazy, and at worst will break things like the series file. Let's just stop the madness.

mq: use %r to format illegal characters instead of manually quoting This will make it easier to ban \r and \n in the next patch and still have a sensible error message.

histedit: abort rather than edit a public changeset (issue4704) This is suboptimal as the user still has to explicitly cancel the histedit afterwards, but it prevents the immediate problem. histedit should probably implicitly do 'hg histedit --abort' if a util.Abort is raised internally.

test-histedit-edit: add test that demonstrates bug in issue4704 Durham and I agree that it's a bug you can fold into a change that's not listed in the histedited set, so we'll follow this up with a change that prevents folds as the first edit step.

histedit: copyedit docstring wording problem I noticed while here

merge with stable

dirstate: avoid invalidating every entries when list is empty Default value was not tested with 'is None', this made empty list seen as default value and result the invalidation of every single entry in the dirstate. On repos with hundred of thousand of files, this results in minutes of lookup time instead nothing. This is a text book example of why we should test 'is None' if this is what we mean.

crecord: fix a typo introduced when moving crecord to core When moving crecord to core, I did a search an replace to remove all of the variable with caps letter. Doing so, I unfortunately changed the 'F' command to 'f' leading to two 'f' commands, one never used. This patch fixes this mistake and re enables the behavior of 'F'.

pull: prevent race condition in bookmark update when using -B (issue4689) We are already fetching remote bookmarks to honor the -B option, we now pass that data to the pull process so it can reuse it. This prevents a race condition between the initial looking and the actual pulling of changesets and bookmarks. Tests are updated to handle this fact.

pull: allow a generic way to pass parameters to the pull operation We have been feeling the need for this in extensions for quite some time. This will be used to pass remote bookmark information around in the next changesets.

pull: skip pulling remote bookmarks with bundle2 if a value already exists For efficiency and consistency purpose, remote bookmarks, retrieved at the time the pull command code is doing lookup, will be reused during the core pull operation. A second step toward this is to avoid requesting bookmark information in the bundle 2 if we already have them locally.

pull: skip pulling remote bookmarks with bundle1 if a value already exist For efficiency and consistency purpose, remote bookmarks, retrieved at the time the pull command code is doing lookup, will be reused during the core pull operation. A first step toward this is to setup the logic avoiding pulling the data again during the discovery phase if some have already been provided.

test: pull through http when testing for race conditions The http server is stateless giving more occasion for race. We switch the test to http before adding extra cases tests.

test: display pre-pull remote bookmark state when testing race condition This makes it easier to read the test. This will be usefull for comming changesets testing more race conditions.

largefiles: replace match.bad() monkey patching with match.badmatch() No known issues with the previous code since it restored the original method, but this is cleaner. The monkey patching in cat is harmless, because it is created locally, and doesn't pass it anywhere (subrepo cat isn't supported with largefiles).

revert: replace match.bad() monkey patching with match.badmatch() No known issues with the previous code since it immediately overwrote the patched, locally create matcher.

cat: replace match.bad() monkey patching with match.badmatch() No known issues with the previous code since it restored the original method, but this is cleaner.

forget: replace match.bad() monkey patching with match.badmatch() The previous code didn't restore the original method, but it looks like the worst that would happen is junk added to a list that had already been processed by previous subrepo invocation(s).

add: replace match.bad() monkey patching with match.badmatch() The previous code didn't restore the original method, but it looks like the worst that would happen is junk added to a list that had already been processed by previous subrepo invocation(s).

context: replace match.bad() monkey patching with match.badmatch() No known issues with the previous code since it restored the original method, but this is cleaner.

addremove: replace match.bad() monkey patching with match.badmatch() No known issues with the previous code since it restored the original method, but this is cleaner.

match: introduce badmatch() to eliminate long callback chains with subrepos Various bit of code replace the bad method on matchers, and then delegate to the original bad method after doing some custom processing. At least some of these forget to restore the original method when the need has passed, and then when the matcher is passed to the next subrepo (even a sibling), another layer is added such that the chain looks like: bad2 -> bad1 -> original At best, this is a waste of processing, but sometimes spurious messages can be emitted (e.g. ccb1623266eb). The trick with this copy of the matcher is to make sure it is *not* passed to any subrepo- the original must be passed instead.

ssl: remove CERT_REQUIRED constant that was necessary for compatibility

ssl: drop try-except clause that was necessary for ancient Python

ssl: drop support for Python < 2.6, require ssl module try-except clause is kept for readability of this patch, and it will be removed soon.

ssl: rename ssl_wrap_socket() to conform to our naming convention I've removed ssl_ prefix because the module name contains ssl.

test-https: kill only hgweb daemon to restart for client-auth test This avoids a test failure on Mac OS X. Because tinyproxy.py isn't detached from the shell, it may complain that a child process is terminated by a signal.

merge with stable

hgwebdir: don't allow the hidden parent of a subrepo to show as a directory Previously, if a subrepo parent had 'web.hidden=True' set, neither the parent nor child had a repository entry. However, the directory entry for the parent would be listed (it wouldn't have the fancy 'web.name' if configured), and that link went to the repo's summary page, effectively making it not hidden. This simply disables the directory processing if a valid repository is present. Whether or not the subrepo should be hidden is debatable, but this leaves that behavior unchanged (i.e. it stays hidden).

pull: avoid race condition with 'hg pull --rev name --update' (issue4706) The previous scheme was: 1) lookup node for all pulled revision, 2) pull said node 3) lookup the node of the checkout target 4) update the repository there. If the remote repo changes between (1) and (3), the resolved name will be different and (3) crash. There is actually no need for a remote lookup during (3), we could just set the value in (1). This prevent the race condition and save a possible network roundtrip.

revert: add an experimental config to use inverted selection We had a discussion on the list about the interactive ui for revert. This patch adds a flag to allow people to test the second alternative (referred to as proposition 2 on the mailing list). It effectively inverts the signs in the

bundle2: stop capturing output for ssh again This backout 36111f98f23d since we can have real time output with ssh again. The tests change is not backed-out because it was a test output fix.