dirstate: move nonnormalentries to dirstatemap As part of moving dirstate storage to its own class, let's move the nonnormalentries logic onto the dirstatemap class. This will let extensions replace the nonnormalentries logic with a persisted cache. Differential Revision: https://phab.mercurial-scm.org/D753

dirstate: create new dirstatemap class This is part of a larger refactor to move the dirstate storage logic to a separate class, so it's easier to rewrite the dirstate storage layer without having to rewrite all the algorithms as well. Step one it to create the class, and replace dirstate._map with it. The abstraction bleeds through in a few places where the main dirstate class has to access self._map._map, but those will be cleaned up in future patches. Differential Revision: https://phab.mercurial-scm.org/D752

style: always use `x is not None` instead of `not x is None` Differential Revision: https://phab.mercurial-scm.org/D842

templatekw: add new-style template expansion to {manifest} The goal is to allow us to easily access to nested data. The dot operator will be introduced later so we can write '{p1.files}' instead of '{revset("p1()") % "{files}"}' for example. In the example above, 'p1' needs to carry a mapping dict along with its string representation. If it were a list or a dict, it could be wrapped semi-transparently with the _hybrid class, but for non-list/dict types, it would be difficult to proxy all necessary functions to underlying value type because several core operations may conflict with the ones of the underlying value: - hash(value) should be different from hash(wrapped(value)), which means dict[wrapped(value)] would be invalid - 'value == wrapped(value)' would be false, breaks 'ifcontains' - len(wrapped(value)) may be either len(value) or len(iter(wrapped(value))) So the wrapper has no proxy functions and its scope designed to be minimal. It's unwrapped at eval*() functions so we don't have to care for a wrapped object unless it's really needed: # most template functions just call evalfuncarg() unwrapped_value = evalfuncarg(context, mapping, args[n]) # if wrapped value is needed, use evalrawexp() maybe_wrapped_value = evalrawexp(context, mapping, args[n]) Another idea was to wrap every template variable with a tagging class, but which seemed uneasy without a static type checker. This patch updates {manifest} to a mappable as an example.

templater: adjust binding strength of '%' and '|' operators (BC) This makes 'foo|bar%baz' parsed as '(foo|bar)%baz', not 'foo|(bar%baz)'. Perhaps it was a mistake that '%' preceded '|'. Both '|' and '%' can be considered a kind of function application, and '|' is more like a '.' operator seen in OO languages. So IMHO '|' should have the same (or higher) binding as '%'. The BC breakage should be minimal since both '|' and '%' operators have strict requirements for their operands and 'foo|bar%baz' was invalid: - right-hand side of '|' must be a symbol - left-hand side of '%' must be a dict or list - right-hand side of '%' must be a string or symbol

templatekw: just pass underlying value (or key) to joinfmt() function Before, iter(hybrid) was proxied to hybrid.gen, which generated formatted strings. That's why we had to apply joinfmt() to the dicts generated by hybrid.itermaps(). Since this weird API was fixed at a0f2d83f8083, we can get rid of the makemap() calls from join().

scmutil: extract helper functions that returns human-readable change id We do "'%d:%s' % (ctx...)" at several places, so let's formalize it. A low- level function, formatrevnode(ui, rev, node), is extracted so we can pass a manifest rev/node pair. Note that hex() for manifest output can be replaced with hexfunc() because it is printed only when debugflag is set. i18n/de.po is updated so test-log.t passes with no error.

templater: extract helper to just evaluate template expression A named function can be easily grepped and is probably good for code readability.

templater: do not destructure operands in buildmap() This makes the next patch slightly simpler.

templater: use helper function to get name of non-iterable keyword

pull: remove inadequate use of operations records to update stepdone The 'stepdone' set is design to be a client side mechanism. If the client used some advanced capabilities to request necessary information (changeset, obsmarkers, phases, etc). It marks the steps as done to avoid having a less advanced mechanism issue a duplicated request. So, the "stepdone.add('phases')" should be the result of a client choice, because only the client can know it has requested all it needed to request. In 4a08cf1a2cfe this principle was broken because any phase-heads part sent by the server to the client would declare the phases retrieval complete. Now that there is an official phases related capability and code associated to it. We do not need the change in 4a08cf1a2cfe anymore and we can back it out. This brings back 'stepdone' management for 'phases' in line with the rest of the code (including other phases handing). Here is an example of potential misbehavior that 4a08cf1a2cfe introduced: Imagine a server that pre-computes bundles. The bundles contains a changegroup part and an (advisory) 'phase-heads' part. When a pull occurs, precomputed bundled are reused if available. As the phase part is advisory it can be sent to all clients. However they could be relevant changesets without phase information. Either because they are already common or because they had no precomputed bundle for them yet. If receiving any 'phase-heads' parts disable subsequent phases re-trivial parts, the client will not request phase data for all relevant changesets. For example common changesets will not turn public.

pull: use 'phase-heads' to retrieve phase information A new bundle2 capability 'phases' has been added. If 'heads' is part of the supported value for 'phases', the server supports reading and sending 'phase- heads' bundle2 part. Server is now able to process a 'phases' boolean parameter to 'getbundle'. If 'True', a 'phase-heads' bundle2 part will be included in the bundle with phase information relevant to the whole pulled set. If this method is available the phases listkey namespace will no longer be listed. Beside the more efficient encoding of the data, this new method will greatly improve the phase exchange efficiency for repositories with non-served changesets (obsolete, secret) since we'll no longer send data about the filtered heads. Add a new 'devel.legacy.exchange' config item to allow fallback to the old 'listkey in bundle2' method. Reminder: the pulled set is not just the changesets bundled by the pull. It also contains changeset selected by the "pull specification" on the client side (eg: everything for bare pull). One of the reason why the 'pulled set' is important is to make sure we can move -common- nodes to public.

bundle2: only grab a transaction when 'phase-heads' affect the repository The next patch will use the 'phase-heads' part to exchange phase data relevant to the pulled set. 'handlephases' currently acquires a transaction even in case of no-op pull, which would results in an empty transaction and messing with the existing journal. Pass the transaction fetcher to updatephases so it can fetch it if necessary.

phases: move the binary decoding function in the phases module We move the decoding function near the encoding one in a place where they can be reused in other place (current target, 'exchange.py').

phases: move binary encoding into a reusable function We want to use binary phases for pushing and pulling. We extract the encoding function out of the bundle2 module first.

phases: use a Struct object for binary encoding and decoding We will move the binary encoding and decoding code to 'phases.py' in order to make it easier to reuse. First, let's cleanup it a bit.

discovery: avoid dropping remote heads hidden locally An extra post processing was added to recognize remote heads that are hidden locally as "common" instead of "unknown". However, this processing was removing such hidden heads from the remote heads sets. It had no impact because we used to pull phase information from all remote heads. This series will replace the phase pulling operation to a more efficient process but requires the unmodified pulled set information.

test-patchbomb: use mocktime The test was using system time for displaying ETAs, which could be flaky if the sysload is high. This patch extracts mocktime.py from test-progress.t to make sure test-patchbomb.t is unaffected by system time. Differential Revision: https://phab.mercurial-scm.org/D844

test-patchbomb: fix the test With the experimental config `progress.estimate` removed, the progress output in `test-patchbomb.t` has a minor change: it shows ETA since the beginning. (This could be folded into f428c3)

progress: make ETA only consider progress made in the last minute This patch limits the estimate time interval to roughly the last minute (configurable by `estimateinterval`) to be more practical. See the test change for why this is better. .. feature:: Estimated time is more accurate with non-linear progress Differential Revision: https://phab.mercurial-scm.org/D820

progress: remove progress.estimate config It was introduced by 98e4d39 ("progress: add speed format" 2011-5-9) and was intended to hide ETA information for the first few seconds. Later 5d261fd ("progress: add a changedelay to prevent parallel topics from flapping (issue2698)" 2011-6-23) introduced `changedelay` config which hides the entire progress bar for the first few seconds. So `progress.estimate` seems somehow duplicated feature-wise. Since it's experimental and duplicated, let's just remove it. This makes the next patch simpler - it no longer needs to make sure `starttimes` is the real start time. Differential Revision: https://phab.mercurial-scm.org/D828

progress: demonstrate non-linear progress has a bad ETA experience Previously, the ETA and speed assumes the progress is linear. Often, due to network or other issues, it could be fast for the most time, and suddenly slow down: [====================================================> ] \___________________________________________/\______/ very fast suddenly much slower This patch adds a test demonstrating the ETA could be way off in those cases. Differential Revision: https://phab.mercurial-scm.org/D819

copytrace: add a a new config to limit the number of drafts in heuristics The heuristics options tries to the default full copytracing algorithm if both the source and destination branches contains of non-public changesets only. But this can be slow in cases when we have a lot of drafts. This patch adds a new config option experimental.copytrace.sourcecommitlimit which defaults to 100. This value will be the limit of number of drafts from c1 to base. Incase there are more changesets even though they are draft, the heuristics algorithm will be used. Differential Revision: https://phab.mercurial-scm.org/D763

mail: encode long unicode lines in emails properly (issue5687) 3e544c074459 introduced a bug: emails Content-Transfer-Encoding is silently replaced with 'quoted-printable' while any other encoding could be used by underlying code. The problem is revealed when a long unicode line is encoded. The patch implements proper check which works for any text and encoding.

chg: show timestamp with debug messages Like `strace -tr`, this helps finding performance bottlenecks. Differential Revision: https://phab.mercurial-scm.org/D807

keepalive: add more context to bad status line errors As the TODO in the test said, the previous error message was not very helpful. Let's improve things. Differential Revision: https://phab.mercurial-scm.org/D811

tests: add interface checks for bundle, statichttp, and union peers I forgot to add these when I initially wrote the test. They inherit from localrepo.localpeer, so they should be explicitly tested. Differential Revision: https://phab.mercurial-scm.org/D810

alias: make alias command lazily resolved With many aliases, resolving them could have some visible overhead. Below is part of traceprof [1] output of `hg bookmark --hidden`: (time unit: ms) 37 \ addaliases dispatch.py:526 37 | __init__ (60 times) dispatch.py:402 33 | findcmd (108 times) cmdutil.py:721 16 | findpossible (49 times) cmdutil.py:683 It may get better by optimizing `findcmd` to do a bisect, but we don't really need to resolve an alias if it's not used, so let's make those command entries lazy. After this patch, `addalias` takes less than 1ms. .. perf:: improved performance when many aliases are defined [1]: https://bitbucket.org/facebook/hg-experimental/src/9aca0dbdbdfc48457e5d2581ca2d6e662fced2e6/hgext3rd/traceprof.pyx Differential Revision: https://phab.mercurial-scm.org/D805

alias: test duplicated definition earlier This patch moves the old definition checking logic introduced by f4b7be3f8430 earlier. So that the test itself does not depend on `aliasdef`. The check is to avoid wrapping a same alias multiple times. It can be done by checking the config name and value (`definition` in code), without constructing a `cmdalias` instance. This makes the next patch easier to review. Differential Revision: https://phab.mercurial-scm.org/D804

uncommit: add a test for uncommit with uncommitondirtywdir config in merge Differential Revision: https://phab.mercurial-scm.org/D809