pull: remove inadequate use of operations records to update stepdone The 'stepdone' set is design to be a client side mechanism. If the client used some advanced capabilities to request necessary information (changeset, obsmarkers, phases, etc). It marks the steps as done to avoid having a less advanced mechanism issue a duplicated request. So, the "stepdone.add('phases')" should be the result of a client choice, because only the client can know it has requested all it needed to request. In 4a08cf1a2cfe this principle was broken because any phase-heads part sent by the server to the client would declare the phases retrieval complete. Now that there is an official phases related capability and code associated to it. We do not need the change in 4a08cf1a2cfe anymore and we can back it out. This brings back 'stepdone' management for 'phases' in line with the rest of the code (including other phases handing). Here is an example of potential misbehavior that 4a08cf1a2cfe introduced: Imagine a server that pre-computes bundles. The bundles contains a changegroup part and an (advisory) 'phase-heads' part. When a pull occurs, precomputed bundled are reused if available. As the phase part is advisory it can be sent to all clients. However they could be relevant changesets without phase information. Either because they are already common or because they had no precomputed bundle for them yet. If receiving any 'phase-heads' parts disable subsequent phases re-trivial parts, the client will not request phase data for all relevant changesets. For example common changesets will not turn public.

pull: use 'phase-heads' to retrieve phase information A new bundle2 capability 'phases' has been added. If 'heads' is part of the supported value for 'phases', the server supports reading and sending 'phase- heads' bundle2 part. Server is now able to process a 'phases' boolean parameter to 'getbundle'. If 'True', a 'phase-heads' bundle2 part will be included in the bundle with phase information relevant to the whole pulled set. If this method is available the phases listkey namespace will no longer be listed. Beside the more efficient encoding of the data, this new method will greatly improve the phase exchange efficiency for repositories with non-served changesets (obsolete, secret) since we'll no longer send data about the filtered heads. Add a new 'devel.legacy.exchange' config item to allow fallback to the old 'listkey in bundle2' method. Reminder: the pulled set is not just the changesets bundled by the pull. It also contains changeset selected by the "pull specification" on the client side (eg: everything for bare pull). One of the reason why the 'pulled set' is important is to make sure we can move -common- nodes to public.

bundle2: only grab a transaction when 'phase-heads' affect the repository The next patch will use the 'phase-heads' part to exchange phase data relevant to the pulled set. 'handlephases' currently acquires a transaction even in case of no-op pull, which would results in an empty transaction and messing with the existing journal. Pass the transaction fetcher to updatephases so it can fetch it if necessary.

phases: move the binary decoding function in the phases module We move the decoding function near the encoding one in a place where they can be reused in other place (current target, 'exchange.py').

phases: move binary encoding into a reusable function We want to use binary phases for pushing and pulling. We extract the encoding function out of the bundle2 module first.

phases: use a Struct object for binary encoding and decoding We will move the binary encoding and decoding code to 'phases.py' in order to make it easier to reuse. First, let's cleanup it a bit.

discovery: avoid dropping remote heads hidden locally An extra post processing was added to recognize remote heads that are hidden locally as "common" instead of "unknown". However, this processing was removing such hidden heads from the remote heads sets. It had no impact because we used to pull phase information from all remote heads. This series will replace the phase pulling operation to a more efficient process but requires the unmodified pulled set information.

test-patchbomb: use mocktime The test was using system time for displaying ETAs, which could be flaky if the sysload is high. This patch extracts mocktime.py from test-progress.t to make sure test-patchbomb.t is unaffected by system time. Differential Revision: https://phab.mercurial-scm.org/D844

test-patchbomb: fix the test With the experimental config `progress.estimate` removed, the progress output in `test-patchbomb.t` has a minor change: it shows ETA since the beginning. (This could be folded into f428c3)

progress: make ETA only consider progress made in the last minute This patch limits the estimate time interval to roughly the last minute (configurable by `estimateinterval`) to be more practical. See the test change for why this is better. .. feature:: Estimated time is more accurate with non-linear progress Differential Revision: https://phab.mercurial-scm.org/D820

progress: remove progress.estimate config It was introduced by 98e4d39 ("progress: add speed format" 2011-5-9) and was intended to hide ETA information for the first few seconds. Later 5d261fd ("progress: add a changedelay to prevent parallel topics from flapping (issue2698)" 2011-6-23) introduced `changedelay` config which hides the entire progress bar for the first few seconds. So `progress.estimate` seems somehow duplicated feature-wise. Since it's experimental and duplicated, let's just remove it. This makes the next patch simpler - it no longer needs to make sure `starttimes` is the real start time. Differential Revision: https://phab.mercurial-scm.org/D828

progress: demonstrate non-linear progress has a bad ETA experience Previously, the ETA and speed assumes the progress is linear. Often, due to network or other issues, it could be fast for the most time, and suddenly slow down: [====================================================> ] \___________________________________________/\______/ very fast suddenly much slower This patch adds a test demonstrating the ETA could be way off in those cases. Differential Revision: https://phab.mercurial-scm.org/D819

copytrace: add a a new config to limit the number of drafts in heuristics The heuristics options tries to the default full copytracing algorithm if both the source and destination branches contains of non-public changesets only. But this can be slow in cases when we have a lot of drafts. This patch adds a new config option experimental.copytrace.sourcecommitlimit which defaults to 100. This value will be the limit of number of drafts from c1 to base. Incase there are more changesets even though they are draft, the heuristics algorithm will be used. Differential Revision: https://phab.mercurial-scm.org/D763

mail: encode long unicode lines in emails properly (issue5687) 3e544c074459 introduced a bug: emails Content-Transfer-Encoding is silently replaced with 'quoted-printable' while any other encoding could be used by underlying code. The problem is revealed when a long unicode line is encoded. The patch implements proper check which works for any text and encoding.

chg: show timestamp with debug messages Like `strace -tr`, this helps finding performance bottlenecks. Differential Revision: https://phab.mercurial-scm.org/D807