windows: read all global config files, not just the first (issue4491) (BC) On windows, hgrc.d/*.rc would not be read if mercurial.ini was found. That was far from obvious from the documentation and different from the behavior on posix systems. As a consequence of this, TortoiseHg cacert configuration placed in hgrc.d would not be read if an old global mercurial.ini still existed. "hg config -g" could also crash when no global configuration files could be found. Instead, make windows behave like posix and read all global configuration files. The documentation was in a way right that individual config settings in the global Mercurial.ini would override settings from for example .hgrc.d\*.rc, but only because the .d files not would be read at all if a Mercurial.ini was found. The ordering in the documentation is thus changed to match the code.

strip: factor out revset calculation for strip -B This will allow reusing it in evolve and overriding it in other extensions.

clonebundles: support for seeding clones from pre-generated bundles Cloning can be an expensive operation for servers because the server generates a bundle from existing repository data at request time. For a large repository like mozilla-central, this consumes 4+ minutes of CPU time on the server. It also results in significant network utilization. Multiplied by hundreds or even thousands of clients and the ensuing load can result in difficulties scaling the Mercurial server. Despite generation of bundles being deterministic until the next changeset is added, the generation of bundles to service a clone request is not cached. Each clone thus performs redundant work. This is wasteful. This patch introduces the "clonebundles" extension and related client-side functionality to help alleviate this deficiency. The client-side feature is behind an experimental flag and is not enabled by default. It works as follows: 1) Server operator generates a bundle and makes it available on a server (likely HTTP). 2) Server operator defines the URL of a bundle file in a .hg/clonebundles.manifest file. 3) Client `hg clone`ing sees the server is advertising bundle URLs. 4) Client fetches and applies the advertised bundle. 5) Client performs equivalent of `hg pull` to fetch changes made since the bundle was created. Essentially, the server performs the expensive work of generating a bundle once and all subsequent clones fetch a static file from somewhere. Scaling static file serving is a much more manageable problem than scaling a Python application like Mercurial. Assuming your repository grows less than 1% per day, the end result is 99+% of CPU and network load from clones is eliminated, allowing Mercurial servers to scale more easily. Serving static files also means data can be transferred to clients as fast as they can consume it, rather than as fast as servers can generate it. This makes clones faster. Mozilla has implemented similar functionality of this patch on hg.mozilla.org using a custom extension. We are hosting bundle files in Amazon S3 and CloudFront (a CDN) and have successfully offloaded >1 TB/day in data transfer from hg.mozilla.org, freeing up significant bandwidth and CPU resources. The positive impact has been stellar and I believe it has proved its value to be included in Mercurial core. I feel it is important for the client-side support to be enabled in core by default because it means that clients will get faster, more reliable clones and will enable server operators to reduce load without requiring any client-side configuration changes (assuming clients are up to date, of course). The scope of this feature is narrowly and specifically tailored to cloning, despite "serve pulls from pre-generated bundles" being a valid and useful feature. I would eventually like for Mercurial servers to support transferring *all* repository data via statically hosted files. You could imagine a server that siphons all pushed data to bundle files and instructs clients to apply a stream of bundles to reconstruct all repository data. This feature, while useful and powerful, is significantly more work to implement because it requires the server component have awareness of discovery and a mapping of which changesets are in which files. Full, clone bundles, by contrast, are much simpler. The wire protocol command is named "clonebundles" instead of something more generic like "staticbundles" to leave the door open for a new, more powerful and more generic server-side component with minimal backwards compatibility implications. The name "bundleclone" is used by Mozilla's extension and would cause problems since there are subtle differences in Mozilla's extension. Mozilla's experience with this idea has taught us that some form of "content negotiation" is required. Not all clients will support all bundle formats or even URLs (advanced TLS requirements, etc). To ensure the highest uptake possible, a server needs to advertise multiple versions of bundles and clients need to be able to choose the most appropriate from that list one. The "attributes" in each server-advertised entry facilitate this filtering and sorting. Their use will become apparent in subsequent patches. Initial inspiration and credit for the idea of cloning from static files belongs to Augie Fackler and his "lookaside clone" extension proof of concept.

sslutil: expose attribute indicating whether SNI is supported This will be used so clone bundles can advertise whether URLs require SNI. This will be explained more in a subsequent patch.

resolve: perform all premerges before performing any file merges (BC) Just like the BC to merge before it, this allows for a maximally consistent state before providing any prompts to the user.

test-resolve.t: add some tests for .orig file contents An upcoming patch will touch some code around this area, and I couldn't find any tests related to this.

test-resolve.t: add some output to show order of operations This basically shows the behavior of resolve with multiple files. An upcoming behavior change will cause this output to also change.

merge.mergestate: perform all premerges before any merges (BC) We perform all that we can non-interactively before prompting the user for input via their merge tool. This allows for a maximally consistent state when the user is first prompted. The test output changes indicate the actual behavior change happening.

merge: introduce a preresolve function The section of code that writes out the version of the file cached in the merge state should only be run at preresolve time. This is so that if the premerge keeps around conflict markers, those don't get overwritten before the main merge.

merge.mergestate._resolve: also return completed status We'll need this for a new 'preresolve' function we're adding.

merge.mergestate: add a wrapper around resolve The resolve function will be broken up into separate pre-resolve and resolve steps.

simplemerge: move conflict warning message to filemerge The current output for a failed merge with conflict markers looks something like: merging foo warning: conflicts during merge. merging foo incomplete! (edit conflicts, then use 'hg resolve --mark') merging bar warning: conflicts during merge. merging bar incomplete! (edit conflicts, then use 'hg resolve --mark') We're going to change the way merges are done to perform all premerges before all merges, so that the output above would look like: merging foo merging bar warning: conflicts during merge. merging foo incomplete! (edit conflicts, then use 'hg resolve --mark') warning: conflicts during merge. merging bar incomplete! (edit conflicts, then use 'hg resolve --mark') The 'warning: conflicts during merge' line has no context, so is pretty confusing. This patch will change the future output to: merging foo merging bar warning: conflicts while merging foo! (edit, then use 'hg resolve --mark') warning: conflicts while merging bar! (edit, then use 'hg resolve --mark') The hint on how to resolve the conflicts makes this a bit unwieldy, but solving that is tricky because we already hint that people run 'hg resolve' to retry unresolved merges. The 'hg resolve --mark' mostly applies to conflict marker based resolution.

filemerge: clean up some dead code We now exit early if we do a premerge, so extra checks are no longer necessary.

run-tests: add b-prefix on two strings to fix python3 support

filemerge: break overall filemerge into separate premerge and merge steps This means that in ms.resolve we must call merge after calling premerge. This doesn't yet mean that all premerges happen before any merges -- however, this does get us closer to our goal. The output differences are because we recompute the merge tool. The only user-visible difference caused by this patch is that if the tool is missing we'll print the warning twice. Not a huge deal, though.

filemerge: only copy to backup during premerge step The premerge might leave the original file in an unclean state. Therefore it's important to only copy the file in the beginning.

filemerge: only print out "merging f" output at premerge step We're soon going to call this function twice, once for premerge and once for merge. This makes sure the "merging" output only gets printed during the premerge step.

filemerge: deindent the parts of filemerge outside the try block It is no longer necessary to indent these parts.

filemerge: introduce a premerge flag and function This flag will let us get to our overall goal of performing all premerges before any merges.

filemerge: also return whether the merge is complete In future patches, we'll pause merges after the premerge step. After the premerge step we'll return complete = False.

filemerge: add a wrapper around the filemerge function We'll introduce a separate premerge function that calls the same code.

context: don't hex encode all unknown 20 char revision specs (issue4890) d3908c911d5e introduced nice hexified display of missing nodes. It did however also make missing 20 character revision specifications be shown as hex - very confusing. Users are often wrong and somehow specify revisions that don't exist. Nodes will however rarely be missing ... and they will only look like a user provided revision specification and be all ascii in 1 of 4*10**9. With this change, missing revisions will only be hexified if they really look like binary nodes. This change will thus improve the error reporting UI in the common case and only very rarely make it confusing in the opposite direction of how it was before.

discovery: put trivial branch first Having the simple and tiny branch of the conditional first help readability. The "else" that appears after a screen of code is harder to relate to a conditional.

shelve: rename 'publicancestors' to something accurate (issue4737) That function is actually not returning public ancestors at all. This is pointed by the second line of the docstring... The bundling behavior was made correct in a5141977198d but with confusion remaining regarding what each function was doing. This close issue4737, because this highlight that shelve is actually -not- bundling too much data (this was actually properly tested).

makefile: add wheel build target

setup: import setup from setuptools if FORCE_SETUPTOOLS is set This should allow easier experimentation with using setuptools in mercurial's build automation, without breaking anything that currently depends on distutils behavior

hgweb: remove obsolete -webkit-border-radius property

monoblue: add a link to the latest file revision For reference, this was added to paper/coal in bb00a159e594 and to gitweb in b3b57ecbda50.

discovery: reference relevant bug in the faulty code We extend the comment about this code flaw with more code flaw.

discovery: fix a typo in a comment The idea here is that the code is imperfect, not that it is impossible to get something behaving properly.

getsubset: get the unpacker version from the bundler The current setup requires to pass both a packer and, optionally, the version of the unpacker. This is confusing and error prone as the two value cannot mismatch. Instead, we simply grab the version from the packer. This fixes a bug where requesting a cg2 from 'hg bundle' were reported as changegroup 1. I should have caught that in the initial changeset but I missed it somehow.

test-convert-cvs: add sleep so cvs notices changes This change makes the test pass on gcc112.fsffrance.org.

cvsps: fix computation of parent revisions when log caching is on cvsps computes the parent revisions of log entries by walking the cvs log sorted by (rcs, revision) and by iteratively maintaining a 'versions' dictionary which maps a (rcs, branch) pair onto the last revision seen for that pair. When log caching is on and a log cache exists, cvsps fails to set the parent revisions of new log entries because it does not iterate over the log cache in the parents computation. A complication is that a file rcs can change (move to/from the attic), with respect to its value in the log cache, if the file is removed/added back. This patch adds an iteration over the log cache to update the rcs of cached log entries, if changed, and to properly populate the 'versions' dictionary.

dirstate: batch calls to statfiles (issue4878) This makes it more interruptible.

parsers: fix infinite loop or out-of-bound read in fm1readmarkers (issue4888) The issue4888 was caused by 0-length obsolete marker. If msize is zero, fm1readmarkers() never ends. This patch adds several bound checks to fm1readmarker(). Therefore, 0-length and invalid-size marker should be rejected.

parsers: read sizes of metadata pair of obsolete marker at once This will make it easy to implement bound checking. Currently fm1readmarker() has no protection for corrupted obsstore and can cause infinite loop or out-of-bound reads.

filemerge: clean up temp files in a finally block This isn't really a big deal because the temp files are created in $TMPDIR, but it makes some upcoming work simpler.

check-code: detect and ban 'util.Abort' We have seen the light, please use the new way.

error: get Abort from 'error' instead of 'util' The home of 'Abort' is 'error' not 'util' however, a lot of code seems to be confused about that and gives all the credit to 'util' instead of the hardworking 'error'. In a spirit of equity, we break the cycle of injustice and give back to 'error' the respect it deserves. And screw that 'util' poser. For great justice.

eol: rename 'error' to 'haserror' The variable 'error' conflict with the module name that we would like to import and use in a coming changeset.

discovery: rename 'error' to 'errormsg' The variable 'error' conflict with the module name that we would like to import and use in a coming changeset.

histedit: delete histedit statefile on any exception during abort When an user aborts a histedit, many things could go wrong. At a minimum, after a histedit abort failure, their repository should be out of that state. We've found situations where the user could not exit the histedit state without manually deleting the histedit state file. This patch ensures that if any exception happens during an abort, the histedit statefile will be deleted so that users are out of the histedit state and can at least manually get the repo back to a workable condition.

histedit: check presence of statefile before deleting it When the histeditstate class instance has it's clear() method called, there is nothing to check to see if the state file exists before deleting it. It may not exist, which would create an exception. This patch allows clear to be called at any time. This will be needed for the following patch.

histedit: add inprogress method to state class If a histedit is progress, the 'histedit-state' file should exist. The patch implements a convenience function to do check if a histedit is in progress. This method will be use in next patch in the series.

commands: use dirstateguard instead of begin/end-parentchange for backout Before this patch, "hg backout" uses 'begin'/'end'-'parentchange()' of 'dirstate' class to avoid writing incomplete dirstate changes out at failure. But this framework doesn't work as expected, if 'dirstate.write()' is invoked between them. In fact, in-memory dirstate changes may be written out at 'repo.status()' implied by 'merge.update()', even before this patch. To restore dirstate as expected at failure of "hg backout", this patch uses 'dirstateguard' instead of 'begin'/'end'-'parentchange()'.

commands: make "hg import" use dirstateguard only for --no-commit Previous patch made dirstate changes in a transaction scope "all or nothing". Therefore, 'dirstateguard' is meaningless, if its scope is as same as one of the related transaction. Before this patch, "hg import" uses 'dirstateguard' always, but transaction is also started if '--no-commit' isn't specified. To avoid redundancy, this patch makes "hg import" use dirstateguard only if transaction isn't started (= '--no-commit' is specified). In this patch, 'if dsguard' can be examined safely, because 'dsguard' is initialized (with None) before outermost 'try'.

cmdutil: stop tryimportone from using dirstateguard (BC) There is no user of 'cmdutil.tryimportone()' other than 'commands.import_()', which can restore dirstate at failure of applying patches by transaction or dirstateguard. Therefore, it is reasonable to stop 'tryimportone()' from using redundant 'dirstateguard', even though it changes behavior of 'tryimportone()'. After this patch, 3rd party extensions should use 'dirstateguard' or so explicitly, if they want to restore dirstate at failure of importing a patch.

dirstate: remove meaningless dirstateguard Previous patch made dirstate changes in a transaction scope "all or nothing". Therefore, 'dirstateguard' is meaningless, if its scope is as same as one of the related transaction. This patch removes such meaningless 'dirstateguard' usage.

localrepo: execute appropriate actions for dirstate at releasing transaction Before this patch, in-memory dirstate changes are still kept over a transaction scope boundary regardless of the result of it. For "all or nothing" policy of the transaction, in-memory dirstate changes should be: - written out at successful closing a transaction, because subsequent 'dirstate.invalidate()' can lose them - discarded at failure of a transaction, because outer 'wlock.release()' or so may write them out To discard all changes in a transaction completely, this patch also restores '.hg/dirstate' by '.hg/journal.dirstate' at failure, because 'transaction' itself does nothing for files related to '.hg/journal.*' in such case (therefore, renaming in this patch is safe enough). This is a part of preparations for "transactional dirstate". See also the wiki page below for detail about it. https://mercurial.selenic.com/wiki/DirstateTransactionPlan This patch also removes redundant 'dirstate.invalidate()' just before aborting a transaction for shelve/unshelve.

transaction: add releasefn to notify the end of a transaction scope 'releasefn' is used by subsequent patch, to do appropriate action according to the result of it at the end of a transaction scope. To ensure that 'releasefn' is invoked only once, this patch invokes it after assignment 'self.journal = None', because such assignment prevents from invoked 'transaction._abort()' again via '__del__()'. def __del__(self): if self.journal: self._abort()

filemerge: move post-merge checks into a separate function This makes the overall filemerge function easier to follow, and makes upcoming work simpler.

filemerge._xmerge: drop no longer necessary 'if r:' check Cleanup from an earlier patch to make premerge be directly called from the main filemerge function.

filemerge._idump: drop no longer necessary 'if r:' check Cleanup from an earlier patch to make premerge be directly called from the main filemerge function.

filemerge._merge: drop no longer necessary 'if r:' check Cleanup from an earlier patch to make premerge be directly called from the main filemerge function.

revset: delete _updatedefaultdest as it has no users The revset is not used anywhere anymore. We delete the function until we use (and therefore test it again).

merge: get the default update destination from the function There is no value in using the revset instead of the extracted function.

update: move default destination computation to a function We ultimately want this to be accessible through a revset, but there is too much complexity here for that to work. Especially we'll have to return more than just the destination to control the behavior (eg: bookmarks to activate, etc). To prevent cycle, a new module is created, it will receive other destination/behavior function in the future.

worker: restore old countcpus code (issue4869) This is a backout of d29859cfcfc2. The stdlib implementation of multiprocessing.cpu_count() attempts to invoke a process on BSD and Darwin platforms (at least on 2.7). Under certain conditions (such as cwd being removed) this could raise. Our old code was silently catching the exception. The old code was more robust, so restore it.

filemerge: call premerge directly from main merge function The merge code currently does (in pseudocode): for f in tomerge: premerge f merge f We'd like to change this to look more like: for f in tomerge: premerge f for f in tomerge: merge f This makes sure as many files are resolved as possible before prompting for the others. This restructuring is also necessary for custom merge drivers. This function separates out the premerge step from the merge step. In future patches we'll actually turn these into separate steps in the merge driver. The 'if r:' occurrences will be cleaned up in subsequent patches.

bundle2: allow lazily acquiring the lock In the external pushrebase extension, it is valuable to be able to do some work without taking the lock (like running expensive hooks). This enables significantly higher commit throughput. This patch adds an option to lazily acquire the lock. It means that all bundle2 part handlers that require writing to the repo must first call op.gettransction(), when in this mode.

bundle2: add op.gettransaction() to handlers that need the lock A future patch will allow the bundle2 lock be taken lazily. We need to introduce transaction-gets to each handler that needs the lock. The tests caught these issues when I added lazy locking.

merge with stable

patchbomb: add a 'bundletype' config under 'patchbomb' patchbomb relies on the 'hg bundle' command to generate an attached bundle using --bundle. However, while 'hg bundle' has a --type option, patchbomb did not. This is becoming very relevant since we are about to issue bundle2 for general-delta repository. This was tracked as issue4863

import: allow processing of extra part header after import As we have a way for extension to add more header, we need a way for them to actually process them. We add a basic hook point to do extra work after the import have been committed.

import: allow processing of extra part header during import As we have a way for extension to add more header, we need a way for them to actually process them. We add a basic hook points to alter the changeset (especially extra) before we commit. There would be more to do for a full featured hooking, but this currently fit my needs.

extract: parse 'nodeid' using the generic mechanism Parsing 'nodeid' is very simple and a good first test.

extract: parse 'branch' using the generic mechanism Parsing 'branch' is very simple and a good first test.

extract: parse 'date' using the generic mechanism Parsing 'date' is very simple and a good first test.

extract: add some facility for extensible header parsing We need a way for extension to extend the header we can parse. We start with a very simple mechanism that will be used in the next changeset.

extract: remove the 'user' variable It is not heavily used enough to justify being something other than a dictionary entry.

extract: use a single return The differences between both returns are now very thin, we factor out that part.

extract: move 'nodeid' assignment where it is parsed The is one setter and no consumer, we can move it there directly.

extract: move 'date' assignment where it is parsed There is one setter and no consumer, we can move it there directly.

extract: assign user to 'data' earlier This is used in both case, we can assign it once.

extract: move 'branch' assignment where it is parsed There is one setter and no consumer, we can move it there directly.

extract: directly assign parent to data dictionary The temporary variables are not adding anything.

extract: assign message only once This return is shared between the two cases. We can move it earlier.

extract: simplify parents assignement We move to the more standard style of: value = default if special: value = other This is more consistent and compact.

patch: move 'extract' return to a dictionnary The final goal here is to be able to parse, return and process arbitrary data from patch. This mirror the recently added ability to add arbitrary data to patch headers. The first step is to return something more flexible, so we return a dict without changing any other logic.

patchbomb: add experimental config of a "pullurl" and export it This config allows to specify a public location where your changeset can be found. It then include a dedicated patch header show a command to be used to retrieve the change. See the test for example. This is flagged as experimental because this feature is not safe until we have more logic to test that: - changeset actually exists on destination - changeset is draft on destination. As all this is experimental, bike shedding can happily happens before we remove the experimental flag.

export: introduce a generic way to add patch header on export Extensions currently have no easy way to add data to exported patch. This is now fixed using a generic mechanism in the same fashion used by bundle2. Tests are coming in the next changeset with its first user.

incoming: request a bundle2 when possible (BC) Incoming was using bundle1 in all cases, as bundle1 is restricted to changegroup1 and does not support general delta, this can lead to significant CPU overhead if the server is using general delta storage. We now properly request and store a bundle2 to disk. If the server include any output or error in the bundle, they will be stored on disk and replayed when the bundle is read. As 'hg incoming' is going to read the bundle right away, we call that 'good' enough and go back to the bigger plan of having general delta on by default. This was tracked as 4864

bundlerepo: indent some code to prepare next patch We are about to add a new condition. Code is indented in a separated patch for readability.

bundle2: add a way to just forward the bundle2 stream to another user There is use case for directly forward and bundle2 stream from the peer to a file (eg: 'hg incoming --bundle'), However ssh peers have no way to know the 'getbundle' is over except for actually interpreting the bundle. So we need to have the unbundle do the interpreting and forward job. The function is marked as private to highlight that this is terrible and that we are sorry.

bundle2: split parameter retrieval and processing We want to introduce a simple way to forward the content of a bundle2 stream. For this purpose, we will need to both yield the parameters block and process it (to apply any behavior change it might indicate).

changegroup: extract the file management part in its own function The current writebundle function do two things: - taking a changegroup-packer instance and storing it into a valid bundle with proper header. - creating a temporary or requested file to store that bundle We would like to make it easier to forward bundle stream directly from a remote peer to a file, so we split the two logic to be able to skip the one about building a valid bundle (the remote is already sending one).

unbundle: properly read head modification result from bundle2 We were reading the wrong key...

revset: strip off "literal:" prefix from bookmark not found error This is what branch() and tag() do.

revset: do not fall through to revspec for literal: branch (issue4838) If "literal:" is specified, it must not be a revset expression. It should error out with a better message.

hgweb: ensure both foreground and background colors are specified (issue4872) When users configure the default foreground or background color to non-default (black on white) values, several hgweb styles lack contrast for headers and table row items. This patch fixes that by ensuring that where either foreground or background colors are specified, both are specified.

templater: do not pre-evaluate generator keyword at runsymbol (issue4868) It was introduced by e06e9fd2d99f, but the important code was removed by a3c2d9211294. So there was no positive effect other than exhausting memory. The problem spotted by e06e9fd2d99f is that you can't use a generator keyword more than once. For example, in hgweb template, "{child} {child}" doesn't work because the first "{child}" consumes the generator. But as a3c2d9211294 says, the fix was wrong because it could overwrite a callable keyword that returns a generator. Also the fix didn't work for a generator of generator such as "{diff}" keyword. So, the proper fix for that problem would be to not put a generator in a keyword table. Instead, it should be a factory of a generator. Note that this should fix the memory issue in hgweb, but my firefox killed by OOM in place. Be careful to not use a modern web browser to test the issue4868.

merge: drop special parent assignment in the obsolete case We can safely drop this because the very same assignment is enforcement later in the function. Dropping it will make it simpler to extract the default destination logic in its own function.

bundle: use bundle2 if repository uses general delta As bundle1 does not support generaldelta, this would mean recomputing delta at bundle time. This is similar to what we do for strip and shelve and was tracked as issue4865.

parsebundletype: add a comment for future generation

bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.

hgweb: fix border-radius for standards-based browsers While here, remove an old gecko-specific property, as gecko has supported the unprefixed version for many years.