exchange: set 'treemanifest' param on pushed changegroups too In 5c0fd878779c (treemanifests: set bundle2 part parameter indicating treemanifest, 2016-01-08), I didn't realize I had to set the parameter separately for getbundle and unbundle. Having the parameter there on push allows us to push to an empty repo and have the requirements updated correctly.

crecord: fix help bar display issue (issue5063) Before this patch, the help bar in crecord wouldn't be printed correctly when the terminal window didn't have enough column to display it. This patch adds logic to make sure that the help bar message is always displayed. We use an ellipsis when it is not possible to display the complete message.

crecord: fix typo in the help text In the crecord help dialog, the toggle all option was wrongfully documented. Instead of using 'a', one must use 'A' to toggle all the hunks. The crecord header that is always displayed on the screen contains the right shortcut and does not need to be changed.

subrepo: better error messages in _ensuregit This patch improves the error messages raised when an OSError occurs, since simply re-raising the exception can be both confusing and misleading. For example, if "hg identify" is run inside a repository that contains a Git subrepository and the git binary could not be found, it'll exit with the message "abort: No such file or directory". That implies "identify" has a problem reading the repository itself. There's no way for the user to know what the real problem is unless they dive into the Mercurial source, which is what I ended up doing after spending hours debugging errors while provisioning a VM with Ansible (turns out I forgot to install Git on it). Descriptive errors are especially important on Windows, since it's common for Windows users to forget to set the "Path" system variable after installing Git.

hgweb: update canvas.width before dynamically redrawing graph (issue2683) After 91ac8cb79125 graph canvas width is decided once on the initial rendering. However, after graph page gets scrolled down to load more, it might need more horizontal space to draw, so it needs to resize the canvas dynamically. The exact problem that this patch solves can be seen using: hg init testfork cd testfork echo 0 > foo hg ci -Am0 echo 1 > foo hg ci -m1 hg up 0 echo 2 > foo hg ci -m2 hg gl -T '{rev}\n' @ 2 | | o 1 |/ o 0 hg serve And then by navigating to http://127.0.0.1:8000/graph/tip?revcount=1 "revcount=1" makes sure the initial graph contains only revision 2. And because the initial canvas width takes only that one revision into count, after the (immediate) AJAX update revision 1 will be cut off from the graph. We can safely set canvas width to the new value we get from the AJAX request because every time graph is updated, it is completely redrawn using all the requested nodes (in the case above it will use /graph/2?revcount=61), so the value is guaranteed not to decrease. P.S.: Sorry for parsing HTML with regexes, but I didn't start it.

run-tests: "fix" race condition in race condition fix Laurent's commit 3203dfe341f9 still suffers from a race: by the time the "job" function tries to assign to channels[channel], that list has been truncated to empty. The result is that every job thread raises an IndexError. Earlier, I tried an approach of correctly locking channels, but that caused run-tests to hang on KeyboardInterrupt sometimes. This approach is strictly hackier, but seems to actually work reliably.

rebase: restore help for rebase w/o args (issue5059) Restoring documentation accidentally removed in a9a047878e14.

shelve: use cg3 for treemanifests Similar to previous change, this teaches shelve to pick the right changegroup version for repos that use treemanifests.

repair: use cg3 for treemanifests The newly created helper changegroup.safeversion() knows to pick version 03 if the repo uses treemanifests, so just using that means we pick the right changegroup version.

changegroup: introduce safeversion() In a few places (at least repair.py and shelve.py), we want to find the best changegroup version that we can assume users of the repo will understand. For example, we choose version 01 by default, but if it's a generaldelta repo, we expect clients to support version 02 anyway, so we choose that for new bundles (for e.g. "hg strip"). Let's create a helper for this functionality in changegroup, so we can reuse it elsewhere later.

changegroup: don't support versions 01 and 02 with treemanifests Since it would be terribly expensive to convert between flat manifests and treemanifests, we have decided to simply not support changegroup version 01 and 02 with treemanifests. Therefore, let's stop announcing that we support these versions on treemanifest repos. Note that this means that older clients that try to clone from a treemanifest repo will fail. What happens is that the server, after this patch, finds that there are no common versions and raises "ValueError: no common changegroup version". This results in "abort: HTTP Error 500: Internal Server Error" on the client. Before this patch, it was no better: The server would instead find that there were directory manifest nodes to put in the changegroup 01 or 02 and raise an AssertionError on changegroup.py#668 (assert not tmfnodes), which would also appear as a 500 to the client.

run-tests: fix crash when --json and --blacklist are both used (issue5050) This patch fixes a crash when both --json and --blacklist were given as arguments of run-tests.py. Now, instead of crashing, we add an entry for blacklisted tests in the json output to show that the tests were skipped.

run-tests: fix race condition Before this patch, it was possible for run-tests to crash on a race condition. The race condition happens in the following case: - the last test finishes and calls: done.put(None) - the context switches to the main thread that clears the channels list - the context switches to the last test mentioned above, it tries to access channels[channel] and crashes This happened to me while running run-tests. This patch fixes the issue by clearing the channel before considering that the test is done.

copyright: update to 2016

transaction: abort transaction during hook exception The new transaction context did not handle the case where an exception during close should still call release. This cause pretxnclose hooks that failed to cause the transaction to fail without aborting, thus requiring a hg recover. I've added a test.

Added signature for changeset 158bdc896572

Added tag 3.7-rc for changeset 158bdc896572

shelve: lowercase flag description The help for --unknown is the *only* command that started with a capital letter

changegroup: cg3 has two empty groups *after* manifests changegroup.getchunks() determines the end of the stream by looking for an empty chunk group (two consecutive empty chunks). It ignores empty groups in the first two groups. Changegroup 3 introduced an empty chunk between the manifests and the files, which confuses getchunks(). Since it comes after the first two, getchunks() will stop there. Fix by rewriting getchunks so it first counts two groups (empty or not) and then keeps antostarts counting empty groups. With this counting, changegroup 1 and 2 have exactly one empty group after the first two groups, while changegroup 3 has two (one for directories and one for files). It's a little hard to test this at this point, but I have verified that this patch fixes narrowhg (which was broken before this patch). Also, future patches will fix "hg strip" with treemanifests, and once that's done, getchunks() will be tested through tests of "hg strip".

mq: check for reserved patch name with qimport -r (issue5033) Fix regression from 143b52fce68e. Catching aborts might not be pretty but it works and is a small change.

mq: refactor makepatchname into class method

repoview: fix corrupted hiddencache crash Mercurial (issue5042) Before this patch if the hiddencache existed but was empty, it would crash mercurial. This patch adds exception handling when reading the hiddencache to avoid the issue. When encountering a corrupted cache file we print a devel warning. There would be no point in issuing a normal warning as the user wouldn't be able to do anything about the situation. The warning looks like: devel-warn: corrupted hidden cache, removing it at: /path/to/repoview.py

repoview: add missing newline character in debug prints

commandserver: drop tell() and seek() from channels (issue5049) These operations are obviously invalid for file-like channels because they will read or write protocol headers. This patch works around the issue that "hg archive" generates a corrupted zip file on Windows commandserver because of unusable tell() implementation. But the problem still occurs without using a commandserver. $ hg archive -R not-small-repo -t zip - | cat > invalid.zip So, this patch cannot fix the issue5049 completely.

crecord: edit during hg crecord should preserve cursor position (issue5041) This patch adds a variable to keep track of what hunk was selected before the edit. We use that variable to select the hunk or its replacement after the edit.

templates: use canvaswidth instead of fixed width for canvas (issue2683) Before this patch, template files for "graph" web page use fixed width size "480" for canvas element. This causes pruned lanes and invisible vertexes, if there are 16 or more vertical lanes at once. In such case, part of graph in right side area over 480 is invisible, even though corresponded summary text blocks are visible correctly. This limitation isn't reasonable for workflow using many branches at once (e.g. "one branch per issue" workflow). There were changes below related to width of canvas: - 7359cb753a54 (templates: widen the graph canvas (issue2683)), released as a part of Mercurial 1.8.2 According to the description, this assumed that 15 parallel branches was enough for ordinary workflow, and bumped width of canvas up from 224 to 480. - d490edc71146 (hgweb: make graph data suitable for template usage), released as a part of Mercurial 2.3 This introduced "canvaswidth" template keyword as a part of refactoring around graph rendering. But 'width="480"' of canvas element in template files wasn't replaced by 'width="{canvaswidth}"' in it (or subsequent one). This patch uses dynamic value "{canvaswidth}" instead of fixed width size "480" for canvas element. This is posted for "stable", because: - this is re-fixing issue2683 - this is simple enough for stable - using "{canvaswidth}" doesn't require any additional cost Calculation of canvaswidth is already implied as a part of "graph" web command.

backout: fix --no-commit option (issue5054)

bundle: exit early when there are no commits to bundle Previously, if you passed a revset that resolved to no nodes, it would get interpreted by the changegroup discovery logic as 'bundle all my heads', which is not what the user asked. Let's exit early when we notice this case. It could be argued that the changeset discovery logic should be smarter and only assume 'all heads' if the incoming heads parameter is None, but that's a much riskier change.

zeroconf: access repo on hgweb_mod properly (issue5036) hgweb_mod.hgweb.repo disappeared in ae33fff17c1e (hg: establish a cache for localrepository instances) and the code for accessing repo instances from hgweb was later refactored to go through a cache-aware context manager. Adapt zeroconf to access the repo instance via the new API.

merge default into stable for 3.7 code freeze

shelve: permit shelves to contain unknown files If an emergency comes in while you're in the middle of an experimental change, it can be useful to shelve not just files hg already tracks but also your unknown files while you handle the emergency. This is especially true if you have hooks intended to prevent you from forgetting to add new code before you push. Teach "hg shelve" to optionally shelve unknown files, not just tracked files. This is functionally similar to --addremove, but with two differences: 1) Deleted files are not removed. 2) Files added during shelve creation are tracked in extra so that they can be forgotten by "hg unshelve". When unshelving, we take care to only forget files if they've been created during the unshelve operation; if you add a file that's being tracked in a shelve as an unknown file, it should not become unknown again when the shelve is unshelved.

localrepo: don't reference transaction from hook closure (issue5043) Before, the hook() closure (which is called as part of locking hooks) would maintain a reference to a transaction instance (which should be finalized by the time lock hooks are called). Because we accumulate hook() instances when there are multiple transactions per lock, this would result in holding references to the transaction instances which would lead to higher memory utilization. Creating a reference to the hook arguments dict minimizes the number of objects that are kept alive until the lock release hook runs, minimizing memory "leaks."

context: don't use util.cachefunc due to cycle creation (issue5043) util.cachefunc stores all arguments as the cache key. For filectxfn functions, the arguments include the memctx instance. This creates a cycle where memctx._filectxfn references self. This causes a memory leak. We break the cycle by implementing our own memoizing function that only uses the path as the cache key. Since each memctx has its own cache instance, there is no concern about invalid cache hits.

largefiles: actions will now always have a file - drop check

largefiles: make prompt order deterministic 42ae1b1f048f introduced iteration of a set. Make it stable.

largefiles: fix commit of missing largefiles 832c98d79587 improved merging of standin files referencing missing largefiles. It did however not test or fix commits of such merges; it would abort. To fix that, change copytostore to skip and warn about missing largefiles with a message similar the one for failing get from remote filestores. (It would perhaps in both cases be better to emit a more helpful warning like "warning: standin file for large1 references 58e24f733a which can't be found in the local store".) To test this, make sure commit doesn't find the "missing" largefile in the global usercache. For further testing, verify that update and status works as expected after this. This will also effectively backout 63116d47cc3f.

diff: don't crash when merged-in addition is copied Similar to what was explained in the previous commit, the diff code expected copy source to be in "ctx1", which is not always the case during a merge. This has been broken since before hg 2.0. Also similar to the previous commit, we fix the problem by fixing up the copy dict.

diff: don't crash when merged-in addition was removed (issue4786) During a merge, if the user removes a file that came from parent 2 and did not exist in parent 1, the file's status will be "removed". This surprises the diff code, which crashes because it expects removed files exist in parent 1. This has been broken since 377124ba6b10 (trydiff: use 'not in addedset' for symmetry with 'not in removedset', 2014-12-23). Fix by fixing up the list of removed file, similar to how we currently fix up the list of modified and added files during a merge.

diff: move status fixup earlier, out of _filepairs() This prepares for future patches, and it also lets us remove the ugly "ctx1" argument to _filepairs() (ugly because of its assymmetry -- there's no "ctx2" argument).

graft: warn when -r is combined with revisions as positional arguments The behaviour in this case is undefined. Instead of silently doing something "random" and surprising, at least issue a warning. (This should perhaps be considered a "deprecation" and turned into an error in a future release.)

graft: clarify in help that `-r` is not just optional Positional parameters are also treated as revisions, but the order of revisions matters and it will often be wrong if the user understands it as `-r` taking multiple revisions as `-r REV1 REV2`. (Alternatively, `-r` could be turned into a no-op flag as the documentation suggests. That would however be less "semantic markup" and I agree with the implementation in 55e7f352b1d3 but not the documentation.)

streamclone: use backgroundfilecloser (issue4889) Closing files that have been appended to is slow on Windows/NTFS. CloseHandle() calls on this platform often take 1-10ms - and that's on my i7-6700K Skylake processor with a modern and fast SSD. Contrast with other I/O operations, such as writing data, which take <100us. This means that creating/appending thousands of files can add significant overhead. For example, cloning mozilla-central creates ~232,000 revlog files. Assuming 1ms per CloseHandle(), that yields 232s (3:52) of wall time waiting for file closes! The impact of this overhead can be measured most directly when applying stream clone bundles. Applying these files is effectively uncompressing a tar archive (read: it's very fast). Using a RAM disk (read: no I/O wait), the difference in wall time for a `hg debugapplystreamclonebundle` for a ~1731 MB mozilla-central bundle between Windows and Linux from the same machine is drastic: Linux: ~12.8s (128MB/s) Windows: ~352.0s (4.7MB/s) Windows is ~27.5x slower. Yikes! After this patch: Linux: ~12.8s (128MB/s) Windows: ~102.1s (16.1MB/s) Windows is now ~3.4x faster. Unfortunately, it is still ~8x slower than Linux. Profiling reveals a few hot code paths that could likely be improved. But those are for other patches. This patch introduces test-clone-uncompressed.t because existing tests of `clone --uncompressed` are scattered about and adding a variation for background thread closing to e.g. test-http.t doesn't feel correct.

streamclone: indent code This will make the subsequent patch easier to read.

scmutil: support background file closing Closing files that have been appended to is relatively slow on Windows/NTFS. This makes several Mercurial operations slower on Windows. The workaround to this issue is conceptually simple: use multiple threads for I/O. Unfortunately, Python doesn't scale well to multiple threads because of the GIL. And, refactoring our code to use threads everywhere would be a huge undertaking. So, we decide to tackle this problem by starting small: establishing a thread pool for closing files. This patch establishes a mechanism for closing file handles on separate threads. The coordinator object is basically a queue of file handles to operate on and a thread pool consuming from the queue. When files are opened through the VFS layer, the caller can specify that delay closing is allowed. A proxy class for file handles has been added. We must use a proxy because it isn't possible to modify __class__ on built-in types. This adds some overhead. But as future patches will show, this overhead is cancelled out by the benefit of closing file handles on background threads.

templatekw: add {namespaces} keyword This provides a general-purpose interface to all custom namespaces. The {namespaces} keyword honors the definition order of namespaces as they are kept by sortdict.

templatekw: move shownames() helper to be sorted alphabetically I'll add shownamespaces(), which is similar to this function. I want to put them nearby.

templater: make get(dict, key) return a single value This is necessary to obtain a _hybrid object from a dict. If get() yields a value, it would be stringified. I see no benefit to make get() lazy, so this patch just changes "yield" to "return".

templater: make _hybrid not callable to avoid conflicting semantics In templater, a callable symbol exists for lazy evaluation, which should have f(**mapping) signature. On the other hand, _hybrid.__call__(), which was introduced by 0b241d7a8c62, generates mapping for each element. This patch renames _hybrid.__call__() to _hybrid.itermaps() so that a _hybrid object can be a value of a mapping dict. {namespaces % "{namespace}: {names % "{name }"}\n"} ~~~~~ a _hybrid object

backout: commit changeset by default (BC) Add --no-commit flag to prevent it. This should make the hg user experience a little better. Some discussion can be found here: http://markmail.org/message/7jm7ro2ias6hxywy

help: move Windows 9x information to appropriate place

shelve: move commitfunc closer to use site Supporting shelving of unknown files needs this code motion. No functional changes.

commands: document clone bundles hooks and rollback behavior The added content is inside a verbose container. I figure it makes sense to explicitly document behavior, including with the caveat it may change later. People can't say they weren't warned!

clonebundles: improve BUNDLESPEC documentation Before the existence of `hg debugbundle --spec`, the process for defining the BUNDLESPEC value in manifests was not very clear and not trivial to automate, especially in the case of stream clone bundles. This patch adds documentation to note the existence of `hg debugbundle --spec`. We drop the reference to stream clone requirements handling because it is now redundant with `hg debugbundle --spec`. While we are here, we further reinforce the importance of defining BUNDLESPEC.

commands: teach debugbundle to print bundle specification This seems like the most logical place to put this functionality. Test coverage over existing known bundle specs has been added.

commands: use context manager for opened bundle file

exchange: implement function for inferring bundle specification We don't currently have a mechanism for inferring bundle spec strings from bundle files. This patch adds one. This will eventually be used to make the producing of clone bundles manifests easier.

streamclone: extract code for reading header fields So it can be called from another consumer in a future patch.

encoding: escape U+007F (DEL) character in JSON RFC 7159 does not state that U+007F must be escaped, but it is widely considered a control character. As '\x7f' is invisible on a terminal, and Python's json.dumps() escapes '\x7f', let's do the same.

run-tests: skip threading for a single test (issue5040) This version backs out 50e621fe0362 and implements it in a more consistent manner.

cleanup: use modern @property/@foo.setter property specification We can use this now that we're 2.6+, and this is more idiomatic modern Python.

buildrpm: use bash shebang, since we use bash features in the script As suggested by Bryan O'Sullivan.

mac: ignore resource fork when checking file sizes Some evil evil awful tool adds resource forks to files it's comparing. Our Mac-specific code to do bulk stats was accidentally using "total size" which includes those forks in the file size, causing them to be reported as modified. This changes it to only care about the normal data size and thus agree with what Mercurial's expecting.

copies: fix detection of divergent directory renames If we move all the files out of one directory, but into two different directories, we should not consider it a directory rename. The detection of this case was broken.

with: use context manager for transaction in strip

with: use context manager for transaction in pushphase

with: use context manager for transaction in strip

with: use a context manager for transaction in strip

with: use context manager in rebuildfncache

with: use context manager for transaction in consumev1

with: use context manager in _histedit

with: use context manager in amend

with: use context manager for transaction in changegroup apply (This needs some line wrapping due to the additional indent level. -mpm)

with: use context manager for transaction in rebase

with: use context manager for transaction in qimport

with: use context manager for transaction in qfinish

with: use context manager for transaction in mercurial_sink

transaction: turn a transaction into a Python context manager This lets us greatly simply acquire/release cycles. If the block completes without raising an exception, the transaction is closed. Code pattern before: try: tr = repo.transaction('x') # zillions of lines of code tr.close() finally: tr.release() And after: with tr.transaction('x'): # ...

with: use context manager for lock in pushphase

with: use context manager in rebuildfncache again

with: use context manager in streamclone consumev1

with: use context manager in manifest

with: use context manager in rename

with: use context manager in resolve

with: use context manager in unbundle

with: use context manager in update

with: use context manager in bisect save_state

with: use context manager in merge update

with: use context manager in _markchanges

with: use context manager in maybeperformlegacystreamclone

with: use context manager in verify

with: use context manager in qrename

with: use context manager in qfinish

with: use context manager in localrepo recover

with: use context manager in streamclone generatev1

with: use context manager in subrepo storeclean

with: use context manager in subrepo _cachestorehash

with: use context manager in largefiles commit

with: use context manager in unshelveabort

with: use context manager for wlock in transplant

with: use context manager for wlock in shelve stripcmd

with: use context manager for lock in continue

with: use context manager for wlock in unshelve

with: use context manager for wlock in shelve deletecmd

with: use context manager for wlock in shelve cleanupcmd

with: use context manager for wlock in shelve createcmd

with: use context manager for lock in histedit cleanupnode

with: use context manager for lock in qimport

with: use context manager for wlock in qfold

with: use context manager for wlock in qrefresh

with: use context manager for wlock in qpop

with: use context manager for wlock in qpush

with: use context manager for wlock in qnew

with: use context manager for wlock in mergeupdate

with: use context manager for wlock in overridepurge

with: use context manager for wlock in cmdutilforget

with: use context manager for wlock in overriderevert

with: use context manager for wlock in removelargefiles

with: use context manager for wlock in addlargefiles

with: use context manager for wlock in updatelfiles

with: use context manager for wlock in kw_dorecord

with: use context manager for wlock in kw_copy

with: use context manager for wlock in kw_amend

with: use context manager for wlock in keyword demo

with: use context manager for wlock in _kwfwrite

with: use context manager for wlock in sign

with: use context manager for wlock in checklookup

with: use context manager for wlock in copy

with: use context manager for wlock in workingctx.undelete

with: use context manager for wlock in workingctx.forget

with: use context manager for wlock in workingctx.add

with: use context manager for wlock in graft

with: use context manager for wlock in debugsetparents

with: use context manager for wlock in debugrebuilddirstate

with: use context manager for wlock in copy

with: use context manager for wlock in branch

with: use context manager for wlock in revert

with: use context manager for wlock in remove

with: use context manager for wlock in recordinwlock

with: use context manager for wlock in _writeactive

with: use context manager for wlock in _writerepo

sshpeer: make remotelock a context manager

lock: turn a lock into a Python context manager This lets us greatly simply acquire/release cycles. Code pattern before: try: lock = repo.lock() # zillions of lines of code finally: lock.release() And after: with repo.lock(): # ...

with: use context manager for I/O in changedfiles in patch

with: use context manager for file I/O in memusage

i18n: don't translate a transaction name

chgserver: mark as a built-in extension

chgserver: import background server extension from cHg This extension is copied from https://bitbucket.org/yuja/chg/ -r 86feb5f2e971 It could be imported as mercurial/chgserver.py, but in that case, we would have to resolve circular import between chgserver and commandserver. So I decided to keep it as an extension. chgserver.chgcmdserver -> commandserver.server commandserver._servicemap -> chgserver.chgunixservice

check-code: export needs a space to avoid false positives

rebase: small refactoring to allow better extensibility from extensions Inhibit, one of evolve's extension, would like to change the way rebase works with obsolete changesets. During a rebase with inhibit, the inhibition of the obsolescence markers should be lifted for the rebase. With this small refactoring, inhibit and can wrap the _filterobsoleterevs function to lift inhibition cleanly and at the same time this change makes rebases' code more legible.

tests: non-GNU mv can't move a file onto a hardlink of itself On Solaris, when you try to mv a file to another path that is a hardlink to the original, it complains that they're identical. GNU mv doesn't complain, but it's simpler to just remove the original file instead.

buildrpm: move creation of RPM directories from dockerrpm Handling the creation of the RPM directories in buildrpm is more consistent and takes care of non-docker builds as well.

i18n: fix a test-gendoc.t failure

test-ctxmanager: fix Python 2.6 compatibility problem

util: rename ctxmanager's __call__ method to enter

obsolete: make _computeobsoleteset much faster This patch makes _computeobsoleteset much faster by looping over the draft and secrets as opposed to looping over the successors. This works because "number of draft and secret" is typically way smaller(<100) than the number of successor in the repo (~90k in my checkout of core mercurial as of today). And also because it is very fast to compute "not public()". I timed the code with the following setup: """ from mercurial import hg, ui, obsolete ui = ui.ui() repo = hg.repository(ui, "~/hg") l = repo.obsstore.successors # This caches the result """ With about 90k successors. k=obsolete._computeobsoleteset(repo) before this patch: 10 loops, best of 3: 33.9 ms per loop k=obsolete._computeobsoleteset(repo) after this patch: 10000 loops, best of 3: 83.3 µs per loop

check-commit: sort errors by line number

check-commit: try to fix multiline handling The old code did not understand the difference between the first line of the summary, and a random line in the summary that happened to include a #, or a random line in the changes that happened to include it. 965788d9ae09 is an example where it fails

check-commit: support REVs as commandline arguments usage: * HG_NODE=REV check-commit * hg export REV | check-commit * check-commit REV ...

check-commit: modularize

check-commit: fix summary length regexp

util: simplify file I/O functions using context managers

run-tests: use a context manager for file I/O in TextTestRunner

bundlerepo: use context manager for file I/O in _writetempbundle

shelve: use a context manager for file I/O in listcmd

largefiles: use util.readfile in lfconvert

run-tests: use a context manager for file I/O

largefiles: use util.readfile in overrides

commands: use a context manager for file I/O in debugdiscovery

check-seclevel: use a context manager for file I/O

largefiles: use a context manager in _getfile

util: replace file I/O with readfile

patchbomb: replace file I/O with util.readfile

util: adjust hgcmd() to handle frozen Mercurial on OS X Previously, 'hg serve -d' was trying to exec the bundled python executable, which failed with: Unknown option: -- usage: python [option] ... Try 'python -h'... abort: child process failed to start See the previous patch for details about the content of the various command variables. Note that unlike the previous patch here an application bundling Mercurial could set $HG in the environment to get the correct result, there isn't anything that a bundling application could do to get the correct result here. 'hg serve -d' now launches under TortoiseHg, and there is a process listed in the background, but a client process cannot connect to it for some reason, so more investigation is needed.

util: adjust hgexecutable() to handle frozen Mercurial on OS X sys.executable is "$appbundle/Contents/MacOS/python" when Mercurial is bundled in a frozen app bundle on OS X, so that isn't appropriate. It appears that this was only visible for things launched via util.system(), like external hooks, where $HG was set wrong. It appears that Mercurial also uses 'sys.modules['__main__'].__file__' (here) and 'sys.argv[0]' (in platform.gethgcmd()) to figure out the command to spawn. In both cases, this points to "$appbundle/Contents/Resources/hg", which invokes the system python since "/usr/bin/env python" is on the shebang line. On my system with a screwed up python install, I get an error importing the os module if this script is invoked. We could take the dirname of sys.executable and join 'hg' instead of this if we want to be paranoid, but py2app boostrap is setting the environment variable since 0.1.6 (current version is 0.9), so it seems safe and we might as well use it.

util: adjust 'datapath' to be correct in a frozen OS X package Apparently unlike py2exe, py2app copies the Mercurial source tree as-is to a Contents/Resources subdirectory of an app bundle, and places its binary stub in Contents/MacOS. (The Windows install has the 'hgext' and 'mercurial' modules in 'lib/library.zip', while the help and templates subdirectories have been moved out of the mercurial directory to the root of the installation. I assume that the python code living in a zip file is why "py2exe doesn't support __file__".) Therefore, prior to this change, Mercurial in a frozen app bundle on OS X would go looking for help *.txt, templates and locale info in Contents/MacOS, where they don't exist. There are only a handful of places that test for frozen, and not all of them are wrong for OS X, so it seems wiser to handle them on a case by case basis, rather that try to change mainfrozen(). The remaining cases are: 1) util.hgexecutable() wrongly points to the bundled python executable, and affects $HG in util.system() launched processes (e.g. external hooks) 2) util.hgcmd() wrongly points to the bundled python executable, but it seems to only affect 'hg serve -d' 3) hook._pythonhook() may be OK, since I didn't see anything outrageous when printing sys.path from an internal hook. I'm not sure if this special case is needed on OS X though. 4) sslutil._plainapplepython() is OK, because sys.executable is not /usr/bin/python, nor is it in /System/Library/Frameworks

help: add --system flag to get help for various platform(s)

help: split config user settings file by platform

help: label windows config help

help: split out unix/plan9 config help

help: move config default marking to its own line

help: split config ui.portablefilenames

test-hgignore: add globs for Windows

tests: run check-commit on secret changes too

util: don't capture exception with a name since we don't use it Spotted by pyflakes.

changegroup3: move treemanifest support into _unpackmanifests() By putting the treemanifest code in _unpackmanifests(), _addchangegroupfiles() will only be about files again, and we get a nice symmetry between _packmanifests() and _unpackmanifest(). The immediate benefit to me is that remotefilelog should not need to be updated to work with treemanifests. It should also make server.validate and progress output easier to get right. Probably bundlerepo too.

changegroup3: add empty chunk separating directories and files Remotefilelog overrides changegroup._addchangegroupfiles(), assuming it is about files, which seems like a natural assumption. However, in changegroup3, directory manifests are sent in the files section of the changegroup. These naturally make remotefilelog unhappy. The fact that the directories are not separated from the files (although they do come before the files) also makes server.validate harder to implement. Since we read one chunk at a time from the steam, once we have found a file (non-directory) entry in the stream, we would have to push the read data back into the stream, or otherwise refactor the code. It will be easier if we add an empty chunk after all directory manifests. This change adds that empty chunk, although we don't yet take advantage of it on the reading side. We will soon move the tree manifest stuff out of _addchangegroupfiles() and into _unpackmanifests().

changegroup3: introduce experimental.changegroup3 boolean config In order to give us the freedom to change the changegroup3 format, let's hide it behind an experimental config. Since it is required by treemanifests, that will override the cg3 config.

changegroup: hide packermap behind methods This is to prepare for hiding changegroup3 behind a config option.

merge with stable

context: check for differing flags a little earlier This makes it clearer that a unchanged file whose flags have changed will be reported as a modification. Also test this.

context: clarify why we don't compare file contents when nodeid differs See previous commit for timing information.

status: back out changeset 89f49813526c This backs out 89f49813526c (status: change + back out == clean (API), 2016-01-04). Although correct, it turned out that it was just too slow. For example, 'hg status --rev .~1000 --rev .' on the Mozilla repo went from <1s to >30s on cold disk. So we go back to reporting reverted changes as modified instead of clean. These are rare anyway, as suggested by the fact that it had been broken since before Mercurial 2.0.

rebase: prevent creating divergence Before this patch rebase would create divergence when you were rebasing obsolete changesets on a destination not containing one of its successors. This patch introduces rebase.allowdivergence to explicitly allow divergence creation with rebase.

rebase: create a new variable to make the next patch more legible

rebase: minor refactoring of _computeobsoletenotrebased This patch is a refactoring of the code skipping obsolete changesets already present in destination. It makes the following patches more legible. Instead of passing all the revs to be rebased to _computeobsoletenotrebased, we only pass the obsolete revisions of the rebaseset.

test-status: stabilize for no-execbit platforms The preceding #if conditional was the only modification to the file, so the "reverting file" line in the subsequent revert command was getting dropped.

merge: split up checks for unknown and ignored files that differ In some real-world cases it is preferable to allow overwriting ignored files while continuing to abort on unknown files. This primarily happens when we're replacing build artifacts (which are ignored) with checked in files, but continuing to abort on differing files that aren't ignored. We're redefining merge.checkunknown to only control the behavior for files that aren't ignored. That's fine because this config was only very recently introduced and has not made its way into any Mercurial releases yet.

merge: determine what untracked conflicts cause warns and aborts separately This is written in a somewhat weird style, but it's designed for code reuse in an upcoming patch.

merge: factor out code to get checkunknown config We're going to reuse this code shortly.

hooks: add HG_NODE_LAST to txnclose and changegroup hook environments Sometimes a txnclose or changegroup hook wants to iterate through all the changesets in transaction: in that situation usually the revset `$HG_NODE:` is used to select the revisions. Unfortunately this revset sometimes may contain too many changesets because we don't have the write lock while the hook runs newer changes may be added to repository in the meantime. That's why there is a need for extra variable carrying the information about the last change in the transaction.

exchange: make clone bundles non-experimental and enabled by default The clone bundles feature was introduced in Mercurial 3.6 behind an experimental and disabled by default flag. The feature has been enabled on hg.mozilla.org for a few months and has served many terabytes of clones. Users have been encouraged to use the feature and reception has been very positive (mainly due to faster clones as a result of connecting to a CDN). I have heard no feedback about changing the feature other than inquiries about when it will be enabled by default. So, I think the feature is ready to be enabled by default. This patch renames experimental.clonebundles to ui.clonebundles, documents the option, and enables it by default. References to the experimental state of clone bundles have been removed. The remaining config option docs in clonebundles.py have been removed because they are redudant with `hg help config`. There are some oddities with behavior of clone bundles. Because clones with clone bundles are effectively 2 `hg pull` operations, there may be 2 transactions. This could result in hooks running twice. If the subsequent pull is aborted, it could result in partial rollback and an incomplete clone. This behavior is a bit wonky and should probably be documented. If this patch is accepted, I'll send a follow-up to document it. I don't think this behavior should prevent the feature being enabled by default. Reworking the clone mechanism to support interrupted or multi-part clones feels like a major new feature and something that when implemented can change the hook and rollback semantics of clone bundles. Besides, partial clone is better than full rollback and hooks running on initial clone are likely rare, so I think the impact is minimal.

exchange: make clonebundleprefers non-experimental In preparation for making the feature enabled by default.

clonebundles: remove advertisement of feature I screwed up. When clone bundles is enabled on the server and a compatible client without the feature enabled clones, the server sends down an advertisement saying to enable the feature. The server creates the message which is printed verbatim on the client as an "output" part. There are 2 problems: 1) The message doesn't respect the client's localization 2) The message contains a reference to the "experimental.clonebundles" option. Since clone bundles is about to be marked as non-experimental and the goal of the advertisement was to encourage clients to test the experimental feature, let's just remove the broken advertisement since it no longer serves a purpose.

changegroup: remove now-unused 'wasempty' variable and parameter

treemanifests: set bundle2 part parameter indicating treemanifest By adding a mandatory 'treemanifest' parameter in the bundle2 part, we make it possible for the recipient to set repo requirements before the manifest revlog is accessed.

docchecker: scan for missing space before :hg:

tests: run docchecker in gendoc.t

tests: move gendoc.t runrst comment next to runrst

docchecker: introduce a way to check for poor markup Specifically, :hg:`foo 'bar baz'` when rendered by `hg help` results in: 'hg foo 'bar baz'' ... which is hard to read. We encourage :hg:`foo "bar baz"` instead.

minirst: change hgrole to use single quotes We decided to reserve double quotes for arguments to hg because cmd does not like single quotes, so switch the outer quotes to single

paths: port to generic templater Embedded passwords are masked only in plain output because we'll want raw values in machine-readable format such as JSON. For custom template, we can add a filter to mask passwords (e.g. "{url|hidepassword}"). path.rawloc field is called as "url" than "path" because we have "pushurl" sub-option. Also, "name" and "url" are not allowed as sub-options as they conflict with the field names.

paths: merge conditions that select visibility of fields Truth table (extracted from the original implementation): search quiet name path subopt ------ ----- ---- ---- ------ f f T T T f T T f f T f f T f T T f f f

paths: use single loop for both search=None|pattern cases This will help porting to the formatter API. This patch adds test for empty pathitems to make sure "hg paths" never say "not found!".

paths: reorder else clause for readability of subsequent patches This prepares for porting to the formatter API. Future patches will use a single loop to handle both search=None|pattern cases because formatter output should be the same. "pathitems" will be switched instead.

paths: drop ui.status label from output of "hg paths name" We just need to not print path if --quiet. ui.status label is unwanted.

unionrepo: fix wrong rev being checked in iscensored (issue5024)

summary: print unstable, bumped and divergent as unconditionally plural This aligns with the unconditional plural output for the update line contents, as well as the incoming/outgoing bookmarks line. It also matches the message in evolve's summary hook as of 4f83b2d2d20d. (Though I thought this was removed recently?)

debugshell: disable demand importer when importing debugger For reasons I can't explain (but likely have something to do with a combination of __import__ inferring default values for arguments and the demand importer mechanism further assuming defaults), the demand importer isn't playing well with IPython. Without this patch, we get a failure "ValueError: Attempted relative import in non-package" when attempting to import "IPython." The stack has numerous demandimport calls on it and adding "IPython" to the exclude list in demandimport isn't enough to make the problem go away, which means the issue is likely somewhere in the bowells of IPython. It's easier to just disable the demand importer when importing the debugger.

status: change + back out == clean (API) After backing out a change, so the file contents is equal to a previous revision of itself, we currently report the status between the two equal revisions as modified. This is because context._buildstatus() reports any file whose new nodeid is not equal to _newnode as modified. That magic nodeid is given only to files added or modified in the working directory, so any file whose nodeid has changed between two revisions will be reported as modified. Fix by simply comparing the file contents for all cases where the nodeid changed, whether they are in the working copy or committed. Marking with (API) as it subtly changes the semantics of the method.

convert: use manifest.diff() instead of ctx.status() mercurial_source.getchanges() seems to care about files whose nodeid has changed even if their contents has not (i.e. it has been reverted/backed out). The method uses ctx1.status(ctx2) to find differencing files. However, that method is currently broken and reports reverted changes as modified. In order to fix that method, we first need to rewrite getchanges() using manifest.diff(), which does report reverted files as modified (because it's about differences in the manifest, so about nodeids).

convert: replace cache of (m,a,r) by (ma,r) The next commit will rewrite the way we find changes between two manifests. By making the cache not care about the difference between added and modified files, we don't require the rewritten code to care about that difference either. Also extract the call to ctx.status() to simplify the next commit.

convert: use _ prefix for private methods in hg sink This makes it clearer which methods are part of the interface defined by the superclass.

children: update help with replacement

graphlog: update help with replacement

histedit: clarify modes also fix terse command syntax

histedit: explain basics of histedit commands This should be hooked into @action via an extended help argument, but that is future work.

histedit: report the unacceptable changeset

mdiff: don't emit a diff header for empty trivial deltas An empty trivial delta, coded as (0, 0, 0) makes the delta application do nothing, but still takes 12 bytes, while skipping it altogether works as much, without taking any space at all.

commands: get rid of empty try/finally block from _dograft This diff is purely an indentation change to clean up a block that was kept in place to make 77995317b374 easier to read.

commands: get rid of empty try/finally block from _dobackout This diff is purely an indentation change to clean up a block that was kept in place to make c7217f1458bf easier to read.

commands: get rid of empty try/finally block from import_ This diff is purely an indentation change to clean up a block that was kept in place to make 713b09fc9fbb easier to read.

streamclone: use context manager for writing files These are the file writes that have the most to gain from background I/O. Plug in a context manager so I can design the background I/O mechanism with context managers in mind.

scmutil: use context managers for file handles Now that we dropped support for Python 2.4, we are able to use context managers. Let's replace the try..finally pattern in scmutil.py with context managers, which close files automatically when the context manager is exited. There should be no change in behavior with this patch. Why convert to context managers if nothing is broken? I'm working on closing file handles in background threads to improve performance on Windows. As part of this, I realized there could be some future issues if the background file closing code isn't designed with context managers in mind. So, I'd like to switch some code to context managers so I can design an API that works with context managers.

statichttprepo: implement __enter__ and __exit__ on httprangeheader httprangeheader behaves like a file object. Implement __enter__ and __exit__ so it can be used as a context manager, just like file objects.

osutil: implement __enter__ and __exit__ on posixfile So they can be used as context managers.

util: introduce ctxmanager, to avoid nested try/finally blocks This is similar in spirit to contextlib.nested in Python <= 2.6, but uses an extra level of indirection to avoid its inability to clean up if an __enter__ method raises an exception. Why add this mechanism? It greatly simplifies scoped resource management, and lets us eliminate several hundred lines of try/finally blocks. In many of these cases the "finally" is separated from the "try" by hundreds of lines of code, which makes the connection between resource acquisition and disposal difficult to follow. (The preferred mechanism would be the "multi-with" syntax of 2.7+, but Mercurial can't move to 2.7 for a while.) Intended use: >>> with ctxmanager(lambda: file('foo'), lambda: file('bar')) as c: >>> f1, f2 = c() This will open both foo and bar when c() is invoked, and will close both upon exit from the block. If the attempt to open bar raises an exception, the block will not be entered - but foo will still be closed.

hghave: support HGMODULEPOLICY for pure HGMODULEPOLICY was introduced in 4374d819ccd5

merge with stable

tests: extend check-commit self-tests This now contains both positive and negative tests for almost all the check-commit rules.

encoding: handle UTF-16 internal limit with fromutf8b (issue5031) Default builds of Python have a Unicode type that isn't actually full Unicode but UTF-16, which encodes non-BMP codepoints to a pair of BMP codepoints with surrogate escaping. Since our UTF-8b hack escaping uses a plane that overlaps with the UTF-16 escaping system, this gets extra complicated. In addition, unichr() for codepoints greater than U+FFFF may not work either. This changes the code to reuse getutf8char to walk the byte string, so we only rely on Python for unpacking our U+DCxx characters.