bundle: fix grammar in help text

bundle: clarify help text The file might not be compressed; the interactions between -a, --base, and a named or default repository weren't clear.

cmdutil: do not duplicate stdout by makefileobj() It made output order unpredictable because two separate buffers are flushed individually. Let's use a thin wrapper that just sends close() to black hole.

cmdutil: reimplement file wrapper that disables close() There's no need to dynamically create wrappedfileobj class and define close() as lambda. Also, __iter__() was missing.

export: remove useless comparison between file object and string It was introduced at 31aa2e5b0750, where "template" argument could be a file object. After that, a306837f8c87 added "len(template)", so "template" must be a string now. Therefore, "fp != template" should always be True. It seems 31aa2e5b0750 was intended to work around a bug in TortoiseHg, and I'm sure I've fixed it completely in TortoiseHg source. https://selenic.com/pipermail/mercurial-devel/2011-February/028467.html

export: do not print '<fdopen>' as an output filename Because makefileobj() duplicates or wraps stdout, "fp != sys.stdout" didn't work correctly. Python doc states that special file objects are named in the form '<...>', and absolute filenames should never start with '<', we can ignore names start with '<'. We can't test fp.fileno() because fp may be a command-server channel. https://docs.python.org/2.7/library/stdtypes.html#file.name In the test output, "exporting patch:" line is printed after patch content. This is caused by fdopen() and will be fixed by the subsequent patch.

commandserver: implement name() to clarify channel is not a plain file Because unknown attributes are delegated to the underlying file object, commandserver channels said they were '<stdout>' or '<stdin>' even though they weren't. This patch makes them say '<X-channel>'.

histedit: delete to drop The default behaviour to forbid this makes a lot of sense for novice users because it's safeguarding them from dangerous behavior but making it configurable will be apprieciated by power users in at least one big organization. It allows an user to look an histedit rules from declarative perspective and make the rules reflect the state after histedit. If we can move lines t move commits why can't we drop lines to drop commits? Let's put this behind config knob and inform users about this feature the very moment they are trying to use it so they can choose desired behaviour.

clonebundles: fix bundlespec typo

record: fix hunk handling to remember the current function

record: turn on showfunc Always try to give diff context when doing an interactive record

parsers: use PyTuple_Pack instead of manual list-filling Suggested by Yuya.

tags: mention --quiet switch in help (issue4920)

merge with stable

histedit: omit useless message from update (edit) specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

histedit: omit useless message from update (_histedit) specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

histedit: omit useless message from update (histeditaction) specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

update: add quietempty flag to _showstats if called with quietempty=True, suppress: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

histedit: omit useless message from abort specifically: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

hg: add quietempty flag to _showstats if called with quietempty=True, suppress: 0 files updated, 0 files merged, 0 files removed, 0 files unresolved

patch: disable nobinary when HGPLAIN=1 The diff output without binaries is definitely great for interactive users - a binary patch is not meaningful for them. Although setting diff.nobinary flag can break the automation. Let's force full output for automation.

help: add missed last new line to "internals" topic Caught by test-gendoc.t.

dirstate: make delaywrite sleep until the next multiple of n seconds Rather than sleep for 2 seconds, we sleep until the next even-numbered second, which has the same effect, but makes tests faster. This removes test-largefiles-update as the long pole of the test suite.

dirstate: only invoke delaywrite if relevant This avoids a significant amount of sleeping in some of our longest tests.

dirstate: move delaywrite logic from write to _write This will allow us to be smarter in upcoming patches.

run-tests: show scheduling with --showchannels This gives one line of output per second with one column per -j level that allows analyzing test scheduling problems. First 24 seconds of output at -j 30 looks like this: 0 . 1 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = s. 2 c c o c r l g r s s = c p = c h c a h c g c h c b c c l l c ss 3 h o b o e a e u u u c o a h o e o c g o l h g h u o = a o = s 4 e n s n b r n n b b m t g n l n l w n o e w e n n e r g i . 5 c t o = a g d - r r = m c w v p v . e v g c e c d v x g . m 6 k r l r s e o t e e b a h e e . e . b e . k b k l e t e . p 7 - i e e e f c e p p u n b b r . r . - r . - - - e r e f . o . 8 p b t v - i . s o o n d o d t . t . c t . c s = 2 t n i . r 9 y - e s c l . t - . d - m i - . - . o - . o y r - - s l . t 10 3 p - e h e . s s . l t b r s . s . m s . d m e f s i e . . 11 - e c t e s . . v . e e . . v . v . m v . e r n o v o s . . 12 c r h . c - . . n . 2 m . . n . n . a n . . e a r n n . . . 13 o f e . k u . . . . - p . . - . - . n - . . v m m - . . . . 14 m . c . - p . . . . e l . . s . m . d s . . . e a e . . . . 15 p . k . r d . . . . x a . . i . o . s o . . . - t n . . . . 16 a . h . e a . . . . c t . . n . v . . u . . . m . c . . . . 17 t . e . s t . . . . h e . . k . e . . r . . . e . o . . . . 18 . . a . t e . . . . a . . . . . . . . c . . . r . d . . . . 19 . . d . o . . . . . n . . . . . . . . e . . . g . i . . . . 20 . . s . r . . . . . g . . . . . . . . . . . . e . n . . . . 21 . . . . e . . . . . e . . . . . . . . . . . . 2 . g . . . . 22 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 . . . . . . . . . . . . . . . . . . . . . . . . . = . . . . ^C Test names read off vertically, beginning with '='. Idle time (not shown) appears as blank space.

tests: avoid duplicate install steps in test-run-tests At several seconds each, this is significantly slowing down the test.

run-tests: add more scheduling weight hints The scheduler would like to order test execution by expected run-time, but doesn't know much about how long a test will run. It thus uses test size as a proxy for run-time. By tweaking these weights we can keep CPUs more evenly busy and thus finish sooner. In particular, this change pushes the three currently longest-running tests closer to the beginning: test-largefiles-update.t test-run-tests.t test-gendoc.t As the largefiles test is currently the long pole of the test suite with higher -j factors, the sooner it's started, the sooner the tests can end. We also up the weight on some shorter but long-running tests that could have previously delayed completion with low -j factors by running very close to the end.

run-tests: report timeouts in a less alarming fashion Rather than report timed-out tests like this: ERROR: test-convert-svn-sink.t output changed ! ..simply put a 't' rather than a '.' in the stream.

ui: try to handle $$ more robustly in prompts (issue4970)

merge with stable

import: reorder help text Try to place key concepts early+together.

import: add word to help text

import: refactor exact flag

help: filter extension commands

test-help: tighten grep patterns Help should output section headings, but no debug commands

summary: add troubles list to the output of hg summary This patch adds troubles information to the output of hg summary. Example line displayed in hg summary: unstable: 1 changeset

wix: add missing template

tests: add coverage to ensure Wix tracks 'help' and 'templates' files This would have caught the problem fixed by 65d2538ac993. There are other *.wxs files that can be checked, but they appear to be more complicated. For example, locale.wxs has what appears to be foreach loop support, as well as variable substitution. By checking `hg files` to determine tracked file, this is able to avoid false failures when other junk is present in the filesystem, like *.orig files. I can't tell if the map-cmdline.status file is not included on purpose, but I don't see the purpose of excluding it. The missing help files seem reasonable for Windows.

perf: adjust perfstartup() for Windows The /dev/null redirect was causing the following error: The system cannot find the path specified. Adjusting HGRCPATH as part of the command line causes the system to try to execute 'HGRCPATH'.

test-hgignore: conditionalize an illegal Windows filename

mac: fix percent-encoding of non-utf-8 characters (issue4999)

help: support loading sub-topics If a sub-topic/section is requested and the main topic corresponds to a topic with sub-topics, we now look for and return content for a sub-topic if found. With this patch, `hg help internals.X` now works. hgweb does not yet render sub-topics, however.

help: pass sub-topic into help query functions While we will likely only use this variable in helptopic(), all these functions are called with the same arguments, so we have to be consistent.

help: pass subtopic into help() Now that we have multiple directories where help topics can live, we need a mechanism to access them. We already use "." to separate topic from section. So it seems logical to also use "." to denote the sub-directory of a topic. This patch teaches the help command to parse out the possible sub-topic and pass it to the help system.

help: add "internals" topic We introduce the "internals" help topic, which renders an index of available sub-topics. The sub-topics themselves are still not reachable via the help system.

help: teach loaddoc to load from a different directory The help system currently only supports showing help topics from a single directory. We'll need to teach it to show results from different directories in order to show the internals topics. The first step is to teach loaddoc() to load documentation from a sub-directory.

setup.py: package internals help files mpm says internal docs should be visible via `hg help` and hgweb. They need to be in the distribution for this to work. Package them.

help: add documentation for bundle types Bundle types and the high-level data format of each bundle isn't documented anywhere. Let's document this as well. Obviously there are many more details about bundles that could be written about. But you have to start somewhere.

help: add documentation for changegroup formats There is no formal location for spec-like technical/internal docs. The repository makes sense as such a location because spec-like documentation should be reviewed (ruling out a wiki). mpm has also stated that he would like this documentation to be part of the built-in help system. So, we establish an "internals" sub-directory to hold this class of documentation. The format of changegroups does not appear to be documented anywhere, even in source code. It therefore seemed like an appropriate first thing to document. This patch adds low-level documentation of versions 1 and 2 of the changegroup foromat. It currently only describes the raw data format. There is probably room to write higher-level documentation on strategies for producing and consuming the data. We'll leave that for another day. The added file is not yet accessible via `hg help` nor via hgweb. Support for this will follow in subsequent patches.

util: reimplement lrucachedict As part of attempting to more aggressively use the existing lrucachedict, collections.deque operations were frequently showing up in profiling output, negating benefits of caching. Searching the internet seems to tell me that the most efficient way to implement an LRU cache in Python is to have a dict indexing the cached entries and then to use a doubly linked list to track freshness of each entry. So, this patch replaces our existing lrucachedict with a version using such a pattern. The recently introduced perflrucachedict command reveals the following timings for 10,000 operations for the following cache sizes for the existing cache: n=4 init=0.004079 gets=0.003632 sets=0.005188 mixed=0.005402 n=8 init=0.004045 gets=0.003998 sets=0.005064 mixed=0.005328 n=16 init=0.004011 gets=0.004496 sets=0.005021 mixed=0.005555 n=32 init=0.004064 gets=0.005611 sets=0.005188 mixed=0.006189 n=64 init=0.003975 gets=0.007684 sets=0.005178 mixed=0.007245 n=128 init=0.004121 gets=0.012005 sets=0.005422 mixed=0.009471 n=256 init=0.004143 gets=0.020295 sets=0.005227 mixed=0.013612 n=512 init=0.004039 gets=0.036703 sets=0.005243 mixed=0.020685 n=1024 init=0.004193 gets=0.068142 sets=0.005251 mixed=0.033064 n=2048 init=0.004070 gets=0.133383 sets=0.005160 mixed=0.050359 n=4096 init=0.004053 gets=0.265194 sets=0.004868 mixed=0.048352 n=8192 init=0.004087 gets=0.542218 sets=0.004562 mixed=0.032753 n=16384 init=0.004106 gets=1.064055 sets=0.004179 mixed=0.020367 n=32768 init=0.004034 gets=2.097620 sets=0.004260 mixed=0.013031 n=65536 init=0.004108 gets=4.106390 sets=0.004268 mixed=0.010191 As the data shows, the existing cache's retrieval performance diminishes linearly with cache size. (Keep in mind the microbenchmark is testing 100% cache hit rate.) The new cache implementation reveals the following: n=4 init=0.006665 gets=0.006541 sets=0.005733 mixed=0.006876 n=8 init=0.006649 gets=0.006374 sets=0.005663 mixed=0.006899 n=16 init=0.006570 gets=0.006504 sets=0.005799 mixed=0.007057 n=32 init=0.006854 gets=0.006459 sets=0.005747 mixed=0.007034 n=64 init=0.006580 gets=0.006495 sets=0.005740 mixed=0.006992 n=128 init=0.006534 gets=0.006739 sets=0.005648 mixed=0.007124 n=256 init=0.006669 gets=0.006773 sets=0.005824 mixed=0.007151 n=512 init=0.006701 gets=0.007061 sets=0.006042 mixed=0.007372 n=1024 init=0.006641 gets=0.007620 sets=0.006387 mixed=0.007464 n=2048 init=0.006517 gets=0.008598 sets=0.006871 mixed=0.008077 n=4096 init=0.006720 gets=0.010933 sets=0.007854 mixed=0.008663 n=8192 init=0.007383 gets=0.015969 sets=0.010288 mixed=0.008896 n=16384 init=0.006660 gets=0.025447 sets=0.011208 mixed=0.008826 n=32768 init=0.006658 gets=0.044390 sets=0.011192 mixed=0.008943 n=65536 init=0.006836 gets=0.082736 sets=0.011151 mixed=0.008826 Let's go through the results. The new cache takes longer to construct. ~6.6ms vs ~4.1ms. However, this is measuring 10,000 __init__ calls, so the difference is ~0.2us/instance. We currently only create lrucachedict for manifest instances, so this regression is not likely relevant. The new cache is slightly slower for retrievals for cache sizes < 1024. It's worth noting that the only existing use of lurcachedict is in manifest.py and the default cache size is 4. This regression is worrisome. However, for n=4, the delta is ~2.9s for 10,000 lookups, or ~0.29us/op. Again, this is a marginal regression and likely not relevant in the real world. Timing `hg log -p -l 100` for mozilla-central reveals that cache lookup times are dominated by decompression and fulltext resolution (even with lz4 manifests). The new cache is significantly faster for retrievals at larger capacities. Whereas the old implementation has retrieval performance linear with cache capacity, the new cache is constant time until much larger values. And, when it does start to increase significantly, it is a few magnitudes faster than the current cache. The new cache does appear to be slower for sets when capacity is large. However, performance is similar for smaller capacities. Of course, caches should generally be optimized for retrieval performance because if a cache is getting more sets than gets, it doesn't really make sense to cache. If this regression is worrisome, again, taking the largest regression at n=65536 of ~6.9ms for 10,000 results in a regression of ~0.68us/op. This is not significant in the grand scheme of things. Overall, the new cache is performant at retrievals at much larger capacity values which makes it a generally more useful cache backend. While there are regressions, their absolute value is extremely small. Since we aren't using lrucachedict aggressively today, these regressions should not be relevant. The improved scalability of lrucachedict should enable us to more aggressively utilize lrucachedict for more granular caching (read: higher capacity caches) in the near future. The impetus for this patch is to establish a cache of decompressed revlog revisions, notably manifest revisions. And since delta chains can grow to >10,000 and cache hit rate can be high, the improved retrieval performance of lrucachedict should be relevant.

record: don't dereference symlinks while copying over stat data Previously, we could be calling os.utime or os.chflags (via shutil.copystat) on a symlink. These functions dereference symlinks, so this would have caused the timestamp of the target to be set. On a read-only or similarly weird filesystem, this might cause an exception to be raised. This is pretty hard to test because conjuring up a read-only filesystem for test purposes is non-trivial.

copyfile: add an optional parameter to copy other stat data Contrary to the comment, I didn't see any evidence that we were copying atime/mtime at all. This adds a parameter to copyfile to optionally copy it and other stat data, with the default being to not copy it. Many systems don't support changing the timestamp of a symlink, but we don't need that in general anyway -- copystat is mostly useful for editors, most of which will dereference symlinks anyway.

tests: move the '-hg' postfix for all style tests We had them on 'test-check-code-hg.t' to avoid collision with the test checking 'check-code' itself. Now that this one have been rename, we can safely remove this suffix for all of them. This get them in line with 'check-pyflakes.t'.

test: rename 'check-code' own test to 'test-contrib-check-code.t' This test (making sure the 'check-code' script run as intended) have been confused with the test making that the mercurial code base comply with our coding still by multiple generations of contributors. We are moving it out of the way so that all tests starting with 'test-check' are now doing compliance testing.

parsers: add a missed PyErr_NoMemory