py3: do I/O in bytes in test-help.t

tests: accept slightly different zip file in Python 3 I added some `unzip -t` here and I *think* the only change is from Python 3 having more data in the zip file headers or something. Sigh. Differential Revision: https://phab.mercurial-scm.org/D5075

webcommands: use stringutil.pprint() to repr invalid archive types Differential Revision: https://phab.mercurial-scm.org/D5074

archival: don't try and fsdecode non-{bytes,str} objects This function accepts both bytes and file-like objects. Differential Revision: https://phab.mercurial-scm.org/D5073

tests: fix last failure in test-tools.t Differential Revision: https://phab.mercurial-scm.org/D5072

run-tests: run tests with as many processes as cores by default This seems like a useful default behavior so tests run faster by default* * Except in special circumstances where the OS/filesystem doesn't scale well to many CPU cores (like APFS *cough* *cough*). Differential Revision: https://phab.mercurial-scm.org/D5071

run-tests: print number of tests and parallel process count This seems like a useful output message to have. I also sneak in a change to lower the parallel process count if it is larger than the number of tests, as that makes no sense and output saying we're running more tests in parallel than there exists tests would be wonky. Differential Revision: https://phab.mercurial-scm.org/D5070

releasenotes: use stringutil.wrap() instead of handcrafted TextWrapper wrapper It's silly to splitlines() a joined string, but we don't care the performance here.

match: optimize matcher when all patterns are of rootfilesin kind Internally at Google, we use narrowspecs with only rootfilesin-kind patterns. Sometimes there are thousands of such patterns (i.e. thousands of tracked directories). In such cases, it can take quite long to build and evaluate the resulting matcher. This patch optimizes matchers that have only patterns of rootfilesin so it instead of creating a regular expression, it matches the given file's directory against the set of directories. In a repo with ~3600 tracked directories, it takes about 1.35 s to build the matcher and 2.7 s to walk the dirstate before this patch. After, it takes 0.04 s to create the matcher and 0.87 s to walk the dirstate. It may be worthwhile to do similar optimizations for e.g. patterns of type "kind:", but that's not a priority for us right now. Differential Revision: https://phab.mercurial-scm.org/D5058

churn: use integer division consistently This results in slight output changes, but it's at least consistent between Python 2 and 3. Since the output is just bar graphs anyway, I'm content with the changes. Differential Revision: https://phab.mercurial-scm.org/D5063

churn: fix stack traces on Python 3 Differential Revision: https://phab.mercurial-scm.org/D5062

py3: moar passing tests Differential Revision: https://phab.mercurial-scm.org/D5055

py3: one new passing tests Differential Revision: https://phab.mercurial-scm.org/D5006

tests: expand our coverage of errors in Python 3 for bad extensions What a mess. Differential Revision: https://phab.mercurial-scm.org/D5005

tests: add lots of globs and conditional output lines The test now passes on Python 3. I'm going to do one follow-up where I'll adjust some grep calls to improve our checking of things on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5004

releasenotes: fix remaining bytes/unicode issues caught by tests All tests now pass. Differential Revision: https://phab.mercurial-scm.org/D5054

relnotes: port to Python 3 The big annoyance here was having to feed textwrap unicodes instead of bytes, but it all seems to work. Differential Revision: https://phab.mercurial-scm.org/D5053

py3: use __code__ instead of func_code to test if func is cachable Surprisingly, this fixes test-cache-abuse.t, which would generate a different revbranch cache if filtered set had no cache.

py3: pass in system string to vars(branchmap).__contains__()

test-bookmarks-pushpull: use correct tmp dir for no-bm-move.sh test-bookmarks-pushpull.t writes a temporary file in $TESTDIR instead of $TESTTMP. Make it use $TESTTMP instead. Differential Revision: https://phab.mercurial-scm.org/D5057

py3: add some glob to make output compatible with python 3 This makes test-pathconflicts-basic.t and test-origbackup-conflict.t passing on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5042

py3: add a b'' prefix in tests/test-requires.t This makes the test pass on Python 3. # skip-blame because just b'' prefix. Differential Revision: https://phab.mercurial-scm.org/D5041

py3: use '%d' for rev nums instead of '%s' This makes test-close-head.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5040

py3: more globing of things to make output compatible between py2 and py3 This also makes the test-ssh-repoerror.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5038

py3: use stringutil.pprint() instead of '%r' This makes test-pager-legacy.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5037

py3: use '%d' for integers instead of '%s' Differential Revision: https://phab.mercurial-scm.org/D5036

py3: add some py3 specific output to test This makes the test work on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5035

py3: use stringutil.pprint() in color.py instead of '%r' This fixes the test-status-color.t on python 3. Differential Revision: https://phab.mercurial-scm.org/D5034

py3: add b'' prefixes in tests/test-progress.t This makes the test pass on Python 3. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5033

py3: add b'' prefixes to tests/test-keyword.t This test is now pretty close to passing on Python 3. # skip-blame because just b'' prefixes. Differential Revision: https://phab.mercurial-scm.org/D5032

py3: use stringutil.pprint() if we are printing bool values Differential Revision: https://phab.mercurial-scm.org/D5031

py3: glob some difference between py2 and py3 output On py2, the directory names are under quotes and on py3 they are not and I don't know why. Differential Revision: https://phab.mercurial-scm.org/D5030

py3: byteify tests/wireprotosimplecache.py # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5029

py3: add b'' prefixes in tests/test-help.t This takes us close to make the test passing. # skip-blame because just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5028

py3: use sys.stdout.buffer for binary output in tests/svnxml.py Just following 6c0b1d964537ab62d76c208f5f04ab414814c94e here. Differential Revision: https://phab.mercurial-scm.org/D5027

py3: use '%s' instead of %r in hgext/convert/subversion.py Using '%r' on bytes in python 3 adds b'' prefixes to output which is bad. I don't see a reason why we want to repr() the string here, so just switched to '%s'. This fixes some output failure in one of the subversion test. Differential Revision: https://phab.mercurial-scm.org/D5026

py3: make test-contrib-perf.t work on python 3 Differential Revision: https://phab.mercurial-scm.org/D5025

py3: add 3 more passing tests to whitelist caught by buildbot Differential Revision: https://phab.mercurial-scm.org/D5024

py3: add some r'' prefixes in hgext/narrow/narrowwirepeer.py This fixes test-narrow-widen-no-ellipsis.t on Python 3. # skip-blame because just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5023

py3: r'' prefix default values for mimetypes.guess_mime() As suggested by @yuja in D4967. Differential Revision: https://phab.mercurial-scm.org/D5020

absorb: update help text Update the help text to reflect the recent changes to absorb in D4999. The `-p` option actually only affects the `-a` option: without `-a`, the changes are printed; with `-a` the changes are only printed if `-p` is specified. Reword the help text for `-p` to reflect this. Differential Revision: https://phab.mercurial-scm.org/D5022

fsmonitor: use vfs instead of opener (issue5938) "opener" of localrepository object was dropped at Mercurial 4.3 (or a7e210167c28). "vfs" should be used instead. wlock is required to write into a file under .hg directory. For efficiency, we should change _cmpsets() from: 1. acquire wlock 2. open log file under .hg directory with write mode 3. compare between result of watchman and Mercurial's dirstate logic 4. write out error info into a file, if error is detected 5. release wlock to: 1. compare between result of watchman and Mercurial's dirstate logic 2. acquire wlock, if error is detected 3. open and write error info into a file 4. release wlock But this is another issue.

tests: configure fsmonitor.mode=paranoid always if fsmonitor is used This forces fsmonitor extension execute "paranoid" code path. Strict speaking, we should make fsmonitor-run-tests.py accept own specific options, but there is no code path, which is disabled in "paranoid" mode, at least now. Therefore, this solution seems reasonable enough.

tests: drop files from exclusion list in test-check-module-imports.t Now, these files has no style issue at checking with import-checker.py. BTW, tests/test-imports-checker.t is still excluded, because almost all code fragments in it has intentional importing style violation. Using NO_CHECK_EOF instead of EOF as heredoc limit mark can make import-checker.py ignore such fragments. But keeping these fragments checkable seems still useful: for example, test-imports-checker.t can be used to test whether import-checker.py can detect erroneous code fragment in test script as expected.

tests: use NO_CHECK_EOF for fragments used to test importing via extension test-extension.t is excluded in test-check-module-imports.t, because import-checker.py reports that some python code fragments in it does not import modules in recommended style. These code fragments are used to test importing modules relatively and absolutely via extension (e.g. issue4029, issue5208, and so on). Test specific package hierarchy becomes complicated for this purpose, and it seems reasonable to avoid checking these code fragments with import-checker.py. But on the other hand, other code fragments in test-extension.t should be checked by import-checker.py. Therefore, this patch uses NO_CHECK_EOF limit mark only for python code fragments, which are used to test importing via extension in test-extension.t. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: use NO_CHECK_EOF for fragments having intentional error Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that code fragments embedded in these test scripts have syntax error. But these syntax error is intentional. This patch uses NO_CHECK_EOF instead of EOF as heredoc limit mark for such fragments, in order to make import-checker.py ignore them. NO_CHECK_EOF limit mark tells import-checker.py that this code fragment should be ignored, via testparseutil.py.

tests: fix style issue of importing order in test-lock.py test-lock.py is excluded in test-check-module-imports.t, because import-checker.py reports that some stdlibs are imported after local silenttestrunner module. I can not found out any reason why tests/silenttestrunner.py should be imported before some stdlibs from the point of functionality view.

tests: fix style issue of importing hgweb in embedded code fragments Some test scripts are excluded in test-check-module-imports.t, because import-checker.py reports that hgweb and/or hgwebdir of mercurial.hgweb are not imported in recommended style. To fix this issues, this patch make python code fragments embedded in these files import hgweb from mercurial package at first, and refer hgweb and hgwebdir via imported hgweb.

py3: encode str to bytes These fields are str on Python 2 and 3. This module doesn't import any Mercurial modules. So I just did the str -> bytes inline. Differential Revision: https://phab.mercurial-scm.org/D5010

py3: sprinkle statprof.py with utf-8 encoding Trying to chase the long tail of unicode badness in this file due to frame / code objects holding str everywhere. Differential Revision: https://phab.mercurial-scm.org/D5016

py3: flush stdout This avoids buffering due to mixed print() and sys.stdout.write() by profiling code. And with this, test-profile.t passes on Python 3 \o/. Honestly, I'd be shocked if there weren't more Python 3 encoding issues lingering in the profiling code. That code isn't super high quality and doesn't seem to have thorough test coverage. Differential Revision: https://phab.mercurial-scm.org/D5018

py3: use raw strings in statprof.py This allows main() and argument parsing to work. Differential Revision: https://phab.mercurial-scm.org/D5017

py3: switch from print(..., file=) to write() Because Python 3's print() doesn't like bytes. Differential Revision: https://phab.mercurial-scm.org/D5015

py3: use %d in a few places Differential Revision: https://phab.mercurial-scm.org/D5014

py3: use raw strings for stack names These are used for simple compares. We don't need to normalize to bytes. As part of this, I expanded a set literal so entries are 1 per line. Differential Revision: https://phab.mercurial-scm.org/D5013

py3: use sysbytes for converting code attributes Differential Revision: https://phab.mercurial-scm.org/D5012

py3: use write() instead of print() Because print() expects str and we want to write bytes. There should be no functional changes as part of this. Differential Revision: https://phab.mercurial-scm.org/D5011

py3: use %d to format ints Differential Revision: https://phab.mercurial-scm.org/D5009

py3: convert sorting field to sysstr This is used as part of an attribute name check and needs to be native str. Differential Revision: https://phab.mercurial-scm.org/D5008

py3: encode json output to bytes and use write() Differential Revision: https://phab.mercurial-scm.org/D5007

absorb: prompt user to accept absorb changes by default Change the default absorb behaviour to print the changes out and then prompt the user if they want to absorb the changes or not. Providing the `-p` option still prints the result and exits. A new `-a` option is provided which applies the changes without printing them, giving the equivalent of the old behaviour. Differential Revision: https://phab.mercurial-scm.org/D4999

templatefuncs: add truncate parameter to pad Add a truncate option to pad that additionally truncates the text to the pad width if it is wider. Since color codes can cause a problem with this, when the text is truncated, the color codes are also stripped. Users of the truncate option should label the text outside the pad. Differential Revision: https://phab.mercurial-scm.org/D5000

absorb: print summary of changesets affected Differential Revision: https://phab.mercurial-scm.org/D4998

absorb: use a formatter to generate output Change absorb to use a formatter to generate its output. This allows the use of templates to customize the output. Differential Revision: https://phab.mercurial-scm.org/D4997

tests: fix up test-bad-extension.t's inline extension Upcoming patches will be simpler, I think. Differential Revision: https://phab.mercurial-scm.org/D5003

polib: update to latest release 1.0.7 (upstream rev d75ce6dbbc2a) # no-check-commit third-party code that doesn't match our style Differential Revision: https://phab.mercurial-scm.org/D5001

tests: glob away some annoying py3 differences Differential Revision: https://phab.mercurial-scm.org/D4994

tests: fix a repr on python3 in test-extension.t Differential Revision: https://phab.mercurial-scm.org/D4993

tests: so many b prefixes in test-extension.t # skip-blame because it's b prefixes Differential Revision: https://phab.mercurial-scm.org/D4992

tests: ensure print() statements in test-extension.t all flush Differential Revision: https://phab.mercurial-scm.org/D4991

revlog: update pure nodecache start lookup offset on insertion test-storage.py is failing in pure builds because the population of the node cache isn't accounting for the new starting offset after a revlog insertion. This commit updates the node cache start offset to account for insertions. I'm not 100% convinced this is the ideal solution. But it works and seems correct. Differential Revision: https://phab.mercurial-scm.org/D4996

exchangev2: use filesdata filesdata is a more efficient mechanism for bulk fetching files data for a range of changesets. Let's use it in exchangev2. With this change, a client performing a full clone of mozilla-unified transmits substantially fewer bytes across the wire: before: 139,124,863 bytes sent after: 20,522,499 bytes sent The bulk of the remaining bytes is likely the transfer of ~1M nodes for changesets and manifests. We can eliminate this by making requests in terms of node ranges instead of explicit node lists... Differential Revision: https://phab.mercurial-scm.org/D4982

wireprotov2: define and implement "filesdata" command Previously, the only way to access file revision data was the "filedata" command. This command is useful to have. But, it only allowed resolving revision data for a single file. This meant that clients needed to send 1 command for each tracked path they were seeking data on. Furthermore, those commands would need to enumerate the exact file nodes they wanted data for. This approach meant that clients were sending a lot of data to remotes in order to request file data. e.g. if there were 1M file revisions, we'd need at least 20,000,000 bytes just to encode file nodes! Many clients on the internet don't have that kind of upload capacity. In order to limit the amount of data that clients must send, we'll need more efficient ways to request repository data. This commit defines and implements a new "filesdata" command. This command allows the retrieval of data for multiple files by specifying changeset revisions and optional file patterns. The command figures out what file revisions are "relevant" and sends them in bulk. The logic around choosing which file revisions to send in the case of haveparents not being set is overly simple and will over-send files. We will need more smarts here eventually. (Specifically, the client will need to tell the server which revisions it knows about.) This work is deferred until a later time. Differential Revision: https://phab.mercurial-scm.org/D4981

wireprotov2: extract file object emission to own function An upcoming commit will introduce another caller. Differential Revision: https://phab.mercurial-scm.org/D4980

wireprotov2: change how revisions are specified to changesetdata Right now, we have a handful of arguments for specifying the revisions whose data should be returned. Defining how all these arguments interact when various combinations are present is difficult. This commit establishes a new, generic mechanism for specifying revisions. Instead of a hodgepodge of arguments defining things, we have a list of dicts that specify revision selectors. The final set of revisions is a union of all these selectors. We implement support for specifying revisions based on: * An explicit list of changeset revisions * An explicit list of changeset revisions plus ancestry depth * A DAG range between changeset roots and heads If you squint hard enough, this problem has already been solved by revsets. But I'm reluctant to expose revsets to the wire protocol because that would require servers to implement a revset parser. Plus there are security and performance implications: the set of revision selectors needs to be narrowly and specifically tailored for what is appropriate to be executing on a server. Perhaps there would be a way for us to express the "parse tree" of a revset query, for example. I'm not sure. We can explore this space another time. For now, the new mechanism should bring sufficient flexibility while remaining relatively simple. The selector "types" are prefixed with "changeset" because I plan to add manifest and file-flavored selectors as well. This will enable us to e.g. select file revisions based on a range of changeset revisions. Differential Revision: https://phab.mercurial-scm.org/D4979

wireprotov2: stop sending phase updates for base revisions This feature is broken and doesn't work properly in all scenarios. e.g. if we have the following DAGs: client server D draft C draft C draft B draft B public A public A public The current code would only send the phase data for C. The client wouldn't see that B moved from draft to public. This feature will be restored in a future commit. For now, it is making refactoring of how revisions are specified in the wire protocol a bit difficult... Differential Revision: https://phab.mercurial-scm.org/D4978

debugcommands: support wrapping long lines If a line within a block is indented more than the line that came before, we automatically concatenate it with the previous line. This allows us to pretty format data. This will make tests easier to read. At some point we may just want to evaluate entire blocks as Python code or something, as even with this change, things aren't perfect, as we can't e.g. have formatting like: foo eval:[ True ] But this is strictly better than before, where we couldn't wrap long lines. Differential Revision: https://phab.mercurial-scm.org/D4977

exchangev2: honor server advertised manifestdata recommended batch size Let's plug the client up to the server-advertised recommended batch size for manifestdata requests. Differential Revision: https://phab.mercurial-scm.org/D4976

wireprotov2: advertise recommended batch size for requests Currently, exchangev2 hardcodes the batch size for how many revisions to fetch per command request. A single value is not appropriate for every repository because some repositories may have a drastically different "shape" from other repositories. e.g. a repo with lots of small files may benefit from larger batch sizes than a repo with lots of large files. And depending on caching used by the server, the server may wish to control the number of commands (to e.g. mitigate overhead of following content redirects). This commit teaches wireprotov2 commands to declare extra metadata which is advertised as part of the command descriptor. The manifestdata command has been taught to advertise a recommended batch size for requests. Differential Revision: https://phab.mercurial-scm.org/D4975

httppeer: expose API descriptor on httpv2peer The API descriptor in wireprotov2 is much more expressive than space-delimited tokens and it will be difficult to define methods to query it in all of the ways we'll want to query it. So let's just declare defeat and expose the API descriptor on the peer instance. As part of this, we define a new interface for version 2 peers, fulfilling a TODO in the process. Differential Revision: https://phab.mercurial-scm.org/D4974

tests: use baseurl instead of advertisedbaseurl The distinction matters for e.g. hosts behind load balancers. But for the test environment, it doesn't matter. For whatever reason, advertisedbaseurl is resolving to http://1.0.0.127.in-addr.arpa:$HGPORT on my MBP. This hostname fails to resolve, causing the test to fail. No clue what's up with that behavior. Differential Revision: https://phab.mercurial-scm.org/D4973

py3: another one started passing Differential Revision: https://phab.mercurial-scm.org/D4990

py3: one more passing test Differential Revision: https://phab.mercurial-scm.org/D4989

scmutil: wrap locker information in bytestr before repr()ing it Differential Revision: https://phab.mercurial-scm.org/D4988

py3: more passing tests Differential Revision: https://phab.mercurial-scm.org/D4987

py3: whitelist another passing test Differential Revision: https://phab.mercurial-scm.org/D4986

obsolete: convert error string to a sysstr This should be okay because we're just %-formatting a hash into a localized string. Differential Revision: https://phab.mercurial-scm.org/D4985

tests: ensure the python 3 passing test list stays sorted Differential Revision: https://phab.mercurial-scm.org/D4984

py3: sort passing test list Differential Revision: https://phab.mercurial-scm.org/D4983

py3: tweak stdout writing in test-hgweb-no-path-info.t We want to write bytes for convenience. This requires sys.stdout.buffer. But using sys.stdout.buffer introducing buffered output. So we sprinkle code with sys.stdout.flush() to force immediate writes. After all that, Python 3 was emitting b'' prefixed output for errors. So we only print errors if there were some. There aren't, so b'' don't come into play and output is identical in Python 2 and 3. Differential Revision: https://phab.mercurial-scm.org/D4972

py3: use .+ instead of .* in regexp pattern Python 3.7 changed the behavior of re.sub(). See https://bugs.python.org/issue33585. The new code should work on old and new Pythons. Differential Revision: https://phab.mercurial-scm.org/D4971

py3: coerce bytestr to bytes to appease urllib.parse.quote_from_bytes() Differential Revision: https://phab.mercurial-scm.org/D4969

py3: pass str and return bytes from mimetypes.guess_type() This function wants a str (which represents a path) and returns a str. We normalize input to str and output to bytes. Differential Revision: https://phab.mercurial-scm.org/D4967

py3: use bytes literal in test-hgweb-json.t # skip-blame just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D4970

py3: convert diff opcode name to bytes Differential Revision: https://phab.mercurial-scm.org/D4968

py3: byteify hgweberror.py # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4966

py3: encode JSON str to bytes json.dumps() will emit UTF-8 str on Python 2 and 3. Use sysbytes to force the .encode('utf-8') on Python 3 and no-op on Python 2. Differential Revision: https://phab.mercurial-scm.org/D4965

py3: use b'' in test-check-interfaces.py # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4964

revlog: optimize ancestors() to not check filtered revisions for each While reviewing the Rust implementation, I noticed iter(ancestors) doesn't need to check filtering state for each parent revision. And doing that appears to have some measurable perf win. $ hg perfancestors -R mercurial (orig) wall 0.038093 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) (this) wall 0.024795 comb 0.020000 user 0.020000 sys 0.000000 (best of 117)

phabricator: drop support for the legacy phabricator.auth.token config (BC) The test for this broke in dc82ad1b7f77 when statistics started being tracked. It wasn't noticed because none of the bots have the vcr module installed. It looks like the custom_patches argument should patch in the custom httpconnection, and I can't figure out what is going on.

absorb: print '{rev}:' as a prefix to the hash I don't see a sane way to print the summary to identify the changed commit, but this at least makes it a bit easier to identify a commit when a group of them changes.

packaging: fix "make centos{5,6,7}" Without this change, make centos{5,6,7} fails with error: cp: cannot stat '<basedir>/rpmbuild/RPMS/*/*': No such file or directory This change is the exact equivalent of bc4bbc42899a (which applied to Fedora).

narrow: don't compress the bundle2 when sending 'error:abort' This is similar to waht getbundle() does and also explicitly specifies that we should get a compressed bundle2 in normal cases when not sending 'error:abort'. Differential Revision: https://phab.mercurial-scm.org/D4934

push: add "remote" to 'repository changed while pushing' messages (issue5971) Differential Revision: https://phab.mercurial-scm.org/D4933

fuzz: try setting PYTHONNOUSERSITE=1 to avoid loading site-packages Right now the fuzzer is crashing trying to look up the home dir for uid 0, which is breaking in the fuzz environment. Differential Revision: https://phab.mercurial-scm.org/D4936

perf: extract result formatting in its own function This will make it easier to reuse in another changesets.

perf: extract the timing of a section in a context manager This makes it easier to reuse it in other (future) part of the code that requires their own time management.

pref: support negative indexing in perfrevlogrevisions This is useful to check for a consistent final slice in multiple different repositories.

perf: accept formatter option for perfmanifest

perf: fix -T json The previous code was mixing formatting and data, breaking `-T json` with unexpected data. We fix the issue and add a test to prevent future regression.

formatter: more details on assertion failure This is useful when the assertion fails.

wireprotov2: raise ProgrammingError on unknown action Suggested by @durin42 in review of D4923. Differential Revision: https://phab.mercurial-scm.org/D4935

wireprotov2: send content encoded frames from server Now that we have support for negotiating encodings and configuring an encoder, we can start sending content encoded frames from the server. This commit teaches the wireprotov2 server code to send content encoded frames. On the mozilla-unified repository with zstd enabled peers, this change reduces the total amount of data transferred from server to client drastically: befor: 7,190,995,812 bytes after: 1,605,508,691 bytes Differential Revision: https://phab.mercurial-scm.org/D4927

wireprotov2: raise exception in objects() if future has been resolved Differential Revision: https://phab.mercurial-scm.org/D4926

wireprotov2: don't emit empty frames Staring at logs revealed the presence of empty frames that should have contained payload. Let's stop that from happening. Differential Revision: https://phab.mercurial-scm.org/D4925

wireprotov2: remove functions for creating response frames from bytes All code in the actual server uses oncommandresponsereadyobjects(). Test code was ported to that method. This resulted in a handful of subtle test changes. Differential Revision: https://phab.mercurial-scm.org/D4924

wireprotov2: handle noop action This action can be returned from the client reactor. We should handle it. Differential Revision: https://phab.mercurial-scm.org/D4923

wireprotov2: send protocol settings frame from client Now that we have client and server reactor support for protocol settings and encoding frames, we can start to send them out over the wire! This commit teaches the client reactor to send out a protocol settings frame when needed. The httpv2 peer has been taught to gather a list of supported content encoders and to advertise them through the client reactor. Because the client is now sending new frame types by default, this constitutes a compatibility break in the framing protocol. The media type version has been bumped accordingly. This will ensure existing clients won't attempt to send the new frames to old servers not supporting this explicit media type. I'm not bothering with the BC annotation because everything wireprotov2 is highly experimental and nobody should be running a server yet. Differential Revision: https://phab.mercurial-scm.org/D4922

wireprotov2: define and use stream encoders Now that we have basic support for defining stream encoding, it is time to start doing something with it. We define various classes implementing stream encoders/decoders for the defined encoding profiles. This is relatively straightforward. We teach the inputstream and outputstream classes how to encode, decode, and flush data. We then teach the clientreactor how to filter received data through the inputstream decoder. One of the features of the framing format is that streams can span requests. This is a differentiating feature from say HTTP/2, which associates streams with requests. By allowing streams to span requests, we can reuse compression context data across requests/responses. But in order to do this, we need a mechanism to "flush" the encoder at logical boundaries so that receivers receive all data where it is expected. And a "flush" event is distinct from a "finish" event from the perspective of certain compressors because a "flush" will retain compression context state whereas a "finish" operation will not. This is why encoders have both a flush() and a finish() and each uses specific flushing semantics on the underlying compressor. The added tests verify various behavior of decoders via clientreactor. These tests do test some compression behavior via use of outputstream. But for all intents and purposes, server reactor support for encoding is not yet implemented. Differential Revision: https://phab.mercurial-scm.org/D4921

wireprotov2: establish dedicated classes for input and output streams Streams are unidirectional. As part of implementing encoding/decoding support, it became clear that it didn't make sense for a generic "stream" class to hold functionality related to both encoding and decoding. So we create new classes to represent the flavor of stream. Differential Revision: https://phab.mercurial-scm.org/D4920

wireprotov2: pass ui into clientreactor and serverreactor This will allow us to use config options to influence compression settings. Differential Revision: https://phab.mercurial-scm.org/D4919

wireprotov2: handle stream encoding settings frames Like what we just did for the server reactor, we teach the client reactor to handle stream encoding settings frames. The code is very similar. We define a method on the stream class to handle processing the data within the decoded frames. However, it doesn't yet do anything useful. Differential Revision: https://phab.mercurial-scm.org/D4918

wireprotov2: document client reactor actions We should document these so consumers have an easier life. Differential Revision: https://phab.mercurial-scm.org/D4917

wireprotov2: handle sender protocol settings frames We teach the server reactor to handle the optional sender protocol settings frames, which can only be sent at the beginning of frame exchange. Right now, we simply decode the data and record the sender protocol settings on the server reactor instance: we don't yet do anything meaningful with the data. Differential Revision: https://phab.mercurial-scm.org/D4916

wireprotov2: update stream encoding specification The encoding of data within streams in the frame-based protocol is not yet defined or implemented. This means that all data in wire protocol version 2 is currently being sent out raw, without compression. That's obviously not ideal. This commit formalizes the beginnings of stream encoding support in the protocol. I suspect we'll change behavior substantially in the future. My goal is to get something landed so we can use compression. We can build out more robust support later. Because the frame type ID changed, this is strictly BC. But existing code wasn't using the frame. I'll bump the framing protocol version later once code is introduced to use the new frame. Differential Revision: https://phab.mercurial-scm.org/D4915

cborutil: cast bytearray to bytes This code didn't like passing in bytearray instances. Let's cast bytearray to bytes so it works. Differential Revision: https://phab.mercurial-scm.org/D4914

tests: disable zstd in test This makes the test pass in pure installs. Differential Revision: https://phab.mercurial-scm.org/D4913

wireprotov2: remove "compression" from capabilities response This is not used. And future commits will change how this mechanism works. Let's remove it. As a bonus, this fixes some test failures on pure installs (due to zstd references). Differential Revision: https://phab.mercurial-scm.org/D4912

zstandard: vendor python-zstandard 0.10.1 This was just released. The upstream source distribution from PyPI was extracted. Unwanted files were removed. The clang-format ignore list was updated to reflect the new source of files. setup.py was updated to pass a new argument to python-zstandard's function for returning an Extension instance. Upstream had to change to use relative paths because Python 3.7's packaging doesn't seem to like absolute paths when defining sources, includes, etc. The default relative path calculation is relative to setup_zstd.py which is different from the directory of Mercurial's setup.py. The project contains a vendored copy of zstandard 1.3.6. The old version was 1.3.4. The API should be backwards compatible and nothing in core should need adjusted. However, there is a new "chunker" API that we may find useful in places where we want to emit compressed chunks of a fixed size. There are a pair of bug fixes in 0.10.0 with regards to compressobj() and decompressobj() when block flushing is used. I actually found these bugs when introducing these APIs in Mercurial! But existing Mercurial code is not affected because we don't perform block flushing. # no-check-commit because 3rd party code has different style guidelines Differential Revision: https://phab.mercurial-scm.org/D4911

rust-chg: install signal handlers to forward signals to server I use sync::Once as a synchronization primitive because it's quite easy to use, and is good enough to prevent data race in these C functions.

rust-chg: remove SIGCHLD handler which won't work in oxidized chg Since pager is managed by the Rust part, the C code doesn't know the pager pid. I could make the Rust part teach the pid to C, but still installing SIGCHLD handler seems horrible idea since we no longer use handcrafted low-level process management functions. Instead, I'm thinking of adding async handler to send SIGPIPE at the exit of the pager.

rust-chg: extract signal handlers from chg/procutil.c abortmsgerrno() and debugmsg() are removed, and the public interface instead returns success/error status. Since signal handlers can't propagate errors, the result of kill() is just ignored.

help: document about "version" template keywords

help: document about "tags" template keywords

help: document about "status" template keywords

help: document about "resolve" template keywords

help: document about "paths" template keywords

help: document about "identify" template keywords

help: document about "grep" template keywords

chgserver: catch Abort while parsing early args to shut down cleanly _loadnewui() calls dispatcher functions, which may raise Abort if unparsable arguments are passed in. The server should catch such errors and translate them to the "exit 255" instruction so the client can finish the IPC session cleanly. Spotted while porting the chg client to Rust.

chg: upgrade client to use "setumask2" command No compatibility code is added to the client side, since it's unlikely for new client to communicate with the old server.

chgserver: add "setumask2" command which uses correct message frame The first 4 bytes should be a length field, not a value. Spotted while porting chg functions to the Rust one.

packaging: "make deb" no longer fails Release 4.7 rationalized the layout of the build scripts. Unfortunately, while "make docker-ubuntu-*" and "make docker-debian-*" worked as expected, "make deb" was broken. Before this change "make deb" was failing with the following error: You are not inside a Mercurial repository! Or, after the latest changes: You are inside <fullpath>, which is not the root of a Mercurial repository Moreover, when "make deb" failed, the cleanup routine deleted the wrong directory (contrib/packaging/debian instead of <reporoot>/debian) resulting in a corrupted working copy that needed to be hg revert-ed. After this change the docker targets continue to work, and the deb one is able to finish.

packaging: cleanup() did not read the value of $CLEANUP When the original author put CLEANUP in a conditional statement he was probably willing to use it to control the "if". This change tries to restore that behaviour: the "rm" clause is triggered if and only if CLEANUP is defined and not empty.

packaging: builddeb's cleanup needs to expand PWD, safely Single quotes would not expand the variable.

packaging: blindly factor out trap's cleanup function in builddeb This commit blindly extracts builddeb's trap routine in a dedicated function. While doing so, I think two bugs are exposed, which will be addressed in the next commits: - single quoting around '$CLEANUP' will always evaluate to the literal '$CLEANUP' regardless of the variable's value. The "if" will always be true. - the removal operation will not expand $PWD (and a variable expansion would need double quotes, anyways.

packaging: print full path to the packages when builddeb finishes successfully

packaging: print more specific error messages when builddeb fails

cmdutil: sort unresolved paths I noticed that `hg status` was printing unresolved paths in a non-deterministic order. This patch fixes that. I'm not sure if the sorting should be done in merge.mergestate.unresolved() instead. Either way fixes the presentation issue. Differential Revision: https://phab.mercurial-scm.org/D4929

fuzz: report error if Python code raised exception I think that's what we wanted to do, given the most of the code block is surrounded by try-except. 'lazymanifest(mdata)' is moved to the try block as it can fail.

revlog: explicitly initialize static variables I know .bss section is zero-filled, but explicit initialization should be better as we rely on that.

tests: do not change sys.path, it can break loading cext.parsers When running this tests with run-tests, the prefix would resolve mercurial.cext to the source tree and the attempt to load mercurial.cext.parsers would therefore fail since it doesn't exist in it. With the regular search path from run-tests, it is picked up from the temporary prefix correctly. Differential Revision: https://phab.mercurial-scm.org/D4910

tests: deal with differences in tic from ncurses and NetBSD Differential Revision: https://phab.mercurial-scm.org/D4909

closehead: fix close-head -r listification Differential Revision: https://phab.mercurial-scm.org/D4908

import-checker: use testparseutil.embedded() to centralize detection logic This patch fixes issues of embedded() in import-checker.py below, too. - overlook (or mis-detect) the end of inline script in doctest style - overlook inline script in doctest style at the end of file (and ignore invalid un-closed heredoc at the end of file, too) - overlook code fragment in styles below - "python <<EOF" (heredoc should be "cat > file <<EOF" style) - "cat > foobar.py << ANYLIMIT" (limit mark should be "EOF") - "cat << EOF > foobar.py" (filename should be placed before limit mark) - "cat >> foobar.py << EOF" (appending is ignored)

tests: use NO_CHECK_EOF as heredoc limit mark to omit checking code fragments This patch uses NO_CHECK_EOF as heredoc limit mark instead of EOF, in order to avoid checking all python code fragments in test-contrib-check-code.t, because almost all of them has un-recommended implementations intentionally.

contrib: add an utility module to parse test scripts This patch centralizes the logic to pick up code fragments embedded in *.t script, in order to: - apply checking with patterns in check-code.py on such embedded code fragments Now, check-code.py completely ignores embedded code fragments. I'll post another patch series to check them. - replace similar code path in contrib/import-checker.py Current import-checker.py has problems below. Fixing each of them is a little difficult, because parsing logic and pattern strings are tightly coupled. - overlook (or mis-detect) the end of inline script in doctest style 8a8dd6e4a97a fixed a part of this issue, but not enough. - it overlooks inline script in doctest style at the end of file (and ignores invalid un-closed heredoc at the end of file, too) - it overlooks code fragment in styles below - "python <<EOF" (heredoc should be "cat > file <<EOF" style) - "cat > foobar.py << ANYLIMIT" (limit mark should be "EOF") - "cat << EOF > foobar.py" (filename should be placed before limit mark) - "cat >> foobar.py << EOF" (appending is ignored) - it is not extensible for other than python code fragments (e.g. shell script, hgrc file, and so on) This new module can detect python code fragments in styles below: - inline script in doctest style (starting by " >>> " line) - python invocation with heredoc script ("python <<EOF") - python script in heredoc style (redirected into ".py" file) As an example of extensibility of new module, this patch also contains implementation to pick up code fragment below. This will be useful to add additional restriction for them, for example. - shell script in heredoc style (redirected into ".sh" file) - hgrc configuration in heredoc style (redirected into hgrc or $HGRCPATH)

tests: use environment variable indirectly Using environment variable directly in heredoc python code will cause syntax error at checking module importation by import-checker.py strictly, because "$varname" is invalid in Python syntax. "$varname" becomes valid after environment variable substitution by shell at writing text into file. Current import-checker.py overlooks code fragment changed in this patch, because of a restriction below for a line starting code fragment. - filename must be specified before limit mark NG: cat <<EOF > FILE.py OK: cat > FILE.py <<EOF import-checker.py itself is fixed in subsequent patch.

tests: import multiple modules separately Current import-checker.py overlooks code fragment changed in this patch, because of restrictions below for a line starting code fragment. - filename must be specified before limit mark NG: cat <<EOF > FILE.py OK: cat > FILE.py <<EOF - limit mark must not be quoted NG: cat > FILE.py <<'EOF' OK: cat > FILE.py <<EOF import-checker.py itself is fixed in subsequent patch.

fuzz: allow manifest fuzzer to detect leaks Huzzah! Differential Revision: https://phab.mercurial-scm.org/D4907

fuzzers: init Python in LLVMFuzzerInitialize and intentionally leak it This sidesteps leaks (or "leaks", I'm not sure) in CPython, and lets our fuzzer work. Differential Revision: https://phab.mercurial-scm.org/D4906

revlog: if the module is initialized more than once, don't leak nullentry Caught (annoyingly) by the manifest fuzzer. Differential Revision: https://phab.mercurial-scm.org/D4905

narrow: move remaining narrow-limited dirstate walks to core In most places we now filter at a higher level (the context object), but there are few places that relied on the dirstate walk to be filtered by the narrowspec. The important cases are those used by `hg add` and `hg addremove`. This patch updates them to pass in a matcher instead of relying on the dirstate to do the filtering. The dirstate filtering is also dropped in narrowdirstate.py. Not always filtering in the dirstate should be useful for a future `hg status --include-outside-narrow` option. These places now end up doing an unrestricted dirstate walk after this patch: * debugfileset * perfwalk * sparse (but restricted to sparse config) * largefiles I'll let anyone who cares about these cases adapt them to work with narrow if necessary. Differential Revision: https://phab.mercurial-scm.org/D4901

narrow: allow repo.narrowmatch(match) to include exact matches from "match" Differential Revision: https://phab.mercurial-scm.org/D4900

narrow: filter files by narrowspec in ctx.matches() This has no effect yet because 1) for committed changes, ctx.matches() just calls ctx.walk(), which we updated in the previous patch, and 2) for the working copy, the filtering is also done in the overridden dirstate.walk() in narrowdirstate. Differential Revision: https://phab.mercurial-scm.org/D4899

narrow: only walk files within narrowspec also for committed revisions Narrow has been walking only paths matching the narrowspec when walking the working copy. We have not done the same filtering when walking committed revisions (e.g. "hg files -r "), which seems a little odd. Let's make it consistent. Differential Revision: https://phab.mercurial-scm.org/D4898

status: intersect matcher with narrow matcher instead of filtering afterwards I seem to have done a very naive move of the code from the narrow extension into core in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). It seems obvious that a better way is to intersect the matchers. Note that this means that when requesting status for the working directory in a narrow repo, we now pass the narrow matcher (possibly intersected with a user-provided matcher) into _buildstatus() and then into dirstate.status() and dirstate.walk(), which will the intersect it again with the narrow matcher. That's functionally fine, but wasteful. I hope to later remove the dirstate wrapping that adds the second layer of matcher intersection. Differential Revision: https://phab.mercurial-scm.org/D4897

localrepo: allow narrowmatch() to accept matcher to intersect with It's pretty common that we need to intersect a matcher we already have (usually from the user) with the narrow matcher. Let's make repo.narrowmatch() take an optional matcher to intersect with. Differential Revision: https://phab.mercurial-scm.org/D4896

obsolete: fix ValueError when stored note contains ':' char (issue5783) The newer version of `amend -n 'Some some'` accepts containing ':' char. The information contained in this note 'Testing::Obstore' gives ValueError, because we are trying to store more than 2 values in key and value. Differential Revision: https://phab.mercurial-scm.org/D4883 Differential Revision: https://phab.mercurial-scm.org/D4882

narrow: update TODO.rst now that we share format with sparse The narrowspec format was unified with the sparse format in f64ebe7d2259 (narrowspec: use sparse.parseconfig() to parse narrowspec file (BC), 2018-08-03). Differential Revision: https://phab.mercurial-scm.org/D4904

narrow: update TODO.rst now that we filter status in ctx The comment referred to was addressed in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). I also think 84092edd5c88 (narrow: drop unnecessary overrides of patch, 2018-09-28) suggests that it was the right thing to do. Differential Revision: https://phab.mercurial-scm.org/D4903

narrow: update TODO.rst now that the narrowspec is in .hg/store We no longer have the unfortunate wrappostshare() and unsharenarrowspec() since 576eef1ab43d (narrow: move .hg/narrowspec to .hg/store/narrowspec (BC), 2018-08-02). Differential Revision: https://phab.mercurial-scm.org/D4902

py3: add 8 new passing tests to whitelist found by buildbot We are getting close! Differential Revision: https://phab.mercurial-scm.org/D4893

py3: use '%f' for floats instead of '%s' I remember Yuya saying we need to use bytestr() or '%r' because '%s' was clever. Not sure it applies to this or not. Differential Revision: https://phab.mercurial-scm.org/D4894

narrow: move adding of narrow server capabilities to core We use the experimental.narrow config option introduced in one of the previous patch and move the functionality of adding narrow server capabilities to core. Differential Revision: https://phab.mercurial-scm.org/D4891

wireprotoserver: move narrow capabilities to wireprototypes.py This is done because wireprotoserver import wireprotov1server, so you cannot import wireprotoserver in wireprotov1server to use the capabilities constants. Differential Revision: https://phab.mercurial-scm.org/D4890

narrow: introduce a config option to check if narrow is enabled or not This patch introduces a new config option experimental.narrow which is set to False by default and set to True by the narrow extension. While moving narrow related logic into core, we need to know at places whether narrow extension is enabled or not. Checking the list of extension enabled is one solution but once narrow is inbuilt, we will definitely want a config option to check whether narrow is turned on or not. So this patch introduces a config option, which will evolve to the main point to turn narrow capability on and off once all the narrow is in core. Differential Revision: https://phab.mercurial-scm.org/D4889

narrow: move the code to generate a widening bundle2 to core This is a part of moving more narrow related bits to core. Differential Revision: https://phab.mercurial-scm.org/D4888

narrow: start returning bundle2 from widen_bundle() Differential Revision: https://phab.mercurial-scm.org/D4838

narrow: the first version of narrow_widen wireprotocol command This patch introduces a wireprotocol command narrow_widen() which will be used to widen a narrow copy using `hg tracked` command provided by narrow extension. The wireprotocol command takes the old and new includes and excludes, common heads, changegroup version, known revs, and a boolean ellipses and generates a bundle2 of the required data and send it. The clients receives the bundle2 and applies that. A bundle2 instead of changegroup because in future we might want to add more things to send while widening. Thanks for martinvonz for the suggestion. I am not sure whether we need changegroup version as an argument to the command as I *think* narrow needs changegroup3 already. The tests shows that we don't exchange phase data now while widening which is nice. Also we don't check for pushkeys, rbc-cache, bookmarks etc. This does not support ellipses cases for now but will be supported in future patches. Since we send bundle2, it won't be hard to plug the ellipses logic in here. The existing code for widening a non-ellipses case is also dropped in this patch. Differential Revision: https://phab.mercurial-scm.org/D4813

remotenames: abort if literal revset pattern matches nothing This is the convention of the other namespace revsets such as tag(). Let's make the remote variants do the same.

remotenames: remove unneeded sorted() from revset implementation The order is constrained by the subset.

remotenames: don't call a set of nodes as "revs"

remotenames: use util.always instead of handcrafted lambda

remotenames: inline _parseargs() into _revsetutil() The _parseargs() function gets quite simple, and the 0/1 loop can be rewritten as "if".

repo: create changectx in a single place in localrepo.__getitem__ Differential Revision: https://phab.mercurial-scm.org/D4885

repo: remove the last few "pass" statements in localrepo.__getitem__ In case of IndexError or LookupError, we used "pass" statements and fell through to the end of localrepo.__getitem__. I find the pass statements easy to miss. Consistently raising and catching exceptions seems easier to follow. Oh -- and I didn't plan this before I wrote the above -- that probably also lets us reuse the "return context.changectx(self, rev, node)" in a later patch. Differential Revision: https://phab.mercurial-scm.org/D4884

filectx: correct docstring about "changeid" The changeid argument must be a revnum (basefile.rev() is defined as "return self._changeid"), so fix the lie in the docstring. It seems to have been incorrect for at least 10 years (I didn't check further back). Differential Revision: https://phab.mercurial-scm.org/D4881

context: drop incorrect and superfluous docstring It's been incorrect at least since 8b86acc7aa64 (context: drop support for looking up context by ambiguous changeid (API), 2018-04-28). Differential Revision: https://phab.mercurial-scm.org/D4880

remotenames: follow-up on D3639 to make revset funcs take only one arg Per the review discussion on D3639, we want this to just take one argument. That ended up simplifying the code, so I'm sharing this as a follow-up to that revision rather than editing in-flight.

remotenames: add names argument to remotenames revset This patch adds names argument to the revsets provided by the remotenames extension. The revsets are remotenames(), remotebranches() and remotebookmarks(). names can be a single names, list of names or can be empty too which means it's an optional argument. If names is/are passed, changesets which have those remotenames will be returned. If names are not passed, changesets from all the remotenames are shown. Passing an invalid remotename does not throw error. The name argument also supports pattern matching. Tests are added for the argument in tests/test-logexchange.t Differential Revision: https://phab.mercurial-scm.org/D3639

copies: add time information to the debug information

copies: add a devel debug mode to trace what copy tracing does Mercurial can spend a lot of time finding renames between two commits. Having more information about that process help to understand what makes it slow in an individual instance. (eg: many files vs 1 file, etc...)

revlog: rewrite censoring logic I was able to corrupt a revlog relatively easily with the existing censoring code. The underlying problem is that the existing code doesn't fully take delta chains into account. When copying revisions that occur after the censored revision, the delta base can refer to a censored revision. Then at read time, things blow up due to the revision data not being a compressed delta. This commit rewrites the revlog censoring code to take a higher-level approach. We now create a new revlog instance pointing at temp files. We iterate through each revision in the source revlog and insert those revisions into the new revlog, replacing the censored revision's data along the way. The new implementation isn't as efficient as the old one. This is because it will fully engage delta computation on insertion. But I don't think it matters. The new implementation is a bit hacky because it attempts to reload the revlog instance with a new revlog index/data file. This is fragile. But this is needed because the index (which could be backed by C) would have a cached copy of the old, possibly changed data and that could lead to problems accessing index or revision data later. One benefit of the new approach is that we integrate with the transaction. The old revlog is backed up and if the transaction is rolled back, the original revlog is restored. As part of this, we had to teach the transaction about the store vfs. I'm not super keen about this. But this was the easiest way to hook things up to the transaction. We /could/ just ignore the transaction like we were doing before. But any file mutation should be governed by transaction semantics, including undo during rollback. Differential Revision: https://phab.mercurial-scm.org/D4869

revlog: move loading of index data into own method This will allow us to "reload" a revlog instance from a rewritten index file, which will be used in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D4868

revlog: clear revision cache on hash verification failure The revision cache is populated after raw revision fulltext is retrieved but before hash verification. If hash verification fails, the revision cache will be populated and subsequent operations to retrieve the invalid fulltext may return the cached fulltext instead of raising. This commit changes hash verification so it will invalidate the revision cache if the cached node fails hash verification. The side-effect is that subsequent operations to request the revision text - even the raw revision text - will always fail. The new behavior is consistent and is definitely less wrong. There is an open question of whether revision(raw=True) should validate hashes. But I'm going to punt on this problem. We can always change behavior later. And to be honest, I'm not sure we should expose raw=True on the storage interface at all. Another day... Differential Revision: https://phab.mercurial-scm.org/D4867

fuzz: new fuzzer for cext/manifest.c This is a bit messy, because lazymanifest is tightly coupled to the cpython API for performance reasons. As a result, we have to build a whole Python without pymalloc (so ASAN can help us out) and link against that. Then we have to use an embedded Python interpreter. We could manually drive the lazymanifest in C from that point, but experimentally just using PyEval_EvalCode isn't really any slower so we may as well do that and write the innermost guts of the fuzzer in Python. Leak detection is currently disabled for this fuzzer because there are a few global-lifetime things in our extensions that we more or less intentionally leak and I didn't want to take the detour to work around that for now. This should not be pushed to our repo until https://github.com/google/oss-fuzz/pull/1853 is merged, as this depends on having the Python tarball around. Differential Revision: https://phab.mercurial-scm.org/D4879

revlog: rename _cache to _revisioncache "cache" is generic and revlog instances have multiple caches. Let's be descriptive about what this is a cache for. Differential Revision: https://phab.mercurial-scm.org/D4866

testing: add file storage integration for bad hashes and censoring In order to implement these tests, we need a backdoor to write data into storage backends while bypassing normal checks. We invent a callable to do that. As part of writing the tests, I found a bug with censorrevision() pretty quickly! After calling censorrevision(), attempting to access revision data for an affected node raises a cryptic error related to malformed compression. This appears to be due to the revlog not adjusting delta chains as part of censoring. I also found a bug with regards to hash verification and revision fulltext caching. Essentially, we cache the fulltext before hash verification. If we look up the fulltext after a failed hash verification, we don't get a hash verification exception. Furthermore, the behavior of revision(raw=True) can be inconsistent depending on the order of operations. I'll be fixing both these bugs in subsequent commits. Differential Revision: https://phab.mercurial-scm.org/D4865

testing: add file storage tests for getstrippoint() and strip() Differential Revision: https://phab.mercurial-scm.org/D4864

wireprotov2: always advertise raw repo requirements I'm pretty sure my original thinking behind making it conditional on stream clone support was that the behavior mirrored wire protocol version 1. I don't see a compelling reason for us to not advertise the server's storage requirements. The proper way to advertise stream clone support in wireprotov2 would be to not advertise the command(s) required to perform stream clone or to advertise a separate capability denoting stream clone support. Stream clone isn't yet implemented on wireprotov2, so we can cross this bridge later. Differential Revision: https://phab.mercurial-scm.org/D4863

tests: don't be as verbose in wireprotov2 tests I don't think that printing low-level I/O and frames is beneficial to testing command-level functionality. Protocol-level testing, yes. But command-level functionality shouldn't care about low-level details in most cases. This output makes tests more verbose and harder to read. It also makes them harder to maintain, as you need to glob over various dynamic width fields. Let's remove these low-level details from many of the wireprotov2 tests. Differential Revision: https://phab.mercurial-scm.org/D4861

repository: define and use revision flag constants Revlogs have a per-revision 2 byte field holding integer flags that define how revision data should be interpreted. For historical reasons, these integer values are sent verbatim on the wire protocol as part of changegroup data. From a semantic standpoint, the flags that go out over the wire are different from the flags stored internally by revlogs. Failure to establish this semantic distinction creates unwanted strong coupling between revlog's internals and the wire protocol. This commit establishes new constants on the repository module that define the revision flags used by the wire protocol (and by some internal storage APIs, sadly). The changegroups internals documentation has been updated to document them explicitly. Various references throughout the repo now use the repository constants instead of the revlog constants. This is done to make it clear that we're operating on generic revision data and this isn't tied to revlogs. Differential Revision: https://phab.mercurial-scm.org/D4860

context: reverse conditional branch order in introrev Positive logic will be simpler to follow. It will help to clarify coming refactoring.

context: drop a redundant fast path in introrev Now that _adjustlinkrev fast path this case itself, we no longer need an extra conditional. A nice side effect is that we are no longer calling `self.rev()`. In case where `_descendantrev` is set, calling `self.rev` will trigger a potentially expensive `_adjustlinkrev` call. So blindly calling `self.rev()` to avoid another `_adjustlinkrev` call can be counterproductive. Note that `_descendantrev` is currently never taken into account in `introrev` so far which is wrong. We'll fix that in changeset later in this series.

context: fast path linkrev adjustement in trivial case If the search starts from the linkrev, there is nothing to adjust.

url: allow to configure timeout on http connection By default, httplib.HTTPConnection opens connection with no timeout. If the server is hanging, Mercurial will wait indefinitely. This may be an issue for automated scripts. Differential Revision: https://phab.mercurial-scm.org/D4878

obsolete: explicitly track folds inside the markers We now record information to be able to recognize "fold" event from obsolescence markers. To do so, we track the following pieces of information: a) a fold ID. Unique to that fold (per successor), b) the number of predecessors, c) the index of the predecessor in that fold. We will now be able to create an algorithm able to find "predecessorssets". We now store this data in the generic "metadata" field of the markers. Updating the format to have a more compact storage for this would be useful. This way of tracking a fold through multiple markers could be applied to split too. This would have two advantages: 1) We get a simpler format, since number of successors is limited to [0-1]. 2) We can better deal with situations where only some of the split successors are pushed to a remote repository. We should look into the relevance of such a change before updating the on-disk format. note: unlike splits, folds do not have to deal with cases where only some of the markers have been synchronized. As they all share the same successor changesets, they are all relevant to the same nodes.

cleanupnodes: update comment to drop mention of filtering Since changeset 1857f50a9643 drop the filtering, we should not longer mention it in code comment.

treemanifests: remove _loadalllazy when doing copies 'before' here is https://phab.mercurial-scm.org/D4845 (not the committed/rebased version) diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.329 s +- 0.011 s | 1.320 s +- 0.010 s | 99.3% m-u | | x | 1.316 s +- 0.005 s | 1.334 s +- 0.018 s | 101.4% m-u | x | | 1.330 s +- 0.021 s | 1.322 s +- 0.005 s | 99.4% m-u | x | x | 87.2 ms +- 0.7 ms | 86.9 ms +- 1.5 ms | 99.7% l-d-r | | | 203.3 ms +- 7.8 ms | 199.4 ms +- 1.8 ms | 98.1% l-d-r | | x | 204.6 ms +- 2.8 ms | 201.7 ms +- 2.1 ms | 98.6% l-d-r | x | | 90.5 ms +- 11.0 ms | 86.2 ms +- 1.0 ms | 95.2% l-d-r | x | x | 66.3 ms +- 2.0 ms | 66.4 ms +- 0.9 ms | 100.2% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 239.4 ms +- 2.0 ms | 241.7 ms +- 4.6 ms | 101.0% m-u | | x | 128.9 ms +- 1.9 ms | 130.9 ms +- 7.7 ms | 101.6% m-u | x | | 241.1 ms +- 1.6 ms | 240.1 ms +- 1.4 ms | 99.6% m-u | x | x | 133.4 ms +- 1.5 ms | 133.4 ms +- 1.2 ms | 100.0% l-d-r | | | 84.3 ms +- 1.5 ms | 83.5 ms +- 1.0 ms | 99.1% l-d-r | | x | 200.9 ms +- 6.3 ms | 203.0 ms +- 4.4 ms | 101.0% l-d-r | x | | 108.1 ms +- 1.4 ms | 108.7 ms +- 2.1 ms | 100.6% l-d-r | x | x | 190.2 ms +- 4.8 ms | 191.6 ms +- 2.0 ms | 100.7% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.655 s +- 0.029 s | 5.640 s +- 0.036 s | 99.7% m-u | | x | 5.813 s +- 0.038 s | 5.773 s +- 0.028 s | 99.3% m-u | x | | 5.593 s +- 0.043 s | 5.589 s +- 0.028 s | 99.9% m-u | x | x | 648.2 ms +- 19.2 ms | 637.3 ms +- 27.7 ms | 98.3% l-d-r | | | 673.3 ms +- 8.0 ms | 673.2 ms +- 6.8 ms | 100.0% l-d-r | | x | 6.583 s +- 0.030 s | 5.721 s +- 0.028 s | 86.9% <-- l-d-r | x | | 277.8 ms +- 6.7 ms | 276.0 ms +- 2.7 ms | 99.4% l-d-r | x | x | 1.692 s +- 0.013 s | 720.9 ms +- 13.3 ms | 42.6% <-- status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 220.9 ms +- 1.6 ms | 219.9 ms +- 2.2 ms | 99.5% m-u | | x | 109.2 ms +- 1.0 ms | 109.4 ms +- 0.8 ms | 100.2% m-u | x | | 222.6 ms +- 1.7 ms | 221.4 ms +- 2.1 ms | 99.5% m-u | x | x | 113.4 ms +- 0.5 ms | 113.1 ms +- 1.1 ms | 99.7% l-d-r | | | 82.1 ms +- 1.7 ms | 82.1 ms +- 1.2 ms | 100.0% l-d-r | | x | 199.8 ms +- 4.0 ms | 200.7 ms +- 3.6 ms | 100.5% l-d-r | x | | 85.4 ms +- 1.5 ms | 85.2 ms +- 0.3 ms | 99.8% l-d-r | x | x | 202.6 ms +- 4.4 ms | 208.0 ms +- 4.0 ms | 102.7% status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.941 s +- 0.014 s | 1.930 s +- 0.009 s | 99.4% m-u | | x | 1.924 s +- 0.007 s | 1.950 s +- 0.010 s | 101.4% m-u | x | | 1.959 s +- 0.085 s | 1.926 s +- 0.009 s | 98.3% m-u | x | x | 96.2 ms +- 1.0 ms | 96.4 ms +- 0.7 ms | 100.2% l-d-r | | | 604.4 ms +- 10.6 ms | 602.6 ms +- 7.1 ms | 99.7% l-d-r | | x | 605.7 ms +- 4.1 ms | 607.4 ms +- 6.1 ms | 100.3% l-d-r | x | | 182.4 ms +- 1.2 ms | 183.4 ms +- 1.2 ms | 100.5% l-d-r | x | x | 150.8 ms +- 2.0 ms | 150.6 ms +- 1.0 ms | 99.9% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.185 s +- 0.027 s | 3.181 s +- 0.017 s | 99.9% m-u | | x | 3.028 s +- 0.021 s | 2.954 s +- 0.010 s | 97.6% m-u | x | | 3.168 s +- 0.010 s | 3.175 s +- 0.023 s | 100.2% m-u | x | x | 317.5 ms +- 3.5 ms | 313.2 ms +- 2.9 ms | 98.6% l-d-r | | | 456.2 ms +- 10.6 ms | 454.4 ms +- 5.8 ms | 99.6% l-d-r | | x | 9.236 s +- 0.063 s | 757.9 ms +- 9.2 ms | 8.2% <-- l-d-r | x | | 257.6 ms +- 2.3 ms | 261.2 ms +- 1.7 ms | 101.4% l-d-r | x | x | 1.614 s +- 0.013 s | 478.0 ms +- 14.3 ms | 29.6% <-- Differential Revision: https://phab.mercurial-scm.org/D4875

treemanifests: store whether a lazydirs entry needs copied after materializing Due to the way that things like manifestlog.get caches its values, without making a copy (if necessary) after calling readsubtree(), we might end up adjusting the state of the same object on different contexts, breaking things like dirty state tracking (and probably other things). Differential Revision: https://phab.mercurial-scm.org/D4874

treemanifests: extract _loaddifflazy from _diff, use in _filesnotin Differential Revision: https://phab.mercurial-scm.org/D4873

identify: show remote bookmarks in `hg id url -Tjson -B` I didn't display bookmarks when `default and not ui.quiet`: it seems strange for templates to depend on --id or -q, and it would take more code for `hg id url -T {node}` to not request remote bookmarks. An alternative I thought of was providing lazy data to the formatter, `fm.data(bookmarks=lambda: fm.formatlist(getbms(), name='bookmark'))`. The plainformatter would naturally not compute it, the templateformatter would compute only what it needs, and the other ones would compute everything, but that's not supported (or I don't see how), so I abandoned this idea. Differential Revision: https://phab.mercurial-scm.org/D4872

showstack: also handle SIGALRM This is looking *very* handy when debugging mysterious hangs in a test: you can wrap a hanging invocation in `perl -e 'alarm shift @ARGV; exec @ARGV' 1` for example, a hanging `hg pull` becomes `perl -e 'alarm shift @ARGV; exec @ARGV' 1 hg pull` where the `1` is the timeout in seconds before the process will be hit with SIGALRM. After making that edit to the test file, you can then use --extra-config-opt on run-tests.py to globaly enable showstack during the test run, so you'll get full stack traces as you force your hg to exit. I wonder (but only a little, not enough to take action just yet) if we should wire up some scaffolding in run-tests itself to automatically wrap all commands in alarm(3) somehow to avoid hangs in the future? Differential Revision: https://phab.mercurial-scm.org/D4870

exchangev2: add progress bar around manifest scanning This can take a long time on large repositories. Let's add a progress bar so we don't have long periods where it isn't obvious what is going on. Differential Revision: https://phab.mercurial-scm.org/D4859

httppeer: report http statistics Now that keepalive.py records HTTP request count and the number of bytes sent and received as part of performing those requests, we can easily print a report on the activity when closing a peer instance! Exact byte counts are globbed in tests because they are influenced by non-deterministic things, such as hostnames and port numbers. Plus, the exact byte count isn't too important anyway. I feel obliged to note that printing the byte count could have security implications. e.g. if sending a password via HTTP basic auth, the length of that password will influence the byte count and the reporting of the byte count could be a side-channel leak of the password length. I /think/ this is beyond our threshold for concern. But if we think it poses a problem, we can teach the byte count logging code to e.g. ignore sensitive HTTP request headers. We could also consider not reporting the byte count of request headers altogether. But since the wire protocol uses HTTP headers for sending command arguments, it is kind of important to report their size. Differential Revision: https://phab.mercurial-scm.org/D4858

keepalive: track number of bytes received from an HTTP response We also bubble the byte count up to the HTTPConnection instance and its parent opener at read time. Unlike sending, there isn't a clear "end of response" signal we can intercept to defer updating the accounting. Differential Revision: https://phab.mercurial-scm.org/D4857

keepalive: track request count and bytes sent I want wire protocol interactions to report the number of requests made and bytes transferred. This commit teaches the very low-level custom HTTPConnection class to track the number of bytes sent to the socket. This may vary from the number of bytes that go on the wire due to e.g. TLS. That's OK. KeepAliveHandler is taught to track the total number of requests and total number of bytes sent across all requests. Differential Revision: https://phab.mercurial-scm.org/D4856

url: have httpsconnection inherit from our custom HTTPConnection This will ensure that any customizations we perform to HTTPConnection will be available to httpsconnection. Differential Revision: https://phab.mercurial-scm.org/D4855

cborutil: change buffering strategy Profiling revealed that we were spending a lot of time on the line that was concatenating the old buffer with the incoming data when attempting to decode long byte strings, such as manifest revisions. Essentially, we were feeding N chunks of size len(X) << len(Y) into decode() and continuously allocating a new, larger buffer to hold the undecoded input. This created substantial memory churn and slowed down execution. Changing the code to aggregate pending chunks in a list until we have enough data to fully decode the next atom makes things much more efficient. I don't have exact data, but I recall the old code spending >1s on manifest fulltexts from the mozilla-unified repo. The new code doesn't significantly appear in profile output. Differential Revision: https://phab.mercurial-scm.org/D4854

cleanup: some Yoda conditions, this patch removes It seems the factor 20 is less than the frequency of " < \d" compared to " \d > ". Differential Revision: https://phab.mercurial-scm.org/D4862

streamclone: don't support stream clone unless repo feature present This change means custom repository types must opt in to enabling stream clone. This seems reasonable, as stream clones are a very low-level feature that has historically assumed the use of revlogs and the layout of .hg/ that they entail. Differential Revision: https://phab.mercurial-scm.org/D4853

localrepo: add repository feature when repo can be stream cloned Right now, the wire protocol server assumes all repository objects can be stream cloned (unless the stream clone feature is disabled via config option). But not all storage backends or repository objects may support stream clone. This commit defines a repository feature denoting whether stream clone is supported. The feature is defined for revlog-based repositories, which should currently be "all repositories." Differential Revision: https://phab.mercurial-scm.org/D4852

wireprotov2: client support for following content redirects And with the server actually sending content redirects, it is finally time to implement client support for following them! When a redirect response is seen, we wait until all data for that request has been received (it should be nearly immediate since no data is expected to follow the redirect message). Then we use a URL opener to make a request. We stuff that response into the client handler and construct a new response object to track it. When readdata() is called for servicing requests, we attempt to read data from the first redirected response. During data reading, data is processed similarly to as if it came from a frame payload. The existing test for the functionality demonstrates the client transparently following the redirect and obtaining the command response data from an alternate URL! There is still plenty of work to do here, including shoring up testing. I'm not convinced things will work in the presence of multiple redirect responses. And we don't yet implement support for integrity verification or configuring server certificates to validate the connection. But it's a start. And it should enable us to start experimenting with "real" caches. Differential Revision: https://phab.mercurial-scm.org/D4778

wireprotov2: server support for sending content redirects A "content redirect" can be sent in place of inline response content. In terms of code, we model a content redirect as a special type of response object holding the attributes describing that redirect. Sending a content redirect thus becomes as simple as the object emission layer sending an instance of that type. A cacher using externally-addressable content storage could replace the outgoing object stream with an object advertising its location. The bulk of the code in this commit is teaching the output layer which handles the object stream to recognize alternate location objects. The rules are that if an alternate location object is present, it must be the first and only object in the object stream. Otherwise the server emits an error. Differential Revision: https://phab.mercurial-scm.org/D4777

wireprotov2: client support for advertising redirect targets With the server now able to emit a redirect target descriptor, we can start to teach the client to recognize it. This commit implements support for filtering the advertised redirect targets against supported features and for advertising compatible redirect targets as part of command requests. It also adds the minimal boilerplate required to fail when a content redirect is seen. The server doesn't yet do anything with the advertised redirect targets. And the client can't yet follow redirects if it did. But at least we're putting bytes on the wire. Differential Revision: https://phab.mercurial-scm.org/D4776

wireprotov2: advertise redirect targets in capabilities This is pretty straightforward. Redirect targets will require an extension to support. So we've added a function that can be wrapped to define redirect targets. To test this, we teach our simple cache test extension to read redirect targets from a file. It's a bit hacky. But it gets the job done. Differential Revision: https://phab.mercurial-scm.org/D4775

wireprotov2: define semantics for content redirects When I implemented the clonebundles feature and deployed it on hg.mozilla.org using Amazon S3 as a content server, server-side CPU and bandwidth usage dropped off a cliff and a ton of server scaling headaches went away pretty much the instant clients with support for clonebundles were rolled out to Firefox CI. An obvious takeaway from that experience was that offloading server load to scalable file servers - potentially backed by a CDN - is a really good idea. Another takeaway was that Mercurial's wire protocol wasn't in a good position to support data offload generally. In wire protocol version 1, there isn't a mechanism in the protocol to say "grab the data from over here instead." For HTTP, we could teach the client to follow HTTP redirects. Or we could invent a media type that encoded redirects inline. But for SSH, we were pretty much out of luck because that protocol wasn't very flexible. Wire protocol version 2 offers the opportunity to do something better. The recent generic server-side content caching layer in the wire protocol version 2 server demonstrated that it is possible to have drop-in caching of responses to command requests. This by itself adds tons of value and already makes the built-in server much more scalable. But I don't want to stop there. The existing server-side caching implementation has a big weakness: it requires the server to send data to the client. This means that the Mercurial server is potentially sending gigabytes of data to thousands of clients. This is problematic because compared to scaling static file servers, scaling dynamic servers is *hard*. A solution to this is to "offload" serving of content to something that isn't the Mercurial server. By offloading content serving, you turn the Mercurial server from a centralized monolithic service to a distributed mostly-indexing service. Assuming high rates of content offload, this should drastically reduce the total work performed by the Mercurial server, both in terms of CPU and data transfer. This will make Mercurial servers vastly easier to scale. This commit defines the semantics for "content redirects" in wire protocol version 2. Essentially: * Servers advertise the set of locations a response could be served from. * When making requests, clients advertise the set of locations they are willing to fetch content from. * Servers can then replace the inline response with one that says "get the response from over here instead." This feature - when fully implemented - will allow extending the server-side caching layer to facilitate such things as integrating your server-side cache with a scalable blob store (such as S3 or a CDN) and offloading most data transfer to that external service. This feature could also be leveraged for load balancing. e.g. requests could come into a central server and then get redirected to an available mirror depending on server availability or locality. There's tons of potential :) Differential Revision: https://phab.mercurial-scm.org/D4774

wireprotov2: support response caching One of the things I've learned from managing VCS servers over the years is that they are hard to scale. It is well known that some companies have very beefy (read: very expensive) servers to power their VCS needs. It is also known that specialized servers for various VCS exist in order to facilitate scaling servers. (Mercurial is in this boat.) One of the aspects that make a VCS server hard to scale is the high CPU load incurred by constant client clone/pull operations. To alleviate the scaling pain associated with data retrieval operations, I want to integrate caching into the Mercurial wire protocol server as robustly as possible such that servers can aggressively cache responses and defer as much server load as possible. This commit represents the initial implementation of a general caching layer in wire protocol version 2. We define a new interface and behavior for a wire protocol cacher in repository.py. (This is probably where a reviewer should look first to understand what is going on.) The bulk of the added code is in wireprotov2server.py, where we define how a command can opt in to being cached and integrate caching into command dispatching. From a very high-level: * A command can declare itself as cacheable by providing a callable that can be used to derive a cache key. * At dispatch time, if a command is cacheable, we attempt to construct a cacher and use it for serving the request and/or caching the request. * The dispatch layer handles the bulk of the business logic for caching, making cachers mostly "dumb content stores." * The mechanism for invalidating cached entries (one of the harder parts about caching in general) is by varying the cache key when state changes. As such, cachers don't need to be concerned with cache invalidation. Initially, we've hooked up support for caching "manifestdata" and "filedata" commands. These are the simplest to cache, as they should be immutable over time. Caching of commands related to changeset data is a bit harder (because cache validation is impacted by changes to bookmarks, phases, etc). This will be implemented later. (Strictly speaking, censoring a file should invalidate caches. I've added an inline TODO to track this edge case.) To prove it works, this commit implements a test-only extension providing in-memory caching backed by an lrucachedict. A new test showing this extension behaving properly is added. FWIW, the cacher is ~50 lines of code, demonstrating the relative ease with which a cache can be added to a server. While the test cacher is not suitable for production workloads, just for kicks I performed a clone of just the changeset and manifest data for the mozilla-unified repository. With a fully warmed cache (of just the manifest data since changeset data is not cached), server-side CPU usage dropped from ~73s to ~28s. That's pretty significant and demonstrates the potential that response caching has on server scalability! Differential Revision: https://phab.mercurial-scm.org/D4773

wireprotov2: define type to represent pre-encoded object An upcoming commit will introduce a caching layer to command serving. This will require the ability to cache pre-encoded data. This commit introduces a type to represent pre-encoded data and teaches the output layer to not CBOR encode an instance of that type. Differential Revision: https://phab.mercurial-scm.org/D4772

wireprotov2: change name and behavior of readframe() In the near future, we will want to support performing I/O from other sources. Let's rename readframe() to readdata() and tweak its logic to support future growth. Differential Revision: https://phab.mercurial-scm.org/D4771

url: move _wraphttpresponse() from httpeer This is a generally useful function. Having it on the url module will make it more accessible outside of the HTTP peers. Differential Revision: https://phab.mercurial-scm.org/D4770

debugcommands: print all CBOR objects application/mercurial-cbor may contain multiple objects. Let's print all of them. Differential Revision: https://phab.mercurial-scm.org/D4769

help: document about "export" template keywords

help: document about "config" template keywords

help: document about "cat" template keywords

help: document about "branches" template keywords

help: document about "bookmarks" template keywords

help: document about "annotate" template keywords

storageutil: pass nodes into emitrevisions() The main emitrevisions() uses nodes. So it makes sense to use nodes for the helper API. Not bothering with API since this function was introduced a few commits ago. Differential Revision: https://phab.mercurial-scm.org/D4805

storageutil: make all callables optional Not all storage backends may implement these callables. That's part of the reason these methods aren't exposed on the storage interface. Differential Revision: https://phab.mercurial-scm.org/D4804

storageutil: extract most of emitrevisions() to standalone function As part of implementing a storage backend, I found myself copying most of revlog.emitrevisions(). This code is highly nuanced and it bothered me greatly to be copying such low-level code. This commit extracts the bulk of revlog.emitrevisions() into a new standalone function. In order to make the function generally usable, all "self" function calls that aren't exposed on the ifilestorage interface are passed in via callable arguments. No meaningful behavior should have changed as part of the port. Upcoming commits will tweak behavior to make the code more generically usable. Differential Revision: https://phab.mercurial-scm.org/D4803

storageutil: invert logic of file data comparison IMO things make more sense when the function is explicitly a test for file data equivalence. Not bothering with API since the function was introduced by the previous commit. Differential Revision: https://phab.mercurial-scm.org/D4802

storageutil: extract filelog.cmp() to a standalone function As part of implementing an alternate storage backend, I found myself reimplementing this code. With a little massaging, we can extract filelog.cmp() to a standalone function. As part of this, the call to revlog.cmp() was inlined (it is just a 2-line function). I also tweaked some variable names to improve readability. I'll further tweak names in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D4801

storageutil: extract copy metadata retrieval out of filelog As part of implementing an alternate storage backend, I found myself reinventing this wheel. Let's create a utility function for doing the work. Differential Revision: https://phab.mercurial-scm.org/D4800

storageutil: extract functionality for resolving strip revisions I found myself having to copy this method as part of implementing a new storage backend. With a little tweaking, we can extract it to a standalone function so it can be reused easily. We'll likely want to implement a better method for removing revisions on the storage interface someday. But until then, let's use what we have. Differential Revision: https://phab.mercurial-scm.org/D4799

storageutil: consistently raise LookupError (API) The interface docs say this is supposed to raise LookupError on failure. But for invalid revision number input, it could raise IndexError because ifileindex.node() is documented to raise IndexError. lookup() for files isn't used that much (pretty much just in basefilectx in core AFAICT). And callers should already be catching LookupError. So I don't anticipate that much fallout from this change. Differential Revision: https://phab.mercurial-scm.org/D4798

storageutil: implement file identifier resolution method (BC) revlog.lookup() has a number of advanced features, including partial node matching. These advanced features aren't needed for file id lookup because file identifiers are almost always from internal sources. (An exception to this is hgweb, which appears to have some code paths that attempt to resolve a user-supplied string to a node.) This commit implements a new function for resolving file identifiers to nodes. It behaves like revlog.lookup() but without the advanced features. Tests reveal behavior changes: * Partial hex nodes are no longer resolved to nodes. * "-1" now returns nullid instead of raising LookupError. * "0" on an empty store now raises LookupError instead of returning nullid. I'm not sure why "-1" wasn't recognized before. But it seems reasonable to accept it as a value since integer -1 resolves to nullid. These changes all seem reasonable to me. And with the exception of partial hex node matching, we may want to consider changing revlog.lookup() as well. Differential Revision: https://phab.mercurial-scm.org/D4797

testing: add more testing for ifileindex.lookup() The tests demonstrate some... questionable behavior of revlog.lookup(). Differential Revision: https://phab.mercurial-scm.org/D4796

dagop: extract DAG local heads functionality from revlog As part of implementing an alternate storage backend, I found myself having to reimplement this function. The DAG traversal logic is generic and can be factored out into a standalone function. We could probably combine this with headrevs(). But I'll leave that for another time. Differential Revision: https://phab.mercurial-scm.org/D4795

dagop: extract descendants() from revlog module This method needs to be implemented in other storage backends and is generic if we parameterize the functions for retrieving revision numbers and parent revision numbers. Let's extract it to dagop so backends don't need to reinvent the wheel. I'm not too worried about performance overhead of the extra function call because I don't expect anyone to be calling descendants() in a tight loop. Differential Revision: https://phab.mercurial-scm.org/D4794

filelog: remove checkhash() (API) It is unused. While a caller may want to ask the store to validate the hash of some provided text, since there are no in-core consumers of this method, let's drop it for now. Differential Revision: https://phab.mercurial-scm.org/D4793

filelog: remove revdiff() (API) This proxy method is no longer used. While it might be useful to query a storage backend for the delta between any 2 revisions because the store could have a delta cached and could compute it more efficiently than the caller calling revision() twice in order to compute a delta, since nothing in core is using this API now, I feel comfortable nuking it. Differential Revision: https://phab.mercurial-scm.org/D4792

localrepo: define storage backend in creation options (API) We add an experimental config option to define the storage backend for new repositories. By default, it uses "revlogv1," which maps to the current and only modern supported repository format. We add a "backend" creation option to control which backend to use. It defaults to using the value from the config option. newreporequirements() will now barf if it sees a "backend" value that isn't "revlogv1." This forces extensions to monkeypatch the function to handle requirements derivation for custom backends. In order for this to "just work," we factored out obtaining the default creation options into its own function and made callers of newreporequirements() responsible for passing in valid data. Without this, direct callers of newreporequirements() wouldn't get the proper results. Differential Revision: https://phab.mercurial-scm.org/D4791

wireprotov2: derive "required" from presence of default value If we define a default value for all optional arguments, we no longer need to explicitly declare whether the argument is required. Instead, we can derive it from the presence of a default value callable. Differential Revision: https://phab.mercurial-scm.org/D4790

localrepo: capture repo interface factory functions as lambas Previously, the list would hold a reference to the original function and attempting to wrap the module function via extension wouldn't result in the wrapped function being called, completely undermining the expected behavior. Differential Revision: https://phab.mercurial-scm.org/D4821

extensions: new closehead module for closing arbitrary heads ``hg close-head`` allows closing arbitrary heads. It is equivalent to checking out the given revisions and commit an empty change with ``hg commit --close-branch``. Differential Revision: https://phab.mercurial-scm.org/D3557

cext: use modern buffer protocol in mpatch_flist() Differential Revision: https://phab.mercurial-scm.org/D4841

cext: use modern buffer protocol in patches() PyObject_AsCharBuffer() is part of the "Old Buffer Protocol," which has been deprecated for years. Let's port away from it. PyBuffer_GetBuffer() must be paired with PyBuffer_Release(), hence the added "goto cleanup" in a failure case. We don't bump the extension version because the API has not changed. Differential Revision: https://phab.mercurial-scm.org/D4840

identify: when using -T, avoid unnecessary remote bookmarks query Differential Revision: https://phab.mercurial-scm.org/D4839

identify: only query remote bookmarks if needed Instead of all the time when operating on a remote repo. This perf regression was introduced in 15a79ac823e8, in 4.3. This datahint method returns nothing for -Tjson, -Tpickle, -Tdebug --config ui.formatdebug=true and --config ui.formatjson, so the bookmarks won't show up. I don't know what these formatters are for. plainformatter and templateformatter work properly, and the few other uses of datahint should have the same kind of problem. There is further weirdness where "--template '{node}'" is not enough to avoid querying the bookmarks, you also need to pass --id or -q. Differential Revision: https://phab.mercurial-scm.org/D4819

py3: whitelist another passing tests caught by buildbot I this indygreg recent patches made this test pass on Python 3. So thanks to him! Differential Revision: https://phab.mercurial-scm.org/D4848

manifest: remove an unused variable caught by pyflakes Uses of todel were removed in D4843 (19103e68a698). This patch will make buildbots green again. Differential Revision: https://phab.mercurial-scm.org/D4847

setup: ignore message about disabling 3rd party extensions because of version I started getting into a bind recently when switching between py2 and py3 because switching requires a `make clean`, which kills __version__.py. But then when running `make local`, it picks up the local hg.exe (MSYS seems to prefix $PATH with '.'), which doesn't know its version. That causes it to emit a warning about needing at least 4.3 to load evolve, which caused setup.py to fail saying there is no working hg executable to figure out the version. If we can ignore general extension import failures, we should be able to ignore this too.

narrow: avoid overwriting a variable I don't like to overwrite variables, especially not with a different type (because it makes it harder to explain what the variable is for, and, therefore, to name it well). Differential Revision: https://phab.mercurial-scm.org/D4846

treemanifests: remove _loadalllazy in _diff() The benchmarks below use a similar setup as in ee7ee0c516ca and my other recent commits. Yes, in some cases this runs in literally 5% of the time it previously took. before = a6f8ab53 diff --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.301 s +- 0.004 s | 1.309 s +- 0.012 s | 100.6% m-u | | x | 1.303 s +- 0.009 s | 1.302 s +- 0.006 s | 99.9% m-u | x | | 1.308 s +- 0.006 s | 1.309 s +- 0.007 s | 100.1% m-u | x | x | 85.7 ms +- 0.6 ms | 86.0 ms +- 0.3 ms | 100.4% l-d-r | | | 197.5 ms +- 0.7 ms | 197.8 ms +- 2.2 ms | 100.2% l-d-r | | x | 199.4 ms +- 0.6 ms | 199.3 ms +- 0.9 ms | 99.9% l-d-r | x | | 86.1 ms +- 0.5 ms | 85.8 ms +- 0.9 ms | 99.7% l-d-r | x | x | 64.4 ms +- 0.4 ms | 64.4 ms +- 0.3 ms | 100.0% diff -c . --git: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 236.7 ms +- 1.1 ms | 236.5 ms +- 1.3 ms | 99.9% m-u | | x | 158.7 ms +- 1.0 ms | 128.0 ms +- 1.0 ms | 80.7% <-- m-u | x | | 239.7 ms +- 1.8 ms | 238.1 ms +- 1.5 ms | 99.3% m-u | x | x | 132.4 ms +- 0.9 ms | 132.3 ms +- 0.6 ms | 99.9% l-d-r | | | 81.8 ms +- 0.4 ms | 81.8 ms +- 0.3 ms | 100.0% l-d-r | | x | 3.894 s +- 0.017 s | 193.6 ms +- 0.4 ms | 5.0% <-- l-d-r | x | | 106.9 ms +- 0.4 ms | 106.6 ms +- 0.3 ms | 99.7% l-d-r | x | x | 182.7 ms +- 0.8 ms | 183.3 ms +- 0.9 ms | 100.3% rebase -r . --keep -d .^^: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 5.615 s +- 0.109 s | 5.562 s +- 0.015 s | 99.1% m-u | | x | 5.701 s +- 0.027 s | 5.715 s +- 0.023 s | 100.2% m-u | x | | 5.572 s +- 0.128 s | 5.613 s +- 0.182 s | 100.7% m-u | x | x | 633.3 ms +- 28.7 ms | 636.2 ms +- 13.8 ms | 100.5% l-d-r | | | 666.4 ms +- 17.0 ms | 658.5 ms +- 9.3 ms | 98.8% l-d-r | | x | 6.520 s +- 0.070 s | 6.505 s +- 0.026 s | 99.8% l-d-r | x | | 279.0 ms +- 13.0 ms | 276.5 ms +- 4.7 ms | 99.1% l-d-r | x | x | 1.636 s +- 0.058 s | 1.657 s +- 0.014 s | 101.3% status --change . --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 218.6 ms +- 1.4 ms | 217.9 ms +- 1.5 ms | 99.7% m-u | | x | 138.5 ms +- 0.4 ms | 108.4 ms +- 0.2 ms | 78.3% <-- m-u | x | | 220.1 ms +- 1.3 ms | 219.7 ms +- 1.5 ms | 99.8% m-u | x | x | 113.2 ms +- 0.4 ms | 112.4 ms +- 0.8 ms | 99.3% l-d-r | | | 80.2 ms +- 0.3 ms | 80.6 ms +- 0.6 ms | 100.5% l-d-r | | x | 3.899 s +- 0.020 s | 194.8 ms +- 4.0 ms | 5.0% <-- l-d-r | x | | 83.4 ms +- 0.8 ms | 83.2 ms +- 0.2 ms | 99.8% l-d-r | x | x | 732.2 ms +- 4.3 ms | 194.9 ms +- 1.0 ms | 26.6% <-- status --copies: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 1.917 s +- 0.005 s | 1.914 s +- 0.004 s | 99.8% m-u | | x | 1.909 s +- 0.012 s | 1.934 s +- 0.004 s | 101.3% m-u | x | | 1.915 s +- 0.005 s | 1.904 s +- 0.004 s | 99.4% m-u | x | x | 94.8 ms +- 0.3 ms | 94.7 ms +- 0.2 ms | 99.9% l-d-r | | | 593.9 ms +- 1.2 ms | 594.6 ms +- 9.4 ms | 100.1% l-d-r | | x | 595.2 ms +- 3.8 ms | 597.2 ms +- 2.6 ms | 100.3% l-d-r | x | | 182.5 ms +- 1.6 ms | 182.1 ms +- 0.6 ms | 99.8% l-d-r | x | x | 149.6 ms +- 0.9 ms | 149.1 ms +- 0.8 ms | 99.7% update $rev^; ~/src/hg/hg{hg}/hg update $rev: repo | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before ------+---+---+------------------------+-----------------------+------------ m-u | | | 3.121 s +- 0.007 s | 3.129 s +- 0.012 s | 100.3% m-u | | x | 2.972 s +- 0.011 s | 2.981 s +- 0.012 s | 100.3% m-u | x | | 3.144 s +- 0.014 s | 3.141 s +- 0.011 s | 99.9% m-u | x | x | 312.2 ms +- 2.4 ms | 312.3 ms +- 2.1 ms | 100.0% l-d-r | | | 444.4 ms +- 4.3 ms | 446.9 ms +- 5.3 ms | 100.6% l-d-r | | x | 9.159 s +- 0.069 s | 9.182 s +- 0.040 s | 100.3% l-d-r | x | | 254.6 ms +- 1.6 ms | 255.2 ms +- 1.6 ms | 100.2% l-d-r | x | x | 1.525 s +- 0.007 s | 1.577 s +- 0.007 s | 103.4% <--? Differential Revision: https://phab.mercurial-scm.org/D4845

treemanifests: skip extraneous check for item before calling _loadlazy Differential Revision: https://phab.mercurial-scm.org/D4844

treemanifests: make _loadchildrensetlazy just call _loadlazy This was suggested on the original review for _loadchildrensetlazy (D4370), it's taken me a while to get to it though. Differential Revision: https://phab.mercurial-scm.org/D4843

treemanifests: make _loadlazy tolerate item not on _lazydirs I'd like to clean up a few cases where we check for an item in _lazydirs before calling _loadlazy - this will remove an extraneous dict lookup and make it slightly more versatile. Differential Revision: https://phab.mercurial-scm.org/D4842

debugcommands: add a debugindexstats command Someone went through the trouble of recording these stats, so let's make them accessible. Differential Revision: https://phab.mercurial-scm.org/D4832

rust-chg: add main program It can at least connect to a *running* command server, and execute Mercurial command. No signal handling nor daemon management is implemented yet.

rust-chg: add interface to chdir the server

rust-chg: add Client extensions to run cHg-specific requests This just provides a nicer way to issue command-server requests.

rust-chg: port basic socket path handling from cHg of C This is basically modeled after setcmdserveropts() of chg.c.

rust-chg: add state machine to handle "runcommand" request with cHg extension This is modeled after tokio-hglib's RunCommand state to support the "S" channel message.

rust-chg: add callback to handle pager and shell command requests This could be inlined into the ChgRunCommand state to be introduced by the next patch, but it seemed good to separate any user interactions from the IPC code.

rust-chg: add low-level function to set pager fd blocking This is necessary because the server expects stdout/stderr to be blocking, whereas we'll use async library to spawn pager, which makes pipes unblocking.

rust-chg: add future that handles "attachio" request This is the sequence to send client-side stdio and pager stdin to the server.

rust-chg: add parser for request messages sent to "S" channel The data structure is documented at chgserver.py:channeledsystem().

rust-chg: add wrapper around C function

rust-chg: add function to send fds via domain socket As a beginning, I wrote some C. It's extracted from attachio() of contrib/chg/hgclient.c. Maybe it could be rewritten in Rust by using the libc (and/or nix) crates, but doing that wouldn't be trivial as the code depends on CMSG_*() macros. IMO, using C is better here.

rust-chg: update dependencies

rust-chg: add project skeleton This directory will host the reimplementation of cHg in Rust. It will use Tokio [1], and tokio-hglib [2] which I wrote for an oxidized CHg, no idea if there's such carbon bonding in nature btw. [1]: https://tokio.rs/ [2]: https://bitbucket.org/yuja/tokio-hglib/ The reasoning for depending on Tokio is that it will allow us to handle Unix signals in a safer way. Well, I believed that until I found a weird function, handlestopsignal(), in cHg codebase. It resends the same signal to the same process by temporarily masking the handler, which can't be inherently async. So the signal handlers will stay in C, which means there isn't actually much reason to write async code right now, other than I've already done most of the async stuff, and slightly easier pager handling. The reasoning for the rewrite is that it will eventually be possible to port server-side config validation back to the client side, which will reduce the complexity of the current daemon management. It will also encourage us to write frontend library (e.g. command line and config parsers) in Rust. The license is GPL2+ because it's likely to include derived work from the cHg of C. The rust/chg crate is excluded from the root workspace as it's unclear how the whole rust packages should be laid out. That can be revisited later.

narrow: move copies overrides to core The copies overrides seems to have been a little complicated just by not being in core. When moved to core, it becomes trivial (at least I think these overrides have the same effect). Differential Revision: https://phab.mercurial-scm.org/D4825

narrow: pass old includes and excludes to _widen() In future patches we will need to pass them in the widen wireprotocol command which we are building. Differential Revision: https://phab.mercurial-scm.org/D4812

narrow: check for servers' narrow support before doing anything (BC) Recently we introduced narrow capabilities for the server. So we can check whether a server has narrow clone support or not before doing anything. This is BC because new clients won't be able to extend from old narrow-enabled servers. I *think* narrow is not used much (maybe just inside Google), also it's experimental so I think we can change this. We will need to this someday anyway. The "doesn't" in error is changed to "does not" because I think that's we do in core. I also changed one more instance of the error message to use 'does not' for consistency. Differential Revision: https://phab.mercurial-scm.org/D4789

narrow: don't do the dirstate dance if ellipses is not enabled I believe we set dirstate parents to nullid before widening pull because in ellipses cases, the parent might be stripped off with a new changeset. However the second ds.setparents() call invalidate my assumption. I am not sure why we do this. So here is a patch. This patch also adds tests showing we break nothing in non-ellipses cases. Differential Revision: https://phab.mercurial-scm.org/D4788

narrow: pass 'narrow_widen' as source while generating changegroup Extensions inspect the `source` parameter and add some custom logic on the basis of value of that parameter. The parameter is also passed to hooks. So let's pass the value as 'narrow_widen' to differentiate widening from pull. Differential Revision: https://phab.mercurial-scm.org/D4787

narrow: factor out logic to create cg while widening into separate fn This patch takes out the logic which generates a changegroup for widening a narrow clone when ellipses are disabled. This is done because future patches will introduce a narrow_widen() wireprotocol command which will send a bundle2 with changegroup and will use this function. The new function for now returns just the changegroup for compatibility with existing code, but in future patches, when we establish a wireprotocol command and call this function from there, this will return the required bundle2. Differential Revision: https://phab.mercurial-scm.org/D4786

narrow: avoid looking up dirstate again when editing dirstate The narrow extension overrides the dirstate editing functions to restrict paths outside the narrowspec. These overrides access the dirstate from repo.dirstate instead of using the "self" reference passed to the overridden functions. I don't see a reason for this and it caused me problems with a later patch (it caused infinite recursion when I modified localrepo.dirstate()), so let's change it. Differential Revision: https://phab.mercurial-scm.org/D4829

repo: move unfiltered-repo optimization to workingctx localrepo.__getitem__ special-cased lookup of the working copy parent to avoid looking up obsmarkers. I think the reason for that code (which I once wrote myself) was to make `hg commit` not load obsmarkers, which it would otherwise do via ctx.p1() in localrepo.commitctx(). That had the somewhat unfortunate consequence of making lookup of an unrelated binary nodeid load the dirstate. Now that changectx's constructor is dumb, we can let workingctx._parents() have the opmtimization instead. This affects two tests, because they no longer end up loading the dirstate and their "warning: ignoring unknown working parent ..." messages therefore go way. Differential Revision: https://phab.mercurial-scm.org/D4828

context: move logic from changectx.__init__ to localrepo.__getitem__ (API) My motivation for this change was to make repo[node] not load the dirstate (more about that in the next patch), but I think it makes more sense this way too. For example, raising RepoLookupError seems to belong better in the repo lookup function (i.e. localrepo.__getitem__). This makes the changectx constructor very simple -- it just assigns the given repo, revnum, and nodeid to properties. Differential Revision: https://phab.mercurial-scm.org/D4827

context: reduce dependence of changectx constructor I want to change the constructor's signature and letting all creation of changectx instances go through the repo simplifies that. Differential Revision: https://phab.mercurial-scm.org/D4826

py3: use system strings for HTTP response header comparison res.headers is holding system strings. Attempting to compare against bytes always fails and we fail to print decoded CBOR responses on Python 3. After this change, various test-wireproto* tests are now properly printing CBOR response objects. Differential Revision: https://phab.mercurial-scm.org/D4835

py3: convert HTTP request headers to str The low-level request object ideally takes system strings for HTTP request headers and values. If we send in bytes, it works. But a duplicate header check fails and various tests emit duplicate user-agent headers. Differential Revision: https://phab.mercurial-scm.org/D4834

py3: use system strings in HTTP server code Previously the source transformer was converting some string literals to bytes and we were comparing a system native string to bytes and this was leading to sending the wrong HTTP response headers on Python 3. After this change, we now properly send Transfer-Encoding and Connection response headers on Python 3. Differential Revision: https://phab.mercurial-scm.org/D4833

py3: byteify test-storage.py The test now passed on Python 3. # skip-blame just b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4836

narrow: remove narrowpatch instead of narrowcopies 84092edd5c88 removed narrowpatch.py and wanted to drop narrowpatch from __init__.py but mistakenly removed narrowcopies. This will make buildbots green again. Differential Revision: https://phab.mercurial-scm.org/D4837

py3: suppress the output from .write() calls in more tests I missed these in 803b7569c9ea because I forgot about `...` lines.

py3: byteify test-lfs.t

lfs: explicitly name a key when sorting blob pointers This is needed to keep py3 happy. The other two instances of sorting already did this.

py3: byteify test-lock.py There's still a stacktrace about not being able to serialize _io.TextIOWrapper.

py3: byteify contrib/dumprevlog

py3: quote $PYTHON in test-merge-symlinks.t

copies: inline a variable that's used only once By inlining it, we also avoid calculating the value when "if of in seen" is false (probably not significant). Differential Revision: https://phab.mercurial-scm.org/D4831

bundlerepo: remove a variable alias "parentrepo" and "repo" were the same thing and I don't see much reason for it (unionrepo has similar structure and a similar alias but there are two repos there so at least it makes a little more sense there). Differential Revision: https://phab.mercurial-scm.org/D4830

merge with stable

narrow: move support for `hg verify` into core Differential Revision: https://phab.mercurial-scm.org/D4824

narrow: drop unnecessary overrides of patch The patch overrides became unnecessary when I made context.status() filter by the narrowspec in e411774a2e0f (narrow: move status-filtering to core and to ctx, 2018-08-02). Differential Revision: https://phab.mercurial-scm.org/D4823

tests: add test showing that paths outside narrowspec are not added We had a test that shows that trying to explicitly add a file outside the narrowspec results in an error, but we didn't have a test that shows that paths from subdirectories outside the narrowspec are not added (on e.g. `hg add .`) but silently ignored. Differential Revision: https://phab.mercurial-scm.org/D4822

catapipe: fix bug introduced when switching to the timer api Differential Revision: https://phab.mercurial-scm.org/D4818

narrow: move the ellipses server capability to core This will be used in core logic to determining whether a server is ellipses enabled or not. And also this will ease moving narrow related things to core. Differential Revision: https://phab.mercurial-scm.org/D4809

narrow: move the wireprotocol narrow capability name to core We are trying to integrate the whole of narrow logic into core and it will be helpful for upcoming patches to have these capability names in core. The next patch will move the ellipses capability to core also. The exact motivation is to know whether we are cloning a ellipses repo or not and adding an ellipses repo requirement. Differential Revision: https://phab.mercurial-scm.org/D4808

help: document about "files" template keywords

formatter: remove experimental marker from -T option I think it's good enough to stabilize the feature, at least for the core and non-experimental commands.

grep: rename {line_number} to {lineno} as well (BC)

annotate: rename {line_number} to {lineno} (BC) I think {lineno} looks more like a common template keyword. It isn't called a {line} to avoid conflicts with the element name of {lines} and the {_|splitlines} filter. https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary

resolve: rename {status} to {mergestatus} to not shadow change status (BC) This is a part of the name unification. I think it's somewhat useful to provide a combined view of change/merge statuses. https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Dictionary

py3: whitelist one additional test Differential Revision: https://phab.mercurial-scm.org/D4817

logtoprocess: connect all fds to /dev/null to avoid bad interaction with pager We detected that pager is waiting for log-to-process script to finish, which is annoying when adding a script on commandfinish that does an HTTP push. There seems to be no workaround on the script side and it will make the behavior on Linux/MacOS closer to the Windows behavior. The drawback is that it makes the related tests more flaky as log-to-process outputs are now really asynchronous. If it's considered a BC change, another option would be to add a config option for this new behavior. I personally think that the different behavior between Windows and Linux is confusing and that it's a bug I would be fine with a new config option. Differential Revision: https://phab.mercurial-scm.org/D4816

logtoprocess: add a test to show pager and ltp bad interaction The next changeset will fix the bad interaction between the pager and log-to- process. Add a test first to make the next changeset easier to understand. Differential Revision: https://phab.mercurial-scm.org/D4815

py3: add one more passing test to whitelist caught by buildbot Thanks to Matt Harbison who fixed the remaining failures of this test. Differential Revision: https://phab.mercurial-scm.org/D4814

cleanupnodes: pass multiple predecessors to `createmarkers` directly

obsolete: allow multiple predecessors in createmarkers The logic for this change is similar to the change to `cleanupnodes` that we did earlier. Now that the rebase code is trying to record a fold, we need to actually record it in the markers. The first step is to have the markers creation API able to receive such fold data. To keep things sane, we restrict fold to on successors.

obsolete: preindent code in createmarkers This will help to make the next change clearer.

obsolete: explicitly pass relation items to effectflag computation To track folds, we are about to change the content of `rel`. To simplify this change, we update the `geteffectflag` function beforehand.

rebase: explicitly track collapses as fold A collapse is a large fold. It is now explicitly tracked (at the cleanupnodes API level).

rebase: use tuple as `replacement` keys Now that `cleanupnodes` support tuples as key, we update the rebase code to use them. No changes in the replacement tracked are introduced yet.

rebase: expand a long "one-liner" When a one-liner gets 3 lines longs, it lose its expressivity benefits. We expand it into a simple for loop. This makes future changes of the code in that area clearer.

cleanupnodes: drop special casing around prune markers (API) The `cleanupnodes` has logic to skip the creation of "prune" markers if the changeset is already obsolete. This feels strange and gets in the way of code changes to tracks folds. Now that callers no longer request such prune, we can drop this logic. In many cases, pruning through cleanupnodes should be replaced by internal phase usage.

rebase: don't try to prune obsolete changeset already in the destination With similar motivations to the previous changesets, we stop marking changeset from pruning when it is not the command intention. In this case, we still need to distinguish between the strip and the obsolete case.

histedit: don't cleanup nodes already disposed of If something else took care of these temporary nodes, we don't need to do anything about it. This less liberal usage of pruning through cleanup nodes will help us further cleanup on the road to explicitly tracks folds.

py3: avoid b'' output in test-hgweb-non-interactive.t I couldn't figure out how to get the list to print without b'' with pycompat.write(), without converted each element to str.

py3: convert print to a function call in a few tests I *think* this is the last of them.

py3: use util.forcebytestr() to convert push lock error to bytes

py3: byteify windows.shelltocmdexe() This makes test-doctest.py happy on Windows.

run-tests: flush output stream before prompting to accept changes With py3 on Windows, the prompt to accept changes in an --interactive run wasn't showing up until after the choice was read from stdin.

py3: use util.forcebytestr() to convert error messages to bytes This makes the python 3 buildbot green again. Differential Revision: https://phab.mercurial-scm.org/D4811

context: stop catching TypeError when converting hex nodeid to binary It has been a programming error to pass a 40-character string that is not a hex nodeid since 8b86acc7aa64 (context: drop support for looking up context by ambiguous changeid (API), 2018-04-28), so we can just let it raise a TypeError. Differential Revision: https://phab.mercurial-scm.org/D4807

context: stop catching and re-raising FilteredRepoLookupError FilteredRepoLookupError is only raised by changectx's constructor and the higher-level scmutil.revsymbol(), so there's no need to catch it in changectx's constructor. Differential Revision: https://phab.mercurial-scm.org/D4806

util: use a context manager in readlock()

py3: convert os.readlink() path to native strings on Windows Windows insisted that it needs to be str. I skipped the stuff in the posix module, and left `tests/f` and `run-tests.py` alone for now.

py3: suppress the output from .write() calls in the remaining tests

py3: conditionalize json float precision difference in test-debugcommands.t

py3: byteify extension in test-debugcommands.t

pullreport: rev duplicated and extinct into account If we already have some obsolete and hidden nodes locally and the server send them again to you, it seems useful to point it out instead of being silent about it.

pullreport: issue a message about "extinct" pulled changesets Changeset pulled from a remote repository while already obsolete locally can end up hidden after the pull. Hiding obsolete changesets is a good behavior but silently "skipping" some of the pulled content can get confusing. We now detect this situation and emit a message about it. The message is simple and the wording could be improved, however, we focus on the detection here. Evolution is still an experimental feature, so the output is open to changes. In particular, we could point out at the latest successors of the obsolete changesets, however, it can get tricky is there are many of them. So we delay these improvements to another adventure. Another easy improvement would be to merge this message with the previous line about the new nodes and their phases. This is a good example of cases where we can only transmit a limited amount of data to users by default. We need some sort of "transaction journal" we could point the user to.

pullreport: skip or rework some early return We are about to add more logic in this report. Before that, we need it to not quit so early.

pullreport: skip filtered revs instead of obsolete ones Obsolescence is closely related to visibility but still a distinct concept. We can receive changesets that are obsolete but visible (eg: when pulling orphans). Such changeset should be reported too. In addition, the filtering level can be anything, we should respect it.

pullreport: add a test to show misreporting of visible changeset The current code ignores all obsolete changesets including the visible one. We add a test showing this behavior before fixing the behavior.

repo: don't look up context for tip node if it's not needed We were doing repo['tip'].node() or similar in a few places where repo.changelog.tip() would be enough. Differential Revision: https://phab.mercurial-scm.org/D4781

repo: look up nullrev context by revnum, not symbolic name I think lookup of the 'null' symbol should be done via scmutil.revsymbol() and repo['null'] is only supported for historical reasons. However, repo[nullrev] is fine, so we can switch to that instead of switching to scmutil.revsymbol('null'). Differential Revision: https://phab.mercurial-scm.org/D4780

bundle: consistently put revnums in "base" collection The "base" collection contains revnums, except that it can also contain the "null" symbol. That's a little weird. Let's be consistent. Differential Revision: https://phab.mercurial-scm.org/D4779

py3: ensure printing to stdout uses str in test-hgweb-no-request-uri.t

scmutil: accept multiple predecessors in 'replacements' (API) This changeset makes 'cleanupnodes' accepts multiple predecessors as `replacements` keys. The same as it accepts multiple successors as `replacements` values. To avoid breaking all callers, the old and new ways are currently valid at the same time. We'll deprecate and drop the old way later. This change is the first step toward a better tracking of "fold" event in the evolution history. While working on the "rewind" command (in the evolve extension), we realized that first class tracking of folds are necessary. We already have good tracking of splits. When walking the evolution history from predecessors to successors, that makes for a clear distinction between having multiple successors because of the actual splitting of a changeset or content-divergences. The "rewind" command allows restoring older evolution of a stack of changesets. One of its mode walks the evolution history to automatically find appropriate predecessors. This means walking from successors to predecessors. In this case, we need to be able to make the same distinction between an actual fold and other cases. So we will have to track folds explicitly. This changesets only focus on making it possible to express fold at the `cleanupnodes` API level. The actual tracking will be implemented later.

scmutil: expand long "one-liner" When a one-liner gets 3 lines longs, it loses its expressivity benefits. We expand it into a simple for loop. This makes future changes of the code in that area clearer.

shelve: find shelvedctx from bundle even if they are already in the repo We use the new "duplicates" node tracking to find the tip of the bundle even if it already exists in the repository. Such logic is not supposed to be needed in theory. If the shelve was made using internal-phase, we already know its node. Otherwise, the bundle content should have been stripped. However, handling it makes the shelve code more robust and provide a good example of "revduplicates" usage.

shelve: return the shelved node as part of bundle application It make sense to have the function in charge of unbundling the shelved revision also return the node of that revision (when the data is in the bundle). This will help us to handle unnatural state where the unshelved change already exists in the repository.

changelog: keep track of duplicated node in the transaction adding them The transaction is already tracking the new nodes. We now tracks the "duplicates" in the same location.

revlog: add a callback "tracking" duplicate node addition If a changegroup contains node already added to the repository, they will be skipped. Skipping them is the right behavior (we don't need to store things twice), but it can hide some information to the code doing the unbundle (eg: shelve looking for the tip of the bundle). The first step to improve this situation is to add a low level callback. We do not need this tracking on all revlog, so actual tracking will be added in the next changeset.

logtoprocess: define $HG for children processes So they can compute the hg version for instance. Differential Revision: https://phab.mercurial-scm.org/D4768

py3: mask out None type when printing in `debuglocks` Apparently, %b doesn't allow None.

py3: ensure standard exceptions use `str` type strings in windows.py See also edaa40dc5fe5.

py3: replace a StandardError reference This doesn't exist on py3, and the standard way of handling this seems to be to catch both exceptions.

storageutil: extract revision number iteration This code is a bit quirky (and possibly buggy). It will likely be used by multiple storage backends. Let's extract it so it is reusable. Differential Revision: https://phab.mercurial-scm.org/D4757

storageutil: new function for extracting metadata-less content from text Other storage backends will want to do this. I'm not concerned about Python function call overhead because I expect self.revision() to dwarf the function call overhead time, since self.revision() requires multiple function calls and may involve decompression in the common case. Differential Revision: https://phab.mercurial-scm.org/D4756

storageutil: move _censoredtext() from revlog This seems like generic functionality we'll want to use from non-revlog storage backends. Differential Revision: https://phab.mercurial-scm.org/D4755

storageutil: move metadata parsing and packing from revlog (API) Parsing and writing of revision text metadata is likely identical across storage backends. Let's move the code out of revlog so we don't need to import the revlog module in order to use it. Differential Revision: https://phab.mercurial-scm.org/D4754

storageutil: new module for storage primitives (API) There will exist common code between storage backends. It would be nice to have a central place to put that code. This commit attempts to create that place by creating the "storageutil" module. The first thing we move is revlog.hash(), which is the function for computing the SHA-1 hash of revision fulltext and parents. Differential Revision: https://phab.mercurial-scm.org/D4753

filelog: stop proxying deltaparent() (API) deltaparent() obtains the revision number of the base revision a delta in storage is stored against. It is highly revlog-centric and may not apply to other storage backends. As a result, it doesn't belong on the generic file storage interface. This method/proxy is no longer used in core. The last consumer was probably changegroup code and went away with the transition to emitrevisions(). Differential Revision: https://phab.mercurial-scm.org/D4751

filelog: stop proxying rawsize() (API) This method is no longer used by external consumers. The API is quite low-level and is effectively len(revision(raw=True)). I don't see a compelling reason to keep it around. Let's drop the API and make the file storage interface simpler. Differential Revision: https://phab.mercurial-scm.org/D4750

filelog: stop proxying "opener" (API) The last consumer of it in upgrade code was removed as part of the previous commit. This attribute is revlog specific (because it assumes the existence of a vfs for performing I/O on tracked file data) and therefore isn't appropriate for a generic storage interface. So nuke it. Differential Revision: https://phab.mercurial-scm.org/D4749

filelog: stop proxying flags() (API) Per-revision storage flags are kinda a revlog-centric API. (Except for the fact that changegroup uses the same integer flags as revlog does and there's minimal verification that the server's flags map to the client's storage flags - but that's another problem.) The last user of flags() was in verify.py and that code was just moved into revlog.py and is accessed behind the verifyintegrity() file storage API. Since there are no more consumers, let's drop the proxy and remove the method from the file storage interface. This commit only drops the dedicated API for reading a single revision's storage flags: we still support reading and writing flags through the bulk data retrieval and add revision APIs. And since changegroups encode revlog integer flags over the wire, we'll always need to support flags at some level. The removal of individual storage flags may be too premature. But since flags() is now unused, I'd like to see how far we can get without that dedicated API - especially since it uses revision numbers instead of nodes. Differential Revision: https://phab.mercurial-scm.org/D4746

revlog: move revision verification out of verify File revision verification is performing low-level checks of file storage, namely that flags are appropriate and revision data can be resolved. Since these checks are somewhat revlog-specific and may not be appropriate for alternate storage backends, this commit moves those checks from verify.py to revlog.py. Because we're now emitting warnings/errors that apply to specific revisions, we taught the iverifyproblem interface to expose the problematic node and to report this node in verify output. This was necessary to prevent unwanted test changes. After this change, revlog.verifyintegrity() and file verify code in verify.py both iterate over revisions and resolve their fulltext. But they do so in separate loops. (verify.py needs to resolve fulltexts as part of calling renamed() - at least when using revlogs.) This should add overhead. But on the mozilla-unified repo: $ hg verify before: time: real 700.640 secs (user 585.520+0.000 sys 23.480+0.000) after: time: real 682.380 secs (user 570.370+0.000 sys 22.240+0.000) I'm not sure what's going on. Maybe avoiding the filelog attribute proxies shaved off enough time to offset the losses? Maybe fulltext resolution has less overhead than I thought? I've left a comment indicating the potential for optimization. But because it doesn't produce a performance regression on a large repository, I'm not going to worry about it. Differential Revision: https://phab.mercurial-scm.org/D4745

tests: de-flake test-narrow-debugrebuilddirstate.t If the dirstate gets written much later (usually 1-2 s, depending on FS) than the working copy file (there's only one), then the `hg debugdirstate` command will include a timestamp. There's nothing wrong with that, so we should just allow it. Differential Revision: https://phab.mercurial-scm.org/D4758

upgrade: use storageinfo() for obtaining storage metadata Let's switch to our new API for obtaining information about storage. This eliminates the last consumer of rawsize() and the opener proxy from the file storage interface! Differential Revision: https://phab.mercurial-scm.org/D4748

revlog: add method for obtaining storage info (API) We currently have a handful of methods on the file and manifest storage interfaces for obtaining metadata about storage. e.g. files() is used to obtain the files backing storage. rawsize() is to quickly compute the size of tracked revisions without resolving their fulltext. Code in upgrade and stream clone make heavy use of these methods. The existing APIs are generic and don't necessarily have the specialization that we need going forward. For example, files() doesn't distinguish between exclusive storage and shared storage. This makes stream clone difficult to implement when e.g. there may be a single file backing storage for multiple tracked paths. It also makes reporting difficult, as we don't know how many bytes are actually used by storage since we can't easily identify shared files. This commit implements a new method for obtaining storage metadata. It is designed to accept arguments specifying what metadata to request and to return a dict with those fields populated. We /could/ make each of these attributes a separate method. But this is a specialized API and I'm trying to avoid method bloat on the interfaces. There is also the possibility that certain callers will want to obtain multiple fields in different combinations and some backends may have performance issues obtaining all that data via separate method calls. Simple storage integration tests have been added. For now, we assume fields can't be "None" (ignoring the interface documentation). We can revisit this later. Differential Revision: https://phab.mercurial-scm.org/D4747

lfs: drop unused import A recent change dropped the last user of this module. Differential Revision: https://phab.mercurial-scm.org/D4744

filelog: drop _generaldelta attribute (API) With changegroup moving to emitrevisions(), this revlog-specific attribute is no longer used and can be deleted. Good riddance. Differential Revision: https://phab.mercurial-scm.org/D4727

revlog: drop emitrevisiondeltas() and associated functionality (API) emitrevisions() is the future! Differential Revision: https://phab.mercurial-scm.org/D4726

changegroup: port to emitrevisions() (issue5976) We now have a unified API for emitting revision data from a storage backend. It handles sorting nodes and the complicated delta versus revision decisions for us. This commit ports changegroup to that API. There should be no behavior changes for changegroups not using ellipsis. And lack of test changes seems to confirm that. There are some changes for ellipsis mode, however. Before, when sending an ellipsis revision, we would always send a fulltext revision (as opposed to a delta). There was a TODO tracking this open item. One of the things the emitrevisions() API does for us is figure out whether we can safely emit a delta. So, it is now possible for ellipsis revisions to be sent as deltas! (It does this by not assuming parent/ancestor revisions are available and tracking which revisions have been sent out.) Because we eliminated the list of revision delta request objects, performance has improved substantially: $ hg perfchangegroupchangelog before: ! wall 24.348077 comb 24.330000 user 24.140000 sys 0.190000 (best of 3) after: ! wall 18.245911 comb 18.240000 user 18.100000 sys 0.140000 (best of 3) That's a lot of overhead for creating a few hundred thousand Python objects! This is still a little slower than 4.7. Probably due to 23d582ca introducing a type for the revision/delta results. There is potentially room to optimize. But at some point we need to abstract storage in order to support alternate storage backends. Unfortunately that means using a Python data structure to represent results. And unfortunately there is overhead with every new Python object created. Differential Revision: https://phab.mercurial-scm.org/D4725

wireprotov2server: port to emitrevisions() We now have a proper storage API to request data on multiple revisions. We can drop it into wire protocol version 2 with minimal effort. The new API handles pretty much everything we were doing manually to build up the delta request. So we were able to delete a lot of code. As a bonus, wireprotov2 code is no longer accessing some low-level storage APIs. This includes the assumption that a node has an associated numeric revision number! This should make it drastically simpler to implement a server that doesn't have the concept of revision numbers. Differential Revision: https://phab.mercurial-scm.org/D4724

tests: use more complex file storage test The previous test was attempting to to test delta storage behavior. It didn't do a very good job at it because there was a good chance a delta wasn't being used in storage. Let's switch the test to yield a delta in storage so an upcoming change to delegate delta logic to storage has the desired effect. Differential Revision: https://phab.mercurial-scm.org/D4723

revlog: new API to emit revision data I recently refactored changegroup generation code to make it more storage agnostic. I made significant progress. But there is still a bit of work to be done. Specifically: * Changegroup code is looking at low-level storage attributes to influence sorting. Sorting should be done at the storage layer. * The linknode lookup and sorting code for ellipsis is very complicated. * Linknodes are just generally wonky because e.g. file storage doesn't know how to translate a linkrev to a changelog node. * We regressed performance when introducing the request-response objects. Having thought about this problem a bit, I think I've come up with a better interface for emitting revision deltas. This commit defines and implements that interface. See the docstring in repository.py for more info. This API adds 3 notable features over the previous one. First, it defers node ordering to the storage implementation in the common case but allows overriding as necessary. We have a facility for requesting an exact ordering (used in ellipsis mode). We have another facility for storage order (used for changelog). Second, we have an argument specifying assumptions about parents revisions. This can be used to force a fulltext revision when we don't know the receiver has a parent revision to delta against. Third, we can control whether revision data is emitted. This makes the API suitable as a generic "index data retrieval" API as well as for producing revision deltas - possibly in the same operation! The new API is much simpler: we no longer need a complicated "request" object to encapsulate the delta generation request. I'm optimistic this will restore performance loss associated with emitrevisiondeltas(). Storage unit tests for the new API have been implemented. Future commits will port existing consumers of emitrevisiondeltas() to the new API then remove emitrevisiondeltas(). Differential Revision: https://phab.mercurial-scm.org/D4722