nodemap: fix validity checking when revlog is too short We cannot check the nodeid of a revision that is not even there. We add a simple fix and simple test.

tests: remove unused creation of file and outdated text It was forgotten to remove this in fb0de0bcd297.

tests: skip pyflakes for mercurial/thirdparty/ The current version of pyflakes (2.2.0) correctly detects one issue: mercurial/thirdparty/selectors2.py:335:40 '...'.format(...) has unused arguments at position(s): 1 But we're not interested in fixing lint errors in third-party code, so we need to exclude at least selectors2.py. And in the discussion for this patch it was decided to just skip the entire thirdparty directory. Differential Revision: https://phab.mercurial-scm.org/D8619

git: add debug logging when there's a mismatch in the cached heads list The dag rebuild can be expensive, so let's try and avoid bugs where it transparently rebuilds all the time for no reason. This would have prevented the issue fixed in D8622. Differential Revision: https://phab.mercurial-scm.org/D8625

py3: make stdout line-buffered if connected to a TTY Status messages that are to be shown on the terminal should be written to the file descriptor before anything further is done, to keep the user updated. One common way to achieve this is to make stdout line-buffered if it is connected to a TTY. This is done on Python 2 (except on Windows, where libc, which the CPython 2 streams depend on, does not properly support this). Python 3 rolls it own I/O streams. On Python 3, buffered binary streams can't be set line-buffered. The previous code (added in 227ba1afcb65) incorrectly assumed that on Python 3, pycompat.stdout (sys.stdout.buffer) is already line-buffered. However the interpreter initializes it with a block-buffered stream or an unbuffered stream (when the -u option or the PYTHONUNBUFFERED environment variable is set), never with a line-buffered stream. One example where the current behavior is unacceptable is when running `hg pull https://www.mercurial-scm.org/repo/hg` on Python 3, where the line "pulling from https://www.mercurial-scm.org/repo/hg" does not appear on the terminal before the hg process blocks while waiting for the server. Various approaches to fix this problem are possible, including: 1. Weaken the contract of procutil.stdout to not give any guarantees about buffering behavior. In this case, users of procutil.stdout need to be changed to do enough flushes. In particular, 1. either ui must insert enough flushes for ui.write() and friends, or 2. ui.write() and friends get split into flushing and fully buffered methods, or 3. users of ui.write() and friends must flush explicitly. 2. Make stdout unbuffered. 3. Make stdout line-buffered. Since Python 3 does not natively support that for binary streams, we must implement it ourselves. (2.) is problematic because using unbuffered I/O changes the performance characteristics significantly compared to line-buffered (which is used on Python 2) and this would be a regression. (1.2.) and (1.3) are a substantial amount of work. It’s unclear whether the added complexity would be justified, given that raw performance doesn’t matter that much when writing to a terminal much faster than the user could read it. (1.1.) pushes complexity into the ui class instead of separating the concern of how stdout is buffered. Other users of procutil.stdout would still need to take care of the flushes. This patch implements (3.). The general performance considerations are very similar to (1.1.). The extra method invocation and method forwarding add a little more overhead if the class is used. In exchange, it doesn’t add overhead if not used. For the benchmarks, I compared the previous implementation (incorrect on Python 3), (1.1.), (3.) and (2.). The command was chosen so that the streams were configured as if they were writing to a TTY, but actually write to a pager, which is also the default: HGRCPATH=/dev/null python3 ./hg --cwd ~/vcs/mozilla-central --time --pager yes --config pager.pager='cat > /dev/null' status --all previous: time: real 7.880 secs (user 7.290+0.050 sys 0.580+0.170) time: real 7.830 secs (user 7.220+0.070 sys 0.590+0.140) time: real 7.800 secs (user 7.210+0.050 sys 0.570+0.170) (1.1.) using Yuya Nishihara’s patch: time: real 9.860 secs (user 8.670+0.350 sys 1.160+0.830) time: real 9.540 secs (user 8.430+0.370 sys 1.100+0.770) time: real 9.830 secs (user 8.630+0.370 sys 1.180+0.840) (3.) using this patch: time: real 9.580 secs (user 8.480+0.350 sys 1.090+0.770) time: real 9.670 secs (user 8.480+0.330 sys 1.170+0.860) time: real 9.640 secs (user 8.500+0.350 sys 1.130+0.810) (2.) using a previous patch by me: time: real 10.480 secs (user 8.850+0.720 sys 1.590+1.500) time: real 10.490 secs (user 8.750+0.750 sys 1.710+1.470) time: real 10.240 secs (user 8.600+0.700 sys 1.590+1.510) As expected, there’s no difference on Python 2, as exactly the same code paths are used: previous: time: real 6.950 secs (user 5.870+0.330 sys 1.070+0.770) time: real 7.040 secs (user 6.040+0.360 sys 0.980+0.750) time: real 7.070 secs (user 5.950+0.360 sys 1.100+0.760) this patch: time: real 7.010 secs (user 5.900+0.390 sys 1.070+0.730) time: real 7.000 secs (user 5.850+0.350 sys 1.120+0.760) time: real 7.000 secs (user 5.790+0.380 sys 1.170+0.710)

simplemerge: rewrite flag merging loop as expression I feel binary operations are more readable.

simplemerge: leverage pycompat function to convert byte string to set

simplemerge: fix function name that tests if ctx is not null revision

git: decode node IDs back into Python strings (issue6349) db.text_factory = bytes, so the database contains only strings. The object IDs we get from pygit2 are Python strings. b'foo' != 'foo' This change allows the "don't reindex" optimization to work by allowing the "cur_cache_heads == cache_heads" comparison a few lines down to succeed. Differential Revision: https://phab.mercurial-scm.org/D8622

phabricator: make it clear what happen when no response Differential Revision: https://phab.mercurial-scm.org/D8621

tests: make it clear what happen when no response entered Differential Revision: https://phab.mercurial-scm.org/D8620

localrepo: handle ValueError during repository opening Python 3.8 can raise ValueError on attempt of an I/O operation against an illegal path. This was causing test-remotefilelog-gc.t to fail on Python 3.8. This commit teaches repository opening to handle ValueError and re-raise an Abort on failure. An arguably better solution would be to implement this logic in the vfs layer. But that seems like a bag of worms and I don't want to go down that rabbit hole. Until users report uncaught ValueError exceptions in the wild, I think it is fine to patch this at the only occurrence our test harness is finding it. Differential Revision: https://phab.mercurial-scm.org/D7944

metadata: filter the `removed` set to only contains relevant data The `files` entry can be bogus and contains too many entries. This can badly combines with the computation of `removed` inflating the set size. The can lead to the changesets centric rename computation to process much more data than needed, slowing it down (and increasing space taken by data storage). In practice newer commits already that reduced set, this applies this "fix" to older changeset. Differential Revision: https://phab.mercurial-scm.org/D8589

files: extract code for extra filtering of the `removed` entry into copies We want to reduce the set of `removed` files that to the set of files actually removed. That `removed` set is used as of the changeset centric algorithm, having smaller sets means less processing and faster computation. In this changeset we extract the code to be a function of it own. We will make use of it in the next changesets. Differential Revision: https://phab.mercurial-scm.org/D8588

metadata: move computation related to files touched in a dedicated module This was suggested by Yuya Nishihara a while back. Since I am about to add more metadata related computation, lets create a new repositories. Differential Revision: https://phab.mercurial-scm.org/D8587

merge: move an inspection of the dirstate from record to calculate phase The intent is clearly to have `calculateupdates()` figure out what actions need to be taken and `recordupdates()` to make necessary modifications to the dirstate. However, in the `ACTION_PATH_CONFLICT_RESOLVE` case, there was one little inspection of copy information done in `recordupdates()`. This patch moves that to `calculateupdates()`. That will help with the next patch, which makes `merge.update()` work better with `overlayworkingctx` (copies should be recorded there too, even though we skip the `recordupdates()` step). Differential Revision: https://phab.mercurial-scm.org/D8615

tests: adjust to the new format in pyflakes output According to the pyflakes' NEWS.rst, the default output format changed recently: 2.2.0 (2020-04-08) - Include column information in error messages So the lines now read: contrib/perf.py:149:15 undefined name 'xrange' mercurial/hgweb/server.py:427:13 undefined name 'reload' mercurial/util.py:2862:24 undefined name 'file' Differential Revision: https://phab.mercurial-scm.org/D8618

tests: consistently use pyflakes as a Python module We check availability of pyflakes as a module, and also running it for real as a module. Only fair to test filterpyflakes.py working correctly when using pyflakes as a module too. Differential Revision: https://phab.mercurial-scm.org/D8617

heptapod-ci: also run tests for chg on python 2 I am not aware of any regular effort to run test with chg. So let's at least do it here. Differential Revision: https://phab.mercurial-scm.org/D8178

rust-dependencies: upgrade `micro-timer` dependency I wanted to to a tour of dependencies to upgrade, but only `micro-timer` has a new release which does not print when the function panics, which should be less misleading. Differential Revision: https://phab.mercurial-scm.org/D8605

context: fix creation of ProgrammingError to not use non-existent field Differential Revision: https://phab.mercurial-scm.org/D8606

help: explain in `hg help flags` that unambiguous prefixes are allowed I used `hg commit --amend` as an example because that's the most frequently used flag I could think of that doesn't yet have a short form. Differential Revision: https://phab.mercurial-scm.org/D8607

git: add coverage for manifest.diff() so we don't regress

git: implement diff manifest method This makes 'hg diff' work.

git: properly visit child tree objects when resolving a path

git: don't yield paths for directories when walking

git: correctly check for type of object when walking

git: implement stub prefetch_parents dirstate method A recent change (35b255e474d9) introduced this new required dirstate method but didn't update the git extension.

rust: leverage .expect() in place of .unwrap() + inline comment For a better error indication in case we made a mistake.

rust: fix false comment about mpsc::Sender We need Sync to share the Sender reference across threads.

resourceutil: fix location of line comments These comments slipped out of position when the sources where formatted with black in 2372284d9457.

rebase: avoid clobbering wdir() with --dry-run or --confirm (issue6291) See 56d3e0b499df for the source of adding originalwd to the list of things that cause wdir to be updated. That change didn't come with tests, and attempts to recreate the scenario described have thus far failed. Differential Revision: https://phab.mercurial-scm.org/D8489

tests: show that rebase --dry-run and --confirm wipeout uncommitted changes It looks like the carnage is limited to rebasing something that is not an ancestor of wdir(), as both of these abort in a preflight check for that case with a dirty working directory. Differential Revision: https://phab.mercurial-scm.org/D8488

Added signature for changeset 065704cbdbdb

Added tag 5.4.1 for changeset 065704cbdbdb

py3: update comment to account for Python 2 and Python 3 differences

py3: add warning about buffering behavior of pycompat.{stdout,stderr}

tests: fix indentation

merge with stable

graft: fix --base value to be saved in state file 'True' just works because it is treated as an integer revision '1' and only the truthiness of the basectx is important. If multiple source revisions were supported with --base, the resumed graft operation would go wrong.

flags: also test merge with executable bit removed This might catch more bug in the future. Differential Revision: https://phab.mercurial-scm.org/D8536

flags: also test the removal of the exec flag Differential Revision: https://phab.mercurial-scm.org/D8535

flags: read flag from dirstate/disk for workingcopyctx (issue5743) In 491855ea9d62, various piece of code are moved from committablectx to workingctx. The reason given is "These read from the dirstate, so they shouldn't be used in other subclasses." At least for `flags` this change introduce a bug, because the value flags end up being read from `_manifest` disregarding the actual state in the working copy (ie: on disk). When merging exec flag change with renames, this means a new files (the local content, renamed) is properly written on disk, with the right flags, but the flags part is later ignored when actually reading flags during merge. It is not clear to me why the `flags` function was moved, because the code does not actually hit the dirstate (the reason given in the changeset description). So I am moving it back to were it comes from and we use a simpler version of that code (that hit the dirstate everytime) in workingcopyctx. This fix the last know bug with merging rename and executable byte changes. Other similar bug might be lurking in 491855ea9d62, but I have not investigated them. Differential Revision: https://phab.mercurial-scm.org/D8534

flags: actually merge flags in simplemerge Since b86fc43e4b73, the local flag were blindly taken. This resulted in bug when rename are involved. exec flag change are now properly merged (when merged from the rename side). Another bug is affecting this when merging from the side without the rename. Differential Revision: https://phab.mercurial-scm.org/D8533

flags: add a test for merging exec flag change with rename and file change Changing the file activate other code path that also have bugs… There are two distinct bugs depending of which side of the merge you stand on. They both leading to exec flag loss. We add tests for both, the fix are coming in later changesets. Differential Revision: https://phab.mercurial-scm.org/D8532

flags: account for flag change when tracking rename relevant to merge There are some logic filtering rename to the one relevant to the merge. That logic was oblivious of flag change, leading to exec flag being dropped when merged with a renamed. There are two others bugs affecting this scenario. This patch fix the was where there is not modification involved except for the flag change. Fixes for the other bug are coming in later changesets. Differential Revision: https://phab.mercurial-scm.org/D8531

flags: also test merging a rename with and exec flag change This case is currently buggy and was not tested. This is probably a quite old regression. The next changeset fix this case. Move exec+rename related bug will gain a test later. To highlight the expected behavior the currently missing line are marked with (false !) and the bad one with (true !) note: we should probably gain explicit "test bool" for this usecases. Differential Revision: https://phab.mercurial-scm.org/D8530

flags: introduce explicit testing for merging change to exec flag It turns out that we do not seems to test the simple case for merging exec flag changes. More advanced case are test (merging exec flag without a common ancestors, merging with a symlink, etc…) but not the basic. We are about introduce various fixes to merging flag change across renames, having the most basic case tested first seems useful. note: We are only testing "adding" an exec flag here, not removing it. We introduce basic test on stable and will consolidate them on default. Differential Revision: https://phab.mercurial-scm.org/D8529

graft-state: save --base in graft's state, fixing bug with graft --continue Without this change, running graft --continue after grafting a merge commit using --base (and encountering conflicts) will output "skipping ungraftable merge revision" even though we specified a base in the initial graft command. Graft's improve behaviour is reflected in test-graft.t. Differential Revision: https://phab.mercurial-scm.org/D8578

relnotes: advertize the possibility to use rust I think the rust work may have been mentioned in the release notes, but if so only in passing, and not as an invitation to try it out. I think the next version is a decent time to do this, because the rust doesn't come with performance regressions AFAIK, speeds up status noticeably when it applies, which is the case for most invocations of status, and doesn't have the undesirable restriction of regex around empty patterns anymore. I am cheating a bit, because I'm giving numbers for `hg status` in mozilla-central, but they have one hgignore pattern that uses lookaround, ".vscode/(?!extensions\.json|tasks\.json", which I took out as it would cause a fallback to python when unknown files are requested. But it seems that they could express their hgignore differently if they were so inclined. Not sure if there are limitation other than linux-only that I am not thinking of but would be worth mentioning upfront, to avoid disappointing users? Differential Revision: https://phab.mercurial-scm.org/D8604

rust: add a pointer for profiling to the README As figuring out how to get useful profiles is not obvious. Differential Revision: https://phab.mercurial-scm.org/D8603

rust: update the mention of hgcli in rust/README.rst This may not be exactly right, but it's better than before. Differential Revision: https://phab.mercurial-scm.org/D8602

sslutil: fix comment to use inclusive or instead of exclusive or The incorrect "either" was introduced by one of my recent patches.

sslutil: propagate return value ssl.PROTOCOL_SSLv23 from protocolsettings() Also, protocolsettings() was renamed to commonssloptions() to reflect that only the options are returned.

sslutil: stop storing protocol and options for SSLContext in settings dict Call protocolsettings() where its return values are needed.

sslutil: rename 'minimumprotocolui' -> 'minimumprotocol' Before, both 'minimumprotocolui' and 'minimumprotocol' were used, but meaning the same.

sslutil: properly detect which TLS versions are supported by the ssl module For the record, I contacted the CPython developers to remark that unconditionally defining ssl.PROTOCOL_TLSv1_1 / ssl.PROTOCOL_TLSv1_2 is problematic: https://github.com/python/cpython/commit/6e8cda91d92da72800d891b2fc2073ecbc134d98#r39569316

sslutil: remove dead code (that failed if only TLS 1.0 is available) We ensure in setup.py that TLS 1.1 or TLS 1.2 is present.

config: remove unused hostsecurity.disabletls10warning config

sslutil: remove dead code (that downgraded default minimum TLS version) We ensure in setup.py that TLS 1.1 or TLS 1.2 is present.