ci: shard the test run on mac os X This should comes with some benefit: - spread the load across more runner, - reduce the real-time CI run, - reduce the "retry" run when we need them. We start with the Mac jobs, but that would be tremendously useful for Windows too. For linux, we need to reduce the startup overhead for this to be worth it. Building smaller image and speeding up clone should help with that.

run-tests: implement crude sharding support It will help to spread the testing load across more CI runners.

ci: have the mac test run if you trigger building the mac wheel The mac test job now depends on the wheel building. And the wheel building is manual. So if the mac test job is set to "on_success" if will be "skipped" by default, and automatically run if the wheel are build. That is especially handy as we are about to shard that test and that the UI for manual sharded test sucks.

ci: adjust the starting port range to runner concurrency If multiple job runs on the same runner, they should not use the port range.

ci: move the "tempory work dir" to "concurrency-safe" location Lets not use a global location and move at the root of the directory dedicated to the job.

ci: abstract the of absolute /tmp/ path We now have a TMP_WORK_DIR directory that we can update to a more sensible value in the next changesets.

ci: rationalize variable usage The usage of "extends" allow to skip a lot of duplication. We also introduce more fine grained variables to help finer override.

ci: move some variables closer to their usage These were defined globaly because we had trouble making them inherited. This is now fixed, so we can get them were they belong.

ci: use extends instead of <<: *x The old form is a yaml construct that make it hard to share variable definition. The "extends:" key is a gitlab specific that preserve the variable definition and just add the new ones. This will help us to reduce duplication. This has the effect of fixing some of variants definition we though we set while we did actually not. Most notably, the "rust" variant for 3.12 and 3.13 seems fully broken in the CI (possibly because some rust-cpython version issue?). This changeset only reveal such breackage and does not introduce them.

ci: use the macos wheel to run tests

wheel: build mac os wheel through the CI Let's start building wheel for mac os X too.

pycompat: drop some now useless workaround for makedirs This `exists_ok` flag was added in Python 3.2

run-tests: install wheel using --prefix instead of --user --user does not work if a venv is enabled when calling run-tests.py

ci: automatically compute the python tag we use to identify tag This make the determination more automatic and less error prone. In addition, this will make it possible to run on a runner without a pre-determined Python version, like what we do for the macos and windows workers.

pytype: add relative timestamp to the output if `ts` is available This should help to identify the module that are the slower to analyze.

ci: build (and use) wheel for all supported version We test wheel building for all supported version and use them where applicable The usage is more verbose than I wish because .gitlab-ci is not that great.

ci: use a pre-setup many-linux image to build wheel This produce wheel that are more universal and identical to the one we want to publish.

ci: use smaller VM to build wheel There is no need for large machine for this job, Python will be mostly singled threaded anyway.

setup: add a way to force the setup to translate (or fail) we add the `MERCURIAL_SETUP_FORCE_TRANSLATIONS` variable that is intended to make sure we don't stop building the translation silently.

ci: pre-adjust some identation We adjust a couple of job to consistently use double space identation. This will make the next changesets clearer.

test-install: glob instance of "python" in warning If run with something else (e.g. python3.12) the message says "python3.12" so we just glob that away.

test-install: use the global hg for the install step This prevent error in some cases.

tests: bump the wait timeouts in test-racy-mutations.t This was done to try to figure out why there's output differences in the previous two commits- low timeouts have been a cause of a lot of problems on Windows. That doesn't seem to be the case here, but I'm leaving it in anyway to avoid sporadic failures.

tests: add a "missing" tests for manifest content in test-racy-mutations.t Trying to figure out why the divergence in behavior on Windows. The first test shows everything is the same on all platforms; the second shows that the other #testcase also diverges. The difference might be that `00manifest.i` doesn't get updated (and copied over) after `pre-race` is created, but I've no idea why that would be either.

tests: conditionalize missing output in test-racy-mutations.t on Windows No idea why this is only missing on Windows. I verified that the line marked as missing directly above this for the changelog is, in fact, missing on both Windows and Linux. So there's probably work to do in this area on all platforms. It would be nice to figure out what is going on, but this appeases CI in the meantime.

tests: disable a section of `test-paths.t` that may hit a zeroconf bug This effectively re-disables the same test as cce9e7d2fb92, but unconditionally because it's not a pyoxidizer-specific problem. 74e16d8ca5f3 disabled the other related test for the same reason- this one was missed because it wasn't failing for the few runs when that was tested.

rust-update: make `update_from_null` respect `worker.numcpu` config option This was overlooked in the original series. This is important for tests (because we run many at once), and for the occasional end user that wants to keep their CPU usage in check. A future series should clean up this `worker` parameter tunelling business by rewriting the config in Rust, but doing so on stable would be a very bad idea.

rust-cpython: add a TODO about repo reuse This will need to be done soon-ish to prevent any surprises.

streamclone: disable the volatile file open handle optimization on Windows Leaving files open caused new failures like this, since a47f09da8bd1: diff --git a/tests/test-persistent-nodemap-stream-clone.t b/tests/test-persistent-nodemap-stream-clone.t --- a/tests/test-persistent-nodemap-stream-clone.t +++ b/tests/test-persistent-nodemap-stream-clone.t @@ -115,7 +115,12 @@ Do a mix of clone and commit at the same $ (hg clone -U --stream ssh://user@dummy/test-repo stream-clone-race-1 --debug 2>> clone-output | grep -E '00(changelog|manifest)' >> clone-output; touch $HG_TEST_STREAM_WALKED_FILE_3) & $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_1 $ hg -R test-repo/ commit -m foo - created new head + transaction abort! + failed to recover 00changelog.n ([WinError 32] The process cannot access the file because it is being used by another process: b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n' -> b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n-f418dcd6') + rollback failed - please run hg recover + (failure reason: [WinError 32] The process cannot access the file because it is being used by another process: b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n' -> b'$STR_REPR_TESTTMP\\test-repo/.hg/store/00changelog.n-f418dcd6') + abort: The process cannot access the file because it is being used by another process: '$TESTTMP\test-repo\.hg\store\00changelog.n' + [255] $ touch $HG_TEST_STREAM_WALKED_FILE_2 $ $RUNTESTDIR/testlib/wait-on-file 10 $HG_TEST_STREAM_WALKED_FILE_3 $ cat clone-output Since the `VolatileManager` falls back to the old copy method when the open file threshold is exceeded, this just drops the threshold so that only 1 file is open. The actual value used (2) is unexpected, and explained inline. I'd like to have a config option for this so that we can test both ways (in theory, it could resort to copies on non-Windows systems too), but I don't see a `uimod.ui` handy. Alternately, I tried replacing the 3 `open()` calls in the `VolatileManager` with `util.posixfile()`, but that simply hung the test on Windows for some reason, I think on the same line that's indicated as failing above. (There was a `grep` command hanging around, as well as `hg -R test-repo serve --stdio`.)

tests: treat `select` as a built-in module on Windows This fixes: --- C:/Users/Matt/hg/tests/test-check-module-imports.t +++ C:/Users/Matt/hg/tests/test-check-module-imports.t.err @@ -43,3 +43,15 @@ > -X tests/test-verify-repo-operations.py \ > -X tests/test-extension.t \ > | sed 's-\\-/-g' | "$PYTHON" "$import_checker" - + hgext/zeroconf/Zeroconf.py:86: stdlib import "socket" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:87: stdlib import "struct" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:88: stdlib import "threading" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:89: stdlib import "time" follows local import: select\r (esc) + hgext/zeroconf/Zeroconf.py:90: stdlib import "traceback" follows local import: select\r (esc) + mercurial/posix.py:18: stdlib import "stat" follows local import: select\r (esc) + mercurial/posix.py:19: stdlib import "sys" follows local import: select\r (esc) + mercurial/posix.py:20: stdlib import "tempfile" follows local import: select\r (esc) + mercurial/posix.py:21: stdlib import "typing" follows local import: select\r (esc) + tests/tinyproxy.py:19: stdlib import "socket" follows local import: select\r (esc) + tests/tinyproxy.py:20: stdlib import "sys" follows local import: select\r (esc) + [1] ERROR: test-check-module-imports.t output changed

hgweb: skip logging ConnectionAbortedError Not stacktracing on `ConnectionResetError` was added in 6bbb12cba5a8 (though it was spelled differently for py2 support), but for some reason Windows occasionally triggers a `ConnectionAbortedError` here across various *.t files (notably `test-archive.t` and `test-lfs-serve-access.t`, but there are others). The payload that fails to send seems to be the html that describes the error to the client, so I suspect some code is seeing the error status code and closing the connection before the server gets to write this html. So don't log it, for test stability- nothing we can do anyway. FWIW, the CPython implementation of wsgihander specifically ignores these two errors, plus `BrokenPipeError`, with a comment that "we expect the client to close the connection abruptly from time to time"[1]. The `BrokenPipeError` is swallowed a level up in `do_write()`, and avoids writing the response following this stacktrace. I'm puzzled why a response is being written after these connection errors are detected- the CPython code referenced doesn't, and the connection is now broken at this point. Perhaps these errors should both be handled with the `BrokenPipeError` after the freeze. (The refactoring away from py2 compat may not be desireable in the freeze, but this is much easier to read, and obviously correct given the referenced CPython code.) I suspect this is what 6bceecb28806 was attempting to fix, but it wasn't specific about the sporadic errors it was seeing. [1] https://github.com/python/cpython/blob/b2eaa75b176e07730215d76d8dce4d63fb493391/Lib/wsgiref/handlers.py#L139

ci: add a runner for Windows 10 This is currently only manually invoked, and allows for failure because we only have a single runner that takes over 2h for a full run, and there are a handful of flakey tests, plus 3 known failing tests. The system being used here is running MSYS, Python, Visual Studio, etc, as installed by `install-windows-dependencies.ps1`. This script installs everything to a specific directory instead of using the defaults, so we adjust the MinGW shell path to compensate. Additionally, the script doesn't install the launcher `py.exe`. It is possible to adjust the script to install it, but it's an option to an existing python install (instead of a standalone installer), and I've had the whole python install fail and rollback when requested to install the launcher if it detects a newer one is already installed. In short, it is a point of failure for a feature we don't (yet?) need. Unlike other systems where the intepreter name includes the version, everything here is `python.exe`, so they can't all exist on `PATH` and let the script choose the desired one. (The `py.exe` launcher would accomplish, using the registry instead of `PATH`, but that wouldn't allow for venv installs.) Because of this, switch to the absolute path of the python interpreter to be used (in this case a venv created from the py39 install, which is old, but what both pyoxidizer and TortoiseHg currently use). The `RUNTEST_ARGS` hardcodes `-j8` because this system has 4 cores, and therefore runs 4 parallel tests by default. However on Windows, using more parallel tests than cores results in better performance for whatever reason. I don't have an optimal value yet (ideally the runner itself can make the adjustment on Windows), but this results in saving ~15m on a full run that otherwise takes ~2.5h. I'm also not concerned about how it would affect other Windows machines, because we don't have any at this point, and I have no idea when we can get more. As far as system setup goes, the CI is run by a dedicated user that lacks admin rights. The install script was run by an admin user, and then the standard user was configured to use it. If I set this up again, I'd probably give the dedicated user admin rights to run the install script, and reset to standard user rights when done. The python intepreter failed in weird ways when run by the standard user until it was manually reinstalled by the standard user: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding Additionally, changing the environment through the Windows UI prompts to escalate to an admin user, and then setting the user level environment variables like `TEMP` and `PATH` (to try to avoid exceeding the 260 character path limit) didn't actually change the user's environment. (Likely it changed the admin user's environment, but I didn't confirm that.) I ended up having to use the registry editor for the standard user to make those changes.

tests: disable a section of `test-hgrc.t` that may hit a zeroconf bug This effectively re-disables the same test as cce9e7d2fb92, but unconditionally because it's not a pyoxidizer-specific problem (see below and 997c9b2069d1). I can run the test locally fine, with the same venv as CI is using, and have had multiple CI runs that don't hit this. But one failed with this: --- /private/tmp/mercurial-ci/tests/test-hgrc.t +++ /private/tmp/mercurial-ci/tests/test-hgrc.t.err @@ -305,5 +305,17 @@ [255] $ HGRCSKIPREPO=1 hg paths --config extensions.zeroconf= + Traceback (most recent call last): + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 966, in run + self.readers[sock].handle_read() + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 1020, in handle_read + msg = DNSIncoming(data) + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 537, in __init__ + self.readOthers() + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 650, in readOthers + self.readCharacterString(), + File "/private/tmp/hgtests.7idf706t/install/lib/python/hgext/zeroconf/Zeroconf.py", line 584, in readCharacterString + length = ord(self.data[self.offset]) + TypeError: ord() expected string of length 1, but int found foo = $TESTTMP/bar The zeroconf extension has bytes vs str problems that are obvious from inspection alone, and nobody has complained, so I'm not going to let this block getting CI for macOS up and running. Given that it's in the packet read code, I suspect that this 1) requires something on the network to speak mDNS, and 2) it is a timing issue if this is seen or not. (The bytes vs str issue itself is real, but only happen if a response is received quickly.)

tests: disable `test-git-interop.t` with a requirements directive Note that the failures in this test affect all platforms. I don't like this, but the test has been broken for awhile because of dirstate API changes, and nobody noticed because the required `pygit2` package isn't installed on the CI systems. I did install it on the mac CI system, which triggers this failure. Disabling it is no worse than not running it due to the missing package, but at least this way the CI systems can get the package installed, and the test can be enabled and fixed eventually, without needing to alter the CI systems. The feature here is kind of abused. I thought about adding one specifically to test for CI, but didn't feel like doing it at this point. Maybe if we need to disable things to get the Windows CI off the ground (but that likely requires testing for CI + platform).

tests: stabilize `test-extdiff.t` on macOS The recent change in the extdiff extension to take into account whether the GUI is accessible in d1b54c152673 started triggering this. I was able to run the test cleanly without this change at the console, but somewhere along the line, I read that the CI runner isn't able to access the GUI when not run as the root user. This is causing CI failures, so we conditionalize these tests out where `DISPLAY` is set to a non empty value to force `procutil.isgui()` to be True, when it in fact doesn't have GUI access.

ci: build a wheel and use it to run c tests First step into building and testing wheel automatically.

ci: split the jobs on more stage We start to have a lot of job, grouping them help to clarifying the pipeline. We don't actually create dependency between each stage, so everything still run concurrently. However we are about to introduce some wheel-building job that will be reused by some tests. So some dependencies are coming.

ci: unify the way `check-pytype` inherit the common setting All the other job use this syntax, so lets us it there too.

run-tests: add a --hg-wheel options to test a pre-built wheel This will be useful to test the wheel we intend to publish. A future changeset will integrate this in the CI.

run-tests: don't use shell call for subprocess This part of the test runner seems to comes for some ages ago.

Added signature for changeset dc97e8670dec

Added tag 6.9rc0 for changeset dc97e8670dec

doc: register the `config-doc` rst directive This was making the build fail because the directive was unknown.

relnotes: add 6.9rc0

branching: merge default into stable We will be releasing 6.9rc0 soon.

branching: merge stable into default

tests: skip doctests that use `time.tzset()` on Windows There's no way to conditionally skip the tests for a function (see the inline feature request). That leaves us with the choice to either put the whole `mercurial.utils.dateutil` module in the skip list of this script (but then this script prints out the module as unexpectedly not tested, and misses a bunch of tests that can be run), blacklist the test entirely (but that makes it harder to work with on Windows), or use this hack to look for the statement that is broken, and skip the test currently attached to one function. (It appears that an example in the list of examples corresponds to a single `>>>` block, and the `test` itself corresponds to a single function. So prescan the examples, and skip all of them when the statement is found in any, since the setup of setting the timezone has an effect on subsequent examples.)

tests: hopefully fix `test-doctest.py` on Windows and more 1. Shell syntax understood by `shell=True` depends on the platform. Instead, pass `shell=False` and call `sh` explicitly to interpret the command correctly. 2. Stop setting `HGRCPATH=/dev/null`, so the setting `experimental.evolution=createmarkers` is set correctly. The reason I set HGRCPATH to /dev/null previously is because of misunderstanding where I thought the Python script had no HGRC to edit. As it turns out, there is in fact a valid temporary HGRC pointed to by HGRCPATH in this context so we don't seem to need this. /shrug

hghave: make the description for "clang-format" ascii test-fix-clang-format.t suddenly started failing on Windows by wiping the whole file content, and replacing with an error: $TESTTMP.sh: $TESTTMP.sh: cannot execute binary file Odd, because I don't have `clang-format` installed, so the test should be skipped. The problem started with 73cf8b56c2f5, and I noticed that running `hghave` manually resulted in a `SyntaxError` (so I can't see how this isn't broken everywhere, but maybe it's because I'm using py3.9 on Windows): $ py hghave --list Traceback (most recent call last): File "hghave", line 8, in <module> import hghave File "c:\Users\Matt\hg\tests\hghave.py", line 627 SyntaxError: Non-ASCII character '\xe2' in file c:\Users\Matt\hg\tests\hghave.py on line 627, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

branching: merge stable into default Hopefully this will bring the last changes necessary to make the 3.13 tests green (on Linux).

tests: enable pytype checking on `mercurial/wireprotov1peer.py`

typing: suppress bogus pytype errors in `mercurial/wireprotov1peer.py` Fixes: File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 100, in result: No attribute '_peerexecutor' on unsentfuture [attribute-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 278, in close: No attribute 'shutdown' on None [attribute-error] Called from (traceback): line 123, in __exit__ File "/mnt/c/Users/Matt/hg/mercurial/wireprotov1peer.py", line 278, in close: No attribute 'shutdown' on None [attribute-error] In Optional[concurrent.futures.thread.ThreadPoolExecutor] We drop the zope decorator on `peerexecutor`, because otherwise it triggers this error: File "/tmp/mercurial-ci/mercurial/wireprotov1peer.py", line 111, in <module>: Invalid type annotation [invalid-annotation] Must be constant Not sure why, because the decorated classes usually get typed as `Any`, which would also be fine here.

tests: enable pytype checking on `mercurial/wireprotoframing.py`

typing: suppress bogus pytype errors in `mercurial/wireprotoframing.py` This fixes: File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 480, in createalternatelocationresponseframe: unsupported operand type(s) for item assignment: bytes [unsupported-operands] No attribute '__setitem__' on bytes File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 510, in createcommanderrorresponse: unsupported operand type(s) for item assignment: bytes [unsupported-operands] No attribute '__setitem__' on bytes File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 776, in __init__: Can't find module 'mercurial.zstd'. [import-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 804, in __init__: Can't find module 'mercurial.zstd'. [import-error] File "/mnt/c/Users/Matt/hg/mercurial/wireprotoframing.py", line 834, in populatestreamencoders: Can't find module 'mercurial.zstd'. [import-error] Using `TypedDict` is tempting here to fix the first two, but requires str keys. The code doing the importing doesn't call the code at the other three locations if the `mercurial.zstd` module fails to import in a place that handles the ImportError.

wireprototypes: make `baseprotocolhandler` methods abstract The documentation says it's an abstract base class, so let's enforce it. The `typing.Protocol` class is already an ABC, but it only prevents instantiation if there are abstract attrs that are missing. For example, from `hg debugshell`: >>> from mercurial import wireprototypes >>> x = wireprototypes.baseprotocolhandler() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class baseprotocolhandler with abstract method name >>> class fake(wireprototypes.baseprotocolhandler): ... pass ... >>> x = fake() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class fake with abstract method name That's great, but it doesn't protect against calling non-abstract methods at runtime, rather it depends on the protocol type hint being added to method signatures or class attrs, and then running a type checker to notice when an instance is assigned that doesn't conform to the protocol. We don't widely use type hints yet, and do have a lot of class hierarchy in the repository area, which could lead to surprises like this: >>> class fake(wireprototypes.baseprotocolhandler): ... @property ... def name(self) -> bytes: ... return b'name' ... >>> z = fake() >>> z.client() >>> print(z.client()) None Oops. That was supposed to return `bytes`. So not only is a bad/unexpected value returned, but it's one that violates the type hints (since the base client() method will be annotated to return bytes). With this change, we get: >>> from mercurial import wireprototypes >>> class fake(wireprototypes.baseprotocolhandler): ... @property ... def name(self) -> bytes: ... return b'name' ... >>> x = fake() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: Can't instantiate abstract class fake with abstract methods addcapabilities, checkperm, client, getargs, getpayload, getprotocaps, mayberedirectstdio So this looks like a reasonable safety harness to me, and lets us catch problems by running the standard tests while the type hints are being added, and pytype is improved. We should probably do this for all Protocol class methods that don't supply a method implementation.

wireprototypes: convert `baseprotocolhandler.name` to an abstract property PyCharm was flagging the subclasses where this was declared as a `@property` with Type of 'name' is incompatible with 'baseprotocolhandler' But pytype didn't complain. This seems more correct, however. Since `Protocol` is already an `abc.ABCMeta` class, we don't need to mess with the class hierarchy.

wireprotoserver: subclass the new `baseprotocolhandler` Protocol class

wireprototypes: convert `baseprotocolhandler` to a Protocol class The methodology for doing this is now known, and this is limited to two implementing classes, so just make the changes.

test: stabilize `test-audit-path.t` in rust (hopefully) We have been seeing flakiness on the file reported for a bit.

tests: conditionalize undesired output on Windows for rbc the mmap cases I don't want to lose sight of this issue, and it's useful to be able to turn on mmap support to hack on the underlying problem. As noted in the previous commit, I think the current usage of `mmap` and `memoryview` needs to be reworked for correctness on posix anyway.