py3: manually import pycompat.setattr where it is needed Continuing to eliminate the implicit import of symbols in the Python 3 source transformer so we can eliminate it. Differential Revision: https://phab.mercurial-scm.org/D7007

py3: stop implicitly importing unicode We should be pycompat.unicode everywhere. It turns out we were doing this everywhere except for one place in templatefilters! Differential Revision: https://phab.mercurial-scm.org/D7006

py3: manually import pycompat.open into files that need it We want to eliminate the source transformer. Currently it inserts a `from mercurial.pycompat import ...` at the top of files to alias some builtins. This commit replaces the implicit import of `open` with an explicit import on files that need it and changes the source transformer to no longer import `open`. As part of this, we needed to store an explicit local for `open` in the Python 2 code path in `pycompat` so the import works. (Builtins that are automatically in scope cannot be imported.) Differential Revision: https://phab.mercurial-scm.org/D7005

import-checker: allow symbol imports from mercurial.pycompat Currently, the source transformer inserts `from mercurial.pycompat import delattr, getattr, hasattr, setattr, open, unicode` to the top of every file. As part of getting rid of the source transformer, we'll need to have source code call these wrappers directly. Rather than rewrite all call sites to call pycompat.*, I think it makes sense to import needed symbols via explicit imports. That requires loosening the import checker to allow this. Differential Revision: https://phab.mercurial-scm.org/D7004

contrib: add a fork of black (as "grey") that includes my changes This is black with https://github.com/psf/black/pull/826 applied as of today. The current git hash of black master is d9e71a75ccfefa3d9156a64c03313a0d4ad981e5, and the hash of my commit is dc1add6e94e212eff37bb3619e1422fb3c6d8dc8. In order to use this, you need to install `black` (from github master) and `typed-ast` using pip, preferably into python3, and then you can run `grey.py` with that Python and you'll have my patched version of black, which is how we've been formatting the codebase. Once my PR is merged, I'll follow up by removing this fork and updating instructions in the example config. # no-check-commit bad style Differential Revision: https://phab.mercurial-scm.org/D7002

rust-cpython: change license of ref_sharing.rs to MIT Since we plan to upstream this feature, it's better to continue further refactoring under the same license as rust-cpython crate. According to the file history, copyright holders are: - Raphaël Gomès <rgomes@octobus.net> - Valentin Gatien-Baron <valentin.gatienbaron@gmail.com> - Yuya Nishihara <yuya@tcha.org>

contrib: fix check-code to be able to detect missing _() with bytestrings Differential Revision: https://phab.mercurial-scm.org/D6997

cleanup: mark some ui.(status|note|warn|write) calls as not needing i18n These used to be marked with no-op parens, but black removes those now and this is more explicit. # skip-blame: fallout from mass reformatting Differential Revision: https://phab.mercurial-scm.org/D6996

ui: define (write|status|warn|note)noi18n aliases We currently use `write(('...'))` to suppress check-code warnings about not using translated strings. However, when we run black, it will strip the `((...))`. In order to placate black, we'll need to use a different mechanism to pass untranslatable strings. This commit introduces a `writenoi18n` alias (and friends) to `write` for that purpose. Differential Revision: https://phab.mercurial-scm.org/D6994

mangler: stop rewriting string constants to be bytes literals We've rewritten everything, so we no longer require this step. Differential Revision: https://phab.mercurial-scm.org/D6973

formatting: byteify all mercurial/ and hgext/ string literals Done with python3.7 contrib/byteify-strings.py -i $(hg files 'set:mercurial/**.py - mercurial/thirdparty/** + hgext/**.py - hgext/fsmonitor/pywatchman/** - mercurial/__init__.py') black -l 80 -t py33 -S $(hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**" - hgext/fsmonitor/pywatchman/**') # skip-blame mass-reformatting only Differential Revision: https://phab.mercurial-scm.org/D6972

formatting: blacken the codebase This is using my patch to black (https://github.com/psf/black/pull/826) so we don't un-wrap collection literals. Done with: hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**"' | xargs black -S # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D6971

style: run a patched black on a subset of mercurial This applied black to the 20 smallest files in mercurial/: ls -S1 mercurial/*.py | tail -n20 | xargs black --skip-string-normalization Note that a few files failed to format, presumably due to a bug in my patch. The intent is to be able to compare results to D5064 with https://github.com/python/black/pull/826 applied to black. I skipped string normalization on this patch for clarity - in reality I think we'd want one pass without string normalization, followed by another to normalize strings (which is basically replacing ' with " globally.) # skip-blame mass-reformatting only Differential Revision: https://phab.mercurial-scm.org/D6342

tests: conditionalize output for Python 3 It appears that the random.randint() behavior is different between Python 2 and Python 3. So make the test conditional on that. This makes the test pass on Python 3.7 (and presumably other Python 3 versions). Differential Revision: https://phab.mercurial-scm.org/D6964

hg: remove HGUNICODEPEDANTRY and RTUNICODEPEDANTRY This was added in 73e4a02e6d23 and 89822d7a9d5f as a preliminary way to better support Python 3. With the Python 3 port nearly done and better procedures in place, we shouldn't need it. Differential Revision: https://phab.mercurial-scm.org/D6991

convert: use pycompat.fsencode() This avoids a `TypeError: Can't mix strings and bytes in path components`. Differential Revision: https://phab.mercurial-scm.org/D6965

tests: make tarball output conditional on Python version Python 3.8 changed the default tar file archive format: https://docs.python.org/3.8/whatsnew/3.8.html#tarfile. This commit teaches our tests about the new behavior. Differential Revision: https://phab.mercurial-scm.org/D6986

tests: conditionalize test-run-tests.t for Python 3.8 It looks like Python 3.8 changed the pretty XML formatting slightly to strip some whitespace. Let's teach our tests about that. Differential Revision: https://phab.mercurial-scm.org/D6988

hgweb: use importlib.reload() if available reload() was nuked in Python 3. We need to use importlib.reload() instead. But pyflakes isn't smart enough to detect our conditional usage, so we allow this error. Differential Revision: https://phab.mercurial-scm.org/D6992

tests: allow xrange warning from perf.py The use of xrange in this file is acceptable. We need to make the output optional - and not conditional on the Python version - because `pyflakes` could be executed by any Python version and pyflakes behaves differently depending on the Python version. Differential Revision: https://phab.mercurial-scm.org/D6990

debugsidedata: fix verbose mode on python3 We need to be gently with python3 for it to display this binary data. Differential Revision: https://phab.mercurial-scm.org/D6995

import: add debug messages when parsing data from patch header

examples: include a sample of how to use black with fix It's commented out for now since my patch hasn't landed, but we can uncomment it when that lands. Differential Revision: https://phab.mercurial-scm.org/D6974

black: add a pyproject.toml that contains our black settings Differential Revision: https://phab.mercurial-scm.org/D6993

contrib: stop whining about whitespace around = This was conflicting with black, so dump it. Differential Revision: https://phab.mercurial-scm.org/D6979

contrib: black manages whitespace after Python keywords now Differential Revision: https://phab.mercurial-scm.org/D6978

contrib: allow trailing commas in lists black manages this for us, so we don't need to worry about this anymore. Differential Revision: https://phab.mercurial-scm.org/D6977

contrib: remove check-code line length limit Both Python and C code are managed by formatters now, so relax this. Differential Revision: https://phab.mercurial-scm.org/D6976

changegroup: use positive logic for treemanifest changegroup3 logic We are about to add more cases, in that context, it is simpler to avoid double negative. En addition, We document the situation for the next soul. Differential Revision: https://phab.mercurial-scm.org/D6938

fix: add :enabled sub-config for fixer tools This allows distributing opt-in fixer tool configurations in .hgrc files. This may be useful for adding default configs in core, or for orgranizations that want to provide configs to their users. Tools are still enabled by default because it would be confusing to add a config and find that it has no effect until you add enabled=true. Differential Revision: https://phab.mercurial-scm.org/D6975

automation: add a command to submit to a Try server The CI code for running the Try Server requires more thorough review. Let's add just the client-side bits for submitting to Try so others can start using it. Differential Revision: https://phab.mercurial-scm.org/D6983

automation: improve documentation for credentials management Differential Revision: https://phab.mercurial-scm.org/D6982

automation: switch to us-west-2 by default This is where the AWS account I'm running lives. Differential Revision: https://phab.mercurial-scm.org/D6981

automation: upgrade various packages Ran `pip-compile -U` to upgrade to latest versions. Differential Revision: https://phab.mercurial-scm.org/D6980

statprof: drop python #! and exec bit we don't use For some reason this wasn't flagged until I ran a reformatter on this file. I'm deeply confused by this, but since we don't execute this drop the support for it. Differential Revision: https://phab.mercurial-scm.org/D6985

cvsps: switch a file open to a with statement For some reason this only showed up in check-code after running black. Puzzling. Differential Revision: https://phab.mercurial-scm.org/D6984

templatefuncs: account for user's diffopts in diff() (BC)

transaction: re-wrap line to avoid a black bug Differential Revision: https://phab.mercurial-scm.org/D6970

discovery: re-wrap expression to avoid a black bug Differential Revision: https://phab.mercurial-scm.org/D6969

sidedata: adjust string for python3 This was pointed out by Gregory Szorc: Python 3 will require a r'' because of source transformation. Differential Revision: https://phab.mercurial-scm.org/D6968

py3: fix phabricator's use of json.loads() for py3.5 Missed this in c340a8ac7ef3 since `loads()` takes bytes from 3.6 onwards. Differential Revision: https://phab.mercurial-scm.org/D6961

run-tests: make coverage work out of tree Currently coverage fails when run on an out-of-tree extension since run-tests.py tries to load sitecustomize.py from self._testdir, which is the dir for the extension's tests. Differential Revision: https://phab.mercurial-scm.org/D6960

unfinishedstate: suggested `hg update .` (including `.`) to complete update `hg update` can update to a different and undesired commit. For users who have commands.update.requiredest=yes, it's even an error to run just `hg update. Differential Revision: https://phab.mercurial-scm.org/D6956

setup: build extensions in parallel by default The build_ext distutils command in Python 3.5+ has a "parallel" option that controls whether to build extensions in parallel. It is disabled by default (None) and can be set to an integer value for number of cores or True to indicate use all available CPU cores. This commit changes our build_ext command override to set "parallel" to True unless a value has been provided by the caller. On my machine, this makes `python setup.py build_ext` 1-4s faster. It is worth noting that at this time, each individual source file constituting the extension is still built serially. For Mercurial, this means that we can't build faster than the slowest-to-build extension, which is the zstd extension by a long shot. This means that setup.py is still not very efficient at utilizing multiple cores. But we're better than before. Differential Revision: https://phab.mercurial-scm.org/D6923 # no-check-commit because of foo_bar naming

merge with stable

sidedata: check that the sidedata safely roundtrip We actually use the data we stored during the test. It is a way to make sure their content is not corrupted. Differential Revision: https://phab.mercurial-scm.org/D6898

sidedata: add a debugsidedata command The command provide basic capabilities to inspect revision side data. Differential Revision: https://phab.mercurial-scm.org/D6897

sidedata: test we can successfully write sidedata For now we just write them and rejoice on the lack of crashes. Differential Revision: https://phab.mercurial-scm.org/D6896