Mercurial > hg
view tests/test-install.t @ 46607:e9901d01d135
revlog: add a mechanism to verify expected file position before appending
If someone uses `hg debuglocks`, or some non-hg process writes to the .hg
directory without respecting the locks, or if the repo's on a networked
filesystem, it's possible for the revlog code to write out corrupted data.
The form of this corruption can vary depending on what data was written and how
that happened. We are in the "networked filesystem" case (though I've had users
also do this to themselves with the "`hg debuglocks`" scenario), and most often
see this with the changelog. What ends up happening is we produce two items
(let's call them rev1 and rev2) in the .i file that have the same linkrev,
baserev, and offset into the .d file, while the data in the .d file is appended
properly. rev2's compressed_size is accurate for rev2, but when we go to
decompress the data in the .d file, we use the offset that's recorded in the
index file, which is the same as rev1, and attempt to decompress
rev2.compressed_size bytes of rev1's data. This usually does not succeed. :)
When using inline data, this also fails, though I haven't investigated why too
closely. This shows up as a "patch decode" error. I believe what's happening
there is that we're basically ignoring the offset field, getting the data
properly, but since baserev != rev, it thinks this is a delta based on rev
(instead of a full text) and can't actually apply it as such.
For now, I'm going to make this an optional component and default it to entirely
off. I may increase the default severity of this in the future, once I've
enabled it for my users and we gain more experience with it. Luckily, most of my
users have a versioned filesystem and can roll back to before the corruption has
been written, it's just a hassle to do so and not everyone knows how (so it's a
support burden). Users on other filesystems will not have that luxury, and this
can cause them to have a corrupted repository that they are unlikely to know how
to resolve, and they'll see this as a data-loss event. Refusing to create the
corruption is a much better user experience.
This mechanism is not perfect. There may be false-negatives (racy writes that
are not detected). There should not be any false-positives (non-racy writes that
are detected as such). This is not a mechanism that makes putting a repo on a
networked filesystem "safe" or "supported", just *less* likely to cause
corruption.
Differential Revision: https://phab.mercurial-scm.org/D9952
author | Kyle Lippincott <spectral@google.com> |
---|---|
date | Wed, 03 Feb 2021 16:33:10 -0800 |
parents | ad107ed7a4aa |
children | fa50f208de73 |
line wrap: on
line source
hg debuginstall $ hg debuginstall checking encoding (ascii)... checking Python executable (*) (glob) checking Python implementation (*) (glob) checking Python version (2.*) (glob) (no-py3 !) checking Python version (3.*) (glob) (py3 !) checking Python lib (.*[Ll]ib.*)... (re) checking Python security support (*) (glob) TLS 1.2 not supported by Python install; network connections lack modern security (?) SNI not supported by Python install; may have connectivity issues with some servers (?) checking Rust extensions \((installed|missing)\) (re) checking Mercurial version (*) (glob) checking Mercurial custom build (*) (glob) checking module policy (*) (glob) checking installed modules (*mercurial)... (glob) checking registered compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob) checking "re2" regexp engine \((available|missing)\) (re) checking templates (*mercurial?templates)... (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob) checking commit editor... (*) (glob) checking username (test) no problems detected hg debuginstall JSON $ hg debuginstall -Tjson | sed 's|\\\\|\\|g' [ { "compengines": ["bz2", "bz2truncated", "none", "zlib"*], (glob) "compenginesavail": ["bz2", "bz2truncated", "none", "zlib"*], (glob) "compenginesserver": [*"zlib"*], (glob) "defaulttemplate": "*mercurial?templates?map-cmdline.default", (glob) "defaulttemplateerror": null, "defaulttemplatenotfound": "default", "editor": "*", (glob) "editornotfound": false, "encoding": "ascii", "encodingerror": null, "extensionserror": null, (no-pure !) "hgmodulepolicy": "*", (glob) "hgmodules": "*mercurial", (glob) "hgver": "*", (glob) "hgverextra": "*", (glob) "problems": 0, "pythonexe": "*", (glob) "pythonimplementation": "*", (glob) "pythonlib": "*", (glob) "pythonsecurity": [*], (glob) "pythonver": "*.*.*", (glob) "re2": (true|false), (re) "templatedirs": "*mercurial?templates", (glob) "username": "test", "usernameerror": null, "vinotfound": false } ] hg debuginstall with no username $ HGUSER= hg debuginstall checking encoding (ascii)... checking Python executable (*) (glob) checking Python implementation (*) (glob) checking Python version (2.*) (glob) (no-py3 !) checking Python version (3.*) (glob) (py3 !) checking Python lib (.*[Ll]ib.*)... (re) checking Python security support (*) (glob) TLS 1.2 not supported by Python install; network connections lack modern security (?) SNI not supported by Python install; may have connectivity issues with some servers (?) checking Rust extensions \((installed|missing)\) (re) checking Mercurial version (*) (glob) checking Mercurial custom build (*) (glob) checking module policy (*) (glob) checking installed modules (*mercurial)... (glob) checking registered compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob) checking "re2" regexp engine \((available|missing)\) (re) checking templates (*mercurial?templates)... (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob) checking commit editor... (*) (glob) checking username... no username supplied (specify a username in your configuration file) 1 problems detected, please check your install! [1] hg debuginstall with invalid encoding $ HGENCODING=invalidenc hg debuginstall | grep encoding checking encoding (invalidenc)... unknown encoding: invalidenc exception message in JSON $ HGENCODING=invalidenc HGUSER= hg debuginstall -Tjson | grep error "defaulttemplateerror": null, "encodingerror": "unknown encoding: invalidenc", "extensionserror": null, (no-pure !) "usernameerror": "no username supplied", path variables are expanded (~ is the same as $TESTTMP) $ mkdir tools $ touch tools/testeditor.exe #if execbit $ chmod 755 tools/testeditor.exe #endif $ HGEDITOR="~/tools/testeditor.exe" hg debuginstall checking encoding (ascii)... checking Python executable (*) (glob) checking Python implementation (*) (glob) checking Python version (2.*) (glob) (no-py3 !) checking Python version (3.*) (glob) (py3 !) checking Python lib (.*[Ll]ib.*)... (re) checking Python security support (*) (glob) TLS 1.2 not supported by Python install; network connections lack modern security (?) SNI not supported by Python install; may have connectivity issues with some servers (?) checking Rust extensions \((installed|missing)\) (re) checking Mercurial version (*) (glob) checking Mercurial custom build (*) (glob) checking module policy (*) (glob) checking installed modules (*mercurial)... (glob) checking registered compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob) checking "re2" regexp engine \((available|missing)\) (re) checking templates (*mercurial?templates)... (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob) checking commit editor... ($TESTTMP/tools/testeditor.exe) checking username (test) no problems detected print out the binary post-shlexsplit in the error message when commit editor is not found (this is intentionally using backslashes to mimic a windows usecase). $ HGEDITOR="c:\foo\bar\baz.exe -y -z" hg debuginstall checking encoding (ascii)... checking Python executable (*) (glob) checking Python implementation (*) (glob) checking Python version (2.*) (glob) (no-py3 !) checking Python version (3.*) (glob) (py3 !) checking Python lib (.*[Ll]ib.*)... (re) checking Python security support (*) (glob) TLS 1.2 not supported by Python install; network connections lack modern security (?) SNI not supported by Python install; may have connectivity issues with some servers (?) checking Rust extensions \((installed|missing)\) (re) checking Mercurial version (*) (glob) checking Mercurial custom build (*) (glob) checking module policy (*) (glob) checking installed modules (*mercurial)... (glob) checking registered compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob) checking "re2" regexp engine \((available|missing)\) (re) checking templates (*mercurial?templates)... (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob) checking commit editor... (c:\foo\bar\baz.exe) (windows !) Can't find editor 'c:\foo\bar\baz.exe' in PATH (windows !) checking commit editor... (c:foobarbaz.exe) (no-windows !) Can't find editor 'c:foobarbaz.exe' in PATH (no-windows !) (specify a commit editor in your configuration file) checking username (test) 1 problems detected, please check your install! [1] debuginstall extension support $ hg debuginstall --config extensions.fsmonitor= --config fsmonitor.watchman_exe=false | grep atchman fsmonitor checking for watchman binary... (false) watchman binary missing or broken: warning: Watchman unavailable: watchman exited with code 1 Verify the json works too: $ hg debuginstall --config extensions.fsmonitor= --config fsmonitor.watchman_exe=false -Tjson | grep atchman "fsmonitor-watchman": "false", "fsmonitor-watchman-error": "warning: Watchman unavailable: watchman exited with code 1", Verify that Mercurial is installable with pip. Note that this MUST be the last test in this file, because we do some nasty things to the shell environment in order to make the virtualenv work reliably. On Python 3, we use the venv module, which is part of the standard library. But some Linux distros strip out this module's functionality involving pip, so we have to look for the ensurepip module, which these distros strip out completely. On Python 2, we use the 3rd party virtualenv module, if available. $ cd $TESTTMP $ unset PYTHONPATH #if py3 ensurepip network-io $ "$PYTHON" -m venv installenv >> pip.log Hack: Debian does something a bit different in ensurepip.bootstrap. This makes it so that pip thinks the 'wheel' wheel is installed so it can build wheels; when it goes to try, however, it shells out to run `python3 -u <setup.py>`, that *doesn't* get the 'wheel' wheel, and it fails with an invalid command 'bdist_wheel'. To fix this, we just delete the wheel from where Debian put it in our virtual env. Then pip doesn't think it's installed and doesn't try to build. $ rm installenv/share/python-wheels/wheel-*.whl >/dev/null 2>&1 || true Note: we use this weird path to run pip and hg to avoid platform differences, since it's bin on most platforms but Scripts on Windows. $ ./installenv/*/pip install $TESTDIR/.. >> pip.log Failed building wheel for mercurial (?) WARNING: You are using pip version *; however, version * is available. (glob) (?) You should consider upgrading via the '$TESTTMP/installenv/bin/python* -m pip install --upgrade pip' command. (glob) (?) $ ./installenv/*/hg debuginstall || cat pip.log checking encoding (ascii)... checking Python executable (*) (glob) checking Python implementation (*) (glob) checking Python version (3.*) (glob) checking Python lib (*)... (glob) checking Python security support (*) (glob) checking Rust extensions \((installed|missing)\) (re) checking Mercurial version (*) (glob) checking Mercurial custom build (*) (glob) checking module policy (*) (glob) checking installed modules (*/mercurial)... (glob) checking registered compression engines (*) (glob) checking available compression engines (*) (glob) checking available compression engines for wire protocol (*) (glob) checking "re2" regexp engine \((available|missing)\) (re) checking templates ($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob) checking default template ($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) (glob) checking commit editor... (*) (glob) checking username (test) no problems detected #endif #if virtualenv no-py3 network-io Note: --no-site-packages is the default for all versions enabled by hghave $ "$PYTHON" -m virtualenv installenv >> pip.log DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. (?) DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support (?) Note: we use this weird path to run pip and hg to avoid platform differences, since it's bin on most platforms but Scripts on Windows. $ ./installenv/*/pip install $TESTDIR/.. >> pip.log DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. (?) DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support (?) DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality. (?) $ ./installenv/*/hg debuginstall || cat pip.log checking encoding (ascii)... checking Python executable (*) (glob) checking Python implementation (*) (glob) checking Python version (2.*) (glob) checking Python lib (*)... (glob) checking Python security support (*) (glob) TLS 1.2 not supported by Python install; network connections lack modern security (?) SNI not supported by Python install; may have connectivity issues with some servers (?) checking Rust extensions \((installed|missing)\) (re) checking Mercurial version (*) (glob) checking Mercurial custom build (*) (glob) checking module policy (*) (glob) checking installed modules (*/mercurial)... (glob) checking registered compression engines (*) (glob) checking available compression engines (*) (glob) checking available compression engines for wire protocol (*) (glob) checking "re2" regexp engine \((available|missing)\) (re) checking templates ($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob) checking default template ($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) (glob) checking commit editor... (*) (glob) checking username (test) no problems detected #endif