Matt Harbison <matt_harbison@yahoo.com> [Mon, 03 Dec 2018 12:48:42 -0500] rev 40832
extdiff: avoid double backslashes in the displayed tool path on Windows
This shows the tool path in the help, and changed in 67b180c0e263. uirepr()
already does the same thing, but that undoes the mangling in its call to repr().
Boris Feld <boris.feld@octobus.net> [Wed, 28 Nov 2018 05:06:58 +0100] rev 40831
contrib: add a helper script that help to build interesting repositories
The script is dedicated to building a couple of repositories that should be
interesting to run discovery from one another. It seems a common enough need
to contribute it upstream.
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 03 Dec 2018 19:42:46 +0300] rev 40830
py3: listify filter() to call len() on it
Differential Revision: https://phab.mercurial-scm.org/D5354
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Nov 2018 18:35:31 +0900] rev 40829
loggingutil: document openlogfile()
This function will be used later for command-server logging.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Nov 2018 18:25:37 +0900] rev 40828
loggingutil: extract openlogfile() and proxylogger to new module
This module isn't placed under the "utils" package since it needs "ui" to
process things. It's called "loggingutil", not "logutil" because the word
"log" is too obscure in our codebase.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Nov 2018 18:21:39 +0900] rev 40827
blackbox: pass in options to _openlogfile() as arguments
This prepares for extracting utility function from the blackbox module.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Nov 2018 22:10:27 +0900] rev 40826
blackbox: just try writing to repo.vfs and update lastlogger on success
This is simpler and more robust. Before, an empty ".hg" directory would be
created if it's removed after checking vfs.isdir('.').
Yuya Nishihara <yuya@tcha.org> [Tue, 20 Nov 2018 22:31:12 +0900] rev 40825
vfs: add option to not create parent directories implicitly
In blackbox, we don't want to create a ".hg" directory by mistake. This
provides a race-safe option to achieve that.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 02:55:33 +0100] rev 40824
repo: add a `wcachevfs` to access the `.hg/wcache/` directory
This wvfs will allow us to migrate various cache to the new `wcache` directory.
Helping with cache issues with "share".
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 02:46:31 +0100] rev 40823
cache: create `wcache` directory at init time
The cache directory will be needed very quickly, so it seems simpler to create
it early to make sure it has the same owner and permission than the other
directory in the repository.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 02:38:55 +0100] rev 40822
cache: create `cache` directory at init time
The cache directory will be needed very quickly, so it seems simpler to create
it early to make sure it has the same owner and permission than the other
directory in the repository.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 17:08:23 +0100] rev 40821
check-exec: write file in 'wcache' instead of 'cache'
Some cache are relevant or affected by the working copy used. So the `.hg/cache`
directory is not the best place for them because multiple shared repository can
end up fighting over them.
To address this issue, we introduce a new 'wcache' directory to host this kind
of cache.
The first user are the `checkisexec` type file. These files describe property of
the working copy and fit the use-case well.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:09:44 +0100] rev 40820
mmapindex: set default to 1MB
mmapping index is more efficient if we only need a small part of it.
The 1MB value has been picked arbitrarily, a lower value might be better.
On a large repository with a 60MB index, we see the following performance
gain:
hg perfindex
before: ! wall 0.032023 comb 0.040000 user 0.000000 sys 0.040000 (best of 100)
after: ! wall 0.000196 comb 0.000000 user 0.000000 sys 0.000000 (best of 1060)
The speed boost benefit all cases, including the one where the full index
needs to be parsed.
hg perfindex --rev 0
before: ! wall 0.040673 comb 0.030000 user 0.000000 sys 0.030000 (best of 100)
after ! wall 0.010713 comb 0.020000 user 0.010000 sys 0.010000 (best of 212)
This gain reflect in higher level operation:
hg perfbookmarks --clear-revlogs
before: ! wall 0.161339 comb 0.160000 user 0.130000 sys 0.030000 (best of 56)
after: ! wall 0.123228 comb 0.120000 user 0.120000 sys 0.000000 (best of 68)
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:07:33 +0100] rev 40819
mmapindex: move the 'mmapindexthreshold' option out of experimental
The option is useful and should be advertised more. We move it out of
experimental as a first step. The `storage` section is selected as this is
related to how the storage is accessed. A new 'performance' section might be
more appropriate.
We move from 'mmapindexthreshold` to `mmap-threshold` as non-index item are
also suitable for mmap (eg: the rev-branch-cache).
If relevant, we can introduce sub-option `mmap-threshold.revlog-index` later.
Boris Feld <boris.feld@octobus.net> [Sat, 01 Dec 2018 15:57:27 +0100] rev 40818
perf: add a --rev attribute to perfindex
This allow for benchmarking the time necessary to look for other version than
the tip.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:03:38 +0100] rev 40817
perf: update perfindex to be more realistic
The previous code was creating a revlog manually, we now use the actual
`localrepo` method to create it.
We have to jump though extra hops to work around the impact of filecache.
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:09:46 -0800] rev 40816
match: drop unnecessary wrapping of regex in group
It seems the regexes have been wrapped in an unnamed group since
b6c42714d900 (Add locate command., 2005-07-05). In that commit, the
grouping was needed because there was a "head" ('^') added before the
group and a "tail" (os.sep) added after it. It seems the head was
moved inside the group in 1c0c413cccdd (Get add and locate to use new
repo and dirstate walk code., 2005-07-18) and the tail was moved
inside the group in 89985a1b3427 (Clean up walk and changes code to
use normalised names properly., 2005-07-31), So it seems to me that
we've carried around the unnecessary group for 13 years. This patch
removes it.
Differential Revision: https://phab.mercurial-scm.org/D5352
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:45:20 -0800] rev 40815
match: use _BASE_SIZE instead of magic value 4
Differential Revision: https://phab.mercurial-scm.org/D5351
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:44:49 -0800] rev 40814
match: make "groupsize" include the trailing "|"
I think this is a little easier to follow and it will simplify later
patches too.
Differential Revision: https://phab.mercurial-scm.org/D5350
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:09:43 -0800] rev 40813
match: fix an unaligned (but harmless) indent
Differential Revision: https://phab.mercurial-scm.org/D5349
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:41:10 +0100] rev 40812
match: raise an Abort error instead of OverflowError
This case of OverflowError (one single pattern being too large) has never been
properly caught in the past.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 21:02:02 +0100] rev 40811
match: avoid translating glob to matcher multiple times for large sets
For hgignore with many globs, the resulting regexp might not fit under the 20K
length limit. So the patterns need to be broken up in smaller pieces.
Before this change, the logic was re-starting the full process from scratch
for each smaller pieces, including the translation of globs into regexp.
Effectively doing the work over and over.
If the 20K limit is reached, we are likely in a case where there is many such
glob, so exporting them is especially expensive and we should be careful not
to do that work more than once.
To work around this, we now translate glob to regexp once and for all. Then,
we assemble the resulting individual regexp into valid blocks.
This raises a very significant performance win for large `.hgignore file`:
Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66)
After: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100)
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:25:49 +0100] rev 40810
match: extract function that group regexps
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:16:05 +0100] rev 40809
match: test for overflow error in pattern
If a single pattern is too large to handle, we raise an exception. This case is
now doctested.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:20:32 +0100] rev 40808
match: extract a literal constant into a symbolic one
Matt Harbison <matt_harbison@yahoo.com> [Sat, 01 Dec 2018 21:42:48 -0500] rev 40807
tests: apply binary mode to output in seq.py
I noticed this when playing with running tests using WSL, and iterating over the
output yielded '0\r', '1\r',... Most of the other *.py tools do this, and `seq`
on MSYS lacks '\r' in the output, so this is more consistent.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 01:09:37 +0100] rev 40806
perf: add a `--clear-caches` to `perfbranchmapupdate`
This flag will help to measure the time we spend loading various cache that
support the branchmap update.
Example for an 500 000 revisions repository:
hg perfbranchmapupdate --base 'not tip' --target 'tip'
! wall 0.000860 comb 0.000000 user 0.000000 sys 0.000000 (best of 336)
hg perfbranchmapupdate --base 'not tip' --target 'tip' --clear-caches
! wall 0.029494 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 21:11:47 +0000] rev 40805
perf: start from an existing branchmap if possible
If the --base set if a superset of one of the cached branchmap, we should use as
a starting point. This greatly help the overall runtime of
`hg perfbranchmapupdate`
For example, for a repository with about 500 000 revisions, using this trick
make the command runtime move from about 200 second to about 10 seconds. A 20x
gain.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 20:35:22 +0000] rev 40804
perf: rely on repoview for perfbranchmapupdate
Using 'repoview' matching the base and target subset make the benchmark more
realistic. It also unlocks optimization to make the command initialization
faster.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 22:56:06 +0100] rev 40803
perf: pre-indent some code in `perfbranchmapupdate`
This make the next patch easier to read.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 12:02:25 +0000] rev 40802
perf: add a `perfbranchmapupdate` command
This command benchmark the time necessary to update the branchmap between two
sets of revisions. This changeset introduce a first version, doing nothing fancy
regarding cache or other internal details.
Anton Shestakov <av6@dwimlabs.net> [Mon, 05 Nov 2018 13:52:19 +0800] rev 40801
push: config option to control behavior when pushing to a publishing server
Pushing to a publishing server by mistake can lead to a difficult situation to
solve because evolution doesn't work on public changesets. This new
experimental config tries to help avoiding unintentionally (or at least being
aware of) pushing to publishing remotes.
`hg push --publish` can be used to make push succeed even when auto-publish is
set to 'abort'.
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 30 Nov 2018 17:42:55 +0300] rev 40800
narrowcommands: remove an unrequired `repo.narrowpats` call
We call that few lines above and do nothing significant in between which can
change the narrowpats. So let's use values returned by that call.
Differential Revision: https://phab.mercurial-scm.org/D5348
Augie Fackler <augie@google.com> [Thu, 29 Nov 2018 16:44:01 -0500] rev 40799
manifest: reject lines shorter than 42 bytes, not 22
Yuya correctly spotted during the review of f27f8e9ef1e73 that we're
dealing with hexlified hashes here, and so it should be 42 bytes not
22.
Differential Revision: https://phab.mercurial-scm.org/D5347
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 20:05:38 +0900] rev 40798
blackbox: initialize logger with repo instance
The blackboxlogger is unusable without a repo. Let's simply initialize it
with a repo instance.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Nov 2018 20:56:25 +0900] rev 40797
blackbox: do not nullify repo to deactivate the logger on failure
The _repo will be a mandatory attribute. Instead, make the logger to not
track any events.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 20:02:34 +0900] rev 40796
blackbox: extract global last logger to proxylogger class
So the blackboxlogger can be instantiated with a repo.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 19:36:21 +0900] rev 40795
ui: pass in bytes opts dict to logger.log()
This is the convention of the Mercurial API.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 19:35:33 +0900] rev 40794
ui: pass in formatted message to logger.log()
This makes sure that all logger instances will handle the message arguments
properly.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:34:46 +0900] rev 40793
blackbox: send debug message to logger by core ui
Since the core ui.log() may recurse into ui.log() through ui.debug(), it
must guard against recursion.
The ui extension class can finally be removed.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Nov 2018 20:23:50 +0900] rev 40792
blackbox: change the way of deactivating the logger on write error
This prepares for the upcoming code move. The recursion guard will be ported
to the core ui.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 28 Nov 2018 10:12:50 -0800] rev 40791
match: remove obsolete catching of OverflowError
Since 0f6a1bdf89fb (match: handle large regexes, 2007-08-19), we catch
an OverflowError from the regex engine and split up the regex if that
happens. In 59a9dc9562e2 (ignore: split up huge patterns, 2008-02-11),
that was extended to raise an OverflowError in our code even if the
regex engine doesn't raise it. It's unclear if there was a range of
regex sizes where the OverflowError would be raised from the regex
engine but that were still below the limit we added in our
code. Either way, both limitations were probably removed in Python
2.7.4 when the regex code width was extended from 16bit to 32bit (or
Py_UCS4) integer (thanks to Yuya for finding that out).
If at least the first limitation was removed, we no longer should be
using OverflowError for flow control, so this patch changes that.
Differential Revision: https://phab.mercurial-scm.org/D5309
Boris Feld <boris.feld@octobus.net> [Tue, 27 Nov 2018 02:10:14 +0100] rev 40790
sparse: raise a move verbose index error from the C code
If we don't like a value we should print it.
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 23:10:56 +0300] rev 40789
narrow: drop the bundle2 capability since we have server capabilities (BC)
This patch drops the narrow bundle2 capabilities since we introduced narrow
server capabilities which are more nice and now used everywhere.
I am not sure what it can affect, so on safe side I marked this as BC. Also I
removed the NARROWCAP constant as that kind of conflicts with the same name
constant in wireprototypes.py.
Differential Revision: https://phab.mercurial-scm.org/D4892
Boris Feld <boris.feld@octobus.net> [Sun, 02 Jul 2017 04:06:24 +0200] rev 40788
vfs: extract the audit path logic into a submethod
This will make it possible to apply it in more cases.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 20:01:28 +0100] rev 40787
subrepo-git: use an official origvfs when appropriate
The origvfs has the auditor properly set and can move file without issue.
The current code is currently working without errors because rename are not
audited, yet.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 19:26:05 +0100] rev 40786
revert: extract origvfs logic in a sub-function
The subrepo's "revert" logic could benefit from it.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 18:44:07 +0100] rev 40785
vfs: treat 'undo.' file the same as 'journal.' file
They are the same kind of file, they are protected by the store lock, but
directly lives inside the '.hg' directory.
No warnings were ever raised about them because `vfs.rename` is not audited.
Something we are trying to change.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 21:00:13 +0100] rev 40784
perf: add a perfignore command
The command is meant to benchmark operations related to hgignore. Right now the
command is benchmarking the loading time of the hgignore rules.
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:36:06 +0300] rev 40783
py3: use pycompat.xrange instead of xrange
xrange does not exists on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5302
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 27 Nov 2018 16:16:13 +0300] rev 40782
store: write fncache only once if there are both adds and removes
Differential Revision: https://phab.mercurial-scm.org/D5307
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 17:44:24 +0000] rev 40781
perf: disable revlogs clearing in `perftags` by default
This aligns things with what `perfbookmarks` does. I decided to disable the
revlogs clearing by default to focus on the core logic by default, ignoring
side effects.
If we prefer to emphasize the side effect, we can instead keep this on in
`perftags` and enable it by default in `perfbookmarks`.
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 10:55:20 +0000] rev 40780
perf: add a `clear-revlogs` flag to `perfbookmarks`
This flag (off by default) makes it possible to enable the refresh of the
changelog and revlog. This is useful to check for costly side effects of
bookmark loading.
Usually, these side effects are shared with other logics (eg: tags).
example output in my mercurial repo (with 1 bookmark, so not a great example):
$ hg perfbookmarks
! wall 0.000044
$ hg perfbookmarks --clear-revlogs
! wall 0.001380
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 10:38:15 +0000] rev 40779
tags: cache `repo.changelog` access when checking tags nodes
The tags reading process checks if the nodes referenced in tags exist. Caching
the access to `repo.changelog` provides a large speedup for repositories with
many tags.
running `hg perftags` in a large private repository
before: ! wall 0.393464 comb 0.390000 user 0.330000 sys 0.060000 (median of 25)
after: ! wall 0.267711 comb 0.270000 user 0.210000 sys 0.060000 (median of 38)
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 10:46:20 +0000] rev 40778
perf: add a `clear-revlogs` flag to `perftags`
This flag (on by default) makes it possible to disable the refresh of the
changelog and revlog. This is useful to check for the time spent in the core
tags logic without the associated side effects. Usually, these side effects
are shared with other logics (eg: bookmarks).
Example output in my Mercurial repository
$ hg perftags
! wall 0.017919 comb 0.020000 user 0.020000 sys 0.000000 (best of 141)
$ hg perftags --no-clear-revlogs
! wall 0.012982 comb 0.010000 user 0.010000 sys 0.000000 (best of 207)
Boris Feld <boris.feld@octobus.net> [Sun, 25 Nov 2018 13:37:53 +0100] rev 40777
perf: stop creating new revlog by hand in perftags
It's better to let the repository logic create its own object. We now just clear
the cache. New object will be automatically created from there.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:23:12 +0100] rev 40776
revlog: update the documentation for `trim_endidx`
The function role drifted since the function was commented.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:21:09 +0100] rev 40775
revlog: properly detect corrupted revlog in `index_get_length`
Pointed out by Yuya Nishihara.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:15:12 +0100] rev 40774
perf: rename `perfhelper-tracecopies` to `perfhelper-pathcopies`
The command it supports is called `perfpathcopies`. It seems better to align the
names.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:13:50 +0100] rev 40773
perf: add a docstring to `perfpathcopies`
This will help people to find this command.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:08:11 +0100] rev 40772
revlog: update the docstring of `ancestors` to match reality
Code using this method expect the revision to be (reverse) sorted. As pointed by
Yuya Nishihara, the docstring should reflect that.
Augie Fackler <augie@google.com> [Mon, 26 Nov 2018 15:53:34 -0500] rev 40771
remotefilelog: fix typo in docstring
Differential Revision: https://phab.mercurial-scm.org/D5306
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 23 Nov 2018 18:58:16 +0300] rev 40770
store: append to fncache if there are only new files to write
Before this patch, if we have to add a new entry to fncache, we write the whole
fncache again which slows things down on large fncache which have millions of
entries. Addition of a new entry is common operation while pulling new files or
commiting a new file.
This patch adds a new fncache.addls set which keeps track of the additions
happening and store them. When we write the fncache, we will just read the addls
set and append those entries at the end of fncache.
We make sure that the entries are new entries by loading the fncache and making
sure entry does not exists there. In future if we can check if an entry is new
without loading the fncache, that will speed up things more.
Performance numbers for commiting a new file:
mercurial repo
before: 0.08784651756286621
after: 0.08474504947662354
mozilla-central
before: 1.83314049243927
after: 1.7054164409637451
netbeans
before: 0.7953150272369385
after: 0.7202838659286499
pypy
before: 0.17805707454681396
after: 0.13431048393249512
In our internal repo, the performance improvement is in seconds.
I have used octobus's ASV perf benchmark thing to get the above numbers. I also
see some minute perf improvements related to creating a new commit without a new
file, but I believe that's just some noise.
Differential Revision: https://phab.mercurial-scm.org/D5301
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:38:35 +0300] rev 40769
py3: fix couple of division operator to do integer divison
Differential Revision: https://phab.mercurial-scm.org/D5305
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:37:48 +0300] rev 40768
py3: use dict.items() instead of dict.iteritems()
dict.iteritems() does not exist on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5304
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:36:51 +0300] rev 40767
py3: convert strings to bytes in tests/test-remotefilelog-histpack.py
Differential Revision: https://phab.mercurial-scm.org/D5303
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 23 Nov 2018 21:54:03 +0300] rev 40766
py3: replace str() with pycompat.bytestr() or ('%d' % int)
After this patch, test-fastannotate-diffopts.t is about to pass. There are some
extra newlines in the output.
Differential Revision: https://phab.mercurial-scm.org/D5300
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 19:32:51 +0100] rev 40765
perf: add a `--timing` argument to `perfhelper-tracecopies`
The new argument will help picking better pair for benchmarking. See
documentation for details.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 18:08:33 +0900] rev 40764
ui: manage logger instances and event filtering by core ui
The setup code in blackbox needs more tweaks since it has lots of black
magics. I'll fix them by follow-up patches.
To be clear, the goal of this series is to provide a proper way for command
server to install its own logger. I need it to debug in-memory repository
cache.
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 21:10:51 +0900] rev 40763
extensions: add "uipopulate" hook, called per instance, not per process
In short, this is the "reposetup" function for ui. It allows us to modify
ui attributes without extending ui.__class__. Before, the only way to do
that was to abuse the config dictionary, which is copied across ui instances.
See the next patch for usage example.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Nov 2018 19:11:45 +0900] rev 40762
hgweb: load globally-enabled extensions explicitly
Before, extensions were loaded as a side effect of hg.repository() if the
hgweb was executed as a CGI/WSGI. I want to make it explicit so that another
ui hook can be inserted after extensions.loadall().
Matt Harbison <matt_harbison@yahoo.com> [Sat, 24 Nov 2018 14:11:02 -0500] rev 40761
tests: disable remotefilelog on Windows
I've spent a non trivial amount of time trying to eliminate the test errors, but
it's looking like this is pretty dependent on Unix support. For example, there
are attempts to delete open files, and uses of threads that report I/O attempts
on closed files. (Maybe this is a race condition? Don't we usually use
processes as workers on Windows?)
In any event, I don't want real new errors elsewhere to be masked by these known
problems.
For some reason $CACHEDIR is reported as missing in test-remotefilelog-repack.t,
but it actually exists in the hgcloneshallow call inside
shallowutil.mkstickygroupdir(). By the time the process exits, it's gone. I
don't see it being removed by code that calls 'rmdir' or 'remove' in the
extension itself.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 23:48:44 +0100] rev 40760
perf: run 'setup' function during stub run
The benchmarked function might need the content of the setup to be run in order
to function properly.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 02:08:01 +0100] rev 40759
perf: fallback to subset if ondisk cache is missing in perfbranchmapload
If there is no branchmap on disk for that filter, it means that the cache from
some subset's filter is relevant for this one. We look for it instead of
aborting.
That way it is much simpler to run the command in an automated way. We can now
add it to `test-contrib-perf.t`.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 02:11:11 +0100] rev 40758
perf: prewarm the branchmap in perfbranchmapload
It is not very interesting to have the command randomly failing because the
branchmap for the tested filter happens to be cold. So we make sure to have a
valid up to date branchmap before going further.
The data might still be missing from disk if a subset was equivalent. See next
changeset for details and fix.
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Nov 2018 22:39:54 +0900] rev 40757
localrepo: correct docstring of filectx()
The same reason as b6c2543e1dd8. It can't be any changeset specifiers but
revision number.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 20:05:55 +0900] rev 40756
annotate: remove dead code to not convert path to relative path
It's annotate. There should be at least one file path specified.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 18:41:44 +0900] rev 40755
annotate: start pager soon after command options are validated
It helps extracting helper class.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 18:38:44 +0900] rev 40754
annotate: check and update command options first
It helps extracting helper class.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 18:36:44 +0900] rev 40753
annotate: format local tables in less-dense form
I'll move these to a class, where one-extra indent will be needed.
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 22 Nov 2018 18:14:21 +0300] rev 40752
perf: rename version flag of perfchangegroupchangelog to cgversion
The version flag conflicts with the global version flag and does not work.
Differential Revision: https://phab.mercurial-scm.org/D5297
Differential Revision: https://phab.mercurial-scm.org/D5298
Matt Harbison <matt_harbison@yahoo.com> [Fri, 23 Nov 2018 22:27:57 -0500] rev 40751
procutil: don't allow the main 'hg' script to be treated as the Windows exe
Previously, there were a handful of errors like this:
$ hg prefetch --repack
(running background incremental repack)
+ abort: %1 is not a valid Win32 application
+ [255]
CreateProcess() doesn't append .exe when `lpApplicationName` contains a path,
and a python script isn't directly executable.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 23 Nov 2018 22:18:56 -0500] rev 40750
remotefilelog: drop some compatibility cruft for finding the hg exeutable
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 21 Nov 2018 17:17:26 +0300] rev 40749
branchmap: build the revbranchcache._namesreverse() only when required
On big repositories with a lot of named branches and that also increasing over
time, building of this dict can be expensive and shows up in profile.
For our internal repository, this saves ~0.05 seconds.
Thanks to Yuya for suggesting using util.propertycache() and
util.clearcachedproperty().
Differential Revision: https://phab.mercurial-scm.org/D5291
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 11:11:38 +0100] rev 40748
sparse-revlog: put the native implementation of slicechunktodensity to use
When possible, the C implementation of delta chain slicing will be used.
providing a large boost in performance for this operation.
To take a practical example of restoring manifest revision '59547c40bc4c' for
a reference NetBeans repository (using sparse-revlog). The media time of the
step `slice-sparse-chain` of `perfrevlogrevision` improve from 0.660 ms to
0.098 ms;
The full series move delta chain slicing from 1.120 ms to 0.098 ms;
Implementing _slicechunktosize into C would yield further improvements.
However, the performance seems good enough for now.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 11:10:52 +0100] rev 40747
cext: increment the `parsers` module version
We introduced a new `slicechunktodensity` method to the index.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 11:09:58 +0100] rev 40746
sparse-revlog: introduce native (C) implementation of slicechunktodensity
This is a C implementation of `_slicechunktodensity` in the
`mercurial/revlogutils/deltas.py` file.
The algorithm involves a lot of integer manipulation and low-level access to
index data. Having a C implementation of it raises a large performance
improvement. See later changeset in this series for details.
Boris Feld <boris.feld@octobus.net> [Fri, 09 Nov 2018 18:45:23 +0100] rev 40745
sparse-revlog: add a `trim_endidx` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first.
This function implement a subpart of the python function `_trimchunk` in
`mercurial/revlogutils/deltas.py`. Handling of actual Python objects is left
to the caller function.
Boris Feld <boris.feld@octobus.net> [Fri, 09 Nov 2018 18:43:16 +0100] rev 40744
sparse-revlog: add a `index_segment_span` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first. This new function provides
an efficient way to retrieve some of the information needed by
`slicechunktodensity`.
Boris Feld <boris.feld@octobus.net> [Fri, 09 Nov 2018 18:42:58 +0100] rev 40743
sparse-revlog: add a `index_get_length` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first. This new function provides
an efficient way to retrieve some of the information needed by
`slicechunktodensity`.
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 14:49:18 +0000] rev 40742
sparse-revlog: add a `index_get_start` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first. This new function provides
an efficient way to retrieve some of the information needed by
`slicechunktodensity`.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:32:32 +0100] rev 40741
perf: add --clear-revlog flag to branchmapload
Having the changelog index already loaded when loading the branchmap can have a
large impact on performance.
Example runs (large private repository):
hg perfbranchmapload -f base
! wall 0.116722 comb 0.120000 user 0.110000 sys 0.010000 (best of 59)
hg perfbranchmapload -f base --clear-revlogs
! wall 0.258246 comb 0.230000 user 0.220000 sys 0.010000 (best of 31)
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:32:28 +0100] rev 40740
perf: introduce a function to fully "unload" a changelog
The function remove various attributes and caches related to changelog.
This is getting a common requirement.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 01:32:57 +0100] rev 40739
perf: use an explicit function in perfbranchmapload
This make things clearer.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 01:20:18 +0100] rev 40738
perf: update function name to match `perfbranchmapload` command
Having function with the same name as the command is simpler.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 11:46:47 +0000] rev 40737
perf: fix a minor typo in perfbranchmapload
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:37 +0200] rev 40736
context: floor adjustlinkrev graph walk during copy tracing
The `_adjustlinkrev` method gains an optional "stoprev" argument. The linkrev
adjustment will give up once this floor is reached. The relevant functions
using `_adjustlinkrev` are updated to pass an appropriate value in the copy
tracing code.
In some private repository, about 10% of the status call triggered the
pathological case addressed by this change. The speedup varies from one call
to another, the best-observed win is moving from 170s to 11s.
The effect of this change can be seen in the public pypy repository, running the
following command:
hg perftracecopies --source 83c9ff0c0206 --destination 59c79103d5b0
before: 3.401753 seconds
after: 2.634897 seconds (-23%)
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 14:30:58 +0000] rev 40735
context: small refactoring of `isintroducedafter`
This make the next change clearer.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:36 +0200] rev 40734
context: split `introrev` logic in a sub function
We want to add a mechanism to stop iteration early associated to intro rev early
in some case. However, it does not make sense to expose it in the public
`filectx` API. So we split the code into an internal method instead.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:35 +0200] rev 40733
context: introduce an `isintroducedafter` method and use it in copies
Right now, copy tracing make effort to not traverse the graph too much to save
performance. It uses a "limit" acting as a floor revision past which data are
no longer relevant to the current copy tracing.
However, to enforce this limit, it does a call to `filectx.rev()` and that
call can trigger a graph traversal on its own. That extra graph traversal is
unaware of the current limit and can become very expensive. That cost is
increased by the nature of work done in adjust link rev, we are not only
walking down the graph, we are also checking the affected file for each
revision we walk through. Something significantly more expensive than the walk
itself.
To work around this we need to make the `filectx` operation aware of the
current limit. The first step is to introduce a dedicated method:
`isintroducedafter`. We'll then rework that method logic to stop traversal as
soon as possible.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:34 +0200] rev 40732
context: take advantage of `_descendantrev` in introrev if available
Before this changeset, `_descendantrev` was ignored and `introrev` could
return a "wrong" result. I was previously fine because there seems to be no
existing code using both `introrev` and `_descendantrev` at the same time.
However, we would like to change that.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:49:30 +0200] rev 40731
context: spell out the logic around linkrev adjustement starting point
We make the intent of the `_changeid` and `_changectx` checks explicit. The
same logic was previously performed by the `self.rev()` call. The new code is
a bit redundant, but much clearer.
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 14:14:56 +0000] rev 40730
perf: add a new `perfhelper-tracecopies` command
The command is not measuring performance itself, it digs interesting statistic
to help pick good arguments for the `perfcopytrace` command.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 21 Sep 2018 09:19:42 -0700] rev 40729
narrow: extract helper for parsing narrowspec file
This will be used for parsing a file that's stored in a different
location.
Differential Revision: https://phab.mercurial-scm.org/D5277
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Nov 2018 22:40:22 +0900] rev 40728
py3: cast attribute name to sysstr in clearcachedproperty()
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 21 Nov 2018 19:12:12 +0300] rev 40727
py3: whitelist three more passing test caught by buildbot
The number of failing test is back to two digits.
Differential Revision: https://phab.mercurial-scm.org/D5293
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 21 Nov 2018 13:08:23 -0500] rev 40726
tests: make test-check-module-imports more robust
It failed for me without this in this way:
tests/test-commandserver.t:19: relative import of stdlib module
tests/test-lfs-serve.t:108: relative import of stdlib module
tests/test-lfs-serve.t:255: relative import of stdlib module
tests/test-lfs-serve.t:362: relative import of stdlib module
tests/test-lfs-serve.t:406: relative import of stdlib module
tests/test-lock.py:9: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-lrucachedict.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-match.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-remotefilelog-datapack.py:15: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-remotefilelog-histpack.py:14: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-simplekeyvaluefile.py:4: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-sshserver.py:6: imports not lexically sorted: silenttestrunner < unittest, True, True
This is because every module is considered a stdlib module, because
the stdlib_prefixes is /usr, and my repo is in /usr/local/home, which
means that sys.path contains a couple of
/usr/local/home/../hg/.. entries that count as "in the stdlib".
Fix this by preventing any path in sys.path that's inside the mercurial
source from being considered "in the stdlib".
Differential Revision: https://phab.mercurial-scm.org/D5294
Anton Shestakov <av6@dwimlabs.net> [Sat, 27 Oct 2018 21:13:23 +0800] rev 40725
push: add --publish flag to change phase of pushed changesets
This flag is copied from topic extension, where it proved to be useful. It
makes all pushed changesets public on both ends if the push succeeds.
Doesn't currently work for any subrepos that are implicitly pushed together
with their owner repo.
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 17:19:54 +0000] rev 40724
match: reformat `syntaxes` dictionary for better maintainability
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 18:54:44 +0000] rev 40723
match: provide and use a quick way to escape a single byte
The previous function has a lot of overhead (including being a function). In
the `_globre` case, we always escape a single byte. So we provide a dictionary
dedicated to this use case. We directly use the dictionary to avoid a function
call, these are expensive in Python.
Again, this raise a very significant performance gain:
Before: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100)
After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146)
Total improvement for the full series:
Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66)
After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146)
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 16:50:21 +0000] rev 40722
perf: make `clearfilecache` helper work with any object
If the object is not a localrepo, it won't have an `unfiltered` method (and
won't need one).
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 10:10:25 +0000] rev 40721
perf: move some of the perftags benchmark to the setup function
Creating fresh objects and clearing the cache should not be part of the
timing.
before: ! wall 0.020851 comb 0.020000 user 0.020000 sys 0.000000 (median of 138)
after: ! wall 0.018740 comb 0.020000 user 0.020000 sys 0.000000 (median of 141)
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 23:14:46 +0000] rev 40720
perf: use the new setup function in "perfbookmarks"
This command was picked arbitrarily to display the usefulness of the new
feature. In my Mercurial repository (with very few bookmarks), moving cache
cleanup in the dedicated setup function has a visible and stable effect on
the benchmark number.
before: ! wall 0.000061 comb 0.000000 user 0.000000 sys 0.000000 (median of 40837)
after: ! wall 0.000058 comb 0.000000 user 0.000000 sys 0.000000 (median of 40500)
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 22:55:07 +0000] rev 40719
perf: add a `setup` argument to run code outside of the timed section
With this new argument, it is possible to perform special setup and cleanup
outside of code actually timed. This is useful to provide more accurate
benchmark.
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 23:02:29 +0000] rev 40718
perf: explicitly pass title as a keyword argument in `perfdiffwd`
This will help to update the timer function arguments in a later changeset.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:59:23 +0900] rev 40717
ui: unify argument name of ui.log()
It's called an "event" in both blackbox and logtoprocess.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:45:18 +0900] rev 40716
logtoprocess: extract logger class from ui wrapper
It provides the same interface as the blackboxlogger. The ui wrapper will
be removed shortly.
Yuya Nishihara <yuya@tcha.org> [Wed, 21 Nov 2018 21:40:16 +0900] rev 40715
merge with stable
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 19 Nov 2018 21:12:13 +0300] rev 40714
py3: use node.hex(m.digest()) instead of m.hexdigest()
hashlib.sha1.hexdigest() returns str on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5287
Boris Feld <boris.feld@octobus.net> [Sun, 18 Nov 2018 02:40:47 +0100] rev 40713
tests: add Balto configuration file
I have been developing a Mercurial test runner plugin for being able to run
Mercurial tests with Balto
(https://bitbucket.org/lothiraldan/balto/src/default/).
Balto requires a configuration file so let's include it, I have added the
required information in the configuration file as comments.
I hope Balto would be an helpful tool for other people than me.
Differential Revision: https://phab.mercurial-scm.org/D5283
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Mon, 19 Nov 2018 13:40:03 -0500] rev 40712
tests: make test-alias.t pass with re2
Locally, these "non-ASCII character in alias" errors don't show up,
though I get them when the alias is defined at the command line rather
than in an hgrc.
The brokenness comes from the fact that hgrcs are parsed with regexes,
and re/re2 differ in this way:
$ python -c 'import re; print(re.compile("(.*)").match("aaa\xc0bbbb").groups())'
('aaa\xc0bbbb',)
$ python -c 'import re2; print(re2.compile("(.*)").match("aaa\xc0bbbb").groups())'
('aaa',)
Apparently re2 stops when it encounters invalid utf8 (which I suppose makes sense
given that '.' matches what appears to be a codepoint rather than a byte). This is
presumably a bug in hg, but not very important, so just change the test to stick
to valid utf8.
Differential Revision: https://phab.mercurial-scm.org/D5288
Martin von Zweigbergk <martinvonz@google.com> [Mon, 19 Nov 2018 23:08:09 -0800] rev 40711
context: remove seemingly impossible code branch
I'm not a Python expert, but I can't think of a way that the following
branch can ever be hit:
def _changeid(self):
if r'_changeid' in self.__dict__:
return self._changeid
It seems to me that if that condition is true, then this function
would not have been called. The only exception I can think of is if a
reference to the function had been stored beforehand, something like this:
c = fctx.__dict__['_changeid']
fctx._changeid
c()
But that seems like very unlikely code to exist.
The condition was added in 921b64e1f7b9 (filecontext: use 'is not
None' to check for filelog existence, 2013-05-01) as a "bonus" change
(in addition to what the patch was actually about)
Differential Revision: https://phab.mercurial-scm.org/D5289
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 19 Nov 2018 21:11:08 +0300] rev 40710
py3: use pycompat.byteskwargs() to covert keys of kwargs to bytes
Differential Revision: https://phab.mercurial-scm.org/D5286
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 19 Nov 2018 20:30:07 +0300] rev 40709
py3: use '%d' for integers instead of '%s'
Differential Revision: https://phab.mercurial-scm.org/D5285
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 19 Nov 2018 19:57:11 +0300] rev 40708
py3: add 9 new passing tests caught by buildbot
Differential Revision: https://phab.mercurial-scm.org/D5284
Anton Shestakov <av6@dwimlabs.net> [Fri, 09 Nov 2018 13:57:13 +0800] rev 40707
branch: allow changing branch of merge commits with --rev
Tests show that changing branch of merge commits works fine with evolution and
without, so let's allow it. Other safeguards should prevent users from shooting
themselves in the foot.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 22:28:38 -0500] rev 40706
lfs: ensure that the return of urlopener.open() is closed
No problem observed, just an oversight noticed while reading documentation.
Kyle Lippincott <spectral@google.com> [Thu, 15 Nov 2018 11:16:42 -0800] rev 40705
changegroup: avoid instantiating storage if we are not using it
Differential Revision: https://phab.mercurial-scm.org/D5280
Matt Harbison <matt_harbison@yahoo.com> [Fri, 16 Nov 2018 17:56:36 -0500] rev 40704
http: allow 'auth.prefix' to have a username consistent with the URI
It may be a little weird to put a username in the prefix, but the documentation
doesn't disallow it, and silently disallowing it has caused confusion[1]. The
username must match what is passed in (which seems to be from the URI via a
circuitous route), as well as 'auth.username' if it was specified. I thought
about printing a warning for a mismatch, but we already don't print a warning if
the 'auth.username' and URI username don't match.
This change allows the first and second last new test cases to work as expected.
It looks like this would have been a problem since at least 0593e8f81c71.
[1] https://www.mercurial-scm.org/pipermail/mercurial/2018-November/051069.html
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 18:14:57 -0500] rev 40703
lfs: make the exception messages consistent
I don't love that it repeats 'HTTP Error' in an already long message, but I
doubt that we should assume that it will always say that on the original
exception message.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 18:08:29 -0500] rev 40702
lfs: handle URLErrors to add additional information
Sometimes the blob server is hit first (e.g. on push), and sometimes it's hit
last (e.g. pull). Throw in depth first subrepo operations, and things quickly
get insane. It wasn't even mentioning LFS, so just saying "connection refused"
can be confusing- especially if the blob server is a secondary server and
connecting to the repo server works.
The exception handler for the transfer handler will print the full path to the
blob, but that seems fine given that it might be necessary to debug a second
server. (We don't yet support a standalone blob server, so the handler for the
Batch API will cover 99.9% of the current problems. But it might as well be
handled now while I'm thinking about it.)
The function for translating to a message was mostly borrowed from
scmutil.catchall().
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 17:58:59 -0500] rev 40701
lfs: improve the hints for common errors in the Batch API
The previous message was too debug-ish and less action oriented than a hint
should be. The remaining errors that aren't handled are more along the lines of
programming errors (not using POST, bad accept type, etc), so I'm not bothering
with that.
The friendly errors purposely use `self.baseurl` instead of the full Batch API
endpoint because I'd expect some copy/paste/modify on the part of the user here,
and it would be more confusing if '/objects/batch' magically appeared, but
shouldn't be used in the config setting. It still seems like the right thing
for debugging in the catchall case.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 17:55:01 -0500] rev 40700
lfs: provide more Batch API error info via a hint in the raised exception
A coworker had a typo in `lfs.url`, forgot it was even set because usually the
blob server is inferred, and then got a 404. It would have been easier to debug
with the failing URL printed.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 17:50:14 -0500] rev 40699
scmutil: display the optional hint when handling StorageError in catchall()
Other than CensoredNodeError (which is also a StorageError), it looks like all
exceptions with a hint display them. I'm not sure that it makes sense to have a
hint for censored nodes, so I'm not bothering with that.
It looks like nobody is using this yet, as the tests don't change.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 14:57:26 +0100] rev 40698
sparse-revlog: align endrevidx usages in the _slicechunktosize
All "startrevidx..endrevidx" ranges in this function are now half-open.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 14:55:11 +0100] rev 40697
sparse-revlog: use `span` variable as intended
The variable was planned to be used in the while condition but was not used
yet.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 17:38:51 -0500] rev 40696
tests: stabilize test-commandserver.t on Windows
It looks like new test coverage in 054d0fcba2c4, rather than a code change.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 15 Nov 2018 17:36:15 -0500] rev 40695
histedit: conditionalize the imports of 'fcntl' and 'termios'
The recent import of chistedit in c36175456350 made Windows sad. I'm not sure
if there's other stuff that needs to be done here (e.g. change the default
interface), but this makes the tests run again.
It would have been nicer if the error message indicated these modules were the
problem, but instead it said "*** failed to import extension histedit: No module
named histedit". I'm not sure if there's anything we can do about that.
Boris Feld <boris.feld@octobus.net> [Fri, 16 Nov 2018 14:21:47 +0100] rev 40694
logtoprocess: update commandfinish options arguments
d2c997b8001f changed the logtoprocess API with the effect of not exposing the
positional arguments to the logtoprocess scripts anymore.
We have some scripts that use the duration and return code of the
"commandfinish" event to monitor hg calls. Update the logging of the
"commandfinish" to expose those values as options argument, which will be
accessible as `OPT_RETURN_CODE` and `OPT_DURATION` in logtoprocess arguments.
The code has been formatted with Black.
Differential Revision: https://phab.mercurial-scm.org/D5282
Kyle Lippincott <spectral@google.com> [Thu, 15 Nov 2018 13:16:46 -0800] rev 40693
rebase: fix two ui.logs to actually have text when using default blackbox log
Some implementations of ui.log record structured information along with the
ui.log which can be used for metrics, but ui.log() as implemented by the
blackbox logging does not do anything special with this, and we end up with a
log line with no text (not even a line break) so it ends up looking something
like:
date time user @node (pid) [rebase]> date time user @node (pid) ...
Differential Revision: https://phab.mercurial-scm.org/D5279
Martin von Zweigbergk <martinvonz@google.com> [Thu, 15 Nov 2018 11:22:32 -0800] rev 40692
wireprotov2server: let repo.narrowmatch(match) do matcher intersection
This is supported since 4fd0fac48922 (localrepo: allow narrowmatch()
to accept matcher to intersect with, 2018-09-28).
Differential Revision: https://phab.mercurial-scm.org/D5281
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:29:46 +0900] rev 40691
blackbox: extract function to test if log event is tracked
This will be a required method of the logger interface.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:25:34 +0900] rev 40690
blackbox: initialize inlog flag properly
And ditch the "bb" prefix as it's no longer a ui extension class.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:24:28 +0900] rev 40689
blackbox: initialize repo attribute properly
And ditch the "bb" prefix as it's no longer a ui extension class.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:22:14 +0900] rev 40688
blackbox: unindent "if True" block
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:17:49 +0900] rev 40687
blackbox: extract logger class from ui wrapper
This moves most functions to new blackboxlogger class. The ui wrapper will
be removed later.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 16:58:22 +0900] rev 40686
blackbox: rename variables to prepare extracting core logic from ui wrapper
I'm going to add ui.setlogger() function so that I can enable logging feature
in command server without extending ui.__class__. This prepares for it.
"self" will be a logger instance, so this patch renames some of them to "ui".
Boris Feld <boris.feld@octobus.net> [Fri, 09 Nov 2018 17:58:37 +0100] rev 40685
sparse-revlog: rework the way we enforce chunk size limit
We move from a O(N) algorithm to a O(log(N)) algorithm.
The previous algorithm was traversing the whole delta chain, looking for the
exact point where it became too big. This would result in most of the delta
chain to be traversed.
Instead, we now use a "binary" approach, slicing the chain in two until we
have a chunk of the appropriate size.
We still keep the previous algorithm for the snapshots part. There are few of
them and they are large bits of data distant from each other. So the previous
algorithm should work well in that case.
To take a practical example of restoring manifest revision '59547c40bc4c' for
a reference NetBeans repository (using sparse-revlog). The media time of the
step `slice-sparse-chain` of `perfrevlogrevision` improve from 1.109 ms to
0.660 ms.
Boris Feld <boris.feld@octobus.net> [Tue, 13 Nov 2018 15:06:29 +0100] rev 40684
doctest: add a `issnapshot` method to _testrevlog
We'll need it soon.
Boris Feld <boris.feld@octobus.net> [Tue, 13 Nov 2018 14:41:04 +0100] rev 40683
tests: add `revlogutils.deltas` module to doctests
The doctest in these module have been from `mercurial.revlog` but the module was
not added to the doctests.
Spotted by Yuya Nishihara.
Yuya Nishihara <yuya@tcha.org> [Thu, 15 Nov 2018 20:20:31 +0900] rev 40682
merge with stable
joco <joco@google.com> [Mon, 05 Nov 2018 22:58:19 +0100] rev 40681
mergetools: adjust Beyond Compare config on Mac/Linux
Set the labels of the Linux and Mac versions of Beyond Compare from
Mercurial's builtin variables, same as the Windows version.
Differential Revision: https://phab.mercurial-scm.org/D5255
Anton Shestakov <av6@dwimlabs.net> [Wed, 14 Nov 2018 15:05:38 +0800] rev 40680
rewriteutil: move publicrevs closer to where it's used
Martin von Zweigbergk <martinvonz@google.com> [Wed, 14 Nov 2018 11:30:46 -0800] rev 40679
requires: use atomictemp=True when writing .hg/requires
We use an unusual file system at Google that allows writes (and
renames) but not deletions (for certain paths). That causes problems
when writing the requires files without atomictemp=True. There doesn't
appear to be any real drawbacks to using atomictemp, so I'm hoping we
can just change it in core.
Differential Revision: https://phab.mercurial-scm.org/D5274
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 16:47:28 +0900] rev 40678
blackbox: extract _log() function which is called after lastui is resolved
This makes sure that self is the solo ui instance used in _log().
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 16:44:30 +0900] rev 40677
blackbox: inline temporary variables which are referenced only once
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 16:43:29 +0900] rev 40676
blackbox: simply update global lastui variable at once
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 16:38:43 +0900] rev 40675
blackbox: consolidate conditions for early return
Just pick the lastui only if it is usable.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 16:34:49 +0900] rev 40674
blackbox: remove redundant check for unassigned repo
Since ui._bbvfs is looked through ui._bbrepo, the repo instance should
exist if ui._bbvfs isn't None.
Augie Fackler <augie@google.com> [Wed, 14 Nov 2018 10:15:28 -0500] rev 40673
tests: fix bytes/str issue I introduced when adding this test
# skip-blame just b prefixes for py3
Differential Revision: https://phab.mercurial-scm.org/D5271
Kyle Lippincott <spectral@google.com> [Tue, 13 Nov 2018 17:14:47 -0800] rev 40672
shelve: use matcher to restrict prefetch to just the modified files
Shelve currently operates by:
- make a temp commit
- identify all the bases necessary to shelve, put them in the bundle
- use exportfile to export the temp commit to the bundle ('file' here means
"export to this fd", not "export this file")
- remove the temp commit
exportfile calls prefetchfiles, and prefetchfiles uses a matcher to restrict
what files it's going to prefetch; if it's not provided, it's alwaysmatcher.
This means that `hg shelve` in a remotefilelog repo can possibly download the
file contents of everything in the repository, even when it doesn't need to. It
luckily is restricted to the narrowspec (if there is one), but this is still a
lot of downloading that's just unnecessary, especially if there's a "smart"
VCS-aware filesystem involved.
exportfile is called with exactly one revision to emit, so we're just
restricting it to prefetching the files from that revision. The base revisions
having separate files should not be a concern since they're handled already;
example:
commit 10 is draft and modifies foo/a.txt and foo/b.txt
commit 11 is draft and modifies foo/a.txt
my working directory that I'm shelving modifies foo/b.txt
By the time we get to exportfile, commit 10 and 11 are already handled, so the
matcher only specifying foo/b.txt does not cause any problems. I verified this
by doing an `hg unbundle` on the bundle that shelve produces, and getting the
full contents of those commits back out, instead of just the files that were
modified in the shelve.
Differential Revision: https://phab.mercurial-scm.org/D5268
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Nov 2018 12:32:05 -0800] rev 40671
revlog: automatically read from opened file handles
The revlog reading code commonly opens a new file handle for
reading on demand. There is support for passing a file handle
to revlog.revision(). But it is marked as an internal argument.
When revlogs are written, we write() data as it is available. But
we don't flush() data until all revisions are written.
Putting these two traits together, it is possible for an in-process
revlog reader during active writes to trigger the opening of a new
file handle on a file with unflushed writes. The reader won't have
access to all "available" revlog data (as it hasn't been flushed).
And with the introduction of the previous patch, this can lead to
the revlog raising an error due to a partial read.
I witnessed this behavior when applying changegroup data (via
`hg pull`) before issue6006 was fixed via different means. Having
this and the previous patch in play would have helped cause errors
earlier rather than manifesting as hash verification failures.
While this has been a long-standing issue, I believe the relatively
new delta computation code has tickled it into being more common.
This is because the new delta computation code will compute deltas
in more scenarios. This can lead to revlog reading. While the delta
computation code is probably supposed to reuse file handles, it
appears it isn't doing so in all circumstances.
But the issue runs deeper than that. Theoretically, any code can
access revision data during revlog writes. It appears we were just
getting lucky that it wasn't. (The "add revision callback" passed to
addgroup() provides an avenue to do this.)
If I changed the revlog's behavior to not cache the full revision
text or to clear caches after revision insertion during addgroup(),
I was able to produce crashes 100% of the time when writing changelog
revisions. This is because changelog's add revision callback attempts
to resolve the revision data to access the changed files list. And
without the revision's fulltext being cached, we performed a revlog
read, which required opening a new file handle. This attempted to read
unflushed data, leading to a partial read and a crash.
This commit teaches the revlog to store the file handles used for
writing multiple revisions during addgroup(). It also teaches the
code for resolving a file handle when reading to use these handles,
if available. This ensures that *any* reads (regardless of their
source) use the active writing file handles, if available. These
file handles have access to the unflushed data because they wrote it.
This allows reads to complete without issue.
Differential Revision: https://phab.mercurial-scm.org/D5267
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Nov 2018 12:30:59 -0800] rev 40670
revlog: detect incomplete revlog reads
_readsegment() is supposed to return N bytes of revlog revision
data starting at a file offset. Surprisingly, its behavior before
this patch never verified that it actually read and returned N
bytes! Instead, it would perform the read(), then return whatever
data was available. And even more surprisingly, nothing in the
call chain appears to have been validating that it received all
the data it was expecting.
This behavior could lead to partial or incomplete revision chunks
being operated on. This could result in e.g. cached deltas being
applied against incomplete base revisions. The delta application
process would happily perform this operation. Only hash
verification would detect the corruption and save us.
This commit changes the behavior of raw revlog reading to validate
that we actually read() the number of bytes that were requested.
We will raise a more specific error faster, rather than possibly
have it go undetected or manifest later in the call stack, at
delta application or hash verification.
Differential Revision: https://phab.mercurial-scm.org/D5266
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 30 Oct 2018 16:50:05 -0700] rev 40669
revlog: use single file handle when de-inlining revlog
_getsegmentforrevs() will eventually call into _datareadfp() to
resolve a file handle to read revision data. If no file handle
is passed into _getsegmentforrevs(), it opens a new one.
Explicit is better than implicit.
This commit changes _enforceinlinesize() to open a file handle
explicitly when converting inline revlogs to split revlogs and
to pass this file handle into _getsegmentforrevs().
I haven't measured, but this change should improve performance,
as we no longer reopen the revlog for reading for every revision
in the revlog when it is converted from inline to split. Instead,
we open it at most once and use it for the duration of the
operation. That being said, I /think/ the chunk cache may mitigate
the number of file opens required.
Differential Revision: https://phab.mercurial-scm.org/D5265
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:44:09 +0300] rev 40668
store: raise ProgrammingError if unable to decode a storage path
Right now, the function magically return False which is dangerous, so let's
raise ProgrammingError.
Suggested by Augie in D5139.
Differential Revision: https://phab.mercurial-scm.org/D5264
Matt Harbison <matt_harbison@yahoo.com> [Tue, 13 Nov 2018 23:54:23 -0500] rev 40667
tests: document a known failing interaction between narrow and lfs
This is one of the two remaining aborts I found looking into issue5794. I've
got no idea what's wrong with the hook, since the changes there fixed the other
two problems noted in that bug report. It seems like it might go away when the
narrow issue is fixed, but let's make sure this doesn't get lost.
The stacktrace for the hook seems to indicate that the missing file *is* in ctx:
remote: Traceback (most recent call last):
remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in checkrequireslfs
remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr>
remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
remote: File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 191, in filectxislfs
remote: return _islfs(self.filelog(), self.filenode())
remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode
remote: return self._filenode
remote: File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__
remote: result = self.func(obj)
remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode
remote: return self._filelog.lookup(self._fileid)
remote: File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 68, in lookup
remote: self._revlog.indexfile)
remote: File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 218, in fileidlookup
remote: raise error.LookupError(fileid, identifier, _('no match found'))
remote: LookupError: data/inside2/f.i@f59b4e021835: no match found
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:55:58 +0900] rev 40666
logtoprocess: drop support for ui.log() call with invalid msg arguments (BC)
Before, the logtoprocess extension put a formatted message into $MSG1, and
its arguments to $MSG2... If the specified arguments couldn't be formatted
because of a caller bug, an unformatted message was passed in to $MSG1
instead of exploding. This behavior doesn't make sense.
Since I'm planning to formalize the ui.log() interface such that we'll no
longer have to extend the ui class, I want to remove any features not
conforming to the ui.log() API. So this patch removes the support for
ill-formed arguments, and $MSG{n} (where n > 1) parameters which seems
useless as long as the message can be formatted. The $MSG1 variable isn't
renamed for the maximum compatibility.
In future patches, a formatted msg will be passed to a processlogger object,
instead of overriding the ui.log() function.
.. bc::
The logtoprocess extension no longer supports invalid ``ui.log()``
arguments. A log message is always formatted and passed in to the
``$MSG1`` environment variable.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:35:38 +0900] rev 40665
py3: byte-stringify inline extension in test-logtoprocess.t
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:33:14 +0900] rev 40664
logtoprocess: rewrite dict building in py3-compatible way
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:27:23 +0900] rev 40663
logtoprocess: leverage procutil.shellenviron() to stringify variables (BC)
This should make the extension more Py3 friendly. The environment variables
of the main process are copied to the dict by shellenviron().
.. bc::
Boolean options passed to the logtoprocess extension are now formatted
as ``0`` or ``1`` instead of ``None``, ``False``, or ``True``.
Yuya Nishihara <yuya@tcha.org> [Tue, 13 Nov 2018 22:15:30 +0900] rev 40662
templater: check invalid use of list expression properly (issue5920)
The error message is still cryptic, but it should be better.
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:08:55 +0300] rev 40661
py3: add b suffix to make sure file is opened in bytes mode
Differential Revision: https://phab.mercurial-scm.org/D5263
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:08:17 +0300] rev 40660
py3: don't use dict.iterkeys()
dict.iterkeys() is not present on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5262
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:07:21 +0300] rev 40659
py3: use node.hex(h.digest()) instead of h.hexdigest()
hashlib.sha1.hexdigest() returns str on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5261
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 17:41:54 +0300] rev 40658
py3: make second argument of fdopen() a str
# skip-blame as just r'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5260
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 17:41:26 +0300] rev 40657
py3: fix keyword arguments handling in hgext/remotefilelog/
Keys of kwargs on Python 3 should be strings. This patch fixes them by appending
r'' prefixes, and using pycompat.byteskwargs() and pycompat.strkwargs().
Differential Revision: https://phab.mercurial-scm.org/D5259
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 05:09:18 +0300] rev 40656
py3: add __pycache__ to .hgignore
Something changed with Python 3 just like million on things. Maybe they should
named Python 3 as anaconda or cobra instead of just increasing the version
number.
This makes test-custom-filters.t pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5039
Boris Feld <boris.feld@octobus.net> [Thu, 08 Nov 2018 16:07:16 +0100] rev 40655
sparse-revlog: stop using a heap to track selected gap
Same logic as for 'gapsheap', we don't actually need a heap.
Boris Feld <boris.feld@octobus.net> [Thu, 08 Nov 2018 16:01:30 +0100] rev 40654
sparse-revlog: stop using a heap to track gaps
The heap doesn't bring any performance advantage as we can simply sort the
final list.
Moreover, the lesser complexity helps a lot when we later implement it in C.
Boris Feld <boris.feld@octobus.net> [Thu, 08 Nov 2018 15:29:58 +0100] rev 40653
sparse-revlog: fast-path before computing payload size
In this fast-path case, we do not need to compute the full delta chain
payload. Since it comes with a significant cost, we avoid doing so if
possible.
Boris Feld <boris.feld@octobus.net> [Tue, 16 Oct 2018 14:46:28 +0200] rev 40652
sparse-revlog: drop unused deltainfo parameter from segmentspan
We no longer need to deal with slicing logic including uncommitted revision.
We drop the associated code.
Boris Feld <boris.feld@octobus.net> [Tue, 16 Oct 2018 14:43:51 +0200] rev 40651
sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
We no longer need to deal with slicing logic including uncommitted revision. We
drop the associated code.
Boris Feld <boris.feld@octobus.net> [Mon, 15 Oct 2018 15:45:08 +0200] rev 40650
sparse-revlog: skip the span check in the sparse-revlog case
This significantly improves the performance on unbundling on smaller
repositories.
Mercurial: unbundling 1K revisions
no-sparse-revlog: 500 ms
sparse-revlog-before: 689 ms
sparse-revlog-after: 484 ms
Pypy: unbundling 1K revisions
no-sparse-revlog: 1.242 s
sparse-revlog-before: 1.135 s
sparse-revlog-after: 0.860 s
NetBeans: unbundling 1K revisions
no-sparse-revlog: 1.386 s
sparse-revlog-before: 2.368 s
sparse-revlog-after: 1.191 s
Mozilla: unbundling 1K revisions
no-sparse-revlog: 3.103 s
sparse-revlog-before: 3.367 s
sparse-revlog-after: 3.093 s
Augie Fackler <augie@google.com> [Wed, 17 Oct 2018 17:15:42 -0400] rev 40649
histedit: import chistedit curses UI from hg-experimental
I don't tend to like curses interfaces, but this gets enough use at
work that it seems like it's worth bringing into core. This is a
minimal import from hg-experimental revision 4c7f33bf5f00, in that
I've done the smallest amount of code movement and editing in order to
import the functionality.
.. feature::
`hg histedit` will now present a curses UI if curses is available
and `ui.interface` or `ui.interface.histedit` is set to `curses`.
Differential Revision: https://phab.mercurial-scm.org/D5146
Augie Fackler <augie@google.com> [Mon, 12 Nov 2018 20:32:58 -0500] rev 40648
manifest: perform cheap checks before potentially allocating memory
Differential Revision: https://phab.mercurial-scm.org/D5258
Augie Fackler <augie@google.com> [Mon, 12 Nov 2018 20:31:57 -0500] rev 40647
manifest: also reject obviously-too-short lines when parsing lines
Differential Revision: https://phab.mercurial-scm.org/D5257
Augie Fackler <augie@google.com> [Mon, 12 Nov 2018 20:35:22 -0500] rev 40646
manifest: make sure there's a filename before bothering to look for newline
There's no valid manifest that would have no characters before the NUL byte on
a line, and this fixes some erratic timeouts in the fuzzer.
Differential Revision: https://phab.mercurial-scm.org/D5256
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 11:12:56 -0500] rev 40645
revlog: replace PyInt_AS_LONG with a more portable helper function
PyInt_AS_LONG disappears on Python, and our previous #define was
producing some problems on Python 3. Let's give up and make an inline
helper function that makes this more sane.
Differential Revision: https://phab.mercurial-scm.org/D5235
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 22:51:36 +0900] rev 40644
help: document weird behavior of uisetup()
While refactoring ui.log() API, it was really annoying that uisetup() is
actually an extsetup() the phase 1. Let's document that. I'm planning to
add another uisetup() which is called per ui instance, though I can't think
of a good name for it.
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 22:26:24 +0900] rev 40643
help: merge section about uisetup() and extsetup()
They are technically the same callback, called only once per process.
The section name "ui setup" is confusing, so shouldn't be used.
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 21:28:54 +0900] rev 40642
help: add internals.extensions topic
I think it's better to include the API overview in core as we now have the
internals section in our help system.
Retrieved from the wiki, and formatted as reST. Several wiki links are
removed since they are invalid in the help. The sections about example
extension are removed at all as they seemed too verbose.
https://www.mercurial-scm.org/wiki/WritingExtensions
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Nov 2018 12:51:14 +0900] rev 40641
commandserver: send raw progress information to message channel
This is pretty basic implementation to support GUI progress bar.
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Nov 2018 12:38:49 +0900] rev 40640
commandserver: make getpass() request distinct from normal prompt
Otherwise, GUI clients would have to parse the prompt text.
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Nov 2018 12:17:20 +0900] rev 40639
commandserver: attach prompt default and choices to message
These attributes are important to provide a GUI prompt to user.
Yuya Nishihara <yuya@tcha.org> [Thu, 08 Nov 2018 22:25:04 +0900] rev 40638
test-commandserver: clean up quoting and location of dbgui extension
This helps embedding '$' in the script.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Jan 2015 17:55:28 +0900] rev 40637
ui: extract helpers to write message with type or label
This provides a 'type' attribute to command-server clients, which seems
more solid than relying on 'ui.<type>' labels. In future patches,
type='progress' will be added to send raw progress information.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Jan 2015 18:49:59 +0900] rev 40636
commandserver: add experimental option to use separate message channel
This is loosely based on the idea of the TortoiseHg's pipeui extension,
which attaches ui.label to message text so the command-server client can
capture prompt text, for example.
https://bitbucket.org/tortoisehg/thg/src/4.7.2/tortoisehg/util/pipeui.py
I was thinking that this functionality could be generalized to templating,
but changed mind as doing template stuff would be unnecessarily complex.
It's merely a status message, a simple serialization option should suffice.
Since this slightly changes the command-server protocol, it's gated by a
config knob. If the config is enabled, and if it's supported by the server,
"message-encoding: <name>" is advertised so the client can stop parsing
'o'/'e' channel data and read encoded messages from the 'm' channel. As we
might add new message encodings in future releases, client can specify a list
of encoding names in preferred order.
This patch includes 'cbor' encoding as example. Perhaps, 'json' should be
supported as well.
Yuya Nishihara <yuya@tcha.org> [Wed, 07 Nov 2018 22:37:51 +0900] rev 40635
commandserver: fix reference before assignment error in pipeservice cleanup
Spotted by the next patch.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Jan 2015 18:43:23 +0900] rev 40634
dispatch: pass around ui.fmsg channel
This will be set by the command server. See the next patch.
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Nov 2018 12:01:21 +0900] rev 40633
test-subrepo: document why share magically works even if subrepos are missing
I was confused how it's working while reviewing fb490d798be0, "share: reload
repo after adjusting it in postshare()."
Augie Fackler <raf@durin42.com> [Sun, 11 Nov 2018 14:13:08 -0500] rev 40632
packaging: add Fedora 29 target
Differential Revision: https://phab.mercurial-scm.org/D5254
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Nov 2018 11:52:32 +0900] rev 40631
store: pass in decoded filename to narrow matcher
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Nov 2018 11:47:36 +0900] rev 40630
test-narrow-clone-stream: include no usefncache case which is broken
Since 9aeb9e2d28a7, encoded filenames are filtered by the narrow matcher,
which is clearly wrong.
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Nov 2018 11:36:22 +0900] rev 40629
test-narrow-clone-stream: include uppercase letter in filename
This makes encoded filenames differ from the original names.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 10 Nov 2018 13:37:18 -0500] rev 40628
phase: use the `phases.cmdphasenames` constant to walk available command flags
This reverts 1ea6772fb415.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 09 Nov 2018 23:19:53 -0500] rev 40627
phases: add a list of names usable by the phase command
This is useful in TortoiseHg for example, which has until recently just added
all names in the `phasenames` list to the context menu that changes the phase.
The ones not supported by the command would just error out.
Boris Feld <boris.feld@octobus.net> [Wed, 07 Nov 2018 19:59:38 +0100] rev 40626
transaction: display data about why the transaction failed to rollback
We saw more of these a while back. Having more data available would be nice.
Yuya Nishihara <yuya@tcha.org> [Sat, 10 Nov 2018 12:07:26 +0900] rev 40625
merge with stable
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Nov 2018 10:47:24 -0800] rev 40624
remotefilelog: avoid accessing repo instance after dispatch
Upstream commit c5e6c1ba1c79 (hg: don't reuse repo instance after
unshare(), 2018-09-12) poisoned the repo instance after
unshare(). That made `hg unshare` fail with remotefilelog because we
tried to close the fileserverclient after dispatch by accessing it via
the repo. This patch fixes that by storing the reference to the
fileserverclient at the beginning of dispatch.
An analogous patch was sent for remotefilelog version in FB's
hg-experimental as D5246.
Differential Revision: https://phab.mercurial-scm.org/D5253
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Nov 2018 11:28:05 -0800] rev 40623
share: avoid a now-unnecessary reload of the repo
hg.share() now returns an already-reloaded repo, so let's just use
that. (This would have failed test-subrepo-recursion.t without the
previous patch.)
Differential Revision: https://phab.mercurial-scm.org/D5252
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Nov 2018 10:46:02 -0800] rev 40622
share: reload repo after adjusting it in postshare()
When sharing a repo that's using remotefilelog, the update that happens
at the end of the `hg share` call does not see the remote repo path
that's copied in hg.postshare(). This patch reloads the repo after
hg.postshare() to address that.
This changes a subrepo test case. Note that `hg share -U; hg co tip`
worked there before, so I don't see see why `hg share` should fail. I
also don't know what a "locally referenced subrepo". So maybe this is
fixing a bug? Hopefully it's not breaking something someone actually
cares about at least. Maybe someone who knows and cares about subrepos
can review this.
Differential Revision: https://phab.mercurial-scm.org/D5251
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Nov 2018 09:36:16 -0800] rev 40621
tests: don't load strip extension via mq extension for remotefilelog tests
The proper way to get the `hg strip` command has been via the "strip"
extension since 2013.
Differential Revision: https://phab.mercurial-scm.org/D5250
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Nov 2018 09:35:05 -0800] rev 40620
remotefilelog: remove unused configs for {data,history}pack version
Differential Revision: https://phab.mercurial-scm.org/D5249
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Nov 2018 09:27:37 -0800] rev 40619
tests: remove obsolete PYTHONPATH override from remotefilelog tests
They were added in
https://bitbucket.org/facebook/hg-experimental/commits/fdcad37a6a68cfbaf5920f3eeaa0cc9cae42bd26,
which says this:
When remotefilelog moved from its own repo, the tests needed to be updated to
adjust the PYTHONPATH to ensure the in-repo remotefilelog was loaded instead of
the system one.
This meant any local runs of remotefilelog tests would've been using the system
remotefilelog unless the user had manually set the PYTHONPATH themselves.
That doesn't seem relevant with remotefilelog in core.
Differential Revision: https://phab.mercurial-scm.org/D5248
Martin von Zweigbergk <martinvonz@google.com> [Fri, 09 Nov 2018 08:19:08 -0800] rev 40618
tests: drop obsolete "hginit" alias in remotefilelog tests
Differential Revision: https://phab.mercurial-scm.org/D5247
Danny Hooper <hooper@google.com> [Thu, 08 Nov 2018 12:35:26 -0800] rev 40617
fix: add extra field to fixed revisions to avoid creating obsolescence cycles
The extra field prevents sequential invocations of fix from producing the same
hash twice. Previously, this could cause problems because it would create an
obsolescence cycle instead of the expected new successor.
This change also adds an explicit check for whether a new revision should be
committed. Until now, the code relied on memctx.commit() to quietly do nothing
if the node already exists. Because of the new extra field, this no longer
covers the case where we don't want to replace an unchanged node.
Differential Revision: https://phab.mercurial-scm.org/D5245
Danny Hooper <hooper@google.com> [Thu, 08 Nov 2018 12:29:56 -0800] rev 40616
cleanup: use revision numbers instead of hashes in test output
This allows changes to the hashes produced by fix to not needlessly modify this
area of the test.
Differential Revision: https://phab.mercurial-scm.org/D5244
Kyle Lippincott <spectral@google.com> [Wed, 07 Nov 2018 15:42:56 -0800] rev 40615
tests: make the commands.resolve.confirm tests not pollute the test-wide hgrc
I want to add tests after these tests that create a new repo and use it, and was
confused for a bit as to why they were seeing different behavior than I
expected.
Differential Revision: https://phab.mercurial-scm.org/D5242
Kyle Lippincott <spectral@google.com> [Wed, 07 Nov 2018 15:41:18 -0800] rev 40614
tests: fix a couple typos in test-resolve.t comments and add a comment
Differential Revision: https://phab.mercurial-scm.org/D5241
Danny Hooper <hooper@google.com> [Tue, 06 Nov 2018 15:50:41 -0800] rev 40613
fix: add suboption for configuring execution order of tools
This could be accomplished by using wrapper scripts, but that would diminish
the usefulness of the incremental formatting logic. Configuring execution order
along with other things in the hgrc is probably more convenient anyway.
This change highlights some awkwardness with suboptions and default values,
which should be addressed separately.
Differential Revision: https://phab.mercurial-scm.org/D5237
Augie Fackler <augie@google.com> [Wed, 07 Nov 2018 14:21:39 -0500] rev 40612
tests: fix up some import statements caught by Python 3
I'm curious how the import checker manages to be so much more
pedantic in Python 3, but not enough to bother exploring.
Differential Revision: https://phab.mercurial-scm.org/D5240
Martin von Zweigbergk <martinvonz@google.com> [Wed, 07 Nov 2018 10:29:38 -0800] rev 40611
tests: replace `tac` reimplementation by `sort -r`
`sort -r` is better code "formatter" than `tac` since it's
stable. It's also portable so we don't need to reimplement it in
Python.
Differential Revision: https://phab.mercurial-scm.org/D5239
Augie Fackler <augie@google.com> [Wed, 07 Nov 2018 12:15:36 -0500] rev 40610
tests: work around `tac` not being portable
Introduce a tac.py helper and use it. Sigh.
Differential Revision: https://phab.mercurial-scm.org/D5238
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 11:22:16 -0500] rev 40609
revlog: give formatting to clang-format
This has two lines that are poorly formatted, both right after
PyObject_HEAD macros. It's possible that in the future there will be a
feature to tell clang-format that a token includes an end-of-line
character (in this case the ;), but for now I'm willing to live with
two poorly-formatted lines in a struct definition in exchange for not
thinking about how this file is formatted.
Differential Revision: https://phab.mercurial-scm.org/D5234
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 11:19:35 -0500] rev 40608
revlog: add blank line in comment to help clang-format
Differential Revision: https://phab.mercurial-scm.org/D5233
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 11:18:12 -0500] rev 40607
revlog: add a comment to help clang-format produce less-awful results
Differential Revision: https://phab.mercurial-scm.org/D5232
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 11:52:41 -0500] rev 40606
py3: ratchet caught two more passing tests
Differential Revision: https://phab.mercurial-scm.org/D5236
Boris Feld <boris.feld@octobus.net> [Mon, 05 Nov 2018 15:57:31 +0100] rev 40605
perf: add a lazydeltabase option to perfrevlogwrite
The option controls if we'll test the provided delta first. This is the current
default behavior for unbundle.
Boris Feld <boris.feld@octobus.net> [Mon, 05 Nov 2018 15:24:09 +0100] rev 40604
perf: add `storage` as possible source for perfrevlogwrite
This source will use the stored delta.
Boris Feld <boris.feld@octobus.net> [Mon, 05 Nov 2018 15:19:44 +0100] rev 40603
perf: add `parent-smallest` as possible source for perfrevlogwrite
This source will use the smallest of the possible diff against parent.
Boris Feld <boris.feld@octobus.net> [Mon, 05 Nov 2018 15:15:18 +0100] rev 40602
perf: add `parent-2` as possible source for perfrevlogwrite
This source will use a diff against p2 if it exists and fall back to p1
otherwise.
Boris Feld <boris.feld@octobus.net> [Mon, 05 Nov 2018 15:15:02 +0100] rev 40601
perf: add `parent-1` as possible source for perfrevlogwrite
This source will use a diff against p1 in all case.
Boris Feld <boris.feld@octobus.net> [Fri, 19 Oct 2018 17:23:29 +0200] rev 40600
perf: add the notion of "source" to perfrevlogwrite
We want to test performance associated witch various way to add a new revision.
They will be specified using this new argument.
Boris Feld <boris.feld@octobus.net> [Tue, 06 Nov 2018 00:57:34 +0100] rev 40599
perf: only display the total time for perfrevlogwrite if quiet
This provide a simple way to get an overview of the total performance.
Boris Feld <boris.feld@octobus.net> [Wed, 03 Oct 2018 11:04:57 +0200] rev 40598
perf: offer full details in perfrevlogwrite
This will be useful for people who want to study the timing pattern more
closely.
Boris Feld <boris.feld@octobus.net> [Wed, 03 Oct 2018 10:53:29 +0200] rev 40597
perf: introduce a perfrevlogwrite command
The command record times taken by adding many revisions to a revlog. Timing
each addition, individually. The "added revision" are recreations of the
original ones.
To time each addition individually, we have to handle the timing and the
reporting ourselves.
This command is introduced to track the impact of sparse-revlog format on
delta computations at initial storage time. It starts with the full text, a
situation similar to the "commit". Additions from an existing delta are better
timed with bundles.
The complaints from `check-perf-code.py` are not relevant. We are accessing
and "revlog" opener, not a repository opener.
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 10:41:00 -0500] rev 40596
tests: fix config knob in test-narrow-clone-stream.t
Two patches landed in parallel and had a semantic conflict. This
resolves the mess and leaves us with passing tests.
Differential Revision: https://phab.mercurial-scm.org/D5231
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 10:26:33 -0500] rev 40595
remotefilelog: fix various whitespace issues in docstring
Differential Revision: https://phab.mercurial-scm.org/D5230
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 19:42:50 +0900] rev 40594
ui: add config knob to redirect status messages to stderr (API)
This option can be used to isolate structured output from status messages.
For now, "stdio" (stdout/err pair) and "stderr" are supported. In future
patches, I'll add the "channel" option which will send status messages to
a separate command-server channel with some metadata attached, maybe in
CBOR encoding.
This is a part of the generic templating plan:
https://www.mercurial-scm.org/wiki/GenericTemplatingPlan#Sanity_check_output
.. api::
Status messages may be sent to a dedicated stream depending on
configuration. Don't use ``ui.status()``, etc. as a shorthand for
conditional writes. Use ``ui.write()`` for data output.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 20:53:31 +0900] rev 40593
ui: hide fin/fout/ferr attributes behind @property functions
This allows keeping references to fout/ferr/fin which are updated when these
properties are changed. See the next patch.
Yuya Nishihara <yuya@tcha.org> [Sun, 18 Jan 2015 17:42:53 +0900] rev 40592
ui: label prompt and echo messages
I'm going to add a dedicated command-server channel for status messages,
which carries metadata alongside a message text. 'ui.*' label provides a
hint how message text should be processed.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 18:17:30 +0900] rev 40591
ui: add inner function to select write destination
I'm going to add a config knob to redirect any status messages to stderr.
This function helps to switch underlying file objects.
# no-check-commit because of existing write_err() function
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 18:04:22 +0900] rev 40590
ui: remove _write() and _write_err() functions
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 18:03:09 +0900] rev 40589
ui: move pre/post processes from low-level write()s to _writenobuf()
This helps adding a dedicated stream for status/error messages. I don't
want to add _write*() function per stream.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 17:56:17 +0900] rev 40588
ui: pass in file object to _writenobuf()
See the subsequent patches for why. The "if" block in _writenobuf() will
be removed soon.
Boris Feld <boris.feld@octobus.net> [Mon, 05 Nov 2018 17:24:39 +0100] rev 40587
perf: fix perfrevlogrevisions --reverse
Currently, 'endrev' equals `len(revlog)`, a revision that does not exist.
When asking for the reverse order, the arguments passed to xrange are
`xrange(len(revlog), startrev)` which then crash.
We need to offset 'endrev' by one so we don't crash anymore. Also, we offset
'startrev' to ensure we get the same number of revisions with and without the
`--reverse` option.
Differential Revision: https://phab.mercurial-scm.org/D5228
Boris Feld <boris.feld@octobus.net> [Tue, 06 Nov 2018 11:54:15 +0100] rev 40586
procutil: import concerns about creationflags on Windows from D1701
I don't have the need anymore for the change in D1701 nor the time to
investigate the changes on all supported Windows platforms.
I import the stuff I learned on D1701 in the `runbgcommand` so the next people
working on it can starts from there.
Differential Revision: https://phab.mercurial-scm.org/D5229
Gregory Szorc <gregory.szorc@gmail.com> [Mon, 05 Nov 2018 14:14:32 -0800] rev 40585
localrepo: extract loading of hgrc files to standalone function
Various 3rd party extensions supplement where per-repo config data
lives. Looking at their sources, they resort to unorthodox means to
inject the config data. And the way they do it is susceptible to
corner cases. e.g. not processing automatic extension loads,
not reacting to new or disabled extensions in configs, etc.
This commit extracts the core logic of loading hgrc files into
a standalone function so there is a clear function that can be
monkeypatched to inject per-repo config data at repository open
time.
Differential Revision: https://phab.mercurial-scm.org/D5221
Martin von Zweigbergk <martinvonz@google.com> [Mon, 05 Nov 2018 09:09:48 -0800] rev 40584
revsets: make bookmark/named('re:nonexistent') not abort (issue6018) (BC)
Foozy documented the differences between revsets branch(), tag(),
bookmark(), and named() in eeb5d5ab14a6 (revset: raise RepoLookupError
to make present() predicate continue the query, 2015-01-31). He seemed
to want tag() to change behavior to not error out on non-matching
regular expressions. I think it's instead bookmark() and named() that
should not error out. So that's what this patch does.
Differential Revision: https://phab.mercurial-scm.org/D5220
Danny Hooper <hooper@google.com> [Mon, 05 Nov 2018 16:05:45 -0800] rev 40583
fix: rename :fileset subconfig to :pattern
This name was always inaccurate, since the config accepts any pattern.
Hopefully so few people use this right now that it won't matter, but there will
now be a warning if the old config name is used.
Differential Revision: https://phab.mercurial-scm.org/D5226
Danny Hooper <hooper@google.com> [Wed, 31 Oct 2018 13:11:51 -0700] rev 40582
fix: add a config to abort when a fixer tool fails
This allows users to stop and address tool failures before proceeding, instead
of the default behavior of continuing to apply any tools that didn't fail. For
example, a code formatting tool could fail if you have syntax errors, and you
might want your repo to stay in its current state while you fix the syntax
error before re-running 'hg fix'. It's conceivable that this would even be
necessary for the correctness of some fixer tools across a chain of revisions.
Differential Revision: https://phab.mercurial-scm.org/D5200
Boris Feld <boris.feld@octobus.net> [Tue, 06 Nov 2018 11:05:13 +0100] rev 40581
perf: measure slicing time in perfrevlogrevision
Slicing a sparse delta chain can be expensive. We now benchmark the associated
time.
Boris Feld <boris.feld@octobus.net> [Tue, 06 Nov 2018 11:04:23 +0100] rev 40580
perf: teach perfrevlogrevision about sparse reading
Before this change, chunks were always read in a single block. Even in the
sparse-read/sparse-revlog case. This gave a false view of the performance and
could lead to memory consumption issue.
Boris Feld <boris.feld@octobus.net> [Tue, 06 Nov 2018 11:13:31 +0100] rev 40579
perf: use the same timer for all section of perfrevlogrevision
Otherwise the -T json output is invalid.
Boris Feld <boris.feld@octobus.net> [Tue, 22 May 2018 15:26:17 +0200] rev 40578
obsutil: clarify the access to "repo"
We use the variable multiple times and we might use it even more in the
future. We use a temporary variable instead.
rdamazio@google.com [Mon, 05 Nov 2018 19:52:42 -0800] rev 40577
pycompat: adding Linux detection and fixing Mac
Python 3 recommends detecting OSs with the prefix of the platform, but we were
comparing the full string for macOS. We also didn't have Linux detection, which
is convenient for extensions to use (rather than have some OSs detected by hg
and some by the extension).
Reference:
https://docs.python.org/3/library/sys.html#sys.platform
Differential Revision: https://phab.mercurial-scm.org/D5227
Augie Fackler <augie@google.com> [Mon, 05 Nov 2018 17:48:23 -0500] rev 40576
remotefilelog: fix various minor py3 problems
# skip-blame b prefixes and pycompat.long, nothing remotely interesting
Differential Revision: https://phab.mercurial-scm.org/D5223
Augie Fackler <augie@google.com> [Mon, 05 Nov 2018 17:37:37 -0500] rev 40575
remotefilelog: rip out a missed mention of lz4
Differential Revision: https://phab.mercurial-scm.org/D5222
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 17:47:23 +0900] rev 40574
ui: wrap whole _write() block with timeblockedsection
I think the cost of color labeling is negligible compared to the I/O
syscalls. Let's simply wrap the whole write() function so that we can
eliminate _write() and _write_err() in later changeset.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 17:43:57 +0900] rev 40573
ui: indent _writenobuf() to prepare moving bits from _write() functions
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 17:42:05 +0900] rev 40572
ui: simply concatenate messages before applying color labels
This should be cheaper in space than applying labels for each message.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 17:36:10 +0900] rev 40571
ui: simplify interface of low-level write() functions
_write() and _write_err() will be replaced with fout.write() and ferr.write()
respectively. This is the first step.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 17:32:35 +0900] rev 40570
ui: factor out function that writes data to fout/ferr with labeling
I'm thinking of adding an option to send status messages to stderr (or a
dedicated command-server channel) so that structured output (e.g. JSON)
would never be interleaved with non-formatter output. A unified write()
interface helps to do that.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Nov 2018 16:04:16 +0900] rev 40569
ui: consolidate places where _progclear() is called
The progress bar has to be cleared when we start writing some data to the
output stream. Let's make it always triggered immediately before switching
by _colormode, so that we can easily factor out helper functions.
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Nov 2018 20:44:26 +0900] rev 40568
templater: compute revset lazily
This speeds up e.g. "{ifcontains(rev, revset('::.'), ...)}" in common cases
where 'rev' is near the working parent.
The templater API is ugly, but it helps here. 'f' can be either a generator
or a function returning a generator.
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 24 Oct 2018 18:48:43 +0300] rev 40567
remotefilelog: drop compat code for "getbundle_shallow" wireprotocol command
Doing some annotate on hgexperimental shows that getbundle_shallow used to exist
in 2013 or before. We don't have any pre-2013 remotefilelog users except Fb
themselves and I doubt they are going to use in-core remotefilelog. So it's safe
to remove this.
Differential Revision: https://phab.mercurial-scm.org/D5193
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 24 Oct 2018 17:54:34 +0300] rev 40566
tests: remove lz4 as dependency while running tests
One of the previous patch authored by Augie rips out the lz4 dependency and
things should work without it.
Now there are just 2-3 tests failing because of same change in emitrevisions()
API.
Differential Revision: https://phab.mercurial-scm.org/D5192
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 24 Oct 2018 17:51:49 +0300] rev 40565
remotefilelogserver: add a matcher argument to _walkstreamfiles()
Implementing narrow stream clones in core, I added an optional matcher argument.
The function in remotefilelogserver.py does not know about that argument and
does not accept that and hence some tests fails.
Differential Revision: https://phab.mercurial-scm.org/D5191
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 05 Nov 2018 11:52:10 -0800] rev 40564
shallowutil: introduce a helper function isenabled()
This patch introduces a inenabled() function which will check whether
remotefilelog is enabled or not. The function is then also used at all the
places where check whether remotefilelog is enabled or not. The new function
makes code easy to read without need to understand what is the constant involved
and why we are checking repo.requirements.
Differential Revision: https://phab.mercurial-scm.org/D5190
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 24 Oct 2018 16:59:05 +0300] rev 40563
remotefilelog: remove some BC code related to streamclone
Since remotefilelog is now moved to core and we have
streamclone._walkstreamfiles() in core, we don't need to have the backward
compatibility code.
People with old mercurial version should use remotefilelog from hg-experimental
repo as IMO remotefilelog will go under a good refactoring and old clients will
break.
# no-check-commit foo_bar function name
Differential Revision: https://phab.mercurial-scm.org/D5189
Augie Fackler <augie@google.com> [Wed, 17 Oct 2018 09:12:29 -0400] rev 40562
remotefilelog: add some docstring
Differential Revision: https://phab.mercurial-scm.org/D5131
Augie Fackler <augie@google.com> [Thu, 04 Oct 2018 10:03:10 -0400] rev 40561
remotefilelog: rename wireproto methods and remaining capabilities
This is a mess, in part because there should be more constants
throughout. I know we typically do exp- instead of the x_ business in
this change, but I also had to use this in some function names, so I
figured until I can break that coupling I'd go with this. If it's too
unpleasant during review, let me know and I can probably clean it up
some more.
# no-check-commit due to new foo_bar naming - too hard to avoid right now :(
Differential Revision: https://phab.mercurial-scm.org/D5129
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 17:02:48 -0400] rev 40560
remotefilelog: consolidate and rename on-disk store requirement
The value of this constant appeared in too many places. While we're
here, rename it to be more consistent with our naming conventions for
experimental functionality.
Differential Revision: https://phab.mercurial-scm.org/D5128
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 17:30:47 -0400] rev 40559
remotefilelog: consolidate and rename bundle2 capability
Bonus: we catch a spot where the shallowrepo requirement was being
misused as the bundle2 capability.
Differential Revision: https://phab.mercurial-scm.org/D5127
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 16:38:40 -0400] rev 40558
remotefilelog: rename capability for legacy ssh file fetching method
I think I want to delete this, but for now we'll just rename this so
it's easy to tell apart from other strings that have the same value.
Differential Revision: https://phab.mercurial-scm.org/D5126
Augie Fackler <augie@google.com> [Thu, 04 Oct 2018 00:11:37 -0400] rev 40557
remotefilelog: rip out lz4 support
Some methods are left teased out so it's easier to build a migration
extension from v1 packs to v2. I also anticipate those methods will be
of use if we get around to adding configurable compression engines
before we end up jettisoning packs.
I think it would make sense to register one-shot `compress` and
`decompress` methods on our compression engines in util.py, but
indygreg mentioned that might not be straightforward.
Differential Revision: https://phab.mercurial-scm.org/D4944
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 23:25:31 -0400] rev 40556
remotefilelog: bump pack file formats to version 2
Drop support for format 0 and 1 packs. Version 2 will be different in
that it'll use zlib compression instead of lz4, as the latter isn't
really suitable for core.
If I can make it work, I'll make compression engines pluggable, but
the logic for handling these files is spread out enough it's a little
tricky to get that right.
Test changes are only due to pack files being renamed. The contents of
the packs are the same.
Differential Revision: https://phab.mercurial-scm.org/D4943
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 23:47:43 -0400] rev 40555
remotefilelog: avoid hard-coding pack names in tests
I'm about to alter the pack format, which alters the content-addressed
name of the pack file. In order to reduce my own confusion I found it
helpful to use an ls | head pipeline since we always want to inspect
the newest pack.
Differential Revision: https://phab.mercurial-scm.org/D4942
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 14:04:41 -0400] rev 40554
remotefilelog: remove now-unused wirepack code
Differential Revision: https://phab.mercurial-scm.org/D5124
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 14:04:11 -0400] rev 40553
remotefilelogserver: remove pack-serving functionality
Differential Revision: https://phab.mercurial-scm.org/D5123
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 13:17:16 -0400] rev 40552
remotefilelog: remove support for fetching packs
We've never used this at Google, and it's strictly an optimization
that indygreg and I are confident we can handle more elegantly
later. Let's rip it out to keep the surface area of the imported
remotefilelog small.
Some tests have changes because the tests for regular packs and
repacking were mixed with tests of fetching packs. I'm pretty sure
I've done the right thing here.
Differential Revision: https://phab.mercurial-scm.org/D5122
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 17:20:44 -0400] rev 40551
remotefilelog: jettison example cache client
We can always reconstruct it from history if it becomes interesting.
Differential Revision: https://phab.mercurial-scm.org/D4941
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 17:18:20 -0400] rev 40550
remotefilelog: prune obsolete method
Differential Revision: https://phab.mercurial-scm.org/D4940
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 14:01:22 -0400] rev 40549
logtoprocess: use new runbgcommand from procutil
Differential Revision: https://phab.mercurial-scm.org/D4939
Augie Fackler <augie@google.com> [Tue, 16 Oct 2018 14:42:57 -0400] rev 40548
procutil: port over windows encoding fixes from logtoprocess
Differential Revision: https://phab.mercurial-scm.org/D5121
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 14:01:04 -0400] rev 40547
remotefilelog: transplant runbgcommand to procutil
While cleaning up the deprecated runshellcommand I noticed a
near-clone of this in logtoprocess, so I'm standardizing on what
appears to be the newer one by moving it to procutil.
Differential Revision: https://phab.mercurial-scm.org/D4938
Augie Fackler <augie@google.com> [Wed, 03 Oct 2018 13:54:45 -0400] rev 40546
remotefilelog: remove function that was described as deprecated
Differential Revision: https://phab.mercurial-scm.org/D4937
Augie Fackler <augie@google.com> [Thu, 27 Sep 2018 13:03:19 -0400] rev 40545
remotefilelog: import pruned-down remotefilelog extension from hg-experimental
This is remotefilelog as of my recent patches for compatibility with
current tip of hg, minus support for old versions of Mercurial and
some FB-specific features like their treemanifest extension and
fetching linkrev data from a patched phabricator. The file extutil.py
moved from hgext3rd to remotefilelog.
This is not yet ready to be landed, consider it a preview for
now. Planned changes include:
* replace lz4 with zstd
* rename some capabilities, requirements and wireproto commands to mark
them as experimental
* consolidate bits of shallowutil with related functions (eg readfile)
I'm certainly open to other (small) changes, but my rough mission is
to land this largely as-is so we can use it as a model of the
functionality we need going forward for lazy-fetching of file contents
from a server.
# no-check-commit because of a few foo_bar functions
Differential Revision: https://phab.mercurial-scm.org/D4782
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 17 Oct 2018 17:42:32 +0300] rev 40544
store: introduce _matchtrackedpath() and use it to filter store files
This patch introduces a function to filter store files on the basis of the path
which they are tracking.
The function assumes that the entries can be of two types, 'meta/*' and 'data/*'
which means it will just work on revlog based storage and not with another
storage ways.
For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the
beginning and the end then pass that to matcher.
For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from
beginning and end then call matcher.visitdir() with it to make sure all the
parent directories are also downloaded.
Since the storage filtering for narrow stream clones is implemented with this
patch, we remove the un-implemented error message, add some more tests and add
the treemanifest case to tests too.
The tests demonstrate that it works correctly.
After this patch, we have now narrow stream clones working. Narrow stream clones
are a very important feature for large repositories who have good internet
connection because they use streamclones for cloning and if they do normal
narrow clone, that takes more time then a full streamclone. Also narrow-stream
clone will drastically speed up clone timings.
Differential Revision: https://phab.mercurial-scm.org/D5139
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 17 Oct 2018 17:24:55 +0300] rev 40543
narrow: only send includepats and excludepats if they are not empty
If we send an empty includepats or excludepats argument to getbundle, it's
translated to `['']` on the server which causes problems because even though
it's empty, bool of that value if True and we end up creating differencematcher
with narrowspec.match() which results in unexpected behavior.
Differential Revision: https://phab.mercurial-scm.org/D5138
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 16 Oct 2018 17:53:26 +0300] rev 40542
exchange: pass includepats and excludepats as arguments to getbundle()
This will help in implementing narrow stream clones. Also narrow extension used
to add these arguments, now we add them by default if they are not empty.
Since reading includepats and excludepats on the server only works when narrow
is enabled, we check if narrow if enabled or not before passing them.
Differential Revision: https://phab.mercurial-scm.org/D5119
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:51:21 -0700] rev 40541
catapult: add a bit more documentation on how to use catapult tracing
Differential Revision: https://phab.mercurial-scm.org/D5217
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:44:02 -0700] rev 40540
catapult: introduce HGTESTCATAPULTSERVERPIPE to control run-tests' tracing
If unset, it will inherit its value from HGCATAPULTSERVERPIPE (which hg itself
also respects). By setting only HGTESTCATAPULTSERVERPIPE, we can get per-command
breakdowns of the test runtimes for the whole test suite without overloading the
trace file with the contents of the tracing from hg (such as demandimport, etc.)
Differential Revision: https://phab.mercurial-scm.org/D5216
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:44:37 -0700] rev 40539
catapult: rename 'active' to 'activetrace'; this isn't storing a boolean state
(This is an attempt to make it more obvious that the indentation is correct, to
prevent a repeat of 97f52862b1bd)
Differential Revision: https://phab.mercurial-scm.org/D5215
Kyle Lippincott <spectral@google.com> [Thu, 01 Nov 2018 16:44:32 -0700] rev 40538
catapult: fix broken run-tests catapult tracing
Commit 97f52862b1bd thought this block was misindented, but it was not; the
indentation after that commit means we never log a catapult trace for individual
commands. Also, we've never emitted the "end" trace line when the test was
finished, so the trace viewer claims that these "did not finish", fixing that as
well.
Differential Revision: https://phab.mercurial-scm.org/D5214
Sangeet Kumar Mishra <mail2sangeetmishra@gmail.com> [Sat, 03 Nov 2018 23:24:15 +0530] rev 40537
test: fix self._testdir to use the right mercurial library during testing
Currently if you run tests from some other directory other than the
`../tests/`, you will get a warning stating
```
warning: Testing with unexpected mercurial lib: mercurial
(expected /tmp/hgtests.xxxxxx/install/lib/python/mercurial)
```
This is because the current directory being added to the 'PATH', if the
`self._testdir != runtestdir`, owing to this line
```
if self._testdir != runtestdir:
path = [self._testdir] + path
```
Also say you ran the tests from the hg base directory,
because directory is being added in the PATH (see the above snippet, at
that stage the `self._testdir` has the value as `cwd`, owing to a faulty
initialization). And since the current directory already has the 'hg',
that is used in place of the hg that is installed for the testing purposes
in `/tmp/hgtests.xxxxxx/...`.
Differential Revision: https://phab.mercurial-scm.org/D5199
Matt Harbison <matt_harbison@yahoo.com> [Sat, 03 Nov 2018 20:50:04 -0400] rev 40536
tests: conditonalize a difference in test-merge-tools.t on Windows
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 17:12:22 -0400] rev 40535
statprof: clean up unicode/bytes a little
I'm not really sure how this worked before, but something perturbed it
and what I've got in this change I believe is a little tidier. This
fixes test-profile.t on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5210
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 17:11:31 -0400] rev 40534
statprof: add a couple of asserts to avoid storing unicodes
I don't feel strongly about this change, so it'd be fine to drop it
during review, but it was _extremely_ helpful in drafting the next
change.
Differential Revision: https://phab.mercurial-scm.org/D5209
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 17:09:55 -0400] rev 40533
tests: add some helpful `|| cat` bits to test-profile.t
This way if the hg invocation crashes, you get to see a stacktrace
without having to edit the test.
Differential Revision: https://phab.mercurial-scm.org/D5208
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 16:13:01 -0400] rev 40532
tests: skip wireproto clientreactor tests on Python 3.6.0-3.6.3 inclusive
See comment next to the if statement for sad details.
Differential Revision: https://phab.mercurial-scm.org/D5207
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 15:58:01 -0400] rev 40531
contrib: tweak import-checker to always use bytes for module names
We could go the other route and make embedded() return this as a
bytes, but then it would be harder to portably doctest, so I prefer
this option.
Differential Revision: https://phab.mercurial-scm.org/D5206
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 15:43:43 -0400] rev 40530
contrib: fix import-checker to not b'' module names on Python 3
Caught by the doctests.
Differential Revision: https://phab.mercurial-scm.org/D5205
Augie Fackler <augie@google.com> [Thu, 01 Nov 2018 15:47:26 -0400] rev 40529
tests: add a critical flush() to run-tests.py to make output stable on py3
Differential Revision: https://phab.mercurial-scm.org/D5204
Kyle Lippincott <spectral@google.com> [Sat, 13 Oct 2018 07:49:20 -0700] rev 40528
merge-tools: when calling external merge tool, describe the resolve inputs
It is a common complaint that a user will be running some operation (histedit,
rebase, evolve, etc.), get into a merge-conflict situation, and not understand
what they are seeing - it is possible that the merge tool is configured to
display the hash, but it's difficult for most merge tools to display a good
snippet of the description.
In the worst case, configuring this template will lead to output that is
immediately covered by a terminal application, maybe the user can hit ctrl-z to
see it. In the common case, the output will be in a terminal window and a GUI
program will start, and it should be possible to view both the terminal and the
GUI program at the same time.
Differential Revision: https://phab.mercurial-scm.org/D5094
Yuya Nishihara <yuya@tcha.org> [Fri, 26 Oct 2018 21:46:37 +0900] rev 40527
templatekw: deprecate p1rev/p2rev/p1node/p2node in favor of p1/p2
Yuya Nishihara <yuya@tcha.org> [Fri, 26 Oct 2018 21:43:20 +0900] rev 40526
templatekw: add p1/p2 keywords which switches the current ctx
This is just an example of mappingdict.
I have no idea what should be displayed as {p1|json}. Currently it is an
empty dict since {'ctx'} isn't displayable, which is clearly useless and
should be changed later.
Yuya Nishihara <yuya@tcha.org> [Fri, 19 Oct 2018 21:11:30 +0900] rev 40525
templater: add wrapper for a single template mapping
This can be used to nest template mappings without inserting a sequence-like
layer. See the next patch for example.
Yuya Nishihara <yuya@tcha.org> [Fri, 26 Oct 2018 21:28:20 +0900] rev 40524
templatekw: extract internal "{rev}:{node|formatnode}" template to constant
This will be used later.
Boris Feld <boris.feld@octobus.net> [Thu, 01 Nov 2018 14:35:18 +0100] rev 40523
test: remove changes preparing the drop of `HGMERGE`
These changes are no longer necessary so we better drop them.
Boris Feld <boris.feld@octobus.net> [Thu, 01 Nov 2018 14:29:21 +0100] rev 40522
run-tests: define the default merge tool through configuration
Using the `HGMERGE` environment variable generates confusion as it overrides
any value set through configuration.
By setting the default value through the default HGRC, the usual overriding
rules apply and it should reduce confusion.
Boris Feld <boris.feld@octobus.net> [Fri, 19 Oct 2018 01:52:16 +0200] rev 40521
run-tests: explicitly declare the list of dropped environment variable
It will make the list clearer and more maintainable.
Boris Feld <boris.feld@octobus.net> [Thu, 01 Nov 2018 14:29:14 +0100] rev 40520
test: drop usage of `HGMERGE` in `test-lfconvert.t`
Using configuration instead of environment variable has a strange effect on
the output, but no impact on the end result.
Boris Feld <boris.feld@octobus.net> [Thu, 01 Nov 2018 14:26:30 +0100] rev 40519
test: drop usage of `HGMERGE` in `test-merge-types.t`
Using configuration instead of environment variable has a strange effect on
the output, but no impact on the end result.
Boris Feld <boris.feld@octobus.net> [Thu, 01 Nov 2018 14:24:34 +0100] rev 40518
test: use the "correct" merge tool in `test-rebase-detach.t`
The value of `HGMERGE` overrides the value passed through --config. We are
about to drop this `HGMERGE` environment variable so we update the value used
in config to match what is currently in use.
This is another example of confusion introduced by the `HGMERGE` environment
variable.
Boris Feld <boris.feld@octobus.net> [Wed, 31 Oct 2018 19:12:32 +0100] rev 40517
test: also deactivate `ui.merge` when testing merge tool
We are about to introduce a default config for `ui.merge`. Since
`test-merge-tools.t` is about testing merge configuration, we need to make sure
it won't interfere.
Boris Feld <boris.feld@octobus.net> [Wed, 31 Oct 2018 18:43:56 +0100] rev 40516
tests: adjust merge tool config in test-largefiles-update.t
Previously, the `ui.merge=internal:fail` configuration was shadowed by the the
`HGMERGE=internal:merge` environment variable. We would like to remove this
usage of `HGMERGE` variable because it is error-prone (eg: this very test). We
start by cleaning up test independently to avoid large churn when the change
happens.
Boris Feld <boris.feld@octobus.net> [Fri, 20 Apr 2018 18:08:54 +0200] rev 40515
extensions: include current version in "invalid version" message
It is "not so rare" for the mercurial version to be badly detected at build
time. In such case, version check for extensions gets confused.
To help pinpoint the error, we now include the Mercurial's version number in
the error message.
Boris Feld <boris.feld@octobus.net> [Fri, 19 Oct 2018 01:07:08 +0200] rev 40514
obsolete: prefetch the repo.obsstore used in phasedivergence loop
The speedup is probably quite negligible, but it cannot hurt.
Boris Feld <boris.feld@octobus.net> [Fri, 19 Oct 2018 00:53:18 +0200] rev 40513
obsutil: prefetch method in allpredecessors loop
We don't expect a massive speedup from this, but the change was laying around
in my repository and it cannot hurt.
Augie Fackler <augie@google.com> [Fri, 02 Nov 2018 14:24:29 -0400] rev 40512
merge with stable
Danny Hooper <hooper@google.com> [Sat, 13 Oct 2018 01:55:40 -0700] rev 40511
beautifygraph: don't substitute anything for 'X' in rendered graphs
It looks like we never actually render graphs this way, although there's an
example in a comment next to the code that prevents it from being needed (see
graphmod.ascii()).
Differential Revision: https://phab.mercurial-scm.org/D5103
Yuya Nishihara <yuya@tcha.org> [Wed, 24 Oct 2018 21:29:04 +0900] rev 40510
py3: do not stringify integers in revlog stats by '%s'
Yuya Nishihara <yuya@tcha.org> [Wed, 24 Oct 2018 21:28:03 +0900] rev 40509
py3: convert revlog stats to a dict of (bytes, int) pairs
Py_DECREF(t) is replaced with Py_CLEAR(t) so that t is set to NULL once
decrefed. Otherwise, it would be excessively decrefed if a subsequent
PyBytes_FromString() failed.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 25 Oct 2018 22:13:22 -0400] rev 40508
py3: roll up threading.Thread constructor args into **kwargs
The constructor doesn't have a `verbose` keyword argument in py3.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 25 Oct 2018 21:04:47 -0400] rev 40507
tests: glob over a single quote vs double quote difference on Windows
Augie Fackler <augie@google.com> [Tue, 23 Oct 2018 11:31:33 -0400] rev 40506
py3: port test-log-exthook.t to Python 3
For once an easy one.
Differential Revision: https://phab.mercurial-scm.org/D5184
Augie Fackler <augie@google.com> [Fri, 12 Oct 2018 12:30:47 -0400] rev 40505
extensions: fix up many many debug logs that use %r
This gets us a bunch closer on Python 3, but I'll still have to use a
ton of sad globs. A previous version of this patch tried to preserve
the %r formatting, but upon review Yuya noted that special characters
in extension names is very unlikely, so we can just use %s and not
sweat the quoting.
Differential Revision: https://phab.mercurial-scm.org/D5002
Matt DeVore <matvore@google.com> [Wed, 17 Oct 2018 15:48:01 -0700] rev 40504
blackbox: add configitem for format of log timestamps
Sometimes blackbox logs are used to report performance problems, but the
timestamps are only at second granularity, so often the timings have to
stated separately by the reporter. This is inconvenient and error-prone,
so I would like to include %f in the date format. This patch makes that
possible.
Martin von Zweigbergk <martinvonz@google.com> [Tue, 23 Oct 2018 14:04:17 -0700] rev 40503
narrow: replace filtering in list comprehension by set operations
I didn't think of this while reviewing the patch.
Differential Revision: https://phab.mercurial-scm.org/D5188
Boris Feld <boris.feld@octobus.net> [Tue, 23 Oct 2018 19:20:22 +0200] rev 40502
storage: update sqlitestore to use the new `deltamode` parameter
While updating the filelog class, I forget to update the sqlitestore, tests
are now passing with this patch.
Differential Revision: https://phab.mercurial-scm.org/D5185
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 23 Oct 2018 16:24:04 +0300] rev 40501
narrow: rework logic to check whether we need to widen and narrow
This patch reworks logic which calculates whether we need to extend or narrow
our working copy or not.
We filter the addincludes, removeincludes, addexcludes and removeexcludes passed
from user to the actual added and removed includes and excludes. What that means
is a user can pass an already included path as addincludes, a path which is not
included as removeincludes etc. In such situations the old logic use to think we
need to do some work, whereas we don't need to do that work.
In old logic, even if we don't have anything new to include but it believes we
need to call widen, this adds some good amount of work on large repository. A
widen calls involves computing incomming csets, calling the narrow_widen() which
in non-ellipses cases goes through all the set of csets which are available
which can take ~2-3 mins on large repos. Those 2-3 minutes are spend on doing
nothing which a client can prevent by checking is there really anything which
needs to be included.
The tests changes shows that we don't go to the server anymore in such cases
which is nice.
Differential Revision: https://phab.mercurial-scm.org/D5183
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 23 Oct 2018 14:26:17 +0300] rev 40500
tests: show that adding an already included path still calls narrow_widen()
This patch adds tests demonstrating that we still go to the server in
non-ellipses widening when we have that path already on the client and there is
nothing new to download.
The next patch will try to make client side logic smart and not go to the server
if we don't need to download anything.
Differential Revision: https://phab.mercurial-scm.org/D5182
Mads Kiilerich <mads@kiilerich.com> [Sun, 14 Oct 2018 17:08:18 +0200] rev 40499
graft: introduce --base option for using custom base revision while merging
The graft command usually performs an internal merge of the current parent
revision with the graft revision, using p1 of the grafted revision as base for
the merge.
As a trivial extension of this, we introduce the --base option to allow for
using another base revision.
This can be used as a building block for grafting and collapsing multiple
changesets at once, or for grafting the resulting change from a merge as a
single simple change. (This is kind of similar to backout --parent ... only
different: this graft base must be an ancestor, but is usually *not* a parent.)
This is probably an advanced use case, and we do thus not show it in the
non-verbose help.
Boris Feld <boris.feld@octobus.net> [Thu, 18 Oct 2018 12:31:06 +0200] rev 40498
changegroup: add a option to create bundle with full snapshot only
This is easy to implement now and can be useful for benchmarking.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:21:02 +0200] rev 40497
changegroup: allow to force delta to be against p1
This new developer option is useful to general more "generic" bundle. Without
this option, a bundle generated from the repository use deltas similar to the
one stored in the specific repository it was generated from. This makes
performance testing a bit tricky.
Using deltas similar to the final result means all delta stored in the bundle
can be applied to the target repository without any further processing (except
for the rare case of a full snapshot). The application of such bundles
(almost) never exercises the (slower) path of searching for a new valid delta.
This result in unrealistic and too favorable timing and profile.
Instead, we introduce an option to make sure all revisions are stored as a
delta against p1. It might not be the best generation option, but it
guarantees that the content will be "generic", not favoring a specific target.
Boris Feld <boris.feld@octobus.net> [Tue, 09 Oct 2018 23:26:35 +0200] rev 40496
storage: also use `deltamode argument` for ifiledata
Now that lower level uses such argument, we can propagate the change to higher
layers.
Boris Feld <boris.feld@octobus.net> [Tue, 09 Oct 2018 22:02:01 +0200] rev 40495
changegroup: refactor emitrevision to use a `deltamode` argument
This new argument gathers the semantic of `sendfulltext` and `deltaprevious`
in a single value. We are about to introduce a new type of constraints.
Avoiding yet another argument sounds like a plus.
Yuya Nishihara <yuya@tcha.org> [Tue, 23 Oct 2018 21:11:13 +0900] rev 40494
branchmap: do not specify changelog as an argument
Since (unfiltered)repo.changelog lookup gets as fast as __dict__ lookup,
there's no point to pass in changelog instance.
$ hg perfbranchmap --clear-revbranch -R mozilla-central
! base
(orig) wall 20.593091 comb 20.600000 user 20.520000 sys 0.080000 (best of 3)
(this) wall 20.129126 comb 20.130000 user 20.020000 sys 0.110000 (best of 3)
This backs out most of the changes in 76d4272bd57b and 47c03042cd1d.
Yuya Nishihara <yuya@tcha.org> [Sat, 20 Oct 2018 17:56:00 +0900] rev 40493
filecache: unimplement __set__() and __delete__() (API)
Implementing __set__() implies that the descriptor can't be overridden by
obj.__dict__, which means any property access involves slow function call.
"Data descriptors with __set__() and __get__() defined always override
a redefinition in an instance dictionary. In contrast, non-data descriptors
can be overridden by instances."
https://docs.python.org/2.7/reference/datamodel.html#invoking-descriptors
This patch basically backs out 236bb604dc39, "scmutil: update cached copy
when filecached attribute is assigned (issue3263)." The problem described
in issue3263 (which is #3264 in Bugzilla) should no longer happen since
repo._bookmarkcurrent has been moved to repo._bookmarks.active. We still
have a risk of introducing similar bugs, but I think that's the cost we
have to pay.
$ hg perfrevset 'branch(tip)' -R mercurial
(orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66)
(prev) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81)
(this) wall 0.099038 comb 0.110000 user 0.100000 sys 0.010000 (best of 93)
Yuya Nishihara <yuya@tcha.org> [Sat, 20 Oct 2018 19:13:05 +0900] rev 40492
filecache: use try-except for faster __dict__ lookup
Python function call is slow, and the cost could be significant here.
$ hg perfrevset 'branch(tip)' -R mercurial
(orig) wall 0.139511 comb 0.140000 user 0.140000 sys 0.000000 (best of 66)
(this) wall 0.114195 comb 0.110000 user 0.110000 sys 0.000000 (best of 81)
rdamazio@google.com [Thu, 18 Oct 2018 19:57:30 -0700] rev 40491
help: displaying extension commands by default
Differential Revision: https://phab.mercurial-scm.org/D5156
rdamazio@google.com [Thu, 18 Oct 2018 19:57:05 -0700] rev 40490
help: displaying documented aliases by default
This makes aliases be displayed in "hg help" when they have a :doc config
entry, and also allows them to be assigned to a category with :category.
Differential Revision: https://phab.mercurial-scm.org/D5087
rdamazio@google.com [Sat, 13 Oct 2018 05:43:39 -0700] rev 40489
help: allow hiding of help topics
Differential Revision: https://phab.mercurial-scm.org/D5077
rdamazio@google.com [Sat, 13 Oct 2018 05:02:55 -0700] rev 40488
help: allow commands to be hidden
This is useful in enterprise environments where some workflows are
discouraged.
Differential Revision: https://phab.mercurial-scm.org/D5076
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 20 Oct 2018 00:12:20 +0300] rev 40487
py3: add one more passing test to whitelist
Caught by python 3 builder.
Differential Revision: https://phab.mercurial-scm.org/D5175
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 20 Oct 2018 00:05:50 +0300] rev 40486
py3: make sure we pass sysstr in sqlite3.connect()
Differential Revision: https://phab.mercurial-scm.org/D5174
Martin von Zweigbergk <martinvonz@google.com> [Tue, 05 Sep 2017 15:24:25 -0700] rev 40485
archive: use manifest.matches() to simplify and speed up matching
manifest.matches() can avoid walking paths the user did not want to
archive.
Differential Revision: https://phab.mercurial-scm.org/D5178
Martin von Zweigbergk <martinvonz@google.com> [Tue, 05 Sep 2017 15:24:22 -0700] rev 40484
archive: create alwaysmatcher when no matcher provided
Differential Revision: https://phab.mercurial-scm.org/D5177
Martin von Zweigbergk <martinvonz@google.com> [Tue, 05 Sep 2017 15:21:21 -0700] rev 40483
archive: change "matcnfn" argument to a real matcher
All callers seem to be passing a real matcher, not just a function. We
were also passing it into match.subdirmatcher(), which assumes it is a
matcher.
Differential Revision: https://phab.mercurial-scm.org/D5176
rdamazio@google.com [Wed, 09 Jan 2019 20:00:35 -0800] rev 40482
histedit: crashing with a more useful error message on empty defaultrev
Before this, `hg --config histedit.defaultrev= histedit` would crash with
File "destutil.py", line 385, in desthistedit
if revs:
UnboundLocalError: local variable 'revs' referenced before assignment
Differential Revision: https://phab.mercurial-scm.org/D5543
Augie Fackler <raf@durin42.com> [Mon, 07 Jan 2019 19:00:54 -0500] rev 40481
Added signature for changeset 197f092b2cd9
Augie Fackler <raf@durin42.com> [Mon, 07 Jan 2019 19:00:51 -0500] rev 40480
Added tag 4.8.2 for changeset 197f092b2cd9
Augie Fackler <raf@durin42.com> [Sun, 06 Jan 2019 14:58:54 -0500] rev 40479
server: always close http socket if responding with an error (issue6033)
It's possible for hgweb to respond _very_ early with an error if we're
catching certain types of errors. When we do, we need to tell the client
the socket is toast when there's a POST involved because otherwise there
can be lingering POST data on the socket that will confuse any future
requests on the socket. This manifested as a flaky failure on Linux in an
lfs extension test and a reliable failure on FreeBSD. With this patch
applied, test-lfs-serve-access.t now passes for me on FreeBSD.
Differential Revision: https://phab.mercurial-scm.org/D5498
Yuya Nishihara <yuya@tcha.org> [Sat, 05 Jan 2019 15:44:55 +0900] rev 40478
match: fix assertion for fileset with no context (issue6046)
A falsy changectx should be allowed.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 04 Jan 2019 21:01:10 -0500] rev 40477
templatekw: fix documentation typos
Yuya Nishihara <yuya@tcha.org> [Wed, 02 Jan 2019 09:41:04 +0900] rev 40476
update: do not pass in user revspec as default destination (issue6044)
When the revsingle() was introduced at 61c0df2b089a, it couldn't handle
revspec=0 (not '0') properly. That's probably why the default was set to
rev.
This is technically BC since "hg update ''" was identical to "hg update '.'"
whereas "hg update -r ''" is "hg update", but I believe that's a bug given
no test fails with this change.
Boris Feld <boris.feld@octobus.net> [Sun, 30 Dec 2018 16:11:06 +0100] rev 40475
revlog: cache delta base value under -1
Such base are invalid so we better report them early.
Boris Feld <boris.feld@octobus.net> [Thu, 27 Dec 2018 23:34:37 +0100] rev 40474
revlog: catch revlog corruption in index_baserev
A revision cannot use a base above itself, it can only happens one corrupted
repository.
Ignoring such corrupted could lead to infinite loop.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 21 Dec 2018 17:36:12 -0500] rev 40473
phabricator: properly encode boolean types in the request body
I tripped over this playing with `hg debugcallconduit` to query for valid
reviewers. If the JSON on stdin is written as 'True' or 'False', python
complains it isn't valid JSON. If it's written as 'true' or 'false', it made it
to the server, but got kicked back with this:
abort: Conduit Error (ERR-CONDUIT-CORE): Error while reading "isBot":
Expected boolean (true or false), got something else.
The test isn't really relevant here (the code can be reverted, and it will
pass), but this gives us coverage for the debug command.
Yuya Nishihara <yuya@tcha.org> [Tue, 11 Dec 2018 22:34:07 +0900] rev 40472
worker: do not swallow exception occurred in main process
Before, SystemExit(255) would be most likely raised since the worker processes
were terminated by the main process and the status would be set to 255 in
response. We should instead re-raise the exception occurred first. It's pretty
hard to debug problems like the issue 6035 with no traceback.
Yuya Nishihara <yuya@tcha.org> [Mon, 17 Dec 2018 20:46:09 +0900] rev 40471
run-tests: fix permission to clean up unreadable directories
I found many hgtests.* directories left in $TMPDIR, which couldn't be deleted
because test-ssh-repoerror.t created some directories with a-rx mode.
Yuya Nishihara <yuya@tcha.org> [Sun, 16 Dec 2018 17:08:46 +0900] rev 40470
context: reimplement memfilectx.cmp()
If I added a sanity check to basefilectx, test-context.py exploded. This
patch copies the naive implementation from overlayworkingfilectx.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Dec 2018 14:55:06 -0500] rev 40469
windows: ensure pure posixfile fd doesn't escape by entering context manager
There are tests in test-revlog-mmapindex.t and test-rebase-mq-skip.t that are
fixed by this, but we usually don't use --pure on Windows. For whatever reason,
the remaining --pure failures are various errors like $ENOTDIR$ and "Access is
denied" have a trailing '.'.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Dec 2018 13:54:37 -0500] rev 40468
vfs: ensure closewrapbase fh doesn't escape by entering context manager
I'm not sure if there's a problem in practice here, as there's no test failure
either way. The __exit__() and close() methods raise an exception, so maybe
__exit__() and close() are being called directly on the underlying handle when
delayclosedfile is used on a context manager? I doubt that was intended.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Dec 2018 13:41:34 -0500] rev 40467
windows: ensure mixedfilemodewrapper fd doesn't escape by entering context mgr
Otherwise it seems that the special read and write handling would be bypassed.
Matt Harbison <matt_harbison@yahoo.com> [Sat, 15 Dec 2018 01:26:18 -0500] rev 40466
py3: ensure the proxied Windows fd doesn't escape by entering context manager
The purpose of the proxy class is to provide the `name` attribute which contains
the file path. But in tests that used a context manager, it still blew up
complaining that 'int' doesn't have a 'startswith' function.
Julien Cristau <jcristau@mozilla.com> [Wed, 12 Dec 2018 06:41:19 +0100] rev 40465
test: fix test-http-bad-server with current python 2.7
https://github.com/python/cpython/pull/2825 changed the exception
message for empty http status line.
Differential Revision: https://phab.mercurial-scm.org/D5412
Matt Harbison <matt_harbison@yahoo.com> [Sun, 09 Dec 2018 23:48:50 -0500] rev 40464
hgweb: register web.comparisoncontext to the config table
This was caught in some server side logging added to debug py3 issues.
Augie Fackler <raf@durin42.com> [Tue, 04 Dec 2018 17:04:19 -0500] rev 40463
Added signature for changeset 1c8c54cf9725
Augie Fackler <raf@durin42.com> [Tue, 04 Dec 2018 17:04:17 -0500] rev 40462
Added tag 4.8.1 for changeset 1c8c54cf9725
Martin von Zweigbergk <martinvonz@google.com> [Tue, 20 Nov 2018 14:43:27 -0800] rev 40461
rebase: fix path auditing to audit path relative to repo root (issue5818)
Before this patch, when rebasing a file called "foo/bar", we would
check e.g. if "/foo" (i.e. rooted at the file system root) was a
symlink.
Differential Revision: https://phab.mercurial-scm.org/D5361
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 Dec 2018 08:56:43 -0800] rev 40460
tests: show bad path auditing in in-memory rebase
Thanks to Yuya for providing this test case in
https://bz.mercurial-scm.org/show_bug.cgi?id=5818.
Differential Revision: https://phab.mercurial-scm.org/D5368
Martin von Zweigbergk <martinvonz@google.com> [Tue, 04 Dec 2018 08:55:48 -0800] rev 40459
tests: add a missing "cd .." to test-rebase-inmemory.t
Differential Revision: https://phab.mercurial-scm.org/D5367
Yuya Nishihara <yuya@tcha.org> [Sun, 28 Oct 2018 21:29:04 +0900] rev 40458
rust: fix possible out-of-bounds read through index_get_parents()
index_get_parents() is an internal function, which doesn't check if the
specified rev is valid. If rustlazyancestors() were instantiated with an
invalid stoprev, it would access to invalid memory region.
This is NOT a security fix as there's no Python code triggering the bug,
but included in this series to not give a notion about the memory issue
fixed by the previous patch.
Yuya Nishihara <yuya@tcha.org> [Thu, 01 Nov 2018 20:32:59 +0900] rev 40457
revlog: fix out-of-bounds access by negative parents read from revlog (SEC)
82d6a35cf432 wasn't enough. Several callers don't check negative revisions
but for -1 (nullrev), which would directly lead to out-of-bounds read, and
buffer overflow could follow. RCE might be doable with carefully crafted
revlog structure, though I don't think this would be useful attack surface.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 03 Dec 2018 11:14:44 -0800] rev 40456
rebase: fix dir/file conflict detection when using in-mem merge
Differential Revision: https://phab.mercurial-scm.org/D5360
Martin von Zweigbergk <martinvonz@google.com> [Mon, 03 Dec 2018 11:11:34 -0800] rev 40455
tests: show that in-mem rebase does not find path dir/file conflicts
Differential Revision: https://phab.mercurial-scm.org/D5359
Matt Harbison <matt_harbison@yahoo.com> [Mon, 03 Dec 2018 20:59:48 -0500] rev 40454
extdiff: register the configuration generated commands with a help category
Otherwise, 'extdiff' shows up under file management and the rest of the commands
are at the bottom under 'Uncategorized'.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 03 Dec 2018 09:36:40 -0800] rev 40453
rebase: abort in-mem rebase if there's a dirty merge state
In-memory merge uses the on-disk merge state, so we should not allow
it run in-memory merge when the merge state is not clean. We should
probably not use the on-disk merge state when running in-memory merge,
but chaning that is not suitable for the stable branch.
Differential Revision: https://phab.mercurial-scm.org/D5357
Martin von Zweigbergk <martinvonz@google.com> [Fri, 30 Nov 2018 16:21:37 -0800] rev 40452
rebase: preserve working copy when redoing in-mem rebase on disk
When in-memory rebase runs into conflicts, we retry it on disk. But
before we do that, we abort the in-memory rebase. That is done because
even though it's mostly in memory, there are still a few state files
written (e.g. the merge state). We should make it not write those
files so we don't need to abort, but for the stable branch, let's
explicitly clear the state we need to clear instead of running the
usual abort code.
Differential Revision: https://phab.mercurial-scm.org/D5356
Martin von Zweigbergk <martinvonz@google.com> [Fri, 30 Nov 2018 15:08:43 -0800] rev 40451
tests: show that in-mem rebase falling back loses state
Both working copy changes and the merge state is lost when in-memory
rebase falls back to on-disk mode.
Differential Revision: https://phab.mercurial-scm.org/D5355
Yuya Nishihara <yuya@tcha.org> [Mon, 03 Dec 2018 21:45:15 +0900] rev 40450
commandserver: get around ETIMEDOUT raised by selectors2
selector.select() should exits with an empty event list on timed out, but
selectors2 raises OSError if timeout expires while recovering from EINTR.
Spotted while debugging new chg feature.
Yuya Nishihara <yuya@tcha.org> [Mon, 03 Dec 2018 21:31:19 +0900] rev 40449
selectors2: backport minimal fix of timeout handling from 2.0.1
The original code would raise TypeError since OSError() doesn't support
keyword arguments.
We can't simply import the selectors 2.0.1, which still spawns "uname -p"
through platform.system(). We could switch to the unreleased version, but
I decided to not right now to minimize the change.