Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:07:33 +0100] rev 40819
mmapindex: move the 'mmapindexthreshold' option out of experimental
The option is useful and should be advertised more. We move it out of
experimental as a first step. The `storage` section is selected as this is
related to how the storage is accessed. A new 'performance' section might be
more appropriate.
We move from 'mmapindexthreshold` to `mmap-threshold` as non-index item are
also suitable for mmap (eg: the rev-branch-cache).
If relevant, we can introduce sub-option `mmap-threshold.revlog-index` later.
Boris Feld <boris.feld@octobus.net> [Sat, 01 Dec 2018 15:57:27 +0100] rev 40818
perf: add a --rev attribute to perfindex
This allow for benchmarking the time necessary to look for other version than
the tip.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:03:38 +0100] rev 40817
perf: update perfindex to be more realistic
The previous code was creating a revlog manually, we now use the actual
`localrepo` method to create it.
We have to jump though extra hops to work around the impact of filecache.
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:09:46 -0800] rev 40816
match: drop unnecessary wrapping of regex in group
It seems the regexes have been wrapped in an unnamed group since
b6c42714d900 (Add locate command., 2005-07-05). In that commit, the
grouping was needed because there was a "head" ('^') added before the
group and a "tail" (os.sep) added after it. It seems the head was
moved inside the group in 1c0c413cccdd (Get add and locate to use new
repo and dirstate walk code., 2005-07-18) and the tail was moved
inside the group in 89985a1b3427 (Clean up walk and changes code to
use normalised names properly., 2005-07-31), So it seems to me that
we've carried around the unnecessary group for 13 years. This patch
removes it.
Differential Revision: https://phab.mercurial-scm.org/D5352
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:45:20 -0800] rev 40815
match: use _BASE_SIZE instead of magic value 4
Differential Revision: https://phab.mercurial-scm.org/D5351
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:44:49 -0800] rev 40814
match: make "groupsize" include the trailing "|"
I think this is a little easier to follow and it will simplify later
patches too.
Differential Revision: https://phab.mercurial-scm.org/D5350
Martin von Zweigbergk <martinvonz@google.com> [Sun, 02 Dec 2018 13:09:43 -0800] rev 40813
match: fix an unaligned (but harmless) indent
Differential Revision: https://phab.mercurial-scm.org/D5349
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:41:10 +0100] rev 40812
match: raise an Abort error instead of OverflowError
This case of OverflowError (one single pattern being too large) has never been
properly caught in the past.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 21:02:02 +0100] rev 40811
match: avoid translating glob to matcher multiple times for large sets
For hgignore with many globs, the resulting regexp might not fit under the 20K
length limit. So the patterns need to be broken up in smaller pieces.
Before this change, the logic was re-starting the full process from scratch
for each smaller pieces, including the translation of globs into regexp.
Effectively doing the work over and over.
If the 20K limit is reached, we are likely in a case where there is many such
glob, so exporting them is especially expensive and we should be careful not
to do that work more than once.
To work around this, we now translate glob to regexp once and for all. Then,
we assemble the resulting individual regexp into valid blocks.
This raises a very significant performance win for large `.hgignore file`:
Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66)
After: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100)
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:25:49 +0100] rev 40810
match: extract function that group regexps
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:16:05 +0100] rev 40809
match: test for overflow error in pattern
If a single pattern is too large to handle, we raise an exception. This case is
now doctested.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 17:20:32 +0100] rev 40808
match: extract a literal constant into a symbolic one
Matt Harbison <matt_harbison@yahoo.com> [Sat, 01 Dec 2018 21:42:48 -0500] rev 40807
tests: apply binary mode to output in seq.py
I noticed this when playing with running tests using WSL, and iterating over the
output yielded '0\r', '1\r',... Most of the other *.py tools do this, and `seq`
on MSYS lacks '\r' in the output, so this is more consistent.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 01:09:37 +0100] rev 40806
perf: add a `--clear-caches` to `perfbranchmapupdate`
This flag will help to measure the time we spend loading various cache that
support the branchmap update.
Example for an 500 000 revisions repository:
hg perfbranchmapupdate --base 'not tip' --target 'tip'
! wall 0.000860 comb 0.000000 user 0.000000 sys 0.000000 (best of 336)
hg perfbranchmapupdate --base 'not tip' --target 'tip' --clear-caches
! wall 0.029494 comb 0.030000 user 0.030000 sys 0.000000 (best of 100)
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 21:11:47 +0000] rev 40805
perf: start from an existing branchmap if possible
If the --base set if a superset of one of the cached branchmap, we should use as
a starting point. This greatly help the overall runtime of
`hg perfbranchmapupdate`
For example, for a repository with about 500 000 revisions, using this trick
make the command runtime move from about 200 second to about 10 seconds. A 20x
gain.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 20:35:22 +0000] rev 40804
perf: rely on repoview for perfbranchmapupdate
Using 'repoview' matching the base and target subset make the benchmark more
realistic. It also unlocks optimization to make the command initialization
faster.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 22:56:06 +0100] rev 40803
perf: pre-indent some code in `perfbranchmapupdate`
This make the next patch easier to read.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 12:02:25 +0000] rev 40802
perf: add a `perfbranchmapupdate` command
This command benchmark the time necessary to update the branchmap between two
sets of revisions. This changeset introduce a first version, doing nothing fancy
regarding cache or other internal details.
Anton Shestakov <av6@dwimlabs.net> [Mon, 05 Nov 2018 13:52:19 +0800] rev 40801
push: config option to control behavior when pushing to a publishing server
Pushing to a publishing server by mistake can lead to a difficult situation to
solve because evolution doesn't work on public changesets. This new
experimental config tries to help avoiding unintentionally (or at least being
aware of) pushing to publishing remotes.
`hg push --publish` can be used to make push succeed even when auto-publish is
set to 'abort'.
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 30 Nov 2018 17:42:55 +0300] rev 40800
narrowcommands: remove an unrequired `repo.narrowpats` call
We call that few lines above and do nothing significant in between which can
change the narrowpats. So let's use values returned by that call.
Differential Revision: https://phab.mercurial-scm.org/D5348
Augie Fackler <augie@google.com> [Thu, 29 Nov 2018 16:44:01 -0500] rev 40799
manifest: reject lines shorter than 42 bytes, not 22
Yuya correctly spotted during the review of f27f8e9ef1e73 that we're
dealing with hexlified hashes here, and so it should be 42 bytes not
22.
Differential Revision: https://phab.mercurial-scm.org/D5347
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 20:05:38 +0900] rev 40798
blackbox: initialize logger with repo instance
The blackboxlogger is unusable without a repo. Let's simply initialize it
with a repo instance.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Nov 2018 20:56:25 +0900] rev 40797
blackbox: do not nullify repo to deactivate the logger on failure
The _repo will be a mandatory attribute. Instead, make the logger to not
track any events.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 20:02:34 +0900] rev 40796
blackbox: extract global last logger to proxylogger class
So the blackboxlogger can be instantiated with a repo.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 19:36:21 +0900] rev 40795
ui: pass in bytes opts dict to logger.log()
This is the convention of the Mercurial API.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 19:35:33 +0900] rev 40794
ui: pass in formatted message to logger.log()
This makes sure that all logger instances will handle the message arguments
properly.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 17:34:46 +0900] rev 40793
blackbox: send debug message to logger by core ui
Since the core ui.log() may recurse into ui.log() through ui.debug(), it
must guard against recursion.
The ui extension class can finally be removed.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Nov 2018 20:23:50 +0900] rev 40792
blackbox: change the way of deactivating the logger on write error
This prepares for the upcoming code move. The recursion guard will be ported
to the core ui.
Martin von Zweigbergk <martinvonz@google.com> [Wed, 28 Nov 2018 10:12:50 -0800] rev 40791
match: remove obsolete catching of OverflowError
Since 0f6a1bdf89fb (match: handle large regexes, 2007-08-19), we catch
an OverflowError from the regex engine and split up the regex if that
happens. In 59a9dc9562e2 (ignore: split up huge patterns, 2008-02-11),
that was extended to raise an OverflowError in our code even if the
regex engine doesn't raise it. It's unclear if there was a range of
regex sizes where the OverflowError would be raised from the regex
engine but that were still below the limit we added in our
code. Either way, both limitations were probably removed in Python
2.7.4 when the regex code width was extended from 16bit to 32bit (or
Py_UCS4) integer (thanks to Yuya for finding that out).
If at least the first limitation was removed, we no longer should be
using OverflowError for flow control, so this patch changes that.
Differential Revision: https://phab.mercurial-scm.org/D5309
Boris Feld <boris.feld@octobus.net> [Tue, 27 Nov 2018 02:10:14 +0100] rev 40790
sparse: raise a move verbose index error from the C code
If we don't like a value we should print it.
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 05 Oct 2018 23:10:56 +0300] rev 40789
narrow: drop the bundle2 capability since we have server capabilities (BC)
This patch drops the narrow bundle2 capabilities since we introduced narrow
server capabilities which are more nice and now used everywhere.
I am not sure what it can affect, so on safe side I marked this as BC. Also I
removed the NARROWCAP constant as that kind of conflicts with the same name
constant in wireprototypes.py.
Differential Revision: https://phab.mercurial-scm.org/D4892
Boris Feld <boris.feld@octobus.net> [Sun, 02 Jul 2017 04:06:24 +0200] rev 40788
vfs: extract the audit path logic into a submethod
This will make it possible to apply it in more cases.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 20:01:28 +0100] rev 40787
subrepo-git: use an official origvfs when appropriate
The origvfs has the auditor properly set and can move file without issue.
The current code is currently working without errors because rename are not
audited, yet.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 19:26:05 +0100] rev 40786
revert: extract origvfs logic in a sub-function
The subrepo's "revert" logic could benefit from it.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 18:44:07 +0100] rev 40785
vfs: treat 'undo.' file the same as 'journal.' file
They are the same kind of file, they are protected by the store lock, but
directly lives inside the '.hg' directory.
No warnings were ever raised about them because `vfs.rename` is not audited.
Something we are trying to change.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 21:00:13 +0100] rev 40784
perf: add a perfignore command
The command is meant to benchmark operations related to hgignore. Right now the
command is benchmarking the loading time of the hgignore rules.
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:36:06 +0300] rev 40783
py3: use pycompat.xrange instead of xrange
xrange does not exists on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5302
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 27 Nov 2018 16:16:13 +0300] rev 40782
store: write fncache only once if there are both adds and removes
Differential Revision: https://phab.mercurial-scm.org/D5307
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 17:44:24 +0000] rev 40781
perf: disable revlogs clearing in `perftags` by default
This aligns things with what `perfbookmarks` does. I decided to disable the
revlogs clearing by default to focus on the core logic by default, ignoring
side effects.
If we prefer to emphasize the side effect, we can instead keep this on in
`perftags` and enable it by default in `perfbookmarks`.
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 10:55:20 +0000] rev 40780
perf: add a `clear-revlogs` flag to `perfbookmarks`
This flag (off by default) makes it possible to enable the refresh of the
changelog and revlog. This is useful to check for costly side effects of
bookmark loading.
Usually, these side effects are shared with other logics (eg: tags).
example output in my mercurial repo (with 1 bookmark, so not a great example):
$ hg perfbookmarks
! wall 0.000044
$ hg perfbookmarks --clear-revlogs
! wall 0.001380
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 10:38:15 +0000] rev 40779
tags: cache `repo.changelog` access when checking tags nodes
The tags reading process checks if the nodes referenced in tags exist. Caching
the access to `repo.changelog` provides a large speedup for repositories with
many tags.
running `hg perftags` in a large private repository
before: ! wall 0.393464 comb 0.390000 user 0.330000 sys 0.060000 (median of 25)
after: ! wall 0.267711 comb 0.270000 user 0.210000 sys 0.060000 (median of 38)
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 10:46:20 +0000] rev 40778
perf: add a `clear-revlogs` flag to `perftags`
This flag (on by default) makes it possible to disable the refresh of the
changelog and revlog. This is useful to check for the time spent in the core
tags logic without the associated side effects. Usually, these side effects
are shared with other logics (eg: bookmarks).
Example output in my Mercurial repository
$ hg perftags
! wall 0.017919 comb 0.020000 user 0.020000 sys 0.000000 (best of 141)
$ hg perftags --no-clear-revlogs
! wall 0.012982 comb 0.010000 user 0.010000 sys 0.000000 (best of 207)
Boris Feld <boris.feld@octobus.net> [Sun, 25 Nov 2018 13:37:53 +0100] rev 40777
perf: stop creating new revlog by hand in perftags
It's better to let the repository logic create its own object. We now just clear
the cache. New object will be automatically created from there.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:23:12 +0100] rev 40776
revlog: update the documentation for `trim_endidx`
The function role drifted since the function was commented.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:21:09 +0100] rev 40775
revlog: properly detect corrupted revlog in `index_get_length`
Pointed out by Yuya Nishihara.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:15:12 +0100] rev 40774
perf: rename `perfhelper-tracecopies` to `perfhelper-pathcopies`
The command it supports is called `perfpathcopies`. It seems better to align the
names.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:13:50 +0100] rev 40773
perf: add a docstring to `perfpathcopies`
This will help people to find this command.
Boris Feld <boris.feld@octobus.net> [Mon, 26 Nov 2018 00:08:11 +0100] rev 40772
revlog: update the docstring of `ancestors` to match reality
Code using this method expect the revision to be (reverse) sorted. As pointed by
Yuya Nishihara, the docstring should reflect that.
Augie Fackler <augie@google.com> [Mon, 26 Nov 2018 15:53:34 -0500] rev 40771
remotefilelog: fix typo in docstring
Differential Revision: https://phab.mercurial-scm.org/D5306
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 23 Nov 2018 18:58:16 +0300] rev 40770
store: append to fncache if there are only new files to write
Before this patch, if we have to add a new entry to fncache, we write the whole
fncache again which slows things down on large fncache which have millions of
entries. Addition of a new entry is common operation while pulling new files or
commiting a new file.
This patch adds a new fncache.addls set which keeps track of the additions
happening and store them. When we write the fncache, we will just read the addls
set and append those entries at the end of fncache.
We make sure that the entries are new entries by loading the fncache and making
sure entry does not exists there. In future if we can check if an entry is new
without loading the fncache, that will speed up things more.
Performance numbers for commiting a new file:
mercurial repo
before: 0.08784651756286621
after: 0.08474504947662354
mozilla-central
before: 1.83314049243927
after: 1.7054164409637451
netbeans
before: 0.7953150272369385
after: 0.7202838659286499
pypy
before: 0.17805707454681396
after: 0.13431048393249512
In our internal repo, the performance improvement is in seconds.
I have used octobus's ASV perf benchmark thing to get the above numbers. I also
see some minute perf improvements related to creating a new commit without a new
file, but I believe that's just some noise.
Differential Revision: https://phab.mercurial-scm.org/D5301
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:38:35 +0300] rev 40769
py3: fix couple of division operator to do integer divison
Differential Revision: https://phab.mercurial-scm.org/D5305
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:37:48 +0300] rev 40768
py3: use dict.items() instead of dict.iteritems()
dict.iteritems() does not exist on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5304
Pulkit Goyal <pulkit@yandex-team.ru> [Mon, 26 Nov 2018 15:36:51 +0300] rev 40767
py3: convert strings to bytes in tests/test-remotefilelog-histpack.py
Differential Revision: https://phab.mercurial-scm.org/D5303
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 23 Nov 2018 21:54:03 +0300] rev 40766
py3: replace str() with pycompat.bytestr() or ('%d' % int)
After this patch, test-fastannotate-diffopts.t is about to pass. There are some
extra newlines in the output.
Differential Revision: https://phab.mercurial-scm.org/D5300
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 19:32:51 +0100] rev 40765
perf: add a `--timing` argument to `perfhelper-tracecopies`
The new argument will help picking better pair for benchmarking. See
documentation for details.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 18:08:33 +0900] rev 40764
ui: manage logger instances and event filtering by core ui
The setup code in blackbox needs more tweaks since it has lots of black
magics. I'll fix them by follow-up patches.
To be clear, the goal of this series is to provide a proper way for command
server to install its own logger. I need it to debug in-memory repository
cache.
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 21:10:51 +0900] rev 40763
extensions: add "uipopulate" hook, called per instance, not per process
In short, this is the "reposetup" function for ui. It allows us to modify
ui attributes without extending ui.__class__. Before, the only way to do
that was to abuse the config dictionary, which is copied across ui instances.
See the next patch for usage example.
Yuya Nishihara <yuya@tcha.org> [Sat, 17 Nov 2018 19:11:45 +0900] rev 40762
hgweb: load globally-enabled extensions explicitly
Before, extensions were loaded as a side effect of hg.repository() if the
hgweb was executed as a CGI/WSGI. I want to make it explicit so that another
ui hook can be inserted after extensions.loadall().
Matt Harbison <matt_harbison@yahoo.com> [Sat, 24 Nov 2018 14:11:02 -0500] rev 40761
tests: disable remotefilelog on Windows
I've spent a non trivial amount of time trying to eliminate the test errors, but
it's looking like this is pretty dependent on Unix support. For example, there
are attempts to delete open files, and uses of threads that report I/O attempts
on closed files. (Maybe this is a race condition? Don't we usually use
processes as workers on Windows?)
In any event, I don't want real new errors elsewhere to be masked by these known
problems.
For some reason $CACHEDIR is reported as missing in test-remotefilelog-repack.t,
but it actually exists in the hgcloneshallow call inside
shallowutil.mkstickygroupdir(). By the time the process exits, it's gone. I
don't see it being removed by code that calls 'rmdir' or 'remove' in the
extension itself.
Boris Feld <boris.feld@octobus.net> [Thu, 22 Nov 2018 23:48:44 +0100] rev 40760
perf: run 'setup' function during stub run
The benchmarked function might need the content of the setup to be run in order
to function properly.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 02:08:01 +0100] rev 40759
perf: fallback to subset if ondisk cache is missing in perfbranchmapload
If there is no branchmap on disk for that filter, it means that the cache from
some subset's filter is relevant for this one. We look for it instead of
aborting.
That way it is much simpler to run the command in an automated way. We can now
add it to `test-contrib-perf.t`.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 02:11:11 +0100] rev 40758
perf: prewarm the branchmap in perfbranchmapload
It is not very interesting to have the command randomly failing because the
branchmap for the tested filter happens to be cold. So we make sure to have a
valid up to date branchmap before going further.
The data might still be missing from disk if a subset was equivalent. See next
changeset for details and fix.
Yuya Nishihara <yuya@tcha.org> [Sun, 25 Nov 2018 22:39:54 +0900] rev 40757
localrepo: correct docstring of filectx()
The same reason as b6c2543e1dd8. It can't be any changeset specifiers but
revision number.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 20:05:55 +0900] rev 40756
annotate: remove dead code to not convert path to relative path
It's annotate. There should be at least one file path specified.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 18:41:44 +0900] rev 40755
annotate: start pager soon after command options are validated
It helps extracting helper class.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 18:38:44 +0900] rev 40754
annotate: check and update command options first
It helps extracting helper class.
Yuya Nishihara <yuya@tcha.org> [Sat, 24 Nov 2018 18:36:44 +0900] rev 40753
annotate: format local tables in less-dense form
I'll move these to a class, where one-extra indent will be needed.
Pulkit Goyal <pulkit@yandex-team.ru> [Thu, 22 Nov 2018 18:14:21 +0300] rev 40752
perf: rename version flag of perfchangegroupchangelog to cgversion
The version flag conflicts with the global version flag and does not work.
Differential Revision: https://phab.mercurial-scm.org/D5297
Differential Revision: https://phab.mercurial-scm.org/D5298
Matt Harbison <matt_harbison@yahoo.com> [Fri, 23 Nov 2018 22:27:57 -0500] rev 40751
procutil: don't allow the main 'hg' script to be treated as the Windows exe
Previously, there were a handful of errors like this:
$ hg prefetch --repack
(running background incremental repack)
+ abort: %1 is not a valid Win32 application
+ [255]
CreateProcess() doesn't append .exe when `lpApplicationName` contains a path,
and a python script isn't directly executable.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 23 Nov 2018 22:18:56 -0500] rev 40750
remotefilelog: drop some compatibility cruft for finding the hg exeutable
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 21 Nov 2018 17:17:26 +0300] rev 40749
branchmap: build the revbranchcache._namesreverse() only when required
On big repositories with a lot of named branches and that also increasing over
time, building of this dict can be expensive and shows up in profile.
For our internal repository, this saves ~0.05 seconds.
Thanks to Yuya for suggesting using util.propertycache() and
util.clearcachedproperty().
Differential Revision: https://phab.mercurial-scm.org/D5291
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 11:11:38 +0100] rev 40748
sparse-revlog: put the native implementation of slicechunktodensity to use
When possible, the C implementation of delta chain slicing will be used.
providing a large boost in performance for this operation.
To take a practical example of restoring manifest revision '59547c40bc4c' for
a reference NetBeans repository (using sparse-revlog). The media time of the
step `slice-sparse-chain` of `perfrevlogrevision` improve from 0.660 ms to
0.098 ms;
The full series move delta chain slicing from 1.120 ms to 0.098 ms;
Implementing _slicechunktosize into C would yield further improvements.
However, the performance seems good enough for now.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 11:10:52 +0100] rev 40747
cext: increment the `parsers` module version
We introduced a new `slicechunktodensity` method to the index.
Boris Feld <boris.feld@octobus.net> [Thu, 15 Nov 2018 11:09:58 +0100] rev 40746
sparse-revlog: introduce native (C) implementation of slicechunktodensity
This is a C implementation of `_slicechunktodensity` in the
`mercurial/revlogutils/deltas.py` file.
The algorithm involves a lot of integer manipulation and low-level access to
index data. Having a C implementation of it raises a large performance
improvement. See later changeset in this series for details.
Boris Feld <boris.feld@octobus.net> [Fri, 09 Nov 2018 18:45:23 +0100] rev 40745
sparse-revlog: add a `trim_endidx` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first.
This function implement a subpart of the python function `_trimchunk` in
`mercurial/revlogutils/deltas.py`. Handling of actual Python objects is left
to the caller function.
Boris Feld <boris.feld@octobus.net> [Fri, 09 Nov 2018 18:43:16 +0100] rev 40744
sparse-revlog: add a `index_segment_span` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first. This new function provides
an efficient way to retrieve some of the information needed by
`slicechunktodensity`.
Boris Feld <boris.feld@octobus.net> [Fri, 09 Nov 2018 18:42:58 +0100] rev 40743
sparse-revlog: add a `index_get_length` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first. This new function provides
an efficient way to retrieve some of the information needed by
`slicechunktodensity`.
Boris Feld <boris.feld@octobus.net> [Tue, 20 Nov 2018 14:49:18 +0000] rev 40742
sparse-revlog: add a `index_get_start` function in C
We are about to implement a native version of `slicechunktodensity`. For
clarity, we introduce the helper functions first. This new function provides
an efficient way to retrieve some of the information needed by
`slicechunktodensity`.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:32:32 +0100] rev 40741
perf: add --clear-revlog flag to branchmapload
Having the changelog index already loaded when loading the branchmap can have a
large impact on performance.
Example runs (large private repository):
hg perfbranchmapload -f base
! wall 0.116722 comb 0.120000 user 0.110000 sys 0.010000 (best of 59)
hg perfbranchmapload -f base --clear-revlogs
! wall 0.258246 comb 0.230000 user 0.220000 sys 0.010000 (best of 31)
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 06:32:28 +0100] rev 40740
perf: introduce a function to fully "unload" a changelog
The function remove various attributes and caches related to changelog.
This is getting a common requirement.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 01:32:57 +0100] rev 40739
perf: use an explicit function in perfbranchmapload
This make things clearer.
Boris Feld <boris.feld@octobus.net> [Fri, 23 Nov 2018 01:20:18 +0100] rev 40738
perf: update function name to match `perfbranchmapload` command
Having function with the same name as the command is simpler.
Boris Feld <boris.feld@octobus.net> [Wed, 21 Nov 2018 11:46:47 +0000] rev 40737
perf: fix a minor typo in perfbranchmapload
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:37 +0200] rev 40736
context: floor adjustlinkrev graph walk during copy tracing
The `_adjustlinkrev` method gains an optional "stoprev" argument. The linkrev
adjustment will give up once this floor is reached. The relevant functions
using `_adjustlinkrev` are updated to pass an appropriate value in the copy
tracing code.
In some private repository, about 10% of the status call triggered the
pathological case addressed by this change. The speedup varies from one call
to another, the best-observed win is moving from 170s to 11s.
The effect of this change can be seen in the public pypy repository, running the
following command:
hg perftracecopies --source 83c9ff0c0206 --destination 59c79103d5b0
before: 3.401753 seconds
after: 2.634897 seconds (-23%)
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 14:30:58 +0000] rev 40735
context: small refactoring of `isintroducedafter`
This make the next change clearer.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:36 +0200] rev 40734
context: split `introrev` logic in a sub function
We want to add a mechanism to stop iteration early associated to intro rev early
in some case. However, it does not make sense to expose it in the public
`filectx` API. So we split the code into an internal method instead.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:35 +0200] rev 40733
context: introduce an `isintroducedafter` method and use it in copies
Right now, copy tracing make effort to not traverse the graph too much to save
performance. It uses a "limit" acting as a floor revision past which data are
no longer relevant to the current copy tracing.
However, to enforce this limit, it does a call to `filectx.rev()` and that
call can trigger a graph traversal on its own. That extra graph traversal is
unaware of the current limit and can become very expensive. That cost is
increased by the nature of work done in adjust link rev, we are not only
walking down the graph, we are also checking the affected file for each
revision we walk through. Something significantly more expensive than the walk
itself.
To work around this we need to make the `filectx` operation aware of the
current limit. The first step is to introduce a dedicated method:
`isintroducedafter`. We'll then rework that method logic to stop traversal as
soon as possible.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:50:34 +0200] rev 40732
context: take advantage of `_descendantrev` in introrev if available
Before this changeset, `_descendantrev` was ignored and `introrev` could
return a "wrong" result. I was previously fine because there seems to be no
existing code using both `introrev` and `_descendantrev` at the same time.
However, we would like to change that.
Boris Feld <boris.feld@octobus.net> [Wed, 10 Oct 2018 00:49:30 +0200] rev 40731
context: spell out the logic around linkrev adjustement starting point
We make the intent of the `_changeid` and `_changectx` checks explicit. The
same logic was previously performed by the `self.rev()` call. The new code is
a bit redundant, but much clearer.
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 14:14:56 +0000] rev 40730
perf: add a new `perfhelper-tracecopies` command
The command is not measuring performance itself, it digs interesting statistic
to help pick good arguments for the `perfcopytrace` command.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 21 Sep 2018 09:19:42 -0700] rev 40729
narrow: extract helper for parsing narrowspec file
This will be used for parsing a file that's stored in a different
location.
Differential Revision: https://phab.mercurial-scm.org/D5277
Yuya Nishihara <yuya@tcha.org> [Thu, 22 Nov 2018 22:40:22 +0900] rev 40728
py3: cast attribute name to sysstr in clearcachedproperty()
Pulkit Goyal <pulkit@yandex-team.ru> [Wed, 21 Nov 2018 19:12:12 +0300] rev 40727
py3: whitelist three more passing test caught by buildbot
The number of failing test is back to two digits.
Differential Revision: https://phab.mercurial-scm.org/D5293
Valentin Gatien-Baron <vgatien-baron@janestreet.com> [Wed, 21 Nov 2018 13:08:23 -0500] rev 40726
tests: make test-check-module-imports more robust
It failed for me without this in this way:
tests/test-commandserver.t:19: relative import of stdlib module
tests/test-lfs-serve.t:108: relative import of stdlib module
tests/test-lfs-serve.t:255: relative import of stdlib module
tests/test-lfs-serve.t:362: relative import of stdlib module
tests/test-lfs-serve.t:406: relative import of stdlib module
tests/test-lock.py:9: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-lrucachedict.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-match.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-remotefilelog-datapack.py:15: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-remotefilelog-histpack.py:14: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-simplekeyvaluefile.py:4: imports not lexically sorted: silenttestrunner < unittest, True, True
tests/test-sshserver.py:6: imports not lexically sorted: silenttestrunner < unittest, True, True
This is because every module is considered a stdlib module, because
the stdlib_prefixes is /usr, and my repo is in /usr/local/home, which
means that sys.path contains a couple of
/usr/local/home/../hg/.. entries that count as "in the stdlib".
Fix this by preventing any path in sys.path that's inside the mercurial
source from being considered "in the stdlib".
Differential Revision: https://phab.mercurial-scm.org/D5294
Anton Shestakov <av6@dwimlabs.net> [Sat, 27 Oct 2018 21:13:23 +0800] rev 40725
push: add --publish flag to change phase of pushed changesets
This flag is copied from topic extension, where it proved to be useful. It
makes all pushed changesets public on both ends if the push succeeds.
Doesn't currently work for any subrepos that are implicitly pushed together
with their owner repo.
Boris Feld <boris.feld@octobus.net> [Mon, 19 Nov 2018 17:19:54 +0000] rev 40724
match: reformat `syntaxes` dictionary for better maintainability