Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 09:17:49 -0400] rev 30348
ui: explicitly open config files in binary mode
This has been working mostly accidentally, but now it works explicitly.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 09 Nov 2016 16:04:44 -0800] rev 30347
help: fix double word usage
"most" was used twice.
(I fixed a grammar error before timeless spotted it!)
Jun Wu <quark@fb.com> [Wed, 09 Nov 2016 22:08:30 +0000] rev 30346
setup: move cffi stuff to mercurial/cffi
This patch moves all setup*cffi stuff to mercurial/cffi to make the root
directory cleaner. The idea was from mpm [1]:
> It seems like we could have a fair amount of cffi definitions, and
> cluttering the root directory (or mercurial/) with them is probably not
> a great long-term solution. We could probably add a cffi/ directory
> under mercurial/ to parallel pure/.
[1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-July/086442.html
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30345
manifest: remove manifest.add and add memmfctx.write
This removes one more dependency on the manifest class by moving the write
functionality onto the memmanifestctx classes and changing the one consumer to
use the new API.
By moving the write path to a manifestctx, we now give the individual manifests
control over how they're read and serialized. This will be useful in developing
new manifest formats and storage systems.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30344
context: add manifestctx property on changectx
This allows us to access the manifestctx for a given commit. This will be used
in a later patch to be able to copy the manifestctx when we want to make a new
commit.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30343
manifest: add copy to mfctx classes
This adds copy functionality to the manifestctx classes. This will be used in an
upcoming diff to copy a manifestctx during commit so we can modify the manifest
before committing.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30342
manifest: introduce memmanifestctx and memtreemanifestctx
This introduces two new classes to represent in-memory manifest instances.
Similar to memchangectx, this lets us prepare a manifest in memory, then in a
future patch we will add the apis that can commit this in memory structure.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30341
manifestctx: add _revlog() function
The `self._repo.manifestlog._revlog` code is getting copy and pasted a lot in
manifestctx. Let's make it a function so it can be reused. This will make future
patches cleaner too.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30340
manifest: remove manifest.find
As part of removing dependencies on manifest, this drops the find function and
fixes up the two existing callers to use the equivalent apis on manifestctx.
Durham Goode <durham@fb.com> [Tue, 08 Nov 2016 08:03:43 -0800] rev 30339
changegroup: remove remaining uses of repo.manifest
The remaining uses of repo.manifest in the changegroup module are treating the
manifest exclusively as a revlog, so let's replace them with instances of the
revlog directly.
This is part of dropping all dependencies on repo.manifest in favor of
repo.manifestlog.
Martin von Zweigbergk <martinvonz@google.com> [Fri, 04 Nov 2016 13:49:15 -0700] rev 30338
treemanifest: fix a "treeinmem" case
f2c5b9d48b29 (manifest: make treemanifestctx store the repo,
2016-10-18) broke most tests when run with treeinmem=True. The
treeinmem mode can not be enabled by the user, so this did not break
anything in practice, but it's useful to have it working for testing
the treemanifest code.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 06 Nov 2016 10:46:55 -0800] rev 30337
perf: support measuring bdiff for all changeset related data
The --all argument changes the behavior of `perfbdiff` to pull
in fulltext revision pairs for all changes related to a changeset.
The p1 and p2 manifests will be bdiffed against current. Every file
that changed between p1 and current will have its file revisions
loaded and bdiffed.
This mode of operation effectively measured the bdiff time required
for `hg commit`.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 06 Nov 2016 11:01:25 -0800] rev 30336
perf: support bdiffing multiple revisions in a single revlog
This is useful for testing bdiff performance on several revision
pairs at a time.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 06 Nov 2016 09:51:14 -0800] rev 30335
perf: prepare to handle multiple pairs in perfbdiff
Before, we only supported benchmarking a single pair of texts
with bdiff. We want to enable feeding larger corpora into this
benchmark. So rewrite the code to support that.
Yuya Nishihara <yuya@tcha.org> [Wed, 09 Nov 2016 22:06:09 +0900] rev 30334
py3: document why os.fsencode() can be used to get back bytes argv
And a possible Windows issue. I'm sad we have to do such ugly hack, but
that's the unicode on Python 3.
Yuya Nishihara <yuya@tcha.org> [Wed, 09 Nov 2016 22:15:51 +0900] rev 30333
py3: update test-check-py3-compat.t output
4b1af1c867fa (scmutil: move util.termwidth()) changed where the import fails.
Mads Kiilerich <madski@unity3d.com> [Mon, 17 Oct 2016 23:16:55 +0200] rev 30332
spelling: fixes of non-dictionary words
Durham Goode <durham@fb.com> [Thu, 03 Nov 2016 17:31:14 -0700] rev 30331
manifest: add __nonzero__ method
This adds a __nonzero__ method to manifestdict. This isn't strictly necessary in
the vanilla Mercurial implementation, since Python will handle nonzero checks by
using __len__, but having it implemented here makes it easier for alternative
implementations to implement __nonzero__ and have them be plug-n-play with the
normal implementation.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 04:36:26 +0530] rev 30330
py3: have bytes version of sys.argv
sys.argv returns unicodes on Python 3. We need a bytes version for us.
There was also a python bug/feature request which wanted then to implement
one. They rejected and it is quoted in one of the comments that we can use
fsencode() to get a bytes version of sys.argv. Though not sure about its
correctness.
Link to the comment: http://bugs.python.org/
issue8776#msg217416
After this patch we will have pycompat.sysargv which will return us bytes
version of sys.argv. If this patch goes in, i will like to make transformer
rewrite sys.argv with pycompat.argv because there are lot of occurences.
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 09:00:47 -0400] rev 30329
util: use '\\' rather than using r'\'
We need bytes, and I find this just a little more immediately obvious
than doing rb'\'.
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 09:03:10 -0400] rev 30328
util: use pycompat urlunquote function
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 09:02:25 -0400] rev 30327
pycompat: introduce an alias for urllib.unquote
We have to use unquote_to_bytes on Python 3, so we need an abstraction
for this.
Christian Ebert <blacktrash@gmx.net> [Mon, 17 Oct 2016 17:42:46 +0200] rev 30326
keyword: handle filectx _customcmp
Suggested by Yuya Nishihara:
https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-October/089461.html
Related to
issue5364.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 22:20:31 +0900] rev 30325
mail: do not print(), use ui.debug() instead
Since print() can't take a bytes output, it's pretty useless in Mercurial
on Python 3. As this is a debug message, switching to ui.debug() seems fine.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 22:12:48 +0900] rev 30324
progress: obtain stderr from ui
This will help Python 3 porting.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 22:09:50 +0900] rev 30323
simplemerge: obtain stdout from ui
This will help Python 3 porting.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 22:07:03 +0900] rev 30322
profiling: obtain stderr from ui
This will help Python 3 porting.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 13 Nov 2016 06:12:22 +0900] rev 30321
scmutil: ignore EPERM at os.utime, which avoids ambiguity at closing
According to POSIX specification, just having group write access to a
file causes EPERM at invocation of os.utime() with an explicit time
information (e.g. working on the repository shared by group access
permission).
To ignore EPERM at closing file object in such case, this patch makes
checkambigatclosing._checkambig() use filestat.avoidambig() introduced
by previous patch.
Some functions below imply this code path at truncation of an existing
(= might be owned by another user) file.
- strip() in repair.py, introduced by
e38d85be978f
- _playback() in transaction.py, introduced by
599912a62ff6
This is a variant of
issue5418.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 13 Nov 2016 06:11:56 +0900] rev 30320
vfs: ignore EPERM at os.utime, which avoids ambiguity at renaming (
issue5418)
According to POSIX specification, just having group write access to a
file causes EPERM at invocation of os.utime() with an explicit time
information (e.g. working on the repository shared by group access
permission).
To ignore EPERM at renaming in such case, this patch makes
vfs.rename() use filestat.avoidambig() introduced by previous patch.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 13 Nov 2016 06:06:23 +0900] rev 30319
util: add utility function to skip avoiding file stat ambiguity if EPERM
Now, advancing stat.st_mtime by os.utime() is used to avoid file stat
ambiguity. But according to POSIX specification, utime(2) with an
explicit time information is permitted only for a process with:
- the effective user ID equal to the user ID of the file, or
- appropriate privileges
http://pubs.opengroup.org/onlinepubs/
9699919799/functions/utime.html
Therefore, just having group write access to a file causes EPERM at
applying os.utime() on it (e.g. working on the repository shared by
group access permission).
This patch adds class filestat utility function avoidamgig() to avoid
file stat ambiguity but skip it if EPERM.
It is reasonable to always ignore EPERM, because utime(2) causes EPERM
only in the case described above (EACCES is used only for utime(2)
with NULL).
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 06 Nov 2016 18:51:57 -0800] rev 30318
bdiff: replace hash algorithm
This patch replaces lyhash with the hash algorithm used by diffutils.
The algorithm has its origins in Git commit
2e9d1410, which is all the
way back from 1992. The license header in the code at that revision
in GPL v2.
I have not performed an extensive analysis of the distribution
(and therefore buckets) of hash output. However, `hg perfbdiff`
gives some clear wins. I'd like to think that if it is good enough
for diffutils it is good enough for us?
From the mozilla-unified repository:
$ perfbdiff -m
3041e4d59df2
! wall 0.053271 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
! wall 0.035827 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
$ perfbdiff
0e9928989e9c --alldata --count 100
! wall 6.204277 comb 6.200000 user 6.200000 sys 0.000000 (best of 3)
! wall 4.309710 comb 4.300000 user 4.300000 sys 0.000000 (best of 3)
From the hg repo:
$ perfbdiff 35000 --alldata --count 1000
! wall 0.660358 comb 0.660000 user 0.660000 sys 0.000000 (best of 15)
! wall 0.534092 comb 0.530000 user 0.530000 sys 0.000000 (best of 19)
Looking at the generated assembly and statistical profiler output
from the kernel level, I believe there is room to make this function
even faster. Namely, we're still consuming data character by character
instead of at the word level. This translates to more loop iterations
and more instructions.
At this juncture though, the real performance killer is that we're
hashing every line. We should get a significant speedup if we change
the algorithm to find the longest prefix, longest suffix, treat those
as single "lines" and then only do the line splitting and hashing on
the parts that are different. That will require a lot of C code,
however. I'm optimistic this approach could result in a ~2x speedup.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 04 Nov 2016 21:44:25 -0700] rev 30317
profiling: make statprof the default profiler (BC)
The statprof sampling profiler runs with significantly less overhead.
Its data is therefore more useful. Furthermore, its default output
shows the hotpath by default, which I've found to be way more useful
than the default profiler's function time table.
There is one behavioral regression with this change worth noting:
the statprof profiler currently doesn't profile individual hgweb
requests like lsprof does. This is because the current implementation
of statprof only profiles the thread that started profiling.
The ability for lsprof to profile individual hgweb requests is
relatively new and likely not widely used. Furthermore, I have plans
to modify statprof to support profiling multiple threads. I expect
that change to go through several iterations. I'm submitting this
patch first so there is more time to test statprof. Perfect is the
enemy of good.
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 04 Nov 2016 20:50:38 -0700] rev 30316
profiling: use vendored statprof and upstream enhancements (BC)
Now that the statprof module is vendored and suitable for use, we
switch our statprof profiler to use it. This required some minor
changes because of drift between the official statprof profiler
and the vendored copy.
We also incorporate Facebook's improvements from the "statprofext"
extension at
https://bitbucket.org/facebook/hg-experimental, notably support for
different display formats.
Because statprof output is different, this is marked as BC. Although
most users likely won't notice since most users don't profile.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 23:16:32 +0900] rev 30315
crecord: use scmutil.termsize()
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 23:09:05 +0900] rev 30314
scmutil: extend termwidth() to return terminal height, renamed to termsize()
It appears crecord.py has its own termsize() function. I want to get rid of it.
The fallback height is chosen from the default of cmd.exe on Windows, and
VT100 on Unix.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 22:57:12 +0900] rev 30313
scmutil: clarify that we explicitly do termwidth - 1 on Windows
I was a bit confused since we didn't add 1 to the width, which is different
from the example shown in StackOverflow.
http://stackoverflow.com/a/
12642749
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 21:57:32 +0900] rev 30312
scmutil: remove superfluous indent from termwidth()
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 21:50:29 +0900] rev 30311
scmutil: narrow ImportError handling in termwidth()
The array module must exist. It's sufficient to suppress the ImportError of
termios. Also salvaged the comment why we have to handle AttributeError, from
7002bb17cc5e.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 21:42:11 +0900] rev 30310
scmutil: make termwidth() obtain stdio from ui
I'm getting rid of direct sys.stderr|out|in references so Py3 porting will
be slightly easier.
Yuya Nishihara <yuya@tcha.org> [Thu, 20 Oct 2016 21:38:44 +0900] rev 30309
scmutil: move util.termwidth()
I'm going to get rid of sys.stderr|out|in references from posix.termwidth().
In order to do that, termwidth() needs to take a ui, but functions in util.py
shouldn't depend on a ui object. So moves termwidth() to scmutil.py.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 06 Nov 2016 00:37:50 -0700] rev 30308
bdiff: don't check border condition in loop
`plast = a + len - 1`. So, this "for" loop iterates from "a" to "plast",
inclusive. So, `p == plast` can only be true on the final iteration
of the loop. So checking for it on every loop iteration is wasteful.
This patch simply decreases the upper bound of the loop by 1 and
adds an explicit check after iteration for the `p == plast` case.
We can't simply add 1 to the initial value for "i" because that
doesn't do the correct thing on empty input strings.
`perfbdiff -m
3041e4d59df2` on the Firefox repo becomes significantly
faster:
! wall 0.072763 comb 0.070000 user 0.070000 sys 0.000000 (best of 100)
! wall 0.053221 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
For the curious, this code has its origins in
8b067bde6679, which is
the changeset that introduced bdiff.c in 2005.
Also, GNU diffutils is able to perform a similar line-based diff in
under 20ms. So there's likely more perf wins to be found in this code.
One of them is the hashing algorithm. But it looks like mpm spent
some time testing hash collisions in
d0c48891dd4a. I'd like to do the
same before switching away from lyhash, just to be on the safe side.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 05 Nov 2016 23:41:52 -0700] rev 30307
perf: add perfbdiff
bdiff shows up a lot in profiling. I think it would be useful to have
a perf command that runs bdiff over and over so we can find hot spots.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 06:54:31 +0530] rev 30306
help: show help for disabled extensions (
issue5228)
This patch does not exactly solve
issue5228 but it results in a better
condition on this issue. For disabled extensions, we used to parse the
module and get the first occurrences of docstring and then return the first
line of that as an introductory heading of extension. This is what we get
today.
This patch returns the whole docstring of the module as a help for extension,
which is more informative. There are some modules which don't have much
docstring at top level except the heading so those are unaffected by this
change. To follow the existing trend of showing commands either we have to
load the extension or have a very ugly parsing method which don't even assure
correctness.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 04:17:19 +0530] rev 30305
py3: make scmutil.rcpath() return bytes
This patch make sure scmutil.rcpath() returns bytes independent of
which platform is used on Python 3. If we want to change type for windows we
can just conditionalize the return variable.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 04:10:33 +0530] rev 30304
py3: use pycompat.ossep at certain places
Certain instances of os.sep has been converted to pycompat.ossep where it was
sure to use bytes only. There are more such instances which needs some more
attention and will get surely.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 03:44:44 +0530] rev 30303
py3: have pycompat.ospathsep and pycompat.ossep
We needed bytes version of os.sep and os.pathsep in py3 as they return
unicodes.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 03:33:22 +0530] rev 30302
py3: add a bytes version of os.name
os.name returns unicodes on py3. Most of our checks are like
os.name == 'nt'
Because of the transformer, on the right hand side we have b'nt'. The
condition will never satisfy even if os.name returns 'nt' as that will be an
unicode.
We either need to encode every occurence of os.name or have a
new variable which is much cleaner. Now we have pycompat.osname.
There are around 53 occurences of os.name in the codebase which needs to
be replaced by pycompat.osname to support Python 3.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 12:18:23 +0900] rev 30301
py3: make util.datapath a bytes variable
In this patch we make util.datapath a bytes variable, but we have to pass a
unicode to gettext.translation otherwise it will cry. Used pycompat.fsdecode()
to decode it back to unicode as it was converted to bytes using
pycompat.fsencode().
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 06 Nov 2016 03:12:40 +0530] rev 30300
py3: add os.fsdecode() as pycompat.fsdecode()
We need to use os.fsdecode() but this was not present in Python 2. So added
the function in pycompat.py
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 04 Nov 2016 20:22:37 -0700] rev 30299
statprof: return state from stop()
I don't like global variables. Have stop() return the captured
state so callers can pass data to the display function.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 05 Nov 2016 09:38:07 -0700] rev 30298
hgweb: cache fctx.parents() in annotate command (
issue5414)
9c37df347485 introduced a call to fctx.parents() for each line in
annotate output. This function call isn't cheap, as it requires
linkrev adjustment.
Since multiple lines in annotate output tend to belong to the same
file revision, a cache of fctx.parents() lookups for each input
should be effective in the common case. So we implement one.
Since the cache has to precompute parents so an aborted generator
doesn't leave an incomplete cache, we could just return a list.
However, we preserve the generator for backwards compatibility.
The effect of this change when requesting /annotate/
96ca0ecdcfa/
browser/locales/en-US/chrome/browser/downloads/downloads.dtd on
the mozilla-aurora repo is significant:
p1(
9c37df347485) 5.5s
9c37df347485: 66.3s
this patch: 10.8s
We're still slower than before. But only by ~2x instead of ~12x.
On the tip revisions of layout/base/nsCSSFrameConstructor.cpp file in
the mozilla-unified repo, time went from 12.5s to 14.5s and back to
12.5s. I'm not sure why the mozilla-aurora repo is so slow.
Looking at the code of basefilectx.parents(), there is room for
further improvements. Notably, we still perform redundant calls to
filelog.renamed() and basefilectx._parentfilectx(). And
basefilectx.annotate() also makes similar calls, so there is potential
for object reuse. However, introducing caches here are not appropriate
for the stable branch.
Yuya Nishihara <yuya@tcha.org> [Sat, 05 Nov 2016 13:20:53 +0900] rev 30297
hghave: check darcs version more strictly
test-convert-darcs.t suddenly started failing on my Debian sid machine. The
reason was Darcs was upgraded from 2.12.0 to 2.12.4 so the original pattern
got to match the last two digits. Fix the pattern to match 2.2+.
Yuya Nishihara <yuya@tcha.org> [Sat, 05 Nov 2016 13:16:40 +0900] rev 30296
tests: silence output of darcs command
It appears darcs is more verbose by default these days. I got test failure
with Darcs 2.12.4.
Durham Goode <durham@fb.com> [Wed, 02 Nov 2016 17:10:47 -0700] rev 30295
manifest: remove manifest.readshallowdelta
This removes manifest.readshallowdelta and converts its one consumer to use
manifestlog instead.
Durham Goode <durham@fb.com> [Wed, 02 Nov 2016 17:10:47 -0700] rev 30294
manifest: get rid of manifest.readshallowfast
This removes manifest.readshallowfast and converts it's one user to use
manifestlog instead.
Durham Goode <durham@fb.com> [Wed, 02 Nov 2016 17:10:47 -0700] rev 30293
manifest: add shallow option to treemanifestctx.readdelta and readfast
The old manifest had different functions for performing shallow reads, shallow
readdeltas, and shallow readfasts. Since a lot of the code is duplicate (and
since those functions don't make sense on a normal manifestctx), let's unify
them into flags on the existing readdelta and readfast functions.
A future diff will change consumers of these functions to use the manifestctx
versions and will delete the old apis.
Durham Goode <durham@fb.com> [Wed, 02 Nov 2016 17:10:47 -0700] rev 30292
manifest: change manifestlog mancache to be directory based
In the last patch we added a get() function that allows fetching directory level
treemanifestctxs. It didn't handle caching at directory level though, so we need to
change our mancache to support multiple directories.
Durham Goode <durham@fb.com> [Wed, 02 Nov 2016 17:24:06 -0700] rev 30291
manifest: add manifestlog.get to obtain subdirectory instances
Previously manifestlog only allowed obtaining root level manifests. Future
patches will need direct access to subdirectory manifests as part of changegroup
creation, so let's add a get() function that knows how to deal with
subdirectories.
Durham Goode <durham@fb.com> [Wed, 02 Nov 2016 17:33:31 -0700] rev 30290
manifest: throw LookupError if node not in revlog
When accessing a manifest via manifestlog[node], let's verify that the node
actually exists and throw a LookupError if it doesn't. This matches the old read
behavior, so we don't accidentally return invalid manifestctxs.
We do this in manifestlog instead of in the manifestctx/treemanifestctx
constructors because the treemanifest code currently relies on the fact that
certain code paths can produce treemanifests without touching the revlogs (and
it has tests that verify things work if certain revlogs are missing entirely, so
they break if we add validation that tries to read them).
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 23 Oct 2016 10:40:33 -0700] rev 30289
revlog: optimize _chunkraw when startrev==endrev
In many cases, _chunkraw() is called with startrev==endrev. When
this is true, we can avoid an extra index lookup and some other
minor operations.
On the mozilla-unified repo, `hg perfrevlogchunks -c` says this
has the following impact:
! read w/ reused fd
! wall 0.371846 comb 0.370000 user 0.350000 sys 0.020000 (best of 27)
! wall 0.337930 comb 0.330000 user 0.300000 sys 0.030000 (best of 30)
! read batch w/ reused fd
! wall 0.014952 comb 0.020000 user 0.000000 sys 0.020000 (best of 197)
! wall 0.014866 comb 0.010000 user 0.000000 sys 0.010000 (best of 196)
So, we've gone from ~25x slower than batch to ~22.5x slower.
At this point, there's probably not much else we can do except
implement an optimized function in the index itself, including in C.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 22 Oct 2016 15:41:23 -0700] rev 30288
revlog: inline start() and end() for perf reasons
When I implemented `hg perfrevlogchunks`, one of the things that
stood out was N * _chunk() calls was ~38x slower than 1
_chunks() call. Specifically, on the mozilla-unified repo:
N*_chunk: 0.528997s
1*_chunks: 0.013735s
This repo has 352,097 changesets. So the average time per changeset
comes out to:
N*_chunk: 1.502us
1*_chunks: 0.039us
If you extrapolate these numbers to a repository with 1M changesets,
that comes out to 1.502s versus 0.039s, which is significant.
At these latencies, Python attribute lookups and function calls
matter. So, this patch inlines some code to cut down on that overhead.
The impact of this patch on N*_chunk() calls is clear:
! wall 0.528997 comb 0.520000 user 0.500000 sys 0.020000 (best of 19)
! wall 0.367723 comb 0.370000 user 0.350000 sys 0.020000 (best of 27)
So, we go from ~38x slower to ~27x. A nice improvement. But there's
still a long way to go.
It's worth noting that functionality like revsets perform changelog
lookups one revision at a time. So this code path is worth optimizing.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 23 Oct 2016 09:34:55 -0700] rev 30287
revlog: reorder index accessors to match data structure order
Index entries are ordered tuples. We have accessors in the revlog
class to map tuple offsets to names. To help reinforce the order,
reorder the methods so they match the order of elements in the
tuple. While I'm here, also sneak in some minimal documentation.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 15:17:02 +0100] rev 30286
color: add the ability to display configured style to 'debugcolor'
The 'hg debugcolor' command gains a '--style' flag to display all the configured
labels and their styles. This have many benefits:
* discovering documented label,
* checking consistency between label's style,
* showing the actual style of a label.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 15:15:47 +0100] rev 30285
color: sort output of 'debugcolor'
The previous ordering were provided by the set. The new output is more stable
and rational. In addition we have some logic to keep the '_background' version
together to help readability.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 14:48:47 +0100] rev 30284
color: extract color and effect display from 'debugcolor'
We are about to introduce a second mode for 'hg debugcolor' that would list the
known label and their configuration, so we split the code related to color and
effect out of the main function.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 14:29:19 +0100] rev 30283
color: restore _style global after debugcolor ran
Before this change, running 'debugcolor' would destroy all color style for the
rest of the process life. We now properly backup and restore the variable
content. Using a global variable is sketchy in general and could probably be
removed. However, this is a quest for another adventure.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 14:12:32 +0100] rev 30282
color: add basic documentation to 'debugcolor'
This does not hurt.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 05:12:23 +0100] rev 30281
tests: merge 'test-push-hook-lock.t' into 'test-push.t'
That test file is very small and is merge with the new 'test-push.t'. No logic
is changed.
We don't register this as a copy because is actually a "ypoc" merging two file
together without replacing the destination and Mercurial cannot express that.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 05:10:14 +0100] rev 30280
tests: merge 'test-push-validation.t' into 'test-push.t'
That test file is very small and is merge with the new 'test-push.t'. No logic
is changed but repository name are update to avoid collision.
We don't register this as a copy because is actually a "ypoc" merging two file
together without replacing the destination and Mercurial cannot express that.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 04:58:46 +0100] rev 30279
test: rename 'test-push-r.t' to 'test-push.t'
We do not have a simple test for 'hg push' but we have multiple tiny tests for
various aspect of it. We'll unify them into a single file, and we start with
'test-push-r.t'. The code is unchanged but we renamed the repository used to
avoid collision with other tests we'll import in coming changesets.
Test timing for the record:
start end cuser csys real Test
1.850 2.640 0.650 0.090 0.790 test-push-validation.t
2.640 3.520 0.760 0.090 0.880 test-push-hook-lock.t
0.000 1.850 1.560 0.210 1.850 test-push-r.t
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 03 Nov 2016 05:05:34 +0100] rev 30278
tests: simplify command script in 'test-push-r.t'
I came across this code by chance. The script of this test is a bit messy with a
lot of unnecessary intermediate commands. We simplify the script and unify
repository access through '-R'.
In the process the update after the unbundle is dropped as it does not add
anything to the tests.
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 03 Nov 2016 03:12:57 +0530] rev 30277
py3: use encoding.environ in ui.py
Using source transformer we add b'' everywhere. So there are no chances
that those bytes string will work with os.environ on Py3 as that returns a
dict of unicodes. We are relying on the errors, even though no error is raised
even in future, these pieces of codes will tend to do wrong things.
if statements can result in wrong boolean and certain errors can be raised
while using this piece of code. Let's not wait for them to happen, fix what is
wrong. If this patch goes in, I will try to do it for all the cases.
Leaving it as it is buggy.
Pulkit Goyal <7895pulkit@gmail.com> [Thu, 03 Nov 2016 02:17:01 +0530] rev 30276
py3: make scmposix.userrcpath() return bytes
We are making sure that we deal with bytes as much we can. This is a part
of fixing functions so that they return bytes if they have to.
Used encoding.environ to return bytes.
After this patch, scmposix.userrcpath() returns bytes and scmutil.osrcpath()
will also return bytes if the platform is posix. Functions is scmposix returns
bytes on Python 3 now.
Jun Wu <quark@fb.com> [Tue, 01 Nov 2016 08:22:50 +0000] rev 30275
adjustlinkrev: remove unnecessary parameters
Since adjustlinkrev has "self", and is a method of a filectx object, it does
not need path, filelog, filenode. They can be fetched from the "self"
easily.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 31 Oct 2016 15:40:30 -0700] rev 30274
repair: make strip() return backup file path
narrowhg wants to strip some commits and then re-apply them after
applying another bundle. Having repair.strip() return the bundle path
will be helpful for it.
timeless <timeless@mozdev.org> [Wed, 02 Nov 2016 18:59:29 +0000] rev 30273
rebase: check for conflicts before continuing
When there are unresolved merge conflicts, there is no reason
to make the user wait for rebase to process all of the
already rebased commits just to complain that it cannot do
anything. Abort early.
timeless <timeless@mozdev.org> [Wed, 02 Nov 2016 18:45:53 +0000] rev 30272
cmdutil: refactor checkunresolved
localrepo.commit had code to check for unresolved merge conflicts,
it would be helpful for at least rebase to be able to use that
code without calling commit().
timeless <timeless@mozdev.org> [Wed, 02 Nov 2016 18:56:07 +0000] rev 30271
rebase: rename merge to mergemod
Mads Kiilerich <mads@kiilerich.com> [Thu, 19 Mar 2015 22:22:50 +0100] rev 30270
context: make sure __str__ works, also when there is no _changectx
Before, it could crash when trying to print the wrong kind of object at the
wrong time.
Mads Kiilerich <madski@unity3d.com> [Tue, 18 Oct 2016 16:45:39 +0200] rev 30269
largefiles: clarify variable name holding file mode
A follow-up to
c01acee367ec.
'st' sounds like the whole stat result while 'mode' is a better name for the
actual file mode.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 01 Nov 2016 18:29:09 -0700] rev 30268
changegroup: use changelogrevision()
Using offsets for accessing changelog entries isn't very readable.
As a bonus, changelog.changelogrevision() also accepts a revision,
so we don't need to perform the inline node resolution either.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 01 Nov 2016 18:28:03 -0700] rev 30267
changegroup: cache changelog and manifestlog outside of loop
History has taught us that repo.changelog can add significant overhead
to loops. So cache the changelog instance outside of the loop to
avoid the lookup. While we're here, do the same for manifestlog,
since each loop would otherwise initialize a new manifestlog instance.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 01 Nov 2016 18:49:23 -0700] rev 30266
make: targets for building packages for ubuntu yakkety
Ubuntu 16.10 Yakkety Yak is out. Let's support it.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 15 Oct 2016 17:24:01 -0700] rev 30265
util: put compression code next to each other
ctxmanager was injecting itself between the compression and
decompression code. Let's restore some order.
Yuya Nishihara <yuya@tcha.org> [Sat, 28 Jun 2014 13:13:32 +0900] rev 30264
hgweb: make log streams compatible with command server
Even though it would be useless to start a web server by a command server,
it should be doable in principle. Also, we can't use sys.stdout/err directly
on Python 3 because they are unicode streams.
Yuya Nishihara <yuya@tcha.org> [Mon, 29 Feb 2016 13:41:54 +0900] rev 30263
cmdserver: write channel header and payload by a single write() call
This makes a channeledoutput thread-safe as long as the underlying fwrite() is
thread-safe. Both POSIX and Windows implementations are documented as MT-safe.
MT-safety is necessary to use ui.fout and ui.ferr in hgweb.
Yuya Nishihara <yuya@tcha.org> [Fri, 18 Mar 2016 17:15:06 -0700] rev 30262
histedit: use ui.fin to read commands from stdin
stdin is wrapped by channeledinput in command-server session.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Oct 2015 14:34:56 +0900] rev 30261
convert: have debugsvnlog obtain standard streams from ui
This will help porting to Python 3, where sys.stdin/out/err are unfortunately
unicode streams so we can't use them directly.
Yuya Nishihara <yuya@tcha.org> [Sat, 03 Oct 2015 14:29:13 +0900] rev 30260
convert: remove superfluous setbinary() calls from debugsvnlog
a3fe91b4f6eb made standard streams set to binary mode globally.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 17 Aug 2016 08:52:15 -0700] rev 30259
tests: explicitly use ls profiler
In preparation for making the statprof profiler the default.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 01 Nov 2016 19:03:11 -0700] rev 30258
statprof: pass data structure to display functions
Currently, statprof maintains a global "state" variable that is used by
several functions. Global variables hinder adaptability of code.
So pass state to display functions so we can make changes to how
"state" works in future patches.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 14 Aug 2016 19:20:12 -0700] rev 30257
statprof: use print function
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 01 Nov 2016 18:55:30 -0700] rev 30256
statprof: use absolute_imports
As part of this, we modify import order to satisfy our import
checker.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 14 Aug 2016 19:14:05 -0700] rev 30255
statprof: require paths to save or load profile data
Upstream appears to aggressively save statprof data in a well-defined
home directory path. Change the code to not do that.
We also change file saving to fail if an error has occurred
instead of silently failing. Callers can catch the exception.
This behavior is more suitable for a generic "library" module.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 14 Aug 2016 19:13:32 -0700] rev 30254
statprof: fix flake8 warnings
My local flake8 hook informed me of these warnings in the upstream
code. Fix them.
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 01 Nov 2016 18:54:03 -0700] rev 30253
statprof: vendor statprof.py
Vendored from https://bitbucket.org/facebook/hg-experimental
changeset
73f9db47ae5a1a9fa29a98dfe92d557ad51234c3 without
modification.
This introduces a number of code style violations. The file
already has the magic words to skip test-check-code.t. I'll
make additional changes to clean up the test-check-py3-compat.t
warnings and to change some behavior in the code that isn't
suitable for general use.
test-check-commit.t also complains about numerous things. But
there's nothing we can do if we're importing as-is.
Augie Fackler <augie@google.com> [Tue, 01 Nov 2016 15:40:21 -0400] rev 30252
merge with stable
Augie Fackler <raf@durin42.com> [Tue, 01 Nov 2016 14:12:39 -0400] rev 30251
Added signature for changeset
eab274469952
Augie Fackler <raf@durin42.com> [Tue, 01 Nov 2016 14:12:37 -0400] rev 30250
Added tag 4.0 for changeset
eab274469952
Kevin Bullock <kbullock+mercurial@ringworld.org> [Tue, 01 Nov 2016 13:03:42 -0500] rev 30249
merge with i18n
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 01 Nov 2016 04:27:41 +0900] rev 30248
i18n-ja: synchronized with
69ffbbe73dd0
Nathan Goldbaum <ngoldbau@illinois.edu> [Mon, 31 Oct 2016 17:12:32 -0500] rev 30247
tag: clarify warning about making a tag on a branch head
Currently the warning is ambiguous about whether the new tag (possibly specified
via --rev) is being added on a branch head or whether the working directory is
based on a branch head. Clarify the error message to eliminate this ambiguity.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 01 Nov 2016 20:39:37 +0900] rev 30246
contrib: check reference to old selenic.com domain
Now, all URL in Mercurial source tree should refer mercurial-scm.org
domain instead of selenic.com.
*.po files are ignored in this patch, because they might contain
msgid/msgstr coming from old source files.
This ignorance seems safe enough, because such msgstr should be
ignored at runtime, because:
- msgid corresponded to it should be invalid, or
- msgstr itself should be marked as fuzzy at synchronized to recent hg.pot
If any additional examination for *.po files is needed in the future,
let i18n/check-translation.py achieve such examination.
BTW, some binary files (e.g. *.png) are meaningless for checking
reference to old domain in this patch, but aren't ignored like as *.po
files, because excluding multiple suffixes is difficult for regexp
matching.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 01 Nov 2016 20:39:36 +0900] rev 30245
check-code: discard filtering result of previous check for independence
Before this patch, check-code.py applies filtering on the file
content, to which filtering of previous check is already applied.
This might hide issues, which should be detected by a subsequent check
in "checks" list.
Fortunately, this problem hasn't appeared, because there is no
overlapping of filename matching (examined in the order below).
1. *.py or *.cgi
2. test-* (not *.t suffix)
3. *.c or *.h
4. *.t
5. *.txt
6. *.tmpl
For example, adding a test, which wants to examine raw comment text in
*.py files, at the end of current "checks" list doesn't work as
expected, because a filter for *.py files normalizes comment text in
them.
Putting such test at the beginning of "checks" list also resolves this
problem, but such dependence on the order decreases maintainability of
check-code.py itself.
This patch discards filtering result of previous check at the
beginning of each checks, for independence of each checks.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 01 Nov 2016 20:39:36 +0900] rev 30244
help: replace selenic.com by mercurial-scm.org in man pages
Source code repository and mailing list services have been already
migrated to mercurial-scm.org domain.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Tue, 01 Nov 2016 20:39:35 +0900] rev 30243
help: replace selenic.com by mercurial-scm.org in command examples
Source code repository service of Mercurial itself has been already
migrated to mercurial-scm.org domain.
Yuya Nishihara <yuya@tcha.org> [Tue, 01 Nov 2016 21:14:33 +0900] rev 30242
hghave: fix 'rmcwd' to ensure temporary directory is removed
On platforms where cwd can't be removed, it should try rmdir() after chdir
to the original cwd.
Mads Kiilerich <madski@unity3d.com> [Mon, 31 Oct 2016 13:43:48 +0100] rev 30241
httppeer: make __del__ access to self.urlopener more safe
Some errors could in some cases show unfortunate scary and confusing warnings
from the httppeer delstructors:
abort: nodename nor servname provided, or not known
Exception AttributeError: "'httpspeer' object has no attribute 'urlopener'" in <bound method httpspeer.__del__ of <mercurial.httppeer.httpspeer object at 0x
106e1f5d0>> ignored```
To mute that, take
7b15dd9125b3 to the next level and use getattr in __del__.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 30 Oct 2016 06:15:09 +0900] rev 30240
tests: test preserving execbit changes at amending only on execbit platform
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 30 Oct 2016 06:15:09 +0900] rev 30239
tests: put temporary file outside the working directory for test portability
test-largefiles-update.t creates temporary file exec-bit.patch inside
the working directory for no-execbit platform specific test, but
subsequent tests aren't aware of it.
On execbit platform, subsequent tests can run successfully, because
exec-bit.patch isn't created.
But on no-execbit platform, this temporary file makes subsequent tests
show "? exec-bit.patch" at each "hg status".
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 30 Oct 2016 06:15:09 +0900] rev 30238
tests: avoid quoting of commit messages for test portability
journal extension uses util.shellquote() to record command line, but
result of it depends on runtime platform: double quotation is used on
Windows and OpenVMS, but single quotation is used otherwise.
test-journal-share.t sometimes specifies commit messages including
white space on command line. It makes journal output depend on runtime
platform, but commit message itself isn't important in this test case.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 30 Oct 2016 06:15:07 +0900] rev 30237
tests: use basic format code "%Y" instead of "%s" for test portability
On Windows, strftime() doesn't support format code "%s", and it causes
"invalid format string" error.
https://msdn.microsoft.com/en-us/library/fe06s4ak.aspx
test-command-template.t examines not seconds value in UTC, but
arithmetic calculation. Therefore, using format code "%Y" instead of
"%s" should be reasonable.
FYI:
- Python standard library reference doesn't list "%s" up in format
code list required for "C standard (1989 version)", even though it
also mentions that additional format codes are required for "C
standard (1999 version)"
https://docs.python.org/2.7/library/datetime.html#strftime-and-strptime-behavior
- The Open Group Base Specifications Issue 7 (IEEE Std 1003.1-2008,
2016 Edition) doesn't require strftime to support format code "%s"
http://pubs.opengroup.org/onlinepubs/
9699919799/functions/strftime.html
- "man strftime" of (Open/Oracle) Solaris and Mac OS X (= UNIX
certified OSs) describes about format code "%s"
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 29 Oct 2016 03:08:08 +0900] rev 30236
tests: add test-commit-interactive-curses.t "require tic" for test portability
Standard library of Python on Windows doesn't have curses module.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 29 Oct 2016 03:04:54 +0900] rev 30235
tests: use "?" to glob both ":" and ";" in output for test portability
If environment variable looks like PATH or so (e.g. any of components
joined by ":" contains "/"), ":" in it is replaced with ";" by MinGW
at spawning Windows native process, to follow path concatenation style
of Windows.
Therefore, "bundle:../full.hg" is converted into "bundle;..\full.hg"
on MinGW.
Difference between "/" and "\" is automatically ignored by "(glob)",
but difference between ":" and ";" should be globed explicitly.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sat, 29 Oct 2016 02:44:45 +0900] rev 30234
tests: invoke printenv.py via sh -c for test portability
On Windows platform, invoking printenv.py directly via hook is
problematic, because:
- unless binding between *.py suffix and python runtime, application
selector dialog is displayed, and running test is blocked at each
printenv.py invocations
- it isn't safe to assume binding between *.py suffix and python
runtime, because application binding is easily broken
For example, installing IDE (VisualStudio with Python Tools, or
so) often requires binding between source files and IDE itself.
This patch invokes printenv.py via sh -c for test portability. This is
a kind of follow up for
d19787db6fe0, which eliminated explicit
"python" for printenv.py. There are already other 'sh -c "printenv.py"'
in *.t files, and this fix should be reasonable.
This changes were confirmed in cases below:
- without any application binding for *.py suffix
- with binding between *.py suffix and VisualStudio
This patch also replaces "echo + redirection" style with "heredoc"
style, because:
- hook command line is parsed by cmd.exe as shell at first, and
- single quotation can't quote arguments on cmd.exe, therefore,
- "printenv.py foobar" should be quoted by double quotation, but
- nested quoting (or tricky escaping) isn't readable
Mads Kiilerich <madski@unity3d.com> [Thu, 27 Oct 2016 20:06:33 +0200] rev 30233
largefiles: handle that a found standin file doesn't exist when removing it
I somehow ended up in a situation where hg crashed on an unlink I introduced in
328545c7d8a1.
I don't know how it happened and can't reproduce it. It seems like it only can
happen when the file is removed between the time of check in a working
directory context walk that finds a standin file, and the time of use when we
try to remove it because the corresponding largefile doesn't exist.
But better safe than sorry: replace the plain unlink with unlinkpath with
ignoremissing=True. That will also remove remaining empty directories, which
arguably is more correct.
Yuya Nishihara <yuya@tcha.org> [Tue, 25 Oct 2016 21:49:30 +0900] rev 30232
templater: use unfiltered changelog to calculate shortest() at constant time
cl._partialmatch() can be pretty slow if hidden revisions are involved. This
patch cancels the slowdown introduced by the previous patch by using an
unfiltered changelog, which means shortest(node) isn't always the shortest.
The result isn't perfect, but seems okay as long as shortest(node) is short
enough to type and can be used as an identifier.
(with hidden revisions)
% hg log -R hg-committed -r0:20000 -T '{node|shortest}\n' --time > /dev/null
(.^^) time: real 1.530 secs (user 1.480+0.000 sys 0.040+0.000)
(.^) time: real 43.080 secs (user 43.060+0.000 sys 0.030+0.000)
(.) time: real 1.680 secs (user 1.650+0.000 sys 0.020+0.000)
Yuya Nishihara <yuya@tcha.org> [Sun, 23 Oct 2016 14:05:23 +0900] rev 30231
templater: do not use index.partialmatch() directly to calculate shortest()
cl.index.partialmatch() isn't a drop-in replacement for cl._partialmatch().
It has no knowledge about hidden revisions, and it raises ValueError if a node
shorter than 4 chars is given. Instead, use index.partialmatch() through
cl._partialmatch(), which has no such problems and gives the identical result
with/without --pure.
The test output was sampled with --pure without this patch, which shows the
most correct result. However, we'll need to switch to using an unfiltered
changelog because _partialmatch() of a filtered changelog can be an order of
magnitude slower.
(with hidden revisions)
% hg log -R hg-committed -r0:20000 -T '{node|shortest}\n' --time > /dev/null
(.^) time: real 1.530 secs (user 1.480+0.000 sys 0.040+0.000)
(.) time: real 43.080 secs (user 43.060+0.000 sys 0.030+0.000)
Yuya Nishihara <yuya@tcha.org> [Wed, 26 Oct 2016 22:50:06 +0900] rev 30230
tests: run "cwd was removed" test only if cwd can actually be removed
On some platforms, cwd can't be removed. In which case, util.unlinkpath()
continues with no error since the failure of directory removal isn't critical.
So it doesn't make sense to run the test added by
90a6c18a7c1d on those
platforms. OTOH, we need to run the test in test-rebase-scenario-global.t
since the repository is referenced after that.
Gábor Stefanik <gabor.stefanik@nng.com> [Tue, 25 Oct 2016 21:01:53 +0200] rev 30229
merge: avoid superfluous filemerges when grafting through renames (
issue5407)
This is a fix for a regression introduced by the patches for
issue4028.
The test changes are due to us doing fewer _checkcopies searches now, which
makes some test outputs revert to the pre-
issue4028 behavior. That issue itself
remains fixed, we only skip copy tracing for files where it isn't relevant.
As a nice side effect, this makes copy detection much faster when tracing
backwards through lots of renames.
Gábor Stefanik <gabor.stefanik@nng.com> [Wed, 19 Oct 2016 18:06:14 +0200] rev 30228
sslutil: guard against broken certifi installations (
issue5406)
Certifi is currently incompatible with py2exe; the Python code for certifi gets
included in library.zip, but not the cacert.pem file - and even if it were
included, SSLContext can't load a cacert.pem file from library.zip.
This currently makes it impossible to build a standalone Windows version of
Mercurial.
Guard against this, and possibly other situations where a module with the name
"certifi" exists, but is not usable.
Mads Kiilerich <madski@unity3d.com> [Tue, 25 Oct 2016 18:56:27 +0200] rev 30227
revset: don't cache abstractsmartset min/max invocations infinitely
There was a "leak", apparently introduced in
ab66c1dee405. When running:
hg = hglib.open('repo')
while True:
hg.log("max(branch('default'))")
all filteredset instances from branch() would be cached indefinitely by the
@util.cachefunc annotation on the max() implementation.
util.cachefunc seems dangerous as method decorator and is barely used elsewhere
in the code base. Instead, just open code caching by having the min/max
methods replace themselves with a plain lambda returning the result.
Kevin Bullock <kbullock+mercurial@ringworld.org> [Mon, 24 Oct 2016 09:14:34 -0500] rev 30226
merge with i18n
Wagner Bruna <wbruna@yahoo.com> [Sat, 22 Oct 2016 23:18:43 -0200] rev 30225
i18n-pt_BR: synchronized with
7b428b00a1d4
Mads Kiilerich <madski@unity3d.com> [Tue, 18 Oct 2016 16:52:35 +0200] rev 30224
dirstate: fix debug.dirstate.delaywrite to use the new "now" after sleeping
It seems like the a regression has sneaked into debug.dirstate.delaywrite in
6c6b48aca328. It would sleep until no files were modified "now" any more, but
when writing the dirstate it would use the old "now" and still mark files as
'unset' instead of recording the timestamp that would make the file show up as
clean instead of unknown.
Instead of getting a new "now" from the file system, we trust the computed end
time as the new "now" and thus cause the actual modification time to be
writiten to the dirstate.
debug.dirstate.delaywrite is undocumented and only used in
test-largefiles-update.t . All tests seems to work fine for me without
debug.dirstate.delaywrite . Perhaps because it not really worked as intended
without the fix in this patch, and code and tests thus have evolved to do fine
without it? It could thus perhaps make sense to drop usage of this setting in
the tests. That could speed the test up a bit.
This functionality (or something very similar) can however apparently be very
convenient in setups where checking dirty-ness is expensive - such as when
using large files and have slow file filesystems or are CPU constrained. Now it
works and we can try it. (But ideally, for the largefile use case, it should
probably only delay lfdirstate writes - not ordinary dirstate.)
Simon Farnsworth <simonfar@fb.com> [Fri, 21 Oct 2016 16:31:16 +0100] rev 30223
tests: fix test-casefolding.t
The message had changed, but the test was not updated. This test does not
run on Linux, but failed on my Mac.
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 19 Oct 2016 15:07:11 -0700] rev 30222
commands: print security protocol support in debuginstall
Over the past week I've had to instruct multiple people to run
Python code to query the ssl module to see what TLS protocol support
is present. I think it would be useful for `hg debuginstall` to print
this info to make it easier to access and debug why Mercurial is
complaining about using an insecure TLS 1.0 protocol.
Ideally we'd also print the path to the CA cert bundle. But the APIs
for querying that in sslutil can emit warnings, making it slightly
more difficult to integrate into `hg debuginstall`. That work will
have to wait for another day.
Durham Goode <durham@fb.com> [Tue, 18 Oct 2016 17:44:42 -0700] rev 30221
manifest: make treemanifestctx store the repo
Same as in the last commit, the old treemanifestctx stored a reference to the
revlog. If the inmemory revlog became invalid, the ctx now held an old copy and
would be incorrect. To fix this, we need the ctx to go through the manifestlog
for each access.
This is the same pattern that changectx already uses (it stores the repo, and
accesses commit data through self._repo.changelog).
Durham Goode <durham@fb.com> [Tue, 18 Oct 2016 17:44:26 -0700] rev 30220
manifest: make manifestctx store the repo
The old manifestctx stored a reference to the revlog. If the inmemory revlog
became invalid, the ctx now held an old copy and would be incorrect. To fix
this, we need the ctx to go through the manifestlog for each access.
This is the same pattern that changectx already uses (it stores the repo, and
accesses commit data through self._repo.changelog).
Durham Goode <durham@fb.com> [Tue, 18 Oct 2016 17:33:39 -0700] rev 30219
manifest: make manifestlog a storecache
The old @property on manifestlog was broken. It meant that we would always
recreate the manifestlog instance, which meant the cache was never hit. Since
we'll eventually remove repo.manifest and make manifestlog the only property,
let's go ahead and make manifestlog the @storecache property, have manifestlog
own the manifest instance, and have repo.manifest refer to it via manifestlog.
This means all accesses go through repo.manifestlog, which is now invalidated
correctly.
Durham Goode <durham@fb.com> [Tue, 18 Oct 2016 17:32:51 -0700] rev 30218
manifest: move manifest creation to a helper function
A future patch will be moving manifest creation to be inside manifestlog as part
of improving our cache guarantees. bundlerepo and unionrepo currently rely on
being able to hook into manifest creation, so let's temporarily move the actual
manifest creation to a helper function for them to intercept.
In the future manifest.manifest() will disappear entirely and this can
disappear.
Kevin Bullock <kbullock@ringworld.org> [Tue, 18 Oct 2016 14:27:30 -0500] rev 30217
Added signature for changeset
438173c41587
Kevin Bullock <kbullock@ringworld.org> [Tue, 18 Oct 2016 14:27:25 -0500] rev 30216
Added tag 4.0-rc for changeset
438173c41587
Kevin Bullock <kbullock+mercurial@ringworld.org> [Tue, 18 Oct 2016 14:15:15 -0500] rev 30215
merge default into stable for 4.0 code freeze
Kevin Bullock <kbullock+mercurial@ringworld.org> [Tue, 18 Oct 2016 14:13:06 -0500] rev 30214
merge with i18n
Wagner Bruna <wbruna@softwareexpress.com.br> [Tue, 11 Oct 2016 20:39:47 -0300] rev 30213
i18n-pt_BR: synchronized with
149433e68974
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 16 Oct 2016 13:35:23 -0700] rev 30212
changegroup: increase write buffer size to 128k
By default, Python defers to the operating system for choosing the
default buffer size on opened files. On my Linux machine, the default
is 4k, which is really small for 2016.
This patch bumps the write buffer size when writing
changegroups/bundles to 128k. This matches the 128k read buffer
we already use on revlogs.
It's worth noting that this only impacts when writing to an explicit
file (such as during `hg bundle`). Buffers when writing to bundle
files via the repo vfs or to a temporary file are not impacted.
When producing a none-v2 bundle file of the mozilla-unified repository,
this change caused the number of write() system calls to drop from
952,449 to 29,788. After this change, the most frequent system
calls are fstat(), read(), lseek(), and open(). There were
2,523,672 system calls after this patch (so a net decrease of
~950k is statistically significant).
This change shows no performance change on my system. But I have a
high-end system with a fast SSD. It is quite possible this change
will have a significant impact on network file systems, where
extra network round trips due to excessive I/O system calls could
introduce significant latency.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Fri, 14 Oct 2016 01:31:11 +0200] rev 30211
changegroup: skip delta when the underlying revlog do not use them
Revlog can now be configured to store full snapshot only. This is used on the
changelog. However, the changegroup packing was still recomputing deltas to be
sent over the wire.
We now just reuse the full snapshot directly in this case, skipping delta
computation. This provides use with a large speed up(-30%):
# perfchangegroupchangelog on mercurial
! wall 2.010326 comb 2.020000 user 2.000000 sys 0.020000 (best of 5)
! wall 1.382039 comb 1.380000 user 1.370000 sys 0.010000 (best of 8)
# perfchangegroupchangelog on pypy
! wall 5.792589 comb 5.780000 user 5.780000 sys 0.000000 (best of 3)
! wall 3.911158 comb 3.920000 user 3.900000 sys 0.020000 (best of 3)
# perfchangegroupchangelog on mozilla central
! wall 20.683727 comb 20.680000 user 20.630000 sys 0.050000 (best of 3)
! wall 14.190204 comb 14.190000 user 14.150000 sys 0.040000 (best of 3)
Many tests have to be updated because of the change in bundle content. All
theses update have been verified. Because diffing changelog was not very
valuable, the resulting bundle have similar size (often a bit smaller):
# full bundle of mozilla central
with delta:
1142740533B
without delta:
1142173300B
So this is a win all over the board.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Fri, 14 Oct 2016 02:25:08 +0200] rev 30210
revlog: make 'storedeltachains' a "public" attribute
The next changeset will make that attribute read by the changegroup packer. We
make it "public" beforehand.
Martin von Zweigbergk <martinvonz@google.com> [Mon, 17 Oct 2016 22:51:22 -0700] rev 30209
manifest: don't store None in fulltextcache
When we read a value from fulltextcache, we expect it to be an array,
so we should not store None in it. Found while working on narrowhg.
Gábor Stefanik <gabor.stefanik@nng.com> [Tue, 18 Oct 2016 02:09:08 +0200] rev 30208
copies: improve assertions during copy recombination
- Make sure there is nothing to recombine in non-graftlike scenarios
- More pythonic assert syntax
Martin von Zweigbergk <martinvonz@google.com> [Mon, 17 Oct 2016 16:12:12 -0700] rev 30207
treemanifest: fix bad argument order to treemanifestctx
Found by running tests with _treeinmem (both of them) modified to be
True.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 16 Oct 2016 11:10:21 -0700] rev 30206
wireproto: compress data from a generator
Currently, the "getbundle" wire protocol command obtains a generator of
data, converts it to a util.chunkbuffer, then converts it back to a
generator via the protocol's groupchunks() implementation. For the SSH
protocol, groupchunks() simply reads 4kb chunks then write()s the
data to a file descriptor. For the HTTP protocol, groupchunks() reads
32kb chunks, feeds those into a zlib compressor, emits compressed data
as it is available, and that is sent to the WSGI layer, where it is
likely turned into HTTP chunked transfer chunks as is or further
buffered and turned into a larger chunk.
For both the SSH and HTTP protocols, there is inefficiency from using
util.chunkbuffer.
For SSH, emitting consistent 4kb chunks sounds nice. However, the file
descriptor it is writing to is almost certainly buffered. That means
that a Python .write() probably doesn't translate into exactly what is
written to the I/O layer.
For HTTP, we're going through an intermediate layer to zlib compress
data. So all util.chunkbuffer is doing is ensuring that the chunks we
feed into the zlib compressor are of uniform size. This means more CPU
time in Python buffering and emitting chunks in util.chunkbuffer but
fewer function calls to zlib.
This patch introduces and implements a new wire protocol abstract
method: compresschunks(). It is like groupchunks() except it operates
on a generator instead of something with a .read(). The SSH
implementation simply proxies chunks. The HTTP implementation uses
zlib compression.
To avoid duplicate code, the HTTP groupchunks() has been reimplemented
in terms of compresschunks().
To prove this all works, the "getbundle" wire protocol command has been
switched to compresschunks(). This removes the util.chunkbuffer from
that command. Now, data essentially streams straight from the
changegroup emitter to the wire, possibly through a zlib compressor.
Generators all the way, baby.
There were slim to no performance changes on the server as measured
with the mozilla-central repository. This is likely because CPU
time is dominated by reading revlogs, producing the changegroup, and
zlib compressing the output stream. Still, this brings us a little
closer to our ideal of using generators everywhere.
Mads Kiilerich <madski@unity3d.com> [Mon, 17 Oct 2016 19:48:36 +0200] rev 30205
revset: optimize for destination() being "inefficient"
destination() will scan through the whole subset and read extras for each
revision to get its source.
Gábor Stefanik <gabor.stefanik@nng.com> [Tue, 11 Oct 2016 04:39:47 +0200] rev 30204
copies: make _checkcopies handle copy sequences spanning the TCA (
issue4028)
When working in a rotated DAG (for a graftlike merge), there can be files
that are renamed both between the base and the topological CA, and between
the TCA and the endpoint farther from the base. Such renames span the TCA
(and thus need both passes of _checkcopies to be fully detected), but may
not necessarily be divergent.
Make _checkcopies return "incomplete copies" and "incomplete divergences"
in this case, and let mergecopies recombine them once data from both passes
of _checkcopies is available.
With this patch, all known cases involving renames and grafts pass.
(Developed together with Pierre-Yves David)
Gábor Stefanik <gabor.stefanik@nng.com> [Tue, 11 Oct 2016 04:25:59 +0200] rev 30203
checkcopies: add logic to handle remotebase
As the two _checkcopies passes' ranges are separated by tca, not base,
only one of the two passes will actually encounter the base.
Pass "remotebase" to the other pass to let it know not to expect passing
over the base. This is required for handling a few unusual rename cases.
Gábor Stefanik <gabor.stefanik@nng.com> [Tue, 04 Oct 2016 12:51:54 +0200] rev 30202
mergecopies: add logic to process incomplete data
We first combine incomplete copies on the two sides of the topological CA
into complete copies.
Any leftover incomplete copies are then combined with the incomplete
divergences to reconstruct divergences spanning over the topological CA.
Finally we promote any divergences falsely flagged as incomplete to full
divergences.
Right now, there is nothing generating incomplete copy/divergence data,
so this code does nothing. Changes to _checkcopies to populate these
dicts are coming later in this series.
Gábor Stefanik <gabor.stefanik@nng.com> [Wed, 12 Oct 2016 11:54:03 +0200] rev 30201
checkcopies: handle divergences contained entirely in tca::ctx
During a graftlike merge, _checkcopies runs from ctx to tca, possibly
passing over the merge base. If there is a rename both before and after
the base, then we're actually dealing with divergent renames.
If there is no rename on the other side of tca, then the divergence is
contained entirely in the range of one _checkcopies invocation, and
should be detected "in the loop" without having to rely on the other
_checkcopies pass.
Gábor Stefanik <gabor.stefanik@nng.com> [Thu, 25 Aug 2016 22:02:26 +0200] rev 30200
update: enable copy tracing for backwards and non-linear updates
As a followup to the
issue4028 series, this fixes a variant of the issue
that can occur when updating with uncommited local changes.
The duplicated .hgsub warning is coming from wc.dirty(). We would previously
skip this call because it's only relevant when we're going to perform copy
tracing, which we didn't do before.
The change to the update summary line is because we now treat the rename as a
proper rename (which counts as a change), rather than an add+delete pair
(which counts as a change and a delete).
Mathias De Maré <mathias.de_mare@nokia.com> [Mon, 26 Sep 2016 10:47:37 +0200] rev 30199
bashcompletion: allow skipping completion for 'hg status'
On systems with large repositories and slow disks,
the calls to 'hg status' make autocomplete annoyingly slow.
This fix makes it possible to avoid the slowdown.
Mads Kiilerich <madski@unity3d.com> [Sun, 21 Aug 2016 01:12:00 +0200] rev 30198
tests: add more test coverage of phase changes when pushing
Prepare for test coverage of phase updates with future push --readonly option,
both with and without actually pushing changesets.
Gábor Stefanik <gabor.stefanik@nng.com> [Thu, 13 Oct 2016 02:19:43 +0200] rev 30197
mergecopies: invoke _computenonoverlap for both base and tca during merges
The algorithm of _checkcopies can only walk backwards in the DAG, never
forward. Because of this, the two _checkcopies patches need to run from
their respective endpoints to the TCA to cover the entire subgraph where
the merge is being performed. However, detection of files new in both
endpoints, as well as directory rename detection, need to run with respect
to the merge base, so we need lists of new files both from the TCA's and
the merge base's viewpoint to correctly detect renames in a graft-like
merge scenario.
(Series reworked by Pierre-Yves David)
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Tue, 18 Oct 2016 00:00:43 +0200] rev 30196
copies: make it possible to distinguish betwen _computenonoverlap invocations
_computenonoverlap needs to be invoked twice during a graft, and debugging
messages should be distinguishable between the two invocations
Gábor Stefanik <gabor.stefanik@nng.com> [Thu, 13 Oct 2016 02:03:54 +0200] rev 30195
copies: make _checkcopies handle simple renames in a rotated DAG
This introduces a distinction between "merge base" and
"topological common ancestor". During a regular merge, these two are
identical. Graft, however, performs a merge in a rotated DAG, where the
merge base will not be a common ancestor at all in the
original DAG.
To correctly find copies in case of a graft, we need to take both the
merge base and the topological CA into account, and track any renames
between them in reverse. Fortunately we can detect this in advance,
see comment in the code about "backwards".
This patch only supports finding non-divergent renames contained entirely
between the merge base and the topological CA. Further patches are coming
to support more complex cases.
(Pierre-Yves David was involved in the cleanup of this patch.)
Gábor Stefanik <gabor.stefanik@nng.com> [Thu, 13 Oct 2016 02:03:49 +0200] rev 30194
copies: compute a suitable TCA if base turns out to be unsuitable
This will be used later in an update to _checkcopies.
(Pierre-Yves David was involved in the cleanup of this patch.)
Gábor Stefanik <gabor.stefanik@nng.com> [Thu, 13 Oct 2016 01:47:33 +0200] rev 30193
copies: detect graft-like merges
Right now, nothing changes as a result of this, but we want to handle
grafts differently from ordinary merges later.
(Series developed together with Pierre-Yves David)
Gábor Stefanik <gabor.stefanik@nng.com> [Wed, 12 Oct 2016 12:41:28 +0200] rev 30192
tests: introduce tests for grafting through renames
These cover all currently known cases of renames being grafted,
or changes being grafted through renames.
Right now, most of these cases are broken. Later patches in this series
will make them behave correctly.
The testcases heavily rely on each other, which would make it very difficult
to separate them and add them one-by-one for each case fixed by a patch.
Separating them should perhaps be a 4.1 task, if it doesn't slow down
the tests too much.
(Developed together with Pierre-Yves David)
Mads Kiilerich <madski@unity3d.com> [Mon, 17 Oct 2016 17:12:24 +0200] rev 30191
largefiles: fix 'deleted' files sometimes persistently appearing with R status
A code snippet that has been around since largefiles was introduced was wrong:
Standins no longer found in lfdirstate has *not* been removed -
they have probably just been deleted ... or not created.
This wrong reporting did that 'up -C' didn't undo the change and didn't sync
the two dirstates.
Instead of reporting such files as removed, propagate the deletion to the
standin file and report the file as deleted.
Mads Kiilerich <madski@unity3d.com> [Sun, 16 Oct 2016 02:29:45 +0200] rev 30190
largefiles: more safe handling of interruptions while updating modifications
Largefiles are fragile with the design where dirstate and lfdirstate must be
kept in sync.
To be less fragile, mark all clean largefiles as unsure ("normallookup") before
updating standins. After standins have been updated and we know exactly which
largefile standins actually was changed, mark the unchanged largefiles back to
clean ("normal").
This will make the failure mode more safe. If interrupted, the next command
will continue to perform extra hashing of all largefiles. That will do that all
largefiles that are out of sync with their standin will be marked dirty and
they will show up in status and can be cleaned with update --clean.
Mads Kiilerich <madski@unity3d.com> [Sun, 16 Oct 2016 02:26:38 +0200] rev 30189
largefiles: test coverage of fatal interruption of update
Test using existing changesets in a clean working directory, revealing problems
with files that don't show up as modified or do show up as removed when they
just not have been written yet.
Gábor Stefanik <gabor.stefanik@nng.com> [Wed, 12 Oct 2016 21:33:45 +0200] rev 30188
checkcopies: add a sanity check against false-positive copies
When grafting a copy backwards through a rename, a copy is wrongly detected,
which causes the graft to be applied inappropriately, in a destructive way.
Make sure that the old file name really exists in the common ancestor,
and bail out if it doesn't.
This fixes the aggravated case of bug 5343, although the basic issue
(failure to duplicate the copy information) still occurs.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 16 Oct 2016 10:38:52 -0700] rev 30187
exchange: refactor APIs to obtain bundle data (API)
Currently, exchange.getbundle() returns either a cg1unpacker or a
util.chunkbuffer (in the case of bundle2). This is kinda OK, as
both expose a .read() to consumers. However, localpeer.getbundle()
has code inferring what the response type is based on arguments and
converts the util.chunkbuffer returned in the bundle2 case to a
bundle2.unbundle20 instance. This is a sign that the API for
exchange.getbundle() is not ideal because it doesn't consistently
return an "unbundler" instance.
In addition, unbundlers mask the fact that there is an underlying
generator of changegroup data. In both cg1 and bundle2, this generator
is being fed into a util.chunkbuffer so it can be re-exposed as a
file object.
util.chunkbuffer is a nice abstraction. However, it should only be
used "at the edges." This is because keeping data as a generator is
more efficient than converting it to a chunkbuffer, especially if we
convert that chunkbuffer back to a generator (as is the case in some
code paths currently).
This patch refactors exchange.getbundle() into
exchange.getbundlechunks(). The new API returns an iterator of chunks
instead of a file-like object.
Callers of exchange.getbundle() have been updated to use the new API.
There is a minor change of behavior in test-getbundle.t. This is
because `hg debuggetbundle` isn't defining bundlecaps. As a result,
a cg1 data stream and unpacker is being produced. This is getting fed
into a new bundle20 instance via bundle2.writebundle(), which uses
a backchannel mechanism between changegroup generation to add the
"nbchanges" part parameter. I never liked this backchannel mechanism
and I plan to remove it someday. `hg bundle` still produces the
"nbchanges" part parameter, so there should be no user-visible
change of behavior. I consider this "regression" a bug in
`hg debuggetbundle`. And that bug is captured by an existing
"TODO" in the code to use bundle2 capabilities.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 13 Oct 2016 01:30:14 +0200] rev 30186
mergecopies: rename 'ca' to 'base'
This variable was named after the common ancestor. It is actually the merge
base that might differ from the common ancestor in the graft case. We rename the
variable before a larger refactoring to clarify the situation. Similar rename
was also applied to 'checkcopies' in a prior changeset.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 13 Oct 2016 01:26:33 +0200] rev 30185
copies: move variable document from checkcopies to mergecopies
It appears that 'mergecopies' is the function consuming these data so we move
the documentation there.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Tue, 11 Oct 2016 02:21:42 +0200] rev 30184
checkcopies: pass data as a dictionary of dictionaries
more are coming
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Tue, 11 Oct 2016 02:15:23 +0200] rev 30183
checkcopies: move 'movewithdir' initialisation right before its usage
The 'movewithdir' had a lot of related logic all around the 'mergecopies'.
However it is actually never containing anything until the very last loop in
that function. We move the (simplified) variable definition there for clarity
Mads Kiilerich <madski@unity3d.com> [Fri, 14 Oct 2016 01:53:15 +0200] rev 30182
cmdutil: satisfy expections in dirstateguard.__del__, even if __init__ fails
Python "delstructors" are terrible - this one because it assumed that __init__
had completed before it was called. That would not necessarily be the case if
the repository was read only or broken and saving the dirstate thus failed in
unexpected ways. That could give confusing warnings about missing '_active'
after failures.
To fix that, make sure all member variables are "declared" before doing
anything that possibly could fail. [Famous last words.]
Mads Kiilerich <madski@unity3d.com> [Fri, 14 Oct 2016 01:53:15 +0200] rev 30181
util: increase filechunkiter size to 128k
util.filechunkiter has been using a chunk size of 64k for more than 10 years,
also in years where Moore's law still was a law. It is probably ok to bump it
now and perhaps get a slight win in some cases.
Also, largefiles have been using 128k for a long time. Specifying that size
multiple times (or forgetting to do it) seems a bit stupid. Decreasing it to
64k also seems unfortunate.
Thus, we will set the default chunksize to 128k and use the default everywhere.
Mads Kiilerich <madski@unity3d.com> [Wed, 12 Oct 2016 12:22:18 +0200] rev 30180
largefiles: always use filechunkiter when iterating files
Before, we would sometimes use the default iterator over large files. That
iterator is line based and would add extra buffering and use odd chunk sizes
which could give some overhead.
copyandhash can't just apply a filechunkiter as it sometimes is passed a
genuine generator when downloading remotely.
Yuya Nishihara <yuya@tcha.org> [Fri, 14 Oct 2016 23:33:00 +0900] rev 30179
revset: for x^2, do not take null as a valid p2 revision
Since we don't count null p2 revision as a parent, x^2 should never return
null even if null is explicitly populated.
Yuya Nishihara <yuya@tcha.org> [Mon, 10 Oct 2016 22:30:09 +0200] rev 30178
revset: make follow() reject more than one start revisions
Taking only the last revision is inconsistent because ancestors(set) follows
all revisions given, and theoretically follow(startrev=set) == ancestors(set).
I'm planning to add a support for multiple start revisions, but that won't fit
to the 4.0 time frame. So reject multiple revisions now to avoid future BC.
len(revs) might be slow if revs were large, but we don't care since a valid
revs should have only one element.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 15 Oct 2016 17:10:53 -0700] rev 30177
bundle2: only emit compressed chunks if they have data
This is similar to
58467204cac0. Not all calls into the compressor
return compressed data, as the compressor may buffer compressed
output internally. It is cheaper to check for empty chunks than to
send empty chunks through the generator.
When generating a gzip-v2 bundle of the mozilla-unified repo, this
change results in 50,093 empty chunks not being sent through the
generator (out of 1,902,996 total input chunks).
Danek Duvall <danek.duvall@oracle.com> [Sat, 15 Oct 2016 15:01:14 -0700] rev 30176
color: add some documentation for custom terminfo codes
Danek Duvall <danek.duvall@oracle.com> [Thu, 13 Oct 2016 13:10:01 -0700] rev 30175
color: debugcolor should emit the user-defined colors
This also fixes a long-standing bug that reversed the sense of the color
name and the label used to print it, which was never relevant before.
Danek Duvall <danek.duvall@oracle.com> [Thu, 13 Oct 2016 12:01:41 -0700] rev 30174
color: ignore effects missing from terminfo
If terminfo mode is in effect, and an effect is used which is missing from
the terminfo database, simply silently ignore the request, leaving the
output unaffected rather than causing a crash.
Danek Duvall <danek.duvall@oracle.com> [Thu, 13 Oct 2016 11:48:17 -0700] rev 30173
color: allow for user-configurable terminfo codes for effects
If the entry in the terminfo database for your terminal is missing some
attributes, it should be possible to create them on the fly without
resorting to just making them a color. This change allows you to have
[color]
terminfo.<effect> = <code>
where <effect> might be something like "dim" or "bold", and <code> is the
escape sequence that would otherwise have come from a call to tigetstr().
If an escape character is needed, use "\E". Any such settings will
override attributes that are present in the terminfo database.
Stanislau Hlebik <stash@fb.com> [Tue, 04 Oct 2016 04:06:48 -0700] rev 30172
update: warn if cwd was deleted
During update directories are deleted as soon as they have no entries.
But if current working directory is deleted then it cause problems
in complex commands like 'hg split'. This commit adds a warning
that will help users figure the problem faster.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 13:34:53 +0200] rev 30171
parsers: avoid PySliceObject cast on Python 3
PySlice_GetIndicesEx() accepts a PySliceObject* on Python 2 and a
PyObject* on Python 3. Casting to PySliceObject* on Python 3 was
yielding a compiler warning. So stop doing that.
With this patch, I no longer see any compiler warnings when
building the core extensions for Python 3!
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 13:27:14 +0200] rev 30170
bdiff: include util.h
Without this, IS_PY3K isn't define and the preprocessor uses the
incorrect module loading code, causing the module fail to load at
run-time.
After this patch, all our C extensions (except for watchman's) appear
to import correctly in Python 3!
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 13:22:40 +0200] rev 30169
parsers: alias more PyInt* symbols on Python 3
I feel dirty for having to do this. But this is currently our approach
for dealing with PyInt -> PyLong in Python 3 for this file.
This removes a ton of compiler warnings by fixing unresolved symbols.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 13:17:23 +0200] rev 30168
manifest: use PyVarObject_HEAD_INIT
More appeasing the Python 3 and compiler overlords. The code is
equivalent.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 13:14:14 +0200] rev 30167
dirs: use PyVarObject_HEAD_INIT
This makes a compiler warning go away on Python 3.
Martijn Pieters <mjpieters@fb.com> [Thu, 13 Oct 2016 09:27:37 +0100] rev 30166
py3: use namedtuple._replace to produce new tokens
Martijn Pieters <mjpieters@fb.com> [Fri, 14 Oct 2016 17:55:02 +0100] rev 30165
py3: refactor token parsing to handle call args properly
The token parsing was getting unwieldy and was too naive about accessing
arguments.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Thu, 13 Oct 2016 13:47:47 +0200] rev 30164
eol: make sure we always release the wlock when writing cache
If any exception were to happen after we acquired the wlock, we could leave it
unreleased. We move the wlock release in a 'finally:' close as it should be.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 21:42:11 +0200] rev 30163
pathencode: use assert() for PyBytes_Check()
This should have been added in
a8c948ee3668. I sent the patch to the
list prematurely.
Mads Kiilerich <madski@unity3d.com> [Wed, 12 Oct 2016 12:22:18 +0200] rev 30162
merge: clarify warning for (not) merging flags without ancestor
Give hints why it can't merge and what it will do instead.
Mads Kiilerich <madski@unity3d.com> [Wed, 12 Oct 2016 12:22:18 +0200] rev 30161
merge: only show "cannot merge flags for %s" warning if flags are different
Mads Kiilerich <madski@unity3d.com> [Wed, 12 Oct 2016 12:22:18 +0200] rev 30160
tests: add test coverage of merging x flag without ancestor
It is more noisy than necessary - we will fix that later.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 08 Oct 2016 17:07:43 +0200] rev 30159
dirs: document Py_SIZE weirdness
Assigning to what looks like a function is clown shoes. Document that
it is a macro referring to a struct member.
Philippe Pepiot <philippe.pepiot@logilab.fr> [Wed, 12 Oct 2016 12:22:54 +0200] rev 30158
record: return code from underlying commit
Philippe Pepiot <philippe.pepiot@logilab.fr> [Fri, 14 Oct 2016 09:52:38 +0200] rev 30157
commit: return 1 for interactive commit with no changes (
issue5397)
For consistency with non interactive commit
Mads Kiilerich <madski@unity3d.com> [Fri, 14 Oct 2016 03:03:39 +0200] rev 30156
demandimport: disable lazy import of __builtin__
Demandimport uses the "try to import __builtin__, else use builtins" trick to
handle Python 3. External libraries and extensions might do something similar.
On Fedora 25 subversion-python-1.9.4-4.fc25.x86_64 will do just that (except
the opposite) ... and it failed all subversion convert tests because
demandimport was hiding that it didn't have builtins but should use
__builtin__.
The builtin module has already been imported when demandimport is loaded so
there is no point in trying to import it on demand. Just always ignore both
variants in demandimport.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 12:50:27 +0200] rev 30155
changelog: disable delta chains
This patch disables delta chains on changelogs. After this patch, new
entries on changelogs - including existing changelogs - will be stored
as the fulltext of that data (likely compressed). No delta computation
will be performed.
An overview of delta chains and data justifying this change follows.
Revlogs try to store entries as a delta against a previous entry (either
a parent revision in the case of generaldelta or the previous physical
revision when not using generaldelta). Most of the time this is the
correct thing to do: it frequently results in less CPU usage and smaller
storage.
Delta chains are most effective when the base revision being deltad
against is similar to the current data. This tends to occur naturally
for manifests and file data, since only small parts of each tend to
change with each revision. Changelogs, however, are a different story.
Changelog entries represent changesets/commits. And unless commits in a
repository are homogonous (same author, changing same files, similar
commit messages, etc), a delta from one entry to the next tends to be
relatively large compared to the size of the entry. This means that
delta chains tend to be short. How short? Here is the full vs delta
revision breakdown on some real world repos:
Repo % Full % Delta Max Length
hg 45.8 54.2 6
mozilla-central 42.4 57.6 8
mozilla-unified 42.5 57.5 17
pypy 46.1 53.9 6
python-zstandard 46.1 53.9 3
(I threw in python-zstandard as an example of a repo that is homogonous.
It contains a small Python project with changes all from the same
author.)
Contrast this with the manifest revlog for these repos, where 99+% of
revisions are deltas and delta chains run into the thousands.
So delta chains aren't as useful on changelogs. But even a short delta
chain may provide benefits. Let's measure that.
Delta chains may require less CPU to read revisions if the CPU time
spent reading smaller deltas is less than the CPU time used to
decompress larger individual entries. We can measure this via
`hg perfrevlog -c -d 1` to iterate a revlog to resolve each revision's
fulltext. Here are the results of that command on a repo using delta
chains in its changelog and on a repo without delta chains:
hg (forward)
! wall 0.407008 comb 0.410000 user 0.410000 sys 0.000000 (best of 25)
! wall 0.390061 comb 0.390000 user 0.390000 sys 0.000000 (best of 26)
hg (reverse)
! wall 0.515221 comb 0.520000 user 0.520000 sys 0.000000 (best of 19)
! wall 0.400018 comb 0.400000 user 0.390000 sys 0.010000 (best of 25)
mozilla-central (forward)
! wall 4.508296 comb 4.490000 user 4.490000 sys 0.000000 (best of 3)
! wall 4.370222 comb 4.370000 user 4.350000 sys 0.020000 (best of 3)
mozilla-central (reverse)
! wall 5.758995 comb 5.760000 user 5.720000 sys 0.040000 (best of 3)
! wall 4.346503 comb 4.340000 user 4.320000 sys 0.020000 (best of 3)
mozilla-unified (forward)
! wall 4.957088 comb 4.950000 user 4.940000 sys 0.010000 (best of 3)
! wall 4.660528 comb 4.650000 user 4.630000 sys 0.020000 (best of 3)
mozilla-unified (reverse)
! wall 6.119827 comb 6.110000 user 6.090000 sys 0.020000 (best of 3)
! wall 4.675136 comb 4.670000 user 4.670000 sys 0.000000 (best of 3)
pypy (forward)
! wall 1.231122 comb 1.240000 user 1.230000 sys 0.010000 (best of 8)
! wall 1.164896 comb 1.160000 user 1.160000 sys 0.000000 (best of 9)
pypy (reverse)
! wall 1.467049 comb 1.460000 user 1.460000 sys 0.000000 (best of 7)
! wall 1.160200 comb 1.170000 user 1.160000 sys 0.010000 (best of 9)
The data clearly shows that it takes less wall and CPU time to resolve
revisions when there are no delta chains in the changelogs, regardless
of the direction of traversal. Furthermore, not using a delta chain
means that fulltext resolution in reverse is as fast as iterating
forward. So not using delta chains on the changelog is a clear CPU win
for reading operations.
An example of a user-visible operation showing this speed-up is revset
evaluation. Here are results for
`hg perfrevset 'author(gps) or author(mpm)'`:
hg
! wall 1.655506 comb 1.660000 user 1.650000 sys 0.010000 (best of 6)
! wall 1.612723 comb 1.610000 user 1.600000 sys 0.010000 (best of 7)
mozilla-central
! wall 17.629826 comb 17.640000 user 17.600000 sys 0.040000 (best of 3)
! wall 17.311033 comb 17.300000 user 17.260000 sys 0.040000 (best of 3)
What about 00changelog.i size?
Repo Delta Chains No Delta Chains
hg 7,033,250 6,976,771
mozilla-central 82,978,748 81,574,623
mozilla-unified 88,112,349 86,702,162
pypy 20,740,699 20,659,741
The data shows that removing delta chains from the changelog makes the
changelog smaller.
Delta chains are also used during changegroup generation. This
operation essentially converts a series of revisions to one large
delta chain. And changegroup generation is smart: if the delta in
the revlog matches what the changegroup is emitting, it will reuse
the delta instead of recalculating it. We can measure the impact
removing changelog delta chains has on changegroup generation via
`hg perfchangegroupchangelog`:
hg
! wall 1.589245 comb 1.590000 user 1.590000 sys 0.000000 (best of 7)
! wall 1.788060 comb 1.790000 user 1.790000 sys 0.000000 (best of 6)
mozilla-central
! wall 17.382585 comb 17.380000 user 17.340000 sys 0.040000 (best of 3)
! wall 20.161357 comb 20.160000 user 20.120000 sys 0.040000 (best of 3)
mozilla-unified
! wall 18.722839 comb 18.720000 user 18.680000 sys 0.040000 (best of 3)
! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)
pypy
! wall 4.828317 comb 4.830000 user 4.820000 sys 0.010000 (best of 3)
! wall 5.415455 comb 5.420000 user 5.410000 sys 0.010000 (best of 3)
The data shows eliminating delta chains makes the changelog part of
changegroup generation slower. This is expected since we now have to
compute deltas for revisions where we could recycle the delta before.
It is worth putting this regression into context of overall changegroup
times. Here is the rough total CPU time spent in changegroup generation
for various repos while using delta chains on the changelog:
Repo CPU Time (s) CPU Time w/ compression
hg 4.50 7.05
mozilla-central 111.1 222.0
pypy 28.68 75.5
Before compression, removing delta chains from the changegroup adds
~4.4% overhead to hg changegroup generation, 1.3% to mozilla-central,
and 2.0% to pypy. When you factor in zlib compression, these percentages
are roughly divided by 2.
While the increased CPU usage for changegroup generation is unfortunate,
I think it is acceptable because the percentage is small, server
operators (those likely impacted most by this) have other mechanisms
to mitigate CPU consumption (namely reducing zlib compression level and
pre-generated clone bundles), and because there is room to optimize this
in the future. For example, we could use the nullid as the base revision,
effectively encoding the full revision for each entry in the changegroup.
When doing this, `hg perfchangegroupchangelog` nearly halves:
mozilla-unified
! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)
! wall 11.196461 comb 11.200000 user 11.190000 sys 0.010000 (best of 3)
This looks very promising as a future optimization opportunity.
It's worth that the changes in test-acl.t to the changegroup part size.
This is because revision 6 in the changegroup had a delta chain of
length 2 before and after this patch the base revision is nullrev.
When the base revision is nullrev, cg2packer.deltaparent() hardcodes
the *previous* revision from the changegroup as the delta parent.
This caused the delta in the changegroup to switch base revisions,
the delta to change, and the size to change accordingly. While the
size increased in this case, I think sizes will remain the same
on average, as the delta base for changelog revisions doesn't matter
too much (as this patch shows). So, I don't consider this a regression.
Gregory Szorc <gregory.szorc@gmail.com> [Sat, 24 Sep 2016 12:25:37 -0700] rev 30154
revlog: add instance variable controlling delta chain use
This is to support disabling delta chains on the changelog in a
subsequent patch.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 12:49:47 +0200] rev 30153
changegroup: document deltaparent's choice of previous revision
As part of debugging low-level changegroup generation, I came across
what I initially thought was a weird behavior: changegroup v2 is
choosing the previous revision in the changegroup as a delta base
instead of p1. I was tempted to rewrite this to use p1, as p1
will delta better than prev in the common case. However, I realized
that taking p1 as the base would potentially require resolving a
revision fulltext and thus require more CPU for e.g. server-side
processing of getbundle requests.
This patch tweaks the code comment to note the choice of behavior.
It also notes there is room for a flag or config option to tweak
this behavior later: using p1 as the delta base would likely make
changegroups smaller at the expense of more CPU, which could be
beneficial for things like clone bundles.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Sun, 09 Oct 2016 03:11:18 +0200] rev 30152
help: backout
f3c4edfd35e1 (mark boolean flags with [no-] in help) for now
The ability to negate any boolean flags itself is great, but I think we are not
ready to expose the help side of it yet.
First, while there exist a handful of such flags whose default value can be
changed (eg: git diff, patchwork confirmation), there is only a few of them. The
users who benefit the most from this change are alias users and large
installation that can deploy extension to change behavior (eg: facebook
tweakdefault). So the majority of user who will be affected by a large change
to command help that is not yet relevant to them. (I expect this to become
relevant when ui.progressive start to exists).
Below is an example of the impact of the new help on 'hg help diff':
-r --rev REV [+] revision
-c --change REV change made by revision
-a --[no-]text treat all files as text
-g --[no-]git use git extended diff format
--[no-]nodates omit dates from diff headers
--[no-]noprefix omit a/ and b/ prefixes from filenames
-p --[no-]show-function show which function each change is in
--[no-]reverse produce a diff that undoes the changes
-w --[no-]ignore-all-space ignore white space when comparing lines
-b --[no-]ignore-space-change ignore changes in the amount of white space
-B --[no-]ignore-blank-lines ignore changes whose lines are all blank
-U --unified NUM number of lines of context to show
--[no-]stat output diffstat-style summary of changes
--root DIR produce diffs relative to subdirectory
-I --include PATTERN [+] include names matching the given patterns
-X --exclude PATTERN [+] exclude names matching the given patterns
-S --[no-]subrepos recurse into subrepositories
Another issue with the current state of help, the default value for the
flag is not conveyed to the user. For example in the 'backout' help, there is
no real distinction between "--[no-]backup" (default to True) and "--[no-]keep"
(default) to False:
--[no-]backup no backups
--[no-]keep do not modify working directory during strip
In addition, I've discussed with Augie Fackler and the last batch of the work on
this have burned him out quite some. Therefore he is not intending to perform
any more work on this topic. Quoting him, he would rather see the help part
backed out than spending more time on it.
I do not think we are ready to expose this to users in 4.0 (freeze in a week),
especially because we cannot expect quick improvement on these aspect as this
topic no longer have an owner. We should be able to reintroduce that change in
the future when someone get back on it and the main issues are solves:
* Introduction of ui.progressive makes it relevant for a majority of user,
* Current default value are efficiently conveyed to the user.
(In addition, the excerpt from diff help show that we still have some issue with
some negative option like '--nodates' so further improvement are probably
welcome there.)
Augie Fackler <augie@google.com> [Mon, 19 Sep 2016 17:15:39 -0400] rev 30151
copy: distinguish "file exists" cases and add a hint (BC)
Users that want to add a copy record to an existing commit with 'hg
commit --amend' should be guided towards this workflow, rather than
reaching for some sort of uncommit-recommit flow. As part of this,
distinguish in the top-line error message whether the file merely
already exists (untracked) on disk or the file already exists in
history.
The full list of copy and rename cases and how they interact with
flags are listed below:
target exists --after --force | action
n n * | copy
n y * | (1)
untracked n n | (4) NEWHINT
untracked n y | (3)
untracked y * | (2)
y n n | (4) NEWHINT
y n y | (3)
y y n | (2)
y y y | (3)
deleted n n | copy
deleted n y | (3)
deleted y n | (1)
deleted y y | (1)
* = don't care
(1) <src>: not recording move - <target> does not exist
(2) preserve target contents
(3) replace target contents
(4) <target>: not overwriting - file {exists,already committed}
Credit to Kevin for wholly rewriting my table to cover more cases we
discovered at the sprint.
I think this change gets the hints correct in all cases, but I'd
appreciate close inspection of the test cases to make sure I haven't
gotten turned around in here.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:20 +0900] rev 30150
perf: make perftags clear tags cache correctly
Before this patch, "hg perftags" command doesn't measure performance
of "repo.tags()" correctly, because it doesn't clear tags cache
correctly.
9dca7653b525 replaced repo._tags with repo._tagscache, but didn't
change the code path to clear tags cache in perftags() at that time.
BTW, full history of "tags cache" is:
-
d7df759d0e97 (or 0.6) introduced repo.tagscache as the first "tags cache"
-
5614a628d173 (or 1.4) replaced repo.tagscache with repo._tags
-
9dca7653b525 (or 2.0) replaced repo._tags with repo._tagscache
-
98c867ac1330 (or 2.5) made repo._tagscache filteredpropertycache
To make perftags clear tags cache correctly, and to increase
"historical portability" of perftags, this patch examines existence of
attributes in repo object, and guess appropriate procedure to clear
tags cache.
To avoid examining existence of attributes at each repetition, this
patch makes repocleartagscachefunc() return the function, which
actually clears tags cache.
mozilla-central repo (85 tags on 308365 revs) with each Mercurial
version between before and after this patch.
==== ========= =========
ver before after
==== ========= =========
1.9 0.476062 0.466464
------- *1 -------
2.0 0.346309 0.458327
2.1 0.343106 0.454489
------- *2 -------
2.2 0.069790 0.071263
2.3 0.067829 0.069340
2.4 0.068075 0.069573
------- *3 -------
2.5 0.021896 0.022406
2.6 0.021900 0.022374
2.7 0.021883 0.022379
2.8 0.021949 0.022327
2.9 0.021877 0.022330
3.0 0.021860 0.022314
3.1 0.021869 0.022669
3.2 0.021831 0.022668
3.3 0.021809 0.022691
3.4 0.021861 0.022916
3.5 0.019335 0.020749
3.6 0.019319 0.020866
3.7 0.018781 0.020251
------- *4 -------
3.8 0.068262 0.072558
3.9 0.069682 0.073773
==== ========= =========
(*1) repo._tags was replaced with repo._tagscache at this point
"repo._tags = None" in perftags "before" this patch doesn't clear
tags cache for Mercurial 2.0 or later. This causes significant
gap of "before" between 1.9 and 2.0 .
(*2) I'm not sure about significant gap at this point, but release
note of 2.2 described "a number of significant performance
improvements for large repositories"
(*3) filtered changelog was cached in repoview as repoview.changelog
at this point (by
4d92e2d75cff)
This avoids calculation of filtered changelog at each repetition
of t().
(*4) calculation of filtered changelog was included into wall time at
this point (by
332926212ef8), again
See below for detail about this significant gap:
https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-April/083410.html
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:19 +0900] rev 30149
perf: replace ui.configint() by getint() for Mercurial earlier than 1.9
Before this patch, using ui.configint() prevents perf.py from
measuring performance with Mercurial earlier than 1.9 (or
fa2b596db182), because ui.configint() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 1.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
61c9bc3da402).
This patch replaces ui.configint() invocations by newly introduced
getint().
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of ui.configint() in perf.py.
BTW, this patch doesn't choose adding configint() method at runtime by
replacing ui.__class__ like below, even though this is the recommended
way to modern Mercurial extensions.
def uisetup(ui):
if not util.safehasattr(ui, 'configint'):
class uiwrap(ui.__class__):
def configint(self, section, name, ....):
....
ui.__class__ = uiwrap
Because changes to ui.__class__ by uisetup() of loaded extension have
been propagated since 1.6.1 (or
d8d0fc3988ca), the recommended way
above doesn't work as expected with Mercurial earlier than it.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:19 +0900] rev 30148
perf: omit copying from ui.ferr to ui.fout for Mercurial earlier than 1.9
Before this patch, referring ui.ferr prevents perf.py from measuring
performance with Mercurial earlier than 1.9 (or
4e1ccd4c2b6d), because
ui.ferr isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 1.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
61c9bc3da402).
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:18 +0900] rev 30147
perf: define formatter locally for Mercurial earlier than 2.2
Before this patch, using ui.formatter() prevents perf.py from
measuring performance with Mercurial earlier than 2.2 (or
ae5f92e154d3), because ui.formatter() isn't available in such
Mercurial, even though there are some code paths for Mercurial earlier
than 2.2 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
61c9bc3da402).
This patch defines formatter class locally, and use it instead of the
value returned by ui.formatter(), if perf.py is used with Mercurial
earlier than 2.2.
In this case, we don't need to think about -T/--template option for
formatter, because previous patch made -T/--template disabled for
perf.py with Mercurial earlier than 3.2 (or
7a7eed5176a4).
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:18 +0900] rev 30146
perf: add functions to get vfs-like object for Mercurial earlier than 2.3
Before this patch, using svfs prevents perf.py from measuring
performance of Mercurial earlier than 2.3 (or
7034365089bf), because
svfs isn't available in such Mercurial, even though there are some
code paths for Mercurial earlier than 2.3 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
61c9bc3da402).
To get appropriate vfs-like object to access files under .hg/store,
this patch adds getsvfs() (and also getvfs(), for future use).
To avoid examining existence of attribute at each repetition while
measuring performance, getsvfs() is invoked outside the function to be
called repeatedly.
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of repo.(vfs|svfs|opener|sopener) in perf.py.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:17 +0900] rev 30145
perf: avoid actual writing branch cache out correctly
Mercurial 2.5 (or
9b6ae29d4801) introduced "perfbranchmap" command,
and tried to avoid actual writing branch cache out by replacing
write() of branchcache class in branchmap.py with no-op function
(probably, for elimination of noisy and heavy file I/O factor).
But its implementation isn't correct, because
9b6ae29d4801 replaced
not branchmap.branchcache.write() but branchmap.write(). The latter
doesn't exist, even at that change.
To avoid actual writing branch cache out correctly, this patch
replaces branchmap.branchcache.write() with no-op function.
To detect mistake of replacement or change of API in the future
quickly, this patch uses safeattrsetter() instead of direct attribute
assignment. For similarity between replacements, this patch also
changes replacement of branchmap.read().
In this patch, replacement of read()/write() can run safely outside
"try" block, because two safeattrsetter() invocations ensure that
replacement doesn't cause exception.
FYI, the table below compares "base" filter wall time of perfbranchmap
on recent mozilla-central repo with each Mercurial version between
before and after this patch.
==== ========= =========
ver before after
==== ========= =========
2.5 18.492334 18.232455
2.6 18.733858 18.156702
2.7 18.245598 18.349210
2.8 18.289070 18.528422
2.9 17.572742 16.989655
3.0 17.406953 17.615012
3.1 17.228419 17.689805
3.2 17.862961 17.718367
3.3 2.632110 2.707960
3.4 3.285683 3.272060
3.5 3.370141 3.352176
3.6 3.366939 3.242455
3.7 3.300778 3.367328
3.8 3.300132 3.267298
3.9 3.418996 3.370265
==== ========= =========
IMHO, there is no serious overlooking performance regression.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:17 +0900] rev 30144
perf: get subsettable from appropriate module for Mercurial earlier than 2.9
Before this patch, using branchmap.subsettable prevents perfbranchmap
from measuring performance of Mercurial earlier than 2.9 (or
175c6fd8cacc), because
175c6fd8cacc moved subsettable from repoview.py
to branchmap.py, even though there are some code paths for Mercurial
earlier than 2.9 in perf.py.
For example, setting "_prereadsize" attribute in perfindex() and
perfnodelookup() is effective only with hg earlier than 1.8 (or
61c9bc3da402).
To get subsettable from appropriate module, this patch examines
existence of subsettable in branchmap and repoview.
This patch also adds check-perf-code.py an extra check entry to detect
direct usage of subsettable attribute in perf.py.
FUJIWARA Katsunori <foozy@lares.dti.ne.jp> [Sun, 09 Oct 2016 01:03:16 +0900] rev 30143
perf: introduce safeattrsetter to replace direct attribute assignment
Referring not-existing attribute immediately causes failure, but
assigning a value to such attribute doesn't.
For example, perf.py has code paths below, which assign a value to
not-existing attribute. This causes incorrect performance measurement,
but these code paths are executed successfully.
- "repo._tags = None" in perftags()
recent Mercurial has tags cache information in repo._tagscache
- "branchmap.write = lambda repo: None" in perfbranchmap()
branchmap cache is written out by branchcache.write() in branchmap.py
"util.safehasattr() before assignment" can avoid this issue, but might
increase mistake at "copy & paste" attribute name or so.
To centralize (1) examining existence of, (2) assigning a value to,
and (3) restoring an old value to the attribute, this patch introduces
safeattrsetter(). This is used to replace direct attribute assignment
in subsequent patches.
Encapsulation of restoring is needed to completely remove direct
attribute assignment from perf.py, even though restoring isn't needed
so often.
Mads Kiilerich <madski@unity3d.com> [Sat, 08 Oct 2016 00:59:41 +0200] rev 30142
largefiles: use context for file closing
Make the code slightly smaller and safer (and more deeply indented).
Mads Kiilerich <madski@unity3d.com> [Sat, 08 Oct 2016 00:59:40 +0200] rev 30141
largefiles: when setting/clearing x bit on largefiles, don't change other bits
It is only the X bit that it matters to copy from the standin to the largefile
in the working directory. While it generally doesn't do any harm to copy the
whole mode, it is also "wrong" to copy more than the X bit we care about. It
can make a difference if someone should try to handle largefiles differently,
such as marking them read-only.
Thus, do similar to what utils.setflags does and set the X bit where there are
R bits and obey umask.
Mads Kiilerich <madski@unity3d.com> [Sun, 09 Oct 2016 15:54:49 +0200] rev 30140
eol: on update, only re-check files if filtering changed
Before, update would mark all files as 'normallookup' in dirstate if .hgeol
changed so all files would get the new filtering applied. That takes some time
... and is pointless if the filtering for that file didn't change.
Instead, keep track of the old filtering and only check files where the
filtering is changed.
To keep the old filtering, change to write the applied .hgeol content to
.hg/eol.cache instead of just touching it. That change is backwards/forwards
compatible.
In a real world test, this takes an update that is changing .hgeol and 30000
files from 12s to 4s - where the remaining eol overhead is 1-2s.
Gregory Szorc <gregory.szorc@gmail.com> [Thu, 13 Oct 2016 10:59:29 +0200] rev 30139
dirs: add comment about _PyBytes_Resize
So readers have a canonical function to compare this code to.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Tue, 11 Oct 2016 01:29:08 +0200] rev 30138
checkcopies: extract the '_related' closure
There is not need for it to be a closure.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Sat, 08 Oct 2016 23:00:55 +0200] rev 30137
checkcopies: add an inline comment about the '_related' call
This helps understanding the flow of the function.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Sat, 08 Oct 2016 19:03:16 +0200] rev 30136
checkcopies: minor change to comment
This helped me understand the refactoring so this must be helpful.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Sat, 08 Oct 2016 18:38:42 +0200] rev 30135
checkcopies: rename 'ca' to 'base'
This variable was named after the common ancestor. It is actually the merge
base that might differ from the common ancestor in the graft case. We rename the
variable before a larger refactoring to clarify the situation.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Wed, 24 Aug 2016 05:09:46 +0200] rev 30134
bisect: extra a small initialisation outside of a loop
Having initialisation done during the first iteration is cute, but can be
avoided.
Martijn Pieters <mjpieters@fb.com> [Mon, 10 Oct 2016 23:11:15 +0100] rev 30133
pycompat: only accept a bytestring filepath in Python 2
Mateusz Kwapich <mitrandir@fb.com> [Mon, 10 Oct 2016 05:31:31 -0700] rev 30132
py3: use raw strings in line continuation (convert ext)
Our py2 to py3 string translations marks those as bytestrings.
Mateusz Kwapich <mitrandir@fb.com> [Mon, 10 Oct 2016 05:30:14 -0700] rev 30131
py3: namedtuple takes unicode (journal ext)
namedtuple usage consistent with changelog.py:141
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 09:42:46 -0400] rev 30130
debuginstall: use %d instead of %s for formatting an int
% formatting on bytes on Python 3 is pickier about which % character
we specify.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 09 Oct 2016 13:59:20 +0200] rev 30129
py3: test to check which commands run
This test helps us to keep track on the commands which runs to Python 3.
The full traceback is hidden. Thanks to Augie and Martijn to wrap it up
in four lines.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Wed, 24 Aug 2016 05:06:21 +0200] rev 30128
bisect: build a displayer only once
There is multiple spot using this, building it early will help to extract more
of the logic into the bisect module.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Wed, 24 Aug 2016 05:04:46 +0200] rev 30127
bisect: factor commonly update sequence
For now, This remains a closure in the module to avoid circular import with used
module.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Wed, 24 Aug 2016 04:25:20 +0200] rev 30126
bisect: move check_state into the bisect module
Now that the function is simpler, we resume our quest to move the logic into the
bisect module. In the process, we add basic documentation.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Wed, 24 Aug 2016 04:23:13 +0200] rev 30125
bisect: simplify conditional in 'check_state'
Now that extra code about "updating" flag have been removed, we can simplify the
condition flow and remove a level.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Wed, 24 Aug 2016 04:22:40 +0200] rev 30124
bisect: remove code about "update-flag" in check_state
Now that the flag dedicated to updating the flag are handled earlier, we do not
need to handle them in the 'check_state' function.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Sun, 09 Oct 2016 03:50:55 +0200] rev 30123
bisect: rename 'check_code' to match our naming scheme
We need to do it early, otherwise 'check-commit' will complain every time we
touch it.
Pierre-Yves David <pierre-yves.david@ens-lyon.org> [Wed, 24 Aug 2016 04:48:17 +0200] rev 30122
bisect: minor movement of code handle flag updating state
The code flag handling is quite complicated, we are moving code around to
prepare further simplification.
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 04:37:02 -0400] rev 30121
test-clone: discard lock-related messages
We can't predict where those will show up and they're not
super-important for the contents of this particular test, so just drop
them. Further reduces the flakiness of the test to zero.
Augie Fackler <augie@google.com> [Sun, 09 Oct 2016 04:25:22 -0400] rev 30120
test-clone: fix some instability in pooled clone race condition test
Healthy output (one log file mentioning "existing pooled" and one
mentioning "new pooled") will now print in a stable order, but
unhealthy output will print some sort of error.
This reduces the flakiness of the test from 55% to 38%. My next patch
makes it completely stable.
Martijn Pieters <mjpieters@fb.com> [Sun, 09 Oct 2016 17:44:23 +0200] rev 30119
py3: add an os.fsencode backport to ease path handling
Martijn Pieters <mjpieters@fb.com> [Sun, 09 Oct 2016 14:10:01 +0200] rev 30118
py3: a second argument to open can't be bytes
This fixes open(filename, 'r'), open(filename, 'w'), etc. calls. In Python
3, that second argument *must* be a string, you can't use bytes.
The fix is the same as used with getattr() (where the second argument must
also always be a string); in the tokenizer, where we detect calls, if there
is something that looks like a call to open (and is not an attribute, so
the previous token is not a "." dot) then make sure that that second
argument is not converted to a `bytes` object instead.
There is some remaining issue where the current transformer will also rewrite
open(f('foo')).
However this also affect function for which we perform similar rewrite
('getattr', 'setattr', 'hasattr', 'safehasattr') and will be dealt with in a
follow up.
Yuya Nishihara <yuya@tcha.org> [Sun, 09 Oct 2016 17:02:34 +0200] rev 30117
py3: make check-py3-compat.py import importlib only if necessary
importlib isn't available on Python 2.6, and it isn't necessary for Py2
checks.
Simon Farnsworth <simonfar@fb.com> [Sun, 09 Oct 2016 08:09:20 -0700] rev 30116
templater: handle division by zero in arithmetic
For now, just turn it to an abort.
Simon Farnsworth <simonfar@fb.com> [Sun, 09 Oct 2016 05:51:04 -0700] rev 30115
templater: provide arithmetic operations on integers
The termwidth template keyword is of limited use without some way to ensure
that margins are respected.
Provide a full set of arithmetic operators (four basic operations plus the
mod function, defined to match Python's // for division), so that you can
create termwidth based layouts that match the user's terminal size
Mads Kiilerich <madski@unity3d.com> [Sun, 09 Oct 2016 15:54:42 +0200] rev 30114
eol: store and reuse pattern matchers instead of creating in tight loop
More "right" and more efficient.
Mads Kiilerich <madski@unity3d.com> [Sun, 09 Oct 2016 15:42:42 +0200] rev 30113
eol: fix variable naming - call it _eolmatch instead of _eolfile
It is not the file but a match object based on it.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 09 Oct 2016 13:50:53 +0200] rev 30112
parsers: move PyInt aliasing out of util.h
The PyInt aliasing is only used by parsers.c. Since we don't want to
encourage the use of PyInt parsing, move the aliasing to parsers.c.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 09 Oct 2016 13:47:46 +0200] rev 30111
osutil: use PyLongObject on Python 3 for listdir_slot
This code looks performance sensitive. So let's retain PyIntObject on
Python 2 and use PyLongObject explicitly on Python 3.
Gregory Szorc <gregory.szorc@gmail.com> [Sun, 09 Oct 2016 13:41:18 +0200] rev 30110
osutil: use PyLongObject in recvfds
PyIntObject doesn't exist in Python 3. While PyIntObject is preferred
on Python 2 because it is a fixed capacity and faster, the difference
between PyIntObject and PyLongObject for scenarios where performance
isn't critical or the caller isn't performing type checking shouldn't
be relevant.
So change recvfds to return a list of longs instead of ints on Python
2.
Pulkit Goyal <7895pulkit@gmail.com> [Sun, 09 Oct 2016 12:37:10 +0200] rev 30109
py3: use encoding.environ instead of os.environ
This complains while running hg version on Python 3.5