Gregory Szorc <gregory.szorc@gmail.com> [Tue, 13 Nov 2018 12:30:59 -0800] rev 40670
revlog: detect incomplete revlog reads
_readsegment() is supposed to return N bytes of revlog revision
data starting at a file offset. Surprisingly, its behavior before
this patch never verified that it actually read and returned N
bytes! Instead, it would perform the read(), then return whatever
data was available. And even more surprisingly, nothing in the
call chain appears to have been validating that it received all
the data it was expecting.
This behavior could lead to partial or incomplete revision chunks
being operated on. This could result in e.g. cached deltas being
applied against incomplete base revisions. The delta application
process would happily perform this operation. Only hash
verification would detect the corruption and save us.
This commit changes the behavior of raw revlog reading to validate
that we actually read() the number of bytes that were requested.
We will raise a more specific error faster, rather than possibly
have it go undetected or manifest later in the call stack, at
delta application or hash verification.
Differential Revision: https://phab.mercurial-scm.org/D5266
Gregory Szorc <gregory.szorc@gmail.com> [Tue, 30 Oct 2018 16:50:05 -0700] rev 40669
revlog: use single file handle when de-inlining revlog
_getsegmentforrevs() will eventually call into _datareadfp() to
resolve a file handle to read revision data. If no file handle
is passed into _getsegmentforrevs(), it opens a new one.
Explicit is better than implicit.
This commit changes _enforceinlinesize() to open a file handle
explicitly when converting inline revlogs to split revlogs and
to pass this file handle into _getsegmentforrevs().
I haven't measured, but this change should improve performance,
as we no longer reopen the revlog for reading for every revision
in the revlog when it is converted from inline to split. Instead,
we open it at most once and use it for the duration of the
operation. That being said, I /think/ the chunk cache may mitigate
the number of file opens required.
Differential Revision: https://phab.mercurial-scm.org/D5265
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:44:09 +0300] rev 40668
store: raise ProgrammingError if unable to decode a storage path
Right now, the function magically return False which is dangerous, so let's
raise ProgrammingError.
Suggested by Augie in D5139.
Differential Revision: https://phab.mercurial-scm.org/D5264
Matt Harbison <matt_harbison@yahoo.com> [Tue, 13 Nov 2018 23:54:23 -0500] rev 40667
tests: document a known failing interaction between narrow and lfs
This is one of the two remaining aborts I found looking into issue5794. I've
got no idea what's wrong with the hook, since the changes there fixed the other
two problems noted in that bug report. It seems like it might go away when the
narrow issue is fixed, but let's make sure this doesn't get lost.
The stacktrace for the hook seems to indicate that the missing file *is* in ctx:
remote: Traceback (most recent call last):
remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in checkrequireslfs
remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr>
remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
remote: File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 191, in filectxislfs
remote: return _islfs(self.filelog(), self.filenode())
remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode
remote: return self._filenode
remote: File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__
remote: result = self.func(obj)
remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode
remote: return self._filelog.lookup(self._fileid)
remote: File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 68, in lookup
remote: self._revlog.indexfile)
remote: File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 218, in fileidlookup
remote: raise error.LookupError(fileid, identifier, _('no match found'))
remote: LookupError: data/inside2/f.i@f59b4e021835: no match found
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:55:58 +0900] rev 40666
logtoprocess: drop support for ui.log() call with invalid msg arguments (BC)
Before, the logtoprocess extension put a formatted message into $MSG1, and
its arguments to $MSG2... If the specified arguments couldn't be formatted
because of a caller bug, an unformatted message was passed in to $MSG1
instead of exploding. This behavior doesn't make sense.
Since I'm planning to formalize the ui.log() interface such that we'll no
longer have to extend the ui class, I want to remove any features not
conforming to the ui.log() API. So this patch removes the support for
ill-formed arguments, and $MSG{n} (where n > 1) parameters which seems
useless as long as the message can be formatted. The $MSG1 variable isn't
renamed for the maximum compatibility.
In future patches, a formatted msg will be passed to a processlogger object,
instead of overriding the ui.log() function.
.. bc::
The logtoprocess extension no longer supports invalid ``ui.log()``
arguments. A log message is always formatted and passed in to the
``$MSG1`` environment variable.
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:35:38 +0900] rev 40665
py3: byte-stringify inline extension in test-logtoprocess.t
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:33:14 +0900] rev 40664
logtoprocess: rewrite dict building in py3-compatible way
Yuya Nishihara <yuya@tcha.org> [Sun, 11 Nov 2018 12:27:23 +0900] rev 40663
logtoprocess: leverage procutil.shellenviron() to stringify variables (BC)
This should make the extension more Py3 friendly. The environment variables
of the main process are copied to the dict by shellenviron().
.. bc::
Boolean options passed to the logtoprocess extension are now formatted
as ``0`` or ``1`` instead of ``None``, ``False``, or ``True``.
Yuya Nishihara <yuya@tcha.org> [Tue, 13 Nov 2018 22:15:30 +0900] rev 40662
templater: check invalid use of list expression properly (issue5920)
The error message is still cryptic, but it should be better.
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:08:55 +0300] rev 40661
py3: add b suffix to make sure file is opened in bytes mode
Differential Revision: https://phab.mercurial-scm.org/D5263
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:08:17 +0300] rev 40660
py3: don't use dict.iterkeys()
dict.iterkeys() is not present on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5262
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 18:07:21 +0300] rev 40659
py3: use node.hex(h.digest()) instead of h.hexdigest()
hashlib.sha1.hexdigest() returns str on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5261
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 17:41:54 +0300] rev 40658
py3: make second argument of fdopen() a str
# skip-blame as just r'' prefixes
Differential Revision: https://phab.mercurial-scm.org/D5260
Pulkit Goyal <pulkit@yandex-team.ru> [Tue, 13 Nov 2018 17:41:26 +0300] rev 40657
py3: fix keyword arguments handling in hgext/remotefilelog/
Keys of kwargs on Python 3 should be strings. This patch fixes them by appending
r'' prefixes, and using pycompat.byteskwargs() and pycompat.strkwargs().
Differential Revision: https://phab.mercurial-scm.org/D5259
Pulkit Goyal <pulkit@yandex-team.ru> [Sat, 13 Oct 2018 05:09:18 +0300] rev 40656
py3: add __pycache__ to .hgignore
Something changed with Python 3 just like million on things. Maybe they should
named Python 3 as anaconda or cobra instead of just increasing the version
number.
This makes test-custom-filters.t pass on Python 3.
Differential Revision: https://phab.mercurial-scm.org/D5039
Boris Feld <boris.feld@octobus.net> [Thu, 08 Nov 2018 16:07:16 +0100] rev 40655
sparse-revlog: stop using a heap to track selected gap
Same logic as for 'gapsheap', we don't actually need a heap.
Boris Feld <boris.feld@octobus.net> [Thu, 08 Nov 2018 16:01:30 +0100] rev 40654
sparse-revlog: stop using a heap to track gaps
The heap doesn't bring any performance advantage as we can simply sort the
final list.
Moreover, the lesser complexity helps a lot when we later implement it in C.
Boris Feld <boris.feld@octobus.net> [Thu, 08 Nov 2018 15:29:58 +0100] rev 40653
sparse-revlog: fast-path before computing payload size
In this fast-path case, we do not need to compute the full delta chain
payload. Since it comes with a significant cost, we avoid doing so if
possible.
Boris Feld <boris.feld@octobus.net> [Tue, 16 Oct 2018 14:46:28 +0200] rev 40652
sparse-revlog: drop unused deltainfo parameter from segmentspan
We no longer need to deal with slicing logic including uncommitted revision.
We drop the associated code.
Boris Feld <boris.feld@octobus.net> [Tue, 16 Oct 2018 14:43:51 +0200] rev 40651
sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity
We no longer need to deal with slicing logic including uncommitted revision. We
drop the associated code.
Boris Feld <boris.feld@octobus.net> [Mon, 15 Oct 2018 15:45:08 +0200] rev 40650
sparse-revlog: skip the span check in the sparse-revlog case
This significantly improves the performance on unbundling on smaller
repositories.
Mercurial: unbundling 1K revisions
no-sparse-revlog: 500 ms
sparse-revlog-before: 689 ms
sparse-revlog-after: 484 ms
Pypy: unbundling 1K revisions
no-sparse-revlog: 1.242 s
sparse-revlog-before: 1.135 s
sparse-revlog-after: 0.860 s
NetBeans: unbundling 1K revisions
no-sparse-revlog: 1.386 s
sparse-revlog-before: 2.368 s
sparse-revlog-after: 1.191 s
Mozilla: unbundling 1K revisions
no-sparse-revlog: 3.103 s
sparse-revlog-before: 3.367 s
sparse-revlog-after: 3.093 s
Augie Fackler <augie@google.com> [Wed, 17 Oct 2018 17:15:42 -0400] rev 40649
histedit: import chistedit curses UI from hg-experimental
I don't tend to like curses interfaces, but this gets enough use at
work that it seems like it's worth bringing into core. This is a
minimal import from hg-experimental revision 4c7f33bf5f00, in that
I've done the smallest amount of code movement and editing in order to
import the functionality.
.. feature::
`hg histedit` will now present a curses UI if curses is available
and `ui.interface` or `ui.interface.histedit` is set to `curses`.
Differential Revision: https://phab.mercurial-scm.org/D5146
Augie Fackler <augie@google.com> [Mon, 12 Nov 2018 20:32:58 -0500] rev 40648
manifest: perform cheap checks before potentially allocating memory
Differential Revision: https://phab.mercurial-scm.org/D5258
Augie Fackler <augie@google.com> [Mon, 12 Nov 2018 20:31:57 -0500] rev 40647
manifest: also reject obviously-too-short lines when parsing lines
Differential Revision: https://phab.mercurial-scm.org/D5257
Augie Fackler <augie@google.com> [Mon, 12 Nov 2018 20:35:22 -0500] rev 40646
manifest: make sure there's a filename before bothering to look for newline
There's no valid manifest that would have no characters before the NUL byte on
a line, and this fixes some erratic timeouts in the fuzzer.
Differential Revision: https://phab.mercurial-scm.org/D5256
Augie Fackler <augie@google.com> [Tue, 06 Nov 2018 11:12:56 -0500] rev 40645
revlog: replace PyInt_AS_LONG with a more portable helper function
PyInt_AS_LONG disappears on Python, and our previous #define was
producing some problems on Python 3. Let's give up and make an inline
helper function that makes this more sane.
Differential Revision: https://phab.mercurial-scm.org/D5235
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 22:51:36 +0900] rev 40644
help: document weird behavior of uisetup()
While refactoring ui.log() API, it was really annoying that uisetup() is
actually an extsetup() the phase 1. Let's document that. I'm planning to
add another uisetup() which is called per ui instance, though I can't think
of a good name for it.
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 22:26:24 +0900] rev 40643
help: merge section about uisetup() and extsetup()
They are technically the same callback, called only once per process.
The section name "ui setup" is confusing, so shouldn't be used.
Yuya Nishihara <yuya@tcha.org> [Mon, 12 Nov 2018 21:28:54 +0900] rev 40642
help: add internals.extensions topic
I think it's better to include the API overview in core as we now have the
internals section in our help system.
Retrieved from the wiki, and formatted as reST. Several wiki links are
removed since they are invalid in the help. The sections about example
extension are removed at all as they seemed too verbose.
https://www.mercurial-scm.org/wiki/WritingExtensions
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Nov 2018 12:51:14 +0900] rev 40641
commandserver: send raw progress information to message channel
This is pretty basic implementation to support GUI progress bar.
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Nov 2018 12:38:49 +0900] rev 40640
commandserver: make getpass() request distinct from normal prompt
Otherwise, GUI clients would have to parse the prompt text.
Yuya Nishihara <yuya@tcha.org> [Sun, 04 Nov 2018 12:17:20 +0900] rev 40639
commandserver: attach prompt default and choices to message
These attributes are important to provide a GUI prompt to user.