Gregory Szorc <gregory.szorc@gmail.com> [Wed, 17 Oct 2018 17:32:15 +0200] rev 40392
sqlitestore: support for storing revisions without their parents
This commit kinda/sorta implements the equivalent of ellipsis nodes for
the SQLite storage backend.
Without implementing full blown ellipsis nodes (and the necessary support for
them in the wire protocol), we instead teach the store to rewrite the p1 and
p2 nodes to nullid when the incoming parent isn't in the local store. This allows
servers to remain dumb and send the real parent and have the clients deal
with the missing parent problem.
This obviously isn't ideal because a benefit of ellipsis nodes is we can
insert a fake parent to ellide missing changesets. But neither solution is
ideal because it drops the original parent from storage. We could probably
teach the SQLite store to retain the original parent and handle missing
parents at read time. However, parent revisions are stored as integers and
it isn't trivial to store an "empty" revision in the store yet, which would
be necessary to represent the "missing" parent.
The store is somewhat intelligent in trying to remove the missing parents
metadata when the revision is re-added. But, revision numbers will be all
messed up in that case, so I'm not sure it is worth it. At some point we'll
likely want to remove the concept of revision numbers from the database and
have the store invent them at index generation time. Or even better, we can
do away with revision numbers from the file storage interface completely.
We'll get there eventually...
Differential Revision: https://phab.mercurial-scm.org/D5168
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 19 Oct 2018 15:38:25 +0200] rev 40391
wireprotov2: support exposing linknode of file revisions
When supporting shallow file storage, clients may fetch file revisions
by changeset. But they may not readily know which changeset introduced a
specific file revision. The "linknode" is used to record which changeset
introduces which file revision.
This commit teaches the "filedata" and "filesdata" wire protocol commands
to expose the linknode for file revisions. The implementation is likely
wrong when hidden changesets are in play, since the linknode may refer to
a hidden changeset. We can deal with this problem later.
Differential Revision: https://phab.mercurial-scm.org/D5167
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 19 Oct 2018 14:59:03 +0200] rev 40390
localrepo: support marking repos as having shallow file storage
Various operations against repositories need to know if repository
storage is full or partial. For example, a checkout (including possibly
a widening of a sparse checkout), needs to know if it can assume all file
revisions are available or whether to look for missing revisions first.
This commit lays the plumbing for doing that.
We define a repo creation option that indicates that shallow file storage
is desired.
The SQLite store uses this creation option to add an extra repo requirement
indicating file storage is shallow.
A new repository feature has been added to indicate that file storage is
shallow. The SQLite store adds this feature when the shallow file store
requirement is present.
Code can now look at repo.features to determine if repo file storage may
be shallow and take additional actions if so.
While we're here, we also teach the SQLite store to handle the narrow repo
requirement, which gets added when making narrow clones.
Differential Revision: https://phab.mercurial-scm.org/D5166
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 26 Sep 2018 14:41:15 -0700] rev 40389
repository: teach addgroup() to receive data with missing parents
The way the narrow extension works today, the server rewrites
outgoing changegroup data to lie about parents when the parents
data is missing. It adds the ellipsis flag to the revision so
it can be recorded as such in the revlog.
In the new wire protocol, such rewriting does not occur on
the server (at least not yet anyway). Instead, it is up to the
client to recognize when it has received a revision without its
parents. This means rewriting will be performed on the client.
Furthermore, the mechanism for storing a shallow revision may
differ from store to store. For example, the revlog store uses
the ellipsis flag to denote a revision's parents have been
rewritten. But a non-revlog store may wish to store things
differently. And, some stores may not even support receiving
shallow revision data!
Therefore, it makes sense for the store itself to be making
decisions about what to do when they receive revision data
without their parents.
This commit teaches the addgroup() bulk insert method to accept
a boolean argument that indicates whether the incoming data may
lack parent revisions. This flag can be set when receiving
"shallow" data from a remote.
The revlog implementation of this method has been taught to rewrite
the missing parent(s) to nullid and to add the ellipsis flag to
the revision when a missing parent is encountered. But it only
does this if ellipsis flags are enabled on the repo and the
incoming data is marked as possibly shallow. An error occurs
otherwise.
Differential Revision: https://phab.mercurial-scm.org/D5165
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 19 Oct 2018 13:44:25 +0200] rev 40388
commands: support passing depth to hg.clone()
This will allow extensions to add --depth or other arguments to control
depth fetching.
Differential Revision: https://phab.mercurial-scm.org/D5164
Gregory Szorc <gregory.szorc@gmail.com> [Wed, 03 Oct 2018 14:57:29 -0700] rev 40387
filelog: add a hasnode() method (API)
Missing in the file storage interface is the ability to query whether
a specified value is a known node.
This commit defines that interface member and implements it on the
revlog and sqlite file stores.
Storage unit tests have been added.
The revlog implementation is a bit more complicated because index lookups
don't consistently raise the same exception. For SQLite, we can simply look
for a key in a dict.
Differential Revision: https://phab.mercurial-scm.org/D5163
Matt Harbison <matt_harbison@yahoo.com> [Sun, 21 Oct 2018 22:26:00 -0400] rev 40386
lfs: consult the narrow matcher when extracting pointers from ctx (
issue5794)
I added a testcase for lfs to all narrow tests, and the following failed:
test-narrow-acl.t
test-narrow-exchange.t
test-narrow-patterns.t
test-narrow-strip.t
test-narrow-trackedcmd.t
test-narrow-widen.t
test-narrow.t
The first two still have errors in the pretxnchangegroup on clone and (receiving
a) push, which I'm still looking into (
4d63f3bc1e1a fixed something in this area
already). These two modified tests seem to cover the things that failed in the
remaining narrow tests, i.e. `hg tracked` and `hg strip`, so I didn't bother
enabling the testcases elsewhere. Maybe we should, but it's 68 tests total.
Yuya Nishihara <yuya@tcha.org> [Sat, 20 Oct 2018 20:25:56 +0900] rev 40385
statprof: fix overflow while skipping boilerplate parts
I got IndexError randomly because of stack[i] where i = len(stack).
Yuya Nishihara <yuya@tcha.org> [Sat, 20 Oct 2018 20:15:48 +0900] rev 40384
statprof: fix indent level of fp.write() (
issue6004)
It was changed at
9d3034348c4f by mistake.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 19 Oct 2018 22:31:47 -0400] rev 40383
py3: stringify setupversion on Windows
This was stringified a few lines above for non Windows platforms, but `version`
remains bytes. The old code effectively undid the conversion, and triggered a
warning in setuptools when building.
Matt Harbison <matt_harbison@yahoo.com> [Fri, 19 Oct 2018 23:47:38 -0400] rev 40382
tests: add coverage for some untested areas of hgweb
The fact that these mimetype guesses weren't blowing up anywhere on py3 prior to
9310037f0636 was the giveaway. The annotate function is a bit unusual in that
it renders the page with a 500 in the middle, so I left the HTML output. For
the other functions, checking the access log is enough.
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 19 Oct 2018 23:30:56 +0300] rev 40381
statprof: update the name as the i increases (
issue6003)
2864f8d3fcd6 while working on py3 fix, take out the name building out of the
loop so we were not building the new stack-name for each i, rather we were using
the first one again and again.
The test changes shows the profile is now working.
Differential Revision: https://phab.mercurial-scm.org/D5172
Pulkit Goyal <pulkit@yandex-team.ru> [Fri, 19 Oct 2018 23:18:29 +0300] rev 40380
test: show more profile lines in test-profile.t
This shows that we don't output anything after the first line and demonstrate
issue6003.
Differential Revision: https://phab.mercurial-scm.org/D5171
Augie Fackler <augie@google.com> [Fri, 19 Oct 2018 11:45:51 -0400] rev 40379
keepalive: use getattr to avoid AttributeErrors when vcr is in use
Fixes test-phabricator.t.
Differential Revision: https://phab.mercurial-scm.org/D5160
Augie Fackler <augie@google.com> [Fri, 19 Oct 2018 11:45:25 -0400] rev 40378
phabricator: do more of the VCR work in demandimport.deactivated()
If I don't do this, VCR gets confused looking for pycurl and other
libraries. I have no idea how this ever worked.
Differential Revision: https://phab.mercurial-scm.org/D5159
Augie Fackler <augie@google.com> [Fri, 19 Oct 2018 11:28:29 -0400] rev 40377
tests: sleep longer in test-logtoprocess.t
We should probably write some sort of helper that can wait N seconds
for all specified values to appear in a file or something, but for now
this will fix the FreeBSD buildbot.
Differential Revision: https://phab.mercurial-scm.org/D5157
Augie Fackler <augie@google.com> [Fri, 19 Oct 2018 11:31:18 -0400] rev 40376
tests: fix pyflakes warning in test-duplicateoptions.py
Differential Revision: https://phab.mercurial-scm.org/D5158
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 19 Oct 2018 16:34:45 +0200] rev 40375
branchmap: avoid changelog and attribute lookups in replacecache()
This should make things faster. I'm not sure which operations would benefit
from it though. Maybe branchmap application on clone?
Differential Revision: https://phab.mercurial-scm.org/D5162
Gregory Szorc <gregory.szorc@gmail.com> [Fri, 19 Oct 2018 16:16:17 +0200] rev 40374
branchmap: pass changelog into branchmap functions
As part of building the branchmap, we loop over revs and call branchmap()
or _branchmap(). Previously, these functions were accessing repo.changelog.
We know from past experience that repo.changelog in loops is bad for
performance.
This commit teaches the branchmap code to pass a changelog instance into
branchmap() and _branchmap() so we don't need to pay this penalty.
On my MBP, this appears to show a speedup on a clone of the
mozilla-unified repo:
$ hg perfbranchmap --clear-revbranch
! base
! wall 21.078160 comb 21.070000 user 20.920000 sys 0.150000 (best of 3)
! wall 20.574682 comb 20.560000 user 20.400000 sys 0.160000 (best of 3)
$ hg perfbranchmap
! base
! wall 4.880413 comb 4.870000 user 4.860000 sys 0.010000 (best of 3)
! wall 4.573968 comb 4.560000 user 4.550000 sys 0.010000 (best of 3)
Differential Revision: https://phab.mercurial-scm.org/D5161
Augie Fackler <augie@google.com> [Thu, 18 Oct 2018 16:36:10 -0400] rev 40373
fuzz: move many initialization steps into LLVMFuzzerInitialize
Doing this means that things we intentionally leak (eg type objects)
no longer confuse AddressSanitizer, so now we can run the fuzzer MUCH
longer.
Differential Revision: https://phab.mercurial-scm.org/D5154
Martin von Zweigbergk <martinvonz@google.com> [Thu, 17 Nov 2016 15:51:33 -0800] rev 40372
bundle2: fix broken compression engine assertion
bundletype() is a function, so it needs to be called, and it is
documented to return a 2-tuple. This code is untested, so that's why
we haven't noticed the bad assertion.
Differential Revision: https://phab.mercurial-scm.org/D5155
Matt Harbison <matt_harbison@yahoo.com> [Thu, 18 Oct 2018 17:54:07 -0400] rev 40371
tests: glob over a difference between Windows 7 and Window 10
The error value is 11001 on Windows 10. I have no idea why it changed, but it
seems unimportant.
Matt Harbison <matt_harbison@yahoo.com> [Thu, 18 Oct 2018 18:11:16 -0400] rev 40370
py3: fix module imports in test-highlight.t
The hash changes are because the *.py file is committed to the repo.
Matt Harbison <matt_harbison@yahoo.com> [Wed, 17 Oct 2018 23:33:43 -0400] rev 40369
py3: fix module imports in tests, as flagged by test-check-module-imports.t
I have no idea why these aren't flagged with python2. I excluded
test-highlight.t for now to make this easier to review- the changed code is
committed to a repo, which has cascading changes on the rest of the test.
There's a mix of bytes and str in the imports dict of contrib/import-checker.py
that crashed it half way through listing out these errors. I couldn't figure
out how to fix that properly, so I was lazy and applied this on py3, to find the
rest of the errors:
diff --git a/contrib/import-checker.py b/contrib/import-checker.py
--- a/contrib/import-checker.py
+++ b/contrib/import-checker.py
@@ -626,7 +626,12 @@ def find_cycles(imports):
top.foo -> top.qux -> top.foo
"""
cycles = set()
- for mod in sorted(imports.keys()):
+ def sort(v):
+ if isinstance(v, bytes):
+ return v.decode('ascii')
+ return v
+
+ for mod in sorted(imports.keys(), key=sort):
try:
checkmod(mod, imports)
except CircularImport as e:
Matt Harbison <matt_harbison@yahoo.com> [Thu, 18 Oct 2018 21:55:47 -0400] rev 40368
lfs: don't add extension to hgrc after conversion (BC)
This is in the spirit of
bcf72d7b1524.
Yuya Nishihara <yuya@tcha.org> [Thu, 18 Oct 2018 21:00:07 +0900] rev 40367
addremove: add "ui." prefix to message color keys
I don't like fully-colorized status/warning messages, and I want to disable
them at all. If we'd supported a syntax like 'color.ui.*=none', I could
easily turn addremove.added/removed off as well as ui.error. This patch is
just for that.
Since addremove colors aren't released yet, which were added at
ddc1da134772,
there are no compatibility concerns.
Martin von Zweigbergk <martinvonz@google.com> [Thu, 09 Feb 2017 09:17:40 -0800] rev 40366
update: clarify update() call sites by specifying argument names
merge.update() takes a lot of parameters and I get confused all the
time which is which.
Differential Revision: https://phab.mercurial-scm.org/D5153
Martin von Zweigbergk <martinvonz@google.com> [Thu, 18 Oct 2018 10:11:08 -0700] rev 40365
debugcommands: avoid stack trace from debugindexstats in pure mode
This has been broken since I added it in
d71e0ba34d9b (debugcommands:
add a debugindexstats command, 2018-08-08). This patch also fixes the
test.
Differential Revision: https://phab.mercurial-scm.org/D5152
Augie Fackler <augie@google.com> [Thu, 18 Oct 2018 11:24:20 -0400] rev 40364
tests: fix up pure case of test-sqlitestore.t
This is clearly what the line should read based on the "force to zlib"
section below, so I'm guessing it just got overlooked during development.
Differential Revision: https://phab.mercurial-scm.org/D5151
Augie Fackler <augie@google.com> [Thu, 18 Oct 2018 11:14:04 -0400] rev 40363
tests: don't emit false failures when sqlite3 is missing
I'm honestly surprised we have buildbot coverage for this, but we do!
Differential Revision: https://phab.mercurial-scm.org/D5150