revlog: automatically read from opened file handles The revlog reading code commonly opens a new file handle for reading on demand. There is support for passing a file handle to revlog.revision(). But it is marked as an internal argument. When revlogs are written, we write() data as it is available. But we don't flush() data until all revisions are written. Putting these two traits together, it is possible for an in-process revlog reader during active writes to trigger the opening of a new file handle on a file with unflushed writes. The reader won't have access to all "available" revlog data (as it hasn't been flushed). And with the introduction of the previous patch, this can lead to the revlog raising an error due to a partial read. I witnessed this behavior when applying changegroup data (via `hg pull`) before issue6006 was fixed via different means. Having this and the previous patch in play would have helped cause errors earlier rather than manifesting as hash verification failures. While this has been a long-standing issue, I believe the relatively new delta computation code has tickled it into being more common. This is because the new delta computation code will compute deltas in more scenarios. This can lead to revlog reading. While the delta computation code is probably supposed to reuse file handles, it appears it isn't doing so in all circumstances. But the issue runs deeper than that. Theoretically, any code can access revision data during revlog writes. It appears we were just getting lucky that it wasn't. (The "add revision callback" passed to addgroup() provides an avenue to do this.) If I changed the revlog's behavior to not cache the full revision text or to clear caches after revision insertion during addgroup(), I was able to produce crashes 100% of the time when writing changelog revisions. This is because changelog's add revision callback attempts to resolve the revision data to access the changed files list. And without the revision's fulltext being cached, we performed a revlog read, which required opening a new file handle. This attempted to read unflushed data, leading to a partial read and a crash. This commit teaches the revlog to store the file handles used for writing multiple revisions during addgroup(). It also teaches the code for resolving a file handle when reading to use these handles, if available. This ensures that *any* reads (regardless of their source) use the active writing file handles, if available. These file handles have access to the unflushed data because they wrote it. This allows reads to complete without issue. Differential Revision: https://phab.mercurial-scm.org/D5267

revlog: detect incomplete revlog reads _readsegment() is supposed to return N bytes of revlog revision data starting at a file offset. Surprisingly, its behavior before this patch never verified that it actually read and returned N bytes! Instead, it would perform the read(), then return whatever data was available. And even more surprisingly, nothing in the call chain appears to have been validating that it received all the data it was expecting. This behavior could lead to partial or incomplete revision chunks being operated on. This could result in e.g. cached deltas being applied against incomplete base revisions. The delta application process would happily perform this operation. Only hash verification would detect the corruption and save us. This commit changes the behavior of raw revlog reading to validate that we actually read() the number of bytes that were requested. We will raise a more specific error faster, rather than possibly have it go undetected or manifest later in the call stack, at delta application or hash verification. Differential Revision: https://phab.mercurial-scm.org/D5266

revlog: use single file handle when de-inlining revlog _getsegmentforrevs() will eventually call into _datareadfp() to resolve a file handle to read revision data. If no file handle is passed into _getsegmentforrevs(), it opens a new one. Explicit is better than implicit. This commit changes _enforceinlinesize() to open a file handle explicitly when converting inline revlogs to split revlogs and to pass this file handle into _getsegmentforrevs(). I haven't measured, but this change should improve performance, as we no longer reopen the revlog for reading for every revision in the revlog when it is converted from inline to split. Instead, we open it at most once and use it for the duration of the operation. That being said, I /think/ the chunk cache may mitigate the number of file opens required. Differential Revision: https://phab.mercurial-scm.org/D5265

store: raise ProgrammingError if unable to decode a storage path Right now, the function magically return False which is dangerous, so let's raise ProgrammingError. Suggested by Augie in D5139. Differential Revision: https://phab.mercurial-scm.org/D5264

tests: document a known failing interaction between narrow and lfs This is one of the two remaining aborts I found looking into issue5794. I've got no idea what's wrong with the hook, since the changes there fixed the other two problems noted in that bug report. It seems like it might go away when the narrow issue is fixed, but let's make sure this doesn't get lost. The stacktrace for the hook seems to indicate that the missing file *is* in ctx: remote: Traceback (most recent call last): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in checkrequireslfs remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr> remote: if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()): remote: File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 191, in filectxislfs remote: return _islfs(self.filelog(), self.filenode()) remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode remote: return self._filenode remote: File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__ remote: result = self.func(obj) remote: File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode remote: return self._filelog.lookup(self._fileid) remote: File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 68, in lookup remote: self._revlog.indexfile) remote: File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 218, in fileidlookup remote: raise error.LookupError(fileid, identifier, _('no match found')) remote: LookupError: data/inside2/f.i@f59b4e021835: no match found

logtoprocess: drop support for ui.log() call with invalid msg arguments (BC) Before, the logtoprocess extension put a formatted message into $MSG1, and its arguments to $MSG2... If the specified arguments couldn't be formatted because of a caller bug, an unformatted message was passed in to $MSG1 instead of exploding. This behavior doesn't make sense. Since I'm planning to formalize the ui.log() interface such that we'll no longer have to extend the ui class, I want to remove any features not conforming to the ui.log() API. So this patch removes the support for ill-formed arguments, and $MSG{n} (where n > 1) parameters which seems useless as long as the message can be formatted. The $MSG1 variable isn't renamed for the maximum compatibility. In future patches, a formatted msg will be passed to a processlogger object, instead of overriding the ui.log() function. .. bc:: The logtoprocess extension no longer supports invalid ``ui.log()`` arguments. A log message is always formatted and passed in to the ``$MSG1`` environment variable.

py3: byte-stringify inline extension in test-logtoprocess.t

logtoprocess: rewrite dict building in py3-compatible way

logtoprocess: leverage procutil.shellenviron() to stringify variables (BC) This should make the extension more Py3 friendly. The environment variables of the main process are copied to the dict by shellenviron(). .. bc:: Boolean options passed to the logtoprocess extension are now formatted as ``0`` or ``1`` instead of ``None``, ``False``, or ``True``.

templater: check invalid use of list expression properly (issue5920) The error message is still cryptic, but it should be better.

py3: add b suffix to make sure file is opened in bytes mode Differential Revision: https://phab.mercurial-scm.org/D5263

py3: don't use dict.iterkeys() dict.iterkeys() is not present on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5262

py3: use node.hex(h.digest()) instead of h.hexdigest() hashlib.sha1.hexdigest() returns str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5261

py3: make second argument of fdopen() a str # skip-blame as just r'' prefixes Differential Revision: https://phab.mercurial-scm.org/D5260

py3: fix keyword arguments handling in hgext/remotefilelog/ Keys of kwargs on Python 3 should be strings. This patch fixes them by appending r'' prefixes, and using pycompat.byteskwargs() and pycompat.strkwargs(). Differential Revision: https://phab.mercurial-scm.org/D5259

py3: add __pycache__ to .hgignore Something changed with Python 3 just like million on things. Maybe they should named Python 3 as anaconda or cobra instead of just increasing the version number. This makes test-custom-filters.t pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5039

sparse-revlog: stop using a heap to track selected gap Same logic as for 'gapsheap', we don't actually need a heap.

sparse-revlog: stop using a heap to track gaps The heap doesn't bring any performance advantage as we can simply sort the final list. Moreover, the lesser complexity helps a lot when we later implement it in C.

sparse-revlog: fast-path before computing payload size In this fast-path case, we do not need to compute the full delta chain payload. Since it comes with a significant cost, we avoid doing so if possible.

sparse-revlog: drop unused deltainfo parameter from segmentspan We no longer need to deal with slicing logic including uncommitted revision. We drop the associated code.

sparse-revlog: drop unused deltainfo parameter from _slicechunktodensity We no longer need to deal with slicing logic including uncommitted revision. We drop the associated code.

sparse-revlog: skip the span check in the sparse-revlog case This significantly improves the performance on unbundling on smaller repositories. Mercurial: unbundling 1K revisions no-sparse-revlog: 500 ms sparse-revlog-before: 689 ms sparse-revlog-after: 484 ms Pypy: unbundling 1K revisions no-sparse-revlog: 1.242 s sparse-revlog-before: 1.135 s sparse-revlog-after: 0.860 s NetBeans: unbundling 1K revisions no-sparse-revlog: 1.386 s sparse-revlog-before: 2.368 s sparse-revlog-after: 1.191 s Mozilla: unbundling 1K revisions no-sparse-revlog: 3.103 s sparse-revlog-before: 3.367 s sparse-revlog-after: 3.093 s

histedit: import chistedit curses UI from hg-experimental I don't tend to like curses interfaces, but this gets enough use at work that it seems like it's worth bringing into core. This is a minimal import from hg-experimental revision 4c7f33bf5f00, in that I've done the smallest amount of code movement and editing in order to import the functionality. .. feature:: `hg histedit` will now present a curses UI if curses is available and `ui.interface` or `ui.interface.histedit` is set to `curses`. Differential Revision: https://phab.mercurial-scm.org/D5146

manifest: perform cheap checks before potentially allocating memory Differential Revision: https://phab.mercurial-scm.org/D5258

manifest: also reject obviously-too-short lines when parsing lines Differential Revision: https://phab.mercurial-scm.org/D5257

manifest: make sure there's a filename before bothering to look for newline There's no valid manifest that would have no characters before the NUL byte on a line, and this fixes some erratic timeouts in the fuzzer. Differential Revision: https://phab.mercurial-scm.org/D5256

revlog: replace PyInt_AS_LONG with a more portable helper function PyInt_AS_LONG disappears on Python, and our previous #define was producing some problems on Python 3. Let's give up and make an inline helper function that makes this more sane. Differential Revision: https://phab.mercurial-scm.org/D5235

help: document weird behavior of uisetup() While refactoring ui.log() API, it was really annoying that uisetup() is actually an extsetup() the phase 1. Let's document that. I'm planning to add another uisetup() which is called per ui instance, though I can't think of a good name for it.

help: merge section about uisetup() and extsetup() They are technically the same callback, called only once per process. The section name "ui setup" is confusing, so shouldn't be used.

help: add internals.extensions topic I think it's better to include the API overview in core as we now have the internals section in our help system. Retrieved from the wiki, and formatted as reST. Several wiki links are removed since they are invalid in the help. The sections about example extension are removed at all as they seemed too verbose. https://www.mercurial-scm.org/wiki/WritingExtensions

commandserver: send raw progress information to message channel This is pretty basic implementation to support GUI progress bar.

commandserver: make getpass() request distinct from normal prompt Otherwise, GUI clients would have to parse the prompt text.

commandserver: attach prompt default and choices to message These attributes are important to provide a GUI prompt to user.

test-commandserver: clean up quoting and location of dbgui extension This helps embedding '$' in the script.

ui: extract helpers to write message with type or label This provides a 'type' attribute to command-server clients, which seems more solid than relying on 'ui.<type>' labels. In future patches, type='progress' will be added to send raw progress information.

commandserver: add experimental option to use separate message channel This is loosely based on the idea of the TortoiseHg's pipeui extension, which attaches ui.label to message text so the command-server client can capture prompt text, for example. https://bitbucket.org/tortoisehg/thg/src/4.7.2/tortoisehg/util/pipeui.py I was thinking that this functionality could be generalized to templating, but changed mind as doing template stuff would be unnecessarily complex. It's merely a status message, a simple serialization option should suffice. Since this slightly changes the command-server protocol, it's gated by a config knob. If the config is enabled, and if it's supported by the server, "message-encoding: <name>" is advertised so the client can stop parsing 'o'/'e' channel data and read encoded messages from the 'm' channel. As we might add new message encodings in future releases, client can specify a list of encoding names in preferred order. This patch includes 'cbor' encoding as example. Perhaps, 'json' should be supported as well.

commandserver: fix reference before assignment error in pipeservice cleanup Spotted by the next patch.

dispatch: pass around ui.fmsg channel This will be set by the command server. See the next patch.

test-subrepo: document why share magically works even if subrepos are missing I was confused how it's working while reviewing fb490d798be0, "share: reload repo after adjusting it in postshare()."

packaging: add Fedora 29 target Differential Revision: https://phab.mercurial-scm.org/D5254

store: pass in decoded filename to narrow matcher

test-narrow-clone-stream: include no usefncache case which is broken Since 9aeb9e2d28a7, encoded filenames are filtered by the narrow matcher, which is clearly wrong.

test-narrow-clone-stream: include uppercase letter in filename This makes encoded filenames differ from the original names.

phase: use the `phases.cmdphasenames` constant to walk available command flags This reverts 1ea6772fb415.

phases: add a list of names usable by the phase command This is useful in TortoiseHg for example, which has until recently just added all names in the `phasenames` list to the context menu that changes the phase. The ones not supported by the command would just error out.

transaction: display data about why the transaction failed to rollback We saw more of these a while back. Having more data available would be nice.

merge with stable

remotefilelog: avoid accessing repo instance after dispatch Upstream commit c5e6c1ba1c79 (hg: don't reuse repo instance after unshare(), 2018-09-12) poisoned the repo instance after unshare(). That made `hg unshare` fail with remotefilelog because we tried to close the fileserverclient after dispatch by accessing it via the repo. This patch fixes that by storing the reference to the fileserverclient at the beginning of dispatch. An analogous patch was sent for remotefilelog version in FB's hg-experimental as D5246. Differential Revision: https://phab.mercurial-scm.org/D5253

share: avoid a now-unnecessary reload of the repo hg.share() now returns an already-reloaded repo, so let's just use that. (This would have failed test-subrepo-recursion.t without the previous patch.) Differential Revision: https://phab.mercurial-scm.org/D5252

share: reload repo after adjusting it in postshare() When sharing a repo that's using remotefilelog, the update that happens at the end of the `hg share` call does not see the remote repo path that's copied in hg.postshare(). This patch reloads the repo after hg.postshare() to address that. This changes a subrepo test case. Note that `hg share -U; hg co tip` worked there before, so I don't see see why `hg share` should fail. I also don't know what a "locally referenced subrepo". So maybe this is fixing a bug? Hopefully it's not breaking something someone actually cares about at least. Maybe someone who knows and cares about subrepos can review this. Differential Revision: https://phab.mercurial-scm.org/D5251

tests: don't load strip extension via mq extension for remotefilelog tests The proper way to get the `hg strip` command has been via the "strip" extension since 2013. Differential Revision: https://phab.mercurial-scm.org/D5250

remotefilelog: remove unused configs for {data,history}pack version Differential Revision: https://phab.mercurial-scm.org/D5249

tests: remove obsolete PYTHONPATH override from remotefilelog tests They were added in https://bitbucket.org/facebook/hg-experimental/commits/fdcad37a6a68cfbaf5920f3eeaa0cc9cae42bd26, which says this: When remotefilelog moved from its own repo, the tests needed to be updated to adjust the PYTHONPATH to ensure the in-repo remotefilelog was loaded instead of the system one. This meant any local runs of remotefilelog tests would've been using the system remotefilelog unless the user had manually set the PYTHONPATH themselves. That doesn't seem relevant with remotefilelog in core. Differential Revision: https://phab.mercurial-scm.org/D5248

tests: drop obsolete "hginit" alias in remotefilelog tests Differential Revision: https://phab.mercurial-scm.org/D5247

fix: add extra field to fixed revisions to avoid creating obsolescence cycles The extra field prevents sequential invocations of fix from producing the same hash twice. Previously, this could cause problems because it would create an obsolescence cycle instead of the expected new successor. This change also adds an explicit check for whether a new revision should be committed. Until now, the code relied on memctx.commit() to quietly do nothing if the node already exists. Because of the new extra field, this no longer covers the case where we don't want to replace an unchanged node. Differential Revision: https://phab.mercurial-scm.org/D5245

cleanup: use revision numbers instead of hashes in test output This allows changes to the hashes produced by fix to not needlessly modify this area of the test. Differential Revision: https://phab.mercurial-scm.org/D5244

tests: make the commands.resolve.confirm tests not pollute the test-wide hgrc I want to add tests after these tests that create a new repo and use it, and was confused for a bit as to why they were seeing different behavior than I expected. Differential Revision: https://phab.mercurial-scm.org/D5242

tests: fix a couple typos in test-resolve.t comments and add a comment Differential Revision: https://phab.mercurial-scm.org/D5241

fix: add suboption for configuring execution order of tools This could be accomplished by using wrapper scripts, but that would diminish the usefulness of the incremental formatting logic. Configuring execution order along with other things in the hgrc is probably more convenient anyway. This change highlights some awkwardness with suboptions and default values, which should be addressed separately. Differential Revision: https://phab.mercurial-scm.org/D5237

tests: fix up some import statements caught by Python 3 I'm curious how the import checker manages to be so much more pedantic in Python 3, but not enough to bother exploring. Differential Revision: https://phab.mercurial-scm.org/D5240