narrowcommands: remove an unrequired `repo.narrowpats` call We call that few lines above and do nothing significant in between which can change the narrowpats. So let's use values returned by that call. Differential Revision: https://phab.mercurial-scm.org/D5348

manifest: reject lines shorter than 42 bytes, not 22 Yuya correctly spotted during the review of f27f8e9ef1e73 that we're dealing with hexlified hashes here, and so it should be 42 bytes not 22. Differential Revision: https://phab.mercurial-scm.org/D5347

blackbox: initialize logger with repo instance The blackboxlogger is unusable without a repo. Let's simply initialize it with a repo instance.

blackbox: do not nullify repo to deactivate the logger on failure The _repo will be a mandatory attribute. Instead, make the logger to not track any events.

blackbox: extract global last logger to proxylogger class So the blackboxlogger can be instantiated with a repo.

ui: pass in bytes opts dict to logger.log() This is the convention of the Mercurial API.

ui: pass in formatted message to logger.log() This makes sure that all logger instances will handle the message arguments properly.

blackbox: send debug message to logger by core ui Since the core ui.log() may recurse into ui.log() through ui.debug(), it must guard against recursion. The ui extension class can finally be removed.

blackbox: change the way of deactivating the logger on write error This prepares for the upcoming code move. The recursion guard will be ported to the core ui.

match: remove obsolete catching of OverflowError Since 0f6a1bdf89fb (match: handle large regexes, 2007-08-19), we catch an OverflowError from the regex engine and split up the regex if that happens. In 59a9dc9562e2 (ignore: split up huge patterns, 2008-02-11), that was extended to raise an OverflowError in our code even if the regex engine doesn't raise it. It's unclear if there was a range of regex sizes where the OverflowError would be raised from the regex engine but that were still below the limit we added in our code. Either way, both limitations were probably removed in Python 2.7.4 when the regex code width was extended from 16bit to 32bit (or Py_UCS4) integer (thanks to Yuya for finding that out). If at least the first limitation was removed, we no longer should be using OverflowError for flow control, so this patch changes that. Differential Revision: https://phab.mercurial-scm.org/D5309

sparse: raise a move verbose index error from the C code If we don't like a value we should print it.

narrow: drop the bundle2 capability since we have server capabilities (BC) This patch drops the narrow bundle2 capabilities since we introduced narrow server capabilities which are more nice and now used everywhere. I am not sure what it can affect, so on safe side I marked this as BC. Also I removed the NARROWCAP constant as that kind of conflicts with the same name constant in wireprototypes.py. Differential Revision: https://phab.mercurial-scm.org/D4892

vfs: extract the audit path logic into a submethod This will make it possible to apply it in more cases.

subrepo-git: use an official origvfs when appropriate The origvfs has the auditor properly set and can move file without issue. The current code is currently working without errors because rename are not audited, yet.

revert: extract origvfs logic in a sub-function The subrepo's "revert" logic could benefit from it.

vfs: treat 'undo.' file the same as 'journal.' file They are the same kind of file, they are protected by the store lock, but directly lives inside the '.hg' directory. No warnings were ever raised about them because `vfs.rename` is not audited. Something we are trying to change.

perf: add a perfignore command The command is meant to benchmark operations related to hgignore. Right now the command is benchmarking the loading time of the hgignore rules.

py3: use pycompat.xrange instead of xrange xrange does not exists on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5302

store: write fncache only once if there are both adds and removes Differential Revision: https://phab.mercurial-scm.org/D5307

perf: disable revlogs clearing in `perftags` by default This aligns things with what `perfbookmarks` does. I decided to disable the revlogs clearing by default to focus on the core logic by default, ignoring side effects. If we prefer to emphasize the side effect, we can instead keep this on in `perftags` and enable it by default in `perfbookmarks`.

perf: add a `clear-revlogs` flag to `perfbookmarks` This flag (off by default) makes it possible to enable the refresh of the changelog and revlog. This is useful to check for costly side effects of bookmark loading. Usually, these side effects are shared with other logics (eg: tags). example output in my mercurial repo (with 1 bookmark, so not a great example): $ hg perfbookmarks ! wall 0.000044 $ hg perfbookmarks --clear-revlogs ! wall 0.001380

tags: cache `repo.changelog` access when checking tags nodes The tags reading process checks if the nodes referenced in tags exist. Caching the access to `repo.changelog` provides a large speedup for repositories with many tags. running `hg perftags` in a large private repository before: ! wall 0.393464 comb 0.390000 user 0.330000 sys 0.060000 (median of 25) after: ! wall 0.267711 comb 0.270000 user 0.210000 sys 0.060000 (median of 38)

perf: add a `clear-revlogs` flag to `perftags` This flag (on by default) makes it possible to disable the refresh of the changelog and revlog. This is useful to check for the time spent in the core tags logic without the associated side effects. Usually, these side effects are shared with other logics (eg: bookmarks). Example output in my Mercurial repository $ hg perftags ! wall 0.017919 comb 0.020000 user 0.020000 sys 0.000000 (best of 141) $ hg perftags --no-clear-revlogs ! wall 0.012982 comb 0.010000 user 0.010000 sys 0.000000 (best of 207)

perf: stop creating new revlog by hand in perftags It's better to let the repository logic create its own object. We now just clear the cache. New object will be automatically created from there.

revlog: update the documentation for `trim_endidx` The function role drifted since the function was commented.

revlog: properly detect corrupted revlog in `index_get_length` Pointed out by Yuya Nishihara.

perf: rename `perfhelper-tracecopies` to `perfhelper-pathcopies` The command it supports is called `perfpathcopies`. It seems better to align the names.

perf: add a docstring to `perfpathcopies` This will help people to find this command.

revlog: update the docstring of `ancestors` to match reality Code using this method expect the revision to be (reverse) sorted. As pointed by Yuya Nishihara, the docstring should reflect that.

remotefilelog: fix typo in docstring Differential Revision: https://phab.mercurial-scm.org/D5306

store: append to fncache if there are only new files to write Before this patch, if we have to add a new entry to fncache, we write the whole fncache again which slows things down on large fncache which have millions of entries. Addition of a new entry is common operation while pulling new files or commiting a new file. This patch adds a new fncache.addls set which keeps track of the additions happening and store them. When we write the fncache, we will just read the addls set and append those entries at the end of fncache. We make sure that the entries are new entries by loading the fncache and making sure entry does not exists there. In future if we can check if an entry is new without loading the fncache, that will speed up things more. Performance numbers for commiting a new file: mercurial repo before: 0.08784651756286621 after: 0.08474504947662354 mozilla-central before: 1.83314049243927 after: 1.7054164409637451 netbeans before: 0.7953150272369385 after: 0.7202838659286499 pypy before: 0.17805707454681396 after: 0.13431048393249512 In our internal repo, the performance improvement is in seconds. I have used octobus's ASV perf benchmark thing to get the above numbers. I also see some minute perf improvements related to creating a new commit without a new file, but I believe that's just some noise. Differential Revision: https://phab.mercurial-scm.org/D5301

py3: fix couple of division operator to do integer divison Differential Revision: https://phab.mercurial-scm.org/D5305

py3: use dict.items() instead of dict.iteritems() dict.iteritems() does not exist on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5304

py3: convert strings to bytes in tests/test-remotefilelog-histpack.py Differential Revision: https://phab.mercurial-scm.org/D5303

py3: replace str() with pycompat.bytestr() or ('%d' % int) After this patch, test-fastannotate-diffopts.t is about to pass. There are some extra newlines in the output. Differential Revision: https://phab.mercurial-scm.org/D5300

perf: add a `--timing` argument to `perfhelper-tracecopies` The new argument will help picking better pair for benchmarking. See documentation for details.

ui: manage logger instances and event filtering by core ui The setup code in blackbox needs more tweaks since it has lots of black magics. I'll fix them by follow-up patches. To be clear, the goal of this series is to provide a proper way for command server to install its own logger. I need it to debug in-memory repository cache.

extensions: add "uipopulate" hook, called per instance, not per process In short, this is the "reposetup" function for ui. It allows us to modify ui attributes without extending ui.__class__. Before, the only way to do that was to abuse the config dictionary, which is copied across ui instances. See the next patch for usage example.

hgweb: load globally-enabled extensions explicitly Before, extensions were loaded as a side effect of hg.repository() if the hgweb was executed as a CGI/WSGI. I want to make it explicit so that another ui hook can be inserted after extensions.loadall().

tests: sniff for libfuzzer actually being available in test-fuzz-targets.t When I upgraded the FreeBSD buildbot to 11.2 it seems we picked up clang6, but the default clang on FreeBSD doesn't include libfuzzer. I can't find a way to sniff for libfuzzer without running a compile, so here we are. Differential Revision: https://phab.mercurial-scm.org/D5270

tests: sniff for /usr/local/bin/gmake and use it in test-fuzz-targets.t This isn't as robust as it probably should be, but for now it'll get the job done on the buildbots. Differential Revision: https://phab.mercurial-scm.org/D5269

tests: stabilize test-inherit-mode.t on FreeBSD and macOS (issue6026) Symbolic links are funny permissions-wise, but on the linked issue Yuya has convinced me that we can ignore this permissions issue on macOS (FreeBSD allows setting permissions bits but ignores them) and we'll be in fine shape.

wireprotov2peer: wait for initial object before resolving future As part of rolling out wireprotov2 with redirect support, I encountered an edge case with regards to future resolution. Essentially, the initial response frame from the server did not fully decode the initial CBOR object. The frame wasn't marked as EOS. In the previous code, we resolved the future for the request to response.objects(), which mapped to the commandresponse instance which would eventually produce a redirect. Upon receiving subsequent data, the initial CBOR object containing the redirect would be decoded and we'd process the redirect. However, the future would already have been resolved with the initial commandresponse.objects() and the client iterating over the objects wouldn't receive any objects from the redirect because the redirect was populating a different commandresponse instance! This commit changes the logic so we don't resolve futures until the initial CBOR response object is fully decoded or until EOS occurs. In cases where there is an empty or partial frame associated with a redirect, the future will now resolve with the commandresponse containing the proper series of decoded objects.

wireprotov2peer: always return a bool from _processredirect() Without this, we may stop servicing the redirect response if the future has already been resolved. And the future will often be resolved very early, since many consumers iterate the decoded CBOR object stream and expect data to lazily arrive.

tests: disable remotefilelog on Windows I've spent a non trivial amount of time trying to eliminate the test errors, but it's looking like this is pretty dependent on Unix support. For example, there are attempts to delete open files, and uses of threads that report I/O attempts on closed files. (Maybe this is a race condition? Don't we usually use processes as workers on Windows?) In any event, I don't want real new errors elsewhere to be masked by these known problems. For some reason $CACHEDIR is reported as missing in test-remotefilelog-repack.t, but it actually exists in the hgcloneshallow call inside shallowutil.mkstickygroupdir(). By the time the process exits, it's gone. I don't see it being removed by code that calls 'rmdir' or 'remove' in the extension itself.

perf: run 'setup' function during stub run The benchmarked function might need the content of the setup to be run in order to function properly.

perf: fallback to subset if ondisk cache is missing in perfbranchmapload If there is no branchmap on disk for that filter, it means that the cache from some subset's filter is relevant for this one. We look for it instead of aborting. That way it is much simpler to run the command in an automated way. We can now add it to `test-contrib-perf.t`.

perf: prewarm the branchmap in perfbranchmapload It is not very interesting to have the command randomly failing because the branchmap for the tested filter happens to be cold. So we make sure to have a valid up to date branchmap before going further. The data might still be missing from disk if a subset was equivalent. See next changeset for details and fix.

localrepo: correct docstring of filectx() The same reason as b6c2543e1dd8. It can't be any changeset specifiers but revision number.

annotate: remove dead code to not convert path to relative path It's annotate. There should be at least one file path specified.

annotate: start pager soon after command options are validated It helps extracting helper class.

annotate: check and update command options first It helps extracting helper class.

annotate: format local tables in less-dense form I'll move these to a class, where one-extra indent will be needed.

perf: rename version flag of perfchangegroupchangelog to cgversion The version flag conflicts with the global version flag and does not work. Differential Revision: https://phab.mercurial-scm.org/D5297 Differential Revision: https://phab.mercurial-scm.org/D5298

procutil: don't allow the main 'hg' script to be treated as the Windows exe Previously, there were a handful of errors like this: $ hg prefetch --repack (running background incremental repack) + abort: %1 is not a valid Win32 application + [255] CreateProcess() doesn't append .exe when `lpApplicationName` contains a path, and a python script isn't directly executable.

remotefilelog: drop some compatibility cruft for finding the hg exeutable

branchmap: build the revbranchcache._namesreverse() only when required On big repositories with a lot of named branches and that also increasing over time, building of this dict can be expensive and shows up in profile. For our internal repository, this saves ~0.05 seconds. Thanks to Yuya for suggesting using util.propertycache() and util.clearcachedproperty(). Differential Revision: https://phab.mercurial-scm.org/D5291

sparse-revlog: put the native implementation of slicechunktodensity to use When possible, the C implementation of delta chain slicing will be used. providing a large boost in performance for this operation. To take a practical example of restoring manifest revision '59547c40bc4c' for a reference NetBeans repository (using sparse-revlog). The media time of the step `slice-sparse-chain` of `perfrevlogrevision` improve from 0.660 ms to 0.098 ms; The full series move delta chain slicing from 1.120 ms to 0.098 ms; Implementing _slicechunktosize into C would yield further improvements. However, the performance seems good enough for now.

cext: increment the `parsers` module version We introduced a new `slicechunktodensity` method to the index.

sparse-revlog: introduce native (C) implementation of slicechunktodensity This is a C implementation of `_slicechunktodensity` in the `mercurial/revlogutils/deltas.py` file. The algorithm involves a lot of integer manipulation and low-level access to index data. Having a C implementation of it raises a large performance improvement. See later changeset in this series for details.

sparse-revlog: add a `trim_endidx` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This function implement a subpart of the python function `_trimchunk` in `mercurial/revlogutils/deltas.py`. Handling of actual Python objects is left to the caller function.

sparse-revlog: add a `index_segment_span` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

sparse-revlog: add a `index_get_length` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

sparse-revlog: add a `index_get_start` function in C We are about to implement a native version of `slicechunktodensity`. For clarity, we introduce the helper functions first. This new function provides an efficient way to retrieve some of the information needed by `slicechunktodensity`.

perf: add --clear-revlog flag to branchmapload Having the changelog index already loaded when loading the branchmap can have a large impact on performance. Example runs (large private repository): hg perfbranchmapload -f base ! wall 0.116722 comb 0.120000 user 0.110000 sys 0.010000 (best of 59) hg perfbranchmapload -f base --clear-revlogs ! wall 0.258246 comb 0.230000 user 0.220000 sys 0.010000 (best of 31)

perf: introduce a function to fully "unload" a changelog The function remove various attributes and caches related to changelog. This is getting a common requirement.

perf: use an explicit function in perfbranchmapload This make things clearer.

perf: update function name to match `perfbranchmapload` command Having function with the same name as the command is simpler.

perf: fix a minor typo in perfbranchmapload

context: floor adjustlinkrev graph walk during copy tracing The `_adjustlinkrev` method gains an optional "stoprev" argument. The linkrev adjustment will give up once this floor is reached. The relevant functions using `_adjustlinkrev` are updated to pass an appropriate value in the copy tracing code. In some private repository, about 10% of the status call triggered the pathological case addressed by this change. The speedup varies from one call to another, the best-observed win is moving from 170s to 11s. The effect of this change can be seen in the public pypy repository, running the following command: hg perftracecopies --source 83c9ff0c0206 --destination 59c79103d5b0 before: 3.401753 seconds after: 2.634897 seconds (-23%)

context: small refactoring of `isintroducedafter` This make the next change clearer.

context: split `introrev` logic in a sub function We want to add a mechanism to stop iteration early associated to intro rev early in some case. However, it does not make sense to expose it in the public `filectx` API. So we split the code into an internal method instead.

context: introduce an `isintroducedafter` method and use it in copies Right now, copy tracing make effort to not traverse the graph too much to save performance. It uses a "limit" acting as a floor revision past which data are no longer relevant to the current copy tracing. However, to enforce this limit, it does a call to `filectx.rev()` and that call can trigger a graph traversal on its own. That extra graph traversal is unaware of the current limit and can become very expensive. That cost is increased by the nature of work done in adjust link rev, we are not only walking down the graph, we are also checking the affected file for each revision we walk through. Something significantly more expensive than the walk itself. To work around this we need to make the `filectx` operation aware of the current limit. The first step is to introduce a dedicated method: `isintroducedafter`. We'll then rework that method logic to stop traversal as soon as possible.

context: take advantage of `_descendantrev` in introrev if available Before this changeset, `_descendantrev` was ignored and `introrev` could return a "wrong" result. I was previously fine because there seems to be no existing code using both `introrev` and `_descendantrev` at the same time. However, we would like to change that.

context: spell out the logic around linkrev adjustement starting point We make the intent of the `_changeid` and `_changectx` checks explicit. The same logic was previously performed by the `self.rev()` call. The new code is a bit redundant, but much clearer.

perf: add a new `perfhelper-tracecopies` command The command is not measuring performance itself, it digs interesting statistic to help pick good arguments for the `perfcopytrace` command.

narrow: extract helper for parsing narrowspec file This will be used for parsing a file that's stored in a different location. Differential Revision: https://phab.mercurial-scm.org/D5277

py3: cast attribute name to sysstr in clearcachedproperty()

py3: whitelist three more passing test caught by buildbot The number of failing test is back to two digits. Differential Revision: https://phab.mercurial-scm.org/D5293

tests: make test-check-module-imports more robust It failed for me without this in this way: tests/test-commandserver.t:19: relative import of stdlib module tests/test-lfs-serve.t:108: relative import of stdlib module tests/test-lfs-serve.t:255: relative import of stdlib module tests/test-lfs-serve.t:362: relative import of stdlib module tests/test-lfs-serve.t:406: relative import of stdlib module tests/test-lock.py:9: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-lrucachedict.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-match.py:5: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-remotefilelog-datapack.py:15: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-remotefilelog-histpack.py:14: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-simplekeyvaluefile.py:4: imports not lexically sorted: silenttestrunner < unittest, True, True tests/test-sshserver.py:6: imports not lexically sorted: silenttestrunner < unittest, True, True This is because every module is considered a stdlib module, because the stdlib_prefixes is /usr, and my repo is in /usr/local/home, which means that sys.path contains a couple of /usr/local/home/../hg/.. entries that count as "in the stdlib". Fix this by preventing any path in sys.path that's inside the mercurial source from being considered "in the stdlib". Differential Revision: https://phab.mercurial-scm.org/D5294

push: add --publish flag to change phase of pushed changesets This flag is copied from topic extension, where it proved to be useful. It makes all pushed changesets public on both ends if the push succeeds. Doesn't currently work for any subrepos that are implicitly pushed together with their owner repo.

match: reformat `syntaxes` dictionary for better maintainability

match: provide and use a quick way to escape a single byte The previous function has a lot of overhead (including being a function). In the `_globre` case, we always escape a single byte. So we provide a dictionary dedicated to this use case. We directly use the dictionary to avoid a function call, these are expensive in Python. Again, this raise a very significant performance gain: Before: ! wall 0.059793 comb 0.060000 user 0.060000 sys 0.000000 (median of 100) After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146) Total improvement for the full series: Before: ! wall 0.153153 comb 0.150000 user 0.150000 sys 0.000000 (median of 66) After: ! wall 0.020390 comb 0.020000 user 0.020000 sys 0.000000 (median of 146)

perf: make `clearfilecache` helper work with any object If the object is not a localrepo, it won't have an `unfiltered` method (and won't need one).

perf: move some of the perftags benchmark to the setup function Creating fresh objects and clearing the cache should not be part of the timing. before: ! wall 0.020851 comb 0.020000 user 0.020000 sys 0.000000 (median of 138) after: ! wall 0.018740 comb 0.020000 user 0.020000 sys 0.000000 (median of 141)

perf: use the new setup function in "perfbookmarks" This command was picked arbitrarily to display the usefulness of the new feature. In my Mercurial repository (with very few bookmarks), moving cache cleanup in the dedicated setup function has a visible and stable effect on the benchmark number. before: ! wall 0.000061 comb 0.000000 user 0.000000 sys 0.000000 (median of 40837) after: ! wall 0.000058 comb 0.000000 user 0.000000 sys 0.000000 (median of 40500)

perf: add a `setup` argument to run code outside of the timed section With this new argument, it is possible to perform special setup and cleanup outside of code actually timed. This is useful to provide more accurate benchmark.

perf: explicitly pass title as a keyword argument in `perfdiffwd` This will help to update the timer function arguments in a later changeset.

ui: unify argument name of ui.log() It's called an "event" in both blackbox and logtoprocess.

logtoprocess: extract logger class from ui wrapper It provides the same interface as the blackboxlogger. The ui wrapper will be removed shortly.

merge with stable

py3: use node.hex(m.digest()) instead of m.hexdigest() hashlib.sha1.hexdigest() returns str on Python 3. Differential Revision: https://phab.mercurial-scm.org/D5287

tests: stabilize the recent checkexec changes on Windows This goes with bd0874977a5e.

tests: add Balto configuration file I have been developing a Mercurial test runner plugin for being able to run Mercurial tests with Balto (https://bitbucket.org/lothiraldan/balto/src/default/). Balto requires a configuration file so let's include it, I have added the required information in the configuration file as comments. I hope Balto would be an helpful tool for other people than me. Differential Revision: https://phab.mercurial-scm.org/D5283

tests: make test-alias.t pass with re2 Locally, these "non-ASCII character in alias" errors don't show up, though I get them when the alias is defined at the command line rather than in an hgrc. The brokenness comes from the fact that hgrcs are parsed with regexes, and re/re2 differ in this way: $ python -c 'import re; print(re.compile("(.*)").match("aaa\xc0bbbb").groups())' ('aaa\xc0bbbb',) $ python -c 'import re2; print(re2.compile("(.*)").match("aaa\xc0bbbb").groups())' ('aaa',) Apparently re2 stops when it encounters invalid utf8 (which I suppose makes sense given that '.' matches what appears to be a codepoint rather than a byte). This is presumably a bug in hg, but not very important, so just change the test to stick to valid utf8. Differential Revision: https://phab.mercurial-scm.org/D5288

context: remove seemingly impossible code branch I'm not a Python expert, but I can't think of a way that the following branch can ever be hit: def _changeid(self): if r'_changeid' in self.__dict__: return self._changeid It seems to me that if that condition is true, then this function would not have been called. The only exception I can think of is if a reference to the function had been stored beforehand, something like this: c = fctx.__dict__['_changeid'] fctx._changeid c() But that seems like very unlikely code to exist. The condition was added in 921b64e1f7b9 (filecontext: use 'is not None' to check for filelog existence, 2013-05-01) as a "bonus" change (in addition to what the patch was actually about) Differential Revision: https://phab.mercurial-scm.org/D5289