win32: split a utility function to obtain the volume out of getfstype() This is only done on Windows because it's simple enough to call statfs() on Unix. The goal is to display this in `hg debugfs`.

util: whitelist NTFS for hardlink creation (issue4580)

win32: implement util.getfstype() This will allow NTFS to be added to the hardlink whitelist, and resume creating hardlinks in transactions (which was disabled globally in 07a92bbd02e5; see also e5ce49a30146). I opted to report "cifs" for remote volumes because this shows in `hg debugfs`, which also reports that hardlinks are supported for these volumes. So being able to distinguish it from "unknown" seems useful. The documentation [1] seems to indicate that SMB isn't supported by these functions, but experimenting shows that mapped drives are reported as "NTFS" on Windows 7. I don't have a second Windows machine, but instead shared a temp directory on C:\. In this setup, both of the following were detected as 'cifs' with the explicit GetDriveType() check: Z:\repo>hg ci -A C:\>hg -R \\hostname\temp\repo ci -A # (without Z:\ being mapped) It looks like this is called 6 times to add and commit a single new file, so I'm a little surprised this isn't cached. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa364993(v=vs.85).aspx

util: move getfstype() to the platform modules This makes room for implementing on Windows using ctypes.

graphlog: add another graph node type, unstable, using character "*" (BC)

largefiles: modernize how capabilities are added to the wire protocol See 982f13bef503, which came well after this code was originally written.

lfs: show a friendly message when pushing lfs to a server without lfs enabled Upfront disclaimer: I don't know anything about the wire protocol, and this was pretty much cargo-culted from largefiles, and then clonebundles, since it seems more modern. I was surprised that exchange.push() will ensure all of the proper requirements when exchanging between two local repos, but doesn't care when one is remote. All this new capability marker does is inform the client that the extension is enabled remotely. It may or may not contain commits with external blobs. Open issues: - largefiles uses 'largefiles=serve' for its capability. Someday I hope to be able to push lfs blobs to an `hg serve` instance. That will probably require a distinct capability. Should it change to '=serve' then? Or just add an 'lfs-serve' capability then? - The flip side of this is more complicated. It looks like largefiles adds an 'lheads' command for the client to signal to the server that the extension is loaded. That is then converted to 'heads' and sent through the normal wire protocol plumbing. A client using the 'heads' command directly is kicked out with a message indicating that the largefiles extension must be loaded. We could do similar with 'lfsheads', but then a repo with both largefiles and lfs blobs can't be pushed over the wire. Hopefully somebody with more wire protocol experience can think of something else. I see 'x-hgarg-1' on some commands in the tests, but not on heads, and didn't dig any further.

lfs: allow non-lfs exchanges when the extension is only enabled on one side Once the 'lfs' requirement is added, the extension must be loaded on both sides, and changegroup3 used. But there's no reason that I can see for bailing with cryptic errors if lfs is not required, but randomly enabled somewhere.

lfs: add the 'lfs' requirement in the changegroup transaction introducing lfs A hook like this is how largefiles manages to do the same. Largefiles uses a changegroup hook, but this uses pretxnchangegroup because that actually causes the transaction to rollback in the unlikely event that writing the requirements out fails. Sadly, the requires file itself isn't rolled back if a subsequent hook fails, but that seems trivial. Now that commit, changegroup and convert are covered, I don't think there's any way to get an lfs repo without the requirement. The grep exit code is blotted out of some test-lfs-serve.t tests now showing the requirement, because run-tests.py doesn't support conditionalizing the exit code.

test-lfs: add tests covering http exchanges This tries to test every combination of having the extension enabled/disabled on each side, and then push/pull/clone/identify lfs and non-lfs content. SSH is ignored here, because there's enough going on as it is. The root issue here is again that requirements are not exchanged and preserved on push/pull/clone. Doing so should eliminate the cryptic error messages when using `hg serve`. The 500 server error is triggered by "ValueError: no common changegroup version", because the extension forces changegroup3. Or, if changegroup3 is enabled manually, it is triggered by "abort: missing processor for flag '0x2000'!". Sadly, run-tests.py doesn't support conditionalizing the exit code like it does lines of output. Therefore, a couple of tests blot out the exit code by appending "|| true", since these failures will go away shortly anyway.

test-lfs: add tests covering local exchanges The root issue here is that requirements are not exchanged and preserved on push/pull. This can be handled with a changegroup hook. Testing for remote exchanges is much more extensive (it's possible for one process or the other to not have the extension loaded at all), so it is added separately.

smartset: split generatorset classes to avoid cycle I uncovered a cycle manifesting in a memory leak by running `hgperfrevset '::tip'`. The cycle was due to generatorset.__init__ assigning a bound method to self.__contains__. Internet sleuthing revealed that assigning a bound method to an instance attribute always creates a cycle. This commit creates two new variants of generatorset for the special cases of ascending and descending generators. The special implementations of __contains__ have been extracted to these classes where they are defined as __contains__. generatorset now implements __new__ and changes the spawned type to one of the new classes if needed. Differential Revision: https://phab.mercurial-scm.org/D1780

merge: raise before running mergedriver if using IMM Merge driver scripts run in the working copy, so disable with IMM for now. Differential Revision: https://phab.mercurial-scm.org/D1781

scmutil: use a tuple of possible values instead of using startswith() This patch also adds a review comment which is helpful as inline comment. Differential Revision: https://phab.mercurial-scm.org/D1761

show: use revlog function to compute length of the longest shortest node As the core part of shortest() was extracted at 448725a2ef73, we no logner need a templater.

commands: use the new API to access hidden changesets in various commands In previous patches, we have added an internal API to unhide hidden changesets. This patch makes the following command use that api in nowarn mode i.e. there will be no warning while accessing hidden changesets. cat, diff, export, files, heads, identify, log, manifest, parents, status This patch also adds test demonstarting the behaviour. .. feature:: Accessing hidden changesets Set config option 'experimental.directaccess = True' to access hidden changesets from read only commands. Differential Revision: https://phab.mercurial-scm.org/D1735

scmutil: add utility fn to return repo object with user passed revs unhidden There has been a need for accessing hidden changesets by default without passing --hidden. This is currently done using the directaccess extension but is bit hacky. This patch adds a utility function to return a repo object having user passed revisions unhidden. This functionality will live behind a config option and won't be the default behaviour. There is also a config option added by this patch which tells whether we want to unhide only those revisions whose hashes are passed or should we consider revisions numbers also. Differential Revision: https://phab.mercurial-scm.org/D1733

repoview: add a new filtername for accessing hidden commits This patch adds a new filter 'visible-hidden' for repository and will be used to return a repo object with user passed revisions unhidden. Unlike the directaccess extension in fb-hgext and previous series adding the functionality, this time we introduce only one new filter as whether to warn user or not is handled by scmutil.unhidehashlikerevs(). Differential Revision: https://phab.mercurial-scm.org/D1734

revsetlang: add utility function to return hash like symbols from the tree Functionalities like unhiding changesets whose rev/hash is passed by the user required the knowledge of rev/hashes in the user provided specs. This patch adds functions which can parse tree object and return a list of such values. Differential Revision: https://phab.mercurial-scm.org/D1732

repoview: add visibilityexception argument to filterrevs() and related fns After this patch, filterrevs() can take an optional argument visibilityexceptions which is a set of revs which should be exception to being hidden. The visibilityexceptions will be passed to the function computing hidden revisions for that filtername and are considered there while calculating the set of hidden revs. Differential Revision: https://phab.mercurial-scm.org/D1747

repoview: add visibilityexceptions as an optional argument to repo.filtered() This will help us in having an API where we can pass the filtername and the visibilityexceptions to get a new repo object. Visibility exceptions are the revs which must be visible even they should in theory belong to the hidden set. They are required as there has been desire to have a functionality to access hidden changesets using certain commands without passing --hidden. After this patch we can make those changesets visibility exceptions so that we can access them without requiring a unfiltered repo. Differential Revision: https://phab.mercurial-scm.org/D1746

streamclone: move wire protocol status code from wireproto command This consolidates the code for the streaming clone wire protocol format into streamclone.py. It also eliminates a generator wrapper, which might make streaming clones slightly faster. Differential Revision: https://phab.mercurial-scm.org/D1754

histedit: add ui.log for action count Differential Revision: https://phab.mercurial-scm.org/D1751

run-tests: extract sorting of tests to own function TestRunner._run() is a large function and is difficult to follow. Let's extract the test sorting to its own function to make it shorter. When I refactored run-tests.py several years ago, I put a lot of functionality in methods. The prevailing Mercurial style is to use functions - not classes - where possible. While refactoring the code, I decided to undo this historical mistake of mine by moving the code to a standalone function. Differential Revision: https://phab.mercurial-scm.org/D1750

run-tests: remove dead code related to temp directory Differential Revision: https://phab.mercurial-scm.org/D1749

rebase: switch ui.log calls to common style The old style raised errors in some cases. Differential Revision: https://phab.mercurial-scm.org/D1748

hgweb: link to successors of obsoleted changesets _siblings() prepare various useful properties to use in templates. This function usually prepares parents and children of changesets for use in hgweb templates, but it can be used for successors too. It's needed because item['successors'] is a _hybrid object that works well when used in regular templates, but in hgweb templates work slightly differently and can't get hex nodes of the successors, which are required for these links to work.

hgweb: display fate of obsolete changesets Operations that obsolete changesets store enough metadata to explain what happened after the fact. One way to get that metadata is showsuccsandmarkers function, which returns a list of successors of a particular changeset and appropriate obsolescence markers. Templates have a set of experimental functions that have names starting with obsfate. This patch uses some of these functions to interpret output of succsandmarkers() and produce human-friendly messages that describe what happened to an obsolete changeset, e.g. "pruned" or "rewritten as 6:3de5eca88c00". In commonentry(), succsandmarkers property is made callable so it's only executed on demand; this saves time when changeset is not obsolete, and also in e.g. /shortlog view, where there are a lot of changesets, but we don't need to show each and every one in detail. In spartan theme, succsandmarkers is used instead of the simple "obsolete: yes", in other themes a new line is added to /rev page.

test-ssh: stabilize for Windows Previously, this complained: remote: '.' is not recognized as an internal or external command, remote: operable program or batch file. Making this a python script apparently revealed some races[1]. Thanks to Yuya for suggesting this. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-December/109094.html

templater: register keywords to defaults table Since the keywords are permanent, there should be no need to pass them by a temporary mapping.

templater: drop unneeded resources from conflict-marker data Follow-up for 32c278eb876f and f1c54d003327.

templater: rewrite docstring of templater.__init__() More importantly, this patch adds what the cache is.

rebase: don't take out a dirstate guard for in-memory rebase Since IMM doesn't affect the dirstate, it's not needed, and might be faster. Differential Revision: https://phab.mercurial-scm.org/D1745

test-split: stabilize for Windows - $PYTHON needs to be quoted when used as an executable in $HGEDITOR. This avoids the error "'c' is not recognized as an internal or external command". - seq.py is printing out CRLF, and then the subsequent `sed` script seems to convert to LF on MSYS. IDK if python print statements can be made to print LF on Windows, and I'm pretty sure CRLF is baked into some other tests. - A stray glob was causing the 'obsstore-off' case to report 'no result code from test'. - When I ran with --debug, the `hg diff` commands in the test both printed color sequences, and paused the output as it was run through the pager.

lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Even though the upload/download message is still in a ui.verbose check, I switched that to ui.note() too so that the 'ui.note' label is applied. The debug message is no longer marked for translation because check-code complained.

lfs: only hardlink between the usercache and local store if the blob verifies This fixes the issue where verify (and other read commands) would propagate corrupt blobs. I originalled coded this to only hardlink if 'verify=True' for store.read(), but then good blobs weren't being linked, and this broke a bunch of tests. (The blob in repo5 that is being corrupted seems to be linked into repo5 in the loop running dumpflog.py prior to it being corrupted, but only if verify=False is handled too.) It's probably better to do a one time extra verification in order to create these files, so that the repo can be copied to a removable drive. Adding the same check to store.write() was only for completeness, but also needs to do a one time extra verification to avoid breaking tests.

lfs: verify lfs object content when transferring to and from the remote store This avoids inserting corrupt files into the usercache, and local and remote stores. One down side is that the bad file won't be available locally for forensic purposes after a remote download. I'm thinking about adding an 'incoming' directory to the local lfs store to handle the download, and then move it to the 'objects' directory after it passes verification. That would have the additional benefit of not concatenating each transfer chunk in memory until the full file is transferred. Verification isn't needed when the data is passed back through the revlog interface or when the oid was just calculated, but otherwise it is on by default. The additional overhead should be well worth avoiding problems with file based remote stores, or buggy lfs servers. Having two different verify functions is a little sad, but the full data of the blob is mostly passed around in memory, because that's what the revlog interface wants. The upload function, however, chunks up the data. It would be ideal if that was how the content is always handled, but that's probably a huge project. I don't really like printing the long hash, but `hg debugdata` isn't a public interface, and is the only way to get it. The filelog and revision info is nowhere near this area, so recommending `hg verify` is the easiest thing to do.

lfs: narrow the exceptions that trigger a transfer retry The retries were added to workaround TCP RESETs in fb-experimental fc8c131314a9. I have no idea if that's been debugged yet, but this wide net caught local I/O errors, bad hostnames and other things that shouldn't be retried. The next patch will validate objects as they are uploaded, and there's no need to retry those errors. The spec[1] does mention that certain http errors can be retried, including 500. But let's work through the corruption detection issues first. [1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md

test-lfs: add tests around corrupted lfs objects These are mostly tests against file:// based remote stores, because that's what we have the most control over. The test uploading a corrupt blob to lfs-test-server demonstrates an overly broad exception handler in the retry loop. A corrupt blob is actually transferred in a download, but eventually caught when it is accessed (only after it leaves the corrupt file in a couple places locally). I don't think we want to trust random 3rd party implementations, and this would be a problem if there were a `debuglfsdownload` command that simply cached the files. And given the cryptic errors, we should probably validate the file hash locally before uploading, and also after downloading.

lfs: add note messages indicating what store holds the lfs blob The following corruption related patches were written prior to adding the user level cache, and it took awhile to track down why the tests changed. (It generally made things more resilient.) But I think this will be useful to the end user as well. I didn't make it --debug level, because there can be a ton of info coming out of clone/push/pull --debug. The pointers are sorted for test stability. I opted for ui.note() instead of checking ui.verbose and then using ui.write() for convenience, but I see most of this extension does the latter. I have no idea what the preferred form is.

tests: teach `f` to handle sha256 checksums

tests: fix a bug in `f` that prevented calculating sha1sum on a file

templater: look up symbols/resources as if they were separated (issue5699) It wouldn't be easy to split the mapping dict into (symbols, resources). This patch instead rejects invalid lookup taking resources.keys() as source of truth. The doctest is updated since mapping['repo'] is now reserved for a repo object.

templater: move repo, ui and cache to per-engine resources

templater: keep default resources per template engine (API) This allows us to register a repo object as a resource in hgweb template, without loosing '{repo}' symbol: symbol('repo') -> mapping['repo'] (n/a) -> defaults['repo'] resource('repo') -> mapping['repo'] (n/a) -> resources['repo'] I'm thinking of redesigning the templatekw API to take (context, mapping) in place of **(context._resources + mapping), but that will be a big change and not implemented yet. props['templ'] is ported to the resources dict as an example. .. api:: mapping does not contain all template resources. use context.resource() in template functions.

templater: look up mapping table through template engine These functions are stub for symbol/resource separation. This series is intended to address the following problems: a) internal data may be exposed to user (issue5699) b) defaults['repo'] (a repository name) will conflict with mapping['repo'] (a repo object) in hgweb

debug: add newlines at the end of three locations that appear to need it Differential Revision: https://phab.mercurial-scm.org/D1720

debug: remove an 'if ui.debug()' that is not doing anything ui.debug() does not return a value. Differential Revision: https://phab.mercurial-scm.org/D1719

paper: minor adjustments to table styles Adding a bit of padding to table columns on e.g. /log means content and headers are better aligned: headers already have this padding. Right margin is removed from #changesetEntry th because elements with display: table-cell (such as <th>) ignore margins anyway.

filemerge: only raise InMemoryMergeConflictsError when running _xmerge The old code here was overly broad and would raise in cases when we didn't end up calling `xmerge` and resolved using an internal tool (such as when `premerge=True`). Instead, let's swap out _xmerge if IMM is enabled and have the new tool raise when called, which is the behavior we want. Differential Revision: https://phab.mercurial-scm.org/D1739

journal: use pager journal output is long and should use a pager. Differential Revision: https://phab.mercurial-scm.org/D1740

commandserver: unblock SIGCHLD This enables the SIGCHLD handler to work properly if some buggy program started chg server with SIGCHLD blocked. A test of this probably requires C code, but we don't have such kind of tests already. Since this is a simple and clear fix, I'm leaving it as "untested" but I did a manual test and there were no longer zombie workers. Differential Revision: https://phab.mercurial-scm.org/D1737

osutil: add a function to unblock signals Signals could be blocked by something like: #include <unistd.h> #include <signal.h> int main(int argc, char * const argv[]) { sigset_t set; sigfillset(&set); sigprocmask(SIG_BLOCK, &set, NULL); execv("/bin/hg", argv); return 0; } One of the problems is if SIGCHLD is blocked, chgserver would not reap zombie workers since it depends on SIGCHLD handler entirely. While it's the parent process to blame but it seems a good idea to just unblock the signal from hg. FWIW git does that for SIGPIPE already [1]. Unfortunately Python 2 does not reset or provide APIs to change signal masks. Therefore let's add one in osutil. Note: Python 3.3 introduced `signal.pthread_sigmask` which solves the problem. `sigprocmask` is part of POSIX [2] so there is no feature testing in `setup.py`. [1]: https://github.com/git/git/commit/7559a1be8a0afb10df41d25e4cf4c5285a5faef1 [2]: http://pubs.opengroup.org/onlinepubs/7908799/xsh/sigprocmask.html Differential Revision: https://phab.mercurial-scm.org/D1736

sshpeer: move docstring to top Also makes it use double quotes consistently.

log: make "slowpath" condition slightly more readable Before 8e0e334bad42 and 6c76c42a5893, the condition was "anypats() or (files() and --removed)". This can be read as "<match is actually slow> or <walk files including removed revs>". So "not always()" (i.e. walk file revs) seems more appropriate here. The logic should be unchanged: not anypats() => always() or isexact() or prefix() isexact() => not always() prefix() => not always()

completion: add support for new "amend" command The command is now shipped with Mercurial, but completion should be helpful (and accurate) for users of the amend command shipped with the evolve extension too. Differential Revision: https://phab.mercurial-scm.org/D1716

completion: don't suggest clean files to revert It looks like we used to suggest only modified, added, removed and deleted files to revert until a821ec835223 (completion: selectively use debugpathcomplete in bash_completion, 2013-03-21). The reasoning in that commit was that getting the status was too slow and the replacement (debugpathcomplete) seems to make sense for the other two commands (remove and forget), but I'm not sure it was intentional to change the behavior of completion for revert. Note that "add" and "diff" already use status-based completion. Differential Revision: https://phab.mercurial-scm.org/D1715

split: new extension to split changesets This diff introduces an experimental split extension to split changesets. The implementation is largely inspired by Laurent Charignon's implementation for mutable-history (changeset 9603aa1ecdfd54b0d86e262318a72e0a2ffeb6cc [1]) This version contains various improvements: - Rebase by default. This is more friendly for new users. Split won't lead to merge conflicts so a rebase won't give the user more trouble. This has been on by default at Facebook for months now and seems to be a good UX improvement. The rebase skips obsoleted or orphaned changesets, which can avoid issues like allowdivergence, merge conflicts, etc. This is more flexible because the user can decide what to do next (see the last test case in test-split.t) - Remove "Done split? [y/n]" prompt. That could be detected by checking `repo.status()` instead. - Works with obsstore disabled. Without obsstore, split uses strip to clean up old nodes, and it can even handle split a non-head changeset with "allowunstable" disabled, since it runs a rebase to solve the "unstable" issue in a same transaction. - More friendly editor text. Put what has been already split into the editor text so users won't lost track about where they are. [1]: https://bitbucket.org/marmoute/mutable-history/commits/9603aa1ecdfd54b Differential Revision: https://phab.mercurial-scm.org/D1082

merge with stable

worker: handle interrupt on windows After applying suggestions from https://phab.mercurial-scm.org/D1564 to catch all exceptions in the same way I actually broke the handling of KeyboardInterrupt on windows. The reason is that KeyboardInterrupt doesn't dervie from Exception, but BaseException: https://docs.python.org/2/library/exceptions.html starting from python 2.5 Test Plan: Run hg on windows and ctrl-c during a large update. No random exceptions from threads surface in the shell. Previously we'd nearly always get stack traces from some of threads Run tests ./run-tests.py [...] Failed test-convert-svn-encoding.t: output changed # Ran 622 tests, 41 skipped, 1 failed. python hash seed: 2962682116 The test failing seems to have nothing to do with the change and fails on base revision as well Differential Revision: https://phab.mercurial-scm.org/D1718

lfs: fix committing deleted files caused by e0a1b9ee93cd e0a1b9ee93cd (lfs: add a repo requirement for this extension once an lfs file is committed) introduced a regression that prevents committing file deletion. This patch fixes that. Differential Revision: https://phab.mercurial-scm.org/D1717

tests: update expected output of svn encoding test The changes I see on the buildbot match the ones I see on my laptop, and all look reasonable. Differential Revision: https://phab.mercurial-scm.org/D1713

run-tests: use context managers for file descriptors I've seen the following error a few times recently when running the tests with `yes | ./run-tests.py --local -j9 -i`: Errored test-add.t: Traceback (most recent call last): File "./run-tests.py", line 821, in run self.runTest() File "./run-tests.py", line 910, in runTest if self._result.addOutputMismatch(self, ret, out, self._refout): File "./run-tests.py", line 1774, in addOutputMismatch rename(test.errpath, test.path) File "./run-tests.py", line 571, in rename os.remove(src) WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\\Users\\Matt\\projects\\hg\\tests\\test-add.t.err' This change doesn't fix the problem, but it seems like a simple enough improvement.

run-tests: add substitution patterns for common '\' path output on Windows The goal is to reduce the amount of hand tuning of new/changed tests that is required on Windows. Since the OS prints the proper paths everywhere else, this is limited to Windows. These are based on the check-code rules that were dropped in 5feb782c7a95. There are some minor tweaks, because those were trying to detect '/' paths without a '(glob)' at the end, whereas these detect '\' paths. Also, it looks like the 'no changes made to subrepo' one was broke, because the path to the subrepo has been getting output but was not in the pattern. End anchors are dropped because '(glob)' is no longer required, but '(feature !)' annotations are a possibility. The 'saved backup bundle' pattern dropped from run-tests.py was simply carrying over the first capture group. The replace() method runs prior to evaluating '\1', but it wasn't doing anything because of the 'r' prefix on '\\'. The 'not recording move' entry is new, because I stumbled upon it searching for some of these patterns. There are probably others.

debuginstall: add a line about re2 availability Using re2 engine can massively speed up regexp. We make it simpler to check if it is available in a given install.

tests: fix the check-code rule for testing non-existent files I missed this in feecfefeba25.

tests: convert the 'file://\$TESTTMP' rule to an automatic substitution The rule only triggered on non Windows platforms, even though Windows also required an adjustment. Automatic seems better. The aggressive globbing in test-subrepo-svn.t was found and rewritten by the substitution.

help: deprecate ui.slash in favor of slashpath template filter (issue5572) > For some reason, I thought someone (Mads?) said we had basically given > up on ui.slash, and some commands didn't support it. (from https://bz.mercurial-scm.org/show_bug.cgi?id=5572#c1) So the ui.slash option doesn't always work and is somewhat confusing. Let's make it clearer we won't improve the situation.

templatefilters: add slashpath() to convert path separator to slash Prepares for deprecating the ui.slash option, which isn't always respected.

check-code: remove unused variable 'winglobmsg' Follows up 5feb782c7a95.

phases: initialize number of loaded revisions to 0 As it isn't a revision number, an empty value should be 0, not -1.

phases: rename _phasemaxrev to _loadedrevslen to clarify it isn't max value "maxrev" sounds like max(0:tip), but it is actually len(0:tip).

lfs: add an experimental config to override User-Agent for the blob transfer This will allow developers to test against various server implementations. I didn't put it under [devel] because it's possible that some user needs to use it in the field.

lfs: add git to the User-Agent header for blob transfers As we were trying to transition off of the non production lfs-test-server for further experimenting, one of the problems we ran into was interoperability. A coworker setup gitbucket[1] to act as the blob server, tested with git, and passed it off to me. But push failed with a message saying "abort: LFS server returns invalid JSON:", and then proceeded to dump a huge HTML page to the screen. It turns out that it is assuming that git is the only thing that wants to do a blob transfer, and everything else is a web browser wanting HTML. It's only a single data point, but I suspect other things may be doing this too. RFC7231 gives an example [2] of listing multiple products in decreasing order of significance. Since the standard provides for this, and since it works with the one problematic server I found, I'm just enabling this by default for a better UX. There's nothing significant about the version of git chosen, other than it is the current version. [1] https://github.com/gitbucket/gitbucket/ [2] https://tools.ietf.org/html/rfc7231#page-46

outgoing: respect ":pushurl" paths (issue5365) Make 'hg outgoing' respect "paths.default:pushurl" in addition to "paths.default-push". 'hg outgoing' has always meant "what will happen if I run 'hg push'?" and it's still documented that way: Show changesets not found in the specified destination repository or the default push location. These are the changesets that would be pushed if a push was requested. If the user uses the now-deprecated "paths.default-push" path, it continues to work that way. However, as described at https://bz.mercurial-scm.org/show_bug.cgi?id=5365, it doesn't behave the same with "paths.default:pushurl". Why does it matter? Similar to the bugzilla reporter, I have a read-only mirror of a non-Mercurial repository: upstream -> imported mirror -> user clone ^-----------------------/ Users push directly to upstream, and that content is then imported into the mirror. However, those repositories are not the same; it's possible that the mirroring has either broken completely, or an import process is running and not yet complete. In those cases, 'hg outgoing' will list changesets that have already been pushed. Mozilla's desired behavior described in bug 5365 can be accomplished through other means (e.g. 'hg outgoing default'), preserving the consistency and meaning of 'hg outgoing'.

tests: test-pathconflicts-merge.t requires symlinks Once we're ready to turn this functionality on more widely, we might want to write a symlink-free version of the test that can run on Windows, but for now we'll just leave it disabled there. Differential Revision: https://phab.mercurial-scm.org/D1710

sshpeer: allow for additional environment passing to ssh exe We already have the ability to customize the ssh command line arguments, let's add the ability to customize its environment as well. Example use-case is ssh.exe from Git on Windows. If `HOME` enviroment variable is present and has some non-empty value, ssh.exe will try to access that location for some stuff (for example, it seems for resolving `~` in `.ssh/config`). Git for Windows seems to sometimess set this variable to the value of `/home/username` which probably works under Git Bash, but does not work in a native `cmd.exe` or `powershell`. Whatever the root cause, setting `HOME` to be an empty string heals things. Therefore, some distributors might want to set `sshenv.HOME=` in the configuration (seems less intrusive that forcing everyone to tweak their env). Test Plan: - rt Differential Revision: https://phab.mercurial-scm.org/D1683

unamend: allow unamending if allowunstable is set I don't see why unamend should be disallowed when allowunstable is set. By switching to rewriteutil.precheck() we fix that and get more consistent error messages (and some additional ones). Differential Revision: https://phab.mercurial-scm.org/D1682

rebase: add ui.log calls for whether IMM used, whether rebasing WCP Make it a bit easy to get metrics from these. Differential Revision: https://phab.mercurial-scm.org/D1681

lfs: using workers in lfs prefetch This significantly speeds up lfs prefetch. With fast network we are seeing ~50% improvement of overall prefetch times Because of worker's API in posix we do lose finegrained progress update and only see progress when a file finished downloading. Test Plan: Run tests: ./run-tests.py -l test-lfs* .... # Ran 4 tests, 0 skipped, 0 failed. Run commands resulting in lfs prefetch e.g. hg sparse --enable-profile Differential Revision: https://phab.mercurial-scm.org/D1568

worker: make windows workers daemons The windows workers weren't daemons and were not correctly killed when ctrl-c'd from the terminal. Withi this change when the main thread is killed, all daemons get killed as well. I also reduced the time we give to workers to cleanup nicely to not have people ctrl-c'ing when they get inpatient. The output when threads clened up nicely: PS C:\<dir>> hg.exe sparse --disable-profile SparseProfiles/<profile>.sparse interrupted! The output when threads don't clenup in 1 sec: PS C:\<dir> hg.exe sparse --enable-profile SparseProfiles/<profile>.sparse failed to kill worker threads while handling an exception interrupted! Exception in thread Thread-4 (most likely raised during interpreter shutdown): PS C:\<dir>> Test Plan: Run hg command on windows (pull/update/sparse). Ctrl-C'd sparse --enable-profile command that was using threads and observed in proces explorer that all threads got killed. ran tests on CentOS Differential Revision: https://phab.mercurial-scm.org/D1564

workers: add config to enable/diable workers This adds config to disable/enable workers with default being enabled. Test Plan: enabled profile without updaing .hg/hgrc (the default should be to use workers) and ran hg sprase --enable-profile <profile>.sparse Watched in the proces explorer that hg started 12 new threads for materializing files (this is my worker.numcpus) value Added [worker] enabled = False to the .hg/hgrc and re ran the command. This time hg didn't spawn any new threads for matreializing of files Differential Revision: https://phab.mercurial-scm.org/D1460

diff: disable diff.noprefix option for diffstat (issue5759) We could use patch.diffhunks() instead of patch.diff() to get filenames without parsing patch content, but that isn't always possible because we sometimes feed raw patch data to patch.diffstat().

hgweb: disable diff.noprefix option for diffstat Copied from cf1e15f91c90.

workers: handling exceptions in windows workers This adds handling of exceptions from worker threads and resurfaces them as if the function ran without workers. If any of the threads throws, the main thread kills all running threads giving them 5 sec to handle the interruption and raises the first exception received. We don't have to join threads if is_alive() is false Test Plan: Ran multiple updates/enable/disable sparse profile and things worked well Ran test on CentOS- all tests passing on @ passed here Added a forged exception into the worker code and got it properly resurfaced and the rest of workers killed: P58642088 PS C:\open\<repo>> ..\facebook-hg-rpms\build\hg\hg.exe --config extensions.fsmonitor=! sparse --enable-profile <profile> updating [==> ] 1300/39166 1m57sException in thread Thread-3: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception Exception in thread Thread-2: Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner self.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run raise e Exception: Forged exception <...> Traceback (most recent call last): File "C:\open\facebook-hg-rpms\build\hg\hgexe.py", line 41, in <module> dispatch.run() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 85, in run status = (dispatch(req) or 0) & 255 File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 173, in dispatch ret = _runcatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 324, in _runcatch return _callcatch(ui, _runcatchfunc) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 332, in _callcatch return scmutil.callcatch(ui, func) File "C:\open\facebook-hg-rpms\build\hg\mercurial\scmutil.py", line 154, in callcatch return func() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 314, in _runcatchfunc return _dispatch(req) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 951, in _dispatch cmdpats, cmdoptions) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 415, in runcommand return orig(lui, repo, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\undo.py", line 118, in _runcommandwrapper result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hgext\journal.py", line 84, in runcommand return orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 268, in _tracksparseprofiles res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 256, in _trackdirstatesizes res = runcommand(lui, repo, *args) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\copytrace.py", line 144, in _runcommand return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbamend\hiddenoverride.py", line 119, in runcommand result = orig(lui, repo, cmd, fullargs, *args) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 712, in runcommand ret = _runcommand(ui, options, cmd, d) File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 959, in _runcommand return cmdfunc() File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 948, in <lambda> d = lambda: util.checksignature(func)(ui, *args, **strcmdopt) File "C:\open\facebook-hg-rpms\build\hg\mercurial\util.py", line 1183, in check return func(*args, **kwargs) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 860, in sparse disableprofile=disableprofile, force=force) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 949, in _config len, _refresh(ui, repo, oldstatus, oldsparsematch, force)) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 1116, in _refresh mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False) File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 311, in applyupdates return orig(repo, actions, wctx, mctx, overwrite, labels=labels) File "C:\open\facebook-hg-rpms\build\hg\mercurial\merge.py", line 1464, in applyupdates for i, item in prog: File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 286, in _windowsworker raise t.exception Exception: Forged exception PS C:\open\ovrsource> Differential Revision: https://phab.mercurial-scm.org/D1459

workers: implemented worker on windows This change implements thread based worker on windows. The handling of exception from within threads will happen in separate diff. The worker is for now used in mercurial/merge.py and in lfs extension After multiple tests and milions of files materiealized, thousands lfs fetched it seems that neither merge.py nor lfs/blobstore.py is thread unsafe. I also looked through the code and besides the backgroundfilecloser (handled in base of this) things look good. The performance boost of this on windows is ~50% for sparse --enable-profile * Speedup of hg up/rebase - not exactly measured Test Plan: Ran 10s of hg sparse --enable-profile and --disable-profile operations on large profiles and verified that workers are running. Used sysinternals suite to see that all threads are spawned and run as they should Run various other operations on the repo including update and rebase Ran tests on CentOS and all tests that pass on @ pass here Differential Revision: https://phab.mercurial-scm.org/D1458

workers: don't use backgroundfilecloser in threads This disables background file closing when in not in main thread Test Plan: Ran pull, update, sparse commands and watched the closer threads created and destroyed in procexp.exe ran test on CentOS. No tests broken compared to the base Differential Revision: https://phab.mercurial-scm.org/D1457

debugssl: convert port number to int (issue5757) It doesn't use util.getport(), which may resolve service name to port number.

debugdiscovery: correct and clean up command synopsis This does a few things: * Changes "-r" to "--rev", since "-r" is not a valid short form * Removes non-existent "-l" and "-b" options * Removes "..." after options, since we don't usually have that Differential Revision: https://phab.mercurial-scm.org/D1706

copies: group wdir-handling in one place I think this makes it both easier to follow and shorter. Differential Revision: https://phab.mercurial-scm.org/D1698

copies: extract method for getting non-wdir forward copies I may add an alternative way of getting copy metadata (from changelog, not filelog) but the chaining with the dirstate copy metadata will be the same, so it will probably help to have this extracted. Even if that doesn't happen, the next patch will show that we can simplify this a bit after this refactoring, so it seems worth it regardless. Differential Revision: https://phab.mercurial-scm.org/D1697

copies: consistently use """ for docstrings Differential Revision: https://phab.mercurial-scm.org/D1696

copies: always respect matcher arg to _forwardcopies() The function would ignore the matcher if the dirstate copies were requested. It doesn't matter in practice because all callers used the returned map only for looking up specific files from and those files had already been filtered by the matcher (AFACT). Still, it's a little confusing, so let's make it clearer by respecting the matcher in this case too. Differential Revision: https://phab.mercurial-scm.org/D1695

debugdiscovery: drop reference to non-existent --remote-head option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1693

debugdiscovery: drop reference to non-existent --serverlog option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1692

debugdiscovery: drop reference to invalid --branch option It seems like it didn't even exist when debugdiscovery was introduced in cb98fed52495 (discovery: add new set-based discovery, 2011-05-02). Differential Revision: https://phab.mercurial-scm.org/D1691

tests: avoid echo with backslash escapes Some shells' echo (e.g. Bash's) do not interpret backslash escapes by default, while others shells' do (e.g. Dash's). This led to a failure in test-run-tests.t. We could switch to the more compatible printf, but using a heredoc seems simpler, so that's what this patch does. For anyone who wants to test this for themselves, remove the "unset HGTEST_SHELL" line in helper-runtests.sh and pass e.g. "--shell=/bin/dash" to the test runner. Differential Revision: https://phab.mercurial-scm.org/D1699

hgweb: stop using HTML comments in <script> Once upon a time, in 1995, there were browsers that didn't understand <script> tags and they would simply show the code inside as text. This started a tradition of wrapping everything inside <script> in <!-- HTML comments -->. Nowadays, it's not only not needed, but can be considered harmful[1]: - within XHTML documents, the source will actually be hidden from all browsers and rendered useless - `--` is not allowed within HTML comments, so any decrement operations in script are invalid [1]: http://www.javascripttoolbox.com/bestpractices/#comments

merge with stable

statprof: remove a line of deadcode Differential Revision: https://phab.mercurial-scm.org/D1689

extdata: abort if external command exits with non-zero status (BC) Per the last discussion, this is more reliable and consistent way than suppressing an error. For grep, erroring out might be inconvenient, but for curl, non-zero exit status should be detected. The latter wouldn't be possible if non-zero status is ignored. https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-October/105727.html

hgweb: update graph function docstring

rebase: fix for hgsubversion 5c25fe7fb1e broke something in the hgsubversion test path, causing it raise an abort (Abort: nothing to merge) during a perfectly good rebase. I tracked it down to this change. It's probably not hgsubversion related. I suspect that using the same `wctx` from before the initial update causes problems with the wctx's cached manifest property. I noticed we also sometimes stick random gunk on the wctx object in other places (like in `copies.py`) so it's probably best to reset it for now. The line I added before was actually useless since we don't pass wctx to the initial `merge.update`, so it defaults to `repo[None]`. So I just removed it. Differential Revision: https://phab.mercurial-scm.org/D1679

hgweb: render next pages on /graph incrementally Previously, when user scrolled down to see the next page on /graph, all hgweb did was re-render everything that would be visible (by simply incrementing revcount). It was not efficient at all, and this patch makes /graph page behave similarly to the regular /log: every new page only consists of new changesets, no duplication, and only jsdata is based on the full set of changesets required to build accurate graph. This is achieved by adding "?graphtop=<node>" to the next page URL template, effectively remembering where the graph started, and using that value to create the new `tree` that covers the whole visible graph. That variable is then used to produce jsdata for redrawing graph client-side. nextentry is used for the same purpose as on /log page (to format the next page URL), but it's not a part of the graph.

hgweb: split graphdata() into jsdata() and nodes() nodes keyword passed to the template can be any iterator, but jsdata needs to be a list because it gets JSONified.

hgweb: calculate <canvas> width and height client-side hgweb determines and passes to templates some variables related to graph appearance, like bg_height, canvaswidth and canvasheight. bg_height was and still is used for graph.scale() call in graph.tmpl, and the two latter variables were used in <canvas> element as width and height properties, and they were set before JS code got to run. Setting these properties server-side doesn't make a lot of sense, because a graph that has been scaled should calculate things like width and height on its own when being rendered. Let's move (re)sizing <canvas> to JavaScript (to Graph.render function) and stop parsing HTML with regular expressions just to know new width and height. That extra loop that only counts cols is required because <canvas> can't be resized after or in the process of rendering (or it gets cleared). Incidentally, SVG doesn't have this problem and I'm hoping to switch graph to using it in future. There also was truecanvasheight, but according to hg grep --all it was never used, see d490edc71146.

hgweb: filter graphmod.colored() output before iterating over it Consumers in this function use output of graphmod.colored(), but only want items with type == CHANGESET, so let's filter it early. This is primarily just a refactoring, but it also fixes a potential small bug with `rows = len(tree)` (this variable is used for "Rows shown" line in raw-graph) if there are items of other types.

py3: introduce pycompat.ziplist as zip is a generator on Python 3 Similar to map, the return value of zip was also turned into a generator on Python 3. We have multiple occurences in our codebase where we retrieve the values of zip multiple times, so let's introduce pycompat.ziplist similar to pycompat.maplist. Differential Revision: https://phab.mercurial-scm.org/D1675

py3: pass bytes from extension to ui.write() in test-revlog-mmapindex.t This makes the test pass on Python 3. Differential Revision: https://phab.mercurial-scm.org/D1674

py3: handle keyword arguments correctly in hgext/record.py Differential Revision: https://phab.mercurial-scm.org/D1673

py3: handle keyword arguments correctly in bundlerepo.py Differential Revision: https://phab.mercurial-scm.org/D1672

py3: handle keyword arguments correctly in debugcommands.py Differential Revision: https://phab.mercurial-scm.org/D1671

memfilectx: make changectx argument mandatory in constructor (API) committablefilectx has three subclasses: workingfilectx, memfilectx, and overlayfilectx. committablefilectx takes an optional (change) ctx instance to its constructor. If it's provided, it's set on the instance as self._changectx. If not, that property is supposed to be defined by the class. However, only workingfilectx does that. The other two will have the property undefined if it's not passed in the constructor. That seems bad to me. This patch makes the changectx argument to the memfilectx constructor mandatory because that fixes the failure I ran into. It seems like we should also fix the overlayfilectx case. Differential Revision: https://phab.mercurial-scm.org/D1658

debugbuilddag: create filectx instance in 'filectxfn' callback Same motivation is previous patch. Differential Revision: https://phab.mercurial-scm.org/D1670

synthrepo: create filectx instance in 'filectxfn' callback I would like to pass the memctx to the memfilectx constructor, but it's not available where we currently create the memfilectx. It is available in the 'filectxfn' callback, so let's create the memfilectx there instead. A later patch will start actually passing the memctx. Differential Revision: https://phab.mercurial-scm.org/D1669

lfs: use 'ui' provided to `upgrade` for output, instead of stealing srcrepo's Also spotted by Yuya.

lfs: correct the directory list value returned by lfsvfs.walk() Spotted by Yuya.

debuglocks: allow setting a lock

debuglocks: add tests (and fix typo in early return)

tests: remove (glob) annotations that were only for '\' matches # skip-blame because this was mechanically rewritten the following script. I ran it on both *.t and *.py, but none of the *.py changes were proper. All *.t ones appear to be, and they run without addition failures on both Windows and Linux. import argparse import os import re ap = argparse.ArgumentParser() ap.add_argument('path', nargs='+') opts = ap.parse_args() globre = re.compile(r'^(.*) \(glob\)(.*)$') for p in opts.path: tmp = p + '.tmp' with open(p, 'rb') as src, open(tmp, 'wb') as dst: for line in src: m = globre.match(line) if not m or '$LOCALIP' in line or '*' in line: dst.write(line) continue if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'): dst.write(line) continue dst.write(m.group(1) + m.group(2) + '\n') os.unlink(p) os.rename(tmp, p)

check-code: drop the rules for adding (glob) for Windows paths I'll probably resurrect at least some of these as replacement patterns to switch '\' to '/' when creating tests at some point. But since this subset of globs isn't needed anymore after dfae14354660, and the rules are blocking the glob removal, just drop them for now. It probably isn't worth turning them into checks for unnecessary globs being present.

run-tests: stop automatically adding a (glob) for bundle backup lines This is the first step to dropping the existing globs for '\' matches, now that it is handled automatically. Instead of just dropping it, this pattern is now used to convert to '/' paths, to reduce the amount of manual cleanup required when creating tests on Windows.

histedit: preserve active branch while histediting The branch information was properly preserved in the changeset, but the "active" branch of the working copy could be lost (the branch of the base being used). Histedit used to behave properly in this regard but the case was not tested and regressed 4 years ago in ab2362e1672e.

hgweb: implement json-graph It's essentially a copy of json-log with graph-related things added (col, row, color, edges).

rebase: replace --inmemory flag with rebase.experimental.inmemory config Differential Revision: https://phab.mercurial-scm.org/D1666

tests: add some commentary and diagnostics to test-run-tests.t Hopefully this will give us a shot at fixing it on the Windows builder. Differential Revision: https://phab.mercurial-scm.org/D1653

templater: fix "one arguments"

tests: add commit hashes to log commands in rebase tests Commit hashes are a useful way to ensure the content of commits made in the tests are not changing, even if we don't query every aspect of every commit. (And some properties, like extras, are rarely printed at all.) Many of the rebase log -G calls didn't show hashes; by adding hashes to places that weren't showing them we can help protect those tests from unwanted changes. Differential Revision: https://phab.mercurial-scm.org/D1650

tests: add a simple test for in-memory rebase This is just a very simple start, but verifies some of the basic cases of an in-memory rebase. Differential Revision: https://phab.mercurial-scm.org/D1652

patch: do not break up multibyte character when highlighting word This changes {\W} to {\W - any 8bit characters} so that multibyte sequences are taken as words. Since we don't know the encoding of user content, this is the most sensible definition of a non-word.

run-tests: accept '\' vs '/' path differences without '(glob)' Having to constantly adjust these is a hassle. It's easy for this to slip by when not testing on Windows, and then when it happens on stable, the tests fail for the next 3 months if we follow the rules for stable. This works the same way the EOL differences are ignored, namely to adjust on the fly and recheck on Windows. I can't think of any situation where there would be a '\' on Windows, a '/' elsewhere, and the '/' should be considered a failure on Windows. This fixes the obvious output problems where (glob) is missing. Without this, test-alias.t, test-remotenames.t and test-largefiles-misc.t are failing. The flip side (not handled by this) is the case where an unnecessary glob is present. There seems to be two separate behaviors. cf300c1ad7bf is an example of where the test has been autocorrecting (with output differences), and d4ec69ff652a is an example where the test fails and reports 'no result code from test'. Hopefully those cases will become even more rare if people don't need to guess at when a glob is needed for a Windows path. It's probably unreasonable to submit a single patch that wipes out all of the (glob) instances that were only used to hide path differences, given the churn from other contributors. Since their presence isn't harming the tests, these can be removed through attrition.

run-tests: suggest a (glob) for os.path.sep mismatches with '\r\n' EOL too We already do this for lines ending in '\n', such that the test only needs to be run with --interactive and the changes accepted at the end. But that wasn't working with list-tree.py output for example, and required manual fixup.

tests: stabilize the sorted output of list-tree.py on Windows The test-largefiles-misc.t test was moving 'dir2\' before 'dir\' because while '/' precedes most of the printable ASCII characters, '\' comes after numbers and capital letters, among other symbols.

upgrade: simplify workaround for repo.ui.copy() Copied from commandserver.py.

debugformat: embed raw values in JSON and template output

debugformat: flush formatter output per item

tests: use Python to write binary data in lfs test instead of shell The shell construct here appears to be unevenly supported: it works in /bin/sh on FreeBSD, but it doesn't seem to work when /bin/sh is dash. Using a Python inline directive works fine, so let's just do that instead. Differential Revision: https://phab.mercurial-scm.org/D1636

py3: add 32 new tests passing to the whitelist Yay, we have crossed 100 in number of tests passing on Python 3. There are 662 tests in our test suite, so there is a lot more which is need to be done. Differential Revision: https://phab.mercurial-scm.org/D1648

py3: handle keyword arguments correctly in wireproto.py Differential Revision: https://phab.mercurial-scm.org/D1647

py3: handle keyword arguments correctly in ui.py Differential Revision: https://phab.mercurial-scm.org/D1646

py3: handle keyword arguments correctly in templater.py Differential Revision: https://phab.mercurial-scm.org/D1645

py3: handle keyword arguments correctly in subrepo.py Differential Revision: https://phab.mercurial-scm.org/D1644

py3: handle keyword arguments correctly in statprof.py Differential Revision: https://phab.mercurial-scm.org/D1643

py3: handle keyword arguments correctly in sslutil.py Differential Revision: https://phab.mercurial-scm.org/D1642

py3: handle keyword arguments correctly in simplemerge.py Differential Revision: https://phab.mercurial-scm.org/D1641

py3: handle keyword arguments correctly in revset.py Differential Revision: https://phab.mercurial-scm.org/D1640

py3: handle keyword arguments correctly in patch.py Differential Revision: https://phab.mercurial-scm.org/D1639

py3: handle keyword arguments correctly in obsolete.py Differential Revision: https://phab.mercurial-scm.org/D1638

py3: handle keyword arguments correctly in keepalive.py Differential Revision: https://phab.mercurial-scm.org/D1637

lfs: restore the local blob store after a repo upgrade This also ends up testing the local extension wrapping for dstrepo during upgrade, which was fixed in 06987c6971be.

lfs: override walk() in lfsvfs In order to fix the missing lfs store after an upgrade, I attempted to walk the store vfs to hardlink to the upgraded repo's store. But the custom join() clashes with the default walk() implementation. First, 'path=None' blew up in the regex matcher, because it wanted a string. But even if that is fixed, the join to walk the root of the vfs wouldn't match the required xx/xx...xx pattern. The first cut of this was a copy/paste/tweak of the base implementation, but this version of walk() hides the internal directories, and treats the vfs as a flat store. I think this makes sense because most vfs methods call join() on input paths, which wants the simple oid format. It also relieves the caller from having to deal with bogus files/directories in the store.

tests: add coverage for preserving 'lfs' requirement on repo upgrade The test also shows that the local blob store is erroneously lost.

test-upgrade-repo: glob away timing values

py3: handle keyword arguments correctly in httppeer.py Differential Revision: https://phab.mercurial-scm.org/D1635

py3: handle keyword arguments correctly in httpconnection.py Differential Revision: https://phab.mercurial-scm.org/D1634

py3: handle keyword arguments correctly in hook.py Differential Revision: https://phab.mercurial-scm.org/D1633

py3: handle keyword arguments correctly in help.py Differential Revision: https://phab.mercurial-scm.org/D1632

py3: handle keyword arguments correctly in exchange.py Differential Revision: https://phab.mercurial-scm.org/D1631

py3: handle keyword arguments correctly in dispatch.py Differential Revision: https://phab.mercurial-scm.org/D1630

py3: handle keyword arguments correctly in context.py Differential Revision: https://phab.mercurial-scm.org/D1629

py3: handle keyword arguments correctly in commands.py Differential Revision: https://phab.mercurial-scm.org/D1628

py3: handle keyword arguments correctly in color.py Differential Revision: https://phab.mercurial-scm.org/D1627

py3: handle keyword arguments correctly in cmdutil.py Differential Revision: https://phab.mercurial-scm.org/D1626

py3: handle keyword arguments correctly in archival.py Differential Revision: https://phab.mercurial-scm.org/D1625

py3: handle keyword arguments correctly in hgext/largefiles/ Keys of keyword arguments must be str(unicode) on Python 3. The transformer which we use on Python 3, appends b'' in front of each string literal, so this may lead in KeyError or None return even when the key is present by we are using bytes value and it's stored in unicodes. This patch and all the similar patches handle this by either converting the keys of kwargs to bytes using 'pycompat.byteskwargs()' or adding r'' so that the transformer won't append b''. This next 23 patches follows the above mentioned way to handle keyword arguments. Differential Revision: https://phab.mercurial-scm.org/D1624

remotenames: rename related file and storage dir to logexchange This patch renames remotenames.py to logexchange.py, test-remotenames.t to test-logexchange.t. Also this patch renames the directory in which the data is stored from remotenames to logexchange. After this patch, data about bookmarks and branches from a server we pull is stored in `.hg/logexchange/(bookmarks|branches)` files. Thanks to smf for the suggestion. Differential Revision: https://phab.mercurial-scm.org/D1607

lfs: allow to run 'debugupgraderepo' on repo with largefiles The extensions wrap the necessary function to ensure the 'lfs' requirements won't be dropped. It is now possible to run `hg debugupgraderepo` on a repository with lfs.

upgrade: add a 'redeltafullall' mode We add a new mode for delta recomputation, when selected, each full text will go through the full "addrevision" mechanism again. This is slower than "redeltaall" but this gives the opportunity for extensions to trigger special logic. For example, the lfs extensions can decide to promote some revision to lfs storage during the upgrade.

upgrade: use actual filelog to convert filelog Extensions can add extra logic related to the config, so we must use the actual class. The path used needs minimal transformation for this to work.

upgrade: more standard creation of the temporary repository By using the standard path to create a repository we fill some hole in the current initialization process. The one who triggered this changeset was the lack of extensions initialization.

upgrade: use the repository 'ui' as the base for the new repository The `repo.baseui` contains all the configuration but the one specific to the repository (so it can be used when dealing with local peer and sub- repository). However, we need the repository config to be taken into account when doing the upgrade. Otherwise, the upgrade related config that exists in the repository config won't be taken into account when performing the update. A buggy and surprising behavior. We had to work around protection set around `repo.ui.copy` since we are an uncommon case.

upgrade: add a test to show the repository config being ignored The upgrade process ignores the config within the repository. The next changeset fixes it, but we introduce this test before to show it actually tests our target.

upgrade: register compression as a format variants Compression is a promising vector for speedup, let us make it easier to check the compression used and upgrade existing repository.

debugformat: handle non-boolean value for variant

debugformat: update label depending on value difference The new label highlight areas where the repo format differs from current config or default. This should help people spot area where a repository mismatch with the expected state.

debugformat: add data about the config when verbose In verbose mode, the command also displays the current configuration choice for the variant and the global Mercurial default for it.

debugformat: add a 'debugformat' command The command displays basic data about all format variants registered for repo upgrades. This gives a quick way to peek into a repository format. The 'fm.write()' calls are very independent because more data will be added in later changeset. Having more separate call make the later patch clearer.

upgrade: rename 'removecldeltachain' to 'plain-cl-delta' The new naming is more descriptive of a "state" while the older one was more about "action". I'm looking into command exposing more of data about the state of the repository so "state" oriented work better there. The key has not been made public anywhere outside the debug area so it is fine to update it.

tests: glob away ' that doesn't occur on all platforms in lfs test Differential Revision: https://phab.mercurial-scm.org/D1623

rebase: extract _assignworkingcopy Slightly cleaner, and makes it easier to hook this in an extension. Differential Revision: https://phab.mercurial-scm.org/D1617

rebase: disable `inmemory` if the rebaseset contains the working copy As described in the comment, rebasing the working copy parent with in-memory merge, and then updating to the new commit, isn't much faster because of the extra overhead of uppdating. Best to leave it off in that case. This commit makes deploying in-memory merge via an extension easier, because you can just set `inmemory=True` based on some config or probability, and this will turn off the cases where it's not desired. Differential Revision: https://phab.mercurial-scm.org/D1616

clone: add support for storing remotenames while cloning If `experimental.remotenames` is set to True, we store the remotenames in case of `hg pull`. This patch adds that support to clone command also. Differential Revision: https://phab.mercurial-scm.org/D1601

revset: use phasecache.getrevset to calculate public() Other revsets like secret(), draft(), _nonpublic() are using phasescache.getrevset already. The latter is more efficient after D1606. So let's migrate the public() revset function too. Tested using: $ hg debugshell --hidden --cwd hg-committed` In [1]: %timeit len(repo.revs('public()')) * Before D1606: 10 loops, best of 3: 22.5 ms per loop * Before this change, after D1606: 10 loops, best of 3: 28.6 ms per loop * After this change: 10 loops, best of 3: 20.2 ms per loop Therefore `public()` revset becomes even slightly faster after the data structure change by D1606. A similar performance win could also be observed on a large repo. A side effect is `phasecache.getrevset` needs to take a `subset` parameter. That was added with a default value so it won't cause BC issues. Differential Revision: https://phab.mercurial-scm.org/D1620

highlight: eagerly discover plugin lexers while demandimport is off Since highlight is only relevant for servers, it seems worthwhile to just trigger this eagerly, which avoids really weird traceback problems caused by demandimport messing with some of the lexer plugins. Differential Revision: https://phab.mercurial-scm.org/D1619

patch: move part of tabsplitter logic in _inlinediff It cannot be entirely moved within _inlinediff as long as worddiff is experimental (when turned off, matches is always an empty dict).

overlayworkingctx: invalidate the manifest cache when changing parents This fixes problems noticeable when rebasing several commits into one destination commit using ``--collapse``. The manifest cache needs to be cleared each time. Differential Revision: https://phab.mercurial-scm.org/D1244

overlayworkingctx: add `_compact()` Alas, presence of a key in the cache isn't sufficient evidence that the file is actually dirty. Differential Revision: https://phab.mercurial-scm.org/D1243

overlayworkingctx: add ``tomemctx()`` Differential Revision: https://phab.mercurial-scm.org/D1242

overlayworkingctx: add _auditconflicts to write() Alas, part of Mercurial's conflict detection (for file<->folder conflicts, for example) depends on the filesystem. We don't have the filesystem with IMM, so we have to run these checks ourselves. Differential Revision: https://phab.mercurial-scm.org/D1241

overlayworkingctx: inherit from committablectx instead of workingctx We should find a better name for this context at some point. (mutablememctx?) Differential Revision: https://phab.mercurial-scm.org/D1240

overlayworkingctx: add a no-op ``clearunknown()`` We no longer inherit ``workingctx``'s version, but we also don't need to do anything anymore. Differential Revision: https://phab.mercurial-scm.org/D1239

overlayworkingctx: add _manifest, files(), added(), removed(), modified() Differential Revision: https://phab.mercurial-scm.org/D1238

rebase: rerun a rebase on-disk if IMM merge conflicts arise Differential Revision: https://phab.mercurial-scm.org/D1249

rebase: add concludememorynode(), and call it when rebasing in-memory Differential Revision: https://phab.mercurial-scm.org/D1248

rebase: pass the wctx object (IMM or on-disk) to merge.update Differential Revision: https://phab.mercurial-scm.org/D1247

rebase: do not update if IMM; instead, set the overlaywctx's parents Differential Revision: https://phab.mercurial-scm.org/D1246

rebase: pass wctx to rebasenode() Differential Revision: https://phab.mercurial-scm.org/D1245

check-code: allow tabs in heredoc

hgweb: rewrite `template = A and B or C` to be a proper ternary operator

fsmonitor: fsmonitor should send wlock notifications to watchman The fsmonitor extension should send state-enter and state-leave notifications to watchman when the wlock is acquired/release, respectively. This will allow watchman and watchman subscribers to customize behavior based on whether source control operations are occurring. Test Plan: Tested checkout, update and working copy changes with extension enabled. Differential Revision: https://phab.mercurial-scm.org/D1612

fsmonitor: remove watchman transaction and working copy change notifications Remove working copy change and transaction notifications. We were relying upon callbacks on transaction function. This caused issues with lock ordering. A different approach will be adopted in a subsequent commit. Differential Revision: https://phab.mercurial-scm.org/D1611

patch: catch unexpected case in _inlinediff If operation is neither 'diff.inserted' or 'diff.deleted', label and token won't be define. This patch explicitely catches that exception.

patch: reverse _inlinediff output for consistency

phases: drop the list with phase of each rev, always comput phase sets Change the C implementation of phasecache.loadphaserevs to provide only the sets for draft and secret phase as well as the number of revisions seen. Change the pure Python implementation of the same functino to compute the sets instead of the list of phases for each revision. Change phasecache.phase to check the phase sets and assume public if the revision is in neither draft nor secret set. This is computationally slightly more expensive. Change phasecache.getrevset for public() based queries to compute the set of non-matching revisions and return the result as filtered fullreposet. A shortcut is taken when no draft or secret revision exists. Bump the module version for the changed interface contract. Overall, this saves around 16 Bytes per revision whenever the phasecache is used, for the test case in issue5691 it is around 3MB. getrevset() for a large repository is around 13% slower here, that seems an acceptable trade off. Performance impact for phase() should be similar. Differential Revision: https://phab.mercurial-scm.org/D1606

transaction: build changes['revs'] as range instead of a set Revisions are added consecutively, so a range can easily represent them in the changes list. This saves around 45 Bytes / revision on 64bit platforms and reduces the memory footprint of issue5691 by 15MB. Don't copy changes['revs'] in getobsoleted. Ranges have a very efficient contains implementation already. Differential Revision: https://phab.mercurial-scm.org/D1615

ui: add diff.showfunc to tweakdefaults This is a little risky, as I think we can have some encoding weirdness crop up. showfunc also isn't the most robust feature, but it's still often useful context... Differential Revision: https://phab.mercurial-scm.org/D1610

ui: add curses interface to tweakdefaults This was part of the original proposal, and while *I* don't like the curses interface, most users anecdotally seem to greatly prefer it to plain text interfaces. Differential Revision: https://phab.mercurial-scm.org/D1609

push: restrict common discovery to the pushed set This changeset make use of the ability of the set discovery to only search common changeset for a subset of the repository. Restricting that search to the pushed set avoid potential waste of time finding out the status of many unrelated related revision. Repository with many heads were especially badly affected by this. Here is an example of findcommonhead discovery for pushing 11 outgoing changeset on a repository with tens of thousand of unrelated heads. (discovery run over a ssh link to localhost). Before: queries: 92 time: 44.1996s After: queries: 3 time: 0.6938s A x63 speedup even with a network link without latency.

setdiscover: allow to ignore part of the local graph Currently, the push discovery first determines the full set of common nodes before looking into what changesets are outgoing. When pushing a specific subset, this can lead to pathological situations where we search for the status of thousand of local heads that are unrelated to the requested pushes. To fix this, we need to teach the discovery to ignores part of the graph. Most of the necessary pieces were already in place. This changeset just makes them available to higher level API and tests them. Change actually impacting pushes are coming in a later changeset.

largefiles: allow to run 'debugupgraderepo' on repo with largefiles The extensions wrap the necessary function to ensure the 'largefiles' requirements won't be dropped. It is now possible to run `hg debugupgraderepo` on a repository with largefiles.

upgraderepo: allow extension to register preserved requirements Some requirement does not directly result from config and needs more advanced logic to be preserved. The current example is 'largefiles'. We add a hook point in the upgrade code so that extensions can handle these cases. The 'largefiles' extension will use it in the next changeset.

revset: make follow() accept empty startrev This is the same behavior as ancestors(emptyset).

revset: alias follow(startrev=rev) to ancestors(rev) This seems natural given 'log -frREV' (with no file pattern) is equivalent to 'log -frREV *'.

revset: make follow() accept keyword arguments Also renamed the argument from 'pattern' to 'file' conforming to followlines().

revset: make follow() accept multiple startrevs The diff might look slightly complicated, but the initial "c = repo['.']" was effective if rev = None.

dagop: use heap to compute max rev in filectxancestors()

dagop: add smartset interface to filectxancestors() The original filectx API is kept public since we'll need it to walk ancestor (rev, match) pairs efficiently. The current implementation scans ancestors twice for 'hg log -fp FILE'.

overlayworkingctx: add `_checkexist(path)` This is in preparation to switch this class to inheriting (and being based off a) `commitctx` instead of a `workingctx`. `filectx` has no `exists` function, so this is how we'll fall back in that case. Differential Revision: https://phab.mercurial-scm.org/D1237

overlayworkingctx: fix a bad reference to `self._path` Differential Revision: https://phab.mercurial-scm.org/D1236

overlayworkingctx: track copy information in the context We can no longer do so in the dirstate, so add the functions to do so here. Differential Revision: https://phab.mercurial-scm.org/D1235

rebaseruntime: raise InMemoryMergeConflictsError on merge conflicts Differential Revision: https://phab.mercurial-scm.org/D1234

rebase: do not bail on uncomitted changes if rebasing in-memory Differential Revision: https://phab.mercurial-scm.org/D1233

rebase: add the --inmemory option flag; assign a wctx object for the rebase In the future, the --inmemory flag might be deprecated in favor of something more intelligent (for example, always rebasing in-memory if the working copy parent isn't in the rebaseset). But we might keep it as a way to explicitly force IMM on or off. Differential Revision: https://phab.mercurial-scm.org/D1232

overlayworkingctx: move _wrappedctx out of the constructor With rebase, we will be setting the _wrappedctx at a different point from the wctx construction (somewhat later, and possibly several times). Move it to a public function. Differential Revision: https://phab.mercurial-scm.org/D1231

merge: don't check for unknown files in IMM Differential Revision: https://phab.mercurial-scm.org/D1214

rebase: enable multidest by default This was intended to be done by D470. But there was a minor documentation issue. The feature is quite usable now so it gets formally documented and enabled. There is no behavior change for people not using the `SRC` or `ALLSRC` in rebase destination revset. .. feature:: Rebase with different destination per source revision Previously, rebase only supports one unique destination. Now ``SRC`` and ``ALLSRC`` can be used in rebase destination revset to precisely define destination per each individual source revision. For example, the following command could move some orphaned changesets to reasonable new places so they become no longer orphaned:: hg rebase -r 'orphan()-obsolete()' -d 'max((successors(max(roots(ALLSRC) & ::SRC)^)-obsolete())::)' Differential Revision: https://phab.mercurial-scm.org/D1063

overlayworkingctx: make clean() public It will be called directly by rebase after concluding a node. Differential Revision: https://phab.mercurial-scm.org/D1230

overlayworkingctx: remove flushall() Differential Revision: https://phab.mercurial-scm.org/D1216

merge: skip subrepo state, update hooks, and updating the dirstate in IMM Differential Revision: https://phab.mercurial-scm.org/D1215

merge: remove calls to flushall() Since D1105, these are unnecessary since IMM will now never use workers. Differential Revision: https://phab.mercurial-scm.org/D1213

filemerge: raise InMemoryMergeConflictsError if we hit merge conflicts in IMM Merge conflicts might be supported in the future, but for now are kept out of scope. Any places where we used to call `flushall()` should be replaced with some kind of exception. At this point, IMM M1 is no longer supported. Differential Revision: https://phab.mercurial-scm.org/D1212

context: switch ctx() use to changectx() I added `ctx()` to `overlayworkingfilectx`, (and before that, `absentfilectx`), because `absentfilectx` had reference to this function in its `cmp()` function. But the standard is actually `changectx()`, and no other class implements `ctx()`. So let's use the standard name. (As a result, I'm not sure that part of the `absentfilectx` comparator ever worked! It was written before I added either function.) This will be necessary in the next patch. Differential Revision: https://phab.mercurial-scm.org/D1211

lfs: introduce a user level cache for lfs files This is the same mechanism in place for largefiles, and solves several problems working with multiple local repositories. The existing largefiles method is reused in place, because I suspect that there are other functions that can be shared. If we wait a bit to identify more before `hg cp lfutil.py ...`, the history will be easier to trace. The push between repo14 and repo15 in test-lfs.t arguably shouldn't be uploading any files with a local push. Maybe we can revisit that when `hg push` without 'lfs.url' can upload files to the push destination. Then it would be consistent for blobs in a local push to be linked to the local destination's cache. The cache property is added to run-tests.py, the same as the largefiles property, so that test generated files don't pollute the real location. Having files available locally broke a couple existing lfs-test-server tests, so the cache is cleared in a few places to force file download.

largefiles: refactor _usercachedir() to allow reuse with lfs Largefiles puts everything into a flat directory, while lfs divides files up by creating subdirectories consisting of the first two characters of the hash. Therefore, pointing at the largefiles cache won't work.

lfs-test: note a problem with unpushed lfs files and cloning/sharing AFAIK, this isn't an issue with largefiles because it knows how to look in the system-wide cache.

patch: add within-line color diff capacity The `diff' command usually writes deletion in red and insertions in green. This patch adds within-line colors, to highlight which part of the lines differ. Lines to compare are decided based on their similarity ratio, as computed by difflib SequenceMatcher, with an arbitrary threshold (0.7) to decide at which point two lines are considered entirely different (therefore no inline-diff required). The current implementation is kept behind an experimental flag in order to test the effect on performance. In order to activate it, set inline-color-diff to true in [experimental].

dagop: extend filectxancestors() to walk multiple files

dagop: put start fctx into visit dict of filectxancestors() Prepares for multiple start revisions/files.

dagop: change visit dict of filectxancestors() indexed solely by rev In future patches, a max heap will be used to compute the next revision to visit.

dagop: use fctx.rev() consistently in filectxancestors() We can't use fctx.linkrev() to sort fctxs coming from multiple files. This was changed at 24b57c3899f8 due to performance issue, but we know we evaluate parent.rev() in revset anyway.