crecord: honor "ui.color = no" config option The current implementation of crecord ignores the ui.color setting. This patch checks the ui.color config option and does the curses setup without colors when the option is set to a falsy value. For other (or missing) setting of ui.color, curses setup is done as before and uses colors.

lfs: use the local store method for opening a blob I noticed that when I cloned without updating and then turned around and pushed that clone to an lfs server, it was only trying to find the blob in the local store. Writes to the dummyremote (file based store) use local.read(), which looks at both the usercache and local store.

lfs: add a local store method for opening a blob The has() and read() methods already dynamically switch between the usercache and local store. This should generally be preferred to directly accessing the vfs instances outside of the store. The file is now explicitly opened in binary mode for clarity. (It was also being opened in binary mode before, but only because vfs.__call__() appends 'b' if needed when not opening with 'text=True'.)

hgdemandimport: use correct hyperlink to python-bug in comments (issue5765) Differential Revision: https://phab.mercurial-scm.org/D1810

monoblue: adjust font size of graph entries /shortlog uses 1.1em font for changesets, so it makes sense to use the same size on /graph too. In practice, 1.2em is 14.784px, and 1.1em is 13.552px.

hgweb: make .info a block element by default Using a <div> instead of a <span> with "display: block" makes more sense. While at it, let's adjust top margin in monoblue so that text in .info doesn't overlap changeset "tags" elements (they are quite large in monoblue).

revlog: don't use slicing to return parents This is the only place we use a slice on index entries, which are currently tuples. In preparation for moving away from tuples, let's stop using slices so we don't have to implement that support on the new type. We also tweak the logic slightly so the exception only catches the IndexError on the index lookup, not on the index entry lookup. The old code should never have been buggy. But it was semantically wrong. Differential Revision: https://phab.mercurial-scm.org/D1764

tests: switch test from '--config ui.editor=~/foo' to HGEDITOR=~/foo The test is currently not testing anything, the '--config ui.editor=~/path' argument is apparently of lower priority than the HGEDITOR that run-tests.py is setting. Differential Revision: https://phab.mercurial-scm.org/D1809

revert: support reverting to hidden cset if directaccess config is set This patch adds support for reverting files to a hidden changeset if `experimental.directacess=True` is set. Differential Revision: https://phab.mercurial-scm.org/D1763

update: support updating to hidden cset if directaccess config is set This patch adds support for updating to a hidden changeset without using --hidden if `experimental.directacces=True` is set. The update command will print out a warning when updating to a hidden changeset saying: `updating to a hidden changeset <hash>` The warning is also printed when directaccess is not used and --hidden is passed which I think is good behaviour. Tests are added for the directaccess part and updated output of tests which have case of updating to hidden cset using `--hidden`. Differential Revision: https://phab.mercurial-scm.org/D1762

commands: check for empty rev before passing to scmutil.unhidehashlikerevs Differential Revision: https://phab.mercurial-scm.org/D1760

test-convert-cvs: change TZ=US/Hawaii to TZ=Pacific/Johnston The former was limited to be known on Linux and the test failed on FreeBSD and Solaris platforms. The newer is known on Linux, FreeBSD and Solaris.

osutil: implement getfsmountpoint() on BSD systems I don't have a BSD system handy to test this, but it looks simple enough from the man page.

debugfs: display the tested path and mount point of the filesystem, if known While implementing win32.getfstype(), I noticed that MSYS path mangling is getting in the way. Given a path \\host\share\dir: - If strong quoted, hg receives it unchanged, and it works as expected - If double quoted, it converts to \host\share\dir - If unquoted, it converts to \hostsharedir The second and third cases are problematic because those are valid paths relative to the current drive letter, so os.path.realpath() will expand it as such. The net effect is to silently turn a network path test into (typically) a "C:\" test. Additionally, the command hangs after printing out 'symlink: no' for the third case (but is interruptable with Ctrl + C). This path mangling only comes into play because of the command line arguments- it won't affect internally obtained paths. Therefore, the simplest thing to do is to provide feedback on what the command is acting on. I also added the mount point, because Windows supports nesting [1] volumes (see the examples in "Junction Points and Mounted Folders"), and it was a useful diagnostic for figuring out why the wrong filesystem was printed out in the cases above. I opted not to call os.path.realpath() on the path argument, to make it clearer that the mangling isn't being done by Mercurial. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa364996(v=vs.85).aspx

util: add a function to show the mount point of the filesystem For now, this is Windows only, since Linux doesn't have the value in its statfs structure, and I don't have a BSD system to test with.

win32: split a utility function to obtain the volume out of getfstype() This is only done on Windows because it's simple enough to call statfs() on Unix. The goal is to display this in `hg debugfs`.

util: whitelist NTFS for hardlink creation (issue4580)

win32: implement util.getfstype() This will allow NTFS to be added to the hardlink whitelist, and resume creating hardlinks in transactions (which was disabled globally in 07a92bbd02e5; see also e5ce49a30146). I opted to report "cifs" for remote volumes because this shows in `hg debugfs`, which also reports that hardlinks are supported for these volumes. So being able to distinguish it from "unknown" seems useful. The documentation [1] seems to indicate that SMB isn't supported by these functions, but experimenting shows that mapped drives are reported as "NTFS" on Windows 7. I don't have a second Windows machine, but instead shared a temp directory on C:\. In this setup, both of the following were detected as 'cifs' with the explicit GetDriveType() check: Z:\repo>hg ci -A C:\>hg -R \\hostname\temp\repo ci -A # (without Z:\ being mapped) It looks like this is called 6 times to add and commit a single new file, so I'm a little surprised this isn't cached. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa364993(v=vs.85).aspx

util: move getfstype() to the platform modules This makes room for implementing on Windows using ctypes.

graphlog: add another graph node type, unstable, using character "*" (BC)

largefiles: modernize how capabilities are added to the wire protocol See 982f13bef503, which came well after this code was originally written.

lfs: show a friendly message when pushing lfs to a server without lfs enabled Upfront disclaimer: I don't know anything about the wire protocol, and this was pretty much cargo-culted from largefiles, and then clonebundles, since it seems more modern. I was surprised that exchange.push() will ensure all of the proper requirements when exchanging between two local repos, but doesn't care when one is remote. All this new capability marker does is inform the client that the extension is enabled remotely. It may or may not contain commits with external blobs. Open issues: - largefiles uses 'largefiles=serve' for its capability. Someday I hope to be able to push lfs blobs to an `hg serve` instance. That will probably require a distinct capability. Should it change to '=serve' then? Or just add an 'lfs-serve' capability then? - The flip side of this is more complicated. It looks like largefiles adds an 'lheads' command for the client to signal to the server that the extension is loaded. That is then converted to 'heads' and sent through the normal wire protocol plumbing. A client using the 'heads' command directly is kicked out with a message indicating that the largefiles extension must be loaded. We could do similar with 'lfsheads', but then a repo with both largefiles and lfs blobs can't be pushed over the wire. Hopefully somebody with more wire protocol experience can think of something else. I see 'x-hgarg-1' on some commands in the tests, but not on heads, and didn't dig any further.

lfs: allow non-lfs exchanges when the extension is only enabled on one side Once the 'lfs' requirement is added, the extension must be loaded on both sides, and changegroup3 used. But there's no reason that I can see for bailing with cryptic errors if lfs is not required, but randomly enabled somewhere.

lfs: add the 'lfs' requirement in the changegroup transaction introducing lfs A hook like this is how largefiles manages to do the same. Largefiles uses a changegroup hook, but this uses pretxnchangegroup because that actually causes the transaction to rollback in the unlikely event that writing the requirements out fails. Sadly, the requires file itself isn't rolled back if a subsequent hook fails, but that seems trivial. Now that commit, changegroup and convert are covered, I don't think there's any way to get an lfs repo without the requirement. The grep exit code is blotted out of some test-lfs-serve.t tests now showing the requirement, because run-tests.py doesn't support conditionalizing the exit code.

test-lfs: add tests covering http exchanges This tries to test every combination of having the extension enabled/disabled on each side, and then push/pull/clone/identify lfs and non-lfs content. SSH is ignored here, because there's enough going on as it is. The root issue here is again that requirements are not exchanged and preserved on push/pull/clone. Doing so should eliminate the cryptic error messages when using `hg serve`. The 500 server error is triggered by "ValueError: no common changegroup version", because the extension forces changegroup3. Or, if changegroup3 is enabled manually, it is triggered by "abort: missing processor for flag '0x2000'!". Sadly, run-tests.py doesn't support conditionalizing the exit code like it does lines of output. Therefore, a couple of tests blot out the exit code by appending "|| true", since these failures will go away shortly anyway.

test-lfs: add tests covering local exchanges The root issue here is that requirements are not exchanged and preserved on push/pull. This can be handled with a changegroup hook. Testing for remote exchanges is much more extensive (it's possible for one process or the other to not have the extension loaded at all), so it is added separately.

smartset: split generatorset classes to avoid cycle I uncovered a cycle manifesting in a memory leak by running `hgperfrevset '::tip'`. The cycle was due to generatorset.__init__ assigning a bound method to self.__contains__. Internet sleuthing revealed that assigning a bound method to an instance attribute always creates a cycle. This commit creates two new variants of generatorset for the special cases of ascending and descending generators. The special implementations of __contains__ have been extracted to these classes where they are defined as __contains__. generatorset now implements __new__ and changes the spawned type to one of the new classes if needed. Differential Revision: https://phab.mercurial-scm.org/D1780

merge: raise before running mergedriver if using IMM Merge driver scripts run in the working copy, so disable with IMM for now. Differential Revision: https://phab.mercurial-scm.org/D1781

scmutil: use a tuple of possible values instead of using startswith() This patch also adds a review comment which is helpful as inline comment. Differential Revision: https://phab.mercurial-scm.org/D1761

show: use revlog function to compute length of the longest shortest node As the core part of shortest() was extracted at 448725a2ef73, we no logner need a templater.

commands: use the new API to access hidden changesets in various commands In previous patches, we have added an internal API to unhide hidden changesets. This patch makes the following command use that api in nowarn mode i.e. there will be no warning while accessing hidden changesets. cat, diff, export, files, heads, identify, log, manifest, parents, status This patch also adds test demonstarting the behaviour. .. feature:: Accessing hidden changesets Set config option 'experimental.directaccess = True' to access hidden changesets from read only commands. Differential Revision: https://phab.mercurial-scm.org/D1735

scmutil: add utility fn to return repo object with user passed revs unhidden There has been a need for accessing hidden changesets by default without passing --hidden. This is currently done using the directaccess extension but is bit hacky. This patch adds a utility function to return a repo object having user passed revisions unhidden. This functionality will live behind a config option and won't be the default behaviour. There is also a config option added by this patch which tells whether we want to unhide only those revisions whose hashes are passed or should we consider revisions numbers also. Differential Revision: https://phab.mercurial-scm.org/D1733

repoview: add a new filtername for accessing hidden commits This patch adds a new filter 'visible-hidden' for repository and will be used to return a repo object with user passed revisions unhidden. Unlike the directaccess extension in fb-hgext and previous series adding the functionality, this time we introduce only one new filter as whether to warn user or not is handled by scmutil.unhidehashlikerevs(). Differential Revision: https://phab.mercurial-scm.org/D1734

revsetlang: add utility function to return hash like symbols from the tree Functionalities like unhiding changesets whose rev/hash is passed by the user required the knowledge of rev/hashes in the user provided specs. This patch adds functions which can parse tree object and return a list of such values. Differential Revision: https://phab.mercurial-scm.org/D1732

repoview: add visibilityexception argument to filterrevs() and related fns After this patch, filterrevs() can take an optional argument visibilityexceptions which is a set of revs which should be exception to being hidden. The visibilityexceptions will be passed to the function computing hidden revisions for that filtername and are considered there while calculating the set of hidden revs. Differential Revision: https://phab.mercurial-scm.org/D1747

repoview: add visibilityexceptions as an optional argument to repo.filtered() This will help us in having an API where we can pass the filtername and the visibilityexceptions to get a new repo object. Visibility exceptions are the revs which must be visible even they should in theory belong to the hidden set. They are required as there has been desire to have a functionality to access hidden changesets using certain commands without passing --hidden. After this patch we can make those changesets visibility exceptions so that we can access them without requiring a unfiltered repo. Differential Revision: https://phab.mercurial-scm.org/D1746

streamclone: move wire protocol status code from wireproto command This consolidates the code for the streaming clone wire protocol format into streamclone.py. It also eliminates a generator wrapper, which might make streaming clones slightly faster. Differential Revision: https://phab.mercurial-scm.org/D1754

histedit: add ui.log for action count Differential Revision: https://phab.mercurial-scm.org/D1751

run-tests: extract sorting of tests to own function TestRunner._run() is a large function and is difficult to follow. Let's extract the test sorting to its own function to make it shorter. When I refactored run-tests.py several years ago, I put a lot of functionality in methods. The prevailing Mercurial style is to use functions - not classes - where possible. While refactoring the code, I decided to undo this historical mistake of mine by moving the code to a standalone function. Differential Revision: https://phab.mercurial-scm.org/D1750

run-tests: remove dead code related to temp directory Differential Revision: https://phab.mercurial-scm.org/D1749

rebase: switch ui.log calls to common style The old style raised errors in some cases. Differential Revision: https://phab.mercurial-scm.org/D1748

hgweb: link to successors of obsoleted changesets _siblings() prepare various useful properties to use in templates. This function usually prepares parents and children of changesets for use in hgweb templates, but it can be used for successors too. It's needed because item['successors'] is a _hybrid object that works well when used in regular templates, but in hgweb templates work slightly differently and can't get hex nodes of the successors, which are required for these links to work.

hgweb: display fate of obsolete changesets Operations that obsolete changesets store enough metadata to explain what happened after the fact. One way to get that metadata is showsuccsandmarkers function, which returns a list of successors of a particular changeset and appropriate obsolescence markers. Templates have a set of experimental functions that have names starting with obsfate. This patch uses some of these functions to interpret output of succsandmarkers() and produce human-friendly messages that describe what happened to an obsolete changeset, e.g. "pruned" or "rewritten as 6:3de5eca88c00". In commonentry(), succsandmarkers property is made callable so it's only executed on demand; this saves time when changeset is not obsolete, and also in e.g. /shortlog view, where there are a lot of changesets, but we don't need to show each and every one in detail. In spartan theme, succsandmarkers is used instead of the simple "obsolete: yes", in other themes a new line is added to /rev page.

test-ssh: stabilize for Windows Previously, this complained: remote: '.' is not recognized as an internal or external command, remote: operable program or batch file. Making this a python script apparently revealed some races[1]. Thanks to Yuya for suggesting this. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-December/109094.html

templater: register keywords to defaults table Since the keywords are permanent, there should be no need to pass them by a temporary mapping.

templater: drop unneeded resources from conflict-marker data Follow-up for 32c278eb876f and f1c54d003327.

templater: rewrite docstring of templater.__init__() More importantly, this patch adds what the cache is.

rebase: don't take out a dirstate guard for in-memory rebase Since IMM doesn't affect the dirstate, it's not needed, and might be faster. Differential Revision: https://phab.mercurial-scm.org/D1745

test-split: stabilize for Windows - $PYTHON needs to be quoted when used as an executable in $HGEDITOR. This avoids the error "'c' is not recognized as an internal or external command". - seq.py is printing out CRLF, and then the subsequent `sed` script seems to convert to LF on MSYS. IDK if python print statements can be made to print LF on Windows, and I'm pretty sure CRLF is baked into some other tests. - A stray glob was causing the 'obsstore-off' case to report 'no result code from test'. - When I ran with --debug, the `hg diff` commands in the test both printed color sequences, and paused the output as it was run through the pager.

lfs: use ui.note() and ui.debug() instead of ui.write() and their flags Even though the upload/download message is still in a ui.verbose check, I switched that to ui.note() too so that the 'ui.note' label is applied. The debug message is no longer marked for translation because check-code complained.

lfs: only hardlink between the usercache and local store if the blob verifies This fixes the issue where verify (and other read commands) would propagate corrupt blobs. I originalled coded this to only hardlink if 'verify=True' for store.read(), but then good blobs weren't being linked, and this broke a bunch of tests. (The blob in repo5 that is being corrupted seems to be linked into repo5 in the loop running dumpflog.py prior to it being corrupted, but only if verify=False is handled too.) It's probably better to do a one time extra verification in order to create these files, so that the repo can be copied to a removable drive. Adding the same check to store.write() was only for completeness, but also needs to do a one time extra verification to avoid breaking tests.

lfs: verify lfs object content when transferring to and from the remote store This avoids inserting corrupt files into the usercache, and local and remote stores. One down side is that the bad file won't be available locally for forensic purposes after a remote download. I'm thinking about adding an 'incoming' directory to the local lfs store to handle the download, and then move it to the 'objects' directory after it passes verification. That would have the additional benefit of not concatenating each transfer chunk in memory until the full file is transferred. Verification isn't needed when the data is passed back through the revlog interface or when the oid was just calculated, but otherwise it is on by default. The additional overhead should be well worth avoiding problems with file based remote stores, or buggy lfs servers. Having two different verify functions is a little sad, but the full data of the blob is mostly passed around in memory, because that's what the revlog interface wants. The upload function, however, chunks up the data. It would be ideal if that was how the content is always handled, but that's probably a huge project. I don't really like printing the long hash, but `hg debugdata` isn't a public interface, and is the only way to get it. The filelog and revision info is nowhere near this area, so recommending `hg verify` is the easiest thing to do.

lfs: narrow the exceptions that trigger a transfer retry The retries were added to workaround TCP RESETs in fb-experimental fc8c131314a9. I have no idea if that's been debugged yet, but this wide net caught local I/O errors, bad hostnames and other things that shouldn't be retried. The next patch will validate objects as they are uploaded, and there's no need to retry those errors. The spec[1] does mention that certain http errors can be retried, including 500. But let's work through the corruption detection issues first. [1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md

test-lfs: add tests around corrupted lfs objects These are mostly tests against file:// based remote stores, because that's what we have the most control over. The test uploading a corrupt blob to lfs-test-server demonstrates an overly broad exception handler in the retry loop. A corrupt blob is actually transferred in a download, but eventually caught when it is accessed (only after it leaves the corrupt file in a couple places locally). I don't think we want to trust random 3rd party implementations, and this would be a problem if there were a `debuglfsdownload` command that simply cached the files. And given the cryptic errors, we should probably validate the file hash locally before uploading, and also after downloading.

lfs: add note messages indicating what store holds the lfs blob The following corruption related patches were written prior to adding the user level cache, and it took awhile to track down why the tests changed. (It generally made things more resilient.) But I think this will be useful to the end user as well. I didn't make it --debug level, because there can be a ton of info coming out of clone/push/pull --debug. The pointers are sorted for test stability. I opted for ui.note() instead of checking ui.verbose and then using ui.write() for convenience, but I see most of this extension does the latter. I have no idea what the preferred form is.

tests: teach `f` to handle sha256 checksums

tests: fix a bug in `f` that prevented calculating sha1sum on a file

templater: look up symbols/resources as if they were separated (issue5699) It wouldn't be easy to split the mapping dict into (symbols, resources). This patch instead rejects invalid lookup taking resources.keys() as source of truth. The doctest is updated since mapping['repo'] is now reserved for a repo object.

templater: move repo, ui and cache to per-engine resources

templater: keep default resources per template engine (API) This allows us to register a repo object as a resource in hgweb template, without loosing '{repo}' symbol: symbol('repo') -> mapping['repo'] (n/a) -> defaults['repo'] resource('repo') -> mapping['repo'] (n/a) -> resources['repo'] I'm thinking of redesigning the templatekw API to take (context, mapping) in place of **(context._resources + mapping), but that will be a big change and not implemented yet. props['templ'] is ported to the resources dict as an example. .. api:: mapping does not contain all template resources. use context.resource() in template functions.

templater: look up mapping table through template engine These functions are stub for symbol/resource separation. This series is intended to address the following problems: a) internal data may be exposed to user (issue5699) b) defaults['repo'] (a repository name) will conflict with mapping['repo'] (a repo object) in hgweb

debug: add newlines at the end of three locations that appear to need it Differential Revision: https://phab.mercurial-scm.org/D1720

debug: remove an 'if ui.debug()' that is not doing anything ui.debug() does not return a value. Differential Revision: https://phab.mercurial-scm.org/D1719

paper: minor adjustments to table styles Adding a bit of padding to table columns on e.g. /log means content and headers are better aligned: headers already have this padding. Right margin is removed from #changesetEntry th because elements with display: table-cell (such as <th>) ignore margins anyway.