util: disable hardlink for copyfile if fstype is outside a whitelist Since osutil.getfstype is available, use it to detect filesystem types. The whitelist currently includes common local filesystems on Linux where they should have good hardlink support. We may add new filesystems for other platforms later.

revlog: use pycompat.maplist to eagerly evaluate map on Python 3 According to Pulkit, this should fix `hg status --all` on Python 3.

py3: stop exporting urlparse from pycompat and util (API) There are no consumers of this in tree. Functions formerly available on this object/module can now be accessed via {pycompat,util}.urlreq.

check-code: recommend util.urlreq when importing urlparse

tests: use urlreq in tinyproxy.py This is our last consumer of util.urlparse.

bugzilla: use util.urlreq.urlparse And stop saving a module variable because it shouldn't be necessary.

pycompat: define urlreq.urlparse and urlreq.unparse aliases Currently, we export urlparse via util.urlparse then call util.urlparse.urlparse() and util.urlparse.urlunparse() in a few places. This is the only url* module exported from pycompat, making it a one-off. So let's transition to urlreq to match everything else. Yes, we double import "urlparse" now on Python 2. This will be cleaned up in a subsequent patch. Also, the Python 3 functions trade in str/unicode not bytes. So we'll likely need to write a custom implementation that speaks bytes. But moving everyone to an abstracted API is a good first step.

pycompat: remove urlunquote alias It is duplicated by urlreq.unquote and is unused. Kill it. We retain the imports because it is re-exported via util.urlparse, which is used elsewhere. Since we no longer access attributes of urlparse at module load time, this change /should/ result in that module reverting to a lazy module.

util: use urlreq.unquote pycompat.urlreq.unquote and pycompat.urlunquote effectively alias the same thing. pycompat.urlunquote is only used once in the code base. So let's switch to urlreq.unquote. "Effectively" in the above paragraph is because pycompat.urlreq.unquote aliases urllib.unquote and pycompat.urlunquote aliases urlparse.unquote on Python 2. You might think one of urllib.unquote and urlparse.unquote is an alias to the other, but you would be incorrect. In fact, these functions are copies of each other. There is even a comment in the CPython source code saying to keep them in sync. You can't make this up.

pycompat: alias urlreq.unquote to unquote_to_bytes Previously, urlreq.unquote aliased to urllib.parse.unquote, which returned a str/unicode. We like bytes, so switch urlreq.unquote to dispatch to urllib.parse.unquote_to_bytes. This required a minor helper function to register an alias under a different name from which it points. If this turns into a common pattern, we could likely teach _registeralias to accept tuple values defining the mapping. Until then, I didn't feel like adding complexity to _registeralias.

revsetlang: portably turn int into bytestring

osutil: export a "getfstype" method This patch exports the "getfstype" method. So we can use it to enable hardlinks for known safe filesystems. The patch was tested manually via debugshell on a Linux system. "mercurial.osutil.getfstype" works as expected. It's hard to mount filesystem on user-space easily. I will add a test for real hardlink support to indirectly test this patch, after turning on real hardlinks support for certain whitelisted filesystems.

osutil: add a C function getting filesystem type Currently it only has Linux filesystems, according to my Linux manpage, built at 2016-03-15. The code uses "if" instead of "switch" because there could be some duplicated values.

setup: test some header files The next patch will use "statfs", which requires different header files on different platforms. Linux: sys/vfs.h or sys/statfs.h FreeBSD or OSX: sys/param.h and sys/mount.h Therefore test them so we can include the correct ones.

setup: detect statfs statfs is not defined by POSIX but is available in various systems to help decide filesystem type. Let's detect it and set the macro HAVE_STATFS.

setup: add a function to test header files

setup: split "hasfunction" to test arbitrary code The next patch wants to test include files.

rebase: add flag to require destination In some mercurial workflows, the default destination for rebase does not always work well and can lead to confusing behavior. With this flag enabled, every rebase command will require passing an explicit destination, eliminating this confusion.

update: add flag to require update destination In some mercurial workflows, the default destination for update does not always work well and can lead to confusing behavior. With this flag enabled, every update command will require passing an explicit destination, eliminating this confusion.

mq: reject new patch name containing leading/trailing whitespace We could create a patch of such name, but it wouldn't be processed properly by mq as parseseries() strips leading/trailing whitespace. The test of default message (added by b9a16ed5acec) is no longer be useful so removed. This issue was reported as: https://bitbucket.org/tortoisehg/thg/issues/4693/

shelve: rename stripnodes to nodestoprune Since we are introducing obs-based shelve, we are no longer stripping temporary nodes, we are obsoleting them. Therefore it looks like stipnodes would be a misleading name, while prune has a connotaion of "strip but with obsolescense", so nodestoprune seems like a good rename.

shelve: add an ability to write key-val data to a new type of shelve files Obsolescense-based shelve only needs metadata stored in .hg/shelved and if feels that this metadata should be stored in a simplekeyvaluefile format for potential extensibility purposes. I want to avoid storing it in an unstructured text file where order of lines determines their semantical meanings (as now happens in .hg/shelvedstate. .hg/rebasestate and I suspect other state files as well). Not included in this series, I have ~30 commits, doubling test-shelve.t in size and testing almost every tested shelve usecase for obs-shelve. Here's the series for the curious now: http://pastebin.com/tGJKx0vM I would like to send it to the mailing list and get accepted as well, but: 1. it's big, so should I send like 6 patches a time or so? 2. instead of having a commit per test case, it more like a commit per some amount of copy-pasted code. I tried to keep it meaningful and named commits somewhat properly, but it is far from this list standards IMO. Any advice on how to get it in without turning it into a 100 commits and spending many days writing descriptions? 3. it makes test-shelve.t run for twice as long (and it is already a slow test). Newest test-shelve.r runs for ~1 minute.

scmutil: add a simple key-value file helper The purpose of the added class is to serve purposes like save files of shelve or state files of shelve, rebase and histedit. Keys of these files can be alphanumeric and start with letters, while values must not contain newlines. In light of Mercurial's reluctancy to use Python's json module, this tries to provide a reasonable alternative for a non-nested named data. Comparing to current approach of storing state in plain text files, where semantic meaning of lines of text is only determined by their oreder, simple key-value file allows for reordering lines and thus helps handle optional values. Initial use-case I see for this is obs-shelve's shelve files. Later we can possibly migrate state files to this approach. The test is in a new file beause I did not figure out where to put it within existing test suite. If you give me a better idea, I will gladly follow it.

graphlog: draw multiple edges towards null node (issue5440) Before, edge (r, null) was processed only once by newparents. However what we really need is just stripping the edge (null, null).

merge with stable

localrepo: use tryunlink

vfs: use tryunlink

server: use tryunlink

dirstate: use tryunlink

debug: use tryunlink

chgserver: use tryunlink

bookmarks: use tryunlink

shelve: use tryunlink

vfs: add tryunlink method Thoughout hg code, we see a pattern of attempting to remove a file and then catching and ignoring any errors due to a missing file in the calling code. Let's unify this pattern in a single implementation in the vfs layer.

util: use tryunlink in unlinkpath We just introduced a func to attempt a file removal. Start using it.

util: add tryunlink function Throughout mercurial cdoe, there is a common pattern of attempting to remove a file and ignoring ENOENT errors. Let's move this into a common function to allow for cleaner code.

util: unify unlinkpath Previously, there were two slightly different versions of unlinkpath between windows and posix, but these differences were eliminated in previous patches. Now we can unify these two code paths inside of the util module.

posix: use local reference to removedirs We have a local reference to os.removedirs in module scope, but we still used os.removedirs inside functions. This changes util to use the local reference, which will pave the way for combining duplicated code in future patches.

posix: use local reference to unlink We have a local reference to os.unlink in module scope, but we still used os.unlink inside functions. This changes util to use the local reference, which will pave the way for combining duplicated code in future patches.

localrepo: improve vfs documentation At the beginning of March, I promised Yuya that I would follow up a comment I made on a patch with improved documention for these vfs objects. Also hat tip to Pierre-Yves for adding the documentation here in the first place.

ui: restrict length of autogenerated blocked tags Long autogenerated blocked tags tend to be because the command has an absolute path; at Facebook, we've had a few where the tag is thousands of characters long (in association with the mergedriver). Change the default to use a suffix of a command as the default tag, limiting us to 85 characters (for a 100 character tag). This is long enough to overflow a standard terminal (thus be obviously autogenerated), but short enough to be readable.

commit: keep opts as byteskwargs as much as possible This fixes many flag-handling bugs on 'hg commit' in Python 3.

util: reference __main__ in sys.modules as a sysstr

ui: convert to/from Unicode on Python 3 in ui.editor() I considered making this I/O be done in terms of bytes, but that would cause an observable regression for Windows users, as non-binary-mode open does EOL conversion there. There are probably new encoding dragons lurking here, so we may want to switch to using binary mode and doing EOL conversion ourselves.

manifest: refer to bytestrings as bytes, not str Required on Python 3.

manifest: use node.hex instead of .encode('hex') The latter doesn't work on Python 3.

pure: use int instead of long Similar to the recent 73aa13bc8dac (revlog: use int instead of long, 2017-03-19).

histedit: inline cleanupnode Move "cleanupnode" (unsafe strip) into "safecleanupnode" so it's impossible to call the unsafe function directly. This helps reduce future programming errors.

histedit: use safecleanupnode in _aborthistedit (issue5500) Now nobody in histedit calls the unsafe cleanupnode directly.

histedit: use safecleanupnode in _finishhistedit This simplifies code a lot.

histedit: add a method to cleanup nodes safely The new method will decide between: - cleanupnode, which calls the unsafe repair.strip - create obsmarkers Ideally, nobody calls "cleanupnode" directly except for "safecleanupnode".

py3: prove hg status works

localrepo: use node.hex instead of awkward .encode('latin1') Spotted as an option by Yuya. Thanks!

py3: prove hg config works

templater: make pad() strip color codes before computing width (issue5416) Tested in ANSI mode. We might have to extend _ansieffectre to support terminfo mode.

templater: make pad() compute actual width str.ljust() and .rjust() are based on byte length, which are valid only for ASCII characters.

templater: reject bad fillchar argument passed to pad() Otherwise TypeError would be raised.

color: insert color code after every "\e[0m" (issue5413) This assumes the last color wins, tested in ANSI mode. I guess terminfo mode would work in the same way.

debugtemplate: pass ui to templater so label() works Follows up 3356bf61fa25.

subrepo: move prompts out of the if (issue5505) Prompts weren't available in the else clause

merge: remove unnecessary matcher checks As part of changing manifest.diff to accept a matcher, a previous patch added matcher calls to each location in merge.manifestmerge that tested if 'x in mf' to maintain the same behavior as before. After analyzing it further, this matcher call isn't needed, and in fact hurts future patches ability to use the matcher here. Basically, all these 'if x in mf' checks were checking if a matched file's copy source was in the matcher as well. This meant if you passed a matcher for just file foo, it would not return file bar even if foo was a copy of bar. Since manifestmerge cares about copy information, let's allow all lookups of copy sources. We also update one spot with a 'is not None' check, since it wasn't obvious that the value could sometimes be None before, which broke when we called matcher(None). A future patch adds matcher optimizations to manifestmerge which causes this code path to get covered by existing tests.

rebase: use one dirstateguard for entire rebase Recently we switched rebases to run the entire rebase inside a single transaction, which dramatically improved the speed of rebases in repos with large working copies. Let's also move the dirstate into a single dirstateguard to get the same benefits. This let's us avoid serializing the dirstate after each commit. In a large repo, rebasing 27 commits is sped up by about 20%. I believe the test changes are because us touching the dirstate gave the transaction something to actually rollback.

histedit: add histedit.singletransaction config option This adds an option (which defaults to False) to run entire histedits in a single transaction. This results in 20-25% faster histedits in large repos where transaction startup cost is expensive. I didn't want to enable this by default because it has some unfortunate side effects. For instance, if a pretxncommit hook throws midway through the histedit, it will rollback the entire histedit and lose any progress the user had made. Same if the user aborts editting a commit message. It's still worth turning this on for large repos, but probably not for normal sized repos. Long term, once we have inmemory merging, we could do the entire histedit in memory, without a transaction, then we could selectively rollback just parts of it in the event of an exception. Tested it by running the tests with `--extra-config-opt=histedit.singletransaction=True`. The only failure was related to the hook rollback issue I mention above.

histedit: pop action after the action is completed We only want to pop the action after the action is completed, since if the action aborts part way through we want it to remain at the front of the list so continue/abort will start with it. Previously we relied on the fact that we only serialized the state file at the beginning of the action, so the pop wasn't serialized until the next iteration of the loop. In a future patch we will be adding a large transaction around this area, which means if we pop the list early it might get serialized if the action throws a user InterventionRequired error, at which point the action is not in the list anymore. So let's only pop it once the action is really truly done.