crecord: use ProgrammingError

transaction: use ProgrammingError

bundle2: use ProgrammingError

merge: use ProgrammingError

repair: use ProgrammingError

py3: abuse r'' to preserve str-ness of literals passed to __setattr__()

py3: fix slicing of byte string in revlog.compress() I tried .startswith('\0'), but data wasn't always a bytes nor a bytearray.

py3: use bytes() to cast to immutable bytes in changelog.appender.write()

py3: use bytes() to cast to immutable bytes in pure.bdiff.bdiff()

bdiff: drop support for array.array argument from pure.bdiff.bdiff() Thanks to 2a18e9e6ca43, we no longer pass array.array('c') object to bdiff().

histedit: test that an aborted histedit can be rerun (with obsolete) In the future, this should help catching issue as the one introduced in 6f0b7475cf9a.

histedit: backout changeset 6f0b7475cf9a Before 6f0b7475cf9a, histedit (like rebase) was only creating markers on final success from the old-rewritten node to the newly created nodes (as of before 6f0b7475cf9a). In case of abort the aborted attempt were stripped to restore the repository in its state prior to the attempt. This use of strip was on purpose. Using markers in this case introduces various issues. The main one is that keeping the partial result of histedit as obsolete prevents us to recreates the same nodes in a second attempt. The same operation will lead to an identical results, using an identical node that already exists in the repository as obsolete. To conclude, we cannot and should not switch to obsolescence markers creation on histedit --abort and we backout 6f0b7475cf9a. A test to catch this class of issue will be introduced in the next changeset.

histedit: backout changeset 2b599f5468a4 Its parent is about to be backedout so this one needs to be removed too.

revsetlang: fix _quote on int on python3 Thanks to Yuya for spotting the need.

runtests: unset editor and pager related environment variables Those environment variables could affect some configuration and future tests. Drop them to avoid issues.

debugfsinfo: improve case-sensitive testing Previously the case-sensitive test was for the current directory, and is fragile with errors, and could remove a real file called ".debugfsinfo". This patch improves the case-sensitive testing so it test the given path using a unique temporary file, and does not crash on errors.

debugfsinfo: show fstype for given path

test-check-py3-commands: cleanup tests related to `hg status` We were testing hg status on Python 3 at two places in the test. Cleaned up one of them.

diff: use pycompat.{byteskwargs, strkwargs} to switch opts b/w bytes and str

patch: make regular expressions bytes by adding b''

dispatch: use pycompat.maplist() instead of map() to get a list

color: fix grammar in help text

statfs: detect more filesystems on Linux Previously, the code only has what manpager says. In <linux/magic.h>, there are more defined. This patch adds filesystems that appear in the current Arch Linux's /proc/filesystems (autofs, overlay, securityfs) and f2fs, which was seen in news.

repair: use context manager for lock management If repo.lock() raised inside of the try block, 'tr' would have been None in the finally block where it tries to release(). Modernize the syntax instead of just winching the lock out of the try block. I found several other instances of acquiring the lock inside of the 'try', but those finally blocks handle None references. I also started switching some trivial try/finally blocks to context managers, but didn't get them all because indenting over 3x for lock, wlock and transaction would have spilled over 80 characters. That got me wondering if there should be a repo.rwlock(), to handle locking and unlocking in the proper order. It also looks like py27 supports supports multiple context managers for a single 'with' statement. Should I hold off on the rest until py26 is dropped?

gitweb: use monospace font for commit messages Commit messages often contain vertically aligned text. The default paper style already uses monospace fonts for rendering commit messages. And, AFAICT, a number of Git servers also render commit messages with monospace. It seems like the reasonable thing to do. This commit converts all instances of the full commit message in the gitweb style to render with monospace.

pager: improve support for various flavors of `more` on Windows Hardcoding 'more' -> 'more.com' means that 'more.exe' from MSYS would need to be configured with its *.exe extension. This will resolve to either one, as cmd.exe would have done if the command ran through the shell. Something that's maybe problematic with this is it comes after 'pageractive' and various ui configs have been set by the calling method. But the other early exits in this method don't undo those changes either.

statfs: avoid static allocation Previously we have "static struct statfs" to return a string. That is not multiple-thread safe. This patch moves the allocation to the caller to address the problem.

statfs: change Linux feature detection Previously we check three things: "statfs" function, "linux/magic.h" and "sys/vfs.h" headers. But we didn't check "struct statfs" or the "f_type" field. That means if a system has "statfs" but "struct statfs" is not defined in the two header files we check, or defined without the "f_type" field, the compilation will fail. This patch combines the checks (2 headers + 1 function + 1 field) together and sets "HAVE_LINUX_STATFS". It makes setup.py faster (less checks), and more reliable (immutable to the issue above).

rebase: don't require destination if commands.rebase.requiredest=False

tests: add tests with commands.{update,rebase}.requiredest=False This shows how rebase is currently broken with commands.rebase.requiredest=False.

rebase: move state serialization to use unfiltered repo Now that rebasestate is serialized as part of the transaction, the repo state it sees is the version at the end of the transaction, which may have hidden nodes. Therefore, it's possible parts of the rebase commit set are no longer visible by the time the transaction is closing, which causes a filtered revision error in this code. I don't think state serialization should be blocked from accessing commits it knows exist, especially if all it's trying to do is get the hex of them, so let's use an unfiltered repo here. Unfortunately, the only known repro is with the fbamend Facebook extension, so I'm not sure how to repro it in core Mercurial for a test.

largefiles: avoid redundant standin() invocations There are some code paths, which apply standin() on same value multilpe times instead of using already standin()-ed value. "fstandin" is common name for "path to standin file" in lfutil.py, to avoid shadowing "standin()".

largefiles: replace hashrepofile by hashfile (API) There is only one user for the former, and repo.wjoin()-ed value is alread known by that user.

largefiles: call readstandin() with changectx itself instead of rev or node readstandin() takes "node" argument to get changectx by "repo[node]". There are some readstandin() invocations, which use ctx.node(), ctx.rev(), or '.' as "node" argument above, even though corresponded changectx object is already looked up on caller side. This patch calls readstandin() with already known changectx itself, to avoid meaningless re-construction of changectx (indirect case via copytostore() is also included). BTW, copytostore() uses "rev" argument only for readstandin() invocation. Therefore, this patch also renames it to "revorctx" to indicate that it can take not only revision ID or so but also changectx, for readability.

largefiles: omit redundant splitstandin() invocations There are 3 splitstandin() invocations in updatestandin() for same "standin" value.

largefiles: replace splitstandin() by isstandin() to omit str creation If splitstandin()-ed str itself isn't used, isstandin() should be used instead of it, to omit meaningless str creation.

largefiles: omit redundant isstandin() before splitstandin() There are many isstandin() invocations before splitstandin(). The former examines whether specified path starts with ".hglf/". The latter returns after ".hglf/" of specified path if it starts with that prefix, or returns None otherwise. Therefore, value returned by splitstandin() can be used for replacement of preceding isstandin(), and this replacement can omit redundant string comparison after isstandin().

misc: update descriptions about removed file for filectxfn Since 650b5b6e75ed, filectxfn for memctx should return None for removed file instead of raising IOError.

osx: always purge build/mercurial before starting build This prevents stray files from previous builds from polluting newer builds.

osx: use more paranoid quoting in test

osx: install completion scripts using install(1) to avoid umask badness This actually fixes some tests that were showing us umask badness that had been overlooked.

merge with stable

py3: prove `hg {add,addremove,commit} all work We can't do a second commit in a repo yet, because pure-Python bdiff is broken on Python 3. That is probably a good next step.

revsetlang: perform quoting using ui.escapestr instead of repr() This changes one of the doctest results, but I'm pretty sure on inspection that it's an equivalent result.

revsetlang: add docstring with some tests to _quote

revsetlang: move quoting function to not be a closure I'm about to change the implementation here and I'd like to add some doctests, which means this needs to not be hidden inside another function.

revsetlang: portably bytestring-ify another pair of int() calls

checkcode: enforce lowercase for extension docstring title This will ensure new extensions are consistent and `hg help -e` has a consistent output. I have to add a new group since the normal "pypats" will be filtered by "pyfilters", which will remove comments and docstrings.

logtoprocess: use lowercase for docstring title

journal: use lowercase for docstring title See the previous patch for why.

automv: use lowercase for docstring title The first line of docstring is used in `hg help -e`. Use lowercase to be consistent with most extensions.

util: enable hardlink for some BSD-family filesystems Since we can now detect filesystems on FreeBSD and OSX. Add their major filesystems (ufs, zfs for FreeBSD; hfs for OSX) to the hardlink whitelist.

osutil: report fstype for BSD and OSX

setup: use a more strict way to test BSD or OSX's statfs We want to use the `f_fstypename` field to get the filesystem type. Test it directly. The new macro HAVE_BSD_STATFS implys the old HAVE_SYS_MOUNT_H and HAVE_SYS_PARAM_H. So the latter ones are removed.

debugfsinfo: use util.getfstype This changes the behavior slightly. It now always prints fstype, regardless of whether osutil.getfstype exists.

util: use util.getfstype

util: add a getfstype method The util version is a thin wrapper of the osutil version, which is not always available.

pager: fix the invocation of `more` on Windows After 9335dc6b2a9c, with 'shell' being (mostly) set to False, invoking `more` no longer worked. Instead, a warning was printed and the pager was disabled. Invoking `more.com` works. Since a user may have configured 'pager.pager=more', do this substitution at the end. Surprisingly, `more` does allow for arguments, so those are preserved. This also allows `more` to work in MSYS. Setting 'shell=False' runs the executable via CreateProcess(), which has rather wonky rules for resolving an executable without an extension [1]. Resolving to *.com is not among them. Since 'shell=True' yields a cryptic error for a bad $PAGER, and a *.exe program will work without specifying the extension, sticking with current 'shell=False' seems like the right thing to do. I don't think there are any other *.com pagers out there, so this one special case seems OK. If somebody wants to do something crazy that requires cmd.exe, I was able to get normal paged output with 'pager.pager="cmd.exe /c more"'. I assume you can replace `more` with *.bat, *.vbs or various other creatures listed in $PATHEXT. [1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

help: format ``commands`` heading correctly The number of dashes under it needs to match exactly for it to be rendered as a heading. Without this change, the dashes end up on the same line as "commands", and "hg help config.commands" does not work.

convert: fix the handling of empty changlist descriptions in P4 Empty changelist descriptions are valid in Perforce. If we encounter one of them we are currently running into an IndexError. In case of empty commit messages set the commit message to **empty changelist description**, which follows Perforce terminology.

status: support commands.status.relative config When the config is set to true, status output becomes relative to the working directory. This has bugged me since I started using hg and it turns it is sillily simple to support it (unless I missed something, of course). We could also add a --relative flag, but I would personally always want that on, and I haven't heard any use for having it sometimes on, so this patch only lets you enable it via config.

plain: ignore [commands] config We only have commands.{update,rebase}.requiredest so far. We should clearly ignore those two if HGPLAIN is in effect, and it seems like we should ignore any future config that will be added in [commands] since that is about changing the behavior of commands. Thanks to Yuya for suggesting to centralize the code in ui.py. While at it, remove the unnecessary False values passed to ui.configbool() for the aforementioned config options.

changegroup: store old heads as a set Previously, the "oldheads" variable was a list. On a repository at Mozilla with 46,492 heads, profiling revealed that list membership testing was dominating execution time of applying small changegroups. This patch converts the list of old heads to a set. This makes membership testing significantly faster. On the aforementioned repository with 46,492 heads: $ hg unbundle <file with 1 changeset> before: 18.535s wall after: 1.303s Consumers of this variable only check for truthiness (`if oldheads`), length (`len(oldheads)`), and (most importantly) item membership (`h not in oldheads` - which occurs twice). So, the change to a set should be safe and suitable for stable. The practical effect of this change is that changegroup application and related operations (like `hg push`) no longer exhibit an O(n^2) CPU explosion as the number of heads grows.

checkheads: extract obsolete post processing in its own function The checkheads function is long and complex, extract that logic in a subfunction is win in itself. As the comment in the code says, this postprocessing is currently very basic and either misbehave or fails to detect valid push in many cases. My deeper motive for this extraction is to be make it easier to provide extensive testing of this case and strategy to cover them. Final test and logic will makes it to core once done.

tests: make test-simplekeyvaluefile.py py2.6-compatible Python 2.6 unittest.TestCase does not have assertRaisesRegexp.

similar: use cheaper hash() function to test exact matches We just need a hash table {fctx.data(): fctx} which doesn't keep fctx.data() in memory. Let's simply use hash(fctx.data()) to put data out from memory, and manage collided fctx objects by list. This isn't significantly faster than using sha1, but is more correct as we know SHA-1 collision attack is getting practical. Benchmark with 50k added/removed files, on tmpfs: $ hg addremove --dry-run --time -q previous: real 12.420 secs (user 11.120+0.000 sys 1.280+0.000) this patch: real 12.350 secs (user 11.210+0.000 sys 1.140+0.000)

similar: take the first match instead of the last It seems more natural. This makes the next patch slightly cleaner.

similar: do not look up and create filectx more than once Benchmark with 50k added/removed files, on tmpfs: $ hg addremove --dry-run --time -q previous: real 16.070 secs (user 14.470+0.000 sys 1.580+0.000) this patch: real 12.420 secs (user 11.120+0.000 sys 1.280+0.000)

similar: use common names for changectx variables We generally use 'wctx' and 'pctx' for working context and its parent respectively.

similar: get rid of quadratic addedfiles.remove() Instead, build a set of files to be removed and recreate addedfiles only if necessary. Benchmark with 50k added/removed files, on tmpfs: $ hg addremove --dry-run --time -q original: real 16.550 secs (user 15.000+0.000 sys 1.540+0.000) previous: real 16.730 secs (user 15.280+0.000 sys 1.440+0.000) this patch: real 16.070 secs (user 14.470+0.000 sys 1.580+0.000)

similar: sort files not by object id but by path for stable result Perhaps the original implementation would want to sort added/removed files alphabetically, but actually it did sort fctx objects by memory location. This patch removes the use of set()s in order to preserve the order of added/removed files. addedfiles.remove() becomes quadratic, but its cost appears not dominant. Anyway, the quadratic behavior will be eliminated by the next patch. Benchmark with 50k added/removed files, on tmpfs: $ mkdir src $ for n in `seq 0 49`; do > mkdir `printf src/%02d $n` > done $ for n in `seq 0 49999`; do > f=`printf src/%02d/%05d $(($n/1000)) $n` > dd if=/dev/urandom of=$f bs=8k count=1 status=none > done $ hg ci -qAm 'add 50k files of random content' $ mv src dest $ hg addremove --dry-run --time -q original: real 16.550 secs (user 15.000+0.000 sys 1.540+0.000) this patch: real 16.730 secs (user 15.280+0.000 sys 1.440+0.000)

debugfsinfo: print fstype information Since we have osutil.getfstype, it'll be handy if "debugfsinfo" prints it.

util: enable hardlink for copyfile This patch removes the global variable "allowhardlinks" that disables hardlink in all cases, so hardlink gets enabled if the filesystem type is whitelisted. Third party extensions wanting to enable hardlink support unconditionally can replace "_hardlinkfswhitelist.__contains__".

hghave: add a check about whitelisted filesystem that supports hardlink This is needed for the test added by the next patch.

util: disable hardlink for copyfile if fstype is outside a whitelist Since osutil.getfstype is available, use it to detect filesystem types. The whitelist currently includes common local filesystems on Linux where they should have good hardlink support. We may add new filesystems for other platforms later.

revlog: use pycompat.maplist to eagerly evaluate map on Python 3 According to Pulkit, this should fix `hg status --all` on Python 3.

py3: stop exporting urlparse from pycompat and util (API) There are no consumers of this in tree. Functions formerly available on this object/module can now be accessed via {pycompat,util}.urlreq.

check-code: recommend util.urlreq when importing urlparse

tests: use urlreq in tinyproxy.py This is our last consumer of util.urlparse.

bugzilla: use util.urlreq.urlparse And stop saving a module variable because it shouldn't be necessary.

pycompat: define urlreq.urlparse and urlreq.unparse aliases Currently, we export urlparse via util.urlparse then call util.urlparse.urlparse() and util.urlparse.urlunparse() in a few places. This is the only url* module exported from pycompat, making it a one-off. So let's transition to urlreq to match everything else. Yes, we double import "urlparse" now on Python 2. This will be cleaned up in a subsequent patch. Also, the Python 3 functions trade in str/unicode not bytes. So we'll likely need to write a custom implementation that speaks bytes. But moving everyone to an abstracted API is a good first step.

pycompat: remove urlunquote alias It is duplicated by urlreq.unquote and is unused. Kill it. We retain the imports because it is re-exported via util.urlparse, which is used elsewhere. Since we no longer access attributes of urlparse at module load time, this change /should/ result in that module reverting to a lazy module.

util: use urlreq.unquote pycompat.urlreq.unquote and pycompat.urlunquote effectively alias the same thing. pycompat.urlunquote is only used once in the code base. So let's switch to urlreq.unquote. "Effectively" in the above paragraph is because pycompat.urlreq.unquote aliases urllib.unquote and pycompat.urlunquote aliases urlparse.unquote on Python 2. You might think one of urllib.unquote and urlparse.unquote is an alias to the other, but you would be incorrect. In fact, these functions are copies of each other. There is even a comment in the CPython source code saying to keep them in sync. You can't make this up.

pycompat: alias urlreq.unquote to unquote_to_bytes Previously, urlreq.unquote aliased to urllib.parse.unquote, which returned a str/unicode. We like bytes, so switch urlreq.unquote to dispatch to urllib.parse.unquote_to_bytes. This required a minor helper function to register an alias under a different name from which it points. If this turns into a common pattern, we could likely teach _registeralias to accept tuple values defining the mapping. Until then, I didn't feel like adding complexity to _registeralias.

revsetlang: portably turn int into bytestring

osutil: export a "getfstype" method This patch exports the "getfstype" method. So we can use it to enable hardlinks for known safe filesystems. The patch was tested manually via debugshell on a Linux system. "mercurial.osutil.getfstype" works as expected. It's hard to mount filesystem on user-space easily. I will add a test for real hardlink support to indirectly test this patch, after turning on real hardlinks support for certain whitelisted filesystems.

osutil: add a C function getting filesystem type Currently it only has Linux filesystems, according to my Linux manpage, built at 2016-03-15. The code uses "if" instead of "switch" because there could be some duplicated values.

setup: test some header files The next patch will use "statfs", which requires different header files on different platforms. Linux: sys/vfs.h or sys/statfs.h FreeBSD or OSX: sys/param.h and sys/mount.h Therefore test them so we can include the correct ones.

setup: detect statfs statfs is not defined by POSIX but is available in various systems to help decide filesystem type. Let's detect it and set the macro HAVE_STATFS.

setup: add a function to test header files

setup: split "hasfunction" to test arbitrary code The next patch wants to test include files.

rebase: add flag to require destination In some mercurial workflows, the default destination for rebase does not always work well and can lead to confusing behavior. With this flag enabled, every rebase command will require passing an explicit destination, eliminating this confusion.

update: add flag to require update destination In some mercurial workflows, the default destination for update does not always work well and can lead to confusing behavior. With this flag enabled, every update command will require passing an explicit destination, eliminating this confusion.

mq: reject new patch name containing leading/trailing whitespace We could create a patch of such name, but it wouldn't be processed properly by mq as parseseries() strips leading/trailing whitespace. The test of default message (added by b9a16ed5acec) is no longer be useful so removed. This issue was reported as: https://bitbucket.org/tortoisehg/thg/issues/4693/

shelve: rename stripnodes to nodestoprune Since we are introducing obs-based shelve, we are no longer stripping temporary nodes, we are obsoleting them. Therefore it looks like stipnodes would be a misleading name, while prune has a connotaion of "strip but with obsolescense", so nodestoprune seems like a good rename.

shelve: add an ability to write key-val data to a new type of shelve files Obsolescense-based shelve only needs metadata stored in .hg/shelved and if feels that this metadata should be stored in a simplekeyvaluefile format for potential extensibility purposes. I want to avoid storing it in an unstructured text file where order of lines determines their semantical meanings (as now happens in .hg/shelvedstate. .hg/rebasestate and I suspect other state files as well). Not included in this series, I have ~30 commits, doubling test-shelve.t in size and testing almost every tested shelve usecase for obs-shelve. Here's the series for the curious now: http://pastebin.com/tGJKx0vM I would like to send it to the mailing list and get accepted as well, but: 1. it's big, so should I send like 6 patches a time or so? 2. instead of having a commit per test case, it more like a commit per some amount of copy-pasted code. I tried to keep it meaningful and named commits somewhat properly, but it is far from this list standards IMO. Any advice on how to get it in without turning it into a 100 commits and spending many days writing descriptions? 3. it makes test-shelve.t run for twice as long (and it is already a slow test). Newest test-shelve.r runs for ~1 minute.

scmutil: add a simple key-value file helper The purpose of the added class is to serve purposes like save files of shelve or state files of shelve, rebase and histedit. Keys of these files can be alphanumeric and start with letters, while values must not contain newlines. In light of Mercurial's reluctancy to use Python's json module, this tries to provide a reasonable alternative for a non-nested named data. Comparing to current approach of storing state in plain text files, where semantic meaning of lines of text is only determined by their oreder, simple key-value file allows for reordering lines and thus helps handle optional values. Initial use-case I see for this is obs-shelve's shelve files. Later we can possibly migrate state files to this approach. The test is in a new file beause I did not figure out where to put it within existing test suite. If you give me a better idea, I will gladly follow it.

graphlog: draw multiple edges towards null node (issue5440) Before, edge (r, null) was processed only once by newparents. However what we really need is just stripping the edge (null, null).

merge with stable

localrepo: use tryunlink

vfs: use tryunlink

server: use tryunlink

dirstate: use tryunlink

debug: use tryunlink

chgserver: use tryunlink

bookmarks: use tryunlink

shelve: use tryunlink

vfs: add tryunlink method Thoughout hg code, we see a pattern of attempting to remove a file and then catching and ignoring any errors due to a missing file in the calling code. Let's unify this pattern in a single implementation in the vfs layer.

util: use tryunlink in unlinkpath We just introduced a func to attempt a file removal. Start using it.

util: add tryunlink function Throughout mercurial cdoe, there is a common pattern of attempting to remove a file and ignoring ENOENT errors. Let's move this into a common function to allow for cleaner code.

util: unify unlinkpath Previously, there were two slightly different versions of unlinkpath between windows and posix, but these differences were eliminated in previous patches. Now we can unify these two code paths inside of the util module.

posix: use local reference to removedirs We have a local reference to os.removedirs in module scope, but we still used os.removedirs inside functions. This changes util to use the local reference, which will pave the way for combining duplicated code in future patches.

posix: use local reference to unlink We have a local reference to os.unlink in module scope, but we still used os.unlink inside functions. This changes util to use the local reference, which will pave the way for combining duplicated code in future patches.

localrepo: improve vfs documentation At the beginning of March, I promised Yuya that I would follow up a comment I made on a patch with improved documention for these vfs objects. Also hat tip to Pierre-Yves for adding the documentation here in the first place.

ui: restrict length of autogenerated blocked tags Long autogenerated blocked tags tend to be because the command has an absolute path; at Facebook, we've had a few where the tag is thousands of characters long (in association with the mergedriver). Change the default to use a suffix of a command as the default tag, limiting us to 85 characters (for a 100 character tag). This is long enough to overflow a standard terminal (thus be obviously autogenerated), but short enough to be readable.

commit: keep opts as byteskwargs as much as possible This fixes many flag-handling bugs on 'hg commit' in Python 3.

util: reference __main__ in sys.modules as a sysstr

ui: convert to/from Unicode on Python 3 in ui.editor() I considered making this I/O be done in terms of bytes, but that would cause an observable regression for Windows users, as non-binary-mode open does EOL conversion there. There are probably new encoding dragons lurking here, so we may want to switch to using binary mode and doing EOL conversion ourselves.

manifest: refer to bytestrings as bytes, not str Required on Python 3.

manifest: use node.hex instead of .encode('hex') The latter doesn't work on Python 3.