graphlog: draw multiple edges towards null node (issue5440) Before, edge (r, null) was processed only once by newparents. However what we really need is just stripping the edge (null, null).

merge with stable

localrepo: use tryunlink

vfs: use tryunlink

server: use tryunlink

dirstate: use tryunlink

debug: use tryunlink

chgserver: use tryunlink

bookmarks: use tryunlink

shelve: use tryunlink

vfs: add tryunlink method Thoughout hg code, we see a pattern of attempting to remove a file and then catching and ignoring any errors due to a missing file in the calling code. Let's unify this pattern in a single implementation in the vfs layer.

util: use tryunlink in unlinkpath We just introduced a func to attempt a file removal. Start using it.

util: add tryunlink function Throughout mercurial cdoe, there is a common pattern of attempting to remove a file and ignoring ENOENT errors. Let's move this into a common function to allow for cleaner code.

util: unify unlinkpath Previously, there were two slightly different versions of unlinkpath between windows and posix, but these differences were eliminated in previous patches. Now we can unify these two code paths inside of the util module.

posix: use local reference to removedirs We have a local reference to os.removedirs in module scope, but we still used os.removedirs inside functions. This changes util to use the local reference, which will pave the way for combining duplicated code in future patches.

posix: use local reference to unlink We have a local reference to os.unlink in module scope, but we still used os.unlink inside functions. This changes util to use the local reference, which will pave the way for combining duplicated code in future patches.

localrepo: improve vfs documentation At the beginning of March, I promised Yuya that I would follow up a comment I made on a patch with improved documention for these vfs objects. Also hat tip to Pierre-Yves for adding the documentation here in the first place.

ui: restrict length of autogenerated blocked tags Long autogenerated blocked tags tend to be because the command has an absolute path; at Facebook, we've had a few where the tag is thousands of characters long (in association with the mergedriver). Change the default to use a suffix of a command as the default tag, limiting us to 85 characters (for a 100 character tag). This is long enough to overflow a standard terminal (thus be obviously autogenerated), but short enough to be readable.

commit: keep opts as byteskwargs as much as possible This fixes many flag-handling bugs on 'hg commit' in Python 3.

util: reference __main__ in sys.modules as a sysstr

ui: convert to/from Unicode on Python 3 in ui.editor() I considered making this I/O be done in terms of bytes, but that would cause an observable regression for Windows users, as non-binary-mode open does EOL conversion there. There are probably new encoding dragons lurking here, so we may want to switch to using binary mode and doing EOL conversion ourselves.

manifest: refer to bytestrings as bytes, not str Required on Python 3.

manifest: use node.hex instead of .encode('hex') The latter doesn't work on Python 3.

pure: use int instead of long Similar to the recent 73aa13bc8dac (revlog: use int instead of long, 2017-03-19).

histedit: inline cleanupnode Move "cleanupnode" (unsafe strip) into "safecleanupnode" so it's impossible to call the unsafe function directly. This helps reduce future programming errors.

histedit: use safecleanupnode in _aborthistedit (issue5500) Now nobody in histedit calls the unsafe cleanupnode directly.

histedit: use safecleanupnode in _finishhistedit This simplifies code a lot.

histedit: add a method to cleanup nodes safely The new method will decide between: - cleanupnode, which calls the unsafe repair.strip - create obsmarkers Ideally, nobody calls "cleanupnode" directly except for "safecleanupnode".

py3: prove hg status works

localrepo: use node.hex instead of awkward .encode('latin1') Spotted as an option by Yuya. Thanks!

py3: prove hg config works

templater: make pad() strip color codes before computing width (issue5416) Tested in ANSI mode. We might have to extend _ansieffectre to support terminfo mode.

templater: make pad() compute actual width str.ljust() and .rjust() are based on byte length, which are valid only for ASCII characters.

templater: reject bad fillchar argument passed to pad() Otherwise TypeError would be raised.

color: insert color code after every "\e[0m" (issue5413) This assumes the last color wins, tested in ANSI mode. I guess terminfo mode would work in the same way.

debugtemplate: pass ui to templater so label() works Follows up 3356bf61fa25.

merge: remove unnecessary matcher checks As part of changing manifest.diff to accept a matcher, a previous patch added matcher calls to each location in merge.manifestmerge that tested if 'x in mf' to maintain the same behavior as before. After analyzing it further, this matcher call isn't needed, and in fact hurts future patches ability to use the matcher here. Basically, all these 'if x in mf' checks were checking if a matched file's copy source was in the matcher as well. This meant if you passed a matcher for just file foo, it would not return file bar even if foo was a copy of bar. Since manifestmerge cares about copy information, let's allow all lookups of copy sources. We also update one spot with a 'is not None' check, since it wasn't obvious that the value could sometimes be None before, which broke when we called matcher(None). A future patch adds matcher optimizations to manifestmerge which causes this code path to get covered by existing tests.

rebase: use one dirstateguard for entire rebase Recently we switched rebases to run the entire rebase inside a single transaction, which dramatically improved the speed of rebases in repos with large working copies. Let's also move the dirstate into a single dirstateguard to get the same benefits. This let's us avoid serializing the dirstate after each commit. In a large repo, rebasing 27 commits is sped up by about 20%. I believe the test changes are because us touching the dirstate gave the transaction something to actually rollback.

histedit: add histedit.singletransaction config option This adds an option (which defaults to False) to run entire histedits in a single transaction. This results in 20-25% faster histedits in large repos where transaction startup cost is expensive. I didn't want to enable this by default because it has some unfortunate side effects. For instance, if a pretxncommit hook throws midway through the histedit, it will rollback the entire histedit and lose any progress the user had made. Same if the user aborts editting a commit message. It's still worth turning this on for large repos, but probably not for normal sized repos. Long term, once we have inmemory merging, we could do the entire histedit in memory, without a transaction, then we could selectively rollback just parts of it in the event of an exception. Tested it by running the tests with `--extra-config-opt=histedit.singletransaction=True`. The only failure was related to the hook rollback issue I mention above.

histedit: pop action after the action is completed We only want to pop the action after the action is completed, since if the action aborts part way through we want it to remain at the front of the list so continue/abort will start with it. Previously we relied on the fact that we only serialized the state file at the beginning of the action, so the pop wasn't serialized until the next iteration of the loop. In a future patch we will be adding a large transaction around this area, which means if we pop the list early it might get serialized if the action throws a user InterventionRequired error, at which point the action is not in the list anymore. So let's only pop it once the action is really truly done.

histedit: add transaction support to writing the state file This will be used in a future diff to enable a single transaction around an entire histedit.

localrepo: forcibly copy list of filecache keys On Python 3, keys() is more like iterkeys(), so we got in trouble for mutating the dict while we're iterating here. Since the list of caches should be relatively small, work around this difference by just forcing a copy of the key list.

localrepo: turn hook kwargs back into strs before calling hook It might be better to ensure that the hook kwargs dict only has str keys on Python 3. I'm torn.

localrepo: ensure transaction id is fully bytes on py3

dirstate: use future-proof next(iter) instead of iter.next The latter has been removed in Python 3.

posix: tiptoe around tempfile module more delicately Some of the values inside the tempfile calls here are str on Python 3, so we've got to pass str in. Use fsdecode to work around the issue.

posix: use open() instead of file()

revlog: use int instead of long By my reading of PEP 237[0], this is completely safe and has been since Python 2.2. 0: https://www.python.org/dev/peps/pep-0237/

error: use r-string to properly pop hints from **kw Fixes the hint mixin on Python 3.

dispatch: use pycompat.maplist to allow summing with args

pycompat: add maplist alias for old map behavior

dispatch: replace mayberepr with shellquote The quoting logic here was actually insufficient, and would have had bogus b-prefixes on Python 3. shellquote seems more appropriate anyway. Surprisingly, only two tests have output changes, and both of them look reasonable to me (both are in blackbox logs). Spotted by Yuya during review.

color: sync text attributes and buffered text output on Windows (issue5508) I originally noticed that log output wasn't being colored after 3a4c0905f357, but there were other complications too. With a bunch of untracked files, only the first 1K of characters were colored pink, and the rest were normal white. A single modified file at the top would also be colored pink. Line buffering and full buffering are treated as the same thing in Windows [1], meaning the stream is either buffered or not. I can't find any explicit documentation to say stdout is unbuffered by default when attached to a console (but some internet postings indicated that is the case[2]). Therefore, it seems that explicit flushes are better than just not reopening stdout. NB: pager is now on by default, and needs to be disabled to see any color on Windows. [1] https://msdn.microsoft.com/en-us/library/86cebhfs(v=vs.140).aspx [2] https://sourceforge.net/p/mingw/mailman/message/27121137/

test-check-help: fix to work on Windows The initial problem was `hg files` prints paths with '\', which gets removed when piped (scanhelptopics.py failed to open 'hgext__init__.py'). Then, xargs was invoking `hg help` with 'backout\r (esc)', which setting binary mode prevents.

branchmap: be more careful about using %d on ints Not doing so breaks Python 3.

util: use bytes re on bytes input in fspath Fixes `hg add` on Python 3.

util: use pycompat.bytestr in checkwinfilename Fixes `hg add` on python3.

dispatch: ensure repr is bytes in _mayberepr Fixes command line arguments containing spaces on Python 3.

dispatch: extract maybe-use-repr formatting to helper function I think this makes the code much clearer. I had to think for a bit to unpack the old-school `condition and if-true or if-false` dance, and formatting argument lists here shouldn't be performance critical.

dispatch: consolidate formatting of arguments This was getting done twice, and it's clever enough I'm about to split it apart and then fix it for Python 3.

py3: make the regular expression bytes to prevent TypeError

pager: flush outputs before firing pager process So that buffered outputs are always sent to the console.

patchbomb: use modern pager to display -n/--test result (BC) This should provide more consistent UX, but is a BC because the pager will no longer be fired up for each message.

httpconnection: make sure to clear progress of httpsendfile at EOF read() should never raise EOFError. If it did, UnboundLocalError would occur due to unassigned 'ret' variable. This issue was originally reported to TortoiseHg: https://bitbucket.org/tortoisehg/thg/issues/4707/

py3: convert log opts to bytes-key dict Now simple log command works.

graphlog: pass function arguments without expansion It's annoying on Python 3 because keys must be unicode type. Let's stop using **opts expansion when not necessary.

py3: call codecs.escape_decode() directly The same rule as 3b7a6941a6ef applies.

util: wrap s.decode('string_escape') calls for future py3 compatibility

merge with stable

hgwebdir: add support for explicit index files This is useful for when repositories are nested in --web-conf, and in the future with hosted subrepositories. The previous behavior was only to render an index at each virtual directory. There is now an explicit 'index' child for each virtual directory. The name was suggested by Yuya, for consistency with the other method names. Additionally, there is now an explicit 'index' child for every repository directory with a nested repository somewhere below it. This seems more consistent with each virtual directory hosting an index, and more discoverable than to only have an index for a directory that directly hosts a nested repository. I couldn't figure out how to close the loop and provide one in each directory without a deeper nested repository, without blocking a committed 'index' file. Keeping that seems better than rendering an empty index.

ui: move configlist parser to config.py The list parser is complex and reusable without ui. Let's move it to config.py. This allows us to parse a list from a "pure" config object without going through ui. Like, we can make "_trustusers" calculated from raw configs, instead of making sure it's synchronized by calling "fixconfig"s.

tests: allow ModuleNotFoundError in addition to ImportError My environment (Python version? PYTHONPATH? something else?) raises ModuleNotFoundError in test-check-py3-compat.t. This patch allows any "*Error". The error string contains "error importing", so it seems specific enough even after.

pager: skip running the pager if it's set to 'cat' Avoid useless uses of cat.

pager: avoid shell=True on subprocess.Popen for better errors (issue5491) man(1) behaves as poorly as Mercurial without this change. This cribs from git's run-command[0], which has a list of characters that imply a string that needs to be run using 'sh -c'. If none of those characters are present in the command string, we can use shell=False mode on subprocess and get significantly better error messages (see the test) when the pager process is invalid. With a complicated pager command (that contains one of the unsafe characters), we behave as we do today (which is no worse than git manages.) I briefly tried tapdancing in a thread to catch early pager exits, but it's just too perilous: you get races between fd duping operations and a bad pager exiting, and it's too hard to differentiate between a slow-bad-pager result and a fast-human-quit-pager-early result. I've observed some weird variation in exit code handling in the "bad experience" case in test-pager.t: on my Mac hg predictably exits nonzero, but on Linux hg always exits zero in that case. For now, we'll work around it with || true. :( 0: https://github.com/git/git/blob/cddbda4bc87b9d2c985b6749b1cf026b15e2d3e7/run-command.c#L201

py3: change explicit conversion of config value from str to pycompat.bytestr

py3: add __bool__ to every class defining __nonzero__ __nonzero__ was renamed to __bool__ in Python 3. This patch simply aliases __bool__ to __nonzero__ for every class implementing __nonzero__.

merge: also allow 'e' action with experimental.updatecheck=noconflict With experimental.updatecheck=noconflict set, if one checks out f3398f1f70a0 (tests: add execute bit and fix shbang line, 2015-12-22) and then try to check out its parent, hg will complain about conflicting changes, even though the working directory is clean. We need to also allow the 'e' action in merge.py. The 'e' action is used when moving to a commit where the only change to the file is to its executable flag, so it's just an optimized 'g' action. Doesn't seem to be worth writing a test for, since the existing setup in test-update-branches.t does not set any flags.

config: honour the trusted flag in ui.configbytes

osutil: fix potential wrong fd close According to POSIX closedir [1]: If a file descriptor is used to implement type DIR, that file descriptor shall be closed. According to POSIX fdopendir [2]: Upon calling closedir() the file descriptor shall be closed. So we should avoid "close(dfd)" after "closedir(dir)". With threads, there could be a race where an innocent fd gets closed. But Python GIL seems to help hiding the issue well. [1]: http://pubs.opengroup.org/onlinepubs/009695399/functions/closedir.html [2]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fdopendir.html

parsers: use Python memory allocator for indexObject->offsets

parsers: use Python memory allocator in commonancestorsheads()

osutil: use Python memory allocator in _listdir The Python memory allocator has performance advantages for small allocations.

bdiff: use Python memory allocator in fixws Python has its own memory allocation APIs. For allocations <= 512 bytes, it allocates memory from arenas. This means that average small allocations don't call the system allocator, which makes them faster. Also, arena allocations cut down on memory fragmentation, which can matter for performance in long-running processes. Another advantage of using the Python memory allocator is that allocations are tracked by Python. This is a bigger deal in Python 3, as modern versions of Python have some decent built-in tools for examining memory usage, leaks, etc. This patch converts a trivial malloc() + free() in the bdiff code to use the Python allocator APIs. Since the object being operated on is a line, chances are it will use an arena. So, this could have a net positive impact on performance (although I didn't measure it).

localrepo: fix deprecation warning version of wfile The patch lingered a bit too long in my local clone and I messed up when I updated the version number. Since nobody caught it, I'm fixing the version after the fact.

util: explicitly tests for None Changeset 8b6927eb7efd removed the mutable default value, but did not explicitly tested for None. Such implicit checking can introduce semantic and performance issue. We move to an explicit check for None as recommended by PEP8: https://www.python.org/dev/peps/pep-0008/#programming-recommendations

context: simplify call to icase matcher in 'match()' The two function takes the very same arguments. We make this clearer and less error prone by dispatching on the function only and having a single call point in the code.

py3: make sure using bytes status char rather than ascii values 'MAR!?IC' is converted to their ascii values when slicing through it. This patch uses pycompat.iterbytestr() to get bytes value.

shelve: get rid of ui.backupconfig

rebase: get rid of ui.backupconfig

mq: get rid of ui.backupconfig

histedit: get rid of ui.backupconfig

record: get rid of ui.backupconfig

import: get rid of ui.backupconfig

clone: get rid of ui.backupconfig

commit: get rid of ui.backupconfig

branchmap: handle nullrev in setcachedata 906be86990 recently changed to switch from: self._rbcrevs[rbcrevidx:rbcrevidx + _rbcrecsize] = rec to pack_into(_rbcrecfmt, self._rbcrevs, rbcrevidx, node, branchidx) This causes an exception if rbcrevidx is -1 (i.e. the nullrev). The old code handled this because python handles out of bound sets to arrays gracefully. The new code throws because the self._rbcrevs buffer isn't long enough to write 8 bytes to. Normally it would've been resized by the immediately preceding line, but because the 0 length buffer is greater than the idx (-1) times the size, no resize happens. Setting the branch for the nullrev doesn't make sense anyway, so let's skip it. This was caught by external tests in the Facebook extensions repo, but I've added a test here that catches the issue.