tests: replace "cp -r" with "cp -R" The POSIX documentation about "cp" [1] says: .... RATIONALE .... Earlier versions of this standard included support for the -r option to copy file hierarchies. The -r option is historical practice on BSD and BSD-derived systems. This option is no longer specified by POSIX.1-2008 but may be present in some implementations. The -R option was added as a close synonym to the -r option, selected for consistency with all other options in this volume of POSIX.1-2008 that do recursive directory descent. The difference between -R and the removed -r option is in the treatment by cp of file types other than regular and directory. It was implementation-defined how the - option treated special files to allow both historical implementations and those that chose to support -r with the same abilities as -R defined by this volume of POSIX.1-2008. The original -r flag, for historic reasons, did not handle special files any differently from regular files, but always read the file and copied its contents. This had obvious problems in the presence of special file types; for example, character devices, FIFOs, and sockets. .... .... Issue 6 The -r option is marked obsolescent. .... Issue 7 .... The obsolescent -r option is removed. .... (No "Issue 8" yet) Therefore it's clear that "cp -R" is strictly better than "cp -r". The issue was discovered when running tests on OS X after 0d87b1caed92. [1]: pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html

posix: give the cached symlink a real target The NamedTemporaryFile file is cleared up so checklink ends up as a dangling symlink, causing cp -r in tests to complain on both Solaris and OS X. Use a permanent file instead when there is a .hg/cache directory.

shelve: move patch extension to a string constant We are using 'name + ".patch"' pattern throughout the shelve code to identify the existence of a shelve with a particular name. In two cases however we use 'name + ".hg"' instead. This commit makes 'patch' be used in all places and "emphasizes" it by moving 'patch' to live in a constant. Also, this allows to extract file name without extension like this: f[:-(1 + len(patchextension))] instead of: f[:-6] which is good IMO. This is a first patch from this initial "obsshelve" series. This series does not include tests, although locally I have all of test-shelve.t ported to test obs-shelve as well. I will send tests later as a separate series.

merge with stable

Added signature for changeset b3b1ae98f6a0

Added tag 4.0.1 for changeset b3b1ae98f6a0

i18n-pt_BR: synchronized with 819f96b82fa4

shelve: fix use of unexpected working dirs in test-shelve.t Fixing some clowniness where we created ~four levels of nested repos and once (my test case :( ) did not even cd into a created repo.

crecord: change the verb according to the operation This will make crecord consistent with record when being used in the revert situation. It will say "Select hunks to revert / discard" accordingly. This should make the revert crecord interface less confusing.

crecord: change help text for the space key dynamically A follow-up of the previous patch, to make the text simple and clear about whether it's to "select" or "deselect".

crecord: rewrite status line text (BC) Previously, we display fixed text in the 2-line status header. Now we want to customize some word there to make the "revert" action clear. However, if we simply replace the verb using '%s' like this: "SELECT CHUNKS: (j/k/up/dn/pgup/pgdn) move cursor; " "(space/A) toggle hunk/all; (e)dit hunk;"), " (f)old/unfold; (c)onfirm %s; (q)uit; (?) help " % verb "| [X]=hunk %s **=folded, toggle [a]mend mode" % verb It could cause trouble for i18n - some languages may expect things like "%(verb) confirm", for example. Therefore, this patch chooses to break the hard-coded 2-line sentences into "segment"s which could be translated (and replaced) separately. With the clean-up, I'm also changing the content being displayed, to make it cleaner and more friendly to (new) users, namely: - Replace "SELECT CHUNKS" to "Select hunks to record". Because: - To eliminate "hunk" / "chunk" inconsistency. - "record" is used in the "text" UI. Do not use "apply", to make it consistent. - To make it clear you are choosing what to record, not revert, or discard etc. This is probably the most important information the user should know. So let's put it first. - "to record" could be replaced to others depending on the operation. The follow-up patches will address them. - Move "[x]" and "**" legends first to explain the current interface. New users should understand what the legends mean, followed by what they can do in the interface. - Replace "j/k/up/dn/pgup/pgdn" with "arrow keys". Because: - "arrow keys" is more friendly to new users. - Mentioning "j/k" first may make emacs users angry. We should stay neutral about editors. - "pgup/pgdn" actually don't work very well. For example, within a hunk of 100-line insertion, "pgdn" just moves one single line. - Left/Right arrow keys are useful for movement and discovery of "expanding" a block. - Replace "fold/unfold" with "collapse/expand", "fold" is well known as a different meaning in histedit and evolve. - Replace "(space/A) toggle hunk/all" to "space: select". Because: - "A: toggle all" is not that useful - It's unclear how "hunk" could be "toggled" to a dumb user. Let's make it clear it's all about "select". - A follow-up will make it use "unselect" when we know the current item is selected. - Remove "(f)old". Use arrow keys instead. - Remove "toggle [a]mend mode". It's just confusing and could be useless. - Remove "(e)dit hunk". It's powerful but not friendly to new users. - Replace "(q)uit" to "q: abort" to make it clear you will lose changes. The result looks like the following in a 73-char-width terminal: Select hunks to record - [x]=selected **=collapsed c: confirm q: abort arrow keys: move/expand/collapse space: select ?: help If the terminal is 132-char wide, the text could fit in a single line.

crecord: make _getstatuslines update numstatuslines We are going to make the text in the status window dynamically generated, so its size would be dynamic. Change getstatuslines to update "numstatuslines" automatically. Fix an issue where "numstatuslines" being 1 makes the chunkpad disappear.

crecord: move status window text calculation to a separate method We will do some changes there in the next patches. The new method would also be the "source of truth" of the content of the status window (so if the status window needs more than 2 lines, it would be calculated from the new method). Also, moved "statuswin.refresh" to make the code compact and easier to read.

revlog: ensure that flags do not overflow 2 bytes This patch adds a line that ensures we are not setting by mistake a set of flags overlfowing the 2 bytes they are allocated. Given the way the data is packed in the revlog header, overflowing 2 bytes will result in setting a wrong offset.

merge with stable

debugcommands: sort command order The diff is a bit large but it is straight code moving without any logical modifications.

tests: add test that @commands in debugcommands.py are sorted I felt like inline Python in test-check-code.t was the most appropriate place for this, as other linters in contrib/ seem to be source file agnostic. The test currently fails.

fsmonitor: be robust in the face of bad state fsmonitor could write out bad state if interrupted part way through, and would then crash when it tried to read it back in. Make both sides of the operation more robust - reading state should fail cleanly, and we can use atomictemp to write out cleanly as the file is small. Between the two, we shouldn't crash with an IndexError any more.

merge: use original file extension for temporary files Some merge tools (like Araxis?) can pick merge mode based on the file extension. That didn't work well when temporary files were given random suffixes. It seems to work better when the random part is before the extension. As usual, when using $output, $local will have the .orig extension. That could perhaps be the subject of another change another day.

ui: use try..finally in configoverride @contextmanager almost always have their "yield" inside a try..finally block. This is because if the calling code inside the activated context manager raises, the code after the "yield" won't get executed. A "finally" block, however, will get executed in this scenario.

util: limit output chunk size in zlib decompression This is essentially a port of 65bd4b8e48bd, which was inadvertently dropped by 8cd7d0fefd30.

crecord: filter text via i18n There are some text in the user interface that are not filtered by i18n. This patch adds the missing "_" call. So the text could be translated.

revert: pass operation to crecord So crecord would know what to display

crecord: add an "operation" field The field would provide extra information to help us to make the curses UI text less confusing.

revert: prompt before removing files in interactive mode Prior to this change, files to be removed (i.e. files added since the revision to revert to) were unconditionally removed despite the interactive mode. Now prompt before actually removing the files, as this is done for other actions (e.g. forget).

revert: indicate the default choice when prompting to forget files

style: avoid an unnecessary line split

debugcommands: move 'debugdeltachain' in the new module

debugcommands: move 'debugindex' and 'debugindexdot' in the new module

debugcommands: move 'debugignore' in the new module

debugcommands: move 'debuggetbundle' in the new module

debugcommands: move 'debugfsinfo' in the new module

debugcommands: move 'debugfileset' in the new module

censor: flag internal documentation

shelve: make --keep option survive user intervention (issue5431) Currently if user runs 'hg unshelve --keep' and merge conflicts occur, the information about --keep provided by user is lost and shelf is deleted after 'hg unshelve --continue'. This is obviously not desired, so this patch fixes it.

worker: use os._exit for posix worker in all cases Like commandserver, the worker should never run other resource cleanup logic. Previously this is not true for workers if they have exceptions other than KeyboardInterrupt. This actually caused a real-world deadlock with remotefilelog: 1. remotefilelog/fileserverclient creates a sshpeer. pipei/o/e get created. 2. worker inherits that sshpeer's pipei/o/e. 3. worker runs sshpeer.cleanup (only happens without os._exit) 4. worker closes pipeo/i, which will normally make the sshpeer read EOF from its stdin and exit. But the master process still have pipeo, so no EOF. 5. worker reads pipee (stderr of sshpeer), which never completes because the ssh process does not exit, does not close its stderr. 6. master waits for all workers, which never completes because they never complete sshpeer.cleanup. This could also be addressed by closing these fds after fork, which is not easy because Python 2.x does not have an official "afterfork" hook. Hacking os.fork is also ugly. Besides, sshpeer is probably not the only troublemarker. The patch changes _posixworker so all its code paths will use os._exit to avoid running unwanted resource clean-ups.

dispatch: move part of callcatch to scmutil Per discussion at 39149b6036e6 [1], we need "callcatch" in worker.py. Move it to scmutil.py to avoid cycles. Note that dispatch's callcatch handles some additional high-level exceptions related to config parsing, and commands. Moving them to scmutil will make scmutil depend on "commands" or require "_formatparse" and "_getsimilar" (and "difflib") to be moved as well. In the worker use-case, it is forked when config and commands are fully loaded. So it should not care about those exceptions. [1]: https://www.mercurial-scm.org/pipermail/mercurial-devel/2016-August/087116.html

py3: use pycompat.getcwd() instead of os.getcwd() We have pycompat.getcwd() which returns bytes path on Python 3. This patch changes most of the occurences of the os.getcwd() with pycompat one.

debugcommands: move 'debugextensions' to the new module

debugcommands: move 'debugdiscovery' in the module And a lot of imports with it.

debugcommands: move 'debugdate' in the new module

debugcommands: move 'debugrevlogopts' into the new module This move contains the first reference to debugrevlogopts in debugcommands.py. We'll eventually want to move that over. We hold off for now because it would introduce a module import cycle.

debugcommands: move 'debugdag' into the new module

chgserver: make it a core module and drop extension flags It was an extension just because there were several dependency cycles I needed to address. I don't add 'chgserver' to extensions._builtin since chgserver is considered an internal extension so nobody should enable it by their config.

chgserver: delay importing commands and dispatch modules This is a workaround for future import cycle: dispatch -> commands -> server -> chgserver -> commands. Some of the problems can be fixed later on pager and chg refactoring.

chgserver: drop CHGINTERNALMARK by chgunixservice() Prepares for the removal of uisetup(). We just need to do that at the start of the chg server, so chgunixservice() should be fine.

server: add public function to select either cmdserver or hgweb

server: move service factory from hgweb

hgweb: extract app factory I'll move createservice() to the server module, but createapp() seems good to remain in the hgweb module because of its dependency on hgweb/hgwebdir_mod.

server: move service table and factory from commandserver This is necessary to solve future dependency cycle between commandserver.py and chgserver.py. 'cmd' prefix is added to table and function names to avoid conflicts with hgweb.

server: move cmdutil.service() to new module (API) And call it runservice() because I'll soon add createservice(). The main reason I'm going to introduce the 'server' module is to solve future dependency cycle between chgserver.py and commandserver.py. The 'server' module sits at the same layer as the cmdutil. I believe it's generally good to get rid of things from the big cmdutil module.

debugcommands: move 'debugcomplete' in the new module

debugcommands: move 'debugcommands' in the new module The commit message isn't an illusion. There is a "debugcommands" module and command.

debugcommands: move 'debugcheckstate' in the new module

debugcommands: move debug{create,apply}streambundleclone to the new module

debugcommands: move 'debugbundle' in the new module

py3: add os.getcwdb() to have bytes path Following the behaviour of Python 3, os.getcwd() return unicodes. We need bytes version as path variables are bytes in UNIX. Python 3 has os.getcwdb() which returns current working directory in bytes. Like rest of the things there in pycompat, like osname, ossep, we need to rewrite every instance of os.getcwd to pycompat.getcwd to make them work correctly on Python 3.

help: clarify contents of revlog index The previous wording indicated that field at index 3 was the size of the decompressed chunk, not the size of the full revision text.

zstd: fix compilation with Solaris Studio Without these changes, Solaris Studio (12.4) gives us "syntax error: empty declaration" on these two lines.

cmdutil: turn forward of checkunresolved into a deprecation warning As with dirstateguard, I really doubt anyone outside core was using this, as my grep over the repositories I keep locally suggests nobody was using this. If others are comfortable with it, let's drop the forward entirely.

localrepo: refer to checkunresolved by its new name

rebase: refer to checkunresolved by its new name

checkunresolved: move to new package to help avoid import cycles This will allow localrepo to stop using cmdutil, which should avoid some future import cycles. There's room for an adventurous soul to delve deeper into merge.py and figure out how to disentangle more of it - it appears to be a nexus of cycle problems. Some of it might be able to move into this new mergeutil package.

cmdutil: mark dirstateguard as deprecated I sincerely doubt this is used in external code, as grepping the extensions I keep locally (including Facebook's hgexperimental and evolve) indicate nobody outside of core uses this. As such, I'd also welcome just dropping this name forward entirely.

localrepo: refer to dirstateguard by its new name

commands: refer to dirstateguard by its new name

rebase: refer to dirstateguard by its new name

mq: refer to dirstateguard by its new name

dirstateguard: move to new module so I can break some layering violations Recently in a review I noticed that localrepo almost has no reason to import cmdutil anymore. Also, cmdutil is a little on the enormous side, so breaking this class out strikes me as a win.

keepalive: discard legacy Python support for error handling We never changed the behavior defined by this attribute anyway, so just jettison all of this support.

mergemod: drop support for merge.update without a target This was to be deleted after 3.9.

dispatch: stop supporting non-use of @command We said we'd delete this after 3.8. It's time.

httppeer: document why super() isn't used Adding a follow-up to document lack of super() per Augie's request.

exchange: add `_getbookmarks()` function This function will be used to generate bookmarks bundle2 part. It is a separate function in order to make it easy to overwrite it in extensions. Passing `kwargs` to the function makes it easy to add new parameters in extensions.

bookmarks: use listbinbookmarks() in listbookmarks()

bookmarks: introduce listbinbookmarks() `bookmarks` bundle2 part will work with binary nodes. To avoid unnecessary conversions between binary and hex nodes let's add `listbinbookmarks()` that returns binary nodes. For now this function is a copy-paste of listbookmarks(). In the next patch this copy-paste will be removed.

ui: add configoverride context manager I feel like this idea might've been discussed before, so please feel free to point me to the right mailing list entry to read about why it should not be done. We have a common pattern of the following code: backup = ui.backupconfig(section, name) try: ui.setconfig(section, name, temporaryvalue, source) do_something() finally: ui.restoreconfig(backup) IMO, this looks better: with ui.configoverride({(section, name): temporaryvalue}, source): do_something() Especially this becomes more convenient when one has to backup multiple config values before doing something. In such case, adding a new value to backup requires codemod in three places.

archival: simplify code and drop message about Python 2.5

bugzilla: stop mentioning Pythons older than 2.6 We don't support those anyway.

tests: update sitecustomize to use uuid1() instead of randrange() The comments mention that uuid would be better, so let's go ahead and make good on an old idea.

win32mbcs: drop code that was catering to Python 2.3 and earlier

httppeer: drop an except block that says it happens only on Python 2.3

windows: do not replace sys.stdout by winstdout Now we use util.stdout everywhere.

py3: bulk replace sys.stdin/out/err by util's Almost all sys.stdin/out/err in hgext/ and mercurial/ are replaced by util's. There are a few exceptions: - lsprof.py and statprof.py are untouched since they are a kind of vendor code and they never import mercurial modules right now. - ui._readline() needs to replace sys.stdin and stdout to pass them to raw_input(). We'll need another workaround here.

py3: provide bytes stdin/out/err through util module Since standard streams are TextIO on Python 3, we can't use sys.stdin/out/err directly. Fortunately we can get the underlying BytesIO via .buffer as long as the streams aren't replaced by e.g. StringIO. stdin/out/err are provided through util so we can wrap them by platform API.

util: rewrite pycompat imports to make pyflakes always happy I'll add more imports which would confuse pyflakes.

windows: do not replace sys.__stdout__ Now we don't use sys.__stdout__ except for getting its fileno(), so we no longer have to wrap it by winstdout. This helps adding pycompat.stdin/out/err.

py3: update test-check-py3-compat.t output This part remains unchanged because it runs in Python 3 only.

py3: use pycompat.sysargv in dispatch.run() Another one to have a bytes result from sys.argv in Python 3. This one is also a part of running `hg version` on Python 3.

py3: use pycompat.sysargv in scmposix.systemrcpath() sys.argv returns unicodes on Python 3. We have pycompat.sysargv which returns bytes encoded using os.fsencode(). After this patch scmposix.systemrcpath() returns bytes in Python 3 world. This change is also a part of making `hg version` run in Python 3.

wireproto: perform chunking and compression at protocol layer (API) Currently, the "streamres" response type is populated with a generator of chunks with compression possibly already applied. This puts the onus on commands to perform chunking and compression. Architecturally, I think this is the wrong place to perform this work. I think commands should say "here is the data" and the protocol layer should take care of encoding the final bytes to put on the wire. Additionally, upcoming commits will improve wire protocol support for compression. Having a central place for performing compression in the protocol transport layer will be easier than having to deal with compression at the commands layer. This commit refactors the "streamres" response type to accept either a generator or an object with "read." Additionally, the type now accepts a flag indicating whether the response is a "version 1 compressible" response. This basically identifies all commands currently performing compression. I could have used a special type for this, but a flag works just as well. The argument name foreshadows the introduction of wire protocol changes, hence the "v1." The code for chunking and compressing has been moved to the output generation function for each protocol transport. Some code has been inlined, resulting in the deletion of now unused methods.

httppeer: use compression engine API for decompressing responses In preparation for supporting multiple compression formats on the wire protocol, we need all users of the wire protocol to use compression engine APIs. This commit ports the HTTP wire protocol client to use the compression engine API. The code for handling the HTTPException is a bit hacky. Essentially, HTTPException could be thrown by any read() from the socket. However, as part of porting the API, we no longer have a generator wrapping the socket and we don't have a single place where we can trap the exception. We solve this by introducing a proxy class that intercepts read() and converts the exception appropriately. In the future, we could introduce a new compression engine API that supports emitting a generator of decompressed chunks. This would eliminate the need for the proxy class. As I said when I introduced the decompressorreader() API, I'm not fond of it and would support transitioning to something better. This can be done as a follow-up, preferably once all the code is using the compression engine API and we have a better idea of the API needs of all the consumers.

httppeer: do decompression inside _callstream The current HTTP transport protocol only compresses certain command responses and requires calls to that command to call "_callcompressable," which zlib decompresses the response transparently. Upcoming changes will enable *any* response to be compressed with varying compression formats. In order to handle this better, this commit moves the decompression bits to the main function performing the HTTP request. We introduce an underscore-prefixed argument to denote this behavior so it doesn't conflict with a named argument to a command.

keepalive: reorder header precedence There are 3 sources of headers used by this function: * The default headers defined by the URL opener * Headers that are copied on redirects * Headers that aren't copied on redirects Previously, we applied the default headers from the URL opener last. This feels wrong to me as those headers are the most low level and something built on top of the URL opener may wish to override them. So, this commit changes the order to apply them with the least precedence. While I was here, I removed a Python version test that is no longer necessary.

debuginstall: print compression engine support Since compression engines may be provided by extensions and since not all registered compression engines may be available to use, it seems useful to provide a mechanism to see the state of known compression engines. This commit teaches `hg debuginstall` to print info on known and available compression engines.

bdiff: don't check border condition in loop This is pretty much a copy of d500ddae7494, just to a different loop. The condition `p == plast` (`plast == a + len - 1`) was only true on the final iteration of the loop. So it was wasteful to check for it on every iteration. We decrease the iteration count by 1 and add an explicit check for `p == plast` after the loop. Again, we see modest wins. From the mozilla-unified repository: $ perfbdiff -m 3041e4d59df2 ! wall 0.035502 comb 0.040000 user 0.040000 sys 0.000000 (best of 100) ! wall 0.030480 comb 0.030000 user 0.030000 sys 0.000000 (best of 100) $ perfbdiff 0e9928989e9c --alldata --count 100 ! wall 4.097394 comb 4.100000 user 4.100000 sys 0.000000 (best of 3) ! wall 3.597798 comb 3.600000 user 3.600000 sys 0.000000 (best of 3) The 2nd example throws a total of ~3.3GB of data at bdiff. This change increases the throughput from ~811 MB/s to ~924 MB/s.