sslutil: use CA loaded state to drive validation logic Until now, sslkwargs may set web.cacerts=! to indicate that system certs could not be found. This is really obtuse because sslkwargs effectively sets state on a global object which bypasses wrapsocket() and is later consulted by validator.__call__. This is madness. This patch introduces an attribute on the wrapped socket instance indicating whether system CAs were loaded. We can set this directly inside wrapsocket() because that function knows everything that sslkwargs() does - and more. With this attribute set on the socket, we refactor validator.__call__ to use it. Since we no longer have a need for setting web.cacerts=! in sslkwargs, we remove that. I think the new logic is much easier to understand and will enable behavior to be changed more easily.

sslutil: handle ui.insecureconnections in validator Right now, web.cacerts=! means one of two things: 1) Use of --insecure 2) No CAs could be found and were loaded (see sslkwargs) This isn't very obvious and makes changing behavior of these different scenarios independent of the other impossible. This patch changes the validator code to explicit handle the case of --insecure being used. As the inline comment indicates, there is room to possibly change messaging and logic here. For now, we are backwards compatible.

sslutil: check for ui.insecureconnections in sslkwargs The end result of this function is the same. We now have a more explicit return branch. We still keep the old code looking at web.cacerts=! a few lines below because we're still setting web.cacerts=! and need to react to the variable. This will be removed in an upcoming patch.

dispatch: set ui.insecureconnections when --insecure is used

ui: add an instance flag to hold --insecure bit Currently, when --insecure is used we set web.cacerts=! and socket validation takes this value into account. web.cacerts=! is not documented AFAICT and is purely an internal implementation detail. Let's be more explicit about what is going on by introducing a dedicated variable outside of the config values to track that --insecure is used.

sslutil: make sslkwargs code even more explicit The ways in which this code can interact with socket wrapping and validation later are mind numbing. This patch helps make it even more clear. The end behavior should be identical.

sslutil: move code examining _canloaddefaultcerts out of _defaultcacerts Before, the return of _defaultcacerts() was 1 of 3 types. This was difficult to read. Make it return a path or None. We had to update hghave.py in the same patch because it was also looking at this internal function. I wasted dozens of minutes trying to figure out why tests were failing until I found the code in hghave.py...

sslutil: further refactor sslkwargs The logic here and what happens with web.cacerts is mind numbing. Make the code even more explicit.

sslutil: document and slightly refactor sslkwargs This will help me and any reviewers keep sane as this code is refactored.

localrepo: remove a couple of local type aliases The local aliases are unused now, and were confusing mypy's type checker.

cmdutil: typo fix in comment

tests: add globs for Windows

chgserver: add [alias] to confighash The [alias] config section affects a global state: commands.table. It's hard to trace whether an alias is added by a config option or an extension, and add/remove aliases in a safe way per chg request. It will hurt performance a bit if we clean the table and parse aliases every time. Therefore let's just add it to confighash. This will make chg pass test-pager.t.

devel: use the 'config' argument for the dirstate normalisation develwarn

devel: use the new 'config' argument for the update develwarn

devel: use the new 'config' argument for the revset develwarn

devel: use the new 'config' argument for the dirstate develwarn

devel: use the new 'config' argument of the develwarn in deprecwarn Controling all deprecation warnings with the same config seems sensible. This mirror a fix (about missing gating) submitted for stable but with the new API.

develwarn: move config gating inside the develwarn function The config gating is almost always the same and contributor tend to forget it. We move the logic inside the function. Call site will be updated in later changeset. We might make the sub config mandatory in the future (once all old call sites are gone).

help: fix the display for `hg help internals.revlogs` (issue5227) It previously aborted saying the help section wasn't found. Credit to Yuya for figuring out the fix.

builddeb: add distroseries to tagged versions This is needed so that launchpad and friends have a unique version number for each distroseries (trusty, wily, xenial, etc). It was discovered when trying to upload 3.8 to launchpad.

debian: forgot to make debian/rules executable in 6b95a623ec90

debian: add wish to suggests Debian maintainers already have this and lintian warns us about not listing 'wish' as a dependency or suggestion so this patch does indeed just that. The issue, by the way, is that we are shipping hgk (which is written in tcl/tk) so we should be good citizens and list wish (a meta package for tcl/tk) as a dependency.

hg-ssh: copy doc string to man page This corrects a warning from lintian that we're shipping an executable without a man page. Since there is a doc string in the text, let's use that for the man page.

progress: stop excessive clearing (issue4801) The progress bar was being cleared on every write(), regardless of whether it was currently displayed. This could foul up the display of any writes that didn't include a linebreak. In particular, the win32 mode of the color extension was turning single prompt string writes into two writes, and the resulting clear/write/clear/write pattern was making the prompt invisible. We fix this by insisting that we have shown a progress bar and haven't just cleared it (setting lastprint to 0). Conveniently, the test suite already had instances of duplicate clears.. that are now cleared up.

chgserver: remove _clearenvaliases Since we expand environment variables in alias lazily, the _clearenvaliases hack is no longer necessary. This resolves an issue that a non-shell alias which has environment variables in its arguments and is set to use pager will not use pager running with chg.

dispatch: defer environment variable resolution in alias commands (BC) Before this patch, if there are environment variables in an alias command, they will be expanded immediately when we first see the alias. This will cause issues with chg, because environment variable updates will not propagate to expanded arguments. This patch makes "args" of "cmdalias" a property that will be calculated every time when accessed.

rollback: add a config knob for entirely disabling the command This is of pretty high value for organizations that used to use p4 (as an example), since `p4 rollback` is what we call `hg backout`.

templater: add separate() template function A pretty common pattern in templates is adding conditional separators like so: {node}{if(bookmarks, " {bookmarks}")}{if(tags, " {tags}")} With this patch, the above can be simplified to: {separate(" ", node, bookmarks, tags)} The function is similar to the already existing join(), but with a few differences: * separate() skips empty arguments * join() expects a single list argument, while separate() expects each item as a separate argument * separate() takes the separator first in order to allow a variable number of arguments after it

bookmarks: jettison bmstore's write() method per deprecation policy

merge with stable

deprecation: gate deprecation warning behind devel configuration Regular users are not supposed to be exposed to the API deprecation warnings. We now only issue them when the developper warnings are enabled.

crecord: update downarrowshiftevent() docstring, remove todo The phrasing is mostly taken from uparrowshiftevent().

crecord: remove things that don't happen in functions from their docstrings Scrolling screen is currently done in a different place. The things that had been described in the docstrings may still happen, but the functions touched by this patch don't do any scrolling, they only set self.currentselecteditem and nothing more.

crecord: remove skipfolded keyword argument from patchnode.previtem() It wasn't used, it wasn't implemented. Probably was a copy-paste bonus from patchnode.nextitem()

crecord: update a copy-pasted comment in downarrowshiftevent()

crecord: drop the version condition for amend The UI is now shipped in core, amend feature is always available.

crecord: add/remove blank lines (coding style)

localrepo: jettison parents() method per deprecation policy (API)

revset: define _parsealias() in _aliasrules class It's short. It doesn't make sense to define _parsealias() outside of the class.

revset: factor out common parsing function

revset: inline _tokenizealias() into _parsealias() This helps factoring out common part between _parsealias() and parse().

store: treat range as a generator instead of a list for py3 compat

ui: add new config option for help text width Before this patch, when printing help text using `hg help`, or `hg log -h`, the output will wrap at 78 chars even if the user has a bigger terminal width and there is no config option to change it, making the experience different from the commonly used `man` tool. This patch introduces a new config option `ui.textwidth`, which replaces the hardcoded number. It's set to 78 by default to maintain compatibility. When set to 0, `hg help` will behave more like `man`.

tests: test histedit base command plan help

largefiles: makes verify batching stat calls to remote Instead of sending stat calls for each files separately, it sends one batch call with stat invocations for all files.

largefiles: change basestore._verifyfile to take list of files to check Makes it easier to use batch stat calls in remotestore to decrease number of round trips.

bookmarks: properly invalidate volatile sets when writing bookmarks This corrects a regression introduced during the 3.7 cycle, but which went undetected due to the surviving-but-deprecated write() method on bmstore.

test-obsolete: update extension in test to actually work This hasn't been testing anything since partway through the 3.7 cycle due to unrelated refactoring. Sadly, the behavior it was trying to prevent reemerged in the codebase at that time. A fix is in the next patch, because proving that the fix was actually correct ended up being trickier than I expected.

bundle2: properly request phases during getbundle getbundle was requesting the "phase" namespace instead of the "phases" namespace, which led to the client still requesting the phases separately after getbundle finished.

rebase: handle successor targets (issue5198) When a parent has a successor (indicated by revprecursor in state), we need to use it.

debugbundle: add tests for debugbundle output with bundle2

revert: mention ui.origbackuppath in the command help

help: wrap ".orig" in rst quotes Apparently, .orig. is a macro for man pages so we need to wrap it in quotes to silence lintian warnings.

parser: shorten prefix of alias parsing errors These messages seemed to be a bit long. We should try making them fit to 80-col console.

parser: rephrase "'$' not for alias arguments" message Say which symbol caused the error. The word "alias" is removed since these messages are prefixed by "failed to parse ... revset alias "...":".

debugbundle: handle the --all option for bundle2

manifest: improve filesnotin performance by using lazymanifest diff lazymanifests can compute diffs significantly faster than taking the set of two manifests and calculating the delta. when running hg diff --git -c . on Facebook's big repo, this reduces the run time from 2.1s to 1.5s.

keyword: replace use of _filerev with _filenode To be independent of rev numbers. Analogous to ba8257cb53e8.

merge with stable

Added signature for changeset a56296f55a5e

Added tag 3.8.1 for changeset a56296f55a5e

convert: pass absolute paths to git (SEC) Fixes CVE-2016-3105 (1/1). Previously, it was possible for the repository path passed to git-ls-remote to be misinterpreted as a URL. Always passing an absolute path to git is a simple way to avoid this.

Added signature for changeset f85de28eae32

Added tag 3.8 for changeset f85de28eae32

debian: alphabetize build deps

debian: fix lintian warning about debhelper It seems this is correct but does it work on older distros? I ran the docker-jessie rule and didn't get any warnings.

builddeb: remove chmod as lintian tells us It turns out we just need debian/rules to be executable, so we do just that.

builddeb: use codename in version Apparently, this is needed to allow ppas to be built for multiple distros.

rebase: restrict rebase destination to the pulled set (issue5214) Before this patch, `hg pull --rebase` would be a strict sequence of `hg pull` followed by `hg rebase` if anything was pulled. Now that rebase pick his default destination the same way than merge, than `hg rebase` step would abort in the case the repo already had multiple anonymous heads (because of the ambiguity). (changed in fac3a24be50e) The intend of the user with `hg pull --rebase` is clearly to rebase on pulled content. This used to be (mostly) enforced by the former default destination for rebase, "tipmost changeset of the branch" as the tipmost would likely a changeset that just got pulled. But this intended was no longer enforced with the new defaul destination (unified with merge). This changeset makes use of the '_destspace' mechanism introduced in the previous changeset to enforce this. This partially fixes issue5214 as no change at all have been made to the new handling of the case with bookmark (unified with merge).

destutil: add the ability to specify a search space for rebase destination In the 'hg pull --rebase', we don't want to pick a rebase destination unrelated to the pull, we lay down basic infrastructure to allow such restriction on stable (before 3.8 release) in this case. See issue 5214 for details. Actual usage and test will be in the next patch.

sslutil: restore old behavior not requiring a hostname argument (issue5210) This effectively backs out changeset 1fde84d42f9c. The http library behind ui.http2=true isn't specifying the hostname. It is the day before the expected 3.8 release and we don't want to ship a regression. I'll try to restore this requirement in the 3.9 release cycle as part of planned improvements to Mercurial's SSL/TLS interactions.

tests: test a variety of cache invariants We've historically had a problem maintaining the expected invariants on our caches, especially when introducing new caches. This tests documents the invariants and exercises them across most of our existing cache files.

repoview: ignore unwritable hidden cache The atomictemp.close() file attempts to do a rename, which can fail. Moving the close inside the exception handler fixes it. This doesn't fit well with the with: pattern, as it's the finalizer that's failing.

tags: silence hgtagsfnodes reading failures tryread() doesn't handle "is a directory" errors and presumably others. We might not want to globally swallow such tryread errors, so we replace with our own try/except handling. An upcoming test will use directories as a portable stand-in for various bizarre circumstances that cache read/write code should be robust to.

tags: silence cache parsing errors Follow our standard STFU cache-handling pattern

i18n-pt_BR: synchronized with 18c1b107898e

ubuntu-xenial-ppa: add makefile rule

ubuntu-wily-ppa: add makefile rule

ubuntu-trusty-ppa: add makefile rule

ubuntu-xenial: add makefile rule to build deb

ubuntu-wily: add makefile rule to build deb

make: turn ubuntu docker into template This allows us to easily add more ubuntu docker targets (which following patches will do). Also, we no longer need the mkdir command.

revsets: add docs for '%' operator

graft: fix printing of --continue command Properly shell quote arguments, to avoid printing commands that won't work when run literally. For example, a date string with timestamp needs to be quoted: --date '1456953053 28800'

hghave: remove unused check for bdist_mpkg

osx: create a modern package including manpages Instead of using bdist_mpkg, we use the modern Apple-provided tools to build an OS X Installer package directly. This has several advantages: * Avoids bdist_mpkg which seems to be barely maintained and is hard to use. * Creates a single unified .pkg instead of a .mpkg. * The package we produce is in the modern, single-file format instead of a directory bundle that we have to zip up for download. In addition, this way of building the package now correctly: * Installs the manpages, bringing the `make osx`-generated package in line with the official Mac packages we publish on the website. * Installs files with the correct permissions instead of encoding the UID of the user who happened to build the package. Thanks to Augie for updating the test expectations.

hghave: add check for OS X packaging tools

tests: add test for Mac OS X package construction

osx: add support for keeping mpkgs This is a bit of a hack, but I don't really want to mount a dmg during a test, and I don't see an option with hdiutil to take a dmg and spit out a folder, so this is what we've got for now.

osx: add support for dumping built dmg into OUTPUTDIR

hghave: add check for bdist_mpkg

verify: don't init subrepo when missing one is referenced (issue5128) (API) Initializing a subrepo when one doesn't exist is the right thing to do when the parent is being updated, but in few other cases. Unfortunately, there isn't enough context in the subrepo module to distinguish this case. This same issue can be caused with other subrepo aware commands, so there is a general issue here beyond the scope of this fix. A simpler attempt I tried was to add an '_updating' boolean to localrepo, and set/clear it around the call to mergemod.update() in hg.updaterepo(). That mostly worked, but doesn't handle the case where archive will clone the subrepo if it is missing. (I vaguely recall that there may be other commands that will clone if needed like this, but certainly not all do. It seems both handy, and a bit surprising for what should be a read only operation. It might be nice if all commands did this consistently, but we probably need Angel's subrepo caching first, to not make a mess of the working directory.) I originally handled 'Exception' in order to pick up the Aborts raised in subrepo.state(), but this turns out to be unnecessary because that is called once and cached by ctx.sub() when iterating the subrepos. It was suggested in the bug discussion to skip looking at the subrepo links unless -S is specified. I don't really like that idea because missing a subrepo or (less likely, but worse) a corrupt .hgsubstate is a problem of the parent repo when checking out a revision. The -S option seems like a better fit for functionality that would recurse into each subrepo and do a full verification. Ultimately, the default value for 'allowcreate' should probably be flipped, but since the default behavior was to allow creation, this is less risky for now.

setup: detect Python DLL filename from loaded DLL Attempting to build Mercurial from source using MinGW from msys2 on Windows produces a hg.exe that attempts to load e.g. python27.dll. MinGW prefixes its library name with "lib" and adds a period between the major and minor versions. e.g. "libpython2.7.dll." Before this patch, hg.exe files in a MinGW environment would either fail to find a Python DLL or would attempt to load a non-MinGW DLL, which would summarily explode. Either way, hg.exe wouldn't work. This patch improves the code that determines the Python DLL filename to actually use the loaded Python DLL instead of inferring it. Basically we take the handle of the loaded DLL from sys.dllhandle and call a Windows API to try to resolve that handle to a filename.

exewrapper: add .dll to LoadLibrary() argument LoadLibrary() changes behavior depending on whether the argument passed to it contains a period. From the MSDN docs: If no file name extension is specified in the lpFileName parameter, the default library extension .dll is appended. However, the file name string can include a trailing point character (.) to indicate that the module name has no extension. When no path is specified, the function searches for loaded modules whose base name matches the base name of the module to be loaded. If the name matches, the load succeeds. Otherwise, the function searches for the file. As the subsequent patch will show, some environments on Windows define their Python library as e.g. "libpython2.7.dll." The existing code would pass "libpython2.7" into LoadLibrary(). It would assume "7" was the file extension and look for a "libpython2.dll" to load. By passing ".dll" into LoadLibrary(), we force it to search for the exact basename we want, even if it contains a period.

update: correct description of --check option The old "update across branches if no uncommitted changes" made it sound like updating across branches (with no uncommitted changes) was allowed only with this option, which was not true. Also, the option did not care whether it was linear or across branches. Instead, it checked that there were no uncommitted changes. Let's explain what it does instead of trying to suggest what happens without it.

util: fix race in makedirs() Update makedirs() to ignore EEXIST in case someone else has already created the directory in question. Previously the ensuredirs() function existed, and was nearly identical to makedirs() except that it fixed this race. Unfortunately ensuredirs() was only used in 3 places, and most code uses the racy makedirs() function. This fixes makedirs() to be non-racy, and replaces calls to ensuredirs() with makedirs(). In particular, mercurial.scmutil.origpath() used the racy makedirs() code, which could cause failures during "hg update" as it tried to create backup directories. This does slightly change the behavior of call sites using ensuredirs(): previously ensuredirs() would throw EEXIST if the path existed but was a regular file instead of a directory. It did this by explicitly checking os.path.isdir() after getting EEXIST. The makedirs() code did not do this and swallowed all EEXIST errors. I kept the makedirs() behavior, since it seemed preferable to avoid the extra stat call in the common case where this directory already exists. If the path does happen to be a file, the caller will almost certainly fail with an ENOTDIR error shortly afterwards anyway. I checked the 3 existing call sites of ensuredirs(), and this seems to be the case for them.

chg: initialize sockdirfd to -1 instead of AT_FDCWD As we don't use sockdirfd yet, this is the simplest workaround to compile chg on old Unices where AT_FDCWD does not exist. Foozy pointed out Mac OS X 10.10 is required for AT_FDCWD as well as xxxat() functions.

bdiff: further restrain potential quadratic performance This causes the longest_match search to limit itself to a window of 30000 lines during search (roughly 1MB), thus avoiding a full O(N*M) search that might occur in repetitive structured inputs. For a particular class of many MB pathological test cases, this generated the following timings: size before after 10x 1.25s 1.24s 100x 57s 33s 1000x >8400s 400s The times on the right quickly become much faster and appear more linear. While windowing means deltas are no longer "optimal", the resulting deltas were within a couple percent of expected size. While we've yet to have a report of a file with the level of repetition necessary to hit this case, some JSON/XML database dump scenario is fairly likely to hit it. This may also slightly improve the average-case performance for deltas of large binaries.

bdiff: balance recursion to avoid quadratic behavior (issue4704) For highly structured files like JSON or XML dumps with large numbers of duplicate lines (eg braces) and isolated matching lines, bdiff could find large numbers of equally good spans. Because it prefers earlier matches, this would result in pathologically unbalance recursion that resulted in quadratic performance. This patch makes it prefer matches closer to the middle that tend to balance recursion. This change improves the speed of a pathological test case from 1100s to 9s. Included is a smaller test that has a roughly 50x safety margin on the performance it accepts. It's likely to fail on pure builds because difflib also has a recursion-balancing problem.

bdiff: deal better with duplicate lines The longest_match code compares all the possible positions in two files to find the best match. Given a pair of sequences, it effectively searches a grid like this: a b b b c . d e . f 0 1 2 3 4 5 6 7 8 9 a 1 - - - - - - - - - b - 2 1 1 - - - - - - b - 1 3 2 - - - - - - b - 1 2 4 - - - - - - . - - - - - 1 - - 1 - Here, the 4 in the middle says "the first four lines of the file match", which it can compute be comparing the fourth lines and then adding one to the result found when comparing the third lines in the entry to the upper left. We generally avoid the quadratic worst case by only looking at lines that match, which is precomputed. We also avoid quadratic storage by only keeping a single column vector and then keeping track of the best match. Unfortunately, this can get us into trouble with the sequences above. Because we want to reuse the '3' value when calculating the '4', we need to be careful not to overwrite it with the '2' we calculate immediately before. If we scan left to right, top to bottom, we're going to have a problem: we'll overwrite our 3 before we use it and calculate a suboptimal best match. To address this, we can either keep two column vectors and swap between them (which significantly complicates bookkeeping), or change our scanning order. If we instead scan from left to right, bottom to top, we'll avoid ever overwriting values we'll need in the future. This unfortunately needs several changes to be made simultaneously: - change the order we build the initial hash chains for the b sequence - change the sentinel values from INT_MAX to -1 - change the visit order in the longest_match inner loop - add a tie-breaker preference for earlier matches This last is needed because we previously had an implicit tie-breaker from our visitation order that our test suite relies on. Later matches can also trigger a bug in the normalization code in diff().

bdiff: fix latent normalization bug This bug is hidden by the current bias towards matches at the beginning of the file. When this bias is tweaked later to address recursion balancing, the normalization code could cause the next block to shrink to a negative length, thus creating invalid delta chunks. We add checks here to disallow that. This bug requires test cases that are an awkwardly large size for the test suite, but is very rapidly picked up by the included torture tester.

bdiff: fold in shift calculation in normalize This just makes the code harder to read without any performance advantage. We're going to make the check here more complex, let's make it simpler first.

bdiff: unify duplicate normalize loops We're about to make the while loop check more complicated, so let's simplify first.

make: backout changeset 51f5fae84e43 Support for '!=' was only added in GNU Make 4.0, and CentOS versions as new as CentOS 7 only carry 3.82. I will leave figuring out compatibility with BSD make as an exercise for interested folks.

tests: test-lock-badness.t message could come later I got this on gcc112: @@ -58,8 +58,8 @@ $ hg -R b ci -A -m b --config hooks.precommit="python:`pwd`/hooks.py:sleepone" > stdout & $ hg -R b up -q --config hooks.pre-update="python:`pwd`/hooks.py:sleephalf" waiting for lock on working directory of b held by '*:*' (glob) - got lock after ? seconds (glob) $ wait + got lock after 1 seconds $ cat stdout adding b

dockerdeb: pass the rest of the args to the builder script It seems this was the original intent of the script so this patch passes the remanining arguments to builddeb.

dockerdeb: fix incorrect number of shifts From the comment, it appears that the original intent was to remove the first two arguments, so this patch does just that.

make: use shell-command assignment instead of $(eval ...) This is portable between BSD and GNU make. As of this change, our Makefile appears to work in both BSD and GNU make, with the caveat that the test-% and testpy-% wildcard rules don't work on BSD make. That said, this still seems worthwhile because it lets the buildbots work more consistently across platforms.

make: do assignment and export in a single statement This is portable between GNU and BSD make, whereas doing the export on its own line confuses BSD make.

make: alter how we compute compiler flags for setup.py This is portable between BSD and GNU make. I'm not thrilled with how it worked out, but it's portable and solves the problem.

revset: unindent "if True" block in sort() It was there to make the previous patch readable.