py3: alias long to int and xrange to range in test-ancestor.py on Python 3

py3: add a new bytesurl() to convert a str url into bytes

py3: add a new strurl() which will convert a bytes url to str

killdaemons: fix WaitForSingleObject() error handling logic on Windows The error return is not 0 for this method, so _check() was doing nothing when an error occurred. This forces the error path, much like the check for OpenProcess(). The only unhandled return is now WAIT_ABANDONED, but I don't see how that could happen in this case.

killdaemons: explicitly set the ctypes signatures When I tried importing util.posixfile to work around removing a file opened by another process on Windows, it brought along the declarations in win32.py, which broke the error handling[1]. It doesn't seem worth hacking killdaemons[2] just to isolate these declarations in win32.py, so just declare them here to prevent any future issues. (win32.py mentions the declarations are required by pypy.) [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097905.html [2] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097907.html

hghave: enable 'serve' on Windows I've been using a local hghaveaddon.py to enable this for a couple of months with reasonable success, and 'killdaemons' is already enabled on Windows. There's one failure[1] in test-http-proxy.t that this adds, which I can't figure out. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096987.html

contrib: add a ratchet for tests in Python 3 This gives us an easy way to automatically update passing tests in Python 3.

contrib: check in a whitelist of passing tests in Python 3

tests: try and fail more gracefully with broken unicode escapes

md5sum: adapt for python 3 support

profiling: allow configuring minimum display threshold for hotpath statprof.display_hotpath() accepts a "limit" function to choose the minimum threshold for samples to display. The default is 0.05, which means you don't need individual items contributing less than 5%. I had a need to adjust this threshold. We already have a config option for it. So let's reuse it. check-config.py doesn't like having multiple defaults for the ui.configwith() calls. The behavior is obviously correct. I'm not sure if it is worth teaching check-config.py how to ignore this. So I've just accepted the new output.

config: document profiling.show{min,max}

check-config: look for ui.configwith We previously weren't looking for this config helper. And, surprise, profiling.py references config options without docs. If I tried hard enough, I could have combined the regexps using a positive lookbehind assertion or something. But I didn't want to make my brain explode. At some point, we should probably do this linting at the tokenizer or ast layer. I'm not willing to open that can of worms right now.

check-config: use named groups in regexp In preparation for making this regexp a bit more complicated.

check-config: use compiled regexp And split the regexp across multiple lines to make it easier to read.

tests: add test coverage for check-config We didn't have explicit test coverage before. I broke check-config.py as part of writing patches and was lucky I realized it.

exchange: use context manager for bundle1 unbundling The lazy locking is not used for bundle1, so using a regular context manager is clearer.

unbundle: use context manager for transaction

clonebundle: use context managers for lock and transaction

changeset_templater: render template specified by templatespec tuple

formatter: render template specified by templatespec tuple

formatter: put topic in templatespec tuple This will allow us to change the initial template reference depending on how the template is looked up. For example, -Tdefault => (ref='changeset', tmpl=None, mapfile='map-cmdline.default') -T'{rev}' => (ref='', tmpl='{rev}', mapfile=None) A literal template given by -T option will be stored as an unnamed template, which will free up the template namespace so that we can load named templates from [templates] section of user config.

cmdutil: pass templatespec tuple directly to changeset_templater (API) A fewer number of arguments should be better.

formatter: wrap (tmpl, mapfile) by named tuple I'm going to add more options to the templatespec tuple. cmdutil.logtemplatespec() is just an alias now, but it will be changed to a factory function later.

cmdutil: factor out helper to create changeset_templater with literal template changeset_templater has lots of arguments, but most callers only need to specify a literal template 'tmpl'. "hg debugtemplate" has no diff option, which means 'opts' were effectively {}, so dropped opts.

test-dirstate-race: stabilize for Windows Without quoting, the output after "custom merge tool" in the next test is an abort trying to open "C:UsersMattAppData..."

formatter: document lookuptemplate()

cmdutil: rename gettemplate() to _lookuplogtemplate() This function is only useful when processing log options.

formatter: inline gettemplater() Since it's highly use-case dependent how template should be looked up, gettemplater() function isn't useful. Keeping it would introduce another bug I've made and fixed earlier in this series.

formatter: factor out function to create templater from literal or map file (tmpl, mapfile) will be packed into a named tuple later.

graphlog: do not look up graphnodetemplate in template paths (BC) ui.graphnodetemplate config should be a literal template as ui.logtemplate is. The use of formatter.gettemplater() is valid only for a template string given by -T/--template option.

formatter: open raw template file in posix semantics This should have no effect, but seems good for code consistency.

formatter: open raw template file in binary mode (BC) I believe it was a mistake to open file in text mode. Now '\r' is preserved on Windows, but it should be okay to print '\r\n' as long as users live in such platform.

formatter: close raw template file explicitly

patchbomb: make getaddrs function easier to work with Prior to this the return value was potentially None, a string, or a list of strings. It now always returns a list of strings where each string is always only one email address

patchbomb: look for non-empty publicurl, not a non-None one Otherwise it's impossible to turn this feature back off, which is making writing of tests awkward.

patchbomb: make variable name for publicurl always be publicurl

bisect: simpler approach for option validation message Yuya Nishihara gave this suggestion on the mailing list after the previous patch was queued, and honestly this seems much simpler and looks more efficient.

pushkey: use False/True for return values from push functions It was particularly unclear in phases.pushphase() whether the 0/1 returned were the 0/1 for public/draft phase or for False/True

smartset: micro optimize spanset.slice() to narrow range accordingly -1ms isn't significant, but seems not bad. revset #0: limit(0:9999, 100, 9000) 6) 0.001145 7) 0.000214 revset #3: last(0:9999, 100) 6) 0.000197 7) 0.000171

smartset: micro optimize baseset.slice() to use slice of list -1ms isn't significant, but seems not bad. revset #1: limit(0::9999, 100, 9000) 5) 0.001681 6) 0.000790

smartset: extract method to slice abstractsmartset Sub classes can provide optimized implementations.

smartset: extract spanset factory to make it constructed without a repo This renames the spanset class to _spanset, and moves its __init__ to new spanset() function. spanset() is now a factory function. This allows us to construct a spanset without keeping a repo instance.

smartset: change repr of spanset to show revisions as half-open range Before, an empty spanset was displayed as '<spanset+ 0:-1>', which seemed confusing.

fsmonitor: don't write out state if identity has changed (issue5581) Inspired by the dirstate fix in dc7efa2826e4, this should fix any race conditions with the fsmonitor state changing from underneath. Since we now grab the wlock for any non-invalidate writes, the only situation this appears to happen in is with a concurrent invalidation. Test that.

fsmonitor: write state with wlock held and dirstate unchanged (issue5581) This means that the state will not be written if: (1) either the wlock can't be obtained (2) something else came along and changed the dirstate while we were in the middle of a status run.

workingctx: add a way for extensions to run code at status fixup time Some extensions like fsmonitor need to run code after dirstate.status is called, but while the wlock is held. The extensions could grab the wlock again, but that has its own peculiar race issues. For example, fsmonitor would not like its state to be written out if the dirstate has changed underneath (see issue5581 for what can go wrong in that sort of case). To protect against these sorts of issues, allow extensions to declare that they would like to run some code to run at fixup time. fsmonitor will switch to using this in the next patch in the series.

workingctx: also pass status tuple into poststatusfixup fsmonitor is going to need this to compute its set of notable files to persist.

workingctx: factor out post-status dirstate fixup We want to allow extensions to be able to add code to run inside the wlock.

merge with stable

profile: properly propagate exception from the sub-context manager Context manager has a mechanism to control extension propagation. It is not used by profiling right now, but making the code correct will help prevent bug in the future.

profile: close 'fp' in all cases There are no way for this to happen today, but better be safe than sorry, no one know how the code will evolve. We now make sure the file pointer is closed even is profiler is None.

profile: close 'fp' on error within '__enter__' Previously, error when initialying the profiler would forgot to explicitly close the file. Thank goes to Yuya Nishihara for spotting this.

profile: indent part of '__enter__' We are about to introduce a try except. We start with adding the indent to make the next patch clearer.

profile: remove now useless indent We no longer rely on the value of '_output' so we can remove this conditional.

profile: use explicit logic to control file closing We make the decision to close 'fp' more explicit instead of relying on the implication of other variable. This makes the overall logic more robust.

profiling: move 'fp' closing logic into its own function We are about to make the logic more robust and reuse it in more place, we start by isolating what we have.

py3: use python3 hg in test-py3-commands.t at places where py2 hg was used This patch fixes my mistakes where I added test in test-py3-commands.t as `hg ...` where I forgot hg here refers to Python 2 mercurial.

test-dirstate-race: back out changeset c82fa7efcbc8 This is non-deterministic. In any case, I switched to using debugrebuilddirstate in my WIP patches, which makes this moot.

revset: fix order of first/last members in compound expression (BC) Suppose len(subset) >> len(ls) in common cases, 'subset & ls' should be avoided whenever possible.

revset: filter first/last members by __and__ operation This replaces 'if y in subset' with '& subset'. first(null) and last(wdir()) are fixed thanks to fullreposet.__and__. This also revealed that first() and last() don't follow the order of the input set. 'ls & subset' is valid only if the ordering requirement is 'define' or 'any'. No performance regression observed: revset #0: limit(0:9999, 100, 9000) 0) 0.001164 1) 0.001135 revset #2: 9000 & limit(0:9999, 100, 9000) 0) 0.001224 1) 0.001181 revset #3: last(0:9999, 100) 0) 0.000237 1) 0.000199

revset: reject negative number to select first/last n members Negative 'lim' doesn't make sense here, and it makes things complicated when using list[:lim].

revset: fix order of last() n members where n > 1 (BC) last() is implemented using a reversed iterator, so the result should be reversed again. I've marked this as BC since it's quite old bug seen in 3.0. The first bad revision is 4849f574aa24 "revset: changed last implementation to use lazy classes."

debugrevspec: add option to suppress list of computed revisions Test will be added later.

debugrevspec: add option to print representation of smartset object It's possible by -v, but -v also prints a parsed tree. Test will be added later.

bookmarks: make sure we close the bookmark file after reading We previously lacked an explicit close of the bookmark file.

bookmarks: rephrase a comment to be shorted and clearer The initial motivation is that I need an initial level of indent in the next changeset o:-) It turn out I like the new version better.

checkheads: use a "lazyancestors" object for allfuturecommon Instead of walking all ancestors to compute the full set we now check membership lazily. This massively speed. On a million-ish revision repository, this remove 14 seconds from the push logic, making the checkheads function disappear from profile.

checkheads: use "revnum" in the "allfuturecommon" set The obsolete post-processing needs to know the extend of the pushed set. The way it is implemented is... suboptimal. It build a full set of all nodes in the pushset and it does so using changectx. We have much better API for this now. The simplest is to use the existing lazy ancestors computation. That logic uses revnum and not node (for good reason) so we start with updating the post-processing code to handle a "allfuturecommon" set containing revision numbers.

checkheads: use 'nodemap.get' to convert nodes to revs We are about to call 'torev' on node that might be locally missing. In this case, 'nodemap.revs' will return None (something valid in our usecase) while 'changelog.rev' would raise an exception. We make this change in a distinct changeset to show it does not impact the tests.

checkheads: pass "ispushed" function to the obsmarkers logic We are about to make "allfuturecommon" a set of revs instead of a set of nodes. The function updated in this patch do not needs to know about these details so we just pass it a 'ispushed(node)' function. This will simplify the next changeset.

profile: drop maybeprofile It seems sufficiently simple to use "profile(enabled=X)" to not justify having a dedicated context manager just to read the config. (I do not have a too strong opinion about this).

profile: support --profile in alias and abbreviated version (--prof) We now process the "--profile" a second time after alias has been processed and the command argument fully parsed. If appropriate we enable profiling at that time. In these situation, the --profile will cover less than if the full --profile flag was passed on the command line. This is better than the previous behavior (flag ignored) and still fullfil multiple valid usecases.

profile: make the contextmanager object available to the callers This will allow calling methods on the object in the code using the context manager.

profile: introduce a knob to control if the context is actually profiling This is a step toward allowing context where the profiling in enabled withing the context range. This also open the way to kill the dedicated "maybeprofile" context manager and keep only one of 'profile' and 'maybeprofile'.

profile: introduce a "start" method to the profile context The start method is doing all profiler setup and activation. It is currently unconditionally called by '__init__' but this will be made more flexible in later changesets.

profile: upgrade the "profile" context manager to a full class So far we have been able to use a simple decorator for this. However using the current context manager makes the scope of the profiling in dispatch constrainted and the time frame to decide to enable profiling quite limited (using "maybeprofile") This is the first step toward the ability to enable the profiling from within the profiling scope. eg:: with maybeprofiling(ui) as profiler: ... bar.foo(): ... if options['profile']: profiler.start() ... fooz() ... My target usecase is adding support for "--profile" to alias definitions with effect. These are to be used with "profiling.output=blackbox" to gather data about operation that get slow from time to time (eg: pull being minutes instead of seconds from time to time). Of course, in such case, the scope of the profiling would be smaller since profiler would be started after running extensions 'reposetup' (and other potentially costly logic), but these are not relevant for my target usecase (multiple second commits, multiple tens of seconds pull). Currently adding '--profile' to a command through alias requires to re-spin a Mercurial binary (using "!$HG" in alias), which as a significant performance impact, especially in context where startup performance is being worked on... An alternative approach would be to stop using the context manager in dispatch and move back to a try/finally setup.

setup: avoid linker warnings on Windows about multiple export specifications The PyMODINIT_FUNC macro contains __declspec(dllexport), and then the build process adds an "/EXPORT func" to the command line. The 64-bit linker flags this [1]. Everything except zstd.c and bser.c are covered by redefining the macro in util.h [2]. These modules aren't built with util.h in the #include path, so the redefining hack would have to be open coded two more times. After seeing that extra_linker_flags didn't work, I couldn't find anything authoritative indicating why, though I did see an offhand comment on SO that CFLAGS is also ignored on Windows. I also don't fully understand the interaction between msvccompiler and msvc9compiler- I first subclassed the latter, but it isn't used when building with VS2008. I know the camelcase naming isn't the standard, but the HackedMingw32CCompiler class above it was introduced 5 years ago (and I think the current style was in place by then), so I assume that there's some reason for it. [1] https://support.microsoft.com/en-us/help/835326/you-receive-an-lnk4197-error-in-the-64-bit-version-of-the-visual-c-compiler [2] https://bugs.python.org/issue9709#msg120859

memctx: always use cache for filectxfn I don't see a downside to doing this unless I'm missing something. Thanks to foozy for correcting my previous bad logic.

test-hardlinks: stabilize for Windows This broke in c2cb0de25120, which breaks hardlinks when the executable bit is toggled.

releasenotes: add more tests for formatting and merging of release notes

releasenotes: command to manage release notes files Per discussion on the mailing list, we want better release notes for Mercurial. This patch introduces an extension that provides a command for producing release notes files. Functionality is implemented as an extension because it could be useful outside of the Mercurial project and because there is some code (like rst parsing) that already exists in Mercurial and it doesn't make sense to reinvent the wheel. The general idea with the extension is that changeset authors declare release notes in commit messages using rst directives. Periodically (such as at publishing or release time), a project maintainer runs `hg releasenotes` to extract release notes fragments from commit messages and format them to an auto-generated release notes file. More details are explained inline in docstrings. There are several things that need addressed before this is ready for prime time: * Moar tests * Interactive merge mode * Implement similarity detection for individual notes items * Support customizing section names/titles * Parsing improvements for bullet lists and paragraphs * Document which rst primitives can be parsed * Retain arbitrary content (e.g. header section/paragraphs) from existing release notes file * Better error messages (line numbers, hints, etc)

packagelib: use LANGUAGE=C for "hg version" If "hg version" does not contain "version" (e.g. Japanese), $hgversion was empty and rpmbuild failed.

rpms: add hgdemandimport in files

rpms: remove '%if "%{?pythonver}" != "2.4"' Mercurial requires python >= 2.7.

obsstore: do not load all markers to detect duplication This will make duplication detection something like O(newmarkers) instead of O(obsstore).

clonebundles: reference correct config option This option is no longer experimental.

filestat: move __init__ to frompath constructor We're going to add a `fromfp` constructor soon, and this also allows a filestat object for a non-existent file to be created.

test-dirstate-race: ensure that a isn't in the lookup set at the end We're going to rely on this in upcoming patches.

hghave: add test for whether fsmonitor is enabled This uses the HGFSMONITOR_TESTS environment variable that fsmonitor-run-tests.py adds.

tests: add a wrapper to run fsmonitor tests This script does a bunch of non-trivial configuration work: in particular, it sets up an isolated instance of Watchman which isn't affected by global state and can be torn down on completion. This script also sets the HGFSMONITOR_TESTS environment variable, which hghave will use in the next patch to allow gating on whether fsmonitor is enabled. With fsmonitor enabled, there appear to be a number of failures in the test suite. It's not yet clear to me why they're happening, but if someone would like to jump in and fix some of them I hope this will be helpful for that.

setdiscovery: improves logged message The 'srvheads' list contains all server heads including the common ones. We adjust 'ui.log' message to provide more useful information about server heads locally unknown. The performance impact of turning the list to set is negligible (about 1e-4s) compared to the rest of the discovery cost, so I'm taking the easy path.

tests: handle variation between pure and normal output in annotate --skip I'm pretty sure that both results are valid, depending on how you slice the edits.

bisect: improve option validation message

context: inline makememctx (API) I have always thought it weird that we have a helper method instead of just using __init__. So, I ripped it out.

context: add convenience method for returning a memfilectx from a patch This is mostly a copy of what makememctx does but refactored to make it behave more like our other convenience methods.

memctx: refactor inline getfilectx into convenience method No actual logic is changing, just moving code so __init__ is easier to read.

hgweb: refresh styling of gitweb's search form gitweb was missing the hint hover box. So that was added. Also, the positioning of the form was absolute and it didn't vertically align on all pages. The element has been moved inline with the navigation links (which now are contained in a div) and flexbox is used to obtain sane alignment of the navigation links and search form. For those new to flexbox, "justify-content: space-between" basically says to maximize space elements. You can use it to easily get left and right justified containers without having to worry about width, floating, etc. "align-items: center" centers all items in a cross-axis. I've literally wasted hours trying to figure out both these problems before flexbox. Flexbox is amazing. Flexbox has been supported by Chrome and Firefox for a few years. But it is only supported by IE 11. I'm willing to wager that people using this either won't be using IE or will be using IE 11. So I'm willing to be a bit aggressive in adopting flexbox because it makes CSS alignment so much easier.

hgweb: consistently add search form to all gitweb pages Paper has it on all pages. Not sure why gitweb doesn't. I think it should be everywhere because it is a useful feature. Also, we weren't consistently adding the HTML in the same place. This was OK since the element is absolutely positioned. But this bothered me a bit, so I went ahead and fixed it.

hgweb: consolidate search form for gitweb

hgweb: consolidate search form for monoblue Same deal as for paper.

hgweb: consolidate search form for paper AFAICT this was mostly a bunch of copy pasta. The only variation is some pages defined a "value" attribute. The "query" variable will just be empty on pages that don't accept it. So let's consolidate the template and remove the redundancy.

test-extension: fix load path for Windows The previous code was trying to load the extension from /tmp/hgtests.xxx/..., but the actual path for tests is C:\Users\...\Temp\hgtests.xxx\... I assume that the former is an MSYS path that maps somewhere under C:\MinGW.

test-setdiscovery: stabilize for Windows Windows wants double quotes here.

convert: correct the documentation about whitespace in branchmap branches Might as well let the users know they can get rid of branch names with spaces.

py3: use pycompat.bytestr() instead of str()

py3: convert bool variables to bytes

context: avoid writing outdated dirstate out (issue5584) Before this patch, workingctx.status() may cause writing outdated dirstate out, if: - .hg/dirstate is changed simultaneously after last loading it, - there is any file, which should be dirstate.normal()-ed Typical issue case is: - the working directory is updated by "hg update" - .hg/dirstate is updated in background (e.g. fsmonitor) This patch compares identities of dirstate before and after acquisition of wlock, and avoids writing outdated dirstate out, if change of .hg/dirstate is detected.

tests: factor external procedures out for portability Fortunately, "&&" is treated as "execute next, if previous doesn't fail" both on POSIX and Windows. But keeping portability of "dirstaterace.command" manually is troublesome. This patch factors external procedures out as a shell script for portability. "sh SCRIPT" always allows scripting in POSIX style. This change is also for convenience. Fixed script name can reduce command line arguments. "r" prefix is needed for "sh '$TESTTMP/dirstaterace.sh'", because $TESTTMP contains backslash on Windows.

dirstate: add identity information to detect simultaneous changing in storage This identity is used to examine whether dirstate is simultaneously changed in storage after previous caching (see issue5584 for detail). util.cachestat can't be used for this purpose, because it has no valuable information on Windows. On the other hand, util.filestat can detect changing dirstate in storage certainly, regardless of platforms. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan Strictly speaking, if underlying filesystem doesn't support ctime/mtime, util.filestat can't detect simultaneous changing in storage as expected. But simultaneous changing on such (very rare) platform can't be detected regardless of this patch series. Therefore, util.filestat should be reasonable identity for almost all usecases.

util: make filestat.__eq__ return True if both of self and old have None stat For convenience to compare two filestat objects regardless of None-ness of stat field.

vfs: create copy at renaming to avoid file stat ambiguity if needed In order to fix issue5418, bff5ccbe5ead made vfs.rename(checkambig=True) omit advancing mtime of renamed file, if renamed file is owned by another (EPERM is raised in this case). But this omission causes rewinding mtime at restoration in such situation, and makes avoiding file stat ambiguity difficult, because ExactCacheValidationPlan assumes that mtime should be advanced, if a file is changed in same ctime. https://www.mercurial-scm.org/wiki/ExactCacheValidationPlan Ambiguity of file stat also requires issue5584 to be fixed with other than file stat, but "hash of file", "generation ID" and so on were already rejected ideas (please see original RFC linked from "Outline of issue" in ExactCacheValidationPlan page). This omission occurs: - only for non append-only files (dirstate, bookmarks, and phaseroots), and - only if previous transaction is rollbacked by another user The latter means "sharing a repository clone via group permission". This is reasonable usecase, but not ordinary for many users, IMHO. "hg rollback" itself has been deprecated since Mercurial 2.7, too. Therefore, increasing the cost at rollbacking previous transaction executed by another a little seems reasonable, for avoidance of file stat ambiguity. This patch does: - create copy of (already renamed) source file, if advancing mtime fails for EPERM - rename from copied file to destination file, and - advance mtime of renamed file, which is now owned by current user This patch also factors "self.join(src)" out to reduce redundancy.

vfs: factor out "rename and avoid ambiguity" to reuse This makes subsequent patch simple.

util: make filestat.avoidambig() return whether ambiguity is avoided or not

debugcommands: issue warning when repo has secret changesets (issue5589) This seems like a prudent thing to do. As the inline comment says, we may want to make this abort once the functionality is stabilized as part of `hg bundle`. Let's save that debate for another day.

streamclone: consider secret changesets (BC) (issue5589) Previously, a repo containing secret changesets would be served via stream clone, transferring those secret changesets. While secret changesets aren't meant to imply strong security (if you really want to keep them secret, others shouldn't have read access to the repo), we should at least make an effort to protect secret changesets when possible. After this commit, we no longer serve stream clones for repos containing secret changesets by default. This is backwards incompatible behavior. In case anyone is relying on the behavior, we provide a config option to opt into the old behavior. Note that this defense is only beneficial for remote repos accessed via the wire protocol: if a client has access to the files backing a repo, they can get to the raw data and see secret revisions.

json: pass formatting options recursively This bug was introduced in 654e9a1c8a6c. It's okay to escape <>, but is unnecessary for command output.

json: avoid extra string manipulation of dict keys A key must be string per JSON spec, and that's also true for template dicts.

test-obsolete: include <> in user field to check JSON escapes I found json() filter doesn't pass formatting options recursively. That's why <> are escaped.

templatefilers: correct filename in header comment