templater: introduce {latesttag()} function to match a pattern (issue4184) This allows the latest class of tag to be found, such as a release candidate or final build, instead of just the absolute latest. It doesn't appear that the existing keyword can be given an optional argument. There is a keyword, function and filter for 'date', so it doesn't seem harmful to introduce a new function with the same name as an existing keyword. Most functions are pretty Mercurial agnostic, but there is {revset()} as precedent. Even though templatekw.getlatesttags() returns a single tuple, one entry of which is a list, it is simplest to present this as a list of tags instead of a single item, with each tag having a distance and change count attribute. It is also closer to how {latesttag} returns a list of tags, and how this function works when not given a '%' operator.

templatekw: introduce showlatesttags() to handle {latesttag} keywords The keywords {changes}, {distance} and {tag} will be available on a future template method that will allow pattern matching against tag names. For consistency, these should be available on the existing {latesttag} keyword as well. I debated whether or not to add {tag} instead of just continuing with the existing {latesttag}. But it seems clearer not to have the same name for two distinct things (a list in the LHS of %, and an individual tag value on the right). The value of latesttags[0] is the date of commit for the cset to which the tag is applied (i.e. not the date the tag was applied), and therefore isn't made visible because it doesn't seem interesting. It appears that this is merely an internal implementation detail for sorting csets in a stable manner when there are different branches.

templatekw: factor out the changessincetag calculation to a private method This will be reused in the next patch.

templatekw: allow getlatesttags() to match a specific tag pattern This will allow the latest class of tag to be found, such as a release candidate or final build, instead of just the absolute latest. It will be exposed in a future patch. It's unfortunate that the original 'latesttags' cache can't be used to determine the proper values, but it isn't fully populated for the entire repo. For example, the {latesttagdistance} keyword on the Mecurial repo builds the cache up back to the revision for 1.4. If the pattern was 're:^0\.\d$', that wouldn't be in the cache. Maybe this can be optimized some other way, but for now, this is the simpliest implementation.

util: extract stringmatcher() from revset This is used to match against tags, bookmarks, etc in revsets. It will be used in a future patch to do the same tag matching in templater.

util.chunkbuffer: avoid extra mutations when reading partial chunks Previously, a read(N) where N was less than the length of the first available chunk would mutate the deque instance twice and allocate a new str from the slice of the existing chunk. Profiling drawed my attention to these as a potential hot spot during changegroup reading. This patch makes the code more complicated in order to avoid the aforementioned 3 operations. On a pre-generated mozilla-central gzip bundle, this series has the following impact on `hg unbundle` performance on my MacBook Pro: before: 358.21 real 317.69 user 38.49 sys after: 301.57 real 262.69 user 37.11 sys delta: -56.64 real -55.00 user -1.38 sys

util.chunkbuffer: refactor chunk handling logic This will make the next patch easier to read. It provides no benefit on its own.

util.chunkbuffer: special case reading everything The new code results in simpler logic within the while loop. It is also faster since we avoid performing operations on the queue and buf collections. However, there shouldn't be any super hot loops for this since the whole point of chunkbuffer is to avoid reading large amounts of data at once. This does, however, make it easier to optimize chunkbuffer in a subsequent patch.

revert: add reference to backout

backout: add reference to revert

localrepo: allow wlock to be inherited This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances. When allowed, the parent process will pass down requisite information to the child process by way of this environment variable.

lock.release: don't call postrelease functions for inherited locks Review feedback from Pierre-Yves David. The postrelease functions typically assume the lock is not held at all.

lock: turn prepinherit/reacquire into a single context manager Review feedback from Pierre-Yves David. This makes the overall code cleaner and less error-prone, and makes a previously explicitly checked error state impossible.

localrepo: add a note about parentenvvar Review feedback from Pierre-Yves David.

exchange: add "streaming all changes" to bundle2 pulling This is the beginning of client-side support for performing a stream clone using bundle2. The main bundle2 pull function checks whether to perform a streaming clone and outputs a message if so. While we have a duplicate message, it seems easier to have all the bundle2 console writing in one location and in an easy-to-read conditional block.

streamclone: move "streaming all changes" message location Previously, the message was printed after we requested and started processing the remote stream. This seems like something that we should do before calling out to the remote. Moving it also makes it easier to deal with the bundle2 implementation.

streamclone: move payload header generation into own function The stream clone data over the wire protocol contains a header line indicating total file count and data size. In bundle2, this metadata can be captured by a part parameter and doesn't need to be in the body. In preparation for bundle2, have generatev1() return the raw metadata and move the header generation to its own function.

streamclone: move payload header line consumption bundle2 parts have parameters. These are a logical place for "header" data such as the file count and payload size of stream clone data. In preparation for supporting stream clones with bundle2, move the consumption of the header line from the payload into maybeperformlegacystreamclone(). Note: the header line is still being emitted by generatev1(). This will be addressed in a subsequent patch.

streamclone: teach canperformstreamclone to be bundle2 aware We add an argument to canperformstreamclone() to return False if a bundle2 stream clone is available. This will enable the legacy stream clone step to no-op when a bundle2 stream clone is supported. The commented code will be made active when bundle2 supports streaming clone. This patch does foreshadow the introduction of the "stream" bundle2 capability and its "v1" sub-capability. The bundle2 capability mirrors the existing "stream" capability and is needed so clients know whether a server explicitly supports streaming clones over bundle2 (servers up to this point support bundle2 without streaming clone support). The sub-capability will denote which data formats and variations are supported. Currently, the value "v1" denotes the existing streaming clone data format, which I intend to reuse inside a bundle2 part. My intent is to eventually introduce alternate data formats that can be produced and consumed more efficiently. Having a sub-capability means we don't need to introduce a new top-level bundle2 capability when new formats are introduced. This doesn't really have any implications beyond making the capabilities namespace more organized.

streamclone: refactor canperformstreamclone to accept a pullop This isn't strictly necessary. But a lot of pull functionality accepts a pulloperation so extra state can be added easily. It also enables extensions to perform more powerful things.

exchange: expose bundle2 availability on pulloperation Like the previous patch, the value is cached and will prevent a function level import in streamclone.py.

exchange: expose bundle2 capabilities on pulloperation This adds a cache and makes accessing the capabilities slightly simpler, as you don't need to directly go through the bundle2 module. This will also help prevent a function-level import in streamclone.py. This patch arguably isn't necessary. But I think it makes things slightly nicer.

keyword: make restrict mode False while updating files for rollback This is a preparation for using 'repo.rollback()' instead of aborting a current running transaction for "shelve" and "unshelve". Before this patch, updating files as a part of 'repo.rollback()' overridden by keyword extension always follows 'restrict' mode of the command currently executed. "merge", "unshelve" and so on should be 'restrict'-ed, because keyword expansion may cause unexpected conflicts at merging while these commands. But, if 'repo.rollback()' is invoked while executing 'restrict'-ed commands, modified files in the working directory are marked as "CLEAN" unexpectedly by code path below: # 'lookup' below is True at updating modified files for rollback kwcmd = self.restrict and lookup # kwexpand/kwshrink : if kwcmd: self.repo.dirstate.normal(f) On the other hand, "rollback" command isn't 'restrict'-ed, because rollbacking itself doesn't imply merging. Therefore, disabling 'restrict' mode while updating files as a part of 'repo.rollback()' regardless of current 'restrict' mode should be reasonable.

streamclone: rename and document maybeperformstreamclone() Upcoming patches will introduce bundle2 based streaming clones. Add "legacy" to the function name and add a docstring clarifying the intent of the function.

streamclone: move applyremotedata() into maybeperformstreamclone() Future work around stream cloning will be implemented in a bundle2 world. This code will only be used in the legacy code path and doesn't need to be abstracted or extensible.

branchmap: move branch cache code out of streamclone.py This is low-level branch map and cache manipulation code. It deserves to live next to similar code in branchmap.py. Moving it also paves the road for multiple consumers, such as a bundle2 part handler that receives branch mappings from a remote. This is largely a mechanical move, with only variable names and indentation being changed.

streamclone: move streamin() into maybeperformstreamclone() streamin() only had a single consumer. And it always only ever will because it is strongly coupled with the current, soon-to-be-superseded-by-bundle2 functionality. The return value has been dropped because nobody was using it.

streamclone: refactor maybeperformstreamclone to take a pullop Just like all the other pull steps. Consistency is good. This seems a little excessive right now since maybeperformstreamclone is such a short function. This will be addressed in a subsequent patch.

demandimport: replace more references to _demandmod instances _demandmod instances may be referenced by multiple importing modules. Before this patch, the _demandmod instance only maintained a reference to its first consumer when using the "from X import Y" syntax. This is because we only created a single _demandmod instance (attached to the parent X module). If multiple modules A and B performed "from X import Y", we'd produce a single _demandmod instance "demandmod" with the following references: X.Y = <demandmod> A.Y = <demandmod> B.Y = <demandmod> The locals from the first consumer (A) would be stored in <demandmod1>. When <demandmod1> was loaded, we'd look at the locals for the first consumer and replace the symbol, if necessary. This resulted in state: X.Y = <module> A.Y = <module> B.Y = <demandmod> B's reference to Y wasn't updated and was still using the proxy object because we just didn't record that B had a reference to <demandmod> that needed updating! With this patch, we add support for tracking which modules in addition to the initial importer have a reference to the _demandmod instance and we replace those references at module load time. In the case of posix.py, this fixes an issue where the "encoding" module was being proxied, resulting in hundreds of thousands of __getattribute__ lookups on the _demandmod instance during dirstate operations on mozilla-central, speeding up execution by many milliseconds. There are likely several other operation that benefit from this change as well. The new mechanism isn't perfect: references in locals (not globals) may likely linger. So, if there is an import inside a function and a symbol from that module is used in a hot loop, we could have unwanted overhead from proxying through _demandmod. Non-global imports are discouraged anyway. So hopefully this isn't a big deal in practice. We could potentially deploy a code checker that bans use of attribute lookups of function-level-imported modules inside loops. This deficiency in theory could be avoided by storing the set of globals and locals dicts to update in the _demandmod instance. However, I tried this and it didn't work. One reason is that some globals are _demandmod instances. We could work around this, but it's a bit more work. There also might be other module import foo at play. The solution as implemented is better than what we had and IMO is good enough for the time being. It's worth noting that this sub-optimal behavior was made worse by the introduction of absolute_import and its recommended "from . import X" syntax for importing modules from the "mercurial" package. If we ever wrote performance tests, measuring the amount of module imports and __getattribute__ proxy calls through _demandmod instances would be something I'd have it check.

demandimport: refactor processfromitem This will match the next patch smaller and easier to read.

demandimport: consolidate code for processing items in fromlist This code was mostly duplicated. An upcoming patch will add more complexity, making the duplication harder to justify. Consolidate the code.

pager: recreate stdout to make it line-buffered We want to see partial command results as soon as possible. But the buffering mode of stdout (= pager's stdin) was set to fully-buffered because it isn't associated with a tty. So, this patch recreates new stdout object to force its buffering mode. Because two file objects are associated with the same stdout fd and their destructors will call close(), one of them must be closed carefully. Python expects that the stdout fd never be closed even after sys.stdout.close() [1], but newstdout has no such hack. So this patch calls newstdout.close() immediately before duplicating the original stdout fd to sys.stdout. operation sys.stdout newstdout fd --------------------- ---------- --------- -------- newstdout.close() open closed closed os.dup2(stdoutfd, ..) open closed open del sys.stdout closed closed open [1] [1]: https://hg.python.org/cpython/file/v2.7.10/Python/sysmodule.c#l1391

pager: rename variables of backup fds The next patch will add backup file objects, so the original variable names would be confusing.

pager: inline _pagersubprocess() into _runpager() We don't need _pagersubprocess() because the fork version was removed at 59d794154e8d.

ui: send traceback of devel warning to appropriate output stream If ui.ferr is a command-server channel, traceback should be written to it.

util.system: compare fileno to see if it needs stdout redirection Future patches will reopen stdout to be line-buffered, so sys.stdout may be different object than sys.__stdout__.

exchange: move stream clone logic into pull code path Stream clones are a special case of clones. Clones are a special case of pull. Most of the logic for deciding what to do at pull time is in exchange.py. It makes sense for the stream clone determination to live there as well. This patch moves the calling of the stream clone code into pull(). The checks in streamclone.canperformstreamclone() ensure that we don't perform a stream clone unless it is possible. A future patch will convert maybeperformstreamclone() to accept a pullop to make it consistent with everything else in pull(). It will also grow some functionality (in case you doubted the necessity of a 4 line function).

exchange: teach pull about requested stream clones An upcoming patch will move the invocation of stream cloning logic to the normal pull code path (from localrepository.clone). In preparation for this, we teach pull() and pulloperation about whether a streaming clone is requested. The return logic in localrepository.clone() has been reformatted slightly because of line length issues.

streamclone: add explicit check for empty local repo Stream clone doesn't work with non-empty local repositories. In upcoming patches, we'll move stream cloning to the regular pull code path. Add an explicit check on the repository being empty to prevent streaming clones to non-empty repos.

streamclone: refactor code for deciding to stream clone Having this in a standalone function will eventually enable bundle2 to share code with the bundle1 code path. While I was here, I also added some comments to add clarity.

streamclone: move streaming clone logic from localrepo This is the last remnants of streaming clone code in localrepo.py. This is a mostly mechanical transplant of code to a new file. Only a rewrite of "self" to "repo" was performed. The code will be significantly refactored in upcoming patches. So don't scrutinize it too closely.

streamclone: move _allowstream() from wireproto While we're moving things into streamclone.py...

streamclone: move code out of exchange.py We bulk move functions from exchange.py related to streaming clones. Function names were renamed slightly to drop a component redundant with the module name. Docstrings and comments referencing old names and locations were updated accordingly.

streamclone: move stream_in() from localrepo Another basic content move. The underscore from the function name was removed to comply with naming standards.

streamclone: move applystreamclone() from localrepo.py Upcoming patches will modernize the streaming clone code. Streaming clone data and code kind of lives in its own world. exchange.py is arguably the most appropriate existing location for it. However, over a dozen patches from now it became apparent that there was a lot of code related to streaming clones and that having it contained within its own module would make it easier to comprehend. So, we establish streamclone.py. It's worth noting that streamclone.py existed a long time ago, last seen in the 1.6 release. It was removed in 04f76a954842. The function was renamed as part of the move because its old name was redundant with the new module name. The only other content change was "self" was renamed to "repo" and minor grammar in the docstring was updated.

exchange: add docstring to pull() This seems like the kind of important function that should be documented better.

localrepo: allow creating inherited locks This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances.

error: make lock inheritance contract violations a subclass of RuntimeError This is more appropriate, per Pierre-Yves David.

templatekw: hide help of "branches" by DEPRECATED marker See 43e872a35f8a for details.

templatekw: remove dockeywords hack Now all template keywords are defined as real functions.

templatekw: port implementation of showparents() from changeset_templater It isn't cool, but we can peek at ui flag via repo.ui. So, it is possible to implement showparents() in templatekw, and therefore we can eliminate the dockeywords hack.

templatekw: reorder stub of showparents() function It will have an implementation.

changeset_printer: move _meaningful_parentrevs() to scmutil It will be used by templatekw.

test-run-tests: add globs for Windows There are two other unrelated failures in this test. For some reason child1 has a space after it, thus 2 spaces before the glob, otherwise the glob is ignored and removed.

test-commandserver: generate files with a fixed EOL for test stability The hashes were different on Windows.

test-debugextensions: sanitize JSON output for Windows Windows is printing a double backslash, which isn't handled by (glob): --- c:/Users/Matt/Projects/hg/tests/test-debugextensions.t +++ c:/Users/Matt/Projects/hg/tests/test-debugextensions.t.err @@ -47,7 +47,7 @@ { "buglink": "", "name": "color", - "source": "*/hgext/color.pyc", (glob) + "source": "c:\\Users\\Matt\\Projects\\hg\\hgext\\color.pyc", "testedwith": "internal" }, {

test-strip: add glob for Windows

bundle2: generate check:heads in a independent function

merge with stable

cmdutil: handle multiline translations of HG: messages safely Before this, if a localizer/localization included a multiline message, and didn't prefix the intermediate lines with 'HG: ', then the line would be a candidate for inclusion in the commit message -- which isn't ideal.

strip: compress bundle2 backup using BZ Storing uncompressed bundle on disk would be a regression. Strip backup using bundle2 are now compressed when requested.

writebundle: add a compression argument for the bundle2 case Bundle2 compression is more complex than the bundle1 one. Therefore it is handled by the bundler itself. Moreover, on-disk bundle2 will probably have a large number of flavors so simply adding a new "format" for it does not seems the way to go. This will be used in the next changeset to compress bundle2 strip backup.

strip: use bundle2 + cg2 by default when repository use general delta The bundle10 format (plain changegroup-01) does not support general delta and result into expensive delta re-computation when stripping. If the repository is general delta, we store backups as bundle20 containing a changegroup-02 payload. We remove the experimental feature related to strip backup bundle format because this achieve the same goal in a leaner way. Removing the experimental option is fine, that is why it experimental in the first place. Compression of these bundles are coming in later changesets.

run-tests: report paths saved by --keep-tmpdir

urls: bulk-change primary website URLs

urls: bulk-change BTS urls to new location

Added signature for changeset 9a466b9f9792

Added tag 3.5.2 for changeset 9a466b9f9792

largefiles: restore archiving largefiles with hgweb (issue4859) This regressed in 7699d3212994, when trying to conditionally disable archiving of largefiles. I'm not sure if wrapfunction() is the right way to do this, but it seems to work. The mysterious issue with lfstatus getting out of sync in the proxy and the unfiltered view crops up again here. See the referenced cset for more info.

i18n-pt_BR: synchronized with f31ddc9bfa5f

help: hide deprecated filesets, revsets and template items if not verbose This allows us to hide {branches} template keyword in a better way. Currently it is achieved by copying keywords table and deleting 'branches' from it.

help: pass around ui to rewriter hooks (API) makeitemsdoc() will hide DEPRECATED items conditionally.

help: pass around ui to doc loader (API) This is necessary to hide DEPRECATED items conditionally. Flagged as API change because it will break "hg help git|subversion".

gendoc: use real ui in place of stdout ui attributes will be required by a help function, so a file object can't be used as a fake ui.

check-seclevel: use ui to show status and error messages Future patches will require ui module to be passed to a help function, so let's use it where appropriate. Additional parens are necessary to silence a check-code warning.

unbundle: test and fix for clean abort on unknown bundle2 feature This introduce a test for the feature and fix the exception name that is different on default.

merge with stable

localrepo: recreate phasecache if changelog was modified (issue4855) Because _phaserevs and _phasesets cache revision numbers, they must be invalidated if there are new commits or stripped revisions. We could do that by calling _phasecache.invalidate(), but it wasn't simple to be integrated with the filecache mechanism. So for now, phasecache will be recreated after repo.invalidate() if 00changelog.i was modified before.

progress: force a repaint of a printed progress bar after a clear() This avoids some visual flickering of the progress bar in convert and probably some other operations. Previously, a line of output would erase the progress bar, and then it would wait `progress.refresh` seconds (default of 0.1) before redrawing the progress bar. Now if we've ever painted a progress bar, we schedule the progress bar for immediate repainting on the next progress call, which helps lend the illusion that the progress bar is "always" there. In practice, it's merely there more of the time, but it ends up being a lot easier to read during convert.

tests: fix test-bundle2-format output Forgot to amend.

localrepo: recreate phasecache if changelog was modified (issue4855) Because _phaserevs and _phasesets cache revision numbers, they must be invalidated if there are new commits or stripped revisions. We could do that by calling _phasecache.invalidate(), but it wasn't simple to be integrated with the filecache mechanism. So for now, phasecache will be recreated after repo.invalidate() if 00changelog.i was modified before.

bundle2: allow compressed bundle This changeset adds support for a 'compression' parameter in bundle2 streams. When set, it controls the compression algorithm used for the payload part of the bundle2. There is currently no usage of this except in tests.

test-bundle2: dump bundle content using f --hexdump Thanks to Greg Szorc for pointing this out.

treemanifest: rework lazy-copying code (issue4840) The old lazy-copy code formed a chain of copied manifests with each copy. Under typical operation, the stack never got more than a couple of manifests deep and was fine. Under conditions like hgsubversion or convert, the stack could get hundreds of manifests deep, and eventually overflow the recursion limit for Python. I was able to consistently reproduce this by converting an hgsubversion clone of svn's history to treemanifests. This may result in fewer manifests staying in memory during operations like convert when treemanifests are in use, and should make those operations faster since there will be significantly fewer noop function calls going on. A previous attempt (never mailed) of mine to fix this problem tried to simply have all treemanifests only have a loadfunc - that caused somewhat weird problems because the gettext() callable passed into read() wasn't idempotent, so the easy solution is to have a loadfunc and a copyfunc.

manifest: rename treemanifest load functions to ease debugging I'm hunting an infinite recursion bug at the moment, and having both of these methods named just _load is muddying the waters slightly.

manifest: add id(self) to treemanifest __repr__ Also rename __str__ to __repr__ since that's what we really want for pdb.

bundlerepo: let bundle repo look in the _mancache When looking up a base revision, we were ignoring the contents that were already available in the manifest's _mancache. This patch allows us to use that data instead of reading from the revlog. This is useful in our pushrebase extension (which allows rebasing on the server side during a push) because it allows us to prefetch the bundle base manifest before aquiring the repo lock (1 second saving), which means doing less work inside the lock, which means a 20% higher commit rate.

check-seclevel: wrap entry point by function This is intended to narrow scope of local variables. The global _verbose flag will be replaced later by ui.verbose.

check-seclevel: set executable bit as it has shebang

bundle20: extract core payload generation in its own function We are about to allow compressing the core of the bundle2. So we extract the generation of this bits in its own parts to make this compression phases easier in a later changesets.

unbundle20: allow registering handlers for stream level parameters As a comment in the code have been asking for, it is now possible to register piece of code that handle parameters for the stream. I've been wondering is such function should be class methods or not. I eventually went for externally decorated methods to stick with the culture of extensibility from an extensions that apply to bundle2.

bundle2: allow to specify unsupported value on error A client may support an argument but not some of its values (eg: coming "compression" parameters). We allow this case to be carried in the exception.

bundle2: rename error exception class for unsupported feature The original name explicitly mention "Part", however it is also used outside of parts related feature. We rename from 'UnsupportedPartError' to 'BundleUnknownFeatureError' to fix this.

changegroup: use a different compression key for BZ in HG10 For "space saving", bundle1 "strip" the first two bytes of the BZ stream since they always are 'BZ'. So the current code boostrap the uncompressor with 'BZ'. This hack is impractical in more generic case so we move it in a dedicated "decompression".

monoblue: provide links to branches, tags and bookmarks by name This is adapted from cd842821db2c, that was added to paper for 3.5 release. It adds another way to refer to branches, tags and bookmarks in urls: by name. It's still possible to navigate to a specific changeset hash, but now you can get more descriptive urls straight from /summary page, for example. branchentry template (and so the whole branches table on /summary and /branches) lost the column that had a plain changeset hash, because tags and bookmarks don't have this column and also because there is already a way to address branch by its changeset hash (changeset link just next to it). Maybe we can instead bring this column with a plain changeset hash to tags and bookmarks, but this, more terse, new look feels fine.

gitweb: provide links to branches, tags and bookmarks by name This is adapted from cd842821db2c, that was added to paper for 3.5 release. It adds another way to refer to branches, tags and bookmarks in urls: by name. It's still possible to navigate to a specific changeset hash, but now you can get more descriptive urls straight from /summary page, for example. branchentry template (and so the whole branches table on /summary and /branches) lost the column that had a plain changeset hash, because tags and bookmarks don't have this column and also because there is already a way to address branch by its changeset hash (changeset link just next to it). Maybe we can instead bring this column with a plain changeset hash to tags and bookmarks, but this, more terse, new look feels fine.