extract: simplify parents assignement We move to the more standard style of: value = default if special: value = other This is more consistent and compact.

patch: move 'extract' return to a dictionnary The final goal here is to be able to parse, return and process arbitrary data from patch. This mirror the recently added ability to add arbitrary data to patch headers. The first step is to return something more flexible, so we return a dict without changing any other logic.

patchbomb: add experimental config of a "pullurl" and export it This config allows to specify a public location where your changeset can be found. It then include a dedicated patch header show a command to be used to retrieve the change. See the test for example. This is flagged as experimental because this feature is not safe until we have more logic to test that: - changeset actually exists on destination - changeset is draft on destination. As all this is experimental, bike shedding can happily happens before we remove the experimental flag.

export: introduce a generic way to add patch header on export Extensions currently have no easy way to add data to exported patch. This is now fixed using a generic mechanism in the same fashion used by bundle2. Tests are coming in the next changeset with its first user.

incoming: request a bundle2 when possible (BC) Incoming was using bundle1 in all cases, as bundle1 is restricted to changegroup1 and does not support general delta, this can lead to significant CPU overhead if the server is using general delta storage. We now properly request and store a bundle2 to disk. If the server include any output or error in the bundle, they will be stored on disk and replayed when the bundle is read. As 'hg incoming' is going to read the bundle right away, we call that 'good' enough and go back to the bigger plan of having general delta on by default. This was tracked as 4864

bundlerepo: indent some code to prepare next patch We are about to add a new condition. Code is indented in a separated patch for readability.

bundle2: add a way to just forward the bundle2 stream to another user There is use case for directly forward and bundle2 stream from the peer to a file (eg: 'hg incoming --bundle'), However ssh peers have no way to know the 'getbundle' is over except for actually interpreting the bundle. So we need to have the unbundle do the interpreting and forward job. The function is marked as private to highlight that this is terrible and that we are sorry.

bundle2: split parameter retrieval and processing We want to introduce a simple way to forward the content of a bundle2 stream. For this purpose, we will need to both yield the parameters block and process it (to apply any behavior change it might indicate).

changegroup: extract the file management part in its own function The current writebundle function do two things: - taking a changegroup-packer instance and storing it into a valid bundle with proper header. - creating a temporary or requested file to store that bundle We would like to make it easier to forward bundle stream directly from a remote peer to a file, so we split the two logic to be able to skip the one about building a valid bundle (the remote is already sending one).

unbundle: properly read head modification result from bundle2 We were reading the wrong key...

revset: strip off "literal:" prefix from bookmark not found error This is what branch() and tag() do.

revset: do not fall through to revspec for literal: branch (issue4838) If "literal:" is specified, it must not be a revset expression. It should error out with a better message.

hgweb: ensure both foreground and background colors are specified (issue4872) When users configure the default foreground or background color to non-default (black on white) values, several hgweb styles lack contrast for headers and table row items. This patch fixes that by ensuring that where either foreground or background colors are specified, both are specified.

templater: do not pre-evaluate generator keyword at runsymbol (issue4868) It was introduced by e06e9fd2d99f, but the important code was removed by a3c2d9211294. So there was no positive effect other than exhausting memory. The problem spotted by e06e9fd2d99f is that you can't use a generator keyword more than once. For example, in hgweb template, "{child} {child}" doesn't work because the first "{child}" consumes the generator. But as a3c2d9211294 says, the fix was wrong because it could overwrite a callable keyword that returns a generator. Also the fix didn't work for a generator of generator such as "{diff}" keyword. So, the proper fix for that problem would be to not put a generator in a keyword table. Instead, it should be a factory of a generator. Note that this should fix the memory issue in hgweb, but my firefox killed by OOM in place. Be careful to not use a modern web browser to test the issue4868.

merge: drop special parent assignment in the obsolete case We can safely drop this because the very same assignment is enforcement later in the function. Dropping it will make it simpler to extract the default destination logic in its own function.

bundle: use bundle2 if repository uses general delta As bundle1 does not support generaldelta, this would mean recomputing delta at bundle time. This is similar to what we do for strip and shelve and was tracked as issue4865.

parsebundletype: add a comment for future generation

bundle: extend the format of --type to support version and compression We had some basic undocumented support for uncompressed bundle2 support. We now have an official extensible syntax to specify both format type and compression (eg: bzip2-v2). In practice, this changeset introduce the 'v1' and 'v2' identifier to make it possible to combine format and compression. The default format is still 'v1'. We'll care about picking 'v1' or 'v2' in regard with general delta in the next changesets.

hgweb: fix border-radius for standards-based browsers While here, remove an old gecko-specific property, as gecko has supported the unprefixed version for many years.

filemerge: move precheck to before files are written out This is much simpler, and also avoids unnecessary disk IO.

filemerge: move 'merging' output to before file creation This has no visible impact but makes some future work simpler.

filemerge.filemerge: make a tuple containing merge paths on disk We're going to need this same tuple elsewhere.

filemerge: switch trymerge boolean to mergetype enum trymerge = False becomes mergetype = nomerge, and trymerge = True becomes mergetype = fullmerge or mergeonly, depending on whether a premerge happens.

filemerge: add some merge types We're going to turn the 'trymerge' boolean into a 'mergetype' enum with these three possible values.

shelve: restore unshelved dirstate explicitly after aborting transaction Before this patch, "hg unshelve" uses aborting a current transaction to discard temporary changes while unshelving. This assumes that dirstate changes in a transaction scope are kept even after aborting it. But this assumption will be broken by "transactional dirstate". See the wiki page below for detail about it. https://mercurial.selenic.com/wiki/DirstateTransactionPlan This patch explicitly saves shelved dirstate just before aborting current transaction, and restore dirstate with it after aborting by utility function '_aborttransaction()' added by previous patch.

shelve: restore shelved dirstate explicitly after aborting transaction Before this patch, "hg shelve" uses aborting a current transaction to discard temporary changes while shelving. This assumes that dirstate changes in a transaction scope are kept even after aborting it. But this assumption will be broken by "transactional dirstate". See the wiki page below for detail about it. https://mercurial.selenic.com/wiki/DirstateTransactionPlan This patch explicitly saves shelved dirstate just before aborting current transaction, and restore dirstate with it after aborting by utility function '_aborttransaction()' added by previous patch. This patch replaces 'if tr: tr.abort()' by 'lockmod.release(tr)', because the former is already done in '_aborttransaction()' (and the latter has no effect), if current transaction is aborted in it successfully. Otherwise, the latter is enough to trigger aborting.

shelve: add utility to abort current transaction but keep dirstate "hg shelve" and "hg unshelve" use aborting a current transaction to discard temporary changes while (un)shelving. This assumes that dirstate changes in a transaction scope are kept even after aborting it. But this assumption will be broken by "transactional dirstate". See the wiki page below for detail about it. https://mercurial.selenic.com/wiki/DirstateTransactionPlan This patch adds utility function "_aborttransaction()" to abort current transaction but keep dirstate changes for (un)shelving. 'dirstate.invalidate()' just after aborting a transaction should be removed soon by subsequent patch, which writes or discards in-memory dirstate changes at releasing transaction according to the result of it. BTW, there are some other ways below, which (seem to, at first glance) resolve this issue. But this patch chose straightforward way for ease of review and future refactorring. - commit transaction at first, and then rollback it It causes unintentional "dirty read" of running transaction to other processes at committing it. - use dirstateguard to save and restore shelved dirstate After DirstateTransactionPlan, making 'dirstate.write()' write in-memory changes into actual file requires 'transaction.writepending()' while transaction running. It causes meaningless writing other in-memory changes out, even though they are never referred. In addition to it, it isn't desirable that scope of dirstateguard and transaction intersects each other. - get list of files changed from the parent, keep it in memory, and emulate that changes after aborting transaction This additional memory consumption may block aborting transaction in large repository (on small resource environment).

dirstate: split write to write changes into files other than .hg/dirstate '_writedirstate()' is used mainly for "transactional dirstate". See the wiki page below for detail about it. https://mercurial.selenic.com/wiki/DirstateTransactionPlan

bookmarks: use recordchange instead of writing if transaction is active Before this patch, 'bmstore.write()' always write in-memory bookmark changes into '.hg/bookmarks' regardless of transaction activity. If 'bmstore.write()' is invoked inside a transaction and it writes changes into '.hg/bookmarks', then: - original bookmarks aren't restored at failure of that transaction This breaks "all or nothing" policy of the transaction. BTW, "hg rollback" can restore bookmarks successfully even before this patch, because original bookmarks are saved into '.hg/journal.bookmarks' at the beginning of the transaction, and it (actually renamed as '.hg/undo.bookmarks') is used by "hg rollback". - uncommitted bookmark changes are visible to other processes This is a kind of "dirty read" For example, 'rebase.rebase()' implies 'bmstore.write()', and it may be executed inside the transaction of "hg unshelve". Then, intentional aborting at the end of "hg unshelve" transaction doesn't restore original bookmarks (this is obviously a bug). This patch uses 'bmstore.recordchange()' instead of actual writing by 'bmstore._writerepo()', if any transaction is active This patch also removes meaningless restoring bmstore explicitly at the end of "hg shelve". This patch doesn't choose fixing each 'bmstore.write()' callers as like below, because writing similar code here and there is very redundant. before: bmstore.write() after: tr = repo.currenttransaction() if tr: bmstore.recordchange(tr) else: bmstore.write() Even though 'bmstore.write()' itself may have to be discarded by putting bookmark operations into transaction scope, this patch chose fixing it to implement "transactional dirstate" at first.

filemerge: run symlink check for :merge3 Just like :merge, :merge3 doesn't support merging symlinks.

filemerge: print correct name of tool for symlink checks Earlier we'd print ':merge' even if the tool was something else like ':union'. That's clearly a bug.

filemerge: normalize 'internal:foo' names to ':foo' In upcoming patches we're going to present these names in the UI -- it would be good not to present deprecated names.

filemerge: use symlinkcheck for :merge and :union This exposes a couple of bugs, both of which will be fixed in upcoming patches.

filemerge: add a precheck for symlinks This will be used by internal merge tools.

filemerge: call precheck if available In upcoming patches we'll define a precheck function for some merge tools.

filemerge: add a before-merge callback to internal merge tools We're going to separate the pre-merge and merge steps for merge tools. The merge step will be specific to the tool, but the pre-merge step will be common to all merge tools that need it. However, some merge tools run checks *before* the pre-merge step. This callback will allow that to continue to work.

filemerge: indent filemerge.filemerge This will make upcoming patches much easier to review.

test-bundle-type: replace unbundle with debugbundle We now have a convenient command to look at bundle contents, let's use it.

bundle: extract the parsing of the bundle type in a function We are going to introduce significant extensions of the bundle parsing code to support creation of bundle2 through the bundle command. As an early step, we extract the logic in its own function.

changegroup: add version argument to getchangegroup For some obscure reasons (probably upsetting a Greek goddess), getchangegroup did not had a 'version' argument to control the changegroup version. We fixes this to allow cg02 to be used with 'hg bundle' in the future.

changegroup: add version argument to getlocalchangegroup For some obscure reasons (probably upsetting a Greek goddess), getlocalchangegroup did not have a 'version' argument to control the changegroup version. We fix this to allow cg02 to be used with 'hg bundle' in the future.

shelve: bundle using bundle2 if repository is general delta (issue4862) This will prevent expensive delta computation on bundling and is similar to what we do for strip backup.

shelve: move changegroup generation inside writebundle We will generate different changegroup if general delta is enabled so we gather this in the lower level function. There wasn't any good reason to have it in the main code anyway.

shelve: compress on-disk bundle Shelve is using 'HG10UN' bundle for no obvious reason. Changing this to 'HG10BZ' as we do for other bundle stored on disk.

merge with stable

notify: fix fromauthor setting for 'incoming' hook type (issue4194) Set the author field in notification emails for the 'incoming' hook type in addition to 'changegroup' and 'outgoing' types.

templater: protect word() from crashing on out of range negative value The function isn't documented to work with negative values at all, but it does, which can be useful. However, the range check didn't account for this.

commands: add a new debug command to print merge state We're going to be extending the merge state very soon, and this will give us a way to test all that.

merge.mergestate: factor out code to validate v1/v2 records We're going to need this in another place in upcoming patches.

localrepo: prevent wlock from being inherited when a transaction is running Review feedback from Pierre-Yves David. A separate line of work is working to ensure that dirstate writes are written to a separate 'pending' file while a transaction is active. Lock inheritance currently conflicts with that, so dodge the issue by simply preventing inheritance while a transaction is running. Custom merge drivers aren't going to run inside a transaction, so this doesn't affect that.

lock: add a way to prevent locks from being inherited We want to prevent locks from being inherited sometimes (e.g. when there's a currently running transaction, which will break a lot of assumptions we're making in here.)

test-fncache: use args/kwargs for lock wrapper This is annoying to keep up to date, and also just plain unnecessary.

rebase: enable histedit for useful help with it

rebase: suggest help -e histedit Users unfamiliar with an extension should be reading the documentation for the feature, not the command.

rebase: factor out histedit help command

util: use tuple accessor to get accurate st_mtime value (issue4836) Because st.st_mtime is computed as 'sec + 1e-9 * nsec' and double is too narrow to represent nanoseconds, int(st.st_mtime) can be 'sec + 1'. Therefore, that value could be different from the one got by osutils.listdir(). This patch fixes the problem by accessing to raw st_mtime by tuple index. It catches TypeError to fall back to st.st_mtime because our osutil.stat does not support tuple index. In dirstate.normal(), 'st' is always a Python stat, but in dirstate.status(), it can be either a Python stat or an osutil.stat. Thanks to vgatien-baron@janestreet.com for finding the root cause of this subtle problem.

util: extract stub function to get mtime with second accuracy This function is trivial but will need a long comment why it can't use st.st_mtime. See the next patch for details.

scmutil: add a way for a repo's wlock to be inherited by a subprocess This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances. In the future, we'll call this for custom merge drivers.

scmutil: add a way for a subprocess to be run with an inheritable lock This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances. In an upcoming patch, we'll add an API for the wlock to be inherited.

localrepo: add a way to get the current wlock if it's held This will be useful to pass around a reference to the lock to some functions we're going to add to scmutil. We don't want those functions to live in localrepo to avoid bloat.

localrepo: add a way to get the current lock if it's held We'll use this in upcoming patches to return a reference to the wlock if it's held.

test-help: verify that extension keywords appear in help templates fixed by a2291c9c85a1

templatekw: add {changes}, {distance} and {tag} to the {latesttag} keyword

templater: introduce {latesttag()} function to match a pattern (issue4184) This allows the latest class of tag to be found, such as a release candidate or final build, instead of just the absolute latest. It doesn't appear that the existing keyword can be given an optional argument. There is a keyword, function and filter for 'date', so it doesn't seem harmful to introduce a new function with the same name as an existing keyword. Most functions are pretty Mercurial agnostic, but there is {revset()} as precedent. Even though templatekw.getlatesttags() returns a single tuple, one entry of which is a list, it is simplest to present this as a list of tags instead of a single item, with each tag having a distance and change count attribute. It is also closer to how {latesttag} returns a list of tags, and how this function works when not given a '%' operator.

templatekw: introduce showlatesttags() to handle {latesttag} keywords The keywords {changes}, {distance} and {tag} will be available on a future template method that will allow pattern matching against tag names. For consistency, these should be available on the existing {latesttag} keyword as well. I debated whether or not to add {tag} instead of just continuing with the existing {latesttag}. But it seems clearer not to have the same name for two distinct things (a list in the LHS of %, and an individual tag value on the right). The value of latesttags[0] is the date of commit for the cset to which the tag is applied (i.e. not the date the tag was applied), and therefore isn't made visible because it doesn't seem interesting. It appears that this is merely an internal implementation detail for sorting csets in a stable manner when there are different branches.

templatekw: factor out the changessincetag calculation to a private method This will be reused in the next patch.

templatekw: allow getlatesttags() to match a specific tag pattern This will allow the latest class of tag to be found, such as a release candidate or final build, instead of just the absolute latest. It will be exposed in a future patch. It's unfortunate that the original 'latesttags' cache can't be used to determine the proper values, but it isn't fully populated for the entire repo. For example, the {latesttagdistance} keyword on the Mecurial repo builds the cache up back to the revision for 1.4. If the pattern was 're:^0\.\d$', that wouldn't be in the cache. Maybe this can be optimized some other way, but for now, this is the simpliest implementation.

util: extract stringmatcher() from revset This is used to match against tags, bookmarks, etc in revsets. It will be used in a future patch to do the same tag matching in templater.

util.chunkbuffer: avoid extra mutations when reading partial chunks Previously, a read(N) where N was less than the length of the first available chunk would mutate the deque instance twice and allocate a new str from the slice of the existing chunk. Profiling drawed my attention to these as a potential hot spot during changegroup reading. This patch makes the code more complicated in order to avoid the aforementioned 3 operations. On a pre-generated mozilla-central gzip bundle, this series has the following impact on `hg unbundle` performance on my MacBook Pro: before: 358.21 real 317.69 user 38.49 sys after: 301.57 real 262.69 user 37.11 sys delta: -56.64 real -55.00 user -1.38 sys

util.chunkbuffer: refactor chunk handling logic This will make the next patch easier to read. It provides no benefit on its own.

util.chunkbuffer: special case reading everything The new code results in simpler logic within the while loop. It is also faster since we avoid performing operations on the queue and buf collections. However, there shouldn't be any super hot loops for this since the whole point of chunkbuffer is to avoid reading large amounts of data at once. This does, however, make it easier to optimize chunkbuffer in a subsequent patch.

revert: add reference to backout

backout: add reference to revert

localrepo: allow wlock to be inherited This is part of a series that will allow locks to be inherited by subprocesses in limited circumstances. When allowed, the parent process will pass down requisite information to the child process by way of this environment variable.

lock.release: don't call postrelease functions for inherited locks Review feedback from Pierre-Yves David. The postrelease functions typically assume the lock is not held at all.

lock: turn prepinherit/reacquire into a single context manager Review feedback from Pierre-Yves David. This makes the overall code cleaner and less error-prone, and makes a previously explicitly checked error state impossible.

localrepo: add a note about parentenvvar Review feedback from Pierre-Yves David.

exchange: add "streaming all changes" to bundle2 pulling This is the beginning of client-side support for performing a stream clone using bundle2. The main bundle2 pull function checks whether to perform a streaming clone and outputs a message if so. While we have a duplicate message, it seems easier to have all the bundle2 console writing in one location and in an easy-to-read conditional block.

streamclone: move "streaming all changes" message location Previously, the message was printed after we requested and started processing the remote stream. This seems like something that we should do before calling out to the remote. Moving it also makes it easier to deal with the bundle2 implementation.

streamclone: move payload header generation into own function The stream clone data over the wire protocol contains a header line indicating total file count and data size. In bundle2, this metadata can be captured by a part parameter and doesn't need to be in the body. In preparation for bundle2, have generatev1() return the raw metadata and move the header generation to its own function.

streamclone: move payload header line consumption bundle2 parts have parameters. These are a logical place for "header" data such as the file count and payload size of stream clone data. In preparation for supporting stream clones with bundle2, move the consumption of the header line from the payload into maybeperformlegacystreamclone(). Note: the header line is still being emitted by generatev1(). This will be addressed in a subsequent patch.

streamclone: teach canperformstreamclone to be bundle2 aware We add an argument to canperformstreamclone() to return False if a bundle2 stream clone is available. This will enable the legacy stream clone step to no-op when a bundle2 stream clone is supported. The commented code will be made active when bundle2 supports streaming clone. This patch does foreshadow the introduction of the "stream" bundle2 capability and its "v1" sub-capability. The bundle2 capability mirrors the existing "stream" capability and is needed so clients know whether a server explicitly supports streaming clones over bundle2 (servers up to this point support bundle2 without streaming clone support). The sub-capability will denote which data formats and variations are supported. Currently, the value "v1" denotes the existing streaming clone data format, which I intend to reuse inside a bundle2 part. My intent is to eventually introduce alternate data formats that can be produced and consumed more efficiently. Having a sub-capability means we don't need to introduce a new top-level bundle2 capability when new formats are introduced. This doesn't really have any implications beyond making the capabilities namespace more organized.

streamclone: refactor canperformstreamclone to accept a pullop This isn't strictly necessary. But a lot of pull functionality accepts a pulloperation so extra state can be added easily. It also enables extensions to perform more powerful things.

exchange: expose bundle2 availability on pulloperation Like the previous patch, the value is cached and will prevent a function level import in streamclone.py.

exchange: expose bundle2 capabilities on pulloperation This adds a cache and makes accessing the capabilities slightly simpler, as you don't need to directly go through the bundle2 module. This will also help prevent a function-level import in streamclone.py. This patch arguably isn't necessary. But I think it makes things slightly nicer.

keyword: make restrict mode False while updating files for rollback This is a preparation for using 'repo.rollback()' instead of aborting a current running transaction for "shelve" and "unshelve". Before this patch, updating files as a part of 'repo.rollback()' overridden by keyword extension always follows 'restrict' mode of the command currently executed. "merge", "unshelve" and so on should be 'restrict'-ed, because keyword expansion may cause unexpected conflicts at merging while these commands. But, if 'repo.rollback()' is invoked while executing 'restrict'-ed commands, modified files in the working directory are marked as "CLEAN" unexpectedly by code path below: # 'lookup' below is True at updating modified files for rollback kwcmd = self.restrict and lookup # kwexpand/kwshrink : if kwcmd: self.repo.dirstate.normal(f) On the other hand, "rollback" command isn't 'restrict'-ed, because rollbacking itself doesn't imply merging. Therefore, disabling 'restrict' mode while updating files as a part of 'repo.rollback()' regardless of current 'restrict' mode should be reasonable.

streamclone: rename and document maybeperformstreamclone() Upcoming patches will introduce bundle2 based streaming clones. Add "legacy" to the function name and add a docstring clarifying the intent of the function.

streamclone: move applyremotedata() into maybeperformstreamclone() Future work around stream cloning will be implemented in a bundle2 world. This code will only be used in the legacy code path and doesn't need to be abstracted or extensible.

branchmap: move branch cache code out of streamclone.py This is low-level branch map and cache manipulation code. It deserves to live next to similar code in branchmap.py. Moving it also paves the road for multiple consumers, such as a bundle2 part handler that receives branch mappings from a remote. This is largely a mechanical move, with only variable names and indentation being changed.

streamclone: move streamin() into maybeperformstreamclone() streamin() only had a single consumer. And it always only ever will because it is strongly coupled with the current, soon-to-be-superseded-by-bundle2 functionality. The return value has been dropped because nobody was using it.

streamclone: refactor maybeperformstreamclone to take a pullop Just like all the other pull steps. Consistency is good. This seems a little excessive right now since maybeperformstreamclone is such a short function. This will be addressed in a subsequent patch.

demandimport: replace more references to _demandmod instances _demandmod instances may be referenced by multiple importing modules. Before this patch, the _demandmod instance only maintained a reference to its first consumer when using the "from X import Y" syntax. This is because we only created a single _demandmod instance (attached to the parent X module). If multiple modules A and B performed "from X import Y", we'd produce a single _demandmod instance "demandmod" with the following references: X.Y = <demandmod> A.Y = <demandmod> B.Y = <demandmod> The locals from the first consumer (A) would be stored in <demandmod1>. When <demandmod1> was loaded, we'd look at the locals for the first consumer and replace the symbol, if necessary. This resulted in state: X.Y = <module> A.Y = <module> B.Y = <demandmod> B's reference to Y wasn't updated and was still using the proxy object because we just didn't record that B had a reference to <demandmod> that needed updating! With this patch, we add support for tracking which modules in addition to the initial importer have a reference to the _demandmod instance and we replace those references at module load time. In the case of posix.py, this fixes an issue where the "encoding" module was being proxied, resulting in hundreds of thousands of __getattribute__ lookups on the _demandmod instance during dirstate operations on mozilla-central, speeding up execution by many milliseconds. There are likely several other operation that benefit from this change as well. The new mechanism isn't perfect: references in locals (not globals) may likely linger. So, if there is an import inside a function and a symbol from that module is used in a hot loop, we could have unwanted overhead from proxying through _demandmod. Non-global imports are discouraged anyway. So hopefully this isn't a big deal in practice. We could potentially deploy a code checker that bans use of attribute lookups of function-level-imported modules inside loops. This deficiency in theory could be avoided by storing the set of globals and locals dicts to update in the _demandmod instance. However, I tried this and it didn't work. One reason is that some globals are _demandmod instances. We could work around this, but it's a bit more work. There also might be other module import foo at play. The solution as implemented is better than what we had and IMO is good enough for the time being. It's worth noting that this sub-optimal behavior was made worse by the introduction of absolute_import and its recommended "from . import X" syntax for importing modules from the "mercurial" package. If we ever wrote performance tests, measuring the amount of module imports and __getattribute__ proxy calls through _demandmod instances would be something I'd have it check.

demandimport: refactor processfromitem This will match the next patch smaller and easier to read.

demandimport: consolidate code for processing items in fromlist This code was mostly duplicated. An upcoming patch will add more complexity, making the duplication harder to justify. Consolidate the code.

pager: recreate stdout to make it line-buffered We want to see partial command results as soon as possible. But the buffering mode of stdout (= pager's stdin) was set to fully-buffered because it isn't associated with a tty. So, this patch recreates new stdout object to force its buffering mode. Because two file objects are associated with the same stdout fd and their destructors will call close(), one of them must be closed carefully. Python expects that the stdout fd never be closed even after sys.stdout.close() [1], but newstdout has no such hack. So this patch calls newstdout.close() immediately before duplicating the original stdout fd to sys.stdout. operation sys.stdout newstdout fd --------------------- ---------- --------- -------- newstdout.close() open closed closed os.dup2(stdoutfd, ..) open closed open del sys.stdout closed closed open [1] [1]: https://hg.python.org/cpython/file/v2.7.10/Python/sysmodule.c#l1391

pager: rename variables of backup fds The next patch will add backup file objects, so the original variable names would be confusing.