revlog: move revision verification out of verify File revision verification is performing low-level checks of file storage, namely that flags are appropriate and revision data can be resolved. Since these checks are somewhat revlog-specific and may not be appropriate for alternate storage backends, this commit moves those checks from verify.py to revlog.py. Because we're now emitting warnings/errors that apply to specific revisions, we taught the iverifyproblem interface to expose the problematic node and to report this node in verify output. This was necessary to prevent unwanted test changes. After this change, revlog.verifyintegrity() and file verify code in verify.py both iterate over revisions and resolve their fulltext. But they do so in separate loops. (verify.py needs to resolve fulltexts as part of calling renamed() - at least when using revlogs.) This should add overhead. But on the mozilla-unified repo: $ hg verify before: time: real 700.640 secs (user 585.520+0.000 sys 23.480+0.000) after: time: real 682.380 secs (user 570.370+0.000 sys 22.240+0.000) I'm not sure what's going on. Maybe avoiding the filelog attribute proxies shaved off enough time to offset the losses? Maybe fulltext resolution has less overhead than I thought? I've left a comment indicating the potential for optimization. But because it doesn't produce a performance regression on a large repository, I'm not going to worry about it. Differential Revision: https://phab.mercurial-scm.org/D4745

tests: de-flake test-narrow-debugrebuilddirstate.t If the dirstate gets written much later (usually 1-2 s, depending on FS) than the working copy file (there's only one), then the `hg debugdirstate` command will include a timestamp. There's nothing wrong with that, so we should just allow it. Differential Revision: https://phab.mercurial-scm.org/D4758

upgrade: use storageinfo() for obtaining storage metadata Let's switch to our new API for obtaining information about storage. This eliminates the last consumer of rawsize() and the opener proxy from the file storage interface! Differential Revision: https://phab.mercurial-scm.org/D4748

revlog: add method for obtaining storage info (API) We currently have a handful of methods on the file and manifest storage interfaces for obtaining metadata about storage. e.g. files() is used to obtain the files backing storage. rawsize() is to quickly compute the size of tracked revisions without resolving their fulltext. Code in upgrade and stream clone make heavy use of these methods. The existing APIs are generic and don't necessarily have the specialization that we need going forward. For example, files() doesn't distinguish between exclusive storage and shared storage. This makes stream clone difficult to implement when e.g. there may be a single file backing storage for multiple tracked paths. It also makes reporting difficult, as we don't know how many bytes are actually used by storage since we can't easily identify shared files. This commit implements a new method for obtaining storage metadata. It is designed to accept arguments specifying what metadata to request and to return a dict with those fields populated. We /could/ make each of these attributes a separate method. But this is a specialized API and I'm trying to avoid method bloat on the interfaces. There is also the possibility that certain callers will want to obtain multiple fields in different combinations and some backends may have performance issues obtaining all that data via separate method calls. Simple storage integration tests have been added. For now, we assume fields can't be "None" (ignoring the interface documentation). We can revisit this later. Differential Revision: https://phab.mercurial-scm.org/D4747

lfs: drop unused import A recent change dropped the last user of this module. Differential Revision: https://phab.mercurial-scm.org/D4744

filelog: drop _generaldelta attribute (API) With changegroup moving to emitrevisions(), this revlog-specific attribute is no longer used and can be deleted. Good riddance. Differential Revision: https://phab.mercurial-scm.org/D4727

revlog: drop emitrevisiondeltas() and associated functionality (API) emitrevisions() is the future! Differential Revision: https://phab.mercurial-scm.org/D4726

changegroup: port to emitrevisions() (issue5976) We now have a unified API for emitting revision data from a storage backend. It handles sorting nodes and the complicated delta versus revision decisions for us. This commit ports changegroup to that API. There should be no behavior changes for changegroups not using ellipsis. And lack of test changes seems to confirm that. There are some changes for ellipsis mode, however. Before, when sending an ellipsis revision, we would always send a fulltext revision (as opposed to a delta). There was a TODO tracking this open item. One of the things the emitrevisions() API does for us is figure out whether we can safely emit a delta. So, it is now possible for ellipsis revisions to be sent as deltas! (It does this by not assuming parent/ancestor revisions are available and tracking which revisions have been sent out.) Because we eliminated the list of revision delta request objects, performance has improved substantially: $ hg perfchangegroupchangelog before: ! wall 24.348077 comb 24.330000 user 24.140000 sys 0.190000 (best of 3) after: ! wall 18.245911 comb 18.240000 user 18.100000 sys 0.140000 (best of 3) That's a lot of overhead for creating a few hundred thousand Python objects! This is still a little slower than 4.7. Probably due to 23d582ca introducing a type for the revision/delta results. There is potentially room to optimize. But at some point we need to abstract storage in order to support alternate storage backends. Unfortunately that means using a Python data structure to represent results. And unfortunately there is overhead with every new Python object created. Differential Revision: https://phab.mercurial-scm.org/D4725

wireprotov2server: port to emitrevisions() We now have a proper storage API to request data on multiple revisions. We can drop it into wire protocol version 2 with minimal effort. The new API handles pretty much everything we were doing manually to build up the delta request. So we were able to delete a lot of code. As a bonus, wireprotov2 code is no longer accessing some low-level storage APIs. This includes the assumption that a node has an associated numeric revision number! This should make it drastically simpler to implement a server that doesn't have the concept of revision numbers. Differential Revision: https://phab.mercurial-scm.org/D4724

tests: use more complex file storage test The previous test was attempting to to test delta storage behavior. It didn't do a very good job at it because there was a good chance a delta wasn't being used in storage. Let's switch the test to yield a delta in storage so an upcoming change to delegate delta logic to storage has the desired effect. Differential Revision: https://phab.mercurial-scm.org/D4723

revlog: new API to emit revision data I recently refactored changegroup generation code to make it more storage agnostic. I made significant progress. But there is still a bit of work to be done. Specifically: * Changegroup code is looking at low-level storage attributes to influence sorting. Sorting should be done at the storage layer. * The linknode lookup and sorting code for ellipsis is very complicated. * Linknodes are just generally wonky because e.g. file storage doesn't know how to translate a linkrev to a changelog node. * We regressed performance when introducing the request-response objects. Having thought about this problem a bit, I think I've come up with a better interface for emitting revision deltas. This commit defines and implements that interface. See the docstring in repository.py for more info. This API adds 3 notable features over the previous one. First, it defers node ordering to the storage implementation in the common case but allows overriding as necessary. We have a facility for requesting an exact ordering (used in ellipsis mode). We have another facility for storage order (used for changelog). Second, we have an argument specifying assumptions about parents revisions. This can be used to force a fulltext revision when we don't know the receiver has a parent revision to delta against. Third, we can control whether revision data is emitted. This makes the API suitable as a generic "index data retrieval" API as well as for producing revision deltas - possibly in the same operation! The new API is much simpler: we no longer need a complicated "request" object to encapsulate the delta generation request. I'm optimistic this will restore performance loss associated with emitrevisiondeltas(). Storage unit tests for the new API have been implemented. Future commits will port existing consumers of emitrevisiondeltas() to the new API then remove emitrevisiondeltas(). Differential Revision: https://phab.mercurial-scm.org/D4722

changegroup: remove reordering control (BC) This logic - including the experimental bundle.reorder option - was originally added in a8e3931e3fb5 in 2011 and then later ported to changegroup.py. The intent of this option and associated logic is to control the ordering of revisions in deltagroups in changegroups. At the time it was implemented, only changegroup version 1 existed and generaldelta revlogs were just coming into the world. Changegroup version 1 requires that deltas be made against the last revision sent over the wire. Used with generaldelta, this created an impedance mismatch of sorts and resulted in changegroup producers spending a lot of time recomputing deltas. Revision reordering was introduced so outgoing revisions would be sent in "generaldelta order" and producers would be able to reuse internal deltas from storage. Later on, we introduced changegroup version 2. It supported denoting which revision a delta was against. So we no longer needed to sort outgoing revisions to ensure optimal delta generation from the producer. So, subsequent changegroup versions disabled reordering. We also later made the changelog not store deltas by default. And we also made the changelog send out deltas in storage order. Why we do this for changelog, I'm not sure. Maybe we want to preserve revision order across clones? It doesn't really matter for this commit. Fast forward to 2018. We want to abstract storage backends. And having changegroup code require knowledge about how deltas are stored internally interferes with that goal. This commit removes reordering control from changegroup generation. After this commit, the reordering behavior is: * The changelog is always sent out in storage order (no behavior change). * Non-changelog generaldelta revlogs are reordered to always be in DAG topological order (previously, generaldelta revlogs would be emitted in storage order for version 2 and 3 changegroups). * Non-changelog non-generaldelta revlogs are sent in storage order (no behavior change). * There exists no config option to override behavior. The big difference here is that generaldelta revlogs now *always* have their revisions sorted in DAG order before going out over the wire. This behavior was previously only done for changegroup version 1. Version 2 and version 3 changegroups disabled reordering because the interchange format supported encoding arbitrary delta parents, so reordering wasn't strictly necessary. I can think of a few significant implications for this change. Because changegroup receivers will now see non-changelog revisions in DAG order instead of storage order, the internal storage order of manifests and files may differ substantially between producer and consumer. I don't think this matters that much, since the storage order of manifests and files is largely hidden from users. Only the storage order of changelog matters (because `hg log` shows the changelog in storage order). I don't think there should be any controversy here. The reordering of revisions has implications for changegroup producers. Previously, generaldelta revlogs would be emitted in storage order. And in the common case, the internally-stored delta could effectively be copied from disk into the deltagroup delta. This meant that emitting delta groups for generaldelta revlogs would be mostly linear read I/O. This is desirable for performance. With us now reordering generaldelta revlog revisions in DAG order, the read operations may use more random I/O instead of sequential I/O. This could result in performance loss. But with the prevalence of SSDs and fast random I/O, I'm not too worried. (Note: the optimal emission order for revlogs is actually delta encoding order. But the changegroup code wasn't doing that before or after this change. We could potentially implement that in a later commit.) Changegroups in DAG order will have implications for receivers. Previously, receiving storage order might mean seeing a number of interleaved branches. This would mean long delta chains, sparse I/O, and possibly more fulltext revisions instead of deltas, blowing up storage storage. (This is the same set of problems that sparse revlogs aims to address.) With the producer now sending revisions in DAG order, the receiver also stores revisions in DAG order. That means revisions for the same DAG branch are all grouped together. And this should yield better storage outcomes. In other words, sending the reordered changegroup allows the receiver to have better storage order and for the producer to not propagate its (possibly sub-optimal) internal storage order. On the mozilla-unified repository, this change influences bundle generation: $ hg bundle -t none-v2 -a before: time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000) after: time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000) before: 7,150,228,967 bytes (uncompressed) after: 7,041,556,273 bytes (uncompressed) before: 1,669,063,234 bytes (zstd l=3) after: 1,628,598,830 bytes (zstd l=3) $ hg unbundle before: time: real 511.910 secs (user 466.750+0.000 sys 32.680+0.000) after: time: real 487.790 secs (user 443.940+0.000 sys 30.840+0.000) 00manifest.d size: source: 274,924,292 bytes before: 304,741,626 bytes after: 245,252,087 bytes .hg/store total file size: source: 2,649,133,490 before: 2,680,888,130 after: 2,627,875,673 We see the bundle size drop. That's probably because if a revlog internally isn't storing a delta, it will choose to delta against the last emitted revision. And on repos with interleaved branches (like mozilla-unified), the previous revision could be an unrelated branch and therefore be a large delta. But with this patch, the previous revision is likely p1 or p2 and a delta should be small. We also see the manifest size drop by ~50 MB. It's worth noting that the manifest actually *increased* in size by ~25 MB in the old strategy and decreased ~25 MB from its source in the new strategy. Again, my explanation for this is that the DAG ordering in the changegroup is resulting in better grouping of revisions in the receiver, which results in more compact delta chains and higher storage efficiency. Unbundle time also dropped. I suspect this is due to the revlog having to work less to compute deltas since the incoming deltas are more optimal. i.e. the receiver spends less time resolving fulltext revisions as incoming deltas bounce around between DAG branches and delta chains. We also see bundle generation time increase. This is not desirable. However, the regression is only significant on the original repository: if we generate a bundle from the repository created from the new, always reordered bundles, we're close to baseline (if not at it with expected noise): $ hg bundle -t none-v2 -a before (original): time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000) after (original): time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000) after (new repo): time: real 362.280 secs (user 260.300+0.000 sys 17.700+0.000) This regression is a bit worrying because it will impact serving canonical repositories (that don't have optimal internal storage unless they are reordered - possibly as part of running `hg debugupgraderepo`). However, this regression will only be noticed by very large changegroups. And I'm guessing/hoping that any repository that large is using clonebundles to mitigate server load. Again, sending DAG order isn't the optimal send order for servers: sending in storage-delta order is. But in order to enable storage-optimal send order, we'll need a storage API that handles sorting. Future commits will introduce such an API. Differential Revision: https://phab.mercurial-scm.org/D4721

filelog: drop index attribute (API) The previous commit removed the last consumer of the "index" attribute on the file storage interface. The index is an extremely low-level data structure that is revlog specific and isn't appropriate to expose as part of a generic storage API. There may be a market for an efficient data structure to obtain metadata on every revision for a file. But if there is, it should be designed using e.g. named attributes for lookup instead of a list-like of 8-tuples. Let's drop the attribute from filelog and remove the attribute from the file storage interface. Differential Revision: https://phab.mercurial-scm.org/D4720

upgrade: use rawsize() instead of revlog index The revlog index is a very low-level data structure and it shouldn't be exposed to the storage interface - at least not in its current form. upgrade.py is the only consumer of the index attribute on file storage in the repository. This commit rewrites that final consumer to use rawsize() instead of going through the index. This is actually the more proper API to use, as rawsize() will accurately report the size of revisions which have a negative size in the index. Differential Revision: https://phab.mercurial-scm.org/D4719

manifest: add rawsize() proxy (API) I'm not keen about doing this. But it unblocks efforts to remove "index" from the file storage interface. We will probably remove this once we have a better upgrade API in place. Differential Revision: https://phab.mercurial-scm.org/D4718

upgrade: report size of backing files, not internal storage size upgrade.py is the only consumer of filelog.index, which I'd like to eliminate from the file storage interface. This commit changes the upgrade code to report the storage size of files by looking at the size of the files backing its storage instead of looking at the index. I'm not convinced the approach in this patch will live very long because it is relying on low-level attributes like "opener" and "files," which may behave very differently on non-revlog storage. But the data is only used for reporting purposes and it does get us one step closer to eliminating "index." A side-effect of this change is we now report the size of the revlog index data - not just the revision data. I think this is more accurate. Differential Revision: https://phab.mercurial-scm.org/D4717

filelog: store filename directly on revlog instance This attribute is only used by LFS. It is used by one of the revlog flag processor functions, which gets an instance of the revlog - not the file storage type. So, it makes sense to store this attribute on the revlog instead of the filelog. With this change, I'm pretty confident that LFS is no longer directly accessing file storage interface members that are revlog centric. i.e. it gets us one step closer to eliminating revlog-centric APIs from the file storage interface! Differential Revision: https://phab.mercurial-scm.org/D4715

lfs: access revlog directly LFS is monkeypatching filelog.filelog and is then accessing various filelog attributes in the monkeypatched function. This is all fine. But some of the attributes being accessed by LFS are revlog centric and shouldn't be exposed on the file storage interface. This commit changes the monkeypatched functions to access proxied attributes on self._revlog instead of self. This should be safe to do because non-revlog repositories should not be using filelog instances: instead they should have a separate class to represent file storage. So it is reasonable for LFS to assume the _revlog attribute exists and points to a revlog. Differential Revision: https://phab.mercurial-scm.org/D4714

largefiles: automatically load largefiles extension when required (BC) This is very similar to what we just did for LFS but for largefiles. See recent commit messages for the rationale here. Differential Revision: https://phab.mercurial-scm.org/D4713

lfs: don't add extension to hgrc after clone or share (BC) Now that repository loading in core supports automatically loading the lfs extension when the "lfs" requirement is present, we no longer need to update the .hg/hgrc of newly-created repos to load the lfs extension! I'm marking this as BC because it is a change in behavior. But users should not notice unless they create an LFS repo with new Mercurial and then attempt to use it with an old versions that doesn't support automatic extension loading. Differential Revision: https://phab.mercurial-scm.org/D4712

localrepo: automatically load lfs extension when required (BC) If an unrecognized requirement is present (possibly due to an unloaded extension), the user will get an error message telling them to go to https://mercurial-scm.org/wiki/MissingRequirement for more info. And some requirements clearly map to known extensions shipped by Mercurial. This commit teaches repository loading to automatically map requirements to extensions. We implement support for loading the lfs extension when the "lfs" requirement is present. This behavior feels more user-friendly to me and I'm having trouble coming up with a compelling reason to not do it. The strongest argument I have against is that - strictly speaking - requirements are general repository features and there could be N providers of that feature. e.g. in the case of LFS, there could be another extension implementing LFS support. And the user would want to use this non-official extension rather than the built-in one. The way this patch implements things, the non-official extension could be missing and Mercurial would load the official lfs extension, leading to unexpected behavior. But this feels like a highly marginal use case to me and doesn't outweigh the user benefit of "it just works." If someone really wanted to e.g. use a custom LFS extension, they could prevent the built-in one from being loaded by either defining "extensions.lfs=/path/to/custom/extension" or "extensions.lfs=!", as the automatic extension loading only occurs if there is no config entry for that extension. Differential Revision: https://phab.mercurial-scm.org/D4711

lfs: add repository feature denoting the use of LFS Whether LFS is enabled seems like a useful feature to expose. This will also facilitate some future work around LFS feature compatibility. Differential Revision: https://phab.mercurial-scm.org/D4710

localrepo: define "features" on repository instances (API) There are a handful of attributes/methods on repository instances that describe the behavior of the repository. Furthermore, there is an unbound set of repository descriptors that we may wish to expose. For example, an extension may wish to add a descriptor and have monkeypatched functions look for the presence of an attribute before taking actions. This commit introduces a "features" mechanism to allow repositories to self-advertise an arbitrary set of strings that describe repository behavior or capabilities. We implement basic support for advertising a few features to give an idea of what I want to use this for. Differential Revision: https://phab.mercurial-scm.org/D4709

localrepo: support writing shared file (API) Now that we can create a shared repository via creation options, we can handle other special actions related to share at repo creation time as well. One of the things we do after creating a shared repository is write out a .hg/shared file containing the list of additional things to share. Of which only "bookmarks" is supported. We add a creation option to hold the set of additional items to share. If items are defined, we write out the .hg/shared file at repo creation time. As part of this, we no longer hold the repo lock when writing the file. I'm pretty sure we don't care about the tiny race condition window. I'm also pretty sure the reason we used the lock was because the vfs auditor on the repo instance complained otherwise. Since the repo creation code doesn't have an audited vfs, we don't need to appease it. Because we no longer need to tell the post share hook what items are shared, the "bookmarks" argument to that function has been dropped, incurring an API change. Differential Revision: https://phab.mercurial-scm.org/D4708

localrepo: support shared repo creation Previously, hg.share() had its own logic for creating a new repository on the filesystem. With the recent introduction of the createopts dict for passing options to influence repository creation, it is now possible to consolidate the repo creation code for both the normal and shared use cases. This commit teaches the repo creation code in localrepo to recognize when we're creating a shared repo and to act appropriately. Meaningful behavior should be identical. However, there are a few subtle changes: * The .hg/requires file is written out in sorted order (rather than having share-related requirements appended at end). * The .hg directory is created with notindexed=True when a shared repo is being created. Differential Revision: https://phab.mercurial-scm.org/D4707

localrepo: validate directories before creating any There is no meaningful change in behavior because wdir would already exist in the case where we raised RepoError. But I think the code is easier to read if we do all validation first then take actions with side-effects. Differential Revision: https://phab.mercurial-scm.org/D4706

localrepo: add missing join() Differential Revision: https://phab.mercurial-scm.org/D4705

revlog: use proper version comparison during verify Verify appears to want to compare the changelog's revlog version number with the version number of filelogs and error if they are different. But what it was actually doing was comparing the full 32-bit header integer, which contains 2 shorts: 1 for the revlog version number and 1 for feature flags. This commit tweaks the verification code so it only looks at the version number component of the header and emits a warning if they differ. The new code is more robust because it accounts for future revlog version numbers without them needing to be special cased. Differential Revision: https://phab.mercurial-scm.org/D4704

filelog: stop proxying checksize() (API) This was only used by verify code. And the check using it is now implemented as part of verifyintegrity(). The method is unused and is revlog-centric, which means it isn't appropriate for the file storage interface. So remove it. Differential Revision: https://phab.mercurial-scm.org/D4703

filelog: remove version attribute (API) This was only used by verify code. The check it was used for is now implemented as part of the verifyintegrity() implementation. The attribute is now unused, is revlog-specific, and isn't appropriate to be exposing on the file storage interface. So drop it. Differential Revision: https://phab.mercurial-scm.org/D4702

verify: start to abstract file verification Currently, the file storage interface has a handful of attributes that are exclusively or near-exclusively used by repo verification code. In order to support verification on non-revlog/alternate storage backends, we'll need to abstract verification so it can be performed in a storage-agnostic way. This commit starts that process. We establish a new verifyintegrity() method on revlogs and expose it to the file storage interface. Most of verify.verifier.checklog() has been ported to this new method. We need a way to represent verification problems. So we invent an interface to represent a verification problem, invent a revlog type to implement that interface, and use it. The arguments to verifyintegrity() will almost certainly change in the future, once more functionality is ported from the verify code. And the "revlogv1" version check is very hacky. (The code in verify is actually buggy because it is comparing the full 32-bit header integer instead of just the revlog version short. I'll likely fix this in a subsequent commit.) Differential Revision: https://phab.mercurial-scm.org/D4701

unionrepo: remove _constructmanifest() It is unused after a previous refactor. Spotted in D4641. Differential Revision: https://phab.mercurial-scm.org/D4700

merge with stable

encoding: remove unnecessary lambdas from _encodingfixers They have been unnecessary since 1a3a08b5d4d5 (encoding: remove workaround for locale.getpreferredencoding(), 2017-05-13). Also rename the variable since "fixer" sounds like a function. Differential Revision: https://phab.mercurial-scm.org/D4743

py3: cast exception to bytes In order to appease Python 3. Differential Revision: https://phab.mercurial-scm.org/D4733

py3: cast exception to bytes This was raising a TypeError on Python 3 and preventing most (all?) wireprotov2 commands from working. Differential Revision: https://phab.mercurial-scm.org/D4731

py3: remove a couple of superfluous calls to pycompat.rapply() These places can only be strings.

py3: byteify an inline python test extension # skip-blame for b'' prefixing

py3: conditionalize access to socketserver.ForkingMixIn This is no longer exported on platforms that don't support forking, as of 3.6. https://github.com/python/cpython/commit/aadff9bea61a2fc9f4cf0f213f0ee50fc54d6574

convert: fix a file descriptor leak test-check-code flagged this after I changed this line for something unrelated.

merge with stable

revlog: catch more specific exception in shortest() Since revlog._partialmatch() catches RevlogError coming from cext and re-raises AmbiguousPrefixLookupError, catching RevlogError here seems less correct. Differential Revision: https://phab.mercurial-scm.org/D4735

py3: update missing module list in test-check-py3-compat.t for Windows

py3: add b'' prefixes to wire protocol test Otherwise the CBOR encoder fails on Python 3 due to lacking support for encoding str/unicode. # skip-blame just some b'' prefixes Differential Revision: https://phab.mercurial-scm.org/D4734

py3: use pycompat.strkwargs() Otherwise we get an error attempting to dispatch a command with arguments because we're passing a dict with bytes keys. Differential Revision: https://phab.mercurial-scm.org/D4732

py3: ensure _start_response() is called with system string This was preventing HTTP 500's from being sent in Python 3. Differential Revision: https://phab.mercurial-scm.org/D4730

py3: convert arguments, cwd and env to native strings when spawning subprocess This keeps Windows happy.

py3: apply byteskwargs to contrib/perf

py3: un-byteify strings around os.system() and os.devnull in contrib/perf

py3: work around the lack of sys.maxint in contrib/perf

py3: switch contrib/perf from xrange to pycompat.xrange

py3: handle sysstr conversion around get/set attr in contrib/perf

py3: proxy posixfile objects to re-add a useful 'name' attribute on Windows This file object is used in the vfs layer, so there are many errors like this: ... File "mercurial\localrepo.py", line 2569, in savecommitmessage return self.pathto(fp.name[len(self.root) + 1:]) TypeError: 'int' object is not subscriptable It looks like the 'name' value is actually the fileno() value, and the documentation says the name parameter to PyFile_FromFd() is ignored. [1] I tried just assigning the attribute after osutil.posixfile() returns, but that crashes saying that it's read-only. [1] https://docs.python.org/3.6/c-api/file.html

py3: don't use os.getcwdb() on Windows to avoid DeprecationWarnings See also ac32685011a3.

py3: rename pycompat.getcwd() to encoding.getcwd() (API) We need to avoid os.getcwdb() on Windows to avoid DeprecationWarnings, and we need encoding.strtolocal() to encode the result of os.getcwd().

py3: whitelist two more passing tests Caught by the ratchet, but initially only the non-legacy path of test-clone-uncompressed.t was passing. That required the preceding patch. Differential Revision: https://phab.mercurial-scm.org/D4729

keepalive: be more careful about self._rbuf when calling super impls In Python 3, HTTPResponse implements read() in terms of readinto(), which was calling back into our readinto(), which duplicates self._rbuf if it's not empty. Before calling into super's read(), ensure self._rbuf is empty. Inheritance is bad, and undocumented self-use of your public API is one of many reasons. Differential Revision: https://phab.mercurial-scm.org/D4728

wireprotov2: teach changesetdata to fetch ancestors until depth For shallow clone, it is useful to specify a starting node and tell the server to send up to N ancestors from that starting point. This enables the server to perform the DAG walk without the client having to discover the base/stop node(s) first. This commit implements support for said queries on the changesetdata command. Differential Revision: https://phab.mercurial-scm.org/D4621

wireprotov2: allow multiple fields to follow revision maps The *data wire protocol commands emit a series of CBOR values. Because revision/delta data may be large, their data is emitted outside the map as a top-level bytestring value. Before this commit, we'd emit a single optional bytestring value after the revision descriptor map. This got the job done. But it was limiting in that we could only send a single field. And, it required the consumer to know that the presence of a key in the map implied the existence of a following bytestring value. This commit changes the encoding strategy so top-level bytestring values in the stream are explicitly denoted in a "fieldsfollowing" key. This key contains an array defining what fields that follow and the expected size of each field. By defining things this way, we can easily send N bytestring values without any ambiguity about their order. In addition, clients only need to know how to parse ``fieldsfollowing`` to know if extra values are present. Because this breaks backwards compatibility, we've bumped the version number of the wire protocol version 2 API endpoint. Differential Revision: https://phab.mercurial-scm.org/D4620

wireprotov2: advertise set of valid values for requestable fields changesetdata, manifestdata, and filedata all allow the caller to specify what data fields to request. Data fields are extensible and may evolve over time. In order to prevent clients from making requests for fields that are not available, the client needs to know what fields are available. This commit teaches the server to declare a set of "valid values" for wire protocol command arguments. That set of values is exposed in the command's capabilities descriptor. The changesetdata, manifestdata, and filedata commands all declare their set of available "fields." Differential Revision: https://phab.mercurial-scm.org/D4619