Mercurial > hg
annotate mercurial/pathutil.py @ 26623:5a95fe44121d
clonebundles: support for seeding clones from pre-generated bundles
Cloning can be an expensive operation for servers because the server
generates a bundle from existing repository data at request time. For
a large repository like mozilla-central, this consumes 4+ minutes
of CPU time on the server. It also results in significant network
utilization. Multiplied by hundreds or even thousands of clients and
the ensuing load can result in difficulties scaling the Mercurial server.
Despite generation of bundles being deterministic until the next
changeset is added, the generation of bundles to service a clone request
is not cached. Each clone thus performs redundant work. This is
wasteful.
This patch introduces the "clonebundles" extension and related
client-side functionality to help alleviate this deficiency. The
client-side feature is behind an experimental flag and is not enabled by
default.
It works as follows:
1) Server operator generates a bundle and makes it available on a
server (likely HTTP).
2) Server operator defines the URL of a bundle file in a
.hg/clonebundles.manifest file.
3) Client `hg clone`ing sees the server is advertising bundle URLs.
4) Client fetches and applies the advertised bundle.
5) Client performs equivalent of `hg pull` to fetch changes made since
the bundle was created.
Essentially, the server performs the expensive work of generating a
bundle once and all subsequent clones fetch a static file from
somewhere. Scaling static file serving is a much more manageable
problem than scaling a Python application like Mercurial. Assuming your
repository grows less than 1% per day, the end result is 99+% of CPU
and network load from clones is eliminated, allowing Mercurial servers
to scale more easily. Serving static files also means data can be
transferred to clients as fast as they can consume it, rather than as
fast as servers can generate it. This makes clones faster.
Mozilla has implemented similar functionality of this patch on
hg.mozilla.org using a custom extension. We are hosting bundle files in
Amazon S3 and CloudFront (a CDN) and have successfully offloaded
>1 TB/day in data transfer from hg.mozilla.org, freeing up significant
bandwidth and CPU resources. The positive impact has been stellar and
I believe it has proved its value to be included in Mercurial core. I
feel it is important for the client-side support to be enabled in core
by default because it means that clients will get faster, more reliable
clones and will enable server operators to reduce load without
requiring any client-side configuration changes (assuming clients are
up to date, of course).
The scope of this feature is narrowly and specifically tailored to
cloning, despite "serve pulls from pre-generated bundles" being a valid
and useful feature. I would eventually like for Mercurial servers to
support transferring *all* repository data via statically hosted files.
You could imagine a server that siphons all pushed data to bundle files
and instructs clients to apply a stream of bundles to reconstruct all
repository data. This feature, while useful and powerful, is
significantly more work to implement because it requires the server
component have awareness of discovery and a mapping of which changesets
are in which files. Full, clone bundles, by contrast, are much simpler.
The wire protocol command is named "clonebundles" instead of something
more generic like "staticbundles" to leave the door open for a new, more
powerful and more generic server-side component with minimal backwards
compatibility implications. The name "bundleclone" is used by Mozilla's
extension and would cause problems since there are subtle differences
in Mozilla's extension.
Mozilla's experience with this idea has taught us that some form of
"content negotiation" is required. Not all clients will support all
bundle formats or even URLs (advanced TLS requirements, etc). To ensure
the highest uptake possible, a server needs to advertise multiple
versions of bundles and clients need to be able to choose the most
appropriate from that list one. The "attributes" in each
server-advertised entry facilitate this filtering and sorting. Their
use will become apparent in subsequent patches.
Initial inspiration and credit for the idea of cloning from static files
belongs to Augie Fackler and his "lookaside clone" extension proof of
concept.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Fri, 09 Oct 2015 11:22:01 -0700 |
parents | 56b2bcea2529 |
children | 6d29ce250a3d |
rev | line source |
---|---|
25964
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
1 from __future__ import absolute_import |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
2 |
25964
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
3 import errno |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
4 import os |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
5 import posixpath |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
6 import stat |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
7 |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
8 from .i18n import _ |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
9 from . import ( |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
10 encoding, |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
11 error, |
25964
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
12 util, |
d740df4e96cf
pathutil: use absolute_import
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25660
diff
changeset
|
13 ) |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
14 |
23598
c02a05cc6f5e
pathauditor: check for codepoints ignored on OS X
Augie Fackler <raf@durin42.com>
parents:
21568
diff
changeset
|
15 def _lowerclean(s): |
c02a05cc6f5e
pathauditor: check for codepoints ignored on OS X
Augie Fackler <raf@durin42.com>
parents:
21568
diff
changeset
|
16 return encoding.hfsignoreclean(s.lower()) |
c02a05cc6f5e
pathauditor: check for codepoints ignored on OS X
Augie Fackler <raf@durin42.com>
parents:
21568
diff
changeset
|
17 |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
18 class pathauditor(object): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
19 '''ensure that a filesystem path contains no banned components. |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
20 the following properties of a path are checked: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
21 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
22 - ends with a directory separator |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
23 - under top-level .hg |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
24 - starts at the root of a windows drive |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
25 - contains ".." |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
26 - traverses a symlink (e.g. a/symlink_here/b) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
27 - inside a nested repository (a callback can be used to approve |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
28 some nested repositories, e.g., subrepositories) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
29 ''' |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
30 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
31 def __init__(self, root, callback=None): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
32 self.audited = set() |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
33 self.auditeddir = set() |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
34 self.root = root |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
35 self.callback = callback |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
36 if os.path.lexists(root) and not util.checkcase(root): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
37 self.normcase = util.normcase |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
38 else: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
39 self.normcase = lambda x: x |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
40 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
41 def __call__(self, path): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
42 '''Check the relative path. |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
43 path may contain a pattern (e.g. foodir/**.txt)''' |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
44 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
45 path = util.localpath(path) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
46 normpath = self.normcase(path) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
47 if normpath in self.audited: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
48 return |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
49 # AIX ignores "/" at end of path, others raise EISDIR. |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
50 if util.endswithsep(path): |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
51 raise error.Abort(_("path ends in directory separator: %s") % path) |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
52 parts = util.splitpath(path) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
53 if (os.path.splitdrive(path)[0] |
23598
c02a05cc6f5e
pathauditor: check for codepoints ignored on OS X
Augie Fackler <raf@durin42.com>
parents:
21568
diff
changeset
|
54 or _lowerclean(parts[0]) in ('.hg', '.hg.', '') |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
55 or os.pardir in parts): |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
56 raise error.Abort(_("path contains illegal component: %s") % path) |
23599
6dad422ecc5a
pathauditor: check for Windows shortname aliases
Matt Mackall <mpm@selenic.com>
parents:
23598
diff
changeset
|
57 # Windows shortname aliases |
6dad422ecc5a
pathauditor: check for Windows shortname aliases
Matt Mackall <mpm@selenic.com>
parents:
23598
diff
changeset
|
58 for p in parts: |
6dad422ecc5a
pathauditor: check for Windows shortname aliases
Matt Mackall <mpm@selenic.com>
parents:
23598
diff
changeset
|
59 if "~" in p: |
6dad422ecc5a
pathauditor: check for Windows shortname aliases
Matt Mackall <mpm@selenic.com>
parents:
23598
diff
changeset
|
60 first, last = p.split("~", 1) |
6dad422ecc5a
pathauditor: check for Windows shortname aliases
Matt Mackall <mpm@selenic.com>
parents:
23598
diff
changeset
|
61 if last.isdigit() and first.upper() in ["HG", "HG8B6C"]: |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
62 raise error.Abort(_("path contains illegal component: %s") |
23599
6dad422ecc5a
pathauditor: check for Windows shortname aliases
Matt Mackall <mpm@selenic.com>
parents:
23598
diff
changeset
|
63 % path) |
23598
c02a05cc6f5e
pathauditor: check for codepoints ignored on OS X
Augie Fackler <raf@durin42.com>
parents:
21568
diff
changeset
|
64 if '.hg' in _lowerclean(path): |
c02a05cc6f5e
pathauditor: check for codepoints ignored on OS X
Augie Fackler <raf@durin42.com>
parents:
21568
diff
changeset
|
65 lparts = [_lowerclean(p.lower()) for p in parts] |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
66 for p in '.hg', '.hg.': |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
67 if p in lparts[1:]: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
68 pos = lparts.index(p) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
69 base = os.path.join(*parts[:pos]) |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
70 raise error.Abort(_("path '%s' is inside nested repo %r") |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
71 % (path, base)) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
72 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
73 normparts = util.splitpath(normpath) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
74 assert len(parts) == len(normparts) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
75 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
76 parts.pop() |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
77 normparts.pop() |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
78 prefixes = [] |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
79 while parts: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
80 prefix = os.sep.join(parts) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
81 normprefix = os.sep.join(normparts) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
82 if normprefix in self.auditeddir: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
83 break |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
84 curpath = os.path.join(self.root, prefix) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
85 try: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
86 st = os.lstat(curpath) |
25660
328739ea70c3
global: mass rewrite to use modern exception syntax
Gregory Szorc <gregory.szorc@gmail.com>
parents:
25286
diff
changeset
|
87 except OSError as err: |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
88 # EINVAL can be raised as invalid path syntax under win32. |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
89 # They must be ignored for patterns can be checked too. |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
90 if err.errno not in (errno.ENOENT, errno.ENOTDIR, errno.EINVAL): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
91 raise |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
92 else: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
93 if stat.S_ISLNK(st.st_mode): |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
94 raise error.Abort( |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
95 _('path %r traverses symbolic link %r') |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
96 % (path, prefix)) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
97 elif (stat.S_ISDIR(st.st_mode) and |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
98 os.path.isdir(os.path.join(curpath, '.hg'))): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
99 if not self.callback or not self.callback(curpath): |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
100 raise error.Abort(_("path '%s' is inside nested " |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
101 "repo %r") |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
102 % (path, prefix)) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
103 prefixes.append(normprefix) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
104 parts.pop() |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
105 normparts.pop() |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
106 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
107 self.audited.add(normpath) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
108 # only add prefixes to the cache after checking everything: we don't |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
109 # want to add "foo/bar/baz" before checking if there's a "foo/.hg" |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
110 self.auditeddir.update(prefixes) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
111 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
112 def check(self, path): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
113 try: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
114 self(path) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
115 return True |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
116 except (OSError, error.Abort): |
20033
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
117 return False |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
118 |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
119 def canonpath(root, cwd, myname, auditor=None): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
120 '''return the canonical path of myname, given cwd and root''' |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
121 if util.endswithsep(root): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
122 rootsep = root |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
123 else: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
124 rootsep = root + os.sep |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
125 name = myname |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
126 if not os.path.isabs(name): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
127 name = os.path.join(root, cwd, name) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
128 name = os.path.normpath(name) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
129 if auditor is None: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
130 auditor = pathauditor(root) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
131 if name != rootsep and name.startswith(rootsep): |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
132 name = name[len(rootsep):] |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
133 auditor(name) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
134 return util.pconvert(name) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
135 elif name == root: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
136 return '' |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
137 else: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
138 # Determine whether `name' is in the hierarchy at or beneath `root', |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
139 # by iterating name=dirname(name) until that causes no change (can't |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
140 # check name == '/', because that doesn't work on windows). The list |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
141 # `rel' holds the reversed list of components making up the relative |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
142 # file name we want. |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
143 rel = [] |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
144 while True: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
145 try: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
146 s = util.samefile(name, root) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
147 except OSError: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
148 s = False |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
149 if s: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
150 if not rel: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
151 # name was actually the same as root (maybe a symlink) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
152 return '' |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
153 rel.reverse() |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
154 name = os.path.join(*rel) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
155 auditor(name) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
156 return util.pconvert(name) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
157 dirname, basename = util.split(name) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
158 rel.append(basename) |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
159 if dirname == name: |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
160 break |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
161 name = dirname |
f962870712da
pathutil: tease out a new library to break an import cycle from canonpath use
Augie Fackler <raf@durin42.com>
parents:
diff
changeset
|
162 |
25011
7d6a507a4c53
pathutil: hint if a path is root relative instead of cwd relative (issue4663)
Matt Harbison <matt_harbison@yahoo.com>
parents:
23602
diff
changeset
|
163 # A common mistake is to use -R, but specify a file relative to the repo |
7d6a507a4c53
pathutil: hint if a path is root relative instead of cwd relative (issue4663)
Matt Harbison <matt_harbison@yahoo.com>
parents:
23602
diff
changeset
|
164 # instead of cwd. Detect that case, and provide a hint to the user. |
7d6a507a4c53
pathutil: hint if a path is root relative instead of cwd relative (issue4663)
Matt Harbison <matt_harbison@yahoo.com>
parents:
23602
diff
changeset
|
165 hint = None |
7d6a507a4c53
pathutil: hint if a path is root relative instead of cwd relative (issue4663)
Matt Harbison <matt_harbison@yahoo.com>
parents:
23602
diff
changeset
|
166 try: |
25022
10bbdcd89164
canonpath: fix infinite recursion
Matt Mackall <mpm@selenic.com>
parents:
25011
diff
changeset
|
167 if cwd != root: |
10bbdcd89164
canonpath: fix infinite recursion
Matt Mackall <mpm@selenic.com>
parents:
25011
diff
changeset
|
168 canonpath(root, root, myname, auditor) |
10bbdcd89164
canonpath: fix infinite recursion
Matt Mackall <mpm@selenic.com>
parents:
25011
diff
changeset
|
169 hint = (_("consider using '--cwd %s'") |
10bbdcd89164
canonpath: fix infinite recursion
Matt Mackall <mpm@selenic.com>
parents:
25011
diff
changeset
|
170 % os.path.relpath(root, cwd)) |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
171 except error.Abort: |
25011
7d6a507a4c53
pathutil: hint if a path is root relative instead of cwd relative (issue4663)
Matt Harbison <matt_harbison@yahoo.com>
parents:
23602
diff
changeset
|
172 pass |
7d6a507a4c53
pathutil: hint if a path is root relative instead of cwd relative (issue4663)
Matt Harbison <matt_harbison@yahoo.com>
parents:
23602
diff
changeset
|
173 |
26587
56b2bcea2529
error: get Abort from 'error' instead of 'util'
Pierre-Yves David <pierre-yves.david@fb.com>
parents:
25964
diff
changeset
|
174 raise error.Abort(_("%s not under root '%s'") % (myname, root), |
25011
7d6a507a4c53
pathutil: hint if a path is root relative instead of cwd relative (issue4663)
Matt Harbison <matt_harbison@yahoo.com>
parents:
23602
diff
changeset
|
175 hint=hint) |
21568
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
176 |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
177 def normasprefix(path): |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
178 '''normalize the specified path as path prefix |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
179 |
23139
e53f6b72a0e4
spelling: fixes from proofreading of spell checker issues
Mads Kiilerich <madski@unity3d.com>
parents:
21568
diff
changeset
|
180 Returned value can be used safely for "p.startswith(prefix)", |
21568
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
181 "p[len(prefix):]", and so on. |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
182 |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
183 For efficiency, this expects "path" argument to be already |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
184 normalized by "os.path.normpath", "os.path.realpath", and so on. |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
185 |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
186 See also issue3033 for detail about need of this function. |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
187 |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
188 >>> normasprefix('/foo/bar').replace(os.sep, '/') |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
189 '/foo/bar/' |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
190 >>> normasprefix('/').replace(os.sep, '/') |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
191 '/' |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
192 ''' |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
193 d, p = os.path.splitdrive(path) |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
194 if len(p) != len(os.sep): |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
195 return path + os.sep |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
196 else: |
8dd17b19e722
subrepo: normalize path in the specific way for problematic encodings
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
20033
diff
changeset
|
197 return path |
25281
660b178f49c7
pathutil: add dirname and join functions
Durham Goode <durham@fb.com>
parents:
25022
diff
changeset
|
198 |
25286
127a11f705d9
pathutil: demote two local functions to just be forwards
Augie Fackler <augie@google.com>
parents:
25285
diff
changeset
|
199 # forward two methods from posixpath that do what we need, but we'd |
127a11f705d9
pathutil: demote two local functions to just be forwards
Augie Fackler <augie@google.com>
parents:
25285
diff
changeset
|
200 # rather not let our internals know that we're thinking in posix terms |
127a11f705d9
pathutil: demote two local functions to just be forwards
Augie Fackler <augie@google.com>
parents:
25285
diff
changeset
|
201 # - instead we'll let them be oblivious. |
127a11f705d9
pathutil: demote two local functions to just be forwards
Augie Fackler <augie@google.com>
parents:
25285
diff
changeset
|
202 join = posixpath.join |
127a11f705d9
pathutil: demote two local functions to just be forwards
Augie Fackler <augie@google.com>
parents:
25285
diff
changeset
|
203 dirname = posixpath.dirname |