Mercurial > hg
annotate tests/f @ 29830:92ac2baaea86
revlog: use an LRU cache for delta chain bases
Profiling using statprof revealed a hotspot during changegroup
application calculating delta chain bases on generaldelta repos.
Essentially, revlog._addrevision() was performing a lot of redundant
work tracing the delta chain as part of determining when the chain
distance was acceptable. This was most pronounced when adding
revisions to manifests, which can have delta chains thousands of
revisions long.
There was a delta chain base cache on revlogs before, but it only
captured a single revision. This was acceptable before generaldelta,
when _addrevision would build deltas from the previous revision and
thus we'd pretty much guarantee a cache hit when resolving the delta
chain base on a subsequent _addrevision call. However, it isn't
suitable for generaldelta because parent revisions aren't necessarily
the last processed revision.
This patch converts the delta chain base cache to an LRU dict cache.
The cache can hold multiple entries, so generaldelta repos have a
higher chance of getting a cache hit.
The impact of this change when processing changegroup additions is
significant. On a generaldelta conversion of the "mozilla-unified"
repo (which contains heads of the main Firefox repositories in
chronological order - this means there are lots of transitions between
heads in revlog order), this change has the following impact when
performing an `hg unbundle` of an uncompressed bundle of the repo:
before: 5:42 CPU time
after: 4:34 CPU time
Most of this time is saved when applying the changelog and manifest
revlogs:
before: 2:30 CPU time
after: 1:17 CPU time
That nearly a 50% reduction in CPU time applying changesets and
manifests!
Applying a gzipped bundle of the same repo (effectively simulating a
`hg clone` over HTTP) showed a similar speedup:
before: 5:53 CPU time
after: 4:46 CPU time
Wall time improvements were basically the same as CPU time.
I didn't measure explicitly, but it feels like most of the time
is saved when processing manifests. This makes sense, as large
manifests tend to have very long delta chains and thus benefit the
most from this cache.
So, this change effectively makes changegroup application (which is
used by `hg unbundle`, `hg clone`, `hg pull`, `hg unshelve`, and
various other commands) significantly faster when delta chains are
long (which can happen on repos with large numbers of files and thus
large manifests).
In theory, this change can result in more memory utilization. However,
we're caching a dict of ints. At most we have 200 ints + Python object
overhead per revlog. And, the cache is really only populated when
performing read-heavy operations, such as adding changegroups or
scanning an individual revlog. For memory bloat to be an issue, we'd
need to scan/read several revisions from several revlogs all while
having active references to several revlogs. I don't think there are
many operations that do this, so I don't think memory bloat from the
cache will be an issue.
author | Gregory Szorc <gregory.szorc@gmail.com> |
---|---|
date | Mon, 22 Aug 2016 21:48:50 -0700 |
parents | 318534bb5dfd |
children | c425b678df7c |
rev | line source |
---|---|
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
1 #!/usr/bin/env python |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
2 |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
3 """ |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
4 Utility for inspecting files in various ways. |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
5 |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
6 This tool is like the collection of tools found in a unix environment but are |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
7 cross platform and stable and suitable for our needs in the test suite. |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
8 |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
9 This can be used instead of tools like: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
10 [ |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
11 dd |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
12 find |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
13 head |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
14 hexdump |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
15 ls |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
16 md5sum |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
17 readlink |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
18 sha1sum |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
19 stat |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
20 tail |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
21 test |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
22 readlink.py |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
23 md5sum.py |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
24 """ |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
25 |
29160
0362605b82cf
py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
28046
diff
changeset
|
26 from __future__ import absolute_import |
0362605b82cf
py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
28046
diff
changeset
|
27 |
0362605b82cf
py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
28046
diff
changeset
|
28 import glob |
29233
318534bb5dfd
tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents:
29230
diff
changeset
|
29 import hashlib |
29160
0362605b82cf
py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
28046
diff
changeset
|
30 import optparse |
0362605b82cf
py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
28046
diff
changeset
|
31 import os |
0362605b82cf
py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
28046
diff
changeset
|
32 import re |
0362605b82cf
py3: make tests/f use absolute_import
Pulkit Goyal <7895pulkit@gmail.com>
parents:
28046
diff
changeset
|
33 import sys |
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
34 |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
35 def visit(opts, filenames, outfile): |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
36 """Process filenames in the way specified in opts, writing output to |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
37 outfile.""" |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
38 for f in sorted(filenames): |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
39 isstdin = f == '-' |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
40 if not isstdin and not os.path.lexists(f): |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
41 outfile.write('%s: file not found\n' % f) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
42 continue |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
43 quiet = opts.quiet and not opts.recurse or isstdin |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
44 isdir = os.path.isdir(f) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
45 islink = os.path.islink(f) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
46 isfile = os.path.isfile(f) and not islink |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
47 dirfiles = None |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
48 content = None |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
49 facts = [] |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
50 if isfile: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
51 if opts.type: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
52 facts.append('file') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
53 if opts.hexdump or opts.dump or opts.md5: |
26950
6686ae524f94
tests: make 'f' tool open files in binary mode when hexdumping
Matt Harbison <matt_harbison@yahoo.com>
parents:
23912
diff
changeset
|
54 content = file(f, 'rb').read() |
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
55 elif islink: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
56 if opts.type: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
57 facts.append('link') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
58 content = os.readlink(f) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
59 elif isstdin: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
60 content = sys.stdin.read() |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
61 if opts.size: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
62 facts.append('size=%s' % len(content)) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
63 elif isdir: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
64 if opts.recurse or opts.type: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
65 dirfiles = glob.glob(f + '/*') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
66 facts.append('directory with %s files' % len(dirfiles)) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
67 elif opts.type: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
68 facts.append('type unknown') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
69 if not isstdin: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
70 stat = os.lstat(f) |
23911
593a5cd709a2
tests: teach f not to report directory size
Matt Mackall <mpm@selenic.com>
parents:
23860
diff
changeset
|
71 if opts.size and not isdir: |
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
72 facts.append('size=%s' % stat.st_size) |
23912
7d0aa6269ece
tests: teach f not to report symlink mode bits
Matt Mackall <mpm@selenic.com>
parents:
23911
diff
changeset
|
73 if opts.mode and not islink: |
28046
742cf5b979ec
f: use modern octal number formatting
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
28044
diff
changeset
|
74 facts.append('mode=%o' % (stat.st_mode & 0o777)) |
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
75 if opts.links: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
76 facts.append('links=%s' % stat.st_nlink) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
77 if opts.newer: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
78 # mtime might be in whole seconds so newer file might be same |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
79 if stat.st_mtime >= os.stat(opts.newer).st_mtime: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
80 facts.append('newer than %s' % opts.newer) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
81 else: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
82 facts.append('older than %s' % opts.newer) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
83 if opts.md5 and content is not None: |
29233
318534bb5dfd
tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents:
29230
diff
changeset
|
84 h = hashlib.md5(content) |
318534bb5dfd
tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents:
29230
diff
changeset
|
85 facts.append('md5=%s' % h.hexdigest()[:opts.bytes]) |
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
86 if opts.sha1 and content is not None: |
29233
318534bb5dfd
tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents:
29230
diff
changeset
|
87 h = hashlib.sha1(content) |
318534bb5dfd
tests: make 'f' utility import hashlib unconditionally
Yuya Nishihara <yuya@tcha.org>
parents:
29230
diff
changeset
|
88 facts.append('sha1=%s' % h.hexdigest()[:opts.bytes]) |
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
89 if isstdin: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
90 outfile.write(', '.join(facts) + '\n') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
91 elif facts: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
92 outfile.write('%s: %s\n' % (f, ', '.join(facts))) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
93 elif not quiet: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
94 outfile.write('%s:\n' % f) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
95 if content is not None: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
96 chunk = content |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
97 if not islink: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
98 if opts.lines: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
99 if opts.lines >= 0: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
100 chunk = ''.join(chunk.splitlines(True)[:opts.lines]) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
101 else: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
102 chunk = ''.join(chunk.splitlines(True)[opts.lines:]) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
103 if opts.bytes: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
104 if opts.bytes >= 0: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
105 chunk = chunk[:opts.bytes] |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
106 else: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
107 chunk = chunk[opts.bytes:] |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
108 if opts.hexdump: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
109 for i in range(0, len(chunk), 16): |
28044
e4f70e79a65f
f: add whitespace around operator
FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
parents:
26950
diff
changeset
|
110 s = chunk[i:i + 16] |
23860
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
111 outfile.write('%04x: %-47s |%s|\n' % |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
112 (i, ' '.join('%02x' % ord(c) for c in s), |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
113 re.sub('[^ -~]', '.', s))) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
114 if opts.dump: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
115 if not quiet: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
116 outfile.write('>>>\n') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
117 outfile.write(chunk) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
118 if not quiet: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
119 if chunk.endswith('\n'): |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
120 outfile.write('<<<\n') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
121 else: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
122 outfile.write('\n<<< no trailing newline\n') |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
123 if opts.recurse and dirfiles: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
124 assert not isstdin |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
125 visit(opts, dirfiles, outfile) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
126 |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
127 if __name__ == "__main__": |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
128 parser = optparse.OptionParser("%prog [options] [filenames]") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
129 parser.add_option("-t", "--type", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
130 help="show file type (file or directory)") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
131 parser.add_option("-m", "--mode", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
132 help="show file mode") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
133 parser.add_option("-l", "--links", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
134 help="show number of links") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
135 parser.add_option("-s", "--size", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
136 help="show size of file") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
137 parser.add_option("-n", "--newer", action="store", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
138 help="check if file is newer (or same)") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
139 parser.add_option("-r", "--recurse", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
140 help="recurse into directories") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
141 parser.add_option("-S", "--sha1", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
142 help="show sha1 hash of the content") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
143 parser.add_option("-M", "--md5", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
144 help="show md5 hash of the content") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
145 parser.add_option("-D", "--dump", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
146 help="dump file content") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
147 parser.add_option("-H", "--hexdump", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
148 help="hexdump file content") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
149 parser.add_option("-B", "--bytes", type="int", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
150 help="number of characters to dump") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
151 parser.add_option("-L", "--lines", type="int", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
152 help="number of lines to dump") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
153 parser.add_option("-q", "--quiet", action="store_true", |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
154 help="no default output") |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
155 (opts, filenames) = parser.parse_args(sys.argv[1:]) |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
156 if not filenames: |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
157 filenames = ['-'] |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
158 |
bead0c7b4f68
tests: add 'f' tool for cross platform file operations in the tests
Mads Kiilerich <madski@unity3d.com>
parents:
diff
changeset
|
159 visit(opts, filenames, sys.stdout) |