tests/list-tree.py
author Arseniy Alekseyev <aalekseyev@janestreet.com>
Thu, 18 May 2023 17:18:54 +0100
changeset 50540 74d8a1b03960
parent 48966 6000f5b25c9b
permissions -rw-r--r--
rust: speed up zstd decompression by re-using the decompression context Admittedly, zstd is already pretty fast, but this change makes it a bit faster yet: it saves ~5% of time it takes to read our large repo. The actual motivating use case is treemanifest: in treemanifest we end up reading *lots* of small directories, and many of them need decompression, and there the saving for [rhg files] is >10%. (which also seems unreasonable, we should probably keep things uncompressed more)

import argparse
import os

ap = argparse.ArgumentParser()
ap.add_argument('path', nargs='+')
opts = ap.parse_args()


def gather():
    for p in opts.path:
        if not os.path.exists(p):
            return
        if os.path.isdir(p):
            yield p + os.path.sep
            for dirpath, dirs, files in os.walk(p):
                for d in dirs:
                    yield os.path.join(dirpath, d) + os.path.sep
                for f in files:
                    yield os.path.join(dirpath, f)
        else:
            yield p


print('\n'.join(sorted(gather(), key=lambda x: x.replace(os.path.sep, '/'))))