contrib/packaging/hg-docker

util: lower water mark when removing nodes after cost limit reached

See the inline comment for the reasoning here. This is a pretty
common strategy for garbage collectors, other cache-like primtives.

The performance impact is substantial:

$ hg perflrucachedict --size 4 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 100
! inserts w/ cost limit
! wall 1.659181 comb 1.650000 user 1.650000 sys 0.000000 (best of 7)
! wall 1.722122 comb 1.720000 user 1.720000 sys 0.000000 (best of 6)
! mixed w/ cost limit
! wall 1.139955 comb 1.140000 user 1.140000 sys 0.000000 (best of 9)
! wall 1.182513 comb 1.180000 user 1.180000 sys 0.000000 (best of 9)

$ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000
! inserts
! wall 0.679546 comb 0.680000 user 0.680000 sys 0.000000 (best of 15)
! sets
! wall 0.825147 comb 0.830000 user 0.830000 sys 0.000000 (best of 13)
! inserts w/ cost limit
! wall 25.105273 comb 25.080000 user 25.080000 sys 0.000000 (best of 3)
! wall  1.724397 comb  1.720000 user  1.720000 sys 0.000000 (best of 6)
! mixed
! wall 0.807096 comb 0.810000 user 0.810000 sys 0.000000 (best of 13)
! mixed w/ cost limit
! wall 12.104470 comb 12.070000 user 12.070000 sys 0.000000 (best of 3)
! wall  1.190563 comb  1.190000 user  1.190000 sys 0.000000 (best of 9)

$ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000 --costlimit 10000 --mixedgetfreq 90
! inserts
! wall 0.711177 comb 0.710000 user 0.710000 sys 0.000000 (best of 14)
! sets
! wall 0.846992 comb 0.850000 user 0.850000 sys 0.000000 (best of 12)
! inserts w/ cost limit
! wall 25.963028 comb 25.960000 user 25.960000 sys 0.000000 (best of 3)
! wall  2.184311 comb  2.180000 user  2.180000 sys 0.000000 (best of 5)
! mixed
! wall 0.728256 comb 0.730000 user 0.730000 sys 0.000000 (best of 14)
! mixed w/ cost limit
! wall 3.174256 comb 3.170000 user 3.170000 sys 0.000000 (best of 4)
! wall 0.773186 comb 0.770000 user 0.770000 sys 0.000000 (best of 13)

$ hg perflrucachedict --size 100000 --gets 1000000 --sets 1000000 --mixed 1000000 --mixedgetfreq 90 --costlimit 5000000
! gets
! wall 1.191368 comb 1.190000 user 1.190000 sys 0.000000 (best of 9)
! wall 1.195304 comb 1.190000 user 1.190000 sys 0.000000 (best of 9)
! inserts
! wall 0.950995 comb 0.950000 user 0.950000 sys 0.000000 (best of 11)
! inserts w/ cost limit
! wall 1.589732 comb 1.590000 user 1.590000 sys 0.000000 (best of 7)
! sets
! wall 1.094941 comb 1.100000 user 1.090000 sys 0.010000 (best of 9)
! mixed
! wall 0.936420 comb 0.940000 user 0.930000 sys 0.010000 (best of 10)
! mixed w/ cost limit
! wall 0.882780 comb 0.870000 user 0.870000 sys 0.000000 (best of 11)

This puts us ~2x slower than caches without cost accounting. And for
read-heavy workloads (the prime use cases for caches), performance is
nearly identical.

In the worst case (pure write workloads with cost accounting enabled),
we're looking at ~1.5us per insert on large caches. That seems "fast
enough."

Differential Revision: https://phab.mercurial-scm.org/D4505

#!/usr/bin/env python3
#
# Copyright 2018 Gregory Szorc <[e-mail]>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.

import argparse
import pathlib
import shutil
import subprocess
import sys

def get_docker() -> str:
    docker = shutil.which('docker.io') or shutil.which('docker')
    if not docker:
        print('could not find docker executable')
        return 1

    try:
        out = subprocess.check_output([docker, '-h'], stderr=subprocess.STDOUT)

        if b'Jansens' in out:
            print('%s is the Docking System Tray; try installing docker.io' %
                  docker)
            sys.exit(1)
    except subprocess.CalledProcessError as e:
        print('error calling `%s -h`: %s' % (docker, e.output))
        sys.exit(1)

    out = subprocess.check_output([docker, 'version'],
                                  stderr=subprocess.STDOUT)

    lines = out.splitlines()
    if not any(l.startswith((b'Client:', b'Client version:')) for l in lines):
        print('`%s version` does not look like Docker' % docker)
        sys.exit(1)

    if not any(l.startswith((b'Server:', b'Server version:')) for l in lines):
        print('`%s version` does not look like Docker' % docker)
        sys.exit(1)

    return docker

def get_dockerfile(path: pathlib.Path, args: list) -> bytes:
    with path.open('rb') as fh:
        df = fh.read()

    for k, v in args:
        df = df.replace(b'%%%s%%' % k, v)

    return df

def build_docker_image(dockerfile: pathlib.Path, params: list, tag: str):
    """Build a Docker image from a templatized Dockerfile."""
    docker = get_docker()

    dockerfile_path = pathlib.Path(dockerfile)

    dockerfile = get_dockerfile(dockerfile_path, params)

    print('building Dockerfile:')
    print(dockerfile.decode('utf-8', 'replace'))

    args = [
        docker,
        'build',
        '--build-arg', 'http_proxy',
        '--build-arg', 'https_proxy',
        '--tag', tag,
        '-',
    ]

    print('executing: %r' % args)
    subprocess.run(args, input=dockerfile, check=True)

def command_build(args):
    build_args = []
    for arg in args.build_arg:
        k, v = arg.split('=', 1)
        build_args.append((k.encode('utf-8'), v.encode('utf-8')))

    build_docker_image(pathlib.Path(args.dockerfile),
                       build_args,
                       args.tag)

def command_docker(args):
    print(get_docker())

def main() -> int:
    parser = argparse.ArgumentParser()

    subparsers = parser.add_subparsers(title='subcommands')

    build = subparsers.add_parser('build', help='Build a Docker image')
    build.set_defaults(func=command_build)
    build.add_argument('--build-arg', action='append', default=[],
                        help='Substitution to perform in Dockerfile; '
                             'format: key=value')
    build.add_argument('dockerfile', help='path to Dockerfile to use')
    build.add_argument('tag', help='Tag to apply to created image')

    docker = subparsers.add_parser('docker-path', help='Resolve path to Docker')
    docker.set_defaults(func=command_docker)

    args = parser.parse_args()

    return args.func(args)

if __name__ == '__main__':
    sys.exit(main())